Question

I have to write a rails task for getting files from s3 bucket but my bucket have more than 1000 object.

.
#Connection codes and configures
.
bucket = AWS::S3::Bucket.find('my_bucket')
puts bucket.size
# => 1000

this code just give me 1000 objects 🙁
how can i get my all objects from s3 bucket ?

Answer 1

As stated in the S3 developer documentation:

To manage large result sets, Amazon S3 uses pagination to split them
into multiple responses. Each list keys response returns a page of up
to 1,000 keys with an indicator indicating if the response is
truncated. You send a series of list keys requests until you have
received all the keys.

The response to a REST GET Bucket operation contains the IsTruncated element which plays the role of the above mentioned indicator.

To retrieve the next set of results, using the AWS::S3 library, use the last key from the current page of results as the marker in your next request.