Question

Is it possible to have growing files on amazon s3?

That is, can i upload a file that i when the upload starts don’t know the final size of. So that I can start writing more data to the file with at an specified offset.

for example write 1000 bytes in one go, and then in the next call continue to write to the file with offset 1001, so that the next bytes being written is the 1001 byte of the file.

Answer 1

Amazon S3 indeed allows you to do that by Uploading Objects Using Multipart Upload API:

Multipart upload allows you to upload a single object as a set of
parts. Each part is a contiguous portion of the object’s data. You can
upload these object parts independently and in any order. If
transmission of any part fails, you can retransmit that part without
affecting other parts. After all parts of your object are uploaded,
Amazon S3 assembles these parts and creates the object. […]

One of the listed advantages precisely addresses your use case, namely to Begin an upload before you know the final object size – You can upload an object as you are creating it.

This functionality is available by Using the REST API for Multipart Upload and all AWS SDKs as well as 3rd party libraries like boto (a Python package that provides interfaces to Amazon Web Services) do offer multipart upload support based on this API as well.