Question

I’m using the aws-sdk gem and I’m trying to basically upload a really large file (it takes 2 days).

The file is uploaded in chunks but sometimes the script will crash and I would like to resume uploading (the next chunk).

During uploading we want to close close the multipart upload (so we can access the s3 data that has been uploaded so far).

Is it possible to add a part after the multipart upload is closed? (say the next day) basically resuming the upload?

Answer 1

Is it possible to add a part after the multipart upload is closed?
(say the next day) basically resuming the upload?

Not as such, but you can simulate the affect you desire.

Background

Uploading Objects Using Multipart Upload API allows you to upload a single object as a set of parts:

Each part is a contiguous portion of the object’s data. You can upload
these object parts independently and in any order. If transmission of
any part fails, you can retransmit that part without affecting other
parts. After all parts of your object are uploaded, Amazon S3
assembles these parts and creates the object. [emphasis mine]

This is further detailed in Complete Multipart Upload:

You first initiate the multipart upload and then upload all parts
using the Upload Parts operation (see Upload Part). […] Upon receiving this request, Amazon S3
concatenates all the parts in ascending order by part number to create
a new object. […] You must ensure the parts list is complete,
this operation concatenates the parts you provide in the list. […] [emphasis mine]

That is, the upload operation is finalized here and cannot be resumed by uploading another part. (Technically speaking the upload ID required for any operation on an initiated multipart upload is not available/valid anymore).

Solution

You can simple initiate a new multipart upload and upload your previously uploaded S3 object as the first part of this new multipart object by means of the Upload Part – Copy operation, which Uploads a part by copying data from an existing object as data source.