Question

I’m quite new to Amazon S3 and I’m having difficulty downloading large files from S3.

I have successfully downloaded a file that is 35MB every time, but when the size of the file is really big around 500 MB – 1.7GB the application crashes.

When trying on the simulator I would get can’t allocate region error after about 1GB of the download.

So then I tried it on the device. Now it seems to just crash at a random time and

no crash report is put in the device, therefor I’m having an issue debugging this problem.

At first I thought it was the device or even the simulator. But i’m not really sure.

Someone mentioned that S3 framework times out the downloads randomly occasionally for large files. Could this be the case?

I’m building the file by opening a data file seeking to the end, adding the data, then closing the file until the download is complete.

I’m not sure how to debug this problem.

Any help would be appreciated.

Thank you.

Answer 1

S3GetObjectRequest has NSMutableData* body where it appends all the data it downloads.

For large files as download progresses data is appended constantly, and it goes over the VM limit of 90MB and then app gets killed by iOS.

Quick and dirty workaround is to create your own S3GetObjectRequest and S3GetObjectResponse classes. AWS framework instantiates Response based on Class Name of Request (Class name of Request without last 7 chars “Request” and appends it with “Response”, and tries to instantiate new class of that name).

Then to override -(void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data to release body all the time.

This is quick and dirty fix simply because you still have constant data allocation, appending and then release. But it works when you are in a pinch. For my usage of downloading files of 150-700mb, this simple hack kept memory usage of the app at 2.55mb average, +/- 0.2mb.

As stated by the author of ASIHTTP library, it is no longer maintained.

Request – LargeFileS3GetObjectRequest.h

@interface LargeFileS3GetObjectRequest : S3GetObjectRequest
@end

Request – LargeFileS3GetObjectRequest.m

@implementation LargeFileS3GetObjectRequest
@end

Response – LargeFileS3GetObjectResponse.h

@interface LargeFileS3GetObjectResponse : S3GetObjectResponse
@end

Response – LargeFileS3GetObjectResponse.m

@implementation LargeFileS3GetObjectResponse

-(void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data
{
    // allow original implementation to send data to delegates
    [super connection:connection didReceiveData:data];

    // release body and set it to NULL so that underlying implementation doesn't
    // append on released object, but instead allocates new one
    [body release];
    body = NULL;
}
@end

Hope it helps.