Question

The first time a user opens my app I need to download lots of data. I get all of this data from the server in JSON form. Depending on the user, these JSON files can be anywhere from 10kb – 30mb each, and there are 10+ of them.

I have no problems doing this when the JSONs have no more than 500 or so records, but like I said some have 10,000+ records and can be up to 30mb in size.

When downloading the larger JSONs, my app allocs a ton of memory, until I eventually get memory warnings and the app blows up.

It seems the CFData has to be the NSMutableData that I am building in didReceiveData. As I download a single JSON the CFData (store) rises– and when I begin parsing it stops rising.

How can I clear out that data before moving on to download & parse the next JSON?

As you can see below, there is 200mb of CFData (store) sitting around in memory:

—

Digging into the CFData doesn’t reveal much to help me:

Here is the code where I create operations to get these various JSONs–

- (void)checkForUpdates
{
    if(!_globals)
        _globals = [MySingleton sharedInstance];

    MyAFHTTPClient* client = [MyAFHTTPClient sharedClient];
    NSString* path = [NSString stringWithFormat:@"cache?deviceUID=%@&token=%@",[_globals getMacAddress], [_globals getToken]];
    NSURLRequest* request = [client requestWithMethod:@"GET" path:path parameters:nil];

     _currentEnvironment = [_globals getActiveEnvironment];       

    if(!_currentEnvironment.didDownloadDataValue)
        [self setupAndShowHUDinView:self.view withText:@"Checking for updates..."];

    AFJSONRequestOperation* operation = [AFJSONRequestOperation JSONRequestOperationWithRequest:request success:^(NSURLRequest *request, NSHTTPURLResponse *response, id JSON) {

        for(NSString *str in [JSON valueForKey:@"Views"])
        {
            //for each table i need to update, create a request to
            NSString* path = [NSString stringWithFormat:@"cache/%@/?deviceUID=%@&token=%@", str, @"00000-00000-0000-00001", [_globals getToken]];
            NSURLRequest* request = [client requestWithMethod:@"GET" path:path parameters:nil];
            AFJSONRequestOperation* operation2 = [AFJSONRequestOperation JSONRequestOperationWithRequest:request success:^(NSURLRequest *request, NSHTTPURLResponse *response, id JSON) 
            {
                    // even if this is comment this out and I do nothing but download, app crashes
                    //[self updateTable:str withJSON:JSON];
            } failure:nil];

            [operation2 setSuccessCallbackQueue:backgroundQueue];
            [client.operationQueue addOperation:operation2];
            numRequests++;
        }
    } failure:^(NSURLRequest *request, NSHTTPURLResponse *response, NSError *error, id JSON) {

    }];

    [operation start];
}

Is that CFData my JSONs responses that are in memory? I have attempted to clear my NSURLCache with no luck–

I have also looked a little into JSON streaming.. would that help me at all to reduce the amount of objects in memory?

Besides that the only other option I can think of is implementing some sort of paging..but I need my users to have all of the data

Any help/suggestions is greatly appreciated! Thanks!

—

EDIT

Ok I decided to use strip out AFNetworking and try to use native functions. In doing this I still get the same build of of CFData. I am using this subclass of NSOperation and I am creating/ adding my operations as follows now:

 NSOperationQueue *operationQueue;
 operationQueue = [[NSOperationQueue alloc]init];
 [operationQueue setMaxConcurrentOperationCount:1];
 for(NSString *str in [views valueForKey:@"Views"])
 {

     NSString* path = [NSString stringWithFormat:@"%@cache/%@/?deviceUID=%@&token=%@", _globals.baseURL, str, @"00000-00000-0000-00001", [_globals getToken]];

     LibSyncOperation *libSyncOperation = [[LibSyncOperation alloc] initWithURL:path];
     [operationQueue addOperation:libSyncOperation];

 }

Answer 1

I use the native JSON conversion functions with massive amounts of data with no memory problems.

I just use a standard NSURLConnection to download the NSData then do the following…

NSData *data = [NSURLConnection sendSynchronous...

// use NSDictionary or NSArray here
NSArray *array = [NSJSONSerialization JSONObjectWithData:data ...

Now deal with the objects.

No leaks as it’s a native function. A lot quicker than third parts frameworks.