I have a problem with an app that takes an XML feed, parses it, and stores the results into Core Data.
The problem only occurs on the very first run of the app when there is nothing in the store and the whole feed is parsed and stored.
The problem is simply that memory allocations build up and up until, on 50% of attempts it crashes the app, usually at around 10Mb.
The objects allocated seem to be CFData(store) objects and I can’t seem to find any way to force a release of them.
If you can get it to run once and successfully complete the parsing and save to core data then every subsequent launch is fine, memory usage never exceeds 2.5Mb
Here’s the general approach I have before we get into code:
Get the feed into an NSData object. Use NSFileManager to store it as a file.
Create a URL from the file path and give it to the parseXMLFile: method. Delegate is self.
On reaching parser:didStartElement:namespaceURI:qualifiedName:attributes: I create a dictionary to catch data from tags I need.
The parser:foundCharacters: method saves the contents of the tag to a string
The parser:didEndElement:… method then determines if the tag is something we need. If so it adds it to the dictionary if not it ignores it. Once it reaches the end of an item it calls a method to add it to the core data store.
From looking at sample code and other peoples postings here it seems there’s nothing in the general approach thats wrong.
The code is below, it comes from a larger context of a view controller but I omitted anything not directly related.
In terms of things I have tried:
The feed is XML, no option to convert to JSON, sorry.
It’s not the images being found and stored in the shared documents area.
Clues as to what it might be:
This is the entry I get from Instruments on the largest most numerous things allocated (256k per item):
Object Address Category Creation Time Live Size Responsible
Library Responsible Caller1 0xd710000 CFData
(store) 16024774912 • 262144 CFNetwork URLConnectionClient::clientDidReceiveData(_CFData
const*,
URLConnectionClient::ClientConnectionEventQueue*)
And here’s the code, edited for anonymity of the content 😉
-(void)parserDidStartDocument:(NSXMLParser *)feedParser { }
-(void)parserDidEndDocument:(NSXMLParser *)feedParser { }
-(void)parser:(NSXMLParser *)feedParser didStartElement:(NSString *)elementName namespaceURI:(NSString *)namespaceURI qualifiedName:(NSString *)qualifiedName attributes:(NSDictionary *)attributeDict
{
eachElement = elementName;
if ( [eachElement isEqualToString:@"item" ] )
{
//create a dictionary to store each item from the XML feed
self.currentItem = [[[NSMutableDictionary alloc] init] autorelease];
}
}
-(void)parser:(NSXMLParser *)feedParser foundCharacters:(NSString *)feedString
{
//Make sure that tag content doesn't line break unnecessarily
if (self.currentString == nil)
{
self.currentString = [[[NSMutableString alloc] init]autorelease];
}
[self.currentString appendString:feedString];
}
-(void)parser:(NSXMLParser *)feedParser didEndElement:(NSString *)elementName namespaceURI:(NSString *)namespaceURI qualifiedName:(NSString *)qName
{
eachElement = elementName;
NSString *tempString = [self.currentString stringByTrimmingCharactersInSet:[NSCharacterSet whitespaceAndNewlineCharacterSet]];
if ( [eachElement isEqualToString:@"title"] )
{
//skip the intro title
if (![tempString isEqualToString:@"Andy Panda UK"])
{
//add item to di citonary output to console
[self.currentItem setValue:tempString forKey:eachElement];
}
}
if ( [eachElement isEqualToString:@"link"] )
{
//skip the intro link
if (![tempString isEqualToString:@"http://andypanda.co.uk/comic"])
{
//add item to dicitonary output to console
[self.currentItem setValue:tempString forKey:eachElement];
}
}
if ( [eachElement isEqualToString:@"pubDate"] )
{
//add item to dicitonary output to console
[self.currentItem setValue:tempString forKey:eachElement];
}
if ( [eachElement isEqualToString:@"description"] )
{
if ([tempString length] > 150)
{
//trims the string to just give us the link to the image file
NSRange firstPart = [tempString rangeOfString:@"src"];
NSString *firstString = [tempString substringFromIndex:firstPart.location+5];
NSString *secondString;
NSString *separatorString = @"\"";
NSScanner *aScanner = [NSScanner scannerWithString:firstString];
[aScanner scanUpToString:separatorString intoString:&secondString];
//trims the string further to give us just the credits
NSRange secondPart = [firstString rangeOfString:@"title="];
NSString *thirdString = [firstString substringFromIndex:secondPart.location+7];
thirdString = [thirdString substringToIndex:[thirdString length] - 12];
NSString *fourthString= [secondString substringFromIndex:35];
//add the items to the dictionary and output to console
[self.currentItem setValue:secondString forKey:@"imageURL"];
[self.currentItem setValue:thirdString forKey:@"credits"];
[self.currentItem setValue:fourthString forKey:@"imagePreviewURL"];
//safety sake set unneeded objects to nil before scope rules kick in to release them
firstString = nil;
secondString = nil;
thirdString = nil;
}
tempString = nil;
}
//close the feed and release all the little objects! Fly be free!
if ( [eachElement isEqualToString:@"item" ] )
{
//get the date sorted
NSDateFormatter *formatter = [[NSDateFormatter alloc] init];
formatter.dateFormat = @"EEE, dd MMM yyyy HH:mm:ss ZZZZ";
NSDate *pubDate = [formatter dateFromString:[currentItem valueForKey:@"pubDate"]];
[formatter release];
NSArray *fetchedArray = [[NSArray alloc] initWithArray:[fetchedResultsController fetchedObjects]];
//build the string to make the image
NSString *previewURL = [@"http://andypanda.co.uk/comic/comics-rss/" stringByAppendingString:[self.currentItem valueForKey:@"imagePreviewURL"]];
if ([fetchedArray count] == 0)
{
[self sendToCoreDataStoreWithDate:pubDate andPreview:previewURL];
}
else
{
int i, matches = 0;
for (i = 0; i < [fetchedArray count]; i++)
{
if ([[self.currentItem valueForKey:@"title"] isEqualToString:[[fetchedArray objectAtIndex:i] valueForKey:@"stripTitle"]])
{
matches++;
}
}
if (matches == 0)
{
[self sendToCoreDataStoreWithDate:pubDate andPreview:previewURL];
}
}
previewURL = nil;
[previewURL release];
[fetchedArray release];
}
self.currentString = nil;
}
- (void)sendToCoreDataStoreWithDate:(NSDate*)pubDate andPreview:(NSString*)previewURL {
NSManagedObjectContext *context = [fetchedResultsController managedObjectContext];
NSEntityDescription *entity = [[fetchedResultsController fetchRequest] entity];
newManagedObject = [NSEntityDescription insertNewObjectForEntityForName:[entity name] inManagedObjectContext:context];
[newManagedObject setValue:[NSDate date] forKey:@"timeStamp"];
[newManagedObject setValue:[[currentItem valueForKey:@"title"] description] forKey:@"stripTitle"];
[newManagedObject setValue:[[currentItem valueForKey:@"credits"] description] forKey:@"stripCredits"];
[newManagedObject setValue:[[currentItem valueForKey:@"imageURL"] description] forKey:@"stripImgURL"];
[newManagedObject setValue:[[currentItem valueForKey:@"link"] description] forKey:@"stripURL"];
[newManagedObject setValue:pubDate forKey:@"stripPubDate"];
[newManagedObject setValue:@"NO" forKey:@"stripBookmark"];
//**THE NEW SYSTEM**
NSString *destinationPath = [(AndyPadAppDelegate *)[[UIApplication sharedApplication] delegate] applicationDocumentsDirectory];
NSString *guidPreview = [[NSProcessInfo processInfo] globallyUniqueString];
NSString *guidImage = [[NSProcessInfo processInfo] globallyUniqueString];
NSString *dpPreview = [destinationPath stringByAppendingPathComponent:guidPreview];
NSString *dpImage = [destinationPath stringByAppendingPathComponent:guidImage];
NSData *previewD = [NSData dataWithContentsOfURL:[NSURL URLWithString:previewURL]];
NSData *imageD = [NSData dataWithContentsOfURL:[NSURL URLWithString:[currentItem valueForKey:@"imageURL"]]];
//NSError *error = nil;
[[NSFileManager defaultManager] createFileAtPath:dpPreview contents:previewD attributes:nil];
[[NSFileManager defaultManager] createFileAtPath:dpImage contents:imageD attributes:nil];
//TODO: BETTER ERROR HANDLING WHEN COMPLETED APP
[newManagedObject setValue:dpPreview forKey:@"stripLocalPreviewPath"];
[newManagedObject setValue:dpImage forKey:@"stripLocalImagePath"];
[newManagedObject release];
before++;
[self.currentItem removeAllObjects];
self.currentItem = nil;
[self.currentItem release];
[xmlParserPool drain];
self.xmlParserPool = [[NSAutoreleasePool alloc] init];
}
[EDIT]
@Rog
I tried playing with NSOperation but no luck so far, the same me,ory build up with the same items, all about 256k each. Here’s the code:
- (void)sendToCoreDataStoreWithDate:(NSDate*)pubDate andPreview:(NSString*)previewURL
{
NSManagedObjectContext *context = [fetchedResultsController managedObjectContext];
NSEntityDescription *entity = [[fetchedResultsController fetchRequest] entity];
newManagedObject = [NSEntityDescription insertNewObjectForEntityForName:[entity name] inManagedObjectContext:context];
[newManagedObject setValue:[NSDate date] forKey:@"timeStamp"];
[newManagedObject setValue:[[currentItem valueForKey:@"title"] description] forKey:@"stripTitle"];
[newManagedObject setValue:[[currentItem valueForKey:@"credits"] description] forKey:@"stripCredits"];
[newManagedObject setValue:[[currentItem valueForKey:@"imageURL"] description] forKey:@"stripImgURL"];
[newManagedObject setValue:[[currentItem valueForKey:@"link"] description] forKey:@"stripURL"];
[newManagedObject setValue:pubDate forKey:@"stripPubDate"];
[newManagedObject setValue:@"NO" forKey:@"stripBookmark"];
NSString *destinationPath = [(AndyPadAppDelegate *)[[UIApplication sharedApplication] delegate] applicationDocumentsDirectory];
NSString *guidPreview = [[NSProcessInfo processInfo] globallyUniqueString];
NSString *guidImage = [[NSProcessInfo processInfo] globallyUniqueString];
NSString *dpPreview = [destinationPath stringByAppendingPathComponent:guidPreview];
NSString *dpImage = [destinationPath stringByAppendingPathComponent:guidImage];
//Create an array and send the contents off to be dispatched to an operation queue
NSArray *previewArray = [NSArray arrayWithObjects:dpPreview, previewURL, nil];
NSOperation *prevOp = [self taskWithData:previewArray];
[prevOp start];
//Create an array and send the contents off to be dispatched to an operation queue
NSArray *imageArray = [NSArray arrayWithObjects:dpImage, [currentItem valueForKey:@"imageURL"], nil];
NSOperation *imagOp = [self taskWithData:imageArray];
[imagOp start];
[newManagedObject setValue:dpPreview forKey:@"stripLocalPreviewPath"];
[newManagedObject setValue:dpImage forKey:@"stripLocalImagePath"];
//[newManagedObject release]; **recommended by stackoverflow answer
before++;
[self.currentItem removeAllObjects];
self.currentItem = nil;
[self.currentItem release];
[xmlParserPool drain];
self.xmlParserPool = [[NSAutoreleasePool alloc] init];
}
- (NSOperation*)taskWithData:(id)data
{
NSInvocationOperation* theOp = [[[NSInvocationOperation alloc] initWithTarget:self
selector:@selector(threadedImageDownload:)
object:data] autorelease];
return theOp;
}
- (void)threadedImageDownload:(id)data
{
NSURL *urlFromArray1 = [NSURL URLWithString:[data objectAtIndex:1]];
NSString *stringFromArray0 = [NSString stringWithString:[data objectAtIndex:0]];
NSData *imageFile = [NSData dataWithContentsOfURL:urlFromArray1];
[[NSFileManager defaultManager] createFileAtPath:stringFromArray0
contents:imageFile
attributes:nil];
//-=====0jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjmn **OLEG**
}
Take a look at AQXMLParser. There is a example app that shows how to integrate it with Core Data on a background thread. It’s a streaming parser so memory usage is minimal.
Set up a second MOC and fetched results controller for the UI and refetch when the parsing is complete to get the new data. (one option is to register for change notifications when the background MOC saves to merge between contexts)
For images, there are a number of third party categories on UIImageVew that support asynchronous downloads and placeholder images. So you would parse the image URL, store it in Core Data, then set the URL on the image view when that item is viewed. This way the user doesn’t need to wait for all the images to be downloaded.