I’ve been struggling with this for a long time now, and after a long research, I think is time to post this question here. So here’s the issue. I’m working on an Android app that’s basically a DB port of a very large system for quick consulting. So I have an SQLite DB that I have to update via webservice calls.
To do this, I call some webservices using SOAP, and parse the response using SAXParser. But here is the problem. Each page or update takes way to long, because it takes aprox 20 seconds to parse a 500 item XML and update the DB. And that is a problem, because in the worst case scenario the app will make 2000 webservice calls.
There are two approches I thought about to do the XML parsing. The first one (let’s call it Approach1) is to parse the whole callback, save the information in an array of objects, and then update the DB via transaction. the second one (Approach2) is to update the db while I parse the XML. So, each time I finish parsing an item, I do a BD insertion.
Here is the code I use
Approach1
public class ContactsParser extends DefaultHandler
{
private String foundData = "";
private ContactObj[] contactObj;
private int objectLength = 3000;
private int totalContacts = 0;
private long initiationTime;
private Context dbContext;
public void setObjectLength(int oLength){
objectLength = oLength;
}
public void setContext(Context ctx) {
dbContext = ctx;
}
public synchronized void parse(InputStream is)
throws ParserConfigurationException, SAXException, IOException
{
SAXParserFactory spf = SAXParserFactory.newInstance();
SAXParser sp = spf.newSAXParser();
XMLReader xr = sp.getXMLReader();
xr.setContentHandler(this);
xr.parse(new InputSource(is));
}
@Override
public void characters(char ch[], int start, int length)
{
foundData += new String(ch, start, length);
}
@Override
public void startDocument() throws SAXException
{
//Initialize the array of objects with the length of the Webservice page.
totalContacts = 0;
initiationTime = System.currentTimeMillis();
contactObj = new ContactObj[objectLength];
}
@Override
public void startElement(String namespaceURI, String localName,
String qName, Attributes atts) throws SAXException
{
foundData = "";
try{
if (totalContacts == 0) {
companiesObj[totalContacts] = new CompaniesObj();
}
}
catch (Exception e){
// Catch error
}
}
@Override
public void endElement(String namespaceURI, String elementName, String qName)
throws SAXException
{
if(qName.equals("contactItem")) {
if (totalContacts < objectLength){
contactObj[totalContacts] = new ContactObj();
}
/**
* <contactItem>
* <contactAtt1>value</contactAtt1>
* <contactAtt2>value</contactAtt2>
* </contactItem>
*/
try {
if(qName.equals("contactAtt1"))
{
contactObj[totalContacts].setContactAtt1(foundData);
}
else if(qName.equals("contactAtt2"))
{
contactObj[totalContacts].setContactAtt2(foundData);
}
catch(Exception e)
{
// catch error
}
}
@Override
public void endDocument() throws SAXException
{
try {
//DBHelper is a class that performs all DB related processes.
//getInstance sees if the instance of the object DBHelper.self exists. If not, it creates it.
DBHelper.getInstance(dbContext).db.beginTransaction();
for (int i = 0; i < totalContacts; i++){
DBHelper.self.sqLiteConctacts.insertRow(contactObj[i]);
}
DBHelper.self.db.setTransactionSuccessful();
} catch (Exception e){
// catch this exception
} finally {
DBHelper.self.db.endTransaction();
DBHelper.self.closeDB();
}
Logger.logMessage("Requried time to complete ContactParseing: "+(System.currentTimeMillis()-initiationTime));
Logger.logMessage("Parsing ended of ContactParser:" + totalContacts);
}
}
This approach has the great advantage of using transactions to update, but has at the cost of having to parse the whole document before I can do the update.
Approach2
public class ContactsParseAndInsert extends DefaultHandler
{
private String foundData = "";
private int totalContacts = 0;
private long initiationTime;
private Context dbContext;
// Contact attributes
private String contactAtt1 = "";
private String contactAtt2 = "";
public void setContext(Context ctx) {
dbContext = ctx;
}
public synchronized void parse(InputStream is)
throws ParserConfigurationException, SAXException, IOException
{
SAXParserFactory spf = SAXParserFactory.newInstance();
SAXParser sp = spf.newSAXParser();
XMLReader xr = sp.getXMLReader();
xr.setContentHandler(this);
xr.parse(new InputSource(is));
}
@Override
public void characters(char ch[], int start, int length)
{
foundData += new String(ch, start, length);
}
@Override
public void startDocument() throws SAXException
{
//Initialize the array of objects with the length of the Webservice page.
totalContacts = 0;
initializationTime = System.currentTimeMillis();
DBHelper.getInstance(dbContext);
Logger.logMessage("Parsing started of ContactParser With instant insert.");
}
@Override
public void startElement(String namespaceURI, String localName,
String qName, Attributes atts) throws SAXException
{
foundData = "";
}
@Override
public void endElement(String namespaceURI, String elementName, String qName)
throws SAXException
{
if(qName.equals("ContactItem")) {
DBHelper.self.sqLiteConctacts.insertRow(contactAtt1, contactAtt2);
}
/**
* <companiesObj>
* <contactAtt1>value</contactAtt1>
* <contactAtt2>value</contactAtt2>
* </companiesObj>
*/
try {
if(qName.equals("contactAtt1"))
{
contactAtt1 = foundData;
}
else if(qName.equals("contactAtt2"))
{
contactAtt2 = foundData;
}
catch(Exception e)
{
// catch error
}
}
@Override
public void endDocument() throws SAXException
{
DBHelper.self.closeDB();
Logger.logMessage("Requried time to complete ContactParseing: "+(System.currentTimeMillis()-initiationTime));
Logger.logMessage("Parsing ended of ContactParser:" + totalContacts);
}
}
What’s great of this second approach is that I can have the XML be as long as I want, and I won’t have any memory issues. But with this it’s impossible to have DB transactions.
Another important thing, is that all this is happening in a Service running it’s own thread and that communicates with my Activities via Messengers.
They both do their work correctly, but I’d rather use the second approach. Can anybody tell me how can I speed up this?? Does anybody know a better way to do this? The following is what I get as a result in my log. (all those dalvikvm pauses concern me a little)
09-13 12:31:42.653: I/MyFaultyApp(6477): Parsing started of ContactParser With instant insert.
09-13 12:31:42.653: I/MyFaultyApp(6477): --------------------------------------------------------
09-13 12:31:43.563: D/dalvikvm(6477): GC_CONCURRENT freed 418K, 29% free 2458K/3459K, paused 11ms+1ms, total 32ms
09-13 12:31:45.213: D/dalvikvm(6477): GC_CONCURRENT freed 427K, 30% free 2432K/3459K, paused 11ms+14ms, total 35ms
09-13 12:31:46.493: D/dalvikvm(6477): GC_CONCURRENT freed 389K, 30% free 2431K/3459K, paused 14ms+12ms, total 36ms
09-13 12:31:47.663: D/dalvikvm(6477): GC_CONCURRENT freed 411K, 30% free 2433K/3459K, paused 1ms+2ms, total 13ms
09-13 12:31:49.213: D/dalvikvm(6477): GC_CONCURRENT freed 420K, 30% free 2432K/3459K, paused 1ms+12ms, total 25ms
09-13 12:31:50.273: D/dalvikvm(6477): GC_CONCURRENT freed 415K, 30% free 2433K/3459K, paused 1ms+1ms, total 14ms
09-13 12:31:51.203: D/dalvikvm(6477): GC_CONCURRENT freed 410K, 30% free 2432K/3459K, paused 1ms+1ms, total 16ms
09-13 12:31:52.183: D/dalvikvm(6477): GC_CONCURRENT freed 415K, 30% free 2432K/3459K, paused 1ms+2ms, total 19ms
09-13 12:31:53.123: D/dalvikvm(6477): GC_CONCURRENT freed 399K, 30% free 2430K/3459K, paused 11ms+12ms, total 34ms
09-13 12:31:53.653: D/dalvikvm(6477): GC_CONCURRENT freed 392K, 30% free 2430K/3459K, paused 11ms+1ms, total 22ms
09-13 12:31:54.423: D/dalvikvm(6477): GC_CONCURRENT freed 407K, 30% free 2433K/3459K, paused 11ms+1ms, total 23ms
09-13 12:31:55.103: D/dalvikvm(6477): GC_CONCURRENT freed 408K, 30% free 2435K/3459K, paused 12ms+12ms, total 36ms
09-13 12:31:55.623: D/dalvikvm(6477): GC_CONCURRENT freed 413K, 30% free 2435K/3459K, paused 11ms+14ms, total 36ms
09-13 12:31:56.793: D/dalvikvm(6477): GC_CONCURRENT freed 410K, 30% free 2435K/3459K, paused 11ms+15ms, total 36ms
09-13 12:31:57.493: D/dalvikvm(6477): GC_CONCURRENT freed 416K, 30% free 2431K/3459K, paused 1ms+2ms, total 14ms
09-13 12:31:58.563: I/MyFaultyApp(6477): Requried time to complete ContactParseing: 15910
09-13 12:31:58.563: I/MyFaultyApp(6477): --------------------------------------------------------
09-13 12:31:58.563: I/MyFaultyApp(6477): Parsing ended of ContactParser:500
09-13 12:31:58.563: I/MyFaultyApp(6477): --------------------------------------------------------
09-13 12:31:58.573: D/dalvikvm(6477): GC_CONCURRENT freed 407K, 30% free 2439K/3459K, paused 13ms+3ms, total 28ms
09-13 12:32:00.503: I/MyFaultyApp(6477): Parsing started of ContactParser With instant insert.
09-13 12:32:00.503: I/MyFaultyApp(6477): --------------------------------------------------------
09-13 12:32:00.803: D/dalvikvm(6477): GC_CONCURRENT freed 399K, 29% free 2460K/3459K, paused 12ms+1ms, total 34ms
09-13 12:32:01.233: D/dalvikvm(6477): GC_CONCURRENT freed 440K, 30% free 2431K/3459K, paused 13ms+1ms, total 26ms
09-13 12:32:01.953: D/dalvikvm(6477): GC_CONCURRENT freed 404K, 30% free 2435K/3459K, paused 11ms+2ms, total 24ms
09-13 12:32:03.643: D/dalvikvm(6477): GC_CONCURRENT freed 413K, 30% free 2432K/3459K, paused 1ms+12ms, total 24ms
09-13 12:32:06.313: D/dalvikvm(6477): GC_CONCURRENT freed 386K, 29% free 2472K/3459K, paused 11ms+1ms, total 31ms
09-13 12:32:08.433: D/dalvikvm(6477): GC_CONCURRENT freed 457K, 30% free 2434K/3459K, paused 11ms+1ms, total 30ms
09-13 12:32:09.733: D/dalvikvm(6477): GC_CONCURRENT freed 423K, 30% free 2436K/3459K, paused 13ms+12ms, total 44ms
09-13 12:32:13.873: D/dalvikvm(6477): GC_CONCURRENT freed 405K, 30% free 2431K/3459K, paused 13ms+2ms, total 25ms
09-13 12:32:15.743: D/dalvikvm(6477): GC_CONCURRENT freed 411K, 30% free 2433K/3459K, paused 1ms+1ms, total 14ms
09-13 12:32:18.593: D/dalvikvm(6477): GC_CONCURRENT freed 412K, 30% free 2435K/3459K, paused 1ms+1ms, total 13ms
09-13 12:32:20.752: D/dalvikvm(6477): GC_CONCURRENT freed 386K, 29% free 2467K/3459K, paused 14ms+11ms, total 34ms
09-13 12:32:22.003: D/dalvikvm(6477): GC_CONCURRENT freed 445K, 30% free 2436K/3459K, paused 1ms+1ms, total 14ms
09-13 12:32:23.113: D/dalvikvm(6477): GC_CONCURRENT freed 421K, 30% free 2435K/3459K, paused 11ms+2ms, total 32ms
09-13 12:32:24.033: D/dalvikvm(6477): GC_CONCURRENT freed 385K, 30% free 2442K/3459K, paused 13ms+5ms, total 27ms
09-13 12:32:24.902: D/dalvikvm(6477): GC_CONCURRENT freed 415K, 30% free 2433K/3459K, paused 11ms+1ms, total 28ms
09-13 12:32:25.203: I/MyFaultyApp(6477): Requried time to complete ContactParseing: 24707
09-13 12:32:25.203: I/MyFaultyApp(6477): --------------------------------------------------------
09-13 12:32:25.203: I/MyFaultyApp(6477): Parsing ended of ContactParser:500
09-13 12:32:25.203: I/MyFaultyApp(6477): --------------------------------------------------------
Thank you and sorry for such a long post.
Use this guy XmlPullParser is more preferable.