Question

I worked in android app that read large text file from server (20 MB) line by line ,
After each line insert into sqlite DB some data extracted from this line and so on .

the problem is, this application takes one hour approximately to finish reading this file .
note :
there are 400,000 line in this file .

how could i fast this operation to take 3 or 4 minutes ?
Thanks for your help .

    public void readArData() {
    try {
        URL url = new URL("http://10.0.2.2/xy.txt");
        BufferedReader in = new BufferedReader(new   
    inputStreamReader(url.openStream()));
        String str;
        while ((str = in.readLine()) != null) {
            // insert into Sqlite DB
        }
        in.close();
    } catch (MalformedURLException e) {
    } catch (IOException e) {
    }

}

Answer 1

Text files are highly compressible so use the gzip compression available on Android.

Other options include:

• splitting your dataset into multiple files and again using gzip download them separately

• seeding the database during the install and creating a synchronization method to update newer records and bring the mobile version up to date.\

• remove redundant data using an efficient database design and only downloading what’s required

UPDATE

To clarify:

1. zip the txt file on the server.
2. change your android code to download the zip file to local storage on device.
3. unzip the local copy to local storage.

You should now have a copy of your txt file on local storage so you can delete the local zip file.

4. Read the local txt file and insert database records.

The major differences in this approach are

a. that you are downloading less data
b. your download is not interrupted by sql inserts.

You can also try using a transaction to improve insert speed as described in Android SQLite database: slow insertion

db.beginTransaction();
for (entry : listOfEntries) {
    db.insert(entry);
}
db.setTransactionSuccessful();
db.endTransaction();