Question

I have a large csv file in sdcard and i want to know the best way to send the data from my csv file to my hosted server through my android app. My app writes to the csv file for an hour and then at the end of every hour i run a service which reads this csv file and sends it to the server.

I was initially not writing anything to the csv file and sending each data as http post request. But it turned out that my app was showing too much of data usage for uploading.

So i switched to writing the data to a file for an hour and at the end of the hour read each line and convert each tuple to json objects , put it into a json array and then send that array as a json object via name value pair.

Problem 1:

My CSV file is too large so I get Out Of Memory Exception when i try to add more than 2000 json objects to the array and send like this (Problem occurs on line marked with **)

nameValuePairs.add(new BasicNameValuePair("myjson", json_array_obj.toString()));

Thread server = new Thread(){
    String line = null;
    public void run()
    {
        try{

            HttpClient httpclient1 = new DefaultHttpClient();
            HttpPost httppost1 = new HttpPost("http://www.example.com/filename.php");


            **httppost1.setEntity(new UrlEncodedFormEntity(nameValuePairs));**
            HttpResponse httpResponse = httpclient1.execute(httppost1);

            HttpEntity httpEntity = httpResponse.getEntity();
            line = EntityUtils.toString(httpEntity);

            Log.e("line" , line);
          }
          catch(Exception e)
          {
            Log.v("catch", "executed");
          } 

    }
};
server.start();

Logcat showed :

`12-28 13:01:18.828: E/AndroidRuntime(2690FATAL EXCEPTION: Thread-16132
 12-28 13:01:18.828: E/AndroidRuntime(26908): java.lang.OutOfMemoryError
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:94)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:145)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at java.lang.StringBuilder.append(StringBuilder.java:216)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at libcore.net.UriCodec.appendHex(UriCodec.java:212)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at libcore.net.UriCodec.appendHex(UriCodec.java:206)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at libcore.net.UriCodec.appendEncoded(UriCodec.java:109)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at libcore.net.UriCodec.encode(UriCodec.java:133)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at java.net.URLEncoder.encode(URLEncoder.java:57)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at org.apache.http.client.utils.URLEncodedUtils.encode(URLEncodedUtils.java:184)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:163)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at org.apache.http.client.entity.UrlEncodedFormEntity.<init>(UrlEncodedFormEntity.java:71)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at com.example.acgylo.CSVtoServer$1.run(CSVtoServer.java:168)8): `

I then found out that json array with large number of objects cannot be sent as namevaluepair as it will cause memory overflow.

So i want to know what is the best way to do this :

option 1 : Send data in batches of 500(i.e 500 lines of csv together).

Is there any other efficient way to achieve sending of 2k tuples(or more) from CSV via HTTP in android ?

Problem 2:

a) I wanted to know how to reduce the data usage of android application ?

b) What uses less data :

One http request with large json object sent as string OR Multiple http requests with smaller json objects?

Please help me find an efficient solution to the above problems. Any help is appreciated.

Thanks in advance.

Answer 1

As you’ve mentioned if you are sending large amounts of data from a mobile device to a server you can either

a) break up the data into pieces and if there is no dependencies between the pieces that could work

or

b) just compress the data into a zip file and send that over to the server. In most cases its probably easier just to modify the server to accept zip files and then do the regular processing.