I have an application which uses a Web Service to get data from SQL Server and exports it to an SQLite Database in my tablet. The thing is, the table has approximately 25,000 records. The code is working just fine when I limit it to, say, 100 records. But I am getting the following error when exporting ALL of the data:
02-13 13:36:31.734: E/AndroidRuntime(12821): FATAL EXCEPTION: AsyncTask #1
02-13 13:36:31.734: E/AndroidRuntime(12821): java.lang.RuntimeException: An error occured while executing doInBackground()
02-13 13:36:31.734: E/AndroidRuntime(12821): at android.os.AsyncTask$3.done(AsyncTask.java:299)
02-13 13:36:31.734: E/AndroidRuntime(12821): at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:352)
02-13 13:36:31.734: E/AndroidRuntime(12821): at java.util.concurrent.FutureTask.setException(FutureTask.java:219)
02-13 13:36:31.734: E/AndroidRuntime(12821): at java.util.concurrent.FutureTask.run(FutureTask.java:239)
02-13 13:36:31.734: E/AndroidRuntime(12821): at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:230)
02-13 13:36:31.734: E/AndroidRuntime(12821): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
02-13 13:36:31.734: E/AndroidRuntime(12821): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
02-13 13:36:31.734: E/AndroidRuntime(12821): at java.lang.Thread.run(Thread.java:856)
02-13 13:36:31.734: E/AndroidRuntime(12821): Caused by: java.lang.OutOfMemoryError
02-13 13:36:31.734: E/AndroidRuntime(12821): at java.lang.ThreadLocal$Values.initializeTable(ThreadLocal.java:244)
02-13 13:36:31.734: E/AndroidRuntime(12821): at java.lang.ThreadLocal$Values.rehash(ThreadLocal.java:325)
02-13 13:36:31.734: E/AndroidRuntime(12821): at java.lang.ThreadLocal$Values.cleanUp(ThreadLocal.java:254)
02-13 13:36:31.734: E/AndroidRuntime(12821): at java.lang.ThreadLocal$Values.getAfterMiss(ThreadLocal.java:438)
02-13 13:36:31.734: E/AndroidRuntime(12821): at java.lang.ThreadLocal.get(ThreadLocal.java:65)
02-13 13:36:31.734: E/AndroidRuntime(12821): at android.database.sqlite.SQLiteDatabase.getThreadSession(SQLiteDatabase.java:359)
02-13 13:36:31.734: E/AndroidRuntime(12821): at android.database.sqlite.SQLiteProgram.<init>(SQLiteProgram.java:58)
02-13 13:36:31.734: E/AndroidRuntime(12821): at android.database.sqlite.SQLiteStatement.<init>(SQLiteStatement.java:31)
02-13 13:36:31.734: E/AndroidRuntime(12821): at android.database.sqlite.SQLiteDatabase.compileStatement(SQLiteDatabase.java:992)
02-13 13:36:31.734: E/AndroidRuntime(12821): at android.database.DatabaseUtils.longForQuery(DatabaseUtils.java:799)
02-13 13:36:31.734: E/AndroidRuntime(12821): at android.database.sqlite.SQLiteDatabase.getVersion(SQLiteDatabase.java:862)
02-13 13:36:31.734: E/AndroidRuntime(12821): at android.database.sqlite.SQLiteOpenHelper.getDatabaseLocked(SQLiteOpenHelper.java:242)
02-13 13:36:31.734: E/AndroidRuntime(12821): at android.database.sqlite.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:164)
02-13 13:36:31.734: E/AndroidRuntime(12821): at com.example.shvalidation.DatabaseHandler.InsertBook(DatabaseHandler.java:75)
02-13 13:36:31.734: E/AndroidRuntime(12821): at com.example.shvalidation.MainMenuScreen$TestThread.doInBackground(MainMenuScreen.java:79)
02-13 13:36:31.734: E/AndroidRuntime(12821): at com.example.shvalidation.MainMenuScreen$TestThread.doInBackground(MainMenuScreen.java:1)
02-13 13:36:31.734: E/AndroidRuntime(12821): at android.os.AsyncTask$2.call(AsyncTask.java:287)
02-13 13:36:31.734: E/AndroidRuntime(12821): at java.util.concurrent.FutureTask.run(FutureTask.java:234)
02-13 13:36:31.734: E/AndroidRuntime(12821): ... 4 more
02-13 13:36:32.804: E/WindowManager(12821): Activity com.example.shvalidation.MainMenuScreen has leaked window com.android.internal.policy.impl.PhoneWindow$DecorView{412636d8 V.E..... R......D 0,0-772,216} that was originally added here
02-13 13:36:32.804: E/WindowManager(12821): android.view.WindowLeaked: Activity com.example.shvalidation.MainMenuScreen has leaked window com.android.internal.policy.impl.PhoneWindow$DecorView{412636d8 V.E..... R......D 0,0-772,216} that was originally added here
02-13 13:36:32.804: E/WindowManager(12821): at android.view.ViewRootImpl.<init>(ViewRootImpl.java:354)
02-13 13:36:32.804: E/WindowManager(12821): at android.view.WindowManagerGlobal.addView(WindowManagerGlobal.java:216)
02-13 13:36:32.804: E/WindowManager(12821): at android.view.WindowManagerImpl.addView(WindowManagerImpl.java:69)
02-13 13:36:32.804: E/WindowManager(12821): at android.app.Dialog.show(Dialog.java:281)
02-13 13:36:32.804: E/WindowManager(12821): at android.app.ProgressDialog.show(ProgressDialog.java:116)
02-13 13:36:32.804: E/WindowManager(12821): at android.app.ProgressDialog.show(ProgressDialog.java:99)
02-13 13:36:32.804: E/WindowManager(12821): at android.app.ProgressDialog.show(ProgressDialog.java:94)
02-13 13:36:32.804: E/WindowManager(12821): at com.example.shvalidation.MainMenuScreen$TestThread.onPreExecute(MainMenuScreen.java:49)
02-13 13:36:32.804: E/WindowManager(12821): at android.os.AsyncTask.executeOnExecutor(AsyncTask.java:586)
02-13 13:36:32.804: E/WindowManager(12821): at android.os.AsyncTask.execute(AsyncTask.java:534)
02-13 13:36:32.804: E/WindowManager(12821): at com.example.shvalidation.MainMenuScreen.onCreate(MainMenuScreen.java:30)
02-13 13:36:32.804: E/WindowManager(12821): at android.app.Activity.performCreate(Activity.java:5104)
02-13 13:36:32.804: E/WindowManager(12821): at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1080)
02-13 13:36:32.804: E/WindowManager(12821): at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2144)
02-13 13:36:32.804: E/WindowManager(12821): at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2230)
02-13 13:36:32.804: E/WindowManager(12821): at android.app.ActivityThread.access$600(ActivityThread.java:141)
02-13 13:36:32.804: E/WindowManager(12821): at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1234)
02-13 13:36:32.804: E/WindowManager(12821): at android.os.Handler.dispatchMessage(Handler.java:99)
02-13 13:36:32.804: E/WindowManager(12821): at android.os.Looper.loop(Looper.java:137)
02-13 13:36:32.804: E/WindowManager(12821): at android.app.ActivityThread.main(ActivityThread.java:5039)
02-13 13:36:32.804: E/WindowManager(12821): at java.lang.reflect.Method.invokeNative(Native Method)
02-13 13:36:32.804: E/WindowManager(12821): at java.lang.reflect.Method.invoke(Method.java:511)
02-13 13:36:32.804: E/WindowManager(12821): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:793)
02-13 13:36:32.804: E/WindowManager(12821): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:560)
02-13 13:36:32.804: E/WindowManager(12821): at dalvik.system.NativeStart.main(Native Method)
I am wondering if anyone knows anyway I can export these records. Yes, I am aware that exporting over 25,000 records to SQLite is ridiculous, but unfortunately this is the project I’ve been assigned. And as such, I have to find a way to do it.
EDIT: Adding the code that uses the Web Service and saves to the SQLite.
package com.example.shvalidation;
import java.util.ArrayList;
import java.util.List;
import org.apache.http.NameValuePair;
import org.apache.http.message.BasicNameValuePair;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import android.app.Activity;
import android.app.ProgressDialog;
import android.content.Intent;
import android.os.AsyncTask;
import android.os.Bundle;
import android.util.Log;
import android.view.Menu;
import android.view.View;
public class MainMenuScreen extends Activity {
//JSON Variables
JSONParser jsonParser = new JSONParser();
String pid;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.main_menu_layout);
new TestThread().execute();
}
@Override
public boolean onCreateOptionsMenu(Menu menu) {
// Inflate the menu; this adds items to the action bar if it is present.
getMenuInflater().inflate(R.menu.main_menu_layout, menu);
return true;
}
public void PlantToDome(View view) {
Intent intent = new Intent(this, SelectLocationScreen.class);
startActivity(intent);
}
//Código del Web Service
public class TestThread extends AsyncTask<Void, Void, Void> {
ProgressDialog dialog;
protected void onPreExecute() {
dialog = ProgressDialog.show(MainMenuScreen.this, "Loading", "Loading data, please wait..");
}
protected Void doInBackground(Void...args0) {
int success;
Book book = new Book();
DatabaseHandler handler = new DatabaseHandler(MainMenuScreen.this);
try {
// Building Parameters
List<NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("pid", pid));
// getting product details by making HTTP request
// Note that product details url will use GET request
JSONObject json = jsonParser.makeHttpRequest(
"http://192.168.1.102:8080/WSBook.php", "GET", params);
// check your log for json response
// json success tag
success = json.getInt("success");
if (success == 1) {
// successfully received product details
JSONArray productObj = json.getJSONArray("record"); // JSON Array
// get first product object from JSON Array
// JSONObject product = productObj.getJSONObject(1);
for (int i = 0; i < productObj.length(); i++) {
JSONObject record = productObj.getJSONObject(i);
book.setAoiName(record.getString("aoi_name"));
book.setExptName(record.getString("expt_name"));
book.setEuID(record.getInt("eu_id"));
book.setX(record.getInt("x"));
book.setY(record.getInt("y"));
book.setZ(record.getInt("z"));
book.setRaplo(record.getString("raplo"));
// Book book = new Book(record.getString("aoi_name"), record.getString("expt_name"), record.getInt("eu_id"),
// record.getInt("x"), record.getInt("y"), record.getInt("z"), record.getString("raplo"));
handler.InsertBook(book);
}
}
} catch (Exception e) {
Log.e("error", e.toString());
}
return null;
}
protected void onPostExecute(Void unused) {
dialog.dismiss();
}
}
}
EDIT #2: My JSON Parser class.
package com.example.shvalidation;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.util.List;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.utils.URLEncodedUtils;
import org.apache.http.impl.client.DefaultHttpClient;
import org.json.JSONException;
import org.json.JSONObject;
import android.util.Log;
public class JSONParser {
static InputStream is = null;
static JSONObject jObj = null;
static String json = "";
// constructor
public JSONParser() {
}
// function get json from url
// by making HTTP POST or GET method
public JSONObject makeHttpRequest(String url, String method,
List<NameValuePair> params) {
// Making HTTP request
try {
// check for request method
if(method == "POST"){
// request method is POST
// defaultHttpClient
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost(url);
httpPost.setEntity(new UrlEncodedFormEntity(params));
HttpResponse httpResponse = httpClient.execute(httpPost);
HttpEntity httpEntity = httpResponse.getEntity();
is = httpEntity.getContent();
}else if(method == "GET"){
// request method is GET
DefaultHttpClient httpClient = new DefaultHttpClient();
String paramString = URLEncodedUtils.format(params, "utf-8");
url += "?" + paramString;
HttpGet httpGet = new HttpGet(url);
HttpResponse httpResponse = httpClient.execute(httpGet);
HttpEntity httpEntity = httpResponse.getEntity();
is = httpEntity.getContent();
}
} catch (UnsupportedEncodingException e) {
Log.e("Unsupported Encoding", Log.getStackTraceString(e));
} catch (ClientProtocolException e) {
Log.e("Client Protocol", Log.getStackTraceString(e));
} catch (IOException e) {
Log.e("IO Exception", Log.getStackTraceString(e));
}
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(
is, "iso-8859-1"), 8);
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
json = sb.toString();
} catch (Exception e) {
Log.e("Buffer Error", "Error converting result " + e.toString());
}
// try parse the string to a JSON object
try {
jObj = new JSONObject(json);
} catch (JSONException e) {
Log.e("JSON Parser", "Error parsing data " + e.toString());
Log.e("JSON Parser", json);
}
// return JSON String
return jObj;
}
}
It would seem you are trying to pull all 25,000 rows’ data into memory first, then write them to SQLite (that is a guess, since you did not show any code, but I bet it’s a good guess). You will have to limit it to, say, 100 rows at a time. That is, read 100 rows from the web service, then write them to SQLite, then read 100 more rows, etc. You could get fancier with threads, but that should get you on the right path.
Edit: Thanks for adding your code. It looks like the culprit is this line:
That tries to read the text of the entire table in from the web service and jam it into a JSONObject, which your phone does not have enough memory to handle.
I see two solutions: