I am developing a Java application using Google App Engine that depends on a largish dataset to be present. Without getting into specifics of my application, I’ll just state that working with a small subset of the data is simply not practical. Unfortunately, at the time of this writing, the Google App Engine for Java development server stores the entire datastore in memory. According to Ikai Lan:
The development server datastore stub is an in memory Map that is persisted
to disk.
I simply cannot import my entire dataset into the development datastore without running into memory problems. Once the application is pushed into Google’s cloud and uses BigTable, there is no issue. But deployment to the cloud takes a long time making development cycles kind of painful. So developing this way is not practical.
I’ve noticed the Google App Engine for Python development server has an option to use SQLite as the backend datastore which I presume would solve my problem.
dev_appserver.py –use_sqlite
But the Java development server includes no such option (at least not documented). What is the best way to get a large dataset working with the Google App Engine Java development server?
There’s no magic solution – the only datastore stub for the Java API, currently, is an in-memory one. Short of implementing your own disk-based stub, your only options are to find a way to work with a subset of data for testing, or do your development on appspot.