The problem:
I need a device agnostic (e.g. HTML5) solution for storing and querying 250,000+ rows of data offline on a phone or tablet type device (e.g. iOS/Android). The idea being I have people working in remote areas without any cellular data connection and they need to run queries on this data and edit it while offline. Partly it will be geo-location based so if there are assets in the area they are in (uses GPS) then it will show those assets and let them be edited. When they return to the office they can sync the data back to the office server.
The reason that I’m approaching this from a web standard point of view is basically to save money and time by writing it once in HTML5 and then it works across multiple platforms rather than writing it twice in Objective C and Java. Also if you write something that’s platform agnostic then you’re not locked in and don’t go down with the ship when everyone moves to a newer one. We had a similar app written for Windows Mobile 5, now it’s useless as that platform is dead.
The offline database on the device needs to be:
- fast (responses under 2 seconds)
- potentially perform joins and have relationships with other tables able to query the database
- select data within a certain range or criteria e.g. by x & y co-ordinate based on the GPS reading.
Options:
HTML5 local storage:
Fine for small amounts of data <5,000 key/values, you can even store arrays/objects in it if you convert it to JSON.
Cons:
- For more than 10,000 rows even on a high end machine the browser will
slow to a crawl. - Can’t do complex queries on the data to pull out the data you want as you have to iterate through the whole storage and manually search for it.
- Limitations with the amount of storage that can be stored
Web SQL Database:
- Meets the requirements.
- Fast to run a query on 250,000 rows (1-2secs)
- Can create complex queries, joins etc
- Supported by Safari, Android and Opera so will work on iOS and Android devices
Cons:
- Deprecated as of November 2010
- Security flaw with cross-directory attacks. Not really an issue as we won’t be on shared hosting
IndexedDB:
Key/value object store similar to local storage except with indexes.
Cons:
- Slow to run a query on 200,000 rows (15-18secs)
- Can’t run complex queries
- Can’t do joins with other tables
- Not supported by main phone or tablet devices e.g. iPad/Android
- Standard not complete
This leaves the only option of implementing the deprecated Web SQL method which may only work for another year or so. IndexedDB and local storage are unusable at present.
I’m not sure how Mozilla and Microsoft got the Web SQL Database standard deprecated and why the W3C let it happen. Supposedly between them they have 77% of the desktop browser market. On advanced mobile devices Mozilla and Microsoft have nearly zero influence as Safari, Opera and Android have over 90% of the market share. How Mozilla & Microsoft can dictate which standard should be used in the mobile market which is where offline storage is most likely to be used doesn’t make any sense.
In the comments from Mozilla about why they wanted to go with IndexedDB instead are mainly about ‘developer aesthetics’ and they don’t like the idea of running SQL in JavaScript. I’m not buying it.
-
Currently the proposed standard is inferior and an extremely basic NoSQL implementation that is slow and doesn’t even support the advanced features people need in a database. There is a lot of boilerplate code to establish the database and get data out but they claim people will write some nice abstraction libraries over the top of it that will provide more advanced features. As of Oct 2011 they’re nowhere to be seen.
-
They’ve deprecated the existing Web SQL standard which actually works and is implemented in the main mobile/tablet browsers. Whereas their ‘new’ and ‘better’ standard is not available in the major mobile browsers.
-
What are we as developers supposed to use for the next 3-5 years which is when the IndexedDB specification might get around to being standardised, have more features, implemented in the main mobile/tablet browsers and there’s some nice libraries to make things easier?
The W3C should keep the Web SQL Database standard running in parallel and just fix the issues. It already has support for the major mobile platforms and it works pretty well. The fact that Mozilla and Microsoft as the two players with the most desktop browser share were able to get this standard scrapped is pretty dubious and could be seen as an attempt to hinder progress on the mobile web platforms until they are able to catch up and offer competing solutions against iOS/Safari and Android.
In conclusion does anyone have a solution for my problem that will work for iOS/Android for phone/tablet devices. Maybe a nice wrapper API that can use multiple database implementations in the background with querying capability and it lets you choose which database has priority. I’ve seen things like lawnchair but I’m pretty sure it only lets you use local storage by default and falls back to the to the others. I think I’d rather it used Web SQL (by default) then the slower options.
Any help for a solution much appreciated, thanks!
I would recommend checking out the JayData library, that actually has the exact purpose of creating a storage agnostic data access layer for mobile devices. JayData provides an abstraction layer with JavaScript Language Query (JSLQ) and JavaScript CRUD support and let’s you work on the exact same way with different offline and online data store types. JayData supports dealing with complex entities and also entity relationships either locally or remotely.
At the time of writing JayData supports the following stores or protocols: webSQL(sqLite)/IndexedDB/OData/YQL/FBQL.
Your particular problem with different systems providing different storage engines can be easily addressed with the provider fallback feature of JayData: it will use whatever storage layer it can find while still provides the same API toward the consumer code.
With regarding WebSQL being deprecated by 2012: at the time of writing it is WebSQL that still has a 95% device coverage including Samsung SmartTV and amazon Kindle. Check out kindle executing WebSQL unit tests with JayData.