im running a multi tenant GAE app where each tenant could have from a few 1000 to 100k documents.
at this moment im trying to make a MVC javascript client app (the admin part of my app with spine.js) and i need CRUD endpoints and the ability to get a big amount of serialized objects at once. for this specific job appengine is way to slow. i tried to store serialized objects in the blobstore but between reading/writing and updating stuff to the blobstore it takes too much time and the app gets really slow.
i thought of using a nosql db on an external machine to do these operations over appengine.
a few options would be mongodb, couchdb or redis. but i am not sure about how good they perform with that much data and concurrent requests/inserts from different tenants.
lets say i have 20 tenants and each tenant has 50k docs. are these dbs capable to handle this load?
is this even the right way to go?
Why not use the much faster regular appengine datastore instead of blobstore? Simply store your documents in regular entities as Blob property. Just make sure the entity size doesn’t exceed 1 MB in which case you have to split up your data into more then one entity. I run an application whith millions of large Blobs that way.
To further speed up things use memcache or even in-memory cache. Consider fetching your entites with eventual consistency which is MUCH faster. Run as many database ops in parallel as possible using either bulk operations or the async API.