Question

I need to read a 5Mb file on Google App Engine (Python 2.7) and use it frequently.

As read a file in GAE is not difficult: Read a file on App Engine with Python?

The difficult part is storing it somewhere so I can access it as fast as possible frequently. It is 5MB so it exceeds the 1MB datastore item limit.

I consider to use Blobstore, but I am afraid it is not fast enough. Is reading Blobstore faster than reading a file?

I am thinking about put the whole file in memcache. Is it possible? Is the memcache big enough to store a 5MB file?

Just like on a computer, I need to put this file in memory not hard disk.

Any suggestions?

Thanks a lot!

Answer 1

If your file does not change, then you can simply put it in your project directory and have it served as a static file.

Now on to questions:

1. Blobstore will be fast enough because all requests (blobstore or user code) on GAE go through a transparent cache. You can simply set an appropriate Cache-control header on the blobstore response to have it cached.

2. Memcache max stored value is 1Mb. Also data in memcache can go away anytime, so you need to store data in a permanent storage anyway. Also, I doubt it would be faster because your frontend instance has to get data from Memcache and then serve it, while Blobstore serving works a bit differently (by intercepting responses and inserting data in body).

3. IMHO, the fastest would be if data is served via transparent cache.

Also, if you want to serve images, then you might want to use Image Service, as it seem to be faster then Blobstore.