Question

I have a ~2MB file that my Google AppEngine server must use (not serve) as part of a computation for a service request.

That is, a client makes a particular request, my GAE server must first get the data from this ~2MB file, do some computations using this data, then serve a small response back to the client.

Where best do I store this data so that it can be quickly read and used by the server in the computation?

Answer 1

If the following assumptions hold true

• the file is not going to require updates outside of appengine code updates
• that the file is read only

Then deploy the file with your code and read the file into memory during startup (ideally using warmup requests) and just operate on it from memory. If you code has to have file based semantics to access the data (read,seek, etc) then read the file contents and wrap it in StringIO.

You will need to assign the value read from the file to a module level variable, that way whenever you get a new request you can just get the files contents by importing the module and referencing the name. ie. mymodule.filecontents