Question

I am new to Python and have been studying its fundementals for 3 months now, learning types, functions and algorithms. Now I started practiciging web app development with GAE framework.

Goal: have a very large dictionary, which can be accessed from all .py files throughout the web app without having it stored more than once or re-created each time when someone visits a URL of the app.

I want to render a simple DB table to a dictionary, with hopes of speed gain as it will be in memory.

Also I am planing on creating an in memory DAWG – TRIE

I don’t want this dictionary to be created each time a page is called, I want it to be stored in memory once, kept there and used and accessed by all sessions and if possible modified too.

How can I achieve this? Like a simple in memory DB but actually a Python dictionary?

Thank you.

Answer 1

I’ll have a stab at this and put forward the standard python shelve module. This provides a simple persistent dictionary that is backed by an dbm file.

There are a few caveats particularly relating to concurrency. It does provide some caching if writeback is enabled, but again there might be concerns re memory consumption. One other limitation is that keys must be strings.

Still, it might be worth a look. It definitely fulfils the description of “pure Python hack”.

Simple example:

import shelve

d = shelve.open('my_shelf')
for i in range(100000):
    d[str(i)] = 'Item %s' % i
d.close()
d = shelve.open('my_shelf')
>>> d['50000']
'Item 50000'