Question

I want to create a very very large dictionary, and I’d like to store it on disk so as not to kill my memory. Basically, my needs are a cross between cPickle and the dict class, in that it’s a class that Python treats like a dictionary, but happens to live on the disk.

My first thought was to create some sort of wrapper around a simple MySQL table, but I have to store types in the entries of the structure that MySQL can’t even hope to support out of the box.

Answer 1

The simplest way is the shelve module, which works almost exactly like a dictionary:

import shelve
myshelf = shelve.open("filename") # Might turn into filename.db
myshelf["A"] = "First letter of alphabet"
print myshelf["A"]
# ...
myshelf.close()   # You should do this explicitly when you're finished

Note the caveats in the module documentation about changing mutable values (lists, dicts, etc.) stored on a shelf (you can, but it takes a bit more fiddling). It uses (c)pickle and dbm under the hood, so it will cheerfully store anything you can pickle.

I don’t know how well it performs relative to other solutions, but it doesn’t require any custom code or third party libraries.