Question

I have some legacy python code that using pypar and mpich2 to transmit data between multiple nodes. For some reason the data is kept in shelves and pypar wants to pickle the shelve to send it to other nodes. Pickling shelves is not allowed. So I want to convert from shelve to something I can send via pypar which pickles it before sending to other nodes. Any suggestions? Could I convert the shelve to json and pickle that?

Answer 1

Try converting the shelve to a dict() and then pickling it:

sdb = shelve.open('foo.db')
sdb['abc'] = {'a': 1, 'b': 2}
tmp = cPickle.dumps(dict(sdb), 2)
print cPickle.loads(tmp)
{'abc': {'a': 1, 'b': 2}}

Updated: (Response to a question in comments): A dict can copy the key-values from any mapping object, so if that object implements the methods keys and __getitem__ it will work. Since a shelve is a mapping object, the dict copy constructor can read the key-values from the shelve, then you can pickle the resulting dict and ship it to other hosts.

The example below shows the minimal interface dict requires to copy an object:

class Foo(object):

    def __init__(self):
        self.value = 0

    def keys(self):
        return ['a', 'b']

    def __getitem__(self, key):
        v = self.value
        self.value += 1
        return self.value

foo = Foo()
print dict(foo)
print dict(foo)

Output

{'a': 1, 'b': 2}
{'a': 3, 'b': 4}

Update: To add the contents of a dict back to a shelve, use update():

d = {'a': 1, 'b': 2}
s = shelve.open('foo.db')
s.update(d)
print s
{'a': 1, 'b': 2}