Question

I have a dictionary let say:

d = {"a": 1, "b": 2, "c": 3 }

All I want to do is to dump this dictionary into a file and then append
it with data produced by my program little by little. I don’t want to have
the dictionary data in memory during the running of the program. By the end I
want to be able to load this file into a python structure from another
program.

Is there a solution to do this using pickle or json or another python
module? I know that you can’t append to a dumped object in a pickle-file.

I know that I can do something like this:

d = pickle.load(open(fileName, 'rb'))
d.update(dictWithNewValues)
pickle.dump(d, open(fileName, 'wb'))

as described here. But I want to do it progressively.. I understand that this may be difficult to do that way because dictionary has unique keys, so to be append it the current keys should be known to avoid conflicts (entries with same keys..).

So, is there a way to do it in a list or another python object?

Answer 1

Assuming the keys of your dictionary are all strings, the simplest option I see is Python’s shelve module. A shelve behaves like a persistent dictionary, and any key can be updated.

Another option is the anydbm module, which also requires all values to be strings.