Question

I have a process which runs and creates three dictionaries: 2 rather small, and 1 large.

I know I can store one dictionary like:

import cPickle as pickle
with open(filename, 'wb') as fp:
  pickle.dump(self.fitResults, fp)

What I’d like to do is store all 3 dictionaries in the same file, with the ability to load in the three dictionaries separately at another time. Something like

with open(filename, 'rb') as fp:
  dict1, dict2, dict3 = pickle.load(fp)

Or even better just load the first two dictionaries, and make it optional whether to load the third (large) one. Is this possible or should I go about this in a completely different way?

Answer 1

Sure, you just dump each one separately and then load them separately:

with open(filename,'wb') as fp:
    pickle.dump(dict1,fp)
    pickle.dump(dict2,fp)
    pickle.dump(dict3,fp)

with open(filename,'rb') as fp:
    d1=pickle.load(fp)
    d2=pickle.load(fp)
    d3=pickle.load(fp)

make sure to dump the big on last so you can load the little ones without loading the big one first. I imagine you could even get clever and store the file positions where each dump starts in a header of sorts and then you could seek to that location before loading (but that’s starting to get a little more complicated).