Question

I am looking around the net (so far I have found pickle ) for an explanation for what I am doing wrong

I am trying to write a very large data structure (nested dictionary/list) to file. Using the below code I discovered that the problem might be because the passed string is too large:

f = open('/path/to/file' , 'w')

try:
 f.write(str(dataStructure))
except:
 try:
  f.write('ABC')
 except:
  print 'Even smaller strings such as ABC did NOT print to the file'
 else:
  print 'Smaller strings such as ABC DID print to the file'

the dataStructure dictionary has a great deal of clique information, in this case, around 10,000 – 100,000 floating point values. The whole reason I am dumping everything into a single file, instead of saving in subfiles is because I want to exec a single file to load it, rather than manually load a few dozen subsets of the file.

Before I start saving each clique (each neuron has several different incoming clique files, such that for a neuron we are looking at 20 indexed files) I was wondering if the file size was actually the problem, or if the problem must be in something else.

Thanks

Answer 1

I’m gonna guess that your problem is that the string you create is too large to exist in memory. For something that big you should write it out piece by piece to the disk.

You could use pickle, json, xml or something which should handle this correctly.