Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Python – Using cPickle to load a previously saved pickle uses too much memory?
My pickle file is about 340MB but takes up 29% of 6gb of memory when loaded. This seems a bit too much. The pickle file is a dictionary of dictionaries. Is this appropriate?
Code used:
import cPickle as pickle
file = pickle.load( file_handle )
Thanks
About 1.7GB seems a bit much, but not impossible. How much memory did the data take before it was pickled?
After unpickling the data should take about the same amount of memory as it took before it was pickled, how big it is in it’s on-disk format is not really that significant.