Question

I have a gzip’d pickled file that is 4mb in size. Each time I open the file and load it the time it takes to unpickle takes longer. The contents of the file never change and the size stays the same.

Test Times:
1st Run: 27.5s
2nd Run: 44.1s
3rd Run: 52.7s

The code I’m using is as follows:

f = gzip.open( filepath, "rb" )
pickleFile = cPickle.Unpickler( f )
paData = pickleFile.load()
f.close()

Any idea why it is taking longer to load each time I run?

Answer 1

I created two test pickle file 3MBs and 22MBs, and zipped it.

I don’t understand what you are doing because I just did a test and this code:

def get_time():
    return int(time.time())

while(count < 10):    
    filepath='/home/shared/word_out.gz'
    start = get_time()
    f = gzip.open( filepath, "rb" )
    pickleFile = cPickle.Unpickler( f )
    paData = pickleFile.load()
    f.close()
    end = get_time()

    print 'Time ellapsed' , end - start
    count+=1

Always prints out the same as I run it.

3 MB File:

Time ellapsed 1
Time ellapsed 1
Time ellapsed 0
Time ellapsed 1
Time ellapsed 1
Time ellapsed 0
Time ellapsed 1
Time ellapsed 1
Time ellapsed 0
Time ellapsed 1

---

22MB File:

Time ellapsed 5
Time ellapsed 4
Time ellapsed 2
Time ellapsed 2
Time ellapsed 3
Time ellapsed 2
Time ellapsed 3
Time ellapsed 2
Time ellapsed 2
Time ellapsed 2