Question

There have been questions asked about memory errors in Python, but I want to ask one more specific to my situation. I am new to programming and Python.

When parsing a large text file (~8GB), the line

mylist = [line.strip('\n').split('|') for line in f]

resulted in “MemoryError”.

I am running the 64-bit of Python [MSC v.1500 64 bit (AMD64)] on Windows XP 64-bit with 12GB of RAM. How can I handle this Memory Error other than installing more RAM?

Answer 1

The memory error is coming because you’re trying to store your whole file in a list(which is in memory). So, try to work on each line instead of storing it:

for line in f:
   data = line.strip('\n').split('|')
   #do something here with data