Question

I’m processing large CSV files (on the order of several GBs with 10M lines) using a Python script.

The files have different row lengths, and cannot be loaded fully into memory for analysis.

Each line is handled separately by a function in my script. It takes about 20 minutes to analyze one file, and it appears disk access speed is not an issue, but rather processing/function calls.

The code looks something like this (very straightforward). The actual code uses a Class structure, but this is similar:

csvReader = csv.reader(open("file","r")
for row in csvReader:
   handleRow(row, dataStructure)

Given the calculation requires a shared data structure, what would be the best way to run the analysis in parallel in Python utilizing multiple cores?

In general, how do I read multiple lines at once from a .csv in Python to transfer to a thread/process? Looping with for over the rows doesn’t sound very efficient.

Thanks!

Answer 1

Try benchmarking reading your file and parsing each CSV row but doing nothing with it. You ruled out disk access, but you still need to see if the CSV parsing is what’s slow or if your own code is what’s slow.

If it’s the CSV parsing that’s slow, you might be stuck, because I don’t think there’s a way to jump into the middle of a CSV file without scanning up to that point.

If it’s your own code, then you can have one thread reading the CSV file and dropping rows into a queue, and then have multiple threads processing rows from that queue. But don’t bother with this solution if the CSV parsing itself is what’s making it slow.