Question

I assumed sorting a CSV file on multiple text/numeric fields using Python would be a problem that was already solved. But I can’t find any example code anywhere, except for specific code focusing on sorting date fields.

How would one go about sorting a relatively large CSV file (tens of thousand lines) on multiple fields, in order?

Python code samples would be appreciated.

Answer 1

Here’s Alex’s answer, reworked to support column data types:

import csv
import operator

def sort_csv(csv_filename, types, sort_key_columns):
    """sort (and rewrite) a csv file.
    types:  data types (conversion functions) for each column in the file
    sort_key_columns: column numbers of columns to sort by"""
    data = []
    with open(csv_filename, 'rb') as f:
        for row in csv.reader(f):
            data.append(convert(types, row))
    data.sort(key=operator.itemgetter(*sort_key_columns))
    with open(csv_filename, 'wb') as f:
        csv.writer(f).writerows(data)

Edit:

I did a stupid. I was playing with various things in IDLE and wrote a convert function a couple of days ago. I forgot I’d written it, and I haven’t closed IDLE in a good long while – so when I wrote the above, I thought convert was a built-in function. Sadly no.

Here’s my implementation, though John Machin’s is nicer:

def convert(types, values):
    return [t(v) for t, v in zip(types, values)]

Usage:

import datetime
def date(s):
    return datetime.strptime(s, '%m/%d/%y')

>>> convert((int, date, str), ('1', '2/15/09', 'z'))
[1, datetime.datetime(2009, 2, 15, 0, 0), 'z']