Question

I have a project that was using Scraperwiki to write to their sqlite store, but I need to just write a CSV. The catch is that all the data is stored in a dict, which works fine for writing to sqlite:

scraperwiki.sqlite.save(unique_keys = ['somekey'], data = data, table_name='fancy')

I run that after I scrape each row. There’s no order to a dict so I can’t just write the values to CSV. I’ve been looking over csv.DictWriter and collections.defaultdict, and I’m still wrapping my head around how I’d refactor my code so that I can write data, which is a dictionary to CSV instead. Here’s an example of my code as structured now:

def store_exception(exception, line_number, some_string):
    data = {
        'timestamp'  : datetime.now(),
        'line_number': line_number,
        'message'    : exception,
        'string'     : some_string
    }
    scraperwiki.sqlite.save(unique_keys = ['timestamp'], data = data, table_name='error_log')

I think I want something like this though:

def store_exception(exception, line_number, some_string):
    data = {
        'timestamp'  : datetime.now(),
        'line_number': line_number,
        'message'    : exception,
        'string'     : some_string
    }
    d = defaultdict(lambda: "")
    d_order = d['timestamp'],d['line_number'],d['message'],d['string']

    with open('some/path.csv', 'w') as csvfile:
        linewriter = csv.DictWriter(csvfile, d_order, delimiter='|',
                            quotechar='"', quoting=csv.QUOTE_MINIMAL)
        linewriter.writerow(data)

That seems inefficient, though. Do I need both collections.defaultdict and csv.DictWriter?

Answer 1

This should do it. You don’t need the default_dict at all, d_order just needs to be a list for field names, or in this case dict keys:

def store_exception(exception, line_number, some_string):
    data = {
        'timestamp'  : datetime.now(),
        'line_number': line_number,
        'message'    : exception,
        'string'     : some_string
    }

    d_order = ['timestamp', 'line_number', 'message', 'string']

    with open('some/path.csv', 'w') as csvfile:
        linewriter = csv.DictWriter(csvfile, d_order, delimiter='|',
                            quotechar='"', quoting=csv.QUOTE_MINIMAL)
        linewriter.writerow(data)