Question

I’ve been sitting for some time with a task to import serverlogs from a .txt file, write it out to a python dictionary and write it further out to a csv file to make it readable in a database. When I open it in fx. excel, I want it to be listed with headers.

EDIT: I need this to be generic in the way that I can load a log file of any length and still get everything. Edit done.

I’m stuck at the part of writing the dictionary down to the csv file. My following script looks like the following, and the seperated, out commented files are tries i’ve used to make it write it out.

    lineArray = []
    timeArray = []
    typeArray = []
    whoArray = []
    idArray = []
    msgArray = []

    infile = open('C:\SomeArea\data.txt', 'r')
    dictionary = {'time','type','who','id','msg'}
    for line in infile:
        timeInString = line[:24]
        timeInStringStripped = timeInString.strip()
        timeArray.append(timeInStringStripped)
        allTimes = ', '.join(timeArray)

#The last 4 fields was removed here to take less space

    infile.close()
    dictionary = {'time': allTimes, 'type': allTypes, 'who': allWhos, 'id': allIds, 'msg': allMsgs}

Half attempts to write it to the csv

#<><><><><><><><><><><><><><><><><><><><><><><>#
#str = 'lement1, lement2'
#arr = ['lement1, lement2']

#import csv
#f = open('sometest2.csv', 'wb')
#wtr = csv.DictWriter(f, dictionary.viewkeys())
#for element in dictionary.viewitems():
#    print element
#    for e in element:
#        print e
#wtr.writerow(0)
#f.close()
#wtr.writerows(allTimes)
#wtr.writerows(allTypes)
#wtr.writerows(allWhos)
#wtr.writerows(allIds)
#wtr.writerows(allMsgs)
#print dictionary

#<><><><><><><><><><><><><><><><><><><><><><><>#

import csv
f = open('sometest2.csv', 'wb')
wtr = csv.DictWriter(f, dictionary.viewkeys())
for row in dictionary: f.write("%s%s" %(delimiter.join([row[name] for name in fieldnames]),lineterminator))

Answer 1

The problem is that the code attempts to write a dictionary of lists and csv.DictWriter expects a list of dictionaries.

The right way to parse your data to work with the csv module would be to store each row as a dictionary. Hence instead of having something like this:

dictionary = {'time': allTimes,
              'type': allTypes,
              'who': allWhos,
              'id': allIds,
              'msg': allMsgs}

You should have something like this:

data = [{'time': time_row_1,
         'type': type_row_1,
         'who': who_row_1,
         'id': id_row_1,
         'msg': msg_row_1},

        ...

        {'time': time_row_n,
         'type': type_row_n,
         'who': who_row_n,
         'id': id_row_n,
         'msg': msg_row_n},
       ]

Hence, the code to read from a file should be like:

data = []
for line in infile:
    row = {}
    row['time'] = ...
    row['type'] = ...
    ...
    ...
    data.append(row)

and the code to write to a file would be:

wtr = csv.DictWriter(open(filename, 'w'), keys)
wtr.writerows(data)