Question

I have a large text file in the below format, which I wish to convert into a CSV File. The column names on the CSV file should corrospond to the first part of the tuples seen below. Its safe to assume the first item in the line, which is not a tuple, will always be in the below format.

Other issues include each line may not have the same fields – some have for example, Statuses, some dont. Some have multiple instances of the same field, in which case I require the second part of the tuples to be concatenated (eg To Mr Smith; Mrs Green) but these are issues which are further away for now.

[' Message  1 '];['Status', 'Read'];['Message ID', '012434'];['Message Truncation', 'OK'];['Priority', 'Low'];['Sent Time', '15/12/2010 05:56:36']
[' Message  2 '];['ColumnName', 'Read'];['ColumnName2', '012434'];['Message Truncation', 'OK'];['Priority', 'Low'];['Sent Time', '15/12/2010 05:56:36']
[' Message  3 '];['To', 'Mr Smith'];['To', 'Mrs green'];['Message Truncation', 'OK'];['Priority', 'Low'];['Sent Time', '15/12/2013 05:56:36']

…

My plan is to iterate thru every block in the file to establish the column names, then start adding data to these column names, leaving blanks when appropriate. Im just wonderint how to go about this in a pythonic manner, as I’ve played around with a list of dictionaries and got stuck.

I think I need to split the line, then add each tuple to a dictionary. Any help?
Thanks!

for line in file:
    line_split = line.split(';')

Answer 1

Solution using pure python…

infile = "listdata.txt"
data = open(infile, "r").readlines()

dataDict = []
columns = []

# Create a dictionary list
for line in data:
    row = line.split(";");
    rowData = {}
    for cell in row:
        cell = cell.strip()[1:-1].split(",")
        if len(cell) > 1:
            rowData[cell[0].strip().strip('"').strip("'")] = cell[1].strip().strip('"').strip("'")
    keys = rowData.keys()
    dataDict.append(rowData)
    columns = list(set(columns) | set(keys))

# Write dictionary list to file
outfile = "listdata.csv"
fp = open(outfile, "w")

for key in columns:
    fp.write(key + ", ")

fp.write("\n")

for data in dataDict:
    for key in columns:
        if key in data:
            fp.write(data[key] + ",")
        else:
            fp.write(",")
    fp.write("\n")

fp.close()

Input:

[' Message  1 '];['Status', 'Read'];['Message ID', '012434'];['Message Truncation', 'OK'];['Priority', 'Low'];['Sent Time', '15/12/2010 05:56:36']
[' Message  2 '];['ColumnName', 'Read'];['ColumnName2', '012434'];['Message Truncation', 'OK'];['Priority', 'Low'];['Sent Time', '15/12/2010 05:56:36']
[' Message  3 '];['To', 'Mr Smith'];['To', 'Mrs green'];['Message Truncation', 'OK'];['Priority', 'Low'];['Sent Time', '15/12/2013 05:56:36']

Output:

Status, Sent Time, To, ColumnName2, Message ID, Message Truncation, Priority, ColumnName, 
Read,15/12/2010 05:56:36,,,012434,OK,Low,,
,15/12/2010 05:56:36,,012434,,OK,Low,Read,
,15/12/2013 05:56:36,Mrs green,,,OK,Low,,

---

Update

This handles multiple entries with same type and join then with ":".

key = cell[0].strip().strip('"').strip("'")
value = cell[1].strip().strip('"').strip("'")
if key in rowData:
    rowData[key] = rowData[key] + ":" + value
else:
    rowData[key] = value