Question

I have a master file which is the one which will constantly be updated and a file which is created every minute. I want to be able to compare the new file which is created every minute to the already existing master file. So far I’ve got:

with open("jobs") as a:
   new = a.readlines()

count=0
for item in new:
   new[count]=new[count].split(",")
   count+=1

This will allow me to compare the first index([0] of each line in my master file. Now at this point I start to confuse myself. I’m guessing it would be something along the lines of:

counter=0
for item in new:
    if new[counter][0] not in master:
        end = open("end","a")
        end.write(str(new[counter]) + "\n")
        counter+=1
        end.close()
    else:
         REPLACE LINES THAT ALREADY EXIST IN MASTER FILE WITH NEW LINE

The IDs won’t necessarily be in the same order every time the new file comes in and the new file may contains more entries than the master file at some point.

If I haven’t made sense or missed some information out then please let me know and I’ll try and clarify. Thanks.

Answer 1

Sounds like a csv problem to me.

unfortunately, it is not clear from your question, if you want to modify the masterfile itself, an out-file, or both.
this does the second (it takes a masterfile and an updatefile, both in csv format, and prints the merged thing unsorted to an out-file). If this is not what you want, or if you got data comma-seperated, but without fieldnames on top, change as you need, should be easy enough.

import csv
with open("master.csv") as m, open("update.csv") as u, open("out.csv", "w") as o:
    master = { line['ID']: line for line in csv.DictReader(m) }
    update = { line['ID']: line for line in csv.DictReader(u) }
    master.update(update)
    fields = csv.DictReader(open("master.csv")).fieldnames
    out = csv.DictWriter(o, fields)
    out.writeheader()
    out.writerows(master.values())

with master.csv as such:

ID,Name,Foo,Bar,Baz,Description
1000001,Name here:1,1001,1,description here
1000002,Name here:2,1002,2,description here
1000003,Name here:3,1003,3,description here
1000004,Name here:4,1004,4,description here
1000005,Name here:5,1005,5,description here
1000006,Name here:6,1006,6,description here
1000007,Name here:7,1007,7,description here
1000008,Name here:8,1008,8,description here
1000009,Name here:9,1009,9,description here

and update.csv as such:

ID,Name,Foo,Bar,Baz,Description
1000003,UPDATED Name here:3,1003,3, UPDATED description here
1000010,NEW ITEM Name here:9,1009,9,NEW ITEM description here 

it outputs to out.csv:

ID,Name,Foo,Bar,Baz,Description
1000010,NEW ITEM Name here:9,1009,9,NEW ITEM description here ,
1000008,Name here:8,1008,8,description here,
1000009,Name here:9,1009,9,description here,
1000006,Name here:6,1006,6,description here,
1000007,Name here:7,1007,7,description here,
1000004,Name here:4,1004,4,description here,
1000005,Name here:5,1005,5,description here,
1000002,Name here:2,1002,2,description here,
1000003,UPDATED Name here:3,1003,3, UPDATED description here,
1000001,Name here:1,1001,1,description here,

Note that the order is not preserved (not clear from question if neccessary). But it is fast and clean.