Question

i have multiple files each containing 8/9 columns.

for a single file : I have to read last column containing some value and count the number of occurrence of each value and then generate an outfile.

I have done it like:

inp = open(filename,'r').read().strip().split('\n')  
out = open(filename,'w')  
from collections import Counter  
C = Counter()  
for line in inp:  
    k = line.split()[-1] #as to read last column  
    C[k] += 1  
for value,count in C.items():  
    x = "%s   %d" % (value,count)  
    out.write(x)  
    out.write('\n')  
out.close()  

now the problem is it works fine if I have to generate one output for one input. But I need to scan a directory using glob.iglobfunction for all files to be used as input. And then have to perform above said program on each file to gather result for each file and then of course have to write all of the analyzed results for each file into a single OUTPUT file.

NOTE: During generating single OUTPUT file if any value is found to be getting repeated then instead of writing same entry twice it is preferred to sum up the ‘count’ only. e.g. analysis of 1st file generate:

123 6  
111 5  
0   6  
45  5  

and 2nd file generate:

121 9  
111 7  
0   1  
22  2  

in this case OUTPUT file must be written such a way that it contain:

123 6  
111 12 #sum up count no. in case of similar value entry  
0   7  
45  5  
22  2  

i have written prog. for single file analysis BUT i’m stuck in mass analysis section.
please help.

Answer 1

Initialise a empty dictionary at the top of the program,
lets say, dic=dict()

and for each Counter update the dic so that the values of similar keys are summed and the new keys are also added to the dic

to update dic use this:

dic=dict( (n, dic.get(n, 0)+C.get(n, 0)) for n in set(dic)|set(C) )

where C is the current Counter, and after all files are finished write the dic to the output file.

import glob
from collections import Counter
dic=dict()
g_iter = glob.iglob(r'c:\\python32\fol\*')
for x in g_iter:

    lis=[]
    with open(x) as f:
        inp = f.readlines()
    for line in inp:
        num=line.split()[-1]
        lis.append(num)
    C=Counter(lis)
    dic=dict( (n, dic.get(n, 0)+C.get(n, 0)) for n in set(dic)|set(C) )
for x in dic:
    print(x,'\t',dic[x])