Question

I need your help please.

I have a text file contains lines of lists, each line represent a list of items. I need to extract all items that have a frequency of >=2 and output them into another file.Here an example.

['COLG-CAD-406', 'CSAL-CAD-030', 'COLG-CAD-533', 'COLG-CAD-188']

['COLG-CAD-188']

['CSAL-CAD-030']

['EPHAG-JAE-004']

['COLG-CAD-188', 'CEM-SEV-004']

['COL-CAD-188', 'COLG-CAD-406']

the output should be

['COLG-CAD-406'], 2

['CSAL-CAD-030'], 2

['COLG-CAD-188'], 3

and so on till the end of the file

Thank you very much for your help in advance.

Answer 1

This is a complete script that does exactly what you want, using regex:

from collections import defaultdict
import re

myarch = 'C:/code/test5.txt'   #this is your archive
mydict = defaultdict(int)

with open(myarch) as f:
    for line in f:
        codes = re.findall("\'(\S*)\'", line)
        for key in codes:
            mydict[key] +=1

out = []
for key, value in mydict.iteritems():
    if value > 1:
        text = "['%s'], %s" % (key, value)
        out.append(text)

#save to a file
with open('C:/code/fileout.txt', 'w') as fo:
    fo.write('\n'.join(out))

This can be simplified as:

from collections import defaultdict
import re

myarch = 'C:/code/test5.txt'
mydict = defaultdict(int)

with open(myarch) as f:
    for line in f:
        for key in re.findall("\'(\S*)\'", line):
            mydict[key] +=1

out = ["['%s'], %s" % (key, value) for key, value in mydict.iteritems() if value > 1]

#save to a file
with open('C:/code/fileout.txt', 'w') as fo:
    fo.write('\n'.join(out))