Question

I want to sort a big text file (about 12 Giga), by counting the occurrences of each line. For that I have used :

sort file.txt | uniq -c > sorted  

But it is taking forever because of geometrical expansion. Any idea?

Answer 1

from collections import defaultdict

d = defaultdict(int)

with open(file.txt) as f:
    for line in f:
        d[line]+=1

d now contains a dictionary, where each key is a unique line, and the value is the count of that line.