Question

I have to remove duplicate strings from extremely big text file (100 Gb+)

Since in memory duplicate removing is hopeless due to size of data, I have tried bloomfilter but of no use beyond something like 50 millions strings ..

total strings are like 1 trillion+

I want to know what are the ways to solve this problem..

My initial attempt is, dividing the file in to number of sub files , sort each file and then merge all files together…

If you have better solution than this please let me know,

Thanks..

Answer 1

The key concept you are looking for here is external sorting. You should be able to merge sort the whole file using the techniques described in that article and then run through it sequentially to remove duplicates.

If the article is not clear enough have a look at the referenced implementations such as this one.