Question

I have a large collection of tab separated text data in the form of DATE NAME MESSAGE. By large I mean, a collection of 1.76GB divided into 1075 actual files. I have to get the NAME data from all the files. Till now I have this:

   File f = new File(directory);
        File files[] = f.listFiles();
        // HashSet<String> all = new HashSet<String>();
        ArrayList<String> userCount = new ArrayList<String>();
        for (File file : files) {
            if (file.getName().endsWith(".txt")) {
                System.out.println(file.getName());
                BufferedReader in;
                try {
                    in = new BufferedReader(new FileReader(file));
                    String str;
                    while ((str = in.readLine()) != null) {
                        // if (all.add(str)) {
                        userCount.add(str.split("\t")[1]);
                        // }

                        // if (all.size() > 500)
                        // all.clear();
                    }
                    in.close();
                } catch (IOException e) {
                    System.err.println("Something went wrong: "
                            + e.getMessage());
                }

            }
        }

My program is always giving out of memory exception even with -Xmx1700. I cannot go beyond that. Is there anyway I can optimize the code so that it can handle the ArrayList<String> of NAMEs?

Answer 1

Since you seem to be allowing alternative solutions than Java, here’s an awk one that should handle it.

cat *.txt | awk -F'\t' '{sum[$2] += 1} END {for (name in sum) print name "," sum[name]}'

Explanation:

-F'\t' - separate on tabs
sum[$2] += 1 - increment the value for the second element (name)

Associative arrays make this extremely succinct. Running it on a test file I created as follows:

import random

def main():
    names = ['Nick', 'Frances', 'Carl']
    for i in range(10000):
        date = '2012-03-24'
        name = random.choice(names)
        message = 'asdf'
        print '%s\t%s\t%s' %(date, name, message)

if __name__ == '__main__':
    main()

I get the results:

Carl,3388
Frances,3277
Nick,3335