Question

Currently I have:

• 1 file with 9 million lines
• BufferedReader.readLine() to read every line
• String.split() to parse every line (columns separated by a pipe)
• A lot of RAM used (because of String interning?)

The problem is: As you may have guessed, I want to read and parse this file a little better…

Questions:

• How do I read this relatively big file using the least amount of resources (knowing that every line will need some kind of “split” on pipe)?
• Can I replace String.split by something else (on lets say, StringBuilder, CharBuffer, …)?
• What’s the best way to avoid using Strings reading the file until I have split them to their final character sequence?
• I don’t mind using something else then String in my POJOs, if you have anything better?
• The file will be reload every few hours, if that helps you in giving me a solution?

Thank you 🙂

Answer 1

A 9 million line file should take less than a few seconds. Most of that time will be spent reading the data into memory. How you split up the data is unlikely to make much difference important.

The BufferedReader and String.split sounds fine to me. I wouldn’t use interning unless you are sure this will help. (It won’t intern() it for you)

The latest version of Java 6 has some performance improvements in the handling of Strings. I would try Java 6 update 25 to see if it is any faster.

---

EDIT: Doing some test finds that split is surprisingly slow and you cna improve on it.

public static void main(String... args) throws IOException {
    long start1 = System.nanoTime();
    PrintWriter pw = new PrintWriter("deleteme.txt");
    StringBuilder sb = new StringBuilder();
    for (int j = 1000; j < 1040; j++)
        sb.append(j).append(' ');
    String outLine = sb.toString();
    for (int i = 0; i < 1000 * 1000; i++)
        pw.println(outLine);
    pw.close();
    long time1 = System.nanoTime() - start1;
    System.out.printf("Took %f seconds to write%n", time1 / 1e9);

    {
        long start = System.nanoTime();
        FileReader fr = new FileReader("deleteme.txt");
        char[] buffer = new char[1024 * 1024];
        while (fr.read(buffer) > 0) ;
        fr.close();
        long time = System.nanoTime() - start;
        System.out.printf("Took %f seconds to read text as fast as possible%n", time / 1e9);
    }
    {
        long start = System.nanoTime();
        BufferedReader br = new BufferedReader(new FileReader("deleteme.txt"));
        String line;
        while ((line = br.readLine()) != null) {
            String[] words = line.split(" ");
        }
        br.close();
        long time = System.nanoTime() - start;
        System.out.printf("Took %f seconds to read lines and split%n", time / 1e9);
    }
    {
        long start = System.nanoTime();
        BufferedReader br = new BufferedReader(new FileReader("deleteme.txt"));
        String line;
        Pattern splitSpace = Pattern.compile(" ");
        while ((line = br.readLine()) != null) {
            String[] words = splitSpace.split(line, 0);
        }
        br.close();
        long time = System.nanoTime() - start;
        System.out.printf("Took %f seconds to read lines and split (precompiled)%n", time / 1e9);
    }
    {
        long start = System.nanoTime();
        BufferedReader br = new BufferedReader(new FileReader("deleteme.txt"));
        String line;
        List<String> words = new ArrayList<String>();
        while ((line = br.readLine()) != null) {
            words.clear();
            int pos = 0, end;
            while ((end = line.indexOf(' ', pos)) >= 0) {
                words.add(line.substring(pos, end));
                pos = end + 1;
            }
            // words.
            //System.out.println(words);
        }
        br.close();
        long time = System.nanoTime() - start;
        System.out.printf("Took %f seconds to read lines and break using indexOf%n", time / 1e9);
    }
}

prints

Took 1.757984 seconds to write
Took 1.158652 seconds to read text as fast as possible
Took 6.671587 seconds to read lines and split
Took 4.210100 seconds to read lines and split (precompiled)
Took 1.642296 seconds to read lines and break using indexOf

So it appears that splitting the string yourself is an improvement and gets you close to treading text as fast as you can. The only way to read it faster is to treat the file as binary/ASCII-7. 😉