Question

I have huge text files– each file is as big as 10 GB.

Now, I have with me 2 strings "StringNumber1" and "StringNumber2"

“Stringnumber1” occurs exactly once in each text file, and "StringNumber2" is always after "StringNumber1" (although there can be more than 1 occurrence of "StringNumber2" in the text file).

I want to extract all the text occurring in each file, starting from the match for “StringNumber1” and up to that occurrence of "StringNumber2" that occurs immediately after the matching “StringNumber1” (The content between these occurrences of "StringNumber1" and "StringNumber2" may be few bytes to many Gigabytes)…

How do I extract such content, using least RAM? I want to do this in Java.

Answer 1

Is just using a BufferedReader not good enough? Well i never had the problem to read really large files. Do you get some kind of error when using this approch?

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;

public class Read {

    /**
     * @param args
     * @throws IOException
     */
    public static void main(String[] args) throws IOException {
        BufferedReader br = new BufferedReader(new FileReader("someFile"));
        try {
            String line = br.readLine();
            while (line != null) {
                if(line.contains("String 1")) {
                    // ...
                } else if (line.contains("String 2")) {
                    // ...
                }               
                line = br.readLine();
            }       
        } finally {
            br.close();
        }
    }
}