Question

Application flow

An input file consists of multiple logical documents.

1. Extract one input logical document.
2. Parse the elements within the document.
3. Build an xml out of the input logical document.
4. Write that document back to a physical file.

What would be a good way to reduce memory needs?

Right now, I save all the logical documents in a physical file in an ArrayList so that I do all the I/O once. But when I write a single logical document to stream after processing, it hits a Java heap space error after 20,000 logical documents. The input logical document count is about 100,000 and I was looking for an efficient way to process & write all of these docs.

Answer 1

Don’t keep everything in memory. Instead, read from and write to disk as you go. For instance:

void split(File inputFile, File outputFile) {
    Inputstream is = new BufferedInputStream(new FileInputStream(inputFile));
    OutputStream os = new BufferedOutputStream(new FileOutputStream(outputFile));
    for (;;) {
        Document doc = readDocument(is);
        if (doc == null) break;
        write(buildXml(doc), os);
    }
    os.close();
    is.close();
}

(You’ll obviously want to add error handling)

That way, only one logical document is in memory at any given time.