Question

I run into the following errors when i try to store a large file into a string.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2882)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
    at java.lang.StringBuffer.append(StringBuffer.java:306)
    at rdr2str.ReaderToString.main(ReaderToString.java:52)

As is evident, i am running out of heap space. Basically my pgm looks like something like this.

FileReader fr = new FileReader(<filepath>);
sb = new StringBuffer();
char[] b = new char[BLKSIZ];

while ((n = fr.read(b)) > 0) 
     sb.append(b, 0, n);    

fileString = sb.toString();

Can someone suggest me why i am running into heap space error? Thanks.

Answer 1

In the OP, your program is aborting while the StringBuffer is being expanded. You should preallocate that to the size you need or at least close to it. When StringBuffer must expand it needs RAM for the original capacity and the new capacity. As TomTom said too, your file is likely 8-bit characters so will be converted to 16-bit unicode in memory so it will double in size.

The program has not even encountered yet the next doubling – that is StringBuffer.toString() in Java 6 will allocate a new String and the internal char[] will be copied again (in some earlier versions of Java this was not the case). At the time of this copy you will need double the heap space – so at that moment at least 4 times what your actual files size is (30MB * 2 for byte->unicode, then 60MB * 2 for toString() call = 120MB). Once this method is finished GC will clean up the temporary classes.

If you cannot increase the heap space for your program you will have some difficulty. You cannot take the “easy” route and just return a String. You can try to do this incrementally so that you do not need to worry about the file size (one of the best solutions).

Look at your web service code in the client. It may provide a way to use a different class other than String – perhaps a java.io.Reader, java.lang.CharSequence, or a special interface, like the SAX related org.xml.sax.InputSource. Each of these can be used to build an implementation class that reads from your file in chunks as the callers needs it instead of loading the whole file at once.

For instance, if your web service handling routes can take a CharSequence then (if they are written well) you can create a special handler to return just one character at a time from the file – but buffer the input. See this similar question: How to deal with big strings and limited memory.