Question

Is there any mechanism in Java to reduce the memory usage while reading large text files?

Almost every program I’ve come across uses String to read text files.But Java reserves space for each String literal.That’s why I think memory usage gets increased since all String objects are stored. All the classes of java.io deals with String. But if we’re not using StringBuilder then how can we reduce memory usage?

After all reducing memory usage is the primary concern of StringBuilder[since it’s not immutable like String]. Then how can we exploit its feature in Java I/O operation without using String i.e. without using something like this: sb.append([String object]);

Answer 1

Assume you have n strings, each of length 1 that you read from your input – for simplicity.

Using operator+ on strigns while reading will create a String object each time you concatenate strings, so you get strings of length 1,2,3,…,n

So the total memory usage of the strings combined is 1 + 2 + .. + n = O(n^2) in addition to the n strings you read from input

while if you use StringBuilder to create the final string, you actually create n – for input [each of length 1] and one object for the final string – of size n, so total memory usage of 1 + 1 + .. + 1 + n = O(n)

So, even if you use sb.append(String) – the space usage is asymptotcally beter then creating all intermediate strings – since you do not need to create intermediate String objects.

In addition – the performance [time] should be better when using StringBuilder – both because you create less objects, and both because of lesser memory usage – the gc doesn’t need to work as hard as when concatenating strings naively.

(*)Note that it is easy to see that the above still holds for any length of strings.