Question

I am using an inner class that is a subclass of a HashMap. I have a String as the key and double[] as the values. I store about 200 doubles per double[]. I should be using around 700 MB to store the keys, the pointers and the doubles. However, memory analysis reveals that I need a lot more than that (a little over 2 GB).

Using TIJmp (profiling tool) I saw there was a char[] that was using almost half of the total memory. TIJmp said that char[] came from Serializable and Cloneable. The values in it ranged from a list of fonts and default paths to messages and single characters.

What is the exact behavior of Serializable in the JVM? Is it keeping a “persistent” copy at all times thus, doubling the size of my memory footprint? How can I write binary copies of an object at runtime without turning the JVM into a memory hog?

PS: The method where the memory consumption increases the most is the one below. The file has around 229,000 lines and 202 fields per line.

public void readThetas(String filename) throws Exception
{
    long t1 = System.currentTimeMillis();
    documents = new HashMapX<String,double[]>(); //Document names to indices.
    Scanner s = new Scanner(new File(filename));
    int docIndex = 0;
    if (s.hasNextLine())
        System.out.println(s.nextLine()); // Consume useless first line :)
    while(s.hasNextLine())
    {
        String[] fields = s.nextLine().split("\\s+");
        String docName = fields[1];
        numTopics = fields.length/2-1;
        double[] thetas = new double[numTopics];
        for (int i=2;i<numTopics;i=i+2)
            thetas[Integer.valueOf(fields[i].trim())] = Double.valueOf(fields[i+1].trim());
        documents.put(docName,thetas);
        docIndex++;
        if (docIndex%10000==0)
            System.out.print("*"); //progress bar ;)
    }
    s.close();
    long t2 = System.currentTimeMillis();
    System.out.println("\nRead file in "+ (t2-t1) +" ms");
}

Oh!, and HashMapX is an inner class declared like this:

public static class HashMapX< K, V> extends HashMap<K,V> {
    public V get(Object key, V altVal) {
        if (this.containsKey(key))
            return this.get(key);
        else
            return altVal;
    }
}

Answer 1

So, I found the answer. It is a memory leak in my code. Had nothing to do with Serializable or Cloneable.

This code is trying to parse a file. Each line contains a set of values which I am trying to extract. Then, I keep some of those values and store them in a HashMapX or some other structure.

The core of the problem is here:

        String[] fields = s.nextLine().split("\\s+");
        String docName = fields[1];

and I propagate it here:

        documents.put(docName,thetas);

What happens is that docName is a reference to an element in an array (fields) and I am keeping that reference for the life of the program (by storing it in the global HashMap documents). As long as I keep that reference alive, the whole String[] fields cannot be garbage collected. The solution:

        String docName = new String(fields[1]); // A copy, not a reference.

Thus copying the object and releasing the reference to the array element. In this way, the garbage collector can free the memory used by the array once I process every field.

I hope this will be useful to all of those who parse large text files using split and store some of the fields in global variables.

Thanks everybody for their comments. They guided me in the right direction.