Question

I create a custom table, not extends from JTable, and i have a huge amount of data approx 4.000.000 string data (10% uniq string). Now i create and index which looks like this:

I create and index for every column. I use treeset for merging data when the user use live search functionality of my table.

Index:

ArrayList which represent the columns:
Column 1 | Column 2 | Column 3 | Column 4
Each element the arraylist contains a HasMap which represent the index:
key -> the data String
value -> the value represent which rows contains this index inside a TreeSet

Example:

Name Column:
Emma 
John
Doe
Emma
Walker
Emma
Doe

HashMap(Emma) -> 0, 3, 5

private void buildIndex()
    {
        if (monitorModel.getMessageIndex() == null)
        {
            ArrayList<HashMap<String, TreeSet<Integer>>> messageIndex = new ArrayList<>(filterableColumn.length);
            for (int i = filterableColumn.length; i >= 0; i--)
            {
                HashMap<String, TreeSet<Integer>> hash = new HashMap<>();
                messageIndex.add(hash);
            }
            // create index for every column
            for (int i = monitorModel.getParser().getMyMessages().getMessages().size() - 1; i >= 0; --i)
            {
                TreeSet<Integer> tempList;

                for (int j = 0; j < filterableColumn.length; j++)
                {
                    String value  = StringPool.getString(getValueAt(i, j).toString());
                    if (!messageIndex.get(j).containsKey(value))
                    {
                        tempList = new TreeSet<>();
                        messageIndex.get(j).put(value, tempList);
                    }
                    else
                    {
                        tempList = messageIndex.get(j).get(value);
                    }

                    tempList.add(i);
                }
            }
            monitorModel.setMessageIndex(messageIndex);
        }
    }

This solution use 500MB heap size which is impossible, how i can optimize this code?

Answer 1

500MB heap size which is impossible

I seriously doubt it uses only that much if you have 4 billion strings. I suspect you application stops at that point.

If you have “4.000.000.000 string data” then this will use about 100 bytes per string to store in a collection (assuming the strings are short)

This means you need 400 GB of memory. The only way to make this more efficient and workable is to use memory mapped files. You can fairly easily hold this much data this way.

On the other hand if you meant 4 million, not 4 billion then a size of 500 MB is quite reasonable. Given 500 MB costs about $10 these days, I wouldn’t worry about it.