Question

Have an application that has a Jtable with the data held in RAM, the table can upto be 1,000,000 rows by 100 columns so this just uses too much memory. So now I’m moving to backing the JTable from a database, but I think getting directly from JTable to database is not going to help much because either all the data from the database is going to get loaded into memory when jtable initilized , or as the user scrolls down the tabel I would do select staements to get the next data, which would be too slow, and how would I handle sorting.

So I think the correct solution is to stick Hibernate between the JTable and Database, but I still can’t quite see how this is going to help with a really large JTable.

Can anyone point me to a good example / have experience of using this with large dataset to keep memory usage down.

EDIT:Ive read some comments on other threads that a table with this much data should have filters so that only a subset of data is ever shown. I agree with that as a general principle and I will provide a filter HOWEVER only the user can decide how they want to filter it, and I still need to provide a ‘ALL’ option, and this is where it could all blow up.

I also remember something about having one table on top of another showing some kind of subset of the data that changes as you scroll down, but not seen a concrete example of this idea.

Answer 1

Normally for a function that displays large amount of records in the table format ,in order to consume less memory, the records will be shown page by page just like the Stackoverflow reputation league

Hibernate ‘s Criteria and Query API provides the following functions to do the pagination:

• setMaxResults(int maxResults) to limit the number of the row to be retrieved
• setFirstResult(int firstResult) to set the first row number to be retrieved (ie. 0 means first row)

In the JTable , you should has the variables for storing the number of records shown per page (pageNum) and the current page number (pageNum). Whenever user changes the pageNum by navigating the page forward or backward , you fetch the records for this page:

Query q = entityManager.createQuery("from someTable tbl order by tbl.id asc");
q.setFirstResult((pageNum -1)*pagesize).setMaxResults(pagesize); 

Important Points:

• Without the "order by" clause , the order of the fetched row is unpredictable . So , it is important that the query has to be sorted by the "order by clause" such that the displaying order of the records can be maintained when the page is browsed forward and backward .

You can refer to this (Use Google translate to translate to English) and its Github for an example about JTable pagination using Hibernate or JDBC.