Question

I have a situation here. I have a huge database with >10 columns and millions of rows. I am using a matching algorithm which matches each input records with the values in database.

The database operation is taking lot of time when there are millions of records to match. I am thinking of using a multi-hash map or any resultset alternative so that i can save the whole table in memory and prevent hitting database again….

Can anybody tell me what should i do??

Answer 1

I don’t think this is the right way to go. You are trying to do the database’s work manually in Java. I’m not saying that you are not capable of doing this, but most databases have been developed for many years and are quite good in doing exactly the thing that you want.

However, databases need to be configured correctly for a given type of query to be executed fast. So my suggestion is that you first check whether you can tweak the database configuration to improve the performance of the query. The most common thing is to add the right indexes to your table. Read How MySQL Uses Indexes or the corresponding part of the manual of your particular database for more information.

The other thing is, if you have so much data storing everything in main memory is probably not faster and might even be infeasible. Not to say that you have to transfer the whole data first.

In any case, try to use a profiler to identify the bottleneck of the program first. Maybe the problem is not even on the database side.