for the application I am working on I have to generate consolidated report where application have to read very large volume of data, some time in the order of over 100,000 records from database. What we are finding is using grails/hibernate (which we are using as defacto ORM for the entire application) is not performing well. In certain cases even to query result set as small as 10,000 records it takes 1.3+ sec. Wondering if there is any tuning I am missing or if its better to move to native sql for such requirements.
Just to mention the table is very simple and in there are no joins. The only reason I prefer to go with Hibernate as criterion’s to filter data is very dynamic and spawns multiple optional columns.
You may want to check out this link. Very insightful. Guy shows how he went from loading 100K record from over 2hrs to just 3 minutes.
http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/