I have developed a 3-tier application and I have an issue about populating data from a database. My application has a window that queries large data from database and this process take a very long time. (My client is so far from server and connection between it is 128 KB/s).
Then I have think for the solution to solving this problem. The solution is populate x records of data every y sec. (eg: populate 10 records to display on UI first and then display more 10 row every 5 sec).
Is this a good solution to solving this problem ? Is there a better way to solving this?
If it is for display purposes you should be using paging. There is a variation of this, where the UI grabs the next ‘page’ of records when the user tries access more info with the scroll bar. If you need to aggregate data, do it on the query, so the whole data doesn’t travels trough the network. If you really/absolutely have to get the whole data and do some processes with it, consider if it or will ever be too big to load all at once. In that case, grab chunks of the data, run related processes, and then grab another chunk (so on). Regardless, you need to do it with an approach that doesn’t blocks the UI (threads / asynchronous operations).