Question

I have a Datatable which is fetching ~5,50,000 records from database(SQLite). When fetching it is slowing down the system.

I am storing these records on backend in SQLite Database and in frontend in Datatable.
Now what should i do so that database creation time(~10.5 hours) in backend and fetching time in Front end reduces.

Is there any other structure that can be used to do so. I have read that Dictionary & Binary File is fast. Can they be used for this purpose & how?

(Its not a web app. its a WPF desktop app where frontend and backend are on same machine).

Answer 1

Your basic problem, I believe, is not the strucutre that you want to maintain but the way you manage your data-flow. Instead of fetching all data from database into some strcuture (DataTable in your case), make some stored procedure to make a calculations you need on server-side and return to you data already calculated. In this way you gain several benefits, like:

• servers holding database servers are usually faster then development or client machine
• huge reduction of data trasmission, as you return only result of the calculation

EDIT

Considering edited post, I would say that DataTable is already highly optimized on in memory access, and I don’t think changing it to something else will bring you a notable benefits. What I think can bring a banefit is a revision of a program flow. In other words, try to answer following questions:

• do I need all that records contemporary ?
• can I run some calculations in service, let’s say, on night ?
• can I use SQL Server Express (just example) and gain benefits of possibility to run a stored procedure, that may (have to be measured) run the stuff faster, even on the same machine ?