I have a text file with 50k+- lines and each line contains data which has to be pulled out of each line as a separate field.
The program runs several times a day.
Since this app is portable I am using SQLIite and reading each of those 50k lines one by one gathering required data and inserting into SQlite DB File as it goes.
I did some tests and found out reading through lines alone only takes like 10% of actual time it takes now, all the overhead comes when I am inserting all that data one by one in SQLite db.
Looking for suggestions for improvement.
You could increase performance using transactions, so that multiple INSERTs are requested at once, instead of one for each line in your text file. This would allow you to batch INSERT statements (try something like 100 per batch) – this will result in a significant performance boost.