Question

I have a CSV file in the assets folder with more than 10000 lines of data. I want to insert this data into a SQLite database when the database is created. I cannot include a database because it is a very generic application and the model demands a CSV file instead. I don’t want to read all 10000 lines of data and insert it from the memory in one stretch. How do I accomplish the task effectively and efficiently?

Answer 1

Just insert immediately once you’ve read the line. So, just don’t store the lines in some arraylist in memory.

E.g.

while ((line = reader.readLine()) != null) {
    insert(line);
}

You might want to do this in a single transaction so that you can rollback whenever it fails halfway. In the JDBC side, you may want to consider PreparedStatement#addBatch()/#executeBatch() to execute the inserts in batches. An example can be found in this answer.

Update: as a completely different but more efficient alternative, you can also leave Java outside the story and use CSV import facility provided by SQLite.