Question

I must parse HTML files, which can be up to 500 000 links.
Of which 400 000 will be desired by me.

Should I put all the links that satisfy the condition for the new list and then for the elements of this list and put it into the database.

Or when I find links to satisfy the condition to add it to the database (sqlite) (and commit it).
Is that a large number of commits is not a problem?

I do not want to lose data in case of failure such as power. Thats why i want commit after insert to the database.

How best to place a large number of items in the database?

Answer 1

Consider just doing a commit after every 1000 records or so