Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I insert 10,000,000+ records into a sqlite base. Is it safe that I wait with connection.commit() until I finish all the insertions? Or there is some out of memory/overflow risk? Is it possible that the uncommitted entries will use all the available memory and cause page swapping? Committing after each INSERT is a performance killer, so I want to avoid it.
I use sqlite3 module with Python
Well, in the first place, sqlite is not really meant to handle that much data. It is generally for smaller data sets. However, committing that much data might take longer when committing all at once then if you were to commit say after every 2 or 3 inserts.
As far as your memory/overflow risk, sqlite dumps everything into a file chache then on commit put’s all the file cache into the sqlitedb file.
However, committing all at once shouldnt give you any issues.