Question

I have a django app that loads a lot of data into a sqlite3 database file.
We are talking about millions of entries, entered in one function that at some point of its execution is throwing:

  File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/transaction.py", line 219, in inner
    self.__exit__(*sys.exc_info())
  File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/transaction.py", line 207, in __exit__
    self.exiting(exc_value, self.using)
  File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/transaction.py", line 286, in exiting
    leave_transaction_management(using=using)
  File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/transaction.py", line 56, in leave_transaction_management
    connection.leave_transaction_management()
  File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/backends/__init__.py", line 114, in leave_transaction_management
    self.rollback()
MemoryError

I examined my function and it is NOT leaking memory. What else can be done here?

Answer 1

The function may not be, but if your realing with such a huge database, the sqlite library may be consuming it in order to update the table. If you need to store such huge amounts of data you may have to move away from SQLite to a fully featured database server such as MySQL.