I’m creating and processing a very large data set, with about 34 million data points, and I’m currently storing them in python dictionaries in memory (about 22,500 dictionaries, with 15 dictionaries in each of 1588 class instances). While I’m able to manage this all in memory, I’m using up all of my RAM and most of my swap.
I need to be able to first generate all of this data, and then do analysis on select portions of it at a time. Would it be beneficial from an efficiency standpoint to write some of this data to file, or store it in a database? Or am I better off just taking the hit to efficiency that comes with using my swap space. If I should be writing to file/a database, are there any python tools that you would recommend to do so?
Get a relational database, fast! Or a whole lot more RAM.
If you’re using Python, then start with Python Database Programming. SQLite would be a choice, but I’d suggest MySQL based upon the amount of data you’re dealing with. If you want an object-oriented approach to storing your data, you might want to look at SQLAlchemy, but you’ll probably get more efficiency if you end up mapping each of your object classes to a table yourself and just coping with rows and columns.