Question

I.e. allowing you to efficiently access and update data as you would in a database.

My specific situation is the following:

I have a very large Monte Carlo simulation + optimization and the data is sorted into a few dictionaries with 600k+ tuple keys each, formatted like this:

simple_dictionary[(year, month, day, hour, minute)] = value_or_small_list_or_small_numpy_array

The above, it turns out, is actually slightly slower than nested dictionaries:

simple_dictionary[year][month][day][hour][minute] = value_or_small_list_or_small_numpy_array

This leads me to believe – and please correct me if I’m wrong – that the latter format checks less keys than the former. This assumption is based on my code (which is too long to post). My code does not create new keys after __init__, but has every key/value pair called at least once each iterations. However, not all values are updated.

Assuming this can/should run from RAM, is there an alternative to the above code that is more efficient?

Answer 1

Have you thought about trying an in-memory sqlite3 database? That would be about as “similar to a database” as you can get, and it’s all in memory. 🙂

Using the connection string :memory: gives you an in memory database.