Question

How should I implement very fast lookups of items in a collection?

class Person(object):
    __all__ = dict()

    def __init__(self, _id, name, age):
        self._id = _id
        self.name = name
        self.age = age
        self.__class__.__all__[_id] = self

Suppose I wish to get the five oldest people. If len(Person.__all__) is quite large, and I need to do this operation frequently, what is the best practice? Currently my code takes about 4hrs to run, and I haven’t even fed the whole dataset in yet.

My current thinking is that I could use a database to maintain indexes, but my feeling is that this would be slower than keeping all objects in ram. (I can comfortably fit every object in ram).

Or, I could have some sort of index based on an auto-sorted list, within Python. So, when I needed to look up people of a certain age, I would query that list, to find ids, then use Person.__all__ to get the object itself.

What would be the best option?

Answer 1

You can make an in memory database with sqlite. It’s easy to move the db to disk later on if you need to