Question

I’m building a web application in python. A part of this application is working with the data that can be described as follows:

Symbol     Begin Date      End Date
AAPL       Jan-1-1985      Dec-27-2010
...

The data is somewhat static – it will be periodically updated, that is: new entries may be added, and the “End Date” field can be updated for all entries.

Now, the question: given the more-or-less static nature of the dataset, what is the best way of storing it and working with it? “Working” means fetching random lines, hopefully more than few times per second.

I can do it with XML file, with SQL DB or SQLite, with JSON object file and some kind of python object in memory.

What are the cons and pros of different solutions? I’ll be thankful for explanations and for the edge cases (such as ‘until 10times/sec XML file is the best, after that SQL DB).

Update: Thanks for all the answers! Just a smallish update: currently the set is around 3K lines. It may grow to, say, 15K lines in a year. Access pattern: updates are regular, once a day, for the complete set; so both adding lines and updating end date will be done at once. Fetching a random line is indeed by the symbol, could be done few times a second.

Answer 1

I would generate a Python source file every time the data changes, and have that file primarily consisting of a dictionary. This assumes that lookup is by symbol, and that the data readily fit into memory.

data = {
  "AAPL":       ("Jan-1-1985",      "Dec-27-2010"),
...
}

To bulk-update the end date, use pprint.pprint, overwriting the entire file.

Edit: To illustrate how such a file can be written, here is a script that fills it out with random data

import random, string, pprint

def randsym():
    res =[]
    for i in range(4):
        res.append(random.choice(string.uppercase))
    return ''.join(res)

months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 
          'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']
days = range(1,29)
years = range(1980,2010)
def randdate():
    return "%s-%s-%s" % (random.choice(months),
                         random.choice(days),
                         random.choice(years))

data = {}
for i in range(15000):
    data[randsym()] = (randdate(), "Dec-27-2010")

with open("data.py", "w") as f:
    f.write("data=")
    f.write(pprint.pformat(data))

To access the data, do from data import data.