Question

I am storing the values of the form given below into a file:

143   800 'Ask'
213   457 'Comment'
424   800 'Hi'

The first column contains unique elements here.
However, the look up on the values of the first column is quite inefficient when I am storing it in file format, is there a more efficient way in Python for a faster look-up.
I am aware of dictionaries in python for accomplishing this, but I am looking for some other method. Since the data I have consits of trillions of records..therefore I can not keep them in dictionary in the RAM. Therefore, I am searching for some other method.
Also with each program exection the rows are going to be inserted in the case of databases, how to overcome that, an example of what I am getting confused about in databases is given below:

 143   800 'Ask'
 213   457 'Comment'
 424   800 'Hi'
 143   800 'Ask'
 213   457 'Comment'
 424   800 'Hi'

Answer 1

Here’s a full code example using sqlite3, showing how to initialise the database, put data into it, and get a single row of data out.

import sqlite3

conn = sqlite3.connect(':memory:')
conn.execute("""CREATE TABLE Widget (id INTEGER PRIMARY KEY,
                                     serial_number INTEGER,
                                     description TEXT);""")

my_data = [ [143, 800, 'Ask'],
            [213, 457, 'Comment'],
            [424, 800, 'Hi'] ]

for row in my_data:
    conn.execute("INSERT INTO Widget (id, serial_number, description) VALUES (?,?,?);" , row )

conn.commit() # save changes

res = conn.execute("SELECT * FROM Widget WHERE id=143")
row = res.next()
print row #prints (143, 800, u'Ask')

Note the use of the special filename :memory: to open a temporary database.