Question

Using Python I managed to make myself a kind of dictionary of terms and their meaning and it’s rather big – x00,000 items (can’t estimate right now as they are stored in multiple files by first letter).
Files are pickled dictionary objects with this structure:

dict{word, (attribute,
            kind,
            [meanings],
            [examples],
            [connections]
            )
    }

If it matters it’s Python dictionary object, with key as string and value as tuple, and then this tuple consists of either string or list objects.

Now I plan to put them all in sqlite3 database as it’s easy with Python. Before I do that I thought to ask for advice if sqlite3 if good choice as I’ve never done any real database task before.

I know that answer depends of what I want to do with this data (besides it’s structure), but let’s say I just want it to be stored locally in one place (file) and be reasonable easy to access (query) and possibly transform.

Answer 1

Yes, I’ve used sqlite3 for this kind of thing. The dictionary values had to first be pickled though:

import sqlite3
import pickle
import collections

class DBDict(collections.MutableMapping):
    'Database driven dictlike object (with non-persistent in-memory option).'

    def __init__(self, db_filename=':memory:', **kwds):
        self.db = sqlite3.connect(db_filename)
        self.db.text_factory = str
        try:
            self.db.execute('CREATE TABLE dict (key text PRIMARY KEY, value text)')
            self.db.execute('CREATE INDEX key ON dict (key)')
            self.db.commit()
        except sqlite3.OperationalError:
            pass                # DB already exists
        self.update(kwds)

    def __setitem__(self, key, value):
        if key in self:
            del self[key]
        value = pickle.dumps(value)
        self.db.execute('INSERT INTO dict VALUES (?, ?)', (key, value))
        self.db.commit()

    def __getitem__(self, key):
        cursor = self.db.execute('SELECT value FROM dict WHERE key = (?)', (key,))
        result = cursor.fetchone()
        if result is None:
            raise KeyError(key)
        return pickle.loads(result[0])

    def __delitem__(self, key):
        if key not in self:
            raise KeyError(key)
        self.db.execute('DELETE FROM dict WHERE key = (?)', (key,))
        self.db.commit()

    def __iter__(self):
        return iter([row[0] for row in self.db.execute('SELECT key FROM dict')])

    def __repr__(self):
        list_of_str = ['%r: %r' % pair for pair in self.items()]
        return '{' + ', '.join(list_of_str) + '}'

    def __len__(self):
        return len(list(iter(self)))



>>> d = DBDict(raymond='red', rachel='blue')
>>> d
{'rachel': 'blue', 'raymond': 'red'}
>>> d['critter'] = ('xyz', [1,2,3])
>>> d['critter']
('xyz', [1, 2, 3])
>>> len(d)
3
>>> list(d)
['rachel', 'raymond', 'critter']
>>> d.keys()
['rachel', 'raymond', 'critter']
>>> d.items()
[('rachel', 'blue'), ('raymond', 'red'), ('critter', ('xyz', [1, 2, 3]))]
>>> d.values()
['blue', 'red', ('xyz', [1, 2, 3])]

The above will keep you database in a single file. You can navigate the object like a regular python dictionary. Since the values are pickled in a single field, sqlite won’t give you any additional query options. Other flatfile storage will have similar restrictions. If you need to write queries that traverse a hierarchical structure, consider using a NoSQL database instead.