Question

I have a program that reads objects (Tasks) from either a CSV file or a DB. Both source have in common that you must explicitely close access to the resource after using it.

I followed the approach of making both the CSV and the DB class iterable, so iterating over them returns tasks.
This is handy for using them, however I’m not convinced it’s clean, and I have the following questions :

• What is the best way to get access to the file or DB ? I use the constructor for that, but I’m not sure about multithreading, etc…
• what is the best way to close the resource (file, cursor). Should the external object notify the access is done, or should the CSV or DB object detect we’re at the end of the file and close it ?

I’m not sure I’m doing this right (it works for single run, but I mean this to be plugged to a website, so with multiple access)

class CSV(AbstractDAO):
    def __init__(self, sourcePath):
        self.sourcePath = sourcePath
        self.csvFile = codecs.open(sourcePath, 'rb', 'UTF-8')

    def __iter__(self):
        return self

    def next(self):
        return self._buildTaskFromLine(self.csvFile.next())

    def deleteAllTasks(self):
        pass

    def loadTask(self, taskID):
        csvFile = codecs.open(self.sourcePath, 'rb', 'UTF-8')
        for line in csvFile:
            taskValues = line.split(";")
            if taskValues[0] == unicode(taskID):
                return self._buildTaskFromLine(line)
            else:
                return None

    def saveTask(self, task):
        pass

    def loadPredecessorsID(self, task):
        csv = codecs.open(self.sourcePath, 'rb', 'UTF-8')
        for line in csv:
            taskValues = line.split(";")
            if taskValues[0] == unicode(task.id):
                return taskValues[2].split(",")
        return None

    def _buildTaskFromLine(self, line):
        taskValues = line.split(";")
        taskID = taskValues[0]
        taskName = taskValues[1]
        taskAncestors = taskValues[2]
        taskDuration = int(taskValues[3])
        return Task(taskID, taskName, taskDuration)

Here is the DB implementation

class SQLite(AbstractDAO):
    def __init__(self, sourcePath):
        self.connection = sqlite3.connect(sourcePath)
        self.cursor = None

    def __iter__(self):
        self.cursor = self.connection.cursor()
        self.cursor.execute("select * from Tasks")
        return self

    def next(self):
        if self.cursor is not None:
            row = self.cursor.fetchone()
            if row is None:
                self.cursor.close()
                raise StopIteration
            else:
                return self._buildTaskFromRow(row)

    def deleteAllTasks(self):
        cursor = self.connection.cursor()
        cursor.execute("delete from Tasks")
        self.connection.commit()
        cursor.close()

    def loadTask(self, id):
        cursor = self.connection.cursor()
        param = (id,)
        cursor.execute("select * from Tasks t where t.id = ? ", param)
        taskRow = cursor.fetchone()
        task = self._buildTaskFromRow(taskRow)
        cursor.close()
        return task

    def saveTask(self, task):
        cursor = self.connection.cursor()
        param = (task.id,)
        cursor.execute("select * from Tasks t where t.id = ? ", param)
        taskRow = cursor.fetchone()
        if taskRow is None:
            param = (task.id, task.name, task.duration)
            cursor.execute("insert into Tasks values (?,?,?)", param)
            self.connection.commit()
            cursor.close()

        else:
            param = (task.id, task.name, task.duration)
            cursor.execute("update Tasks \
            set description = ?, duration = ?  \
            where id = ? ", param)
            self.connection.commit()
            cursor.close()

    def loadPredecessors(self, task):
        pass

    def _buildTaskFromRow(self, row):
        taskId = row[0]
        taskName = row[1]
        taskDuration = row[2]
        return Task(taskId, taskName, taskDuration)

Finally, the code above is for instance called like this by ma TaskTree (it’s an object holding all tasks)

def loadTreeFrom(self, source, sourcePath):
    if source not in ('CSV', 'DB'):
        raise AttributeError('Unknown source : supported sources are CSV or DB')

    dao = None
    if source == 'CSV':
        dao = CSV(sourcePath)
    elif source == "DB":
        dao = SQLite(sourcePath)

    #populate the tasks first
    for task in dao:
        self.tasks[unicode(task.id)] = task

    # then populate the dependencies
    for item in self.tasks.iteritems():
        ancestorsID = dao.loadPredecessorsID(item[1])
        self.addDependencies(item[1], ancestorsID)

Answer 1

This is sort of a sideways answer to your question, but based on your description, I think you should consider making these objects into context managers. That way, instead of having your “external object notify the access is done,” you can simply use a with block. When the block is entered, your context manager object’s __enter__ method is called; when it is exited, your context manager object’s __exit__ method is called. Here’s a (very) simple example:

>>> class DummyManager(object):
...     def __enter__(self):
...         print 'entering with block!'
...         return 'foo'
...     def __exit__(self, exc_type, exc_val, exc_tb):
...         print 'exiting with block!'
...         print 'this is exception info, if an exception was raised:'
...         print exc_type, exc_val, exc_tb
... 
>>> with DummyManager() as dummy:
...     print dummy
... 
entering with block!
foo
exiting with block!
this is exception info, if an exception was raised:
None None None

This is a nice way for the iterating object to let your DB/CVS object know that it’s no longer needed.

Honestly, I have no precise idea how you should handle concurrent access, etc — since I don’t know your overall design. But the __enter__ and __exit__ methods are potentially good places to handle locks, etc., if you need them.

One way to restructure your classes based on this system would be to write the methods assuming that the resource is open, instead of opening and closing it all the time. Then always refer to instances of the object within a with block; the with statement takes care of initializing and opening resources, and closing them when control leaves the block. So for example your loadTask and saveTask methods would no longer require x.open(...) and x.close() lines at the beginning and end, and the resource is open exactly as long as it is being used.

If you like, you can create public open and close methods, and then just have __enter__ and __exit__ call them. Then your user (or you) can decide whether to use a with block or to open and close the object in the classic style. Either way, the object behaves an a way analogous to a file in Python. (And did I mention that files are also context managers?)

Based on your new code, you would then call the resources something like this:

def loadTreeFrom(self, source, sourcePath):
    if source not in ('CSV', 'DB'):
        raise AttributeError('Unknown source : supported sources are CSV or DB')

    dao = None
    if source == 'CSV':
        dao = CSV            # dao just a reference to the class now
    elif source == "DB":
        dao = SQLite

    with dao() as tasks:
        #populate the tasks first
        for task in tasks:
            self.tasks[unicode(task.id)] = task

        # then populate the dependencies
        for item in self.tasks.iteritems():
            ancestorsID = dao.loadPredecessorsID(item[1])
            self.addDependencies(item[1], ancestorsID)

Now when you call loadPredecessorsID, it doesn’t open and close the resource every time.