Question

I have built little custom web framework on top of Python 3.2 using Cherrypy to built WSGI application and SQLAlchemy Core (just for connection pooling and executing text SQL statements).

Versions I am using:

• Python: 3.2.3
• CherryPy: 3.2.2
• SQL Alchemy: 0.7.5
• Psycopg2: 2.4.5

For every request, a DB connection is retrieved from pool using sqlalchemy.engine.base.Engine´s connect method. After request handler finishes, the connection is closed using close method. Pseudocode for example:

with db.connect() as db:
    handler(db)

Where db.connect() is context manager defined like this:

@contextmanager
def connect(self):
    conn = self.engine.connect()
    try: 
        yield conn
    finally:
        conn.close()

I hope that this is correct practice for doing this task. It worked until things went more complicated in page handlers.

I am getting weird behavior. Because of uknown reason, connection is sometimes closed before the handler finishes it´s work. But not every time!

By observation, this happens only when making requests quickly consecutively. If I make small pause between requests, the connection is not closed and request is finished successfully. But anyway, this does not happen every time. I have not found more specific pattern in failures/successes of requests.

I observed that the connection is not closed by my context manager. It is already closed at that point.

My question:
How to figure out when, why and by what code is my connection closed?

I tried debugging. I put breakpoint on sqlalchemy.engine.base.Connection´s close method but the connection is closed before it reach this code. Which is weird.

I will appreciate any tips or help.

*edit *
Information requested by zzzeek:

symptom of the “connection being closed”:
Sorry for not clarifying this before. It is the sqlalchemy.engine.Connection that is closed.

In handlers I am calling sqlalchemy.engine.base.Connection´s execute method to get data from database (select statements). I can say that sqlalchemy.engine.Connection is closed, because I am checking it’s closed property before calling execute.

I can post here traceback, but only thing that you will probably see in it is that Exception is raised before the execute in my DB wrapper library (because connection is closed).

If I remove this check (and let the execute method execute), SQLAlchemy raises this exception: http://pastebin.com/H6052yca

Regarding the concurency problem that zzzeek mentioned. I must apologize. After more observation the situation is slightly different.

This is exact procedure how to invoke the error:

Request for HandlerA. Everything ok. 
Wait moment (about 10-20s).
Request for HandlerB. Everything ok.

Request for HandlerA. Everything ok.
Immediate request for HandlerB. Error!
Immediate request for HandlerB. Error!
Immediate request for HandlerB. Error!
Wait moment (about 10-20s).
Request for HandlerB. Everything ok.

I am using default SQLAlchemy pooling class with pool_size = 5.

I know that you cannot do miracles when you don’t have the actual code. But unfortunately, I cannot share it. Is there any best practice for debugging this type of error? Or the only option is to debug more deeply step by step and try to figure it out?

Another observation:

When I start the server in debugger (WingIDE), I cannot bring up the error. Probably because the the debugger is so slow when interpreting the code, that the connection is somehow “repaired” before second request (RequestB) is handled.

Answer 1

After daylong debugging. I found out the problem.

Unfortunatelly it was not related to SQLAlchemy directly. So the question should be deleted. But you guys tried to help me, so I will answer my own question. And maybe, somebody will find this helpfull some day.

Basically, Error was caused by my custom publish/subscribe methods which did not play nicely in multi threaded enviorment.

I tried stepping code line by line… which was not working (as i described in the question). So I started generating very detailed log of what is going on.

Even then, everything looked normal, until I noticed that few lines before crash, the address of Connection object referenced in the model changed. Which practically meant that something assigned another Connection object to model and that connection object was already closed.

So the lesson is. When everything looks correct, print out / log the repr() of objects which are problematic.

Thanks to commenters for their time.