Question

Background

We have a C#/VB.net client application consuming a WCF service which connects to an Oracle database. The web service connects to the database using the .NET framework’s data provider for Oracle (not to be confused with ODP). Our testers have experienced sporadic Oracle account locking which seems to happen shortly after changing the user’s Oracle password. The dba_audit_trail logs have revealed what seem to be automated connection attempts at very regular intervals (i.e. every two minutes on the dot) without any user or client activity. Numerous logs (IIS, WCF tracing, message logging, etc.) have confirmed these connection attempts are not initiated directly by the client application; they must be coming independently from the web service or from inside the System.Data.OracleClient library. The automatic attempts continue forever until the web service’s worker process (single worker) dies from inactivity.

In some instances, these automatic attempts get started before the password change, and they connect successfully to the database, but as soon as the password changes, the next attempt fails for invalid username/password. After three attempts, the account gets locked. We are trying to find the source of these periodic connection attempts.

I found a very similar, but unanswered, problem on Oracle’s forum here.

Current Thoughts

Our latest investigations have led us to believe it is an unexpected behavior from connection pooling. If the user connected to the web service before the password change, a connection pool would be created for the original connection string. After changing the password and logging back into the web service, the data provider would create a new connection pool based on the new connection string.

Could something inside the data provider be trying to keep the old connection alive from the first connection pool? Perhaps the first connection pool is discarding the old connection and attempting to replenish it with a new one (with the now invalid connection string). What could cause this? Note: we are using the default settings for min/max pool size (0/100).

We do not believe our code is directly attempting to access a connection from the first connection pool. The user’s session does not have any memory of the previous session’s password, and therefore would not be using the old connection string to reference the first connection pool. Additionally, nothing in our code would explain the very precise connection intervals we are seeing.

Answer 1

The underlying problem ended up being unreleased database connections. When a connection is opened, it gets checked out of the connection pool. If the connection is never closed, the pool thinks it is still being used. This causes the pool management logic to periodically re-authenticate with the database using the original connection string. When the password changes, this quickly leads to failed login attempts and account locking.

// Problem logic; connection is never closed/returned to the connection pool.
public static void ConnPoolTest1()
{
    OracleConnection conn = new OracleConnection(connectionStringWithPooling);
    conn.Open();

    //...Do some work

    // Sit on this line for 5-10 minutes and examine Oracle's dba_audit_trail.
    Console.ReadKey(); // Since connection was never released back to the connection pool, the
                       // data provider's pool management will regularly re-authenticate with DB.
                       // If user's password changes before this process dies (releasing the
                       // connection pools), you start accumulating failed password attempts.
}

The proper fix for this problem is to ensure connections are always returned to the pool when you are done with them!

// Best practice: ALWAYS CLOSE YOUR CONNECTIONS WHEN YOU ARE DONE!
public static void ConnPoolTest2()
{
    OracleConnection conn = new OracleConnection(connectionStringWithPooling);
    conn.Open();

    //...Do some work

    conn.Close();

    // Sit on this line for 5-10 minutes and examine Oracle's dba_audit_trail.
    Console.ReadKey(); // No problem here! No recurring authentication attempts because the
                       // connection has been returned to the pool.
}

NOTE: Other answers suggested turning off pooling and clearing old connection pools when the password changed. These suggestions worked for us as a temporary patch while we searched for the unreleased resources, and they greatly helped us to isolate the problem.