Recently, I asked (and answered) a question on StackOverflow about why a unit test would work when run by itself and then fail sporadically when run with the whole batch of unit tests. See here: SQL Server and TransactionScope (with MSDTC): Sporadically can't get connection
Unit tests passing when running one at a time and then failing when run together is a classic sign that something is seriously wrong with the code.
I discovered that there is a bit of a resource-leak. Because of a subtle error causing connections to a SQL server to not be released, I was running out of connections and tests were failing. AFAIK, this works almost exactly like a memory leak; connections are allocated from a connection pool and never freed just as memory can be allocated and then not freed.
However, this does leave me with a puzzling question? What is the difference between running my tests one at a time and running them as a suite? If the tests pass when run one at a time and then fail when run together, then there must be some sort of clean-up happening between test runs that happens only when the tests are run one at a time.
I conjecture that this could have something to do with what the .net garbage collector does or doesn’t do between the tests. In one case, connections are freed between tests; in another case, they aren’t.
How can I explain this?
Update: To those of you asking about the specifics of the code, it’s rather simple. I declare a new TransactionScope object in my Setup method and dispose it in my Teardown method. However, The problem test was a data-driven test with 100 test cases; the code under test populated a SqlDataReader object from a select statement using the SqlHelper class and then didn’t call the close method on the SqlDataReader. Because I used the SqlHelper class to get the SqlDataReader, I expected that the connections were handled for me. Not so!
But to clarify, I’m not asking about my specific situation. What I want to know is: generally, how are resources freed between tests? I would imagine this would be some application of the garbage collector. I wonder if the garbage collector could still be cleaning up a previous test as the next test runs (race condition?)
Update: What I know about garbage collection with Unit Tests. Following my own curiosity, I pulled out the unit tests that were failing because a connection was left open by the SqlDataReader object. I tried adding System.GC.Collect() to the end of each test. This successfully freed the connections, but does impose an ~50% performance penalty.
Garbage collection is a periodic background task. In specific, there’s a thread that does nothing but finalize objects that have already been marked as dead. By running one test at a time, you’re giving that thread a chance finalize objects so as to close the connections.