Question

I’m working on a project at work where there’s a performance issue with the code.

I’ve got some changes I think will improve performance, but no real way of gauging how my changes affect it.

I wrote a unit test that does things the way they’re currently implemented, with a Stopwatch to monitor how fast the function runs. I’ve also wrote a similar unit test that does things slightly differently.

If the tests are ran together, one takes 1s to complete, the other takes 73 ms.
If the tests are ran separately, they both take around 1s to complete (yea.. that change i made didn’t seem to change much).

If the tests are identical, I have the same issue, one runs faster than the other.

Is visual studio doing something behind the scenes to improve performance? Can I turn it off if it is?

I’ve tried moving tests into different files, which didn’t fix the issue I’m having.

I’d like to be able to run all the tests, but have them run as if there’s only one test running at a time.

Answer 1

My guess: it’s likely down to dll loading and JIT compiling

1. Assembly loading.

.NET lazily loads assemblies (dll’s). If you add reference to FooLibrary, it doesn’t mean it gets loaded when your code loads.
Instead, what happens is that the first time you call a function or instantiate a class from FooLibrary, then the CLR will go and load the dll it lives in. This involves searching for it in the filesystem, possible security checks, etc.
If your code is even moderately complex, then the "first test" can often end up causing dozens of assemblies to get loaded, which obviously takes some time.
Subsequent tests appear fast because everything’s already loaded.

2. JIT Compiling

Remember, your .NET assemblies don’t contain code that the CPU can directly execute. Whenever you call any .NET function, the CLR takes the MSIL bytecode and compiles it into executable machine code, and then it goes and runs this machine code. It does this on a per-function basis.
So, if you consider that the first time you call any function, there will be a small delay while it JIT compiles, these things can add up. This can be particularly bad if you’re calling a lot of functions or initializing a big third party library (think entity framework, etc).
As above, subsequent tests appear fast, because many of the functions will have already been JIT compiled, and cached in memory.

So, how can you get around this?

You can improve the assembly loading time by having fewer assemblies. This means fewer file searches and so on. The microsoft .NET performance guidelines go into more detail.
Also, I believe installing them in the global assembly cache may (??) help, but I haven’t tested that at all so please take it with a large grain of salt.
Installing into the GAC requires administrative permissions and is quite a heavyweight operation. You don’t want to be doing it during development, as it will cause you problems (assemblies get loaded from the GAC in preference to the filesystem, so you can end up loading old copies of your code without realizing it).

You can improve the JIT time by using ngen to pre-compile your assemblies. However, like with the GAC, this requires administrative permissions and takes some time, so you do not want to do it during development either.

My advice?

Firstly, measuring performance in unit tests is not a particularly good or reliable thing to be doing. Who knows what else visual studio is doing in the background that may or may not affect your tests.

Once you’ve got your code you’re trying to benchmark out into a standalone app, have it loop and run all the tests twice, and discard the first result 🙂