Question

I tried to create class with similar implementation.
One using boxing/unboxing at constructor, while other using Generics.

Then tried to create the object for both class 1,000,000 times.
when i run the ‘for loop’ for Obj class first i am always getting longer time for Obj creation, But when i run the ‘for loop’ for Gen class first 75% times i am getting longer time for Obj. Please help me why this behavior.
Here is the code:-

    class Obj
{
    public string _t;
    public int _u;

    public Obj(Object t, Object u) //Boxing taking place implicitly
    {
        _t = (string)t; 
        _u = (int)u; //unboxing
    }
}
class Gen<T, U> 
{
    public T _t;
    public U _u;

    public Gen(T t, U u)
    {
        _t = t;
        _u = u;
    }
}
static void Main(string[] args)
    {
        long before_Obj = DateTime.Now.Ticks;
        for (int i = 0; i < 1000000; i++)
        {
            new Obj("hi", i);
        }
        long after_Obj = DateTime.Now.Ticks - before_Obj;
        Console.WriteLine(after_Obj);

        long before_Gen = DateTime.Now.Ticks;
        for (int i = 0; i < 1000000; i++)
        {
            new Gen<string, int>("hi", i);
        }
        long after_Gen = DateTime.Now.Ticks - before_Gen;
        Console.WriteLine(after_Gen);
    }

OUTPUT:
2656250
1406250

Answer 1

This is a poorly designed and poorly implemented test, which is probably why you are getting strange results. Consider some of the ways in which the test is poorly designed.

1) You lump jitter costs in with regular costs. The jitter only runs once for each method, but running the jitter can be quite expensive, particularly compared to the cost of a very simple method — like these. I have seen benchmarks in which the jitter cost dominated the cost of every subsequent run. You should run the test scenarios at least once before you start your measuring loop.

2) DateTime is designed to measure times on the scale of “is it time for lunch”, not times on the sub-microsecond scale. Always use Stopwatch for a high-precision timer.

3) The first test creates two million objects — a million “Obj” and a million boxed integers. That means that there might be as many as two million objects for the garbage collector to deal with by the time the second test runs, which then allocates another million objects. If that triggers a collection, then you’ve got a cost incurred by the first test that is actually being charged to the second test! If we’re testing which one of the two of us can make a cake faster, and I go first and leave the messy kitchen for you to clean up, of course you’re going to be slowed down.

Moreover, the second test creates another million objects, but perhaps the garbage collector does not run until after the test is completed; now you’ve taken a cost that should have been charged to the second test and are failing to account for it anywhere.

Either write two different programs, so that the GC cost of one is not charged to the other, or force a full collection after each test, and charge the cost of the collection to the test. Remember, the cost of boxing is not just the allocation and the copy; it’s also the collection pressure and the cleanup.

4) The test is not real-world, which creates all kinds of problems. You are creating millions of objects, which means that a significant fraction of the cost of the program will be dealing with the collection pressure caused by all those objects. Is that realistic? In a realistic program you might not be creating millions of these objects, and some other factor might be the thing producing collection pressure.

5) Nothing useful is done with any object that is created; the objects are only being created for the purpose of observing the side effect of the time needed to create and collect them. The jitter is permitted to realize this fact — it can do a flow analysis and determine that the result of the ctor call is never observed, and elide the whole thing, reducing its cost to zero. Clearly it does not do so here, but it could in some other version of the jitter. I have seen plenty of performance tests where people concluded that X was much faster than Y when in fact the cost of X was zero because the jitter removed the never-observed computation entirely.

And of course it should go without saying — but unfortunately it does not — do not under any circumstances run your test program in the debugger. Because then you are measuring the cost of running the debugger as much as the cost of running the program. The jitter de-optimizes programs that are running in the debugger to make them easier to debug. Test under realistic conditions; your customers won’t be running your program in the debugger, so your performance tests should not run in the debugger either.