Question

I’m trying to test what built-in collections perform best under certain applications, such as intersection. To do so, I built the following test:

private static void Main(string[] args)
{
    LoadTest<HashSet<object>>();
    ClearEverythingHere(); // <<-- what can go here?
    LoadTest<LinkedList<object>>();
    Console.ReadKey(true);
}

private static void LoadTest<T>() where T : ICollection<object>, new()
{
    const int n = 1 << 16;
    const int c = 1 << 3;

    var objs = new object[n << 1];
    for (int i = 0; i < n << 1; i++)
        objs[i] = new object();

    var array = new T[c];
    var r = new Random(123);
    for (int s = 0; s < c; s++)
    {
        array[s] = new T();
        for (int i = 0; i < n; i++)
            array[s].Add(objs[r.Next(n << 1)]);
    }

    var sw = Stopwatch.StartNew();
    IEnumerable<object> final = array[0];
    for (int s = 1; s < c; s++)
        final = final.Intersect(array[s]);
    sw.Stop();
    Console.WriteLine("Ticks elapsed: {0}", sw.ElapsedTicks);
}

If I uncomment both test methods from Main, the second test always completes much faster than the first, no matter which order I test the structures. Generally, the first intersection runs in a few hundred ticks, and the second finishes in less than ten. I would have thought having the tests in completely separate scopes would have prevented at least some of the (what I’m presuming is) caching that leads to such different results.

Is there an easy way to reset the application so that I don’t have to worry about caching or optimizing for testing? I would like to be able to run one test, print the results, clear it out, and run another test? Yes, I could comment and uncomment, or possibly spawn two separate applications, but that’s a lot of work for simple console tests.

---

Edit: I’ve modified the tests as per the suggestions in the answers.

private static void Main(string[] args)
{
    const int n = 1 << 17;
    const int c = 1 << 4;

    var objs = new Item[n << 1];
    for (int i = 0; i < (n << 1); i++)
        objs[i] = new Item(i);

    var items = new Item[c][];
    var hash = new HashSet<Item>[c];
    var list = new LinkedList<Item>[c];

    var r = new Random();
    for (int s = 0; s < c; s++)
    {
        items[s] = new Item[n];
        for (int i = 0; i < n; i++)
            items[s][i] = objs[r.Next(n << 1)];
        hash[s] = new HashSet<Item>(items[s]);
        list[s] = new LinkedList<Item>(items[s]);
    }

    Stopwatch stopwatch = Stopwatch.StartNew();
    HashSet<Item> fHash = hash[0];
    for (int s = 1; s < hash.Length; s++)
        fHash.IntersectWith(hash[s]);
    stopwatch.Stop();

    Console.WriteLine("Intersecting values: {0}", fHash.Count);
    Console.WriteLine("Ticks elapsed: {0}", stopwatch.ElapsedTicks);

    stopwatch = Stopwatch.StartNew();
    IEnumerable<Item> iEnum = list[0];
    for (int s = 1; s < list.Length; s++)
        iEnum = iEnum.Intersect(list[s]);
    Item[] array = iEnum.ToArray();
    stopwatch.Stop();

    Console.WriteLine("Intersecting values: {0}", array.Length);
    Console.WriteLine("Ticks elapsed: {0}", stopwatch.ElapsedTicks);
    Console.ReadKey(true);
}

[DebuggerDisplay("Value = {_value}")]
private class Item
{
    private readonly int _value;

    public Item(int value)
    {
        _value = value;
    }

    public override bool Equals(object obj)
    {
        if (ReferenceEquals(null, obj))
            return false;
        if (ReferenceEquals(this, obj))
            return true;
        if (obj.GetType() != typeof(Item))
            return false;
        return Equals(obj);
    }

    public override int GetHashCode()
    {
        return _value;
    }

    public override string ToString()
    {
        return _value.ToString();
    }
}

This solved most of my problems. (And if you’re wondering, HashSet.IntersectWith with appears much faster than IEnumerable.Intersect.)

Answer 1

There are few errors in your code.

1. Intersects is LINQ functions, so it means it is lazily evaluated. That means it gets executed only when the data is accesed. This can be done by either looping over the data or calling ToList or ToArray on this enumerable. By adding this, you get different result
2. Testing must be always done on same data. Try creating your data outside your test method and pass it as parameter.
3. First pass of code is usualy considered wrong, because JITing and such.
4. Try creating your own object and override Equals and GetHashCode. Like this it might not be correct to test it.