Question

Eqatec shows several thousand anonymous method closures being called each time a method is called that includes a simple LINQ ‘Where’ statement in my program. Pseudo code example:

Class1
{
    //foo and bar are both EF model classes
    List<foo> aList; // n = 2000
    List<bar> bList; // n = ~4000

    void aMethod() 
    {  
        foreach (var item in aList)
        {
            Class2.DoSomeWork(item, bList);
        }
    }
}

Class2
{
    static void DoSomeWork(foo item, List<bar> bList)
    {
     var query = bList.where(x => x.prop1 == item.A && x.prop2 = item.B).toList(); // <--- Calls thousands of anonymous method closures each method call.

     if (query.any()) <--- Calls only 1 anonymous method closure.
        DoSomethingElse(); 
    } 
}

I don’t understand why 2,000 calls to ‘DoSomeWork’ called some 8 million anonymous method closures (even 1 causes several thousand).

As a fix, I simply rewrote the statement without using LINQ which eliminated the need for closures and produced a 10x fold increase in performance.

I would still like to understand why this occurred in the first place if anyone has some theories they would like to share.

Answer 1

I think the 8M is referring to the number of times a method was executed on a closure class, and not the number of closure instances created. Firstly, let’s make the code compile:

class Class2
{
    public static void DoSomeWork(foo item, List<bar> bList)
    {
        var query = bList.Where(x => x.prop1 == item.A && x.prop2 == item.B)
                         .ToList();

        if (query.Any())
            DoSomethingElse();
    }
    static void DoSomethingElse() { }
}
class foo { public int A { get; set; } public int B { get; set; } }
class bar { public int prop1 { get; set; } public int prop2 { get; set; } }

Now, we can discard the original ” // <— Calls only 1 anonymous method closure.” comment, because actually no anonymous method closures are used by the .Any() – that just checks whether a list has contents: no closures required.

Now; let’s manually rewrite the closure to show what is happening in the compiler:

class Class2
{
    class ClosureClass
    {
        public foo item; // yes I'm a public field
        public bool Predicate(bar x)
        {
            return x.prop1 == item.A && x.prop2 == item.B;
        }
    }
    public static void DoSomeWork(foo item, List<bar> bList)
    {
        var ctx = new ClosureClass { item = item };
        var query = bList.Where(ctx.Predicate).ToList();

        if (query.Any()) {
            DoSomethingElse();
        }
    }
    static void DoSomethingElse() { }
}

You can see that 1 ClosureClass is created per DoSomeWork, which maps directly to how the only captured variable (item) is scoped at the method level. The predicate (ctx.Predicate) is obtained once (only), but is invoked for every item in bList. So indeed, 2000 * 4000 is 8M calls to a method; however, 8M calls to a method is not necessarily slow.

However! I think the biggest problem is that you are creating a new list just to check for existence. You don’t need that. You can make your code much more efficient by moving the Any earlier:

if (bList.Any(x => x.prop1 == item.A && x.prop2 == item.B)) {
    DoSomethingElse();
}

This now only invokes the predicate enough times until a match is found, which we should anticipate to be less than all of them; it also doesn’t fill a list unnecessarily.

Now; yes, it will be be a bit more efficient to do this manually, i.e.

bool haveMatch = false;
foreach(var x in bList) {
    if(x.prop1 == item.A && x.prop2 == item.B) {
        haveMatch = true;
        break;
    }
}
if(haveMatch) {
    DoSomethingElse();
}

but note that this change between Any and foreach is not the critical difference; the critical difference is that I’ve removed the ToList() and the “keep reading, even if you’ve already found a match”. The Any(predicate) usage is a lot more concise and is easy to read etc. It isn’t typically a performance issue, and I doubt it is here either.