Question

when using the method below for an OR operation, I am getting back duplicate records. do I have to specify a custom IEqualityComparer? a simple distinct() does not work

internal static IQueryable<T> FilterEntity<T>(filters filters, IQueryable<T> entities)
    {
        if (filters.groupOp == "AND")
            foreach (var rule in filters.rules)
                entities = entities.Where<T>(
                    rule.field, rule.data,
                    (WhereOperation)StringEnum.Parse(typeof(WhereOperation), rule.op)
                    );
        else
        {
            //Or
            IQueryable<T> temp = (new List<T>()).AsQueryable();
            foreach (var rule in filters.rules)
            {
                var t = entities.Where<T>(
                    rule.field, rule.data,
                    (WhereOperation)StringEnum.Parse(typeof(WhereOperation), rule.op)
                    );

                    temp = temp.Concat<T>(t).AsQueryable();
            }
            entities = temp;
        }
        return entities;
    }

EDITED AFTER SUGGESTION FROM @usr below – This gives me the correct query in sql profiler (with a distinct) but this starts to look too convoluted – I wd like a cleaner solution

    internal static IQueryable<T> FilterEntity<T>(filters filters, IQueryable<T> entities)
    {
        if (filters.groupOp == "AND")
            foreach (var rule in filters.rules)
                entities = entities.Where<T>(
                    rule.field, rule.data,
                    (WhereOperation)StringEnum.Parse(typeof(WhereOperation), rule.op)
                    );
        else
        {
            //Or
            var t1 = entities.Where<T>(filters.rules[0].field,filters.rules[0].data,
                (WhereOperation)StringEnum.Parse(typeof(WhereOperation),filters.rules[0].op)
                );
            for (int i = 1; i<filters.rules.Count(); i++)
            {

                var t = t1.Where<T>(filters.rules[i].field, filters.rules[i].data,
                (WhereOperation)StringEnum.Parse(typeof(WhereOperation), filters.rules[i].op)
                );
              t1.Concat<T>(t).AsQueryable();
            }
           entities  = t1;
        }
        return entities.Distinct<T>();
    }

Answer 1

This is not related to Distinct() or IEqualityComparer. It is the modified closure gotcha, that is: the loop variable rule must be copied in the loop body:

foreach (var rule in filters.rules)
{
    var rule1 = rule;
    // work with rule1 only.

You can follow Usr’s advice by doing this:

IQueryable<T> temp = null;
    ....
    foreach (var rule in filters.rules)
    {
        var rule1 = rule;
        var t = entities.Where<T>(rule1.field, rule1.data,
            (WhereOperation)StringEnum.Parse(typeof(WhereOperation), rule1.op));

            if (temp == null)
                temp = t;
            else
                temp = temp.Union(t); // Union!!
    }
}
return temp;

I wonder if it solves your issue. Note the use of Union (which is an implicit Distinct). If it does not solve your issue I think there is some code invisible to us (e.g. in your WhereOperation) that interferes.