Question

I have a set of data all doubles:
100 rows 20 columns

I’m pulling the data into a IEnumerable list with:

var RowsOfData = File.ReadLines(dll.Globals.OutputDir +     dll.Globals.filename).Select(a => a.Split(',').ToList());

var FilteredRowsToday = (from n in RowsOfData
       where n[1] == 1
       orderby n[0] descending
       select n);

I then have a set of functions, which do simple check on each the data rows and each returns a Bool.
What I want is a count of the number of rows for which each of the functions evaluated true.
And then when I scale my project up I want this processed asap in parallel if possible, I’ve tried:

foreach (var row in FilteredRowsToday) {  
is f1() true, is f2() true 
etc
}

Seems slow
I’ve tried to do in parallel

foreach (var row in FilteredRowsToday.AsParallel())

no faster

I’m now thinking something like:

var TotalTrue = FilteredRowsToday.Select(item => f1() & f2() & f3()).Count();

I can pre-process the data to provide the results of the evaluations of each function as a sort of binary grid if that’s a better stating point?

F1, f2, f3 etc
1, 0, 0 row 1
1, 1, 1 row 2 etc

suggestions welcome!

Answer 1

If you’re just interested in the count where all three functions evaluate to true, then this should be sufficient:

var TotalTrue = FilteredRowsToday.Count(item => f1() & f2() & f3());

As for why it’s slow, your functions could be the reason behind this.

You could try only evaluating the rows until either all three functions return true, or at least one of them return false, e.g.

var TotalTrue = FilteredRowsToday.Count(item => f1() && f2() && f3());

I.e. If f1() evaluates to false, then don’t bother doing the rest of the validations.

UPDATE: If your functions aren’t doing any resource-intensive checks, then parallel LINQ won’t do you much good (more info here).