Question

I have an extension method which takes in two list and compares them for modifications then outputs a new list. Here is the code

public static List<Member> GetModifiedRecords(this List<Member> LocalMemberData, List<Member> RemoteMemberData)
    {
        var result = (from localdata in LocalMemberData
                      from remotedata in RemoteMemberData
                      where
                      ((
                      localdata.Card != remotedata.Card ||
                      localdata.DateJoined != remotedata.DateJoined ||
                      localdata.DatePaidUpTo != remotedata.DatePaidUpTo ||
                      localdata.Forename != remotedata.Forename ||
                      localdata.Postcode != remotedata.Postcode ||
                      localdata.State != remotedata.State ||
                      localdata.Street != remotedata.Street ||
                      localdata.Surname != remotedata.Surname ||
                      localdata.Title != remotedata.Title ||
                      localdata.Town != remotedata.Town
                      )
                      && (localdata.MemberNumber == remotedata.MemberNumber
                      ))
                      select localdata).Distinct();

        List<Member> modifiedMembers = new List<Member>(result);
        return modifiedMembers;
    } 

What’s strange is when running it fails on the line

List<Member> modifiedMembers = new List<Member>(result);

With the error

“The CLR has been unable to transition from COM context 0x3b4668 to COM context 0x3b44f8 for 60 seconds. The thread that owns the destination context/apartment is most likely either doing a non pumping wait or processing a very long running operation without pumping Windows messages. This situation generally has a negative performance impact and may even lead to the application becoming non responsive or memory usage accumulating continually over time. To avoid this problem, all single threaded apartment (STA) threads should use pumping wait primitives (such as CoWaitForMultipleHandles) and routinely pump messages during long running operations.”

FYI both Lists that are being compared have over 100,000 records. Am I thinking about this wrong?

Answer 1

Consider a join, instead of filtering the cartesian product. For instance:

var r = new Random();
var list1 = Enumerable.Range(0,10000).OrderBy(_=>r.Next()).ToList();
var list2 = Enumerable.Range(0,10000).OrderBy(_=>r.Next()).ToList();
var sw = Stopwatch.StartNew();
var c1 = from x1 in list1 from x2 in list2 where x1==x2 select x1;
var j1 = c1.ToList();
sw.ElapsedMilliseconds.Dump();
sw=Stopwatch.StartNew();
var c2 = from x1 in list1 join x2 in list2 on x1 equals x2 select x1;
var j2 = c2.ToList();
sw.ElapsedMilliseconds.Dump();

Gives timings of

3584
1

i.e. joins are super fast (complexity O(n)), your pseudo-join isn’t (complexity O(n²)).

At 100000 items, my machine choked (I quit after 5 minutes) on the cartesian product filtering approach, but completed the join in 21ms.

So, rewriting your query as follows should really turbo things up:

(from localdata in LocalMemberData
join remotedata in RemoteMemberData 
on localdata.MemberNumber equals remotedata.MemberNumber
where
(
localdata.Card != remotedata.Card ||
localdata.DateJoined != remotedata.DateJoined ||
localdata.DatePaidUpTo != remotedata.DatePaidUpTo ||
localdata.Forename != remotedata.Forename ||
localdata.Postcode != remotedata.Postcode ||
localdata.State != remotedata.State ||
localdata.Street != remotedata.Street ||
localdata.Surname != remotedata.Surname ||
localdata.Title != remotedata.Title ||
localdata.Town != remotedata.Town
)
select localdata).Distinct()