Question

I have two entities, Class and Student, linked in a many-to-many relationship.

When data is imported from an external application, unfortunately some classes are created in duplicate. The ‘duplicate’ classes have different names, but the same subject and the same students.

For example:

{ Id = 341, Title = ’10rs/PE1a’, SubjectId = 60, Students = { Jack, Bill, Sarah } }

{ Id = 429, Title = ’10rs/PE1b’, SubjectId = 60, Students = { Jack, Bill, Sarah } }

There is no general rule for matching the names of these duplicate classes, so the only way to identify that two classes are duplicates is that they have the same SubjectId and Students.

I’d like to use LINQ to detect all duplicates (and ultimately merge them). So far I have tried:

var sb = new StringBuilder();
using (var ctx = new Ctx()) {
  ctx.CommandTimeout = 10000; // Because the next line takes so long!
  var allClasses = ctx.Classes.Include("Students").OrderBy(o => o.Id);
  foreach (var c in allClasses) {
    var duplicates = allClasses.Where(o => o.SubjectId == c.SubjectId && o.Id != c.Id && o.Students.Equals(c.Students));
    foreach (var d in duplicates)
      sb.Append(d.LongName).Append(" is a duplicate of ").Append(c.LongName).Append("<br />");
  }
}
lblResult.Text = sb.ToString();

This is no good because I get the error:

NotSupportedException: Unable to create a constant value of type ‘TeachEDM.Student’. Only primitive types (‘such as Int32, String, and Guid’) are supported in this context.

Evidently it doesn’t like me trying to match o.SubjectId == c.SubjectId in LINQ.

Also, this seems a horrible method in general and is very slow. The call to the database takes more than 5 minutes.

I’d really appreciate some advice.

Answer 1

The comparison of the SubjectId is not the problem because c.SubjectId is a value of a primitive type (int, I guess). The exception complains about Equals(c.Students). c.Students is a constant (with respect to the query duplicates) but not a primitive type.

I would also try to do the comparison in memory and not in the database. You are loading the whole data into memory anyway when you start your first foreach loop: It executes the query allClasses. Then inside of the loop you extend the IQueryable allClasses to the IQueryable duplicates which gets executed then in the inner foreach loop. This is one database query per element of your outer loop! This could explain the poor performance of the code.

So I would try to perform the content of the first foreach in memory. For the comparison of the Students list it is necessary to compare element by element, not the references to the Students collections because they are for sure different.

var sb = new StringBuilder();
using (var ctx = new Ctx())
{
    ctx.CommandTimeout = 10000; // Perhaps not necessary anymore
    var allClasses = ctx.Classes.Include("Students").OrderBy(o => o.Id)
        .ToList(); // executes query, allClasses is now a List, not an IQueryable

    // everything from here runs in memory
    foreach (var c in allClasses)
    {
        var duplicates = allClasses.Where(
           o => o.SubjectId == c.SubjectId &&
           o.Id != c.Id &&
           o.Students.OrderBy(s => s.Name).Select(s => s.Name)
            .SequenceEqual(c.Students.OrderBy(s => s.Name).Select(s => s.Name)));

        // duplicates is an IEnumerable, not an IQueryable
        foreach (var d in duplicates)
            sb.Append(d.LongName)
              .Append(" is a duplicate of ")
              .Append(c.LongName)
              .Append("<br />");
    }
}
lblResult.Text = sb.ToString();

Ordering the sequences by name is necessary because, I believe, SequenceEqual compares length of the sequence and then element 0 with element 0, then element 1 with element 1 and so on.

---

Edit To your comment that the first query is still slow.

If you have 1300 classes with 30 students each the performance of eager loading (Include) could suffer from the multiplication of data which are transfered between database and client. This is explained here: How many Include I can use on ObjectSet in EntityFramework to retain performance? . The query is complex because it needs a JOIN between classes and students and object materialization is complex as well because EF must filter out the duplicated data when the objects are created.

An alternative approach is to load only the classes without the students in the first query and then load the students one by one inside of a loop explicitely. It would look like this:

var sb = new StringBuilder();
using (var ctx = new Ctx())
{
    ctx.CommandTimeout = 10000; // Perhaps not necessary anymore
    var allClasses = ctx.Classes.OrderBy(o => o.Id).ToList(); // <- No Include!
    foreach (var c in allClasses)
    {
        // "Explicite loading": This is a new roundtrip to the DB
        ctx.LoadProperty(c, "Students");
    }

    foreach (var c in allClasses)
    {
        // ... same code as above
    }
}
lblResult.Text = sb.ToString();

You would have 1 + 1300 database queries in this example instead of only one, but you won’t have the data multiplication which occurs with eager loading and the queries are simpler (no JOIN between classes and students).

Explicite loading is explained here:

• http://msdn.microsoft.com/en-us/library/bb896272.aspx
• For POCOs (works also for EntityObject derived entities): http://msdn.microsoft.com/en-us/library/dd456855.aspx
• For EntityObject derived entities you can also use the Load method of EntityCollection: http://msdn.microsoft.com/en-us/library/bb896370.aspx

If you work with Lazy Loading the first foreach with LoadProperty would not be necessary as the Students collections will be loaded the first time you access it. It should result in the same 1300 additional queries like explicite loading.