Question

I have this class that exposes an IEnumerable<Record> as follows (implementation details left out):

public class SomeFileReader() {
    public IEnumerable<Record> Records()
    {
        using (StreamReader sr = new StreamReader(this.Path, this.Encoding, true))
        {
            var hdr = this.HeaderParser.Parse(sr.ReadLine());  //Parse, but further ignore header (the HeaderParser might throw though)
            while (!sr.EndOfStream)
                yield return this.RecordParser.Parse(sr.ReadLine()) as Record;
        }
    }

A Record has, amongst many other properties (and thus being quite large ‘memory/storage wise’), an Id property (which is a Key object that consists of 2 “parts”). For completeness this looks something like:

public class Key : IEquatable<Key>
{
    public string OperatorCode { get; set; }
    public string Key { get; set; }

    public bool Equals(Key other)
    {
        return (this.OperatorCode.Equals(other.OperatorCode, StringComparison.OrdinalIgnoreCase))
            && (this.Key.Equals(other.Key, StringComparison.OrdinalIgnoreCase));
    }
}

The file contains the records in “key order”, so it is (guaranteed to be) sorted by the Record’s ID on-disk.

I also have, in memory, a HashSet<Key> of records that I want to process from the SomeFileReader. Currently my testfile is only a few megabytes large but I am forseeing this to grow very large in the near future. At this moment I just read the entire file into memory using a Dictionary<Key, Record> for easy/fast retrieval of specific records I want to process from my “list” of “to be processed” records. This would be similar to:

var recordsfromfile = MyFileImporter.Records().ToDictionary(k => k.Key.Key);

This will be problematic once the file grows (too) large ofcourse.

But since I am exposing an IEnumerable<Record> I was thinking… I shouldn’t have to read the file into memory completely since the records are in key order. A simple Intersect() with my “list” of to-be-processed Keys should suffice. The Key already implements IEquatable and should I need an IEQualityComparer<Key> that wouldn’t be hard to implement at all. But I (think I) digress..

The Intersect() documentation tells me:

When the object returned by this method is enumerated, Intersect
enumerates first, collecting all distinct elements of that sequence.
It then enumerates second, marking those elements that occur in both
sequences. Finally, the marked elements are yielded in the order in
which they were collected.

^{(Emphasis mine)}

So, if I understand correcly, if first would be my IEnumerable<Record> the file will still be read into memory completely. And even if it would be second all matches with my ‘to-be-processed’ “list” would still be read into memory which could still be a very large amount of data. Or am I misreading the documentation and is this “Finally” tripping me up and/or am I misunderstanding the documentation?

What I want to prevent, obviously, is

• a) not reading a large wad of data into memory for the sole purpose of processing some of the records in them one-by-one after which I don’t care about those records (the processing will write the result out somewhere else for example)
• b) not (re)opening the same file again and again for each record on my ‘to-be-processed’ “list” (so I want to be careful not to reset my iterator)

Long story short; will the Intersect() do what I want it to do? Should I use another method? Nested for-loop? Any other ideas on how to handle this efficiently?

EDIT: Updated to make clear that the “list of to be processed keys” is actually a HashSet<Key>.

---

P.s. I was just hit by a brainwave about using Linq for this purpose in bed and can’t get to sleep before I figure this out. Unfortunately, I am on vacation and miles away from a decent Visual Studio instance to just simply test this out. That will have to wait until after my vaction (so the misses says… we’ll see about that…)

Answer 1

EDIT: I suspect you actually want:

var records = new SomeFileReader().Records()
                                  .Where(record => keys.Contains(record.Key));

foreach (var record in records)
{
    Process(record);
}

---

The Intersect documentation is wrong, I’m afraid. It actually enumerates second first, collecting everything in that… and then streams first, yielding any intersecting values.

It also doesn’t wait until it’s seen all the elements before it yields them. See my Edulinq blog post on Intersect for more details of what it actually does.

In a TL;DR sense, it’s:

• Create a HashSet<T> from second
• Iterate over first
  • For each item, try to remove it from the set
  • If it was in the set, yield it; otherwise, don’t

The fact that items are removed from the set as we go stops the same element from being yielded twice (even if it occurred more than once in both first and second, due to it being a set).

Basically, I think you’ll be okay so long as you reverse the order of the operands, so you do:

var result = streamingRecordsFromFile.Intersect(smallCollectionInMemory);