Question

I have an piece of code that does calculations on assets. There are many millions of those so I want to compute everything in streams. My current ‘pipeline’ looks like this:

I have a query that is executed as a Datareader.

Then my Asset class has a constructor that accepts an IDataReader;

Public Asset(IdataReader rdr){
  // logic that initiates fields
}

and a method that converts the IDataReader to an IEnumerable<Asset>

public static IEnumerable<Asset> ToAssets(IDataReader rdr) {

    // make sure the reader is in the right formt
    CheckReaderFormat(rdr);

    // project reader into IEnumeable<Asset>
    while (rdr.Read()) yield return new Asset(rdr);

}

That then gets passed into a function that does the actually calculations and then projects it into a IEnumerable<Asnwer>

That then gets a wrapper the exposes the Answers as an IDataReader and that then that gets passed to a OracleBulkCopy and the stream is written to the DB.

So far it works like a charm. Because of the setup I can swap the DataReader for an IEnumerable that reads from a file, or have the results written to a file etc. All depending on how I string the classes/ functions together.

Now: There are several thing I can compute, for instance besides the normal Answer I could have a DebugAnswer class that also outputs some intermediate numbers for debugging. So what I would like to do is project the IEnumerable into several output streams so I can put ‘listeners’ on those. That way I won’t have to go over the data multiple times. How can I do that? Kind of like having several Events and then only fire certain code if there’s a listeners attached.

Also sometimes I write to the DB but also to a zipfile just to keep a backup of the results. So then I would like to have 2 ‘listeners’ on the IEnumerable. One that projects is as an IDataReader and another one that writes straight to the file.

How do I output multiple output streams and how can I put multiple listeners on one outputstream? What lets me compose streams of data like that?

edit

so some pseudocode of what I would like to do:

foreach(Asset in Assets){
   if(DebugListener != null){
     // compute 
     DebugAnswer da = new DebugAnswer {result = 100};
     yield da to DebugListener;  // so instead of yield return yield to that stream

   }

   if(AnswerListener != null){
     // compute basic stuff 
     Answer a = new Answer { bla = 200 };
     yield a to AnswerListener;
   }
}

Thanks in advance,

Gert-Jan

Answer 1

Your example rewritten using Rx:

// The stream of assets
IObservable<Asset> assets = ...

// The stream of each asset projected to a DebugAnswer
IObservable<DebugAnswer> debugAnswers = from asset in assets
                                        select new DebugAnswer { result = 100 };

// Subscribe the DebugListener to receive the debugAnswers
debugAnswers.Subscribe(DebugListener);

// The stream of each asset projected to an Anwer
IObservable<Answer> answers = from asset in assets
                              select new Answer { bla = 200 };

// Subscribe the AnswerListener to receive the answers
answers.Subscribe(AnswerListener);