Question

 public interface Event
 {
      Guid identifier;
      Timestamp ts;
 }

We’re thinking of using Reactive Extensions for a rewrite of a problem at my financial firm.

The premise is that we get events identified by a Guid (stock symbol + uniqueness entropy embedded into it), a timestamp, and a Value field. These come at a high rate, and we cannot act on these objects until “at least” after X seconds (10 seconds), after which we have to act on them, and remove them from the system.

Think about it like two windows, an initial window of “10 seconds” (for example T0 to T10), where we identify all the unique events (basically, group by guid), then we look into the next “10 seconds”, “secondary window” (T10-T20), to make sure we’re implementing the policy of “at least” 10 seconds. From the “initial window”, we remove all the events (because we’ve accounted for them), and then from the “secondary window”, we remove the ones that occurred in the “initial window”. And we keep on moving 10 second sliding windows, so now we’re looking at window T20-T30, repeat and rinse.

How could I implement this in Rx, because it seems like the way to go.

Answer 1

If you can rely on the server clock and the timestamp in the message (that is, we’re in ‘real life’ mode), and you’re after a sliding 10 second delay as opposed to a jumping 10 second window, then you can just delay the events 10 seconds:

var events = new Subject<Event>();  
var delayedEvents = events.Delay(TimeSpan.FromSeconds(10));

Checking for unique events etc is just a matter of adding them to a set of some sort:

var guidSet = new HashSet<Guid>();  
delayedEvents.Do(e => guidSet.Add(e.identifier));

If you’re problem is instead that you must wait 10 seconds and then process the last 10 seconds at once, then you just want to buffer for 10 seconds instead:

var bufferedEvents = events.Buffer(TimeSpan.FromSeconds(10));
bufferedEvents.Do(es => { foreach (var e in es) guidSet.Add(e.identifier); });

I haven’t shown the example of a sliding 10 second window as I can’t imagine that’s what you want (events get processed more than once).

---

Now we get serious. Let’s say you don’t want to rely on wall time and instead want to use the time within your events to drive your logic. Assuming event is redefined as:

 public class Event
 {
      public Guid identifier;
      public DateTime ts;
 }

Create the historical scheduler and feed the scheduled events from the original ones:

var scheduler = new HistoricalScheduler();
var driveSchedule = events.Subscribe(e => scheduler.AdvanceTo(e.ts));   
var target = events.SelectMany(e => Observable.Timer(e.ts, scheduler).Select(_ => e));

Now you can simply use the regular Rx combinators on target instead of event, and just pass through the scheduler so they are triggered appropriately, for example:

var bufferedEvents = target.Buffer(TimeSpan.FromSeconds(10), scheduler);

Here’s a simple test. Create a hundred events each ‘virtually’ 30 seconds apart but in real-time triggered every second:

var now = DateTime.Now;
var test = Enumerable.Range(0,99).Select(i =>
    Scheduler.ThreadPool.Schedule(
        TimeSpan.FromSeconds(i), 
        () => events.OnNext(new Event() { 
            identifier = Guid.NewGuid(), 
            ts = now.AddSeconds(i * 30) 
        })
    )
).ToList();

Subscribe to it and ask for 60 seconds of buffered events – and actually receive 2 events every 2 ‘real’ seconds (60 virtual seconds):

target.Select(e => String.Format("{0} {1}", e.identifier, e.ts.ToString()))
      .Buffer(TimeSpan.FromSeconds(60), scheduler)
      .Select(es => String.Join(" - ", es))
      .DumpLive();