Question

I’m looking for an efficient way of solving a particular problem in Python. I have a stream of events arriving each with an associated timestamp. Conceptually I add these events to the end of a time ordered list and do some processing on them, making some changes to various counts and averages (depending on the type of event). An event is timed out after 15 minutes when I remove the event at the start of the list and make the relevant adjustments to my counts and averages. The deque class in the collections module is a really good fit for this.

Now to the problem, I want to extend this idea but also keep counts and averages for events over the last 5 and 1 minute periods. As I see it I can’t efficiently time out events over these periods whilst using a single deque. I could keep a pair of indices into the deque of the next events to be timed out at the 5 minute and 1 minute boundaries, but as I understand things, indexing into a deque is an O(n) operation. I could also use 3 separate deques, one for each time period with events (or references to events) duplicated on each list. This just feels ugly.

The solution I’m toying with is to use a linked list but this seems pretty low level for Python; I don’t think Python provides a linked list structure so I’ll need to write my own. There are some additional constraints in that there are a large number of events and memory is fairly limited. I’d appreciate any other suggestions or insights into how I might solve this.

Answer 1

You can use three deques but like this:

• the original one for 1 minutes interval (instead of 15)
• another one for 5 minutes, will receive events from the previous deque
• another one for 15 minutes, will receive events from the previous deque.

This way, no duplicates and no indexing is needed. And this can be extended too.