I find myself needing to process network traffic captured with tcpdump. Reading the traffic is not hard, but what gets a bit tricky is spotting where there are “spikes” in the traffic. I’m mostly concerned with TCP SYN packets and what I want to do is find days where there’s a sudden rise in the traffic for a given destination port. There’s quite a bit of data to process (roughly one year).
What I’ve tried so far is to use an exponential moving average, this was good enough to let me get some interesting measures out, but comparing what I’ve seen with external data sources seems to be a bit too aggressive in flagging things as abnormal.
I’ve considered using a combination of the exponential moving average plus historical data (possibly from 7 days in the past, thinking that there ought to be a weekly cycle to what I am seeing), as some papers I’ve read seem to have managed to model resource usage that way with good success.
So, does anyone knows of a good method or somewhere to go and read up on this sort of thing.
The moving average I’ve been using looks roughly like:
avg = avg+0.96*(new-avg)
With avg being the EMA and new being the new measure. I have been experimenting with what thresholds to use, but found that a combination of “must be a given factor higher than the average prior to weighing the new value in” and “must be at least 3 higher” to give the least bad result.
This is widely studied in intrusion detection literature. This is a seminal paper on the issue which shows, among other things, how to analyze tcpdump data to gain relevant insights.
This is the paper: http://www.usenix.org/publications/library/proceedings/sec98/full_papers/full_papers/lee/lee_html/lee.html here they use the RIPPER rule induction system, I guess you could replace that old one for something newer such as http://www.newty.de/pnc2/ or http://www.data-miner.com/rik.html