Question

How do I delete array items I’m not interested in? If I would leave them—my memory would get overflowed with unnecessary items.

I need to implement in Perl one task. One file is being continuously filled with messages containing:

 "IP - URL"

I need to continuously read this file and measure if there were more than, say five, same IP - URL pairs in, say, a five second interval.

If I read the file from last position every five seconds and count duplicates then I can run into the situation when there were eight same line pairs during five seconds but during first read there were four of them and another four were during second read after five seconds. Thus I need to check the interval between last five duplicate lines.

What I can:

$pairs[$ip_url_line] = ['time-stamp',....,'time-stamp-N']

Then get last five array items for this hash key and calculate time shift. If it exceeds five seconds—do something.

Sure I can run through all hash elements and all array items in a loop and check whether it’s older then 5 seconds but it’s too resource-expensive.

Answer 1

1. Store the timestamps per IP address in order. You were probably going to do that anyway.
2. Whenever you do get a log line and you add a new entry, remove any stale entries right there, before you check how many entries there are. You can do it easily with a grep.
3. Periodically (once a minute?) delete any IP addresses from the hash that have a last (newest) timestamp more than 5 minutes ago, because that means that all the entries are more than 5 minutes old and that address hasn’t been seen for a while.

It’s simple, it’s easy to prove correct, it tries to avoid doing too much work at one time, and it keeps your tables from getting unreasonably large. With a 1-minute interval for step 3, no entry can possibly live more than 11 minutes. (If the first entry for 1.2.3.4 was added at 00:00:00, the latest an an entry could be added without shifting off the first one would be 00:04:59. The latest a step 3 sweep can run without deleting the whole array would then be 00:09:58; assuming that worst case, the next sweep would be at 00:10:58.) If you can keep 11 minutes of data in memory, you’re golden.