Question

I’m trying to compare 2 mysql tables with the same scheme and columns. The table contains ip addresses, the ports they communicate on, if it’s in or outbound and the number of connections over a 1 month period. Below is a small example table with made up numbers (the real tables are approx 100k rows).

+---------------+------+----------+-------------+-----------+
| ip_address    | port | protocol | connections | direction |
+---------------+------+----------+-------------+-----------+
| 123.17.19.6    | 123  | 17       | 31972       | IN        |
| 123.17.19.6    | 22   | 6        | 4           | IN        |
| 123.17.19.6    | 25   | 6        | 206969      | IN        |
| 123.17.19.10   | 135  | 6        | 2997        | OUT       |
| 123.17.19.10   | 389  | 17       | 4965        | OUT       |
| 123.17.19.10   | 389  | 6        | 7089        | OUT       |
| 123.17.19.11   | 139  | 6        | 1           | OUT       |
| 123.17.19.10   | 135  | 6        | 1102        | OUT       |
| 123.17.19.11   | 389  | 17       | 2993        | OUT       |
| 123.17.19.11   | 389  | 6        | 1629        | OUT       |
| 123.17.19.11   | 443  | 6        | 28          | OUT       |
| 123.17.19.11   | 445  | 6        | 4267        | OUT       |
| 123.17.19.11   | 53   | 17       | 5230        | OUT       |
| 123.17.19.11   | 53   | 6        | 10          | OUT       |
| 123.17.19.11   | 80   | 6        | 11          | OUT       |
| 123.17.19.12   | 135  | 6        | 1640        | OUT       |
| 123.17.19.12   | 22   | 6        | 2           | OUT       |
| 123.17.19.10   | 22   | 6        | 6           | OUT       |
| 123.17.19.12   | 389  | 17       | 2539        | OUT       |
+---------------+------+----------+-------------+-----------+

What I want to do is compare 2 months to see what IP, Port, Proto & Direction combinations are new / no longer present and for any matches see the variation in the number of connections

My original thought was to simply just loop through each row and then run a query against the other table to see if that connection exists but this would lead to literally hundreds of thousands of queries being run. I just feel like there has to be a much simpler way to do this. (example below)

use strict;
use warnings;
use DBI;

my ($db1_list,$db2_list,@compare_list1,@compare_list2);
my $db1 = "Jan";
my $db2 = "Feb";

$db2_list = login()->prepare(qq(select * from $db2));
$db2_list->execute;

while (@compare_list2 = $db2_list->fetchrow()){
  $db1_list = login()->prepare(qq(select * from $db2 where ip_address = "@compare_list2[0]" and port = @compare_list2[1] and protocol = @compare_list2[2] and direction = "@compare_list2[4]"));
  $db1_list->execute;

  while (@compare_list1 = $db1_list->fetchrow()){
    if (@compare_list1[0] ~~ @compare_list2[0]);
      @compare_list[3] -= @compare_list[3];
      print "@compare_list[3]\n";
    }
    else {
      print "@compare_list2[0], @compare_list2[1], @compare_list2[2], @compare_list2[3], @compare_list2[4] was seen in $db2 and not in $db1\n";
    }
  }
}

Answer 1

MySQL can do this within a single query:

SELECT *
FROM Feb
WHERE NOT EXISTS (SELECT 1 FROM Jan
    WHERE Feb.ip = Jan.ip
    AND Feb.protocol = Jan.protocol
    AND Feb.direction = Jan.direction
)

Now you’ve got a list of everything in Feb that does NOT have a match in month2 (so it’s “new” in Feb).