Home/ Questions/Q 8671069
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-12T18:51:29+00:00

I have pmacct running summarizing network traffic on an hourly basis into a postgres

  • 0

I have pmacct running summarizing network traffic on an hourly basis into a postgres database.
I need to write a script/query to move that data in a different format into a mysql database. I want to do as much of the data processing as possible using SQL, as this dataset is going to rapidly grow.

I have a perl script running to add an additional field (agent_id) to track the zone the data is in (local/national/international), which will show as either 0, 1, or 2.

The relevant fields from the schema of the table I’m pulling this data out from is:

ip_src, ip_dst, agent_id, bytes, stamp_updated, processed

The schema I want to insert the data into is:

ip, local_down_mb, nat_down_mb, int_down_mb, local_up_mb, nat_up_mb, int_up_mb, timestamp

As I’m only looking for the traffic where the source or destination is one of my ranges, I have a query at present which gets the upload data out of the postgres database in a way that I want it:

SELECT DISTINCT ip_src, agent_id, SUM(bytes), stamp_updated FROM acct
WHERE ip_src <<= '192.168.0.0/22'
   OR ip_src <<= '10.1.2.0/24'
   OR ip_src <<= '1.2.3.4/32'
GROUP BY ip_src, agent_id, stamp_updated
ORDER BY ip_src, agent_id, stamp_updated

A sample output of that query is:

   ip_src     | agent_id |    sum    |    stamp_updated    
--------------+----------+-----------+---------------------
10.1.2.134    |        2 |      3192 | 2012-09-13 21:20:01
10.1.2.134    |        2 |      3192 | 2012-09-13 22:20:01
10.1.2.134    |        2 |      3192 | 2012-09-13 23:20:01
10.2.3.252    |        2 |       448 | 2012-09-11 06:00:01
10.2.3.252    |        2 |       448 | 2012-09-11 07:20:01
10.2.3.252    |        2 |       448 | 2012-09-11 08:20:01
10.2.3.252    |        2 |      8112 | 2012-09-11 09:20:01

At this stage, I know I could run the same query for ip_dst, and then have a bit of a manual process when reinserting the data into mysql in the new format to ensure that the ip source and destination were matched for a timestamp, and then use the combination of agent_id and whether it was the ip source or ip destination that I was inserting to know whether it was inbound or outbound, and if the traffic was local, national, or international.

What I’d like, however, is a query which will do all of that for me. The limit of my SQL knowledge is having gone through the W3C website tutorials months ago, which has gotten me to the point where I can write a query as above, but not much further.

From what I can tell, what I need some help with is writing a join between the two sets of results, one for ip_src and one for ip_dst, and then doing some magic to use the information of which direction traffic was going in conjunction with the agent_id to have an output which will match the schema of the mysql database.

Is there someone who can either (very kindly) write what query they think might work for accomplishing this, or at least point me towards relevant documentation and give me a head start on what functions I might need to use to make this work?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team
    • To address your primary concern of how to glue the ip_src to the ip_dst searches, you want to use a FULL OUTER JOIN on the the two queries you already had to handle the case where an IP only had traffic in one direction for a given timestamp. If your data loader could guarantee matching data, you could get away with an INNER JOIN, but why risk it?
    • Since you want to pivot the agent_id into 3 separate columns in your target schema, I showed one approach with conditions inside the aggregate function.

    • I made assumptions about transforming the byte counts to rounded up megabytes in the final output based on the column names.

        SELECT down.ip,
         ceil(down.lb/1048576) AS local_down_mb,
         ceil(down.nb/1048576) AS nat_down_mb,
         ceil(down.ib/1048576) AS int_down_mb,
         ceil(up.lb/1048576) AS local_up_mb,
         ceil(up.nb/1048576) AS nat_up_mb,
         ceil(up.ib/1048576) AS int_up_mb,
         down.timestamp
    FROM (SELECT ip_src AS ip,
                 SUM(CASE WHEN agent_id=0 THEN bytes ELSE 0 END) AS lb,
                 SUM(CASE WHEN agent_id=1 THEN bytes ELSE 0 END) AS nb,
                 SUM(CASE WHEN agent_id=2 THEN bytes ELSE 0 END) AS ib,
                 stamp_updated AS timestamp
            FROM acct
           WHERE ip_src <<= '192.168.0.0/22'
              OR ip_src <<= '10.1.2.0/24'
              OR ip_src <<= '1.2.3.4/32'
        GROUP BY ip,timestamp) down
    FULL OUTER JOIN
         (SELECT ip_dst AS ip,
                 SUM(CASE WHEN agent_id=0 THEN bytes ELSE 0 END) AS lb,
                 SUM(CASE WHEN agent_id=1 THEN bytes ELSE 0 END) AS nb,
                 SUM(CASE WHEN agent_id=2 THEN bytes ELSE 0 END) AS ib,
                 stamp_updated AS timestamp
            FROM acct
           WHERE ip_dst <<= '192.168.0.0/22'
              OR ip_dst <<= '10.1.2.0/24'
              OR ip_dst <<= '1.2.3.4/32'
        GROUP BY ip,timestamp) up
   USING (ip,timestamp)
ORDER BY ip,timestamp;
    • 0