Question

I was trying to get multiple counts of occuring fields from 2 or more tables and I solved that issue from this question,

Nested queries to get count with two conditions

The solution worked but when tried in the following data structure,

It executes but never displays any result. but with the previous question i posted this kind of solution worked fine..Can anyone please help on this issue..it doesn’t even say there is a syntax error.

EDIT: sql query

SELECT t1.timeStamp, t1.localIp, t2.localPort, t3.localGeo, t4.isp, t5.foreignIp, t6.foreignPort, t7.foreignGeo, t8.infection,t1.timeStampCount, t1.localIpCount, t2.localPortCount, t3.localGeoCount, t4.ispCount, t5.foreignIpCount, t6.foreignPortCount, t7.foreignGeoCount, t8.infectionCount
FROM
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(timeStamp) AS 'timeStampCount',COUNT(localIp) AS 'localIpCount' 
FROM (
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'  
    FROM tbl_shadowserver_bot_geo 
    UNION ALL 
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'  
    FROM tbl_www_cymru_com 
) c  
GROUP BY timeStamp,localIp)t1  
JOIN 
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(localPort) AS 'localPortCount' 
FROM (
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'  
    FROM tbl_shadowserver_bot_geo 
    UNION ALL 
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'  
    FROM tbl_www_cymru_com 
) c  
GROUP BY timeStamp,localIp,localPort)t2  
ON t1.timeStamp=t2.timeStamp 
JOIN 
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(localGeo) AS 'localGeoCount' 
FROM (
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'  
    FROM tbl_shadowserver_bot_geo 
    UNION ALL 
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'  
    FROM tbl_www_cymru_com 
) c  
GROUP BY timeStamp,localIp,localPort,localGeo)t3  
ON t1.timeStamp=t3.timeStamp 
JOIN 
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(isp) AS 'ispCount' 
FROM (
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'  
    FROM tbl_shadowserver_bot_geo 
    UNION ALL 
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'  
    FROM tbl_www_cymru_com 
) c  
GROUP BY timeStamp,localIp,localPort,localGeo,isp)t4  
ON t1.timeStamp=t4.timeStamp 
JOIN 
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(foreignIp) AS 'foreignIpCount' 
FROM (
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'  
    FROM tbl_shadowserver_bot_geo 
    UNION ALL 
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'  
    FROM tbl_www_cymru_com 
) c  
GROUP BY timeStamp,localIp,localPort,localGeo,isp,foreignIp)t5  
ON t1.timeStamp=t5.timeStamp 
JOIN 
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(foreignPort) AS 'foreignPortCount' 
FROM (
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'  
    FROM tbl_shadowserver_bot_geo 
    UNION ALL 
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'  
    FROM tbl_www_cymru_com 
) c  
GROUP BY timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort)t6  
ON t1.timeStamp=t6.timeStamp 
JOIN 
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(foreignGeo) AS 'foreignGeoCount' 
FROM (
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'  
    FROM tbl_shadowserver_bot_geo 
    UNION ALL 
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'  
    FROM tbl_www_cymru_com 
) c  
GROUP BY timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo)t7  
ON t1.timeStamp=t7.timeStamp JOIN 
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(infection) AS 'infectionCount' 
FROM (
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'  
    FROM tbl_shadowserver_bot_geo 
    UNION ALL 
    SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'  
    FROM tbl_www_cymru_com 
    ) c  
    GROUP BY timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection)t8 
ON t1.timeStamp=t8.timeStamp ORDER BY timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection

Answer 1

I’ll first try to restate your question in my own words, to make it clear what this answer here does. For each record in the union of two tables, you want one row in the result set. That row should contain additional information: for a given set of selected columns you want to count the rows from the input which match the current rows in all selected columns. In each step you add to the set of selected rows. So first you count all rows with the same time stamp. Next you count all the rows with the same time stamp and the same IP address. And so on.

As you repeatedly select from the same union of tables, it might be prudent to introduce a short name for it. You can do so by creating a VIEW. Next, you want to use that view as the first factor of your join. This gives you one row of output for every row of input. For the added statistics, you join to that a subquery which counts rows, grouping by all the selected columns for that count. You want to join that subquery to the rest of your query using all these selected columns. So for example:

CREATE VIEW view_both_tables AS
SELECT date_format(timestamp, '%Y-%m-%d %h') AS `timeStamp`, ip AS localIp, …
FROM tbl_shadowserver_bot_geo 
UNION ALL 
SELECT date_format(timestamp, '%Y-%m-%d %h') AS `timeStamp`, ip AS localIp, …
FROM tbl_www_cymru_com;

SELECT t0.*,
 t1.countSameTime,
 t2.countSameTimeAndLocalIp,
 t3.countSameTimeLocalIpAndInfection
FROM view_both_tables t0
LEFT JOIN (
  SELECT `timeStamp`,
   COUNT(*) AS `countSameTime`
  FROM view_both_tables
  GROUP BY `timeStamp`
 ) t1 USING (`timeStamp`)
LEFT JOIN (
  SELECT `timeStamp`, localIp,
   COUNT(*) AS `countSameTimeAndLocalIp`
  FROM view_both_tables
  GROUP BY `timeStamp`, localIp
 ) t2 USING (`timeStamp`, localIp)
LEFT JOIN (
  SELECT `timeStamp`, localIp, infection,
   COUNT(*) AS `countSameTimeLocalIpAndInfection`
  FROM view_both_tables
  GROUP BY `timeStamp`, localIp, infection
 ) t3 USING (`timeStamp`, localIp, infection)
;

This query was taken from my SQL fiddle. It only covers up to three selected columns, as the other columns didn’t contain any data in your example, and as the idea should be clear from this.

In general, it is more efficient to COUNT(*) instead of counting one particular column. The only exceptions are when the column you want to count may contain NULL values which should be omitted from the count, or when you’re using COUNT(DISTINCT …). But as my interpretation of your question referred to counting records, not rows, neither of these exceptions applies.

As a further note on style, using single quotation marks to quote column names is rather confusing SQL style. Usually, strings are delimited using single quotation marks, and columns uisng backticks. What you wrote is valid syntax for the default MySQL configuration, but confusing nonetheless.

If performance is an issue, you might try executing a single query using GROUP BY … WITH ROLLUP. Storing the result to a temporary table, you could derive the result you asked for from repeatedly joining that table to itself. No sure whether there would be any performance gain, but it might be worth a try.