Question

I have a huge table containing 161 million rows. The table structure is entryid(varchar), dater(date), region(char), detail(varchar), views(int), and the first four columns are together as the primary key. I want to do the following processing:

1. detail includes URL such as www.google.com, www.google.ca, www.google.uk, www.facebook.com, facebook.com, etc. I want to group those rows from the same domain, for example, using LIKE "%.google.%" for google and LIKE "%.facebook.%" OR LIKE "facebook.%" for facebook are good enough. But if update the rows, there might be duplicate key, how to use ON DUPLICATE KEY UPDATE to sum the views? And is there a faster way other than LIKE?

2. I want to remove the region column and group the rows and sum the views and insert into a new table. How to do that in one command?

3. I want to group the rows (and update the views) by month.

Answer 1

Try this, for MYSQL, assuming that for every row, the detail field looks like http://www.something.com.

SELECT
   MONTH(dater),
   SUBSTRING_INDEX(SUBSTRING_INDEX(detail, '.', 2), -1) as url,
   sum(views)
from table
group by 
   MONTH(dater),
   SUBSTRING_INDEX(SUBSTRING_INDEX(detail, '.', 2), -1)

Or you could use a subquery to make it look a bit cleaner:

SELECT mth, url, sum(views) from
(
SELECT
   MONTH(dater) as mth,
   SUBSTRING_INDEX(SUBSTRING_INDEX(detail, '.', 2), -1) as url,
   views
from table) subq
group by 
   mth, url