Question

The following query is being run on about 4 million rows. The first two CTE statements execute in about an hour. The final one however is on track to last more than 15 years.

WITH parsed AS (
   SELECT name, array(...) description FROM import
), counts AS (
   SELECT unnest(description) token, count(*) FROM parsed GROUP BY 1
) 
INSERT INTO table (name, description) 
SELECT name, ARRAY(
    SELECT ROW(token, count)::a 
    FROM (
        SELECT token, (
            SELECT count 
            FROM counts 
            WHERE a.token=counts.token
            ) 
        FROM UNNEST(description) a(token)
        ) _
    )::a[] description 
FROM parsed;

                                                                  QUERY PLAN                                                                   
-----------------------------------------------------------------------------------------------------------------------------------------------
 Insert on table  (cost=55100824.40..162597717038.41 rows=3611956 width=96)
   CTE parsed
     ->  Seq Scan on import  (cost=0.00..51425557.67 rows=3611956 width=787)
           Filter: ((name IS NOT NULL) AND (description IS NOT NULL))
           SubPlan 1
             ->  HashAggregate  (cost=11.59..12.60 rows=101 width=55)
                   ->  Append  (cost=0.00..11.34 rows=101 width=55)
                         ->  Result  (cost=0.00..0.01 rows=1 width=0)
                         ->  Index Scan using import_aliases_mid_idx on import_aliases  (cost=0.00..10.32 rows=100 width=56)
                               Index Cond: (mid = "substring"(import.mid, 5))
           SubPlan 2
             ->  HashAggregate  (cost=0.78..1.30 rows=100 width=0)
                   ->  Result  (cost=0.00..0.53 rows=100 width=0)
   CTE counts
     ->  HashAggregate  (cost=3675165.23..3675266.73 rows=20000 width=32)
           ->  CTE Scan on parsed  (cost=0.00..1869187.23 rows=361195600 width=32)
   ->  CTE Scan on parsed  (cost=0.00..162542616214.01 rows=3611956 width=96)
         SubPlan 6
           ->  Function Scan on unnest a  (cost=0.00..45001.25 rows=100 width=32)
                 SubPlan 5
                   ->  CTE Scan on counts  (cost=0.00..450.00 rows=100 width=8)
                         Filter: (a.token = token)

There are about 4 million rows in both parsed and counts. The query’s currently running, and the final statement’s inserting a row roughly every 2 minutes. It’s barely touching disk, but eating CPU like crazy, and I’m confused.

What’s wrong with the query?

The final statement’s supposed to look up each element of description in counts, transforming something like this [a,b,c] to something like this [(a,9),(b,4),(c,0)] and inserting it.

Edit

With parsed and counts as tables, and token in counts indexed, this is the plan:

explain INSERT INTO table (name, mid, description) SELECT name, mid, ARRAY(SELECT ROW(token, count)::a FROM (SELECT token, (SELECT count FROM counts WHERE a.token=counts.token) FROM UNNEST(description) a(token)) _)::a[] description FROM parsed;
                                              QUERY PLAN                                              
------------------------------------------------------------------------------------------------------
 Insert on table  (cost=0.00..5761751808.75 rows=4002061 width=721)
   ->  Seq Scan on parsed  (cost=0.00..5761751808.75 rows=4002061 width=721)
         SubPlan 2
           ->  Function Scan on unnest a  (cost=0.00..1439.59 rows=100 width=32)
                 SubPlan 1
                   ->  Index Scan using counts_token_idx on counts  (cost=0.00..14.39 rows=1 width=4)
                         Index Cond: (a.token = token)

Which is much more reasonable. The arrays have an average of 57 elements, so I guess it was just the sheer number of lookups against the presumable fairly inefficient CTE table that was killing performance. It’s now going at 300 rows per second, which I’m happy with.

Answer 1

As stated in my edit to the question, with parsed and counts as tables, and token in counts indexed it’s far faster. I was assuming CTE joins were cleverer than they are.