Home/ Questions/Q 9006979
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-16T01:33:34+00:00

I have a table in BigQuery with the following fields: time,a,b,c,d time is a

  • 0

I have a table in BigQuery with the following fields:

time,a,b,c,d

time is a string in ISO8601 format but with a space, a is an integer from 1 to 16000, and the other columns are strings. The table contains one month’s worth of data, and there are a few million records per day.

The following query fails with “response too large”:

select UTC_USEC_TO_DAY(PARSE_UTC_USEC(time)) as day,b,c,d,count(a),count(distinct a, 1000000)
from [myproject.mytable]
group by day,b,c,d
order by day,b,c,d asc

However, this query works (the data starts at 2012-01-01)

select UTC_USEC_TO_DAY(PARSE_UTC_USEC(time)) as day,
  b,c,d,count(a),count(distinct a)
from [myproject.mytable]
where UTC_USEC_TO_DAY(PARSE_UTC_USEC(time)) = UTC_USEC_TO_DAY(PARSE_UTC_USEC('2012-01-01 00:00:00'))
group by day,b,c,d
order by day,b,c,d asc

This looks like it might be related to this issue. However, because of the group by clause, the top query is equivalent to repeatedly calling the second query. Is the query planner not able to handle this?

Edit: To clarify my test data:

I am using fake test data I generated. I originally used several fields and tried to get hourly summaries for a month (group by hour, where hour is defined using as in the select part of the query). When that failed I tried switching to daily. When that failed I reduced the columns involved. That also failed when using a count (distinct xxx, 1000000), but it worked when I just did one day’s worth. (It also works if I remove the 1000000 parameter, but since that does work with the one-day query it seems the query planner is not separating things as I would expect.)

The one checked for count (distinct) has cardinality 16,000, and the group by columns have cardinality 2 and 20 for a total of just 1200 expected rows. Column values are quite short, around ten characters.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    How many results do you expect? There is currently a limitation of about 64MB in the total size of results that are allowed. If you’re expecting millions of rows as a result, than this may be an expected error.

    If the number of results isn’t extremely large, it may be that the size problem is not the final response, but the internal calculation. Specifically, if there are too many results from the GROUP BY, the query can run out of memory. One possible solution is to change “GROUP BY” to “GOUP EACH BY” which alters the way the query is executed. This is a feature that is currently experimental, and as such, is not yet documented.

    For your query, since you reference fields named in the select in the group by, you might need to do this:

    select day, b,c,d,day,count(a),count(distinct a, 1000000) 
FROM (
    select UTC_USEC_TO_DAY(PARSE_UTC_USEC(time)) as day, b, c, d
    from [myproject.mytable]
)
group EACH by day,b,c,d
order by day,b,c,d asc
    • 0