Home/ Questions/Q 7872633
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-03T02:15:45+00:00

Im trying to do logs analysis with BigQuery. Specifically, I have an appengine app

  • 0

Im trying to do logs analysis with BigQuery. Specifically, I have an appengine app and a javascript client that will be sending log data to BigQuery. In bigquery, I’ll store the full log text in one column but also extract important fields into other columns. I then want to be able to do adhoc queries over those columns.

Two questions:

1) Is BigQuery particularly good or particularly bad at this use case?
2) How do I setup revolving logs? I.e. I want to only store the last N logs or the last X GB of log data. I see delete is not supported.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    1. Just so you know, there is an excellent demo of moving App Engine Log data to BigQuery via App Engine MapReduce called log2bq (http://code.google.com/p/log2bq/)

    2. Re: “use case” – Stack Overflow is not a good place for judgements about best or worst, but BigQuery is used internally at Google to analyse really really big log data.

    3. I don’t see the advantage of storing full log text in a single column. If you decide that you must set up revolving “logs,” you could ingest daily log dumps by creating separate BigQuery tables, perhaps one per day, and then delete the tables when they become old. See https://developers.google.com/bigquery/docs/reference/v2/tables/delete for more information on the Table.delete method.

    • 0