Question

I have using the bigquery uploading system over 3 months , I push data to bigquery for every one hour and it has been working fine . But since yesterday , all my Uploads are failing . Error at the bigquery console says ” Too few columns: expected 11 column(s) but got 9 column(s)” , The ninth field has a new line character and it is inside double quotes . This has been working for the past 3 months and It is failing from yesterday . Any help guys ?

Answer 1

Yesterday we started enforcing not allowing quoted newlines unless the allowQuotedNewlines flag is passed.
If you are using the bq tool, you can add the –allow_quoted_newlines flag. If you are using java, you can set this here. The flag is documented here.

Disallowing quoted newlines by default allows us to import csv files much more efficiently because we can split them into pieces and import in parallel. This translates into faster imports and allows us to scale to much larger imports.