Question

We are loading datastore backups into big query with the big query v2 api. We are specifying this JSON configuration:

{'configuration': {
    'load': {
        'sourceFormat'     : 'DATASTORE_BACKUP',
        'writeDisposition' : 'WRITE_TRUNCATE',
        'sourceUris'       : sourceUris,
        'destinationTable' : {
            'projectId': settings.PROJECT_ID,
            'datasetId': datasetId,
            'tableId'  : entityKind
            }
        }
    }
}

We have already loaded this entity into BigQuery once and are now expecting further loads to replace the existing table with the new data. We are not seeing this but an error in the insert job request:

u'status': {
u'state': u'DONE',
u'errors': [
  {
    u'reason': u'invalid',
    u'message': u'Cannot import a datastore backup to a table that already has a schema.'
  }
],
u'errorResult': {
  u'reason': u'invalid',
  u'message': u'Cannot import a datastore backup to a table that already has a schema.'
}

},

Is this expected that writeDisposition == ‘WRITE_TRUNCATE’ does not work with DATASTORE_BACKUP?

Thanks,

Braden

Answer 1

Yes, at the moment, ingestion jobs using Datastore admin backups as source files currently only support the creation of new BigQuery tables (or replacing the entire contents of an existing BigQuery table) – currently it’s not possible to append to existing tables.

Note that this feature in an early experimental stage, and we appreciate your feedback as we plan new iterations of this feature.