Question

I have some big query jobs that have failed, I’m trying to develop a retry mechanism in which I will save all jobs that have failed in the data store and later on reattempt to run them.
so, I have my own object “JobFailed” which have some properties that “Job” is one of them.
“Job” is not serialize able and cannot be serialized or to be embed in a data store entity (or at least I wasn’t able to do so..)
so what are my options?
must I make my own object to save in the data store and parse it when its being pulled out to build a job object?
can I inherit job into my own and put an Entity annotation on it and save it as a standalone entity which is connected to my JobFailed entity? will that work?

Another possibility is saving only JobId in dataStore and getting the job when I want it ,with the “get” method in the BigQuery Api . (but I prefer saving it to datastore..)

thanks.

Answer 1

I might be misunderstanding the question, but can’t you just pull out the status and error message from the API response?

project_number = 'XXXXXXXX'
job_id = 'job_600b919ee97a4eXXXXXXXX'

bigquery_service = build('bigquery', 'v2', http=http)
jobs = bigquery_service.jobs()

job = jobs.get(projectId=project_number,
               jobId=job_id).execute()

# Result
job_state = job['state']
job_error = job['errorResult']['reason'] if job.get('errorResult') else ''

And then store this information in a Datastore model such as:

class BigQueryJob(db.Model):
  job_id = db.StringProperty(required=True)
  status = db.StringProperty(required=True)
  error = db.StringProperty()