Question

I have a model in App Engine that I am exporting in CSV format and then importing to my local dev environment using appcfg.py and bulkloader.yaml.

I can import and export most models, but I am having issues with models that have a foreign key which is not always present. I can use the lambda import below to always import the foreign key as None or use the create_foreign_key() transform to import the foreign key when every line in my csv file has the foreign key.

How do I configure bulkloader.py to import the foreign key when it is present and ignore it when it is not present?

- kind: MyModel
  connector: csv
  connector_options:
  property_map:
    - property: myOtherModel
      external_name: myOtherModel
      import_transform: "lambda x: x is None and None or None"
      #import_transform: transform.create_foreign_key('MyOtherModel', key_is_id=True)
      export_transform: transform.key_id_or_name_as_string

Just uncommenting out the second import_transform in place of the lambda transform will produce this error.

File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/transform.py", line 127, in generate_foreign_key_lambda
    value = int(value)
ValueError: invalid literal for int() with base 10: ''

This error occurs when I run appcfg.py. All other imports without foreign keys or where foreign keys are always present are working properly.

appcfg.py upload_data --config_file=bulkloader.yaml --num_threads=1 --batch_size=50 --url=http://localhost:8080/remote_api --email=Chris --passin --kind=MyModel --filename=MyModel.csv

The myOtherModel column in the csv file sometimes contains the MyOtherModel.key().id() and sometimes not.

Eg.

myOtherModel
1234
4567

2345

5678

Answer 1

Default behivour when foreign key value does not exist, reference property takes value: None and code for this:

import_transform: transform.create_foreign_key('MyOtherModel')

As in your error message: value does not exist shows that you uses
'othercolumn','','anohtercolumn' instead of 'othercolumn',,'anohtercolumn'

So if the source ” is then to handle this:

from this link : http://eikke.com/python-ifelse-in-lambda/
as Thomas Thurman comments, lambda expression should look like:

import-transform: "lambda x: [x, None][x=='']"

I hope it works for you