Question

I have harvested data, its not particularly clean data, and this has been bulk uploaded into the DataStore. However I am getting the following issue when trying to simply loop through all the records. I don’t much care about validation at this point as all I want to do is perform a bulk operation but GAE appears not to let me even loop through the data records. I want to get at the bottom of this. To my knowledge all records have data in the field for the country and I could switch of validation, but can someone explain why this is happening and GAE is being sensitive. Thanks

result = Company.all()
        my_count = result.count()
        if result:
            for r in result:
                self.response.out.write("hello")

The data model has these properties:

class Company(db.Model):
    companyurl = db.LinkProperty(required=True)
    companyname = db.StringProperty(required=True)
    companydesc = db.TextProperty(required=True)
    companyaddress = db.PostalAddressProperty(required=False)
    companypostcode = db.StringProperty(required=False)
    companyemail = db.EmailProperty(required=True)
    companycountry = db.StringProperty(required=True)

The error message is below

Traceback (most recent call last):
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 701, in __call__
    handler.get(*groups)
  File "/base/data/home/apps/XXX/1.358667163009710608/showcompanies.py", line 99, in get
    for r in result:
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 2312, in next
    return self.__model_class.from_entity(self.__iterator.next())
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1441, in from_entity
    return cls(None, _from_entity=entity, **entity_values)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 973, in __init__
    prop.__set__(self, value)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 613, in __set__
    value = self.validate(value)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 2815, in validate
    value = super(StringProperty, self).validate(value)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 640, in validate
    raise BadValueError('Property %s is required' % self.name)
BadValueError: Property companycountry is required

Answer 1

If you have the bulk process you wish to run in its own script, you can construct a modified version of your Company class without validation. Since db.Model classes are just wrappers to the datastore based on the name of the class, you can have different classes in different parts of your code with different behaviors.

So you might have a model.py file with:

class Company(db.Model):
  companyurl = db.LinkProperty(required=True)
  # ...
  companycountry = db.StringProperty(required=True)

# Normal operations go here

And, another bulk_process.py file with:

class Company(db.Model):
  companyurl = db.LinkProperty()
  # ...
  companycountry = db.StringProperty()

result = Company.all()
my_count = result.count()
if result:
  for r in result:
    self.response.out.write("hello")

Because this second model class lacks the validation, it should run just fine. And, because the code is logically separated you don’t have to worry about unintentional side-effects from removing the validation in the rest of your code. Just make sure that your bulk process doesn’t accidentally write back data without validation (unless you’re OK with this).