Question

Attempting to import a TextProperty, similar to another Stack Overflow question (http://stackoverflow.com/questions/3434090/app-engine-badvalueerror-on-bulk-data-upload-textproperty-being-construed-as-s)

After adding the import_transform: db.Text, I still get an error. The story_html is a TextProperty()

BadValueError: Property story_html is 3132 bytes long; it must be 500 or less. Consider Text instead, which can store strings of any length.

My Bulkloader.yaml file.

python_preamble:
- import: base64
- import: re
- import: google.appengine.ext.bulkload.transform
- import: google.appengine.ext.bulkload.bulkloader_wizard
- import: google.appengine.ext.db
- import: google.appengine.api.datastore
- import: google.appengine.api.users

transformers:
- kind: Story   connector: csv   connector_options:
    encoding: utf-8
    columns: from_header   property_map:
    - property: publication
      external_name: publication

    - property: title
      external_name: title

    - property: author
      external_name: author

    - property: date
      external_name: date

    - property: url
      external_name: url
      import_transform: db.Link

    - property: story_html
      external_name: story_html

    - property: story_text
      external_name: story_text

    - property: link_text
      external_name: link_text

    - property: story_text_word
      external_name: story_text_word

    - property: story_text_frequency
      external_name: story_text_frequency

    - property: link_text_word
      external_name: link_text_word

    - property: link_text_frequency
      external_name: link_text_frequency

    - property: tags
      external_name: tags

    - property: companies
      external_name: companies

    - property: companies_links
      external_name: companies_links

My Models.py file

from google.appengine.ext import db

class Story(db.Model):
    publication = db.StringProperty()
    title = db.StringProperty()
    author = db.StringProperty()
    date = db.DateProperty()
    url = db.LinkProperty()
    story_html = db.TextProperty()

    # Processed Attributes
    story_text = db.TextProperty()
    link_text = db.TextProperty()

    # Word Frequency
    story_text_word = db.ListProperty(unicode)
    story_text_frequency = db.ListProperty(int)

    link_text_word = db.ListProperty(unicode)
    link_text_frequency = db.ListProperty(int)

    tags = db.ListProperty(unicode)
    companies = db.ListProperty(unicode)
    companies_links = db.ListProperty(unicode)

And my loaders.py file

 import datetime from google.appengine.ext import db 
 from google.appengine.tools import bulkloader 
 import re

 import models

 class StoryLoader(bulkloader.Loader):
     def __init__(self):
         bulkloader.Loader.__init__(self, 'Story',
                                    [('publication', lambda x: x.decode('utf-8')),
                                    ('title', lambda x: x.decode('utf-8')),
                                    ('author', lambda x: x.decode('utf-8')),
                                    ('date', lambda x: datetime.datetime.strptime(x, '%Y-%m-%d').date()),
                                    ('url', str),
                                    ('story_html', lambda x: db.Text(x.decode('utf-8'))),
                                    ('story_text', lambda x: db.Text(x.decode('utf-8'))),
                                    ('link_text', lambda x: db.Text(x.decode('utf-8')))

                                    ('story_text_word', lambda x: x.split('|')),
                                    ('story_text_frequency', lambda x: list(eval(x))),

                                    ('link_text_word', lambda x: x.split('|')),
                                    ('link_text_frequency', lambda x: x.split('|')),
                                    ('tags', lambda x: x.split('|')),
                                    ('companies', lambda x: x.split('|')),
                                    ('companies_links', lambda x: x.split('|'))])

 loaders = [StoryLoader]

Answer 1

You say you’ve set the import_transform, but there’s no transform set on the story_html property. You need to set it per-property.