Question

in order to migrate our application (2 years old, fair amount of data), I plan to use the recommended tool.

Some of our entities store serialized dictionaries with references to keys, so we’ll need to convert those to change the application name part of the key. My question is, will changing data that has been migrated mess up the migration tool’s understanding of what needs to be migrated during an incremental copy? E.g, is it safe to:

• Start the migration tool
• run our custom tool to migrate stored keys (in the new app)
• verify our app works
• run an incremental copy
• run our tool again

Answer 1

It turns out that during the migration, the target app is in read only mode, so running any repair tools on the target hrd app before the migration is complete is not an option.

Instead, my approach is now to monkey patch db.get to rewrite keys on the fly, and then repair the references in the target app post migration. So far I’ve verified my new hrd app works fine with the monkey patch in place. It makes me wary to have to fix the references once the new version is live, but so it goes.

from google.appengine.ext import db
import os

APP_ID = os.getenv('APPLICATION_ID')

def replace_key(key_str, app_id=None):
    app_id = app_id or APP_ID
    oldkey = db.Key(key_str) if not isinstance(key_str, db.Key) else key_str
    if oldkey.app() != app_id:
        fixedkey = str(db.Key.from_path(oldkey.kind(), oldkey.id_or_name(), _app=app_id))
        logging.info("rewriting %s to %s", str(oldkey), fixedkey)
        return fixedkey
    else:
        return key_str

old_get = db.get

def monkeyed_get(keys, **kwargs):
    single = False
    if isinstance(keys, (basestring, db.Model, db.Key)):
        single = True
        keys = [keys]

    results = old_get(map(lambda k: replace_key(str(k)), keys), **kwargs)
    return results[0] if single else results

db.get = monkeyed_get