Question

I’m converting a library to use SQLAlchemy as the datastore. I like the flexibility of the PickleType column, but it doesn’t seem to work well when pickling SA objects (table rows). Even if I overload setstate and getstate to do a query + session merge when unpickling, there’s no referential integrity across that pickle boundary. That means that I can’t query collections of objects.

class Bar(Base):
    id = Column(Integer, primary_key=True)
    __tablename__ = 'bars'
    foo_id = Column(Integer, ForeignKey('foos.id'), primary_key=True)

class Foo(Base):
    __tablename__ = 'foos'
    values = Column(PickleType)
    #values = relationship(Bar)  # list interface (one->many), but can't assign a scalar or use a dictionary
    def __init__(self):
        self.values = [Bar(), Bar()]

        # only allowed with PickleType column
        #self.values = Bar()
        #self.values = {'one' : Bar()}
        #self.values = [ [Bar(), Bar()], [Bar(), Bar()]]

# get all Foo's with a Bar whose id=1
session.query(Foo).filter(Foo.values.any(Bar.id == 1)).all()

One workaround would be to implement my own mutable object type as is done here. I’m imagining having some kind of flattening scheme which traverses the collections and appends them to a simpler one->many relationship. Perhaps the flattened list might have to be weakrefs to the pickled collection’s objects?

Tracking changes and references sounds like no fun and I can’t find any examples of people pickling SA rows anywhere else (perhaps indicative of bad design on my part?). Any advice?

EDIT 1:
After some discussion I’ve simplified the request. I’m looking for a single property that can behave as either a scalar or a collection. Here is my (failing) attempt:

from sqlalchemy import MetaData, Column, Integer, PickleType, String, ForeignKey, create_engine
from sqlalchemy.orm import relationship, Session
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm.collections import attribute_mapped_collection


# from http://www.sqlalchemy.org/trac/browser/examples/vertical
from sqlalchemy_examples.vertical import dictlike_polymorphic as dictlike

metadata = MetaData()
Base = declarative_base()
engine = create_engine('sqlite://', echo=True)
Base.metadata.bind = engine
session = Session(engine)


class AnimalFact(dictlike.PolymorphicVerticalProperty, Base):
    """key/value attribute whose value can be one of several types"""
    __tablename__ = 'animalfacts'
    type_map = {#str: ('string', 'str_value'),
                list: ('list', 'list_value'),
                tuple: ('tuple', 'tuple_value')}
    id = Column(Integer, primary_key=True)
    animal_id = Column(Integer, ForeignKey('animal.id'), primary_key=True)
    key = Column(String, primary_key=True)
    type = Column(String)
    #str_value = Column(String)
    list_value = relationship('StringEntry')
    tuple_value = relationship('StringEntry2')


class Animal(Base, dictlike.VerticalPropertyDictMixin):
    __tablename__ = 'animal'
    _property_type = AnimalFact
    _property_mapping = 'facts'

    id = Column(Integer, primary_key=True)
    name = Column(String)
    facts = relationship(AnimalFact, backref='animal',
                          collection_class=attribute_mapped_collection('key'))

    def __init__(self, name):
        self.name = name


class StringEntry(Base):
    __tablename__ = 'stringentry'
    id = Column(Integer, primary_key=True)
    animalfacts_id = Column(Integer, ForeignKey('animalfacts.id'))
    value = Column(String)

    def __init__(self, value):
        self.value = value


class StringEntry2(Base):
    __tablename__ = 'stringentry2'
    id = Column(Integer, primary_key=True)
    animalfacts_id = Column(Integer, ForeignKey('animalfacts.id'))
    value = Column(String)

    def __init__(self, value):
        self.value = value

Base.metadata.create_all()


a = Animal('aardvark')
a['eyes'] = [StringEntry('left side'), StringEntry('right side')]  # works great
a['eyes'] = (StringEntry2('left side'), StringEntry2('right side'))  # works great
#a['cute'] = 'sort of'  # failure

Answer 1

The PickleType is really a hacky way around edge cases where you have some arbitrary object you’d just like to shove away. It’s a given that when you use PickleType, you’re giving up any relational advantages, including being able to filter/query on them, etc.

So putting an ORM mapped object in a Pickle is basically a terrible idea.

If you want a collection of scalar values, use traditional mappings and relationship() in combination with association_proxy. See http://docs.sqlalchemy.org/en/rel_0_7/orm/extensions/associationproxy.html#simplifying-scalar-collections .

“or dictionaries”. Use attribute_mapped_collection: http://docs.sqlalchemy.org/en/rel_0_7/orm/collections.html#dictionary-collections

“dictionaries plus scalars”: combine both attribute_mapped_collection and association_proxy: http://docs.sqlalchemy.org/en/rel_0_7/orm/extensions/associationproxy.html#proxying-to-dictionary-based-collections

Edit 1:
Well, you dug into a really esoteric and complex example there. association_proxy is a much easier way to get around these cases where you want an object to act like a scalar, so here’s that, without all that crazy boilerplate of the “vertical” example, which I’d avoid as it is really too complex. Your example seemed undecided about primary key style so I went with the composite version. Surrogate + composite can’t be mixed in a single table (well it can, but its relationally incorrect. The key should be the smallest unit that identifies a row – http://en.wikipedia.org/wiki/Unique_key is a good top level read into various subjects regarding this).

from sqlalchemy import Integer, String, Column, create_engine, ForeignKey, ForeignKeyConstraint
from sqlalchemy.orm import relationship, Session
from sqlalchemy.orm.collections import attribute_mapped_collection
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.associationproxy import association_proxy

Base = declarative_base()

class AnimalFact(Base):
    """key/value attribute whose value can be either a string or a list of strings"""
    __tablename__ = 'animalfacts'

    # use either surrogate PK id, or the composite animal_id/key - but
    # not both.   id/animal_id/key all together is not a proper key.
    # Personally I'd go for "id" here, but here's the composite version.

    animal_id = Column(Integer, ForeignKey('animal.id'), primary_key=True)
    key = Column(String, primary_key=True)

    # data
    str_value = Column(String)
    _list_value = relationship('StringEntry')

    # proxy list strings
    list_proxy = association_proxy('_list_value', 'value')

    def __init__(self, key, value):
        self.key = key
        self.value = value

    @property
    def value(self):
        if self.str_value is not None:
            return self.str_value
        else:
            return self.list_proxy

    @value.setter
    def value(self, value):
        if isinstance(value, basestring):
            self.str_value = value
        elif isinstance(value, list):
            self.list_proxy = value
        else:
            assert False

class Animal(Base):
    __tablename__ = 'animal'

    id = Column(Integer, primary_key=True)
    name = Column(String)
    _facts = relationship(AnimalFact, backref='animal',
                          collection_class=attribute_mapped_collection('key'))
    facts = association_proxy('_facts', 'value')

    def __init__(self, name):
        self.name = name

    # dictionary interface around "facts".
    # I'd just use "animal.facts" here, but here's how to skip that.
    def __getitem__(self, key):
        return self.facts.__getitem__(key)

    def __setitem__(self, key, value):
        self.facts.__setitem__(key, value)

    def __delitem__(self, key):
        self.facts.__delitem__(key)

    def __contains__(self, key):
        return self.facts.__contains__(key)

    def keys(self):
        return self.facts.keys()


class StringEntry(Base):
    __tablename__ = 'myvalue'
    id = Column(Integer, primary_key=True)
    animal_id = Column(Integer)
    key = Column(Integer)
    value = Column(String)

    # because AnimalFact has a composite PK, we need
    # a composite FK.
    __table_args__ = (ForeignKeyConstraint(
                        ['key', 'animal_id'],
                        ['animalfacts.key', 'animalfacts.animal_id']),
                    )
    def __init__(self, value):
        self.value = value

engine = create_engine('sqlite://', echo=True)
Base.metadata.create_all(engine)

session = Session(engine)


# create a new animal
a = Animal('aardvark')

a['eyes'] = ['left side', 'right side']

a['cute'] = 'sort of'

session.add(a)
session.commit()
session.close()

for animal in session.query(Animal):
    print animal.name, ",".join(["%s" % animal[key] for key in animal.keys()])