Question

Database structure

I have two classes, A and B, that have two different relationships:

1) Many-to-many relationship using an association table (associations) to store information relevant only to that particular association (association_property_1) and instanced through backrefs in A and B.

2) One-to-one relationship between A and B using a foreign key in table_b, such that only B ‘knows’ about this relationship. I don’t care if A knows about it, but it just seemed simpler this way.

My classes look like this:

class A(Base):
  __tablename__ = 'table_a'

  id = Column(Integer, primary_key=True)
  a_property_1 = Column(Float)
  a_property_2 = Column(Float)
  a_property_special = Column(Float)

  # Many-to-many relationship with B through an Association
  associated_bs = relationship('Association', backref='a')

class B(Base):
  __tablename__ = 'table_b'

  id = Column(Integer, primary_key=True)
  b_property_1 = Column(Float)
  b_property_2 = Column(Float)

  # One-to-one relationship with A
  a_id = Column(Integer, ForeignKey('table_a.id'))
  a = relationship('A', uselist=False, backref='b')

  # Many-to-many relationship with A through an Association
  associated_as = relationship('Association', backref='b')

class Association(Base):
  __tablename__ = 'associations'

  a_id = Column(Integer, ForeignKey('table_a.id'), primary_key=True)
  b_id = Column(Integer, ForeignKey('table_b.id'), primary_key=True)

  association_property_1 = Column(Float)

Procedure

I want to run a query on all associations, where I have access to the special property of A through the one-to-one relationship with B. So basically I want to be able to access the property

B.a.a_property_special

inside a query.

An example of a particular query could be the following:

session.query(Association.association_property_1,
  func.abs(A.a_property_special - B.a.a_property_special).\
  filter(B.a.a_property_special > 3.0)

where A and B are joined using the many-to-many relationship and B.a is joined through the one-to-one. Obviously this query won’t work as B is not instanced, so I won’t have access to B.a.a_property_special.

If I did not have the many-to-many relationship I could just join A on B and be done with it. My problem is that I want to query both A and B using the association, but I still need the scalar B.a.a_property_special through the one-to-one relationship.

Possible solutions

I have tried several different solutions, but all have proved unsatisfactory for various reasons.

• Copy column ‘a_property_special’ to table B. This I don’t like because it duplicates information and does not present a nice logical data structure if the one-to-one relationship between A and B changes (which it might during runtime).
• Use a column_property or an association_proxy. Seem nice and clean, but I can only get it to work properly on instanced objects. When using them in a query I get problems constructing binary expressions etc.
• Using subqueries. I have fiddled around with this a bit, but haven’t been able to produce anything that works well. Maybe I’m just not doing it right, but it seems to always end up being very cluttered and slow.
• Simply query all Association(s) and do the math, logical expressions and filtering in python. My feeling is that this would be less efficient than doing it in SQL, but I could be wrong..

Requirements

• It needs to be fast (duh). My tables have a few times 100,000 records each.
• The query has to be as simply as possible, so that it is easy to debug and modify, while still reflecting the logical structure of the database. I would prefer to keep as much code as possible tucked away inside the class definitions.
• I do not have any particular preference for the structure of the relationships, I simply need a one-to-one and a many-to-many (including its own associated properties).

I have a feeling that this is really simple, but I just can’t seem to find a good solution. Any help or comments are welcomed.

Answer 1

SQLAlchemy is explicit about joins so when you see something like:

session.query(B).filter(B.a.a_property_special > 3.0)

that really means this:

session.query(B).join(B.a).filter(A.a_property_special > 3.0)

there is a subquery case also, which is not as efficient as a join. The subquery case always requires the usage of correlated subqueries, like this:

subq = session.query(A.a_property_special).where(A.id == B.a_id).correlate(B).as_scalar()
session.query(B).filter(subq > 3.0)

When using relationships, you also have access to the any() and has() methods, which render an EXISTS subquery for one-to-many, many-to-one, respectively:

session.query(B).filter(B.a.has(A.a_property_special > 3.0))

the above is the equivalent to this:

from sqlalchemy import exists

session.query(B).filter(exists().where(B.a_id==A.id, A.a_property_special > 3.0))

the advantage with the subquery is that it can be used to create self-contained filter criterion, whereas when relying upon a join(), there’s no way for that to happen implicitly. But the subquery approach doesn’t perform as well on the database side.

There’s of course lots of simple cases where joins could be implicitly added to the enclosing query based on various things present, and this is what ORMs like Django do, but SQLAlchemy’s take on it is that you very quickly get into an ocean of cases where a simplistic approach like that breaks down, so we instead don’t make guesses like that in the library.

So to take your original query example:

session.query(Association.association_property_1,
  func.abs(A.a_property_special - B.a.a_property_special)).\
  filter(B.a.a_property_special > 3.0)

You’re actually trying to hit on A in two different ways, so when doing the explicit join route, you need to make an alias of it so that it can be targeted twice:

from sqlalchemy.orm import aliased
a_alias = aliased(A)
session.query(
      Association.association_property_1,
      func.abs(A.a_property_special - a_alias.a_property_special)
     ).\
     join(Association.a).\
     join(Association.b).join(a_alias, B.a).\
     filter(a_alias.a_property_special > 3.0)

this builds the same way you’d do it in SQL, basically. The SQL is this:

SELECT associations.association_property_1 AS associations_association_property_1, abs(table_a.a_property_special - table_a_1.a_property_special) AS abs_1 
FROM associations JOIN table_a ON table_a.id = associations.a_id JOIN table_b ON table_b.id = associations.b_id JOIN table_a AS table_a_1 ON table_a_1.id = table_b.a_id 
WHERE table_a_1.a_property_special > :a_property_special_1

The subquery route here would be hard on the database. While you could wire up attributes on Association that render subqueries, they’d all need to be called as correlated subqueries, which would perform terribly especially if you referred to them multiple times in one query. Here is how to do that using hybrid attributes:

class Association(Base):
    __tablename__ = 'associations'

    a_id = Column(Integer, ForeignKey('table_a.id'), primary_key=True)
    b_id = Column(Integer, ForeignKey('table_b.id'), primary_key=True)

    association_property_1 = Column(Float)

    @hybrid.hybrid_property
    def a_property_special(self):
        return self.a.a_property_special

    @a_property_special.expression
    def a_property_special(cls):
        return select([A.a_property_special]).where(A.id==cls.a_id).as_scalar()

    @hybrid.hybrid_property
    def b_a_property_special(self):
        return self.b.a.a_property_special

    @b_a_property_special.expression
    def b_a_property_special(cls):
        return select([A.a_property_special]).where(A.id==B.a_id).where(B.id==cls.b_id).as_scalar()

session.query(
  Association.association_property_1,
  func.abs(Association.a_property_special - Association.b_a_property_special)
 )

SQL here is:

SELECT associations.association_property_1 AS associations_association_property_1, abs((SELECT table_a.a_property_special 
FROM table_a 
WHERE table_a.id = associations.a_id) - (SELECT table_a.a_property_special 
FROM table_a, table_b 
WHERE table_a.id = table_b.a_id AND table_b.id = associations.b_id)) AS abs_1 
FROM associations

the database is given less information about how the rows from these three tables are related to each other for the purpose of this query, therefore it has to do more work when fetching rows. The join case, while it requires that you lay out “A” as a target in two different ways and also specify how things join, gives the database a simpler task, as joins are more efficient than calculating correlation of a related SELECT for each row of a parent rowset.