Question

I am building a distributed accounting system.
In terms of the DB structure and requirements, it’s probably easiest to describe the app as a Twitter-like app, but with a hierarchical DB structure of 14 tables.
A company that uses the app may have 1 and more users, all sharing the company information.

Currently, each entity represents a record type, i.e. customers, invoices etc.
All entities have a parent which is the user of the app. (for HRD query consistency reasons)

Each query to the DB consists of 14 AppEngine queries. One for each table.
The query involves property filtering.

A new requirement is that a query of a user may require different property values based on each of the other users.
This means that we need (14 x the number of company-users) AppEngine queries. This seems far too much.

Kindless Ancestor queries that can filter by properties would be really nice, alas, no can do 🙂

My options are:

1. Set entity kind to User. No parent. This means that all record types are mixed. (The filtered fields exist in all record types).
   This is not pretty. But would you consider that?

2. Have a fixed entity kind and query by filter only. The result is the equivalent of the Kindless Ancestor queries. However, I fear that it will be slow in a multi user use.

Some numbers:
We plan for 10,000 companies, average of 5 users per company and 1 to 5 million records per record type. (x 14 for total)

Thank you for your patience so far.. 🙂

Answer 1

Honestly I found it challenging to follow your description, so this may be off base. Seeing your existing code might help. But I get that you want an efficient alternative to propertied kindless ancestor queries, so let’s start there.

Consider de-normalizing your data model to include a meta-entity just for querying:

class User(db.Model):
  pass

class OwnedObject(db.Expando):
  object_key = db.StringProperty()

class Customer(db.Model):
  name = db.StringProperty()
  created_on = db.DateProperty()

class Invoice(db.Model):
  amount = db.IntegerProperty()
  created_on = db.DateProperty()

# on write
customer = Customer()
customer.name = name
customer.created_on = date.today()
customer.put()

user = User(key_name=users.get_current_user().user_id())

owned_object = OwnedObject(parent=user)
owned_object.object_key = customer.key()
owned_object.created_on = customer.created_on
owned_object.put()

# on read
query = OwnedObject.all()
query.ancestor(user)
query.filter('created_on =', date.today())

entities = db.get([x.object_key for x in query])

So here you’re doing more work on write and less on read.

Each real entity is coupled with an OwnedObject entity that descends from the appropriate ancestor and points to the real entity’s key. OwnedObject is an expando, so you’ll eager-assign any properties that you want to query on (in this example, created_on).

On read, you can query for any properties that you’ve copied to the expando meta-entity, and you can pull back all of a user’s objects with a fixed overhead of one query and one batch get.

Edit: You can accomplish something similar without the meta-entity by using PolyModel.