Question

For a model like:

class Thing(ndb.Model):
    visible = ndb.BooleanProperty()
    made_by = ndb.KeyProperty(kind=User)
    belongs_to = ndb.KeyProperty(kind=AnotherThing)

Essentially performing an ‘or’ query, but comparing different properties so I can’t use a built in OR… I want to get all Thing (belonging to a particular AnotherThing) which either have visible set to True or visible is False and made_by is the current user.

Which would be less demanding on the datastore (ie financially cost less):

1. Query to get everything, ie: Thing.query(Thing.belongs_to == some_thing.key) and iterate through the results, storing the visible ones, and the ones that aren’t visible but are made_by the current user?

2. Query to get the visible ones, ie: Thing.query(Thing.belongs_to == some_thing.key, Thing.visible == "True") and query separately to get the non-visible ones by the current user, ie: Thing.query(Thing.belongs_to == some_thing.key, Thing.visible == "False", Thing.made_by = current_user)?

Number 1. would get many unneeded results, like non-visible Things by other users – which I think is many reads of the datastore? 2. is two whole queries though, which is also possibly unnecessarily heavy, right? I’m still trying to work out what kinds of interaction with the database cause what kinds of costs.

I’m using ndb, tasklets and memcache where necessary, in case that’s relevant.

Answer 1

Number two is going to be financially less for two reasons. First you pay for each read of the data store and for each returned entity in a query, therefore you will be paying more for the first one which you have to Read all data and query all data. The second way you only pay for what you need.

Secondly you also pay for backend or frontend time, and you will be using time to iterate through all your results in the first method, where as you need to spend no time for the second method.

I can’t see a way where the first option is better. (maybe if you only have a few entities??)

To understand how reads and queries cost you scroll down a little on:
https://developers.google.com/appengine/docs/billing

You will see how Read, Writes and Smalls are added up for reads, writes and queries.

I would also just query for ones that are owned by the current user instead of visible=false and owner=current, this way you don’t need a composite index which will save some time. You can also make visible a partial index this was saving some space as well (only index it when true, assuming you never need to query for false ones). You will need to do a litte work to remove duplicates, but that is probably not to bad.