Question

I have a problem with my app on Google AppEngine, especially with the datastore.
The story is a bit long. My data model looks like this:

1. Users can create entities. Each entity has the following properties:
   #1). the auto generated key. #2). The creator. #3). A number.
2. In the app, I’m doing the following queries:
   a). Query entities by creator (#2). b). Query entities by the number property (#3). c). Query the entity by id(#1).

I’m using the low level datastore API. Now, here are several problematic approaches I have tried:

[Approach 1]: With the Datastore, if I make the entities parentless (no ancester), then things are good, especially the query ‘c’ can be easily achieved by Datastore.get(k).

Problem of Approach1: Since the datastore is “eventually consistent”, for query ‘a’ the result is ALWAYS not update to date. If a user creates an entity, then lists all entities he creates, the newly created one is always missing. It only appears by refresh in a few seconds later…Tried using memcache, does not help the case (or I’m not finding a good way?).

To overcome the problem above, I changed the data structure. Here’s [Approach 2]: when creating each entity, create a key based on the creator, and put the new entity as child of the key. So all entities created by a user are entities with an ancester. Thus query ‘a’ works in a timely manner, and query ‘b’ also seems to work.

The new problem of Approach2: For query ‘c’, it always returns nothing. Here’s my query ‘c’. Both of them can not find the entity, always:

method 1:
Entity en = datastore.get(k)

method2:

DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Query q = new Query(kind)
    .setFilter(FilterOperator.EQUAL.of(Entity.KEY_RESERVED_PROPERTY, k));
Entity en = ds.prepare(q).asSingleEntity();

Another try: Approach3:
Group all entities under one ancestor. This overcomes the problem in Approach1, and can be queried with an ancestor filter so it’s ok to retrieve the result.

Problem of Approach3: Such structure groups all entities as one entity group. Per Google datastore specification, write operation can happen 5 times at the most per second. This is a limit I’d not want to hit…

I think this kind of data model is really common. Is there a proper way to structure the data? Any help is appreciated. Thanks…

Answer 1

I’m not quite sure what you are asking due to the formatting of your question but…

When you query by ID you also have to include the parent key. The key includes both the parents and the child data, and if you are only using the child data then it won’t return what you are looking for as that’s not it’s “name”. The full path (ancestor/child) is it’s name.

So include the parent in the key you are constructing, then get it.

key = parent, parent_ID, child, child_ID

as

key = child, ID

won’t work if it has a parent, as the key is incomplete and it can’t return the data you want like that.