Question

We have some part of our application that need to load a large set of data (>2000 entities) and perform computation on this set. The size of each entity is approximately 5 KB.

On our initial, naïve, implementation, the bottleneck seems to be the time required to load all the entities (~40 seconds for 2000 entities), while the time required to perform the computation itself is very small (<1 second).

We had tried several strategies to speed up the entities retrieval:

• Splitting the retrieval request into several parallel instances and then merging the result: ~20 seconds for 2000 entities.
• Storing the entities at an in-memory cache placed on a resident backend: ~5 seconds for 2000 entities.

The computation needs to be dynamically computed, so doing a precomputation at write time and storing the result does not work in our case.

We are hoping to be able to retrieve ~2000 entities in just under one second. Is this within the capability of GAE/J? Any other strategies that we might be able to implement for this kind of retrieval?

UPDATE: Supplying additional information about our use case and parallelization result:

• We have more than 200.000 entities of the same kind in the datastore and the operation is retrieval-only.
• We experimented with 10 parallel worker instances, and a typical result that we obtained could be seen in this pastebin. It seems that the serialization and deserialization required when transferring the entities back to the master instance hampers the performance.

UPDATE 2: Giving an example of what we are trying to do:

1. Let’s say that we have a StockDerivative entity that need to be analyzed to know whether it’s a good investment or not.
2. The analysis performed requires complex computations based on many factors both external (e.g. user’s preference, market condition) and internal (i.e. from the entity’s properties), and would output a single “investment score” value.
3. The user could request the derivatives to be sorted based on its investment score and ask to be presented with N-number of highest-scored derivatives.

Answer 1

In the end, it does not appear that we could retrieve >2000 entities from a single instance in under one second, so we are forced to use in-memory caching placed on our backend instance, as described in the original question. If someone comes up with a better answer, or if we found a better strategy/implementation for this problem, I would change or update the accepted answer.