Question

I have a problem with my code running on google app engine. I dont know how to modify my code to suit GAE. The following is my problem

for j in range(n):
 for d in range(j):
  for d1 in range(d):
   for d2 in range(d1):
    # block which runs in O(n^2)

Efficiently the entire code block is O(N^6) and it will run for more than 10 mins depending on n. Thus I am using task queues. I will also be needing a 4 dimensional array which is stored as a list (eg A[j][d][d1][d2]) of n x n x n x n ie needs memory space O(N^4)

Since the limitation of put() is 10 MB, I cant store the entire array. So I tried chopping into smaller chunks and store it and when retrieve combine them. I used the json function for this but it doesnt support for larger n (> 40).

Then I stored the whole matrix as individual entities of lists in datastore ie each A[j][d][d1] entity. So there is no local variable. When i access A[j][d][d1][d2] in my code I would call my own functions getitem and putitem to get and put data from datastore (used caching also). As a result, my code takes more time for computation. After few iterations, I get the error 203 raised by GAE and task fails with code 500.

I know that my code may not be best suited for GAE. But what is the best way to implement it on GAE ?

Answer 1

There may be even more efficient ways to store your data and to iterate over it.

Questions:

• What datatype are you storing, list of list ... of int?
• What range of the nested list does your innermost loop O(n^2) portion typically operate over?
• When you do the putitem, getitem how many values are you retrieving in a single put or get?

Ideas:

• You could try compressing your json (and base64 for cut and pasting). 'myjson'.encode('zlib').encode('base64')
• Using a divide and conquer (map reduce) as @Robert suggested. You may be able to use a dictionary with tuples for keys, this may be fewer lookups then A[j][d][d1][d2] in your inner loop. It would also allow you to sparsly populate your structure. You would need to track and know your bounds of what data you loaded in another way. A[j][d][d1][d2] becomes D[(j,d,d1,d2)] or D[j,d,d1,d2]