Question

I need help with the Pythonic looping overhead of the following problem: I’m writing a function that calculates a pixel flow algorithm that’s a classic dynamic programming algorithm on a 2D Numpy array. It requires:

1) visiting all the elements of the array at least once like this:

for x in range(xsize):
    for y in range(ysize):
         updateDistance(x,y)

2) potentially following a path of elements based on the values of the neighbors of an element which looks like this

while len(workingList) > 0:
   x,y = workingList.pop()
   #if any neighbors of x,y need calculation, push x,y and neighbors on workingList 
   #else, calculate flow on pixels as a sum of flow on neighboring pixels

Unfortunately, I seem to be getting a lot of Pythonic loop overhead on #1 even if the call to updateDistance is pass. I figure this is a classic enough algorithm that there must be a good approach to it in Python that can avoid some of the looping overhead. I’m also worried that if I can fix #1 I’m going to get busted on the loop in #2.

Any suggestions about quickly looping through elements in a 2D numpy array and potentially updating chains of elements in that array?

Edit: Flushing out more details of #2

It seems I might be able to vectorize the first loop, perhaps with vectorizing an np.meshgrid call.

The loop in part is a little complicated, but here’s a simplified version. I’m concerned about both the loop and the indexing into the neighboring elements:

#A is a 2d cost matrix
workingList = [(x,y)]
while len(workingList) > 0:
   x,y = workingList.pop()
   neighborsToCalculate = []
   for n in neighborsThatNeedCalculation(x,y): #indexes A to check neighbors of (x,y)
      neighborsToCalculate.append(n)
   if len(neighborstToCalculate) != 0:
       workingList.append((x,y))
       workingList.extend(neighborsToCalculate)
   else:
       for xn,yn in neighbors(x,y):
          A[x,y] += 1+A[xn,yn]

This is a classic breadth first search problem. It would be great if this could be parallelized. It probably can’t in its current form because it follows a path, but I’d be delighted for suggestions.

Answer 1

you won’t get any speed boost from numpy if you use python loops in your algorithm. You need to parallelize your problem.

In image processing, parallelization means using the same function on all pixels, i.e using kernels. In numpy, instead of doing:

for x in range(xsize):
    for y in range(ysize):
         img1[y, x] = img2[y, x] + img3[y, x]

you do:

img1 = img2 + img3 # add 2 images pixelwise

so that the loop happens in c. The fact that you have a list of neighbors with unknown length for each pixel make your problem difficult to parallelize in this way. You should either rework your problem (could you be a bit more specific about you algorithm?), or use another language, like cython.

edit:

you won’t get benefit from Numpy without changing your algorithm. Numpy allows you to perform linear algebra operations. You can’t avoid looping overhead with this library when performing arbitrary operations.

To optimize this, you may consider:

• switching to another language like cython (which is specialized in python extensions) to get rid of looping costs

• optimizing your algorithm: If you can get the same result using only linear algebra operations (this depend on the neighborsThatNeedCalculation function), you may use numpy, but you will need to work out a new architecture.

• using parallelization techniques like MapReduce. With python you may use a pool of workers (available in the multiprocessing module), you will get more speed gains if you switch to another language, since python will have other bottlenecks.

In the case you want something easy to setup and to integrate, and you just need to have c-like performances, I strongly suggest cython if you can’t rework your algorithm.