Question

I have a dictionary that is keyed by date and filled with classes that have an attribute that is a numpy.array. I want to use np.dstack to make one large array from all the arrays in the dictionary. My current code is like this:

import numpy as np
#PARTS is my dictionary
#the .partposit is the attribute that is an array of shape (50000, 12)
ks = sorted(PARTS.keys())
p1 = PARTS[ks[0]].partposit
for k in ks[1:]:
    p1 = np.dstack((p1, PARTS[k].partposit))

My result is as I expect:

In [67]: p1.shape
Out[67]: (50000, 12, 163)

However, it is quite slow. Is there a more efficient way to do this?

Answer 1

you could try this:

>>> import numpy as np
>>> class A:
...     def __init__(self, values):
...         self.partposit = values
... 
>>> PARTS = dict((index, A(np.zeros((50000, 12)))) for index in xrange(163))
>>> p1 = np.dstack((PARTS[k].partposit for k in sorted(PARTS.keys())))
>>> p1.shape
(50000, 12, 163)
>>> 

it took a few seconds to stack it on my machine.

>>> import timeit
>>> timeit.Timer('p1 = np.dstack((PARTS[k].partposit for k in sorted(PARTS.keys())))', "from __main__ import np, PARTS").timeit(number = 1)
2.1245520114898682

numpy.dstack takes in a sequence of arrays and stacks them together as such it would be much faster if we just give it the list instead of continuously stacking them ourselves.

numpy.dstack(tup)

Stack arrays in sequence depth wise (along third axis).
Takes a sequence of arrays and stack them along the third axis to make a single array.

http://docs.scipy.org/doc/numpy/reference/generated/numpy.dstack.html

I was also curious as to see how long your method would be:

>>> import timeit
>>> setup = """
... import numpy as np
... #PARTS is my dictionary
... #the .partposit is the attribute that is an array of shape (50000, 12)
... 
... class A:
...     def __init__(self, values):
...         self.partposit = values
... 
... PARTS = dict((index, A(np.zeros((50000, 12)))) for index in xrange(163))
... ks = sorted(PARTS.keys())
... """
>>> stack = """
... p1 = PARTS[ks[0]].partposit
... for k in ks[1:]:
...     p1 = np.dstack((p1, PARTS[k].partposit))
... """
>>> timeit.Timer(stack, setup).timeit(number = 1)
67.69684886932373

ouch!

>>> numpy.__version__
'1.6.1'

$ python --version
Python 2.6.1

I hope this helps.