Question

Hello I am using something the Grouper function from python’s itertools to cut large chunks of select where in(idlist) queries down for sqlite performance. The problem is grouper fills in the whole space of the chunksize even if the list is much smaller so I had to add a loop and comparison before which now I want to optimize.

# input list shorter than grouper chunk size
input = (1,2,3,4,5)

grouper(10,input)
# desired output = (1,2,3,4,5)
# actual output = (1,2,3,4,5,None,None,None,None,None)

# current fix for this issue
list_chunks = tuple(tuple(n for n in t if n) for t in grouper(10, input))

I think there has to be a way to do this without this loop and comparison.

Note: using python 2.5

Answer 1

Something like this?

>>> filter(bool, (1,2,3,4,5,None,None,None,None,None))
(1, 2, 3, 4, 5)

For more complicated case (eg, with 0 in list or you need to drop something differ from None) you can implement own lambda:

>>> filter(lambda n: n is not None, (0,1,2,3,4,5,None,None,None,None,None))
(0, 1, 2, 3, 4, 5)

Or even

>>> from functools import partial
>>> from operator import ne
>>> filter(partial(ne, None), (0,1,2,3,4,5,None,None,None,None,None))
(0, 1, 2, 3, 4, 5)