Question

I am using in-mapper combining in a Map Reduce job via the Python mrjob module. Because I wrote a mapper_final function that emits a single pair, I am sure that only a single key-value pair is emitted to my reducers.

However, my reduce function is erring:

  def reducer(self, key, occurrences):
    '''
    Calculates the final value.
    '''
    yield 'Final Value: ', occurrences[0] / 2

The error reads

File "calculateFinalValue.py", line 354, in reducer
    yield 'Final Value: ', occurrences[0] / 2
TypeError: 'generator' object has no attribute '__getitem__'

Why can I not index into occurrences? There should only be a single pair in that list, right?

Answer 1

occurrences is not a list, it is a generator. If you want a list, you need to assemble the generator results into a list. Something like:

list_occurrences = [ occ for occ in occurrences ]

or

list_occurrences = list(occurrences)

yield 'Final Value: ', list_occurrences[0] / 2

Or you can get the first value of occurrences with occurrences.next():

yield 'Final Value: ', occurrences.next() / 2