I want to implement a hadoop reducer for word counting.
In my reducer I use a hash table to count the words.But if my file is extremely large the hash table will use extreme amount of memory.How I can address this issue ?
(E.g A file with 10 million lines each reducer receives 100million words how can he count the words a hash table requires 100million keys)
My current implementation is in python.
Is there a smart way to reduce the amount of memory?
I want to implement a hadoop reducer for word counting. In my reducer I
Share
The most efficient way to do this is to maintain a hash map of word frequency in your mappers, and flush them to the output context when they reach a certain size (say 100,000 entries). Then clear out the map and continue (remember to flush the map in the cleanup method too).
If you still truely have 100 of millions of words, then you’ll either need to wait a long time for the reducers to finish, or increase your cluster size and use more reducers.