Question

I want to add something to the hadoop counters from outside a mappper.
So, I want to access the getCounter on context object like this:

context.getCounter(counter, key).increment(amount)

I’m not able to get the context object from where I start the job. I can only do

job.getCounters().findCounter()

which doesn’t let me add something to the hadoop counters.

Answer 1

You can only use/write to the counters from within the mapper/reducer tasks. The job tracker has built in capabilities to interactz with the counters and you don’t really want to interfere with what is already a complex setup.

I had exactly this issue a few months ago, trying to use the counters to store interim information, but I decided to write the informtion I needed to a defined hdfs directory and read that once my job was complete.

EDIT: why and whatdo you want to use the counter for outside of the mapper?

EDIT #2: if you want stats from a finished job, then counters are not the right place for that, as a) they don’t seem to bewritable once the job tracker is done collecting data and b) they are intended to be used for aggregating metrics across tasks. I had a similar need recently and endedup doing my stats sums in the job set-up class (on my edge node) and then writing the data to the logs.