Home/ Questions/Q 7893271
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-03T07:05:24+00:00

I have a relatively simple program written in C++ and I have been using

  • 0

I have a relatively simple program written in C++ and I have been using Hadoop Streaming for MapReduce jobs (my version of Hadoop is Cloudera).

Recently, I found that a lot of streaming tasks are keep failing and restarted by task tracker while they finish successfully at the end. I tracked the user logs and it seems some MapReduce tasks are getting zero input. Specifically the error message looks like this:

HOST=null
USER=mapred
HADOOP_USER=null
last Hadoop input: |null|
last tool output: |TCGA-06-0216-0000024576-0000008192   0   27743   10716|
Date: Sun Apr 29 15:55:51 EDT 2012
java.io.IOException: Broken pipe

Sometimes the error rate is pretty high (nearly 50%). I don’t think it’s normal.
Does anyone know

a) What’s going on ?

b) How can I fix it ?

Thanks

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Does your data have a lot of characters in other languages (e.g. Chinese)?

    If so, check your character encoding setting in
    (1) JVM for your Hadoop cluster : it is likely to be set at UTF-8 by default.
    (2) your mapper / reducer : make sure your mapper / reducer emits characters in UTF-8 (or whichever char encoding you have set your JVM)

    • 0