Question

I was trying out the simplistic word count example for hadoop pipes. Unfortunately it is erroring out with java.lang.NullPointerException and /usr/lib64/libstdc++.so.6: no version information available errors when I try to run it as:

$ bin/hadoop pipes -Dhadoop.pipes.java.recordreader=true \
                   -Dhadoop.pipes.java.recordwriter=true \ 
                   -input <inputHDFSPath> \               
                   -output <outputHDFSPath> \
                   -program <cppBinaryHDFSPath>

The error details are here.

Any suggestions on how to fix this would be greatly appreciated. Thanks.

Edit1: Added command used to run.

Edit2: It turns out that the C++ binary wasn’t statically linked. After rectifying that I am now seeing this error.

Edit3: Added hadoop version being used. I have also added relevant portions of the jobtracker logs here. Also, stderr for each of the failed attempts has this:

$more /usr/local/dp/hadoop/userlogs/job_201107211831_0013/\
attempt_201107211831_0013_m_‌​000000_0/stderr 
Hadoop Pipes Exception: Aborted by driver

Answer 1

So I gave it another shot and started from scratch. This time round I had better success. Here is the code and the config files I used.

Also, here’s the command I used to run the job:

hadoop pipes -conf path/to/dp_hpt_conf.xml 
             -input /path/to/input 
             -output /path/to/output

The input files in the /path/to/input were:

$hadoop fs -ls /path/to/input
Found 3 items
-rw-r--r--   3 dp dpgrp         25 2011-07-26 23:24 /path/to/input/bar
-rw-r--r--   3 dp dpgrp         87 2011-07-26 23:24 /path/to/input/baz
-rw-r--r--   3 dp dpgrp         44 2011-07-26 23:24 /path/to/input/foo
$hadoop fs -get /path/to/input/foo .
$cat foo 
The quick brown fox jumps over the lazy dog

And the output was:

$ hadoop fs -get /path/to/output/part-00000 .
$ cat part-00000 
another 3
be      2
brown   2
dog     2
end     2
for     2
fox     2
input   2
is      3
jumps   2
lazy    2
of      2
over    2
program 2
quick   2
sentence        3
should  2
test    2
that    2
the     5
this    4
yet     2

(Apologies for the wonky paths. My actual paths were different – the paths here are just my lazy attempts at anonymizing them.)