Question

I am trying to read a file with hadoop fs -text. The file is a sequence file defined in my own classes – for that I am using parameter -libjars. Normally. This time I am getting InvocationTargetException with no stacktrace:

# hadoop fs -libjars myjar.jar -text /tmp/part-r-00000
12/09/21 17:24:18 WARN snappy.LoadSnappy: Snappy native library is available
12/09/21 17:24:18 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/09/21 17:24:18 INFO snappy.LoadSnappy: Snappy native library loaded
text: java.lang.reflect.InvocationTargetException

It is hard to see what is going on.

Answer 1

Probably something to do with nested classes or dependencies classes in your Key / Value class.

the -libjars option when used the FsShell (hadoop fs) adds your jar to the classpath (as you’d expect), so your root Key / Value class is now available. What is probably happening here is your Key / Value class depends on another class either not in the jar listed in libjars, or is in a nested jar (in the libs folder in the jar).

This is made confusing by the fact the hadoop jar command will unpack you jar and add all the nested jars in the lib folder to the classpath when it submits a job, but the FsShell command does not unpack the jar.

Annoyingly the FsShell doesn’t dump full stack traces to stderr, but if you enable debug level logging you should see the full stack trace (not sure how you do this though)