Home/ Questions/Q 8513629
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-11T04:37:43+00:00

We’re using cdh3u4, Hadoop and HBase. I’m trying to run a unit test that

  • 0

We’re using cdh3u4, Hadoop and HBase. I’m trying to run a unit test that kicks off a MapReduce job after starting up the miniMapReduceCluster provided by the HBaseTestingUtility.

The jobs fail with this in the map and reducer task stderr logs:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Child
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Child
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
 Could not find the main class: org.apache.hadoop.mapred.Child.  Program will exit.
java.lang.Throwable: Child Error

I’ve been trying to figure this out for days. I’m guessing its a mis – Configuration and that the cluster is not finding any of my jars due to misconfigured fs / hdfs config values.
My test setup code looks like this (excuse typos as this is translation from Scala):

HBaseTestingUtility htest = new HBaseTestingUtility();
Configuration c = htest.getConfiguration();
c.set("hadoop.log.dir", "/tmp/hadoop-test-logs"); // required or else can't start the miniMapReduceCluster
htest.startMiniCluster();
htest.startMiniMapReduceCluster();

// create and run a MapReduce job that works in production but not in test

In case this is important, we are using Play! framework 2.0 (which uses SBT) with Specs2 testing framework and Scala. I don’t think it should matter (that we aren’t using Java + JUnit).

Has anyone seen this before? Any pointers on where to look?

Thanks in advance,

Mark

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Turns out we have to manually setup the classpath on the minicuster. This may be an SBT / Ivy thing – since all examples I’ve seen don’t have to do this and are presumably using Maven (and Java).

    Here’s how I resolved the classpath issue (in Scala):

    // unit test setup:
val htest = new HBaseTestingUtility
htest.startMiniCluster()

val conf = htest.getConfiguration
conf.set("hadoop.log.dir", "/tmp/hadoop-test-logs") // required to prevent NPE when starting miniMapReduceCluster
htest.startMiniMapReduceCluster()

// Set up cluster classpath:
val fs = FileSystem.get(conf)
val jarsDir = new Path("/tmp/hadoop-test-lib")
fs.mkdirs(jarsDir)

// copy jars over to hdfs and add to classpath:
for (jar <- myjars) {
    val localJarPath = new Path("file://%s".format(jar.getAbsolutePath))
    val hdfsJarPath = new Path("/tmp/hadoop-test-lib/%s".format(jar.getName))
    fs.copyFromLocalFile(localJarPath, hdfsJarPath)
    DistributedCache.addFileToClassPath(hdfsJarPath)
}

// now Map and Reduce tasks can find classes
    • 0