Question

when you use pigServer.registerFunction, you’re not supposed to explicitly call pigServer.registerJar, but rather have pig automatically detect the jar using jarManager.findContainingJar.

However, we have a complex UDF who’s class is dependent on other classes from multiple jars. So we created a jar-with-dependencies with the maven-assembly. But this causes the entire jar to enter pigContext.skipJars (as it contains the pig.jar itself) and not being sent to the hadoop server 🙁

What’s the correct approach here? Must we manually call registerJar for every jar we depend on?

Answer 1

not sure what’s the certified way, but here’s some pointers:

• when you use pigServer.registerFunction pig automatically detects the jar that contain the udfs and sends it to the jobTracker
• pig also automatically detects the jar that contains PigMapReduce class (JarManager.createJar), and extracts from it only the classes that start with org/apache/pig, org/antlr/runtime, etc. and sends them to the jobTracker as well
• so, if your UDF sits in the same jar as PigMapReduce your’e screwed, because it won’t get sent
• our conclusion: don’t use jar-with-dependencies

HTH