Home/ Questions/Q 811369
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-15T01:02:05+00:00

I have a quick Hadoop Streaming question. If I’m using Python streaming and I

  • 0

I have a quick Hadoop Streaming question. If I’m using Python streaming and I have Python packages that my mappers/reducers require but aren’t installed by default do I need to install those on all the Hadoop machines as well or is there some sort of serialization that sends them to the remote machines?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    If they’re not installed on your task boxes, you can send them with -file. If you need a package or other directory structure, you can send a zipfile, which will be unpacked for you. Here’s a Haddop 0.17 invocation:

    $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-0.17.0-streaming.jar -mapper mapper.py -reducer reducer.py -input input/foo -output output -file /tmp/foo.py -file /tmp/lib.zip

    However, see this issue for a caveat:

    https://issues.apache.org/jira/browse/MAPREDUCE-596

    • 0