Home/ Questions/Q 8140629
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-06T12:13:34+00:00

I get the following error when running mongo-hadoop streaming: java.io.IOException: Cannot run program "mapper.py":

  • 0

I get the following error when running mongo-hadoop streaming:

java.io.IOException: Cannot run program "mapper.py": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
    at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:53)
    at java.lang.ProcessImpl.start(ProcessImpl.java:91)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
    ... 24 more

I have no problem running standard Hadoop python streaming via dumbo or the regular way.

This error was mentioned in Hadoop python streaming in another post

I m running the job like this:

hadoop jar /Volumes/Locodrive/hadoop/mongo-hadoop/streaming/target/mongo-hadoop-streaming-assembly-1.1.0-SNAPSHOT.jar -mapper mapper.py -file mapper.py -reducer reducer.py -file reducer.py -inputURI mongodb://localhost:27017/testdb.docs -outputURI mongodb://localhost:27017/testdb.testhadoop

Using path/absolute path on mapper.py/reducer.py , adding abolute path in -file argument does not help. Standard Hadoop streaming works without any problems so I do not get the error.

Adding mapper.py and reducer.py to hdfs did not help as well.

mapper.py and reducer.py are execuatable and do have a shebang in the first line:

mapper.py

#!/usr/bin/env python

import sys
sys.path.append(".")


from pymongo_hadoop import BSONMapper


def mapper(documents):

    i = 0
    for doc in documents:
        i += 1
        yield {"_id": "test", "count": 1}


BSONMapper(mapper)
print >> sys.stderr, "Done Mapping!!!"

reducer.py

#!/usr/bin/env python
# encoding: utf-8

import sys
sys.path.append('.')


from pymongo_hadoop import BSONReducer


def reducer(key, values):
    print >> sys.stderr, "Processing key %s" % key
    _count = 0
    for v in values:
        _count += v["count"]

    return {"_id": key, "count": _count}

BSONReducer(reducer)

I am running cloudera Hadoop CDH3u3 on OSX.
The Java examples work with no issue

UPDATE

I tried 0.23.1 and get the same error.

running -debug does not delete PackagedJobJar streamjob.jar

When I extract it, mapper.py and reducer.py are in there

These files are also there when running a std streaming job. mongo-haddoop-streaming still generates the above error

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    k got it working
    it should be -files instead -file

    [http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/util/GenericOptionsParser.html][1]

    hadoop jar /Volumes/Locodrive/hadoop/mongo-hadoop/streaming/target/mongo-hadoop-streaming-assembly-1.1.0-SNAPSHOT.jar -files mapper.py,reducer.py -inputURI mongodb://127.0.0.1:27017/mongo_hadoop.yield_historical.in -outputURI mongodb://127.0.0.1:27017/mongo_hadoop.testhadoop -mapper mapper.py -reducer reducer.py -verbose -debug
    • 0