Home/ Questions/Q 7646851
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-31T10:18:55+00:00

When I create a streaming job with Amazon Elastic MapReduce (Amazon EMR), using the

  • 0

When I create a streaming job with Amazon Elastic MapReduce (Amazon EMR), using the Ruby command line interface, how can I specify to use only EC2 spot instances (except for master)? The command below is working, but it “forces” me to use at lease 1 core instance…

./elastic-mapreduce --create --stream          \
--name    n2_3                             \
--input   s3://mr/neuron/2              \
--output  s3://mr-out/neuron/2          \
--mapper  s3://mr/map.rb         \
--reducer s3://mr/noop_reduce.rb \
--instance-group master --instance-type m1.small --instance-count 1 \
--instance-group core   --instance-type m1.small --instance-count 1 \
--instance-group task   --instance-type m1.small --instance-count 18 --bid-price 0.028

Thanks

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Both CORE and TASKS nodes run TaskTrackers but only CORE nodes run DataNodes so, yes, you need at least one CORE node.

    So you could run spot core nodes?

    ./elastic-mapreduce --create --stream \
...
--instance-group master --instance-type m1.small --instance-count 1 \
--instance-group core   --instance-type m1.small --instance-count 19 --bid-price 0.028

    p.s. you also could run one CORE and many TASK nodes but, depending on how much reading/writing you’re doing, you’ll have pain since 18 nodes will be reading/writing to 1 node.

    # expect problems....
./elastic-mapreduce --create --stream \
...
--instance-group master --instance-type m1.small --instance-count 1 \
--instance-group core   --instance-type m1.small --instance-count 1  --bid-price 0.028
--instance-group task   --instance-type m1.small --instance-count 18 --bid-price 0.028
    • 0