Question

How can I add steps to a waiting Amazon EMR job flow using boto without the job flow terminating once complete?

I’ve created an interactive job flow on Amazon’s Elastic Map Reduce and loaded some tables. When I pass in new steps to the job flow using Boto’s emr_conn.add_jobflow_steps(...), the job flow terminates after it finishes or fails.

I know I can start a job flow with boto using run_jobflow with the keep_alive parameter — but I’d like to work with flows that are already running.

Answer 1

If it finishes correctly, it should not terminate with keep_alive=True. That said, it would normally exit on failure, so you want to add terminate_on_failure="CONTINUE" to your add_job_steps arguments.