Home/ Questions/Q 8089337
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-05T19:21:52+00:00

I have been looking into AWS spot instances for some jobs however instead of

  • 0

I have been looking into AWS spot instances for some jobs however instead of having a static cluster which works from SQS and is mantained through a cronjob (or sorts) on my main server to ensure the number of servers and jobs being done etc I wish to make the servers create and dismantle per job I have to do.

Now I understand how to spin up new spot instances specifying a certain AMI, and I have on this AMI a PHP script in say /home/ubuntu/ called do_job.php but it requires a few parameters (like it would from an SQS message).

Is there a good way, in my server spawner, to call an asynchronous command to the new spot instance once I have verified it is actually running which does not lock my spawner?

Maybe I am looking at this from the wrong perspective? I am a bit of a noob at aws clusters so I might be barking up the wrong tree here…

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I thought I would follow up on this question.

    No there is no easy way of doing this. Currently the easiest way is to have a basic Apache or lightttp or whatever webserver installed on your server, get it’s internal public DNS and call it from your spawning EC2 instance via a async cURL method or what not.

    The more acceptable way of doing something similar is to use a MQ like SQS and create a visibility lock on the job server. When you come to process your MQ on the worker server or whatever spot/on-demand instance you are running to process information you set the visibility for the length of the script run. You can do this with a simple despatcher which makes the processing of the information transform to a background process while your despatcher goes into a while() loop resetting the visibility of the Message everytime it detects the background process is still running. This allows for single server integrity of messages.

    In order to get this to work you would have to scale based upon the size and depth of MessagesVisible within the Amazon Cloudwatch metric telling you how many messages are visible in the queue and have been for a while. It is best to have a delay period on this metric so:

    • for every 60 secs check the metric
    • for every 10 mins the average is greater than 10 visible messages scale up the servers.

    That is a more acceptable way to scale a job cluster.

    • 0