Question

bellow is my cronjob script what i want to only

wait for 3rd_job completion for starting 4th,5th

& wait for 2nd_job completion for starting 6,7,8th job.

how can i achieve this thanks in advance

currently for starting 4,5th job it is waiting for the completion of 1,2,3th job but i just want to wait for 3rd for starting 4,5th and wait for only 2nd completion for stating 6,7,8th

    #!/bin/bash
    cd /home/ubuntu/
    PATH=$PATH:/usr/local/bin
    export PATH
    nohup scrapy crawl 1st_job &
    nohup scrapy crawl 2nd_job &
    nohup scrapy crawl 3rd_job &
    wait $(pgrep 3rd_job)
    nohup scrapy crawl 4th_job &
    nohup scrapy crawl 5th_job &
    wait $(pgrep 2nd_job)
    nohup scrapy crawl 6th_job &
    nohup scrapy crawl 7th_job &
    nohup scrapy crawl 8th_job &

Answer 1

You can use the $! variable that contains the PID of the last spawned asynchronous task:

#!/bin/bash
cd /home/ubuntu/
PATH=$PATH:/usr/local/bin
export PATH
nohup scrapy crawl 1st_job &
nohup scrapy crawl 2nd_job &
pid_2nd="$!"
nohup scrapy crawl 3rd_job &
wait $!
nohup scrapy crawl 4th_job &
nohup scrapy crawl 5th_job &
wait $pid_2nd
nohup scrapy crawl 6th_job &
nohup scrapy crawl 7th_job &
nohup scrapy crawl 8th_job &

If you want something a little more complex, for instance execute in a directed acyclic graph fashion (for example as soon as 3rd finishes start 4-5 and as soon 2nd finishes start 6-8) I suggest you reorganize the calls into functions, so that they can be called asynchronously:

#!/bin/bash
second () {
    nohup scrapy crawl 2nd_job
    nohup scrapy crawl 6th_job &
    nohup scrapy crawl 7th_job &
    nohup scrapy crawl 8th_job &
}

third () {
    nohup scrapy crawl 3rd_job
    nohup scrapy crawl 4th_job &
    nohup scrapy crawl 5th_job &
}

cd /home/ubuntu/
PATH=$PATH:/usr/local/bin
export PATH
nohup scrapy crawl 1st_job &
third &
second &

Hope this helps =)