Question

A HPC task either succeeds or fails, but how do I communicate “try later” back to the scheduler, sure I could use fail and resubmit the task but I need a way determine if I’ve failed because something is broken (give up) or try again shortly because this task is waiting on another task (and keep trying until we either error or successfully complete)

Is there a way to achieve this using the HPC API or similar? From what I’ve heard anything non-zero is failure and zero is success, that’s it, surely there must be a nice way to achieve this “try later” behavour.

Background

We are attempting to run a number of HPC tasks in a single job that have complex interdepencences between them in that as the first task is executing other tasks sit and wait until the first task as processed enough of the data so they can make a start (sort of a cascading execution but not in any easy order so we can’t define dependences in HPC).

Initally I was trying to get these multiple tasks shared across multiple cores in such a way they could sleep while waiting for the main task to complete the task they are interested in. Similar to how Windows would timeshare processes. It’s now clear HPC (by design!) only allows one task per core so if you have an eight core machine you can only run eight tasks at once.

The solution appears to be use a batch file or similar to spawn multiple processes, however before I go down that path I’d like to know if the above question is feasible.

Answer 1

1. Unfortunately, there’s no way for a task to fail in a way that will cause the scheduler to retry it.
2. As you suspected, the recommended way to do it would be to have each task run a batch file or powershell script that starts all the processes you want started.
3. If you don’t want to do that, the HPC scheduler in SP2 now allows core over-subscription (more than one task per core), which might be applicable to solving your problem. See here for a guide on how to set it up: Oversubscribe core counts on cluster nodes