Question

I am having 8 powershell scripts. Few of them having dependencies. It means they can’t be executed in parallel. They should be executed on after another.

Some of the Powershell scripts has no dependency and it can be executed in parallel.

Following is the dependency explained in detail

    Powershell scripts 1, 2, and 3 depend on nothing else
    Powershell script 4 depends on Powershell script 1
    Powershell script 5 depends on Powershell scripts 1, 2, and 3
    Powershell script 6 depends on Powershell scripts 3 and 4
    Powershell script 7 depends on Powershell scripts 5 and 6
    Powershell script 8 depends on Powershell script 5

I knew that by manually hard coding the dependency is possible. But 10 more powershell scripting may be added and dependency among them may added.

Has any one acheived parallelism by finding dependency? If so please share me how to proceed.

Answer 1

How familiar with parallel programming in general are you? Have you heard of and used the concept of mutual exclusion? The concept in general is to use some kind of messaging/locking mechanism to protect a shared resource among different parallel threads.

In your case, you’re making the dividing lines be the scripts themselves – which I think may make this much simpler than most of the techniques outlined in that wikipedia article. Would this simple template work for what you’re looking for?

1. Define a folder in the local file system. This location will be known to all scripts (default parameter).
2. Before running any of the scripts, make sure any files in that directory are deleted.
3. For each script, as the very last step of their execution, they should write a file in the shared directory with their script name as the name of the file. So script1.ps1 would create script1 file, for example.
4. Any script that has a dependency on another script will define these dependencies in terms of the file names of the scripts. If script3 is dependent on script1 and script2, this will be defined as a dependency parameter in script3.
5. All scripts with dependencies will run a function that checks if the files exist for the scripts it’s dependent on. If they are, it proceeds with the execution of the script, otherwise it pauses until they are complete.
6. All scripts get kicked off simultaneously by a master script / batch file. All of the scripts are ran as PowerShell jobs so that the OS will run their execution in parallel. Most of the scripts will start up, see they have dependencies, and then wait patiently for these to get resolved before continuing with the actual execution of the script body.

The good news is that this would allow for flexible changing of dependencies. Every script writes a file, making no assumption about whether someone else is waiting for them or not. Changing the dependency of a particular script would be a simple one-line change or change of input parameter.

This is definitely not a perfect solution though. For instance what would happen if a script fails (or your script can exit in multiple different code paths but you forget to write the file in one of them)? This could cause a deadlock situation where no dependent scripts will get kicked off. The other bad thing is the busy wait of sleeping or spinning while waiting for the right files to get created – this could be corrected by implementing an Event-based approach where you have the OS watch the directory for changed.

Hope this helps and isn’t all garbage.