I’m trying to think of a solution regarding a way to run existing executables in parallel. What I came up so far is making a wrapper that puts the specified executable in a queue (managed by a ‘server’). The problem that I have is when I need to run multiple executables whose output is another’s executable input.
Example:
I have an executable A whose output is a file (an image).
I have another executable B whose input is a list of files (consider it creates thumbnails from A‘s images, stores them in an archive, etc)
The question is: is there a way to make the B executable wait for the multiple A processes to finish?
Is there a standard way of doing this? I’m interested in the concept and if it is possible, rather than the language. It would be nice if the solution is cross platform but for the moment I don’t have any solutions so feel free to share your ideas. I assume this will be somehow done in C/C++ as it requires some low level interaction.
SIMPLIFIED:
I have some executables (A) running in background and another executable (B) that uses their output as input. How can I block the execution of B until all the needed input files are available.
NOTE:
executables A run continuously on the machine but some of the input files are ‘collected’ periodically by B.
Thanks for any suggestions,
Iulian
If you’re talking about a program that does this for you, GNU Parallel will do this for you. You can set it to run on a cron job, or whatever you think you need. We frequently run it from scripts to rewrite large amounts of text (or code) and fully utilize all of the machine’s cores to do the job. Frequently you can use Parallel along with scripts and sed / awk to get whatever you really need done. I’m not entirely clear if you want to do this programmatically, however, so perhaps this isn’t the best answer.