What is the best/easiest way to build a minimal task queue system for Linux using bash and common tools?
I have a file with 9’000 lines, each line has a bash command line, the commands are completely independent.
command 1 > Logs/1.log
command 2 > Logs/2.log
command 3 > Logs/3.log
...
My box has more than one core and I want to execute X tasks at the same time. I searched the web for a good way to do this. Apparently, a lot of people have this problem but nobody has a good solution so far.
It would be nice if the solution had the following features:
- can interpret more than one command (e.g.
command; command) - can interpret stream redirects on the lines (e.g.
ls > /tmp/ls.txt) - only uses common Linux tools
Bonus points if it works on other Unix-clones without too exotic requirements.
Can you convert your command list to a Makefile? If so, you could just run “make -j X”.