Question

I need to start many short-lived processes to be started by some kind of spawner process, those processes by itselves do some work, save result (or delegate saving to some saver process) and then exit.

And there also should be some global limit of the number of parallel running processes. And i suppose some kind of queue, because when limit is reached, then spawner could continue to spawn new processes while they i guess should suspend and wait for the room to run.

Could that scheme be implemented easily in erlang ? Does it make sense to have pool of workers to gain some speed or just spawn new process every time is okay ? Could supervisor simple_one_for_one help here ?

Answer 1

You can spawn process easily, as already pointed out, and your spawning process could use monitor/2 and a counter to limit and maintain your worker process pool. Your spawning process would spawn an initial set of processes, Monitoring the resulting Pids (i.e., monitor(process, Pid)) and decrementing a counter as appropriate. It would then go into a receive waiting on a {‘DOWN’, MonitorRef, Type, Object, Info} message indicating that a worker had terminated. As in…

loop(MaxProcesses, LiveProcesses) ->
    receive
       {'DOWN', _Ref, process, _Pid, _Info} ->
           case LiveProcesses of
               N when N =< MaxProcesses -> 
                   spawn_another() %and monitor new pid
                   loop(MaxProcess, LiveProcesses);
               N when N > MaxProcess -> 
                   loop(MaxProcesses, LiveProcesses -1)
           end
        end
    after  %optional if you want the spawner to potentially do something else
        Timeout->
           do more work
           loop(MaxProcesses, LiveProcesses)
        end
    end.