Question

I have a pipeline which at some point splits work into various sub-processes that do the same thing in parallel. Thus their output should go into the same file.

Is it too risky to say all of those processes should write into the same file? Or does python try and retry if it sees that this resource is occupied?

Answer 1

In general, this is not a good idea and will take a lot of care to get right. Since the writes will have to be serialized, it might also adversely affect scalability.

I’d recommend writing to separate files and merging (or just leaving them as separate files).