When designing a feed forward neural network with multiple outputs, is there a conceptual difference (other than computational efficency) between having a single network with multiple outputs, and having multiple networks, each having a single output?
Although output neurons in the same network don’t affect each other “on the fly”, they do affect training, since the error from each output backpropagates and affects the weights of the hidden layer, which in turn affect the value of the other outputs.
Are there problems that are better solved either way? Intuitively, I’d say that a single network better fits problems where only one output should be active at a time (i.e OCR), where multiple networks better fit problems where multiple outputs can be active simultanously (i.e. when each output correspond to some characteristic in the input, where several of them can be present simultaniously). But this is mere intuition. Does it practically hold?
The conceptual difference, as you noted yourself, is joint training vs. separate training. I think people find that in most cases joint training helps if the problems are related (e.g. if all outputs are about recognizing characters). So I’d think that that in both problems you mention joint training will help, regardless of whether multiple outputs can be active simultaneously.
A case where joint training wouldn’t help and will probably hurt is when the tasks are not related at all, e.g. if one output is about ‘character a’ and another one is about ‘sound x’.