just considering starting to learning python but I have one concern before I invest more time. Let me phrase this as a statement followed by a concern for others to comment on as perhaps the assumptions in the statement are invalid:
I have read about GIL and the consensus seems to be if you require concurrent solutions in python your best bet is to fork a new process to avoid GIL.
My concern is that if I have a problem I’d like to split into N*2 pieces across N processors (assume for example I have a single server running a *nix o/s with say 8 cores) I will incur context switching penalties between processes rather than between threads, which is more costly, which will limit performance.
I ask this because other languages are out there that claim to excel in such a scenario and I wonder is python appropriate for this arena.
Python is not very good for CPU-bound concurrent programming. The GIL will (in many cases) make your program run as if it was running on a single core – or even worse. Even Unladen Swallow will (probably) not solve that problem (quote from their project plan: “we are no longer as optimistic about our chances of removing the GIL completely”).
As you already stated, other languages claim to be better in concurrent programming. Haskell, for example, has built-in functionality for programming concurrent applications. You could also try C++ with OpenMP, which I think makes parallelization very simple.
If your application is I/O-bound, Python may be a serious solution as the GIL is normally released while doing blocking calls.