Question

Does pypy handle threads and sockets quickly compared to hand written C? Compared to normal python?

I would just try it, but the python code in question was written for a small cluster of computers on which I am not an admin. I’m asking here because my attempts to google only provided comparisons to cython, unladen swallow, ect., and I don’t want to bug the admin about it if this is unlikely to work.

I don’t actually need pypy to be as good at C; I’m looking to use it because right now the interpreter’s overhead is completely overshadowing the computation I’m trying to time. I just need pypy to get me in the neighborhood of handwritten C.

Answer 1

Does pypy handle threads and sockets quickly compared to hand written C?

No. It’s usually the same or worse.

PyPy retains the Global Interpreter Lock (GIL) that CPython has. This means native threads cannot run Python code in parallel. Python threads also have additional semantics that come at a cost. Much synchronization surrounds Python thread start-up, shutdown, and tracking of the thread objects. By comparison, C threads will start up faster, are cheaper to use, and can run completely in parallel.

Efficient socket handling requires minimizing time not spent waiting for the next socket event. Since PyPy’s threading model is still bound by the GIL, this means threads that return from blocking socket calls can’t proceed until they acquire the GIL. Equivalent C code generally executes faster, and can return to waiting on socket events sooner.

Compared to normal python?

Yes. But not much.

For the reasons above, PyPy will, except for occasional spikes due to JIT and and other overheads, require less CPU time for the equivalent code. Therefore both thread and socket handling is faster and more responsive.

I would just try it, but the python code in question was written for a small cluster of computers on which I am not an admin. I’m asking here because my attempts to google only provided comparisons to cython, unladen swallow, ect., and I don’t want to bug the admin about it if this is unlikely to work.

PyPy will only make noticeable performance improvements if your code is CPU bound. PyPy is the fastest Python implementation I know of that runs natively. You could investigate some of the other implementations, or consider writing C extensions if true threading parallelism is a high priority for you.

I don’t actually need pypy to be as good at C; I’m looking to use it because right now the interpreter’s overhead is completely overshadowing the computation I’m trying to time. I just need pypy to get me in the neighborhood of handwritten C.

Closing the performance gap with C is currently PyPy’s single greatest feature. I highly recommend you try it out.