Question

I did see this post but it does not answer my question: C/Python Socket Performance?

I have been tasked with creating an application that can create thousands of connections based on sockets. I can do this in Python but I want to have room for performance improvements. I know it’s possible in Python because of my past projects, but I’m curious how much of a performance improvement this would be if I was to do this project in C (not C++)?

Answer 1

It really depends on what you’re doing with the sockets.

The best generic answer is: Usually Python is good enough that it doesn’t matter, but sometimes it’s not.

The overhead in the time taken to create and connect sockets is minimal, and reading and writing isn’t much worse. But that doesn’t matter, that’s pretty much never any significant time spent doing that anyway.

There are reactors and proactors for Python every bit as good as the general-purpose ones available for C (and half of the C libraries have Python bindings). If you’re not doing much significant work beyond the sockets, this is often your main bottleneck. If you’ve got a very specific use pattern and/or very tightly specified hardware, you might be able to write a custom reactor or proactor that beats out anything general-purpose. In that case, you pretty much have to go with C, not Python.

But usually, you’ve got significant work to do beyond just manipulating sockets.

If that work is mostly independent and highly parallelizable, C obviously beats Python (because of the GIL), unless the jobs are heavy enough that you can multi-process them (and keep in mind that “heavy enough” can be pretty heavy on Windows platforms). Except, of course, that it’s incredibly easy to screw up performance (not to mention stability) writing multi-threaded C code; really, something like Erlang or Haskell is probably a better bet here than either C or Python. (If you’re about to say, “But we’ve got people who are experienced at C but they can’t learn Haskell”, then those people are probably not good enough programmers to write multi-threaded code.)

If that work is mostly memory copying between socket buffers, and you can deal with a tightly-specified system, you may be able to write C code that optimizes zero-copies, and there’s no way to do that in Python.

But if it’s mostly typical things like waiting on disk or serialized computation, then it scarcely matters how you write the socket-stuff, because it’s going to end up waiting on the real code anyway.

So, without any more information, I’d go with Python, because the time you save getting things up and running and debugged vs. C can be spent optimizing or otherwise improving whatever turns out to matter.