I have written an HTTP proxy that does some stuff that’s not relevant here, but it is increasing the client’s time-to-serve by a huge amount (600us without proxy vs 60000us with it). I think I have found where the bulk of that time is coming from – between my proxy finishing sending back to the client and the client finishing receiving it. For now, server, proxy and client are running on the same host, using localhost as the addresses.
Once the proxy has finished sending (once it has returned from send() at least), I print the result of gettimeofday which gives an absolute time. When my client has received, it prints the result of gettimeofday. Since they’re both on the same host, this should be accurate. All send() calls are with no flags, so they are blocking. The difference between the two is about 40000us.
The proxy’s socket on which it listens for client connections is set up with the hints AF_UNSPEC, SOCK_STREAM and AI_PASSIVE. Presumably a socket from accept()ing on that will have the same parameters?
If I’m understanding all this correctly, Apache manages to do everything in 600us (including the equivalent of whatever is causing this 40000us delay). Can anybody suggest what might be causing this? I have tried setting the TCP_NODELAY option (I know I shouldn’t, it’s just to see if it made a difference) and the delay between finishing sending and finishing receiving went right down, I forget the number but <1000us.
This is all on Ubuntu Linux 2.6.31-19. Thanks for any help
You can’t really do meaningful performance measurements on a proxy with the client, proxy and origin server on the same host.
Place them all on different hosts on a network. Use real hardware machines for them all, or specialised hardware test systems (e.g. Spirent).
Your methodology makes no sense. Nobody has 600us of latency to their origin server in practice anyway. Running all the tasks on the same host creates contention and a wholly unreaslistic network environment.