We are load testing our web application(Java, Struts, Spring, Java EE). The results show that, out of 11,000 odd requests sent to the server, only 1500 odd pass through; most of them fail throwing SocketException.
What are the possible reasons this could occur ?
I have no idea where to look in the application to fix it.
It’s likely that you are simply overwhelming your server. The operating system will only maintain a bounded queue of pending connections, and your application almost certainly has a bounded number of threads processing incoming connections.
I’d suggest you start recording metrics on how many idle processing threads you have, and (if you can) how many pending connections are in the operating system queue. If you consistently have zero idle threads and an OS queue at or near the maximum then you know that you’ve hit the limits of your hardware and/or software and/or database.
You should also watch the operating system’s CPU and I/O load metrics. That may help narrow down where your bottleneck is. If you haven’t maxed out the CPU but you have a lot of I/O wait, then you need to optimize I/O operations. If you consistently hit 100% CPU then you need to either make your code more efficient or throw more hardware at it. You’ll also need to keep an eye on the garbage collector, because if it starts spinning then that will degrade performance.
Performance tuning is hard. You really need to be very careful about what you measure, but CPU load, I/O wait, network activity and memory consumption are a good start.