First, sorry for my bad english.
I am using the mpstat tool in Ubuntu server to print the value of cpu percent idle every second. On the client side, I tried to send a UDP request/second, that the number of request increases every 60s. The server is using 8 cores, and the result of mpstat is a decrease of idle state (from 100 % -> stabilize in 80% ). I believe to count the total cpu usage is by (100% minus idle state). The software that runs in the server runs as ROOT (so I can’t use us% parameter in mpstat)
I was thinking that this might be the case as however I increase the load, the cpu usage will peak in around 80% idle state. Is this the correct assumption ? I tried to see some reference about the OS behaviour regarding the performance, but I can’t find any good explanation why after jumping from 100 % to 80%, it stabilize whatever the number of load I send. I assume this is because the locking mechanism that the software on the server side uses, thus no more increment can be achieved.
Would appreciate of any pointers..
My first guess would be that your application is IO-bound and not CPU-bound, as is typical with networking software. This would mean that the throughput of the network card is the bottleneck, not processor performance. You can only push a certain number of packets through the network and if your operations on each packet are not very complicated, it is quite probable that you will saturate your NIC before you saturate your CPU.
Have you checked that the number of packets actually sent indeed still increases at the time your CPU profile flattens? If the network card is the cause, you should see at some point that the sending side cannot send any more packets in a time unit, because the receiver can’t receive them that fast. In the short run this may be smoothed out by buffering, but not in the long run. One possibility is that since you are using UDP and not TCP, perhaps the packets are silently dropped if the network can’t handle them. So from the sender’s point of view it looks like they’re sending lots and lots of packets but in reality only few arrive. Maybe you should put a counter in your app and display the number of packets correctly received per unit of time once every 10000 packets or so. Then, you would know if you are loosing packets due to reaching the network’s throughput (which would then explain a limit on CPU usage). If you are actually correctly receving these packets, then the cause is probably something else – but I’d check this first.