I have a handful of apache processes using way too much cpu. I’d like to somehow log what’s going on with these process to diagnose and debug the bottleneck. How can I dive into what’s slowing these processes down?
Example from top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13768 apache 16 0 275m 11m 4264 S 37.4 0.1 1:22.87 httpd
15904 apache 15 0 274m 9620 4188 S 19.8 0.1 0:14.72 httpd
13321 apache 16 0 275m 11m 4264 S 19.5 0.1 1:57.66 httpd
14552 apache 15 0 275m 11m 4244 S 19.2 0.1 0:52.49 httpd
15794 apache 15 0 274m 9648 4200 S 18.8 0.1 0:19.94 httpd
15296 apache 16 0 274m 9704 4256 S 18.5 0.1 0:39.45 httpd
15782 apache 16 0 274m 9660 4208 R 18.2 0.1 0:17.56 httpd
15797 apache 15 0 274m 9632 4184 S 17.9 0.1 0:18.35 httpd
14951 apache 15 0 274m 9712 4260 S 17.2 0.1 0:41.50 httpd
15493 apache 16 0 282m 13m 4276 R 17.2 0.2 0:34.45 httpd
15630 apache 16 0 274m 9692 4244 S 16.9 0.1 0:24.02 httpd
14768 apache 16 0 274m 9728 4276 S 16.6 0.1 0:52.33 httpd
15494 apache 15 0 274m 9644 4196 S 16.6 0.1 0:36.35 httpd
15152 apache 15 0 275m 10m 4216 S 16.0 0.1 0:39.28 httpd
15785 apache 15 0 274m 9612 4176 S 15.7 0.1 0:20.02 httpd
15634 apache 16 0 292m 27m 4188 R 15.0 0.3 0:22.73 httpd
13346 apache 15 0 275m 11m 4272 S 14.7 0.1 2:00.32 httpd
13068 mysql 15 0 1433m 290m 3508 S 12.8 3.5 7:45.70 mysqld
15752 apache 16 0 287m 22m 4192 R 9.9 0.3 0:19.73 httpd
15757 apache 15 0 275m 10m 4268 S 9.3 0.1 0:18.43 httpd
13841 apache 16 0 283m 18m 4268 R 6.7 0.2 1:20.26 httpd
15742 apache 16 0 283m 18m 4264 R 6.7 0.2 0:21.13 httpd
15903 apache 15 0 283m 18m 4220 R 6.4 0.2 0:17.96 httpd
15987 apache 15 0 280m 13m 4212 S 2.6 0.2 0:12.98 httpd
15751 apache 16 0 280m 13m 4284 R 2.2 0.2 0:19.02 httpd
15986 apache 16 0 274m 9628 4180 S 2.2 0.1 0:17.88 httpd
15872 apache 16 0 280m 13m 4220 R 1.9 0.2 0:18.79 httpd
16255 apache 16 0 280m 13m 4200 R 1.9 0.2 0:11.95 httpd
14885 apache 15 0 275m 11m 4272 S 1.6 0.1 0:44.75 httpd
13107 apache 16 0 280m 13m 4280 R 1.3 0.2 1:53.03 httpd
14160 apache 16 0 280m 13m 4304 R 1.3 0.2 0:56.41 httpd
15407 apache 16 0 280m 13m 4276 R 1.3 0.2 0:33.79 httpd
15770 apache 16 0 280m 13m 4200 R 1.3 0.2 0:19.03 httpd
15985 apache 16 0 280m 13m 4284 R 1.3 0.2 0:19.56 httpd
14952 apache 15 0 280m 13m 4300 S 1.0 0.2 0:37.95 httpd
15205 apache 16 0 280m 13m 4216 R 1.0 0.2 0:34.74 httpd
15758 apache 16 0 280m 13m 4272 R 1.0 0.2 0:21.21 httpd
15784 apache 15 0 280m 13m 4280 S 1.0 0.2 0:13.76 httpd
13771 apache 16 0 280m 13m 4304 R 0.6 0.2 1:16.60 httpd
13857 apache 16 0 280m 13m 4284 R 0.6 0.2 1:17.89 httpd
15199 apache 16 0 280m 13m 4236 R 0.6 0.2 0:31.85 httpd
15405 apache 16 0 280m 13m 4244 R 0.6 0.2 0:32.25 httpd
15616 apache 16 0 280m 13m 4220 R 0.6 0.2 0:28.39 httpd
It seems you can log the time spent per request, see: Is there a way to have Apache log slow requests?
One process will handle many requests during its life, you have to do that kind of logging/profiling at the application level. Which technology do you use? PHP(mod_php)?
I’ve seen this tool recently, Amon it seems very promising.