We are having performance issues on our web app during peak times, which is currently split between one 2003 IIS6 Web Server, and one SQL Server 2005 DB Server. The query times on the DB server look fine (around 30ms), and the CPU is low (under 20%), yet queries can take a long time to execute on the web server (over 1 second).
I was looking in perfmon for clues, and saw that the Average Network IO wait time is ~400ms, which could well explain the performance difference. I also ran some queries inside SSMS and see that every second there are roughly 10,000ms of cumulative ASYNC_NETWORK_IO Wait Types.
From some research, it seems that the problem is either insufficient Network capacity, or that the web server isn’t processing the results fast enough. How do I go about investigating which it is, and then resolving?
We use NHibernate for our data layer, so there are no DataReaders used as far as I know. There are no known queries that return large results sets, although there is one table that has a column containing compressed XML documents.
Thanks in advance
Additional Requested Information
- Sql Server is set up to use TCP/IP
and Shared Memory protocols - We make around 4 DB requests per page, nothing ridiculous
- At peak time DB Server sends 1.5MB/sec
- At peak, Web server CPU is around 60%
- AT peak, Web server NIC load is 13MB/sec. 8MB is sending, 5MB is receiving. The same NIC handles SQL and HTTP traffic
- We do use some caching but the application is pretty dynamic so requires up to date data most of the time
We resolved the problem by separating the SQL traffic on to a different NIC. In addition, this NIC connects to a local IP Address, whereas before it was hitting the public IP of the database server.
Thanks to @Remus Rusanu for the tip.