What is the best way to log http requests for a web application, including ajax requests, so that I can later go back and query “I want to know how many times this request was made, and how long it took to complete on average”, or “show me the top 5 highest average time requests”
Would you use a separate database from the current production db to log these things to prevent all of those inserts causing IO slowdown, or does this end up not really making a big impact?
Would you bulk up requests and then push to the DB or would you do a single insert for each request?
Is there a better way to add this request logging in with timings besides wrapping each request handler in the application logic like:
start = CurrentTime()
/* request handler code */
end = CurrentTime()
Insert(requestName, start, (end - start))
You should be able to use your web server logs for this purpose. Apache and IIS logs both capture URL, query string, response code and duration. If the AJAX requests receive data via HTTP POST you will need to change the web server’s configuration to capture that data if it’s important to you. Then the best tool I’ve found for log analysis is Microsoft’s Log Parser, which does a great job working with large files with a SQL syntax to calculate answers to the kinds of questions you’re asking.
However, if you’re intent on rolling your own, use some sort of local logging. And use a logging framework – such as log4J – which is smart enough to buffer out disk writes to minimize I/O, roll the log files, and delete old files. This is more scalable approach for clustered servers, otherwise each is hitting the database constantly. If you want to put the data in a table, make it a batch process, once an hour or day for instance.