Question

I run a data-intensive site with CMS built by outsourcing. Now as the no. of users grows it is getting slower. It goes down during major product launches. What services that I can turn to, to analyze the bottleneck of the site and analyze SQL queries, etc? Service which can provide solutions like separating load between redundant servers, configure master-slave for db, etc? I am new to this.

Answer 1

If you have a custom-built CMS, you’re going to need software engineers to analyze the problems and propose solutions; there’s no off-the-shelf solution for that. The engineers you’re looking for would need to understand the programming language your CMS is built in, and have experience in web scalability. Obviously, your outsource partner would be a good candidate here…

If the CMS is an off the shelf solution, the vendor might be able to recommend specialist service providers, independent of the outsource partner.

In general, the performance / scalability process is:

• understand the targets, ideally represented as page generations per
  second, subdivided into page types if necessary (e.g. “50 CMS product
  pages / second, 20 logins / second”). Establish the maximum acceptable response time (e.g. 1 second average, 4 seconds max).
• create a test environment which you understand completely, and which you can relate to the target production system. The test environment should be easy to work with, and accessible to the team; typically I recommend using a developer work station, or a low-powered VM. The purpose is to bring bottlenecks to light, not to handle huge amounts of traffic.
• establish test targets for your test environment – e.g. if “production” needs to handle 100 page generations / second, your test environment might only need to handle 20 page generations / second.
• deploy your application to the test environment, and set up a way of collecting performance information, e.g. CPU, memory, disk, network, etc.
• run load tests on the test environment; increase load until you exceed the response time targets. Use the monitoring tools to identify the bottleneck.
• fix bottleneck
• rinse & repeat until you hit your performance targets
• deploy application to production-like environment – similar capacity and architecture – if you’re lucky enough to have one. Set up monitoring and performance capture tools.
• run load test, increasing load until you exceed your performance targets
• if you’ve met your goals, congratulations!
• if you haven’t met your goals, it suggests your production environment has a different bottleneck than your test environment (often the database). Find out what the bottleneck is; try to replicate on your test environment.
• restart testing on test environment.

Quite often, you will find you have to make architectural or infrastructure changes to reach your targets; I’ve used the following:
– run the solution on bigger hardware.
– introduce a CDN to offload traffic – some CMS-driven sites can be cached almost entirely on a CDN
– introduce caching into the application, ideally at the page generation level, but a typically web app has many places where caching can help
– add more front-end web servers (assuming application was built with load balancing in mind)
– add more database servers (this is nearly always a major intervention unless the app was built with this in mind)

Load testing tools are available as services you can hire (Keynote is one I’ve used), or tools you can run yourself (JMeter is my favourite).