Question

We currently have a system with 20 SOA services and having a single master mysql database and 2 slave nodes. We currently have 10 GB of data in the database. We have a requirement in which the data in a table is going to be significantly increased. We want to stress test the system before proceeding with the implementation. What kind of stress testing makes sense for this kind of distributed environment?

Also, when performing the stress testing, I can look out for latency and metrics like what is the latency for servicing 90% of the requests for the services. Are there any other good metrics for services? What metrics should I look for the mysql database?

Thank you

Answer 1

Here are a few ideas:

1. Set up a test database and load the additional data into the table where you’re expecting the increase; use a multiple of the increase you’re expecting, e.g if you’re expecting the table to increase by 2000 rows then add 4000 rows to your test table.
2. Enable slow query logging in MySQL.
3. Make sure the logging levels in your SOA servers are detailed enough to debug errors from your stress test.
4. Use a load test tool such as JMeter to run multiple requests against each service in rapid succession. Use a multiple of the number of requests per second you’re expecting; I usually ramp up with 2x, 4x, 8x etc. times the number of expected requests.
5. Repeat the above test against each indivudual service in turn.
6. Repeat the above test with a “typical” mix of services – e.g. if you expect twice as many requests to service 1 than to service 2 then reflect that in the test.
7. If you want to test reliability as well, try repeating the JMeter tests with one or both of the MySQL slave nodes taken offline.

JMeter should give you all the latency info you need. Another useful “real world” data point form JMeter that I like to use is the 90% query time, which is the response time value that’s greater than or equal to 90% of the test responses.