Question

I am doing performance test against our site. The site exposes many Web services APIs. Our product has several workflows and each complete workflow is composed of calling more than one APIs.

I am wondering which of the following apporoaches is reasonable:

• Test against each single API in a standalone way.

• Test against each complete workflow which involves several APIs.

I think the latter one make more sense since it mimic the real scenario.
I’d like to hear your comments.

Thanks.

Answer 1

There are (as usual) pro’s and cons of each approach. Personally I’d consider doing both, starting with the single API test.

The single API is probably the easiest to build, and it has the benefit of exactly pinpointing out where your performance issues are. It’s also useful to so spot regressions in performance during development. If there are unit tests for the application consider using those instead. When there is a performance regression there usually also is a unit test which suddenly became slower.

Once you’ve done that you will still need to do the more complex tests. Firstly because you need to know if the performance of a certain flow is acceptable, but also because there can be unexpected interactions between different API’s. Depending on you application there may be nasty concurrency issues, throughput bottlenecks etc. Make sure you run several flows concurrently, that’s what happens in really live and it’s only way to find issues related to locking in the database, I/O bottlenecks etc.

But before you start make sure you have a realistic idea of what the performance should be, how many concurrent users there will be and what the hardware requirements are. There is no limit to improving performance, so you have to decide what is good enough or you will never stop optimizing.