Question

Here is the scenario

We are load testing a web application. The application is deployed on two VM servers with a a hardware load balancer distributing the load.

There are tow tools used here
1. HP Load Runner (an expensive tool).
2. JMeter – free

JMeter was used by development team to test for a huge number of users. It also does not have any licensing limit like Load Runner.

How the tests are run ?
A URL is invoked with some parameters and web application reads the parameter , process results and generates a pdf file.

When running the test we found that for a load of 1000 users spread over period of 60 seconds, our application took 4 minutes to generate 1000 files.
Now when we pass the same url through JMeter, 1000 users with a ramp up time of 60 seconds,
application takes 1 minutes and 15 seconds to generate 1000 files.

I am baffled here as to why this huge difference in performance.

Load runner has rstat daemon installed on both servers.

Any clues ?

Answer 1

You really have four possibilities here:

1. You are measuring two different things. Check your timing record structure.
2. Your request and response information is different between the two tools. Check with Fiddler or Wireshark.
3. Your test environment initial conditions are different yielding different results. Test 101 stuff, but quite often overlooked in tracking down issues like this.
4. You have an overloaded load generator in your loadrunner environment which is causing all virtual users to slow. For example you may be logging everything resulting in your file system becoming a bottleneck for the test. Deliberately underload your generators, reduce your logging levels and watch how you are using memory for correlations so you don’t create a physical memory oversubscribed condition which results in high swap activity.

As to the comment above as to JMETER being faster, I have benchmarked both and for very complex code the C based solution for Loadrunner is faster upon execution from iteration to iteration than the Java based solution in JMETER. (method: complex algorithm for creating data files on the fly for upload for batch mortgage processing. p3: 800Mhz. 2GB of RAM. LoadRunner 1.8 million iterations per hour ungoverned for a single user. JMETER, 1.2 million) Once you add in pacing it is the response time of the server which is determinate to both.

It should be noted that LoadRunner tracks its internal API time to directly address accusations of the tool influencing the test results. If you open the results set database set (.mdb or Microsoft SQL server instance as appropriate) and take a look at the [event meter] table you will find a reference for “Wasted Time.” The definition for wasted time can be found in the LoadRunner documentation.