Question

I’m relatively new to Java and I was curious how Java performed on a mundane task vs C++. So I compared 2 C++ string formatting methods (sprintf and ostringstream) against using Java’s StringBuilder. For each executable, the first argument I pass is the number of iterations to measure, so all the code looks roughly like

 int numIterations = args[0]; // or argv[1] in C++

 // measure begin here, ie: 
 int begin = System.nanoTime();
 // loop to measure
 for (int i = 0; i < numIterations; ++i)
 {
      // formatting code
 }
 // measure end here, ie
 int end = System.nanoTime();

This lines up with how I’ve seen other people profile Java code. I noticed that for small number of iterations (say 1000), Java does very poorly. For larger number of iterations, (say 1,000,000) Java does relatively well:

With 1000 iterations

$ ./spf.out 1000
C++  took: 1412618 ns
$ ./oss.out 1000
C++  took: 1816222 ns
$ java StringBuilderTest 1000
Java took: 25787951 ns

With 1000000 iterations

$ ./spf.out 1000000
C++  took: 1658699148 ns
$ ./oss.out 1000000
C++  took: 2053606449 ns
$java StringBuilderTest 1000000
Java took:  595965442 ns

I’m suspicious of the method of using begin/end timers to profile java code as I’m not sure exactly how the JITer works. Is this a vaild method of measuring Java code? Does Java have more “warming up” to do when running the code with a JIT’er? What is the canonical way of profiling Java code. Is there a way to warm up the JITer before measuring performance? Or is it just understood that for small N JITing is going to be part of the performance numbers measured?

Answer 1

Is this a vaild method of measuring Java code?

For the code shown yes. Java is much smarter than C++ at eliminating code which doesn’t do anything. 9 out of 10 benchmarks where Java is significantly faster, this is the case.

Does Java have more “warming up” to do when running the code with a JIT’er?

Yes, the warm up goes through stages. Make sure you run the test for 2-10 seconds and it should have been warm enough. You might ignore the first 10K – 20K runs.

What is the canonical way of profiling Java code. Is there a way to warm up the JITer before measuring performance?

There is many ways. This is one of the simplest.

 long begin = 0;
 // loop to measure
 for (int i = -11000; i < numIterations; ++i)
    if (i == 0) begin = System.nanoTime();

Or is it just understood that for small N JITing is going to be part of the performance numbers measured?

It depends on how realistic you want to make your test. If in production the code will not be called 10,000 times and it doesn’t warm up completely, that is what you should measure.