Question

I’m wondering why this code returns different times:

My outputs are:

Time1: 511 Time2: 228 for N: 100000000
Time1: 509 Time2: 229 for N: 100000001
Time1: 503 Time2: 229 for N: 100000002

I have:

java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.4) (6b24-1.11.4-1ubuntu0.12.04.1)
OpenJDK Server VM (build 20.0-b12, mixed mode)
Intel® Pentium(R) CPU B960 @ 2.20GHz × 2 
Linux Ubuntu 12.04

My code looks like:

public class test {

    public static void main(String[] arg)
    {
        for(long N=100000000;N<2000000000;++N){

            long time2 = System.currentTimeMillis();
            double d = 1.0; 
            double z = 1.0/3.0;
            for(long i = 0; i < N; i++)
            {
                d = d*z;
            }
            long result2 = System.currentTimeMillis() - time2;


            long time1 = System.currentTimeMillis();
            double x = 1.0;
            double f = 1.0/3.0;
            for(long i = 0; i < N; i++)
            {
                x = x*f;
            }
            long result1 = System.currentTimeMillis() - time1;



            System.out.println("Time1: " + result1 + " Time2: " + result2 + " bigger: " + (result1 > result2) + " for N: " + N);
        }

    }
}

Answer 1

Benchmarking code in Java is hard.

The JVM does lots of things in the background:

• it optimizes code on the fly,
• compile methods after a certain number of times they are called (so methods can suddenly get faster, but compilation consumes resources, so everything might get slow for some time),
• it can move variables outside loops if it sees fit,
• it can even simply remove some code if it sees that it does nothing (for instance: you are calculating d, but you don’t do anything with it, so why calculate it? the JVM is allowed to simply remove that code; read about “escape analysis”).
• There might also happen some garbage collection, etc.

Also, your operating system might interfere with the JVM’s execution, etc.

So, the bottom line is: without studying a lot on how the JVM and your operating system works, forget about doing precise benchmarks. You can’t expect to get meaningful results.

For more information, check this question and answers. It contains links to very good papers.