Question

I am doing a simple coin flipping experiment for class that involves flipping a certain number of coins on a certain number of threads. To run our performance tests for speedup, we use a fixed number of coinflips (I’ve been using a billion) and change the number of threads. We use AWS extra High CPU instances with 8 cores to run these tests. For some reason, as soon as I use more than 6 threads, I get significant slowdown. Worse than that, it is inconsistent. Sometimes I will get 14 seconds, sometime 2 for the same number of threads and flips. It makes no sense. I have tried using different JVM’s (OpenJRE and Sun JVM) and trying a new instance. Below is my code and the benchmark results (in ms). I would love some help. Thanks.

EDIT: So it seems that I solved it, thanks in big part to the suggestions of yadab and Bruno Reis. They suggested using a local variable to keep track of the number of heads, which I think could have been a factor. They also suggested running all my tests from within the same JVM session, which almost definitely was a factor. Thank you for your help everyone.

Speedup:
Threads | Flips | Time
1       1000000000  16402 16399  16404
2       1000000000  8218  8216   8217
3       1000000000  5493  5483   5492
4       1000000000  4125  4127   4140
5       1000000000  3306  3304   3311
6       1000000000  2758  2766   2756
7       1000000000  8346  7874   10617
8       1000000000  14370 14414  17831
9       1000000000  14956  14764  15316
10      1000000000  13595 14491  14031
11      1000000000  12642 11188   10625
12      1000000000  10620 10629  10876
13      1000000000  8422  9950   9756
14      1000000000  9284  9546   10194
15      1000000000  8524  4134   8046
16      1000000000  6915  6361   7275

Code:

import java.util.Random;

public class CoinFlip implements Runnable {
    private final long iterations; //iterations is the number of times the program will run, numHeads is the number of heads counted
    private long numHeads;
    public CoinFlip(long iterations) {
        this.iterations = iterations;
    }

    @Override
    public void run() {
        Random rand = new Random();
        numHeads = 0;
        for (long i = 0; i < iterations; i++) {
            if (rand.nextBoolean()) { //True represents heads, false represents a tails
                numHeads++;
            }
        }
    }

    public long getHeads() { //numHeads getter
        return numHeads;
    }

    public static void main(String[] args) {
        final long numIterations , itersPerThread; //iterations: number of iterations, threads: number of threads to run on, itersPerThread: how many iterations each thread is responsible for
        final int threads;
        if (args.length != 2) {
            System.out.println("Usage: java CoinFlip #threads #iterations");
            return;
        }
        try {
            threads = Integer.parseInt(args[0]);
            numIterations = Long.parseLong(args[1]);
        } catch (NumberFormatException e) {
            System.out.println("Usage: java CoinFlip #threads #iterations");
            System.out.println("Invalid arguments");
            return;
        }
        itersPerThread = numIterations / ((long)threads); //Might cause rounding errors, but we were told to ignore that
        Thread[] threadList = new Thread[threads]; //List of running threads so we can join() them later
        CoinFlip[] flipList = new CoinFlip[threads]; //List of our runnables so that we can collect the number of heads later
        for (int i = 0; i < threads; i++) { //create each runnable
            flipList[i] = new CoinFlip(itersPerThread);
        }
        long time = System.currentTimeMillis(); //start time
        for (int i = 0; i < threads; i++) { //create and start each thread
            threadList[i] = new Thread(flipList[i]);
            threadList[i].start();
        }
        for (int i = 0; i < threads; i++) { //wait for all threads to finish
            try {
                threadList[i].join();
                System.out.println("Collected thread " + i);
            } catch (InterruptedException e) {
                System.out.println("Interrupted");
                return;
            }
        }
        time = System.currentTimeMillis() - time; //total running time
        long totHeads = 0; 
        for (CoinFlip t : flipList) { //Collect number of heads from each CoinFlip object
            totHeads += t.getHeads();
        }

        //Print results
        System.out.println(totHeads + " heads in " + (numIterations / threads)
                * threads + " coin tosses on " + threads + " threads");
        System.out.println("Elapsed time: " + time + "ms");
    }
}

Answer 1

Any Java tests that run in a short amount of time (less than 30 seconds) are just not suitable for performance testing. The hotspot compiler and other java runtime mechanisms are optimizing your code during the the first large number of seconds your application is running. Your timing deviations can easily be attributed to JVM startup, optimizations, and shutdown.

If you want a more realistic timing then you are going to have to run for 30 or so seconds and then start your timing run. Also, I would recommend that you multiply the test runs by at least an order of magnitude to better average out the affects of OS overhead, GC, background tasks, etc.. So warm up your application letting it run for 30 seconds, start your testing and the timer, let it run for at least a minute, stop the timer and record your results, and then shutdown the JVM.

Also, it makes more sense to graph how many coin flips you do in a certain amount of time then to see how long it takes to do a certain number of coin flips. The difference is that you want all of the tests to be running for the same amount of time if you can.