Question

I’m using java 6 random (java.util.Random,linux 64) to randomly decide between serving one version of a page to a second one (Normal A/B testing), technically i initialize the class once with the default empty constructor and it’s injected to a bean (Spring) as a property .
Most of the times the copies of the pages are within 8%(+-) of each other but from time to time i see deviations of up to 20 percent , e.g :

I now have two copies that split : 680 / 570 is that considered normal ?
Is there a better/faster version to use than java random ?

Thanks

Answer 1

A deviation of 20% does seem rather large, but you would need to talk to a trained statistician to find out if it is statistically anomalous.

UPDATE – and the answer is that it is not necessarily anomalous. The statistics predict that you would get an outlier like this roughly 0.3% of the time.

---

It is certainly plausible for a result like this to be caused by the random number generator. The Random class uses a simple “linear congruential” algorithm, and this class of algorithms are strongly auto-correlated. Depending on how you use the random number, this could lead anomalies at the application level.

If this is the cause of your problem, then you could try replacing it with a crypto-strength random number generator. See the javadocs for SecureRandom. SecureRandom is more expensive than Random, but it is unlikely that this will make any difference in your use-case.

---

On the other hand, if these outliers are actually happening at roughly the rate predicted by the theory, changing the random number generator shouldn’t make any difference.

If these outliers are really troublesome, then you need to take a different approach. Instead of generating N random choices, generate a list of false / true with exactly the required ratio, and then shuffle the list; e.g. using Collections.shuffle.