Question

I’ve had an A/B Test running in Google Web Optimizer for six weeks now, and there’s still no end in sight. Google is still saying: “We have not gathered enough data yet to show any significant results. When we collect more data we should be able to show you a winning combination.”

Is there any way of telling how close Google is to making up its mind? (Does anyone know what algorithm does it use to decide if there’s been any “high confidence winners”?)

According to the Google help documentation:

Sometimes we simply need more data to
be able to reach a level of high
confidence. A tested combination
typically needs around 200 conversions
for us to judge its performance with
certainty.

But all of our conversions have over 200 conversations at the moment:

230 / 4061 (Original)
223 / 3937 (Variation 1)
205 / 3984 (Variation 2)
205 / 4007 (Variation 3)

How much longer is it going to have to run??

Thanks for any help.

Answer 1

Is there any way of telling how close Google is to making up its mind?

You can use the GWO calculator to help determine how long a test will take based on a number of assumptions that you provide. Keep in mind though that it is possible that there is not significant difference between your test combination, in which case a test to see which is best would take an infinite amount of time, because it is not possible to find a winner.

(Does anyone know what algorithm does it use to decide if there’s been any “high confidence winners”?)

That is a mystery, but with most, if not all, statistical tests, there is what’s called a p-value which is the probability of obtaining a result as extreme as the one observed by chance alone. GWO tests run until the p-value passes some threshold, probably 5%. To be more clear, GWO tests run until a combination is significantly better than the original combination, such that the result only has a 5% chance of occurring by chance alone.

For your test there appears to be no significant winner, it’s a tie.