Question

Please explain (with examples) what is meant by A/B testing of a website.

Answer 1

In the context of a web site it means that some users of the site/webapp get slightly modified version(s) and you verify how each modification affects the usability/user’s behavior compared to baseline “control” version.

As always with A/B testing, which is a statistical technique, you need a large enough sample to properly assess impact of change.

This was done by both Amazon and Google among well known web companies.

A VERY good article (with marketing angle) on web A/B testing is : http://www.clickz.com/3349901

In the interest of completeness, here’s the full A/B testing definition from Wikipedia:

A/B testing or bucket testing is a
method of marketing testing by which a
baseline control sample is compared to
a variety of single-variable test
samples in order to improve response
rates. A classic direct mail tactic,
this method has been recently adopted
within the interactive space to test
tactics such as banner ads, emails and
landing pages.

Significant improvements can be seen
through testing elements like copy
text, layouts, images and colors.
However, not all elements produce the
same improvements, and by looking at
the results from different tests, it
is possible to identify those elements
that consistently tend to produce the
greatest improvements.

Employers of this A/B testing method
will distribute multiple samples of a
test, including the control, to see
which single variable is most
effective in increasing a response
rate or other desired outcome. The
test, in order to be effective, must
reach an audience of a sufficient size
that there is a reasonable chance of
detecting a meaningful difference
between the control and other tactics