Multivariate Test experiment

Tomáš Veselý - podpořen AI
2 days ago
4 min read

We're building a comprehensive knowledge library about product development as part of our mission. The library is for anyone looking to make better decisions — primarily decisions about product development. Whether you're an inventor, a product manager, or a Chief Product Officer, using the right research methods and experiments increases your chances of building the right things for the right audience. Today we'll introduce the multivariate test validation method.

When to Use This Experiment?

Multivariate testing is the right choice when you need to find out which combination of several simultaneously changed elements on a single page produces the best result (often a conversion). It makes sense under these conditions:

The page has high traffic, enough to gather sufficient data for every tested combination.
Multiple elements on one page change at once (for example the headline, the main visual, and the button), and you suspect they influence one another.
The goal is a single, clearly measurable conversion (sign-up, purchase, click).
The product or site already has a real audience to run the test on.

Unlike A/B testing, which changes a single variable to isolate its individual effect (control version A vs. variant version B), multivariate testing changes several elements at once and compares all of their combinations. This also reveals the interactions between elements — but at the cost of far higher traffic requirements. That's why A/B testing is the practical choice for most situations; multivariate testing only pays off on high-traffic pages where the interplay of elements matters. Large internet companies with millions of visits a day routinely run multivariate tests on most of their features, every minute of operation.

Basic Experiment Principles

The essence of multivariate testing is comparing all possible combinations of several changed elements at once, in order to reveal not only the effectiveness of individual changes but also how they interact.

Define the conversion goal and the page. Pick one page and one conversion action the test should improve.
Choose the elements and their variants. Identify the elements that will change (headline, main visual, button text) and prepare several variants of each.
Build the combinations. Produce the test variants — for example several versions of the homepage with its changed elements.
Split the visitors. Each incoming visitor is randomly shown one of the combinations, and whether they convert is recorded.
Collect the data. Track unique views and the conversion rate for each combination. The decisive signal is the combination that beats the baseline version with statistical significance; alongside the winner, the test also reveals which elements drive the result upward and which work best together. Let the test run until every combination has gathered enough data for significance.
Identify the risks. Multivariate testing requires far higher traffic than an A/B test, because the number of combinations grows exponentially and each one needs its own sample. Interaction effects tend to be subtler than main effects, so proving them takes even more data. On low-traffic pages the test runs unbearably long or never reaches significance. The results are also harder to interpret, and the method won't tell you why a given combination works — only that it does.

Real-World Experiment Example

Link to research: Obama's $60 million dollar experiment — Dan Siroker, Optimizely

In December 2007, Barack Obama's presidential campaign team ran its first experiment on the website's splash page. The goal was to increase the share of visitors who signed up for the email list. Director of Analytics Dan Siroker tested two parts of the page at the same time: the media block at the top and the call-to-action button text.

It was a full-factorial multivariate test in Google Website Optimizer — four button variants and six media variants (three images and three videos), so 4 × 6 = 24 combinations, all tested against each other at once. During the test, over 310,000 people visited the page, so each combination was seen by roughly 13,000 visitors. Going in, the team had favored one of the videos.

The winning combination — an image of the family and a "Learn More" button — reached a sign-up rate of 11.6% versus the original 8.26%, a 40.6% improvement. The favored video variant, meanwhile, performed worse than every image. This is exactly where the multivariate test showed its strength: it revealed not just the best individual elements, but their most effective combination.

What Can Be Tested With This Experiment?

The main strength of multivariate testing is uncovering how elements on a page work together — not just which one wins on its own. Specifically, it's useful for:

Interaction between elements: whether, for example, a particular combination of headline, visual, and button works better than another.
Relative weight of individual elements: which of the changed elements raises the conversion rate the most.
Landing page design: which specific landing page layout maximizes a single action, such as a sign-up or purchase.
Disproving internal design assumptions: whether an element the team subjectively favors actually leads to higher conversion; the signal is when the data contradicts the assumption.
Conversion sensitivity to small tweaks: how strongly the result shifts with button text, color, or visual.