Multivariate testing enables you to test several components of a website to determine the optimal combination for increasing conversion. But unlike a traditional split test, which tests one creative versus another, multivariate testing enables you to test the combination of elements on a page and then will determine the optimal combination of those elements for increasing conversion.

Here’s a quick example:
Let’s say you are running a paid search campaign and you have set up a landing page specifically for your paid search visitors. The page currently converts at .75%. You’re not thrilled… In addition, your analytics package shows that you have a 60% bounce rate on the landing page. Again, you’re not thrilled. With multivariate testing, you can take that landing page and then determine key components of the page that you would like to test in order to increase conversion. For example, you might want to test the header graphic, the headline, and a call to action on the page. So, you have 3 components (or page sections) to test and then you can create multiple versions of each component. Let’s say you tested 4 versions of each component, which would mean that you have 81 possible combinations of elements. Your testing application would automatically change the content for different visitors and then measure the effectiveness of each element and the combination of elements. Cool, right? Note, the more components you test and more versions of components will extend the length of your test. There needs to be statistical significance in order to accurately determine the best combination of elements.

Enter Google Website Optimizer:
There are several tools on the market to help you with multivariate testing, including Offermatica and Optimost (probably the most popular services.) However, you might want to consider a great starter application before diving into the more complex applications listed above. That’s where Google Website Optimizer comes in. It’s Google’s free optimization tool that does a great job with basic multivariate testing. It’s a great way to start your in-house testing program. Heck, it’s free! I’ll take you through a quick example below.

A Real World Example, Step by Step:
One of my clients sells software (B2B) and they noticed a lot of people visiting the demo page, which also has customer testimonials as part of the page. After utilizing their web analytics package to analyze the past few months of activity, I could clearly see that visitors weren’t taking the next step and clicking the call to action to buy the software from the page. So, to increase conversion, we decided to use multivariate testing using Google Website Optimizer.

Here is what I did and how it worked:

1. I broke down the demo page into the core components we wanted to test. This ended up being the header graphic, the headline, and 2 calls to action (one at the top of the page and one below the fold). Note that mapping out your test is probably the toughest part since if you test the wrong components, then you probably won’t achieve your goals. :-) I recommend including everyone at this stage, including the designers, the marketing group, your VP, developers, etc. You don’t have to take everyone’s recommendation, but it typically gives you a good view of the possibilities.

2. Access Google Website Optimizer, which can be found in your AdWords account under campaign management. Create a new experiment and follow the steps for setting up your experiment. There are 5 steps and I briefly touch on each of them below.

3. Click the button for identifying your experiment pages. This will include the page you are testing and the conversion page.

4. Name your experiment and locate your test page and conversion page.

5. The second step (at least in Google Website Optimizer) is to plan your experiment. I briefly explained my client’s experiment above, but this is the step where you figure out which components to test in order to increase conversion. I determined that the header graphic, the headline and 2 calls to action would be the components to test and that we would test 3 different versions of each (giving us 81 possible combinations).

6. Add the JavaScript tags to both the test page and the conversion page. Google gives you detailed instructions for doing this, so I won’t add each step of accomplishing this task. Basically, you’ll be adding some JavaScript to each page and then identifying the unique sections of your page that you will be testing. Once you identify a section, such as the headline, then you will add specific JavaScript so Google Website Optimizer can swap out content on the fly for testing purposes. If you are familiar with HTML and JavaScript, this process is fairly simple. If you aren’t familiar with HTML and JavaScript, then have a developer help you.

7. Then you can click “Check tags on page” once they are uploaded and Google will automatically check them for you. A nice feature…Note, you can also check local pages, if needed by browsing your computer for the webpages.

8. Now is the fun part. You will create the different versions of each component. Don’t be shy here…make sure each version is significantly different or your results may not tell you anything! Remember, we are testing conversion, not how pretty something is. To me, I love this part of website testing. Why? Because it’s sometimes the ugliest component that pulls the best response. Nothing drives designers crazier that showing them an ugly element that converted twice the rate of the prettiest graphic. :-) And remember you are always starting with the original page and elements as a comparison.

9. Once you create and enter each variation for your test components, then you can preview the experiment. You can also tell Google Website Optimizer how much traffic to use for the test. For example, 50% of the traffic to the page, 75%, or 100% of the traffic. You can determine this, based on your exact experiment.

10. Preview your experiment. Having 4 components and 3 variations of each component yields 81 possible combinations. And, Google Website Optimizer enables you to view each combination in a slick preview screen. You can use dropdowns to select each combination or specific versions of each component. **Note, if you see something wrong, change it now. Close out the preview and click the back button to re-edit your versions. If you go live with the experiment, you will NOT be able to change the component versions.

11. When you are ready and everything looks perfect, simply click the Launch Now button to start your test. Now all you need is traffic. :-O By the way, you should start with a page that gets a lot of traffic. In order to scientifically determine which combination of elements works best, you need enough traffic to thoroughly test all of the combinations. The more components and variations you have, the more traffic and time you will need.

Go have a snack, grab lunch, hit the local bar, or wherever you want to boast about the multivariate test you just set up. :-) Check back a few hours later and you can start to review the reporting for your experiment. I plan to write a post that extensively covers the reporting in Google Website Optimizer, but in a nutshell, you can see the following statistics:

Under the Combinations Tab:
* which combinations are performing the best
* which combinations are performing the worst
* each combination’s chance of beating the original
* each combination’s chance of beating the other combinations
* observed improvement over the original
* conversions per visitor

Under the Page Sections Tab:
* the estimated conversion rate for each element within each component
* each variation’s chance of beating the original
* each variation’s chance of beating the other combinations
* observed improvement over the original
* conversions per visitor
* section relevance rating, which basically tells you how important variations in that section are to the overall experiment.


The results of my experiment:
And why I love multivariate testing…we never would have picked the combination that pulled the best response.

1. A header graphic that was relatively cluttered from a design standpoint, but one that builds credibility, performed the best. Needless to say, the designers never would have chosen this header. :-)

2. All 3 of the headlines performed well. The original actually performed better than the rest, but not much better…

3. A simple call to action (as compared to the other variations) performed the best at the top of the page.

4. For the lengthier call to action below the fold, a conversational call to action performed the best. This played on the fact that if you actually got down to the second call (below the fold), then you probably went through most of the content on the page.

Taking Action:
Based on the multivariate test, the optimal combination (out of 81 possible combinations) showed a 60% observed improvement over the original. So we reviewed the results and recently implemented the winning combination.

In Closing…
I hope this post helped you understand what multivariate testing is, how you can use it, and leaves you wanting to set up your first experiment. What I like best about scientific marketing is that you take opinion out of the equation (or mostly out of the equation). If your VP loves one piece of creative and you believe it’s not the right one…test it! It’s hard to argue with real data…and the reporting can act as nice buffer, which will help you keep your job! ;-)