Rafael Alvarez

Share this post

Testing Silver Bullets

www.jralvarezc.com

Discover more from Rafael Alvarez

Blog, Newsletter, Journal
Continue reading
Sign in

Testing Silver Bullets

A/B testing

Rafael Alvarez
Sep 14, 2015
Share this post

Testing Silver Bullets

www.jralvarezc.com
Share

When creating a website for the first time, it’s common to turn to well-known experts in the field to get their magical solutions or “silver bullets” on how to improve it.  

Thanks to the possibilities of A/B testing, I decided to experiment by performing multiple tests on some of their advice. And to my surprise, the results were different from predicted.  I guess the experts weren't wrong, just that testing shows that in the business world, silver bullets don't exist.

bala de oro y plata sobre superficie blanca

The experiment I'm going to tell you about here was very simple.  Should the action on a landing page be at the top or the bottom?  The conventional wisdom of my close advisers was, unanimously, that it should be at the top.

To test this silver bullet, we designed an experiment where we ran the variations simultaneously for 40 weeks after first restarting the statistics.  The action with which we measured the conversion was the page display vs click on the "I WANT TO BE HEALTHY" button. (Here shown in green, “QUIERO SER SALUDABLE”). What does the experiment tell us about this silver bullet?:

abtests

In the table above we see three variants: A, AA and B. Variants A and AA have the button at the top of the page:

header

Variant B has the button at the bottom of the page:

footer

As we can see in the first image, the conversion rate of the A and AA variants is between 6.10% and 7.63% (average 6.86%). However, the B variant has a conversion rate of 13.70%, a 79.55% improvement over the best previous conversion.

The difference between variants A and AA followed another, more subtle suggestion, where I was told I should change the order of the sections:

orden

Again, the change proved to be insubstantial and contrary to the recommendation. The difference in conversion rate between Variant AA and Variant A was marginal (1.53%), that is, practically statistical equality. Andin any case the original Variant A, where “investment” appeared before “experience”, proved to be better.

The simple conclusion is that if someone gives you a silver bullet solution, you have to test it out.  In the long run the circumstances surrounding a business context are usually different and the important thing is to make identifiable changes one at a time, measure, compare, go back and repeat.

Share this post

Testing Silver Bullets

www.jralvarezc.com
Share
Previous
Next
Comments
Top
New

No posts

Ready for more?

© 2023 Rafael Alvarez
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing