Case Study: A/B Testing on Sim Only

Every business has a chance to sweep their customers off of their feet…they just need the right broom.

Many online companies have done one too many strategies to sell over the Internet. Affiliate marketing, Search Engine Optimization, Ad placements, and social media usage are all tossed in the basket. But what most of them fail to realize is that eventually, these ploys get old. Such tactics may only be effective during the early phases, but what happens after the consumers have gotten used to seeing the website?

This is the reason why many businesses falter after a few months. Webmasters may exert some effort to update the contents regularly, but these are just not enough to keep the buyers (and profits) coming.

Remember that customer preferences are dynamic. As such, each move should be latched on what the majority clamours. It may sound difficult given the vastness of the Internet market, but this is why methods such as A/B testing exist.

Having people who doubt the efficiency of split testing is inevitable. But companies who have tried running the procedure emerged victorious. The best case in point is that of Sim Only’s A/B Testing.

About Sim Only

Sim Only is the company behind SIM ONLY. They run an e-commerce site to sell mobile phones from colossal companies like Blackberry and Apple Macintosh. Apart from their homepage, therefore, the company also sets its eye on the traffic of its external partnering domains.

Before the split test was conducted, Sim Only’s homepage was garnering an average of 18,000 hits per month. The click out rates, on the other hand, was at 54,000 monthly.

The goal of the company is then to see which type of design can motivate the consumers into checking the external websites. It then partnered with Maxymiser – an infamous company known for multivariate testing – to carry out the process of testing the homepage through a split test that transpired for 6 weeks.

The Plan: How A/B Testing is Conducted

As mentioned, Sim Only aimed to observe how the differences in their main page’s design can lure visitors into clicking the icons for the links.

Vertical vs. Horizontal Layout

The first couple of versions were set to have different presentations of products. One of them bore a vertical layout of the products on the right hand side of the page. The icons were arranged according to popularity, with the best seller situated at the top.

The second version was tailored to have a horizontal design, with the icons running across the page. The products were still arranged according to popularity from left to right.

Both designs came with a check box that allows users to select more than one item at the same time.

It is a must to remember that the webpage can be tested one component at a time. Sim Only managed to conduct more test for the icons, but the differences between the vertical and horizontal produced more significant results.

The Findings

This table shows how the 1,596 number of generations increased to 1,722 with the vertical layout. As a result, the conversion rates (or the ratio of visits to purchases made) rose from 13.35% to 15.97%. More than 50 sales was made with the vertical presentation of products (from 213 of the default to 275 of the vertical layout).

The first variant also touts the lowest conversion rate error, highest uplift, and most ideal confidence error. This can be accounted to the 20% increase in the click out rates of Sim Only’s homepage.

What does this mean, exactly?

Adopting a vertical presentation of the products appears to have greater impact on the audiences compared to the horizontal layout. It potentially motivates more customers to buy a phone of SIM Card, thereby contributing to the volume of sales made by the company online. It’s understandable that it’s not a concept not everyone will grasp, but Maxymiser have a nice A/B testing guide.

Some might think that the results were acquired from pure chance, but the six-week span of the experiment makes it fool-proof and concretely conclusive. Sim Only explained that they cannot go beyond this time allotted because they can lost a huge pool of potential clients when from flashing the horizontal design that does not work as effectively.


Ruben Corbo is a freelance writer and writes for a number of online marketing websites including those that help online businesses improve A/B testing techniques to increase sales conversions. When Ruben is not writing, he’s producing or composing music for short films or other visual arts.


  1. Such multivariate split testing always pays off in the long run. Thanks for a great post. Dan