When you and your team are deciding which imagery, copy or color variation to go with, opinions and biases can often come into play and prevent an objective decision being made. In situations like this, data is often the best way to settle discussions about design.
You wouldn’t be arguing whether light blue or dark blue would be better for your main call to action if you knew that 10% more users noticed the darker blue. Or if a certain copy variation increased conversion rates and reduced the cost of customer acquisition.
A/B Testing has recently exploded in popularity because it allows your team to specifically gather this type of data: It allows you to test two variants of the same page – or same specific element – and see which one performs better. (In the case of multi-variant testing, it allows you to test between two or more variants to see which performs better).
It’s not just great to be able to settle the type of discussions mentioned above, but it can also optimize your websites and make sure they are delivering the best results possible.
When you decide to start running A/B tests, it’s especially important that you decide on the right tool for your use. There are dozens (maybe even hundreds) of different tools out there and it can be a challenge to not just filter out the bad ones, but also to understand which ones better suit your needs and those of your organization.
Below, we have listed our favorite five tools to help you make that choice.
Top A/B Testing tools
Here’s the list:
Optimizely was the first A/B testing tool I worked with. It’s a phenomenal starting point for most testers while also having features to support advanced users. It allows you to use a visual editor (no need to dive into coding) to set up experiments on different elements and then track specific performance indicators – clicks, conversions, revenue, engagement and more.
It’s fairly easy to learn how to use and get going with it. The starter package contains a good number of essential features and even the advanced packages are priced according to the amount of page views you get. In addition, according to a study conducted by ConversionXL on how much overall site speed is affected, it ranks decently among other testing tools – that is, it doesn’t affect your site performance very much.
All in all, Optimizely is the complete package: if you want to get started with A/B testing, it’s definitely worth trying out, considering the pricing and trial plans. If you are a more of a power user with the budget to support the top tiers, Optimizely’s advanced features might also be for you.
Visual Website Optimizer
Visual Website Optimizer (often referred to as just VWO) is a full-featured A/B testing tool that’s similar to Optimizely in several ways.
It’s a full-fledged testing tool and surely one of the most complete solutions out there. It has all the basics a beginner would need to get started with A/B testing – and that includes for example, a visual campaign builder, which means you don’t have to worry about coding to start testing.
You’ll find an Idea Factory there too, which is a list of best-practices and ideas that you can use in your website right away. If you’re not sure how to get started, it will provide test suggestions for headlines, color schemes, user flows and others.
If you are an intermediate or more advanced user, VWO has a suite of features that can help you do the power testing you want to. It supports A/B, Multivariate and Split URL Testing, which is good for large design changes and testing between two completely different domains. In case you want to dive deeper than the visual editor, it also allows developers to make more sophisticated and complex changes in code.
VWO goes as far as offering an in-built heatmap tool to track click behavior and browsing habits and there’s a dashboard too, with access to insights from the reports generated from your campaigns and more.
Starting at $49 per month, it might be a little too much for really small websites, but it fits most budgets. On the downside, according to the same ConversionXL study mentioned earlier, VWO can oddly, have a negative impact in your website’s performance. It ranks in 7 out of 8.
Adobe Target is part of Adobe’s Marketing Cloud that’s a set of digital marketing solutions provided by Adobe. It covers all the basic features a good A/B testing tool should have: It supports Multivariate Testing, it’s mobile-optimized, allows you to create specific profiles and audiences to run deeply-customized tests. It doesn’t require coding to get set up and ready to go, offers geolocation targeting and more.
You can also use Adobe Analytics, (which is also a part of Adobe’s Marketing Cloud) to understand the data, do better targeting and create more precise filtering.
However, the reason why we decided to include it in this list is because it has a focus on automating the optimization of user experiences. The idea is to create customized user experiences that will serve specific target groups.
Their so-called “Auto Target” feature uses machine learning to automate optimization across the entire tool.
For example: If previously you’d run a test to see whether a 75-percent-off coupon would drive more sales than a 25-percent-off coupon to a specific type of user, with Auto Target you can include a unlimited number of variables in the test (other than the coupon value, different copy, images, layouts, buttons and so on) and to multiple types of users.
Furthermore, it will automatically track responses to these variations and point out to the ones with best performance.
Adobe Target is an advanced tool, and it is especially great if you use other software from the Marketing Cloud. The Auto Target really makes it worth giving it a try. You will need to contact them to learn which pricing best applies to you.
Convert is an A/B testing tool mainly targeted towards enterprise clients. It has a robust feature set that suits advanced users very well. It has all the basics: A Visual Editor for making changes, support for A/B, Split URL and Multivariate Testing (it’s specially great for this type of testing), and it integrates with dozens of tools – all the way from Hubspot and Mixpanel to Drupal and Twilio – and Conversion (with Support for multiple conversion goals) and Revenue Tracking.
And goes beyond the basics too: It offers auto bounce and engagement measurement, allows for behavioral and segmented testing, geo targeting capabilities, control tests and deep analytics integration. You will also find:
- Unlimited Projects and Users
- Flicker-Free Tests. When running A/B tests, sometimes a screen render might flick or blip, which in turn could affect a controlled experiment and produce unreliable results. Convert focuses on making sure flickering doesn’t affect tests.
- Almost no performance impact. It ranks in 3rd place in the ConversionXL study mentioned previously and its impact on how fast the website will load isn’t worrisome. When you consider enterprise clients whose websites have millions of visitors, this is especially important.
Convert probably isn’t the best tool for beginners looking to try out A/B testing and get going quickly but for larger organizations, it is the robust tool that’s made for them. It starts at $599/month (on annual billing) for up to 12M tested visitors, but it has a top tier plan that starts at $6,999/month that could be expanded to support up to 1.2B tested visitors.
Kissmetrics is a broadly used tool that focuses on tracking specific audiences throughout your funnels to help you increase retention, conversions and optimize for growth. They say that Google Analytics will tell you what’s happening, but that they tell you who’s doing it.
Until they released their own A/B testing tool, the only way to run these tests using data from Kissmetrics was by running a funnel report then exporting that to an external tool to analyse it. Now, all the A/B testing reporting can be done from within Kissmetrics.
You can dive deep and work with different types of reports:
- Revenue: Compare various segments to visualize which ones are bringing the most revenue, the lowest churn, highest lifetime value and so on.
- Cohort: Measure groups of people with similar characteristics or experiences. For instance, ex- “Customers who signed up in September”.
- Funnel: Identify problem areas in your business by creating funnels and observing how many people drop-off from one event to another.
Kissmetrics is highly efficient for segmenting your visitors.
Pricing can be a restrictive factor here. Kissmetrics’ basic plan starts at $220/month for up to 300k events – but package doesn’t include A/B testing. For you to access that feature, you have to choose their mid-tier plan which starts at $500/month for up to 300k/events.
Which tool to choose
Which one should you be using? These are our favorites, but for you to understand which one will better suit your needs, you should be asking the following assessment questions:
- What is my knowledge level?
Are you an absolute beginner looking to get started with A/B testing and just try it out? Or are you a seasoned marketer, experienced with testing multiple variants for different scenarios and audiences? You need to assess yourself to understand which tools will best serve you.
- What is my budget?
This question is equally important – what budget does your organization have for testing? Advanced tools for high-traffic websites could easily go up to the thousands of dollars per month. If you don’t have the budget, you should focus and make a better use of your time by only thinking about tools within a certain price range.
- What are my needs?
If you are an enterprise company with millions of visitors per month that go through highly complex funnels and can be segmented into hundreds of profiles, you should be looking into a very different tool than if you just want to see which copy works best for your app landing page.
- What do I hope to achieve?
What results are you looking for? This ties back to the complexity mentioned in the previous question. Have a clear picture of your goals and then look for tools that will help you accomplish them. Increasing conversions in a landing page is a far less complex task than increasing trial to paid conversions in a SaaS product, for example.
- Which one works best for me?
Lastly, once you have answers to the questions above, you will probably have identified a few tools that might suit you. The next step is to test them. Most mentioned above will give you either a free trial period or at least a demonstration. Which one seems to feel and work better? In the end of the day, you need to be comfortable with the tool to get the most out of it.