Overview
An A/B test is a great way to test two variations of a design against one another to measure the impact on a pre-determined metric. During an A/B test, anyone matching your target audience will see one variation of the campaign.
These types of tests are great at helping you understand which copy, creative, position or format is best to do a particular job. Oftentimes, we find that we can unconsciously decide that a certain format is better than another - A/B testing helps us validate these assumptions.
Running an A/B test is not always the best option. Sometimes, you may need to consider adding in a control split to these tests to ensure that both the creatives are improving upon the standard site experience. Whilst it may be tempting to add a control group to every A/B test, the trade off with introducing additional splits is that your campaign will take longer to hit confidence.
Key Metrics Used in Calculations
It's important to understand where the calculations compare to with an A/B test. If you do not include a control split, the calculations will compare to one another (which is why below you can see one negative, and one positive version). Including a control split means that both A and B will automatically compare to the control group.
NB The comparative numbers are not identical because the calculations are based on impressions, and there is always some slight variance.
Sessions - The number of sessions where a user successfully triggered a campaign event. If a user triggers a campaign multiple times in the same session, this will still only count as one session.
Sales - The number of sessions in which a user triggers a campaign event and completes a sale. This does not include any sales made in future sessions.
Revenue - The amount of revenue generated from sales attributed to a campaign.
Clicks - Any time a user clicks on any CTA on a campaign. A CTA must be selected as Count Towards CTR to be tracked as a click.
Leads - Any event in which a user fills out a form and submits the information via a Submit CTA. This includes both campaigns that request an Email/SMS or survey campaigns.
Closes - Any time a user clicks on a campaign close button. This does not include Dismiss CTAs (a click) or if someone clicks outside of an overlay.