A/B testing is often used to come up with better headlines or layout, increasing conversion. However, even today A/B testing isn't always implemented in an optimal way, negatively impacting your business instead.
Creating different A/B variants is often given a lot of thought. But the technical characteristics isn't always given a lot of consideration. Which is a shame, as it can easily impact real user experience. This means Core Web Vitals will be impacted, impacting your revenue and SEO as well.
A/B testing characteristics
To be able to optimally implement A/B testing, it might be convenient to know how browsers work. So let's dive into some A/B testing best practices:
- A/B testing should never prevent the browser from doing the most important work: parsing. Be sure to use the defer or at least async attribute at all times;
- If elements above the fold are being A/B tested, then you might want to use the async attribute;
- And that's where the so-called async hiding snippet comes in: making the page invisible while A/B variants are being applied while the browser can continue parsing-work.
Google Optimize even has a Q&A on which implementation to use. Keep in mind though that it should never block the browser from doing its work as mentioned above.
A/B testing providers
Some common A/B testing providers and their documentation pages regarding the implementation of their A/B testing services:
- Google Optimize
- Visual Web Optimizer
Prevent page flicker;
Non blocking snippet;
- Adobe Target
Manage flicker with at.js.
Other implementation best practices
The implementation mentioned on their guides is called hiding-snippet or anti-flicker script. But these aren't always followed up properly, or just not read at all. Some best practices:
- Especially when using inlined JS, the script-element should come before any other JS or CSS files;
- Be sure to not cause any chained requests. It's better to directly embed A/B testing, instead of loading it via Google Tag Manager or Adobe Tag Manager;
When the exact resource path changes per request or session, you might want to just do a domain name lookup in advance by using the preconnect resource hint.
But do you really need a hiding snippet?
Hiding snippet is necessary to prevent users from seeing page flickering. Page flicker could distract users or skew your A/B testing results. A hiding snippet will prevent this, but as it is impacting user engagement and perceived performance, it's just the lesser of two evils.
And maybe you don't even need this. So, also note:
- if no A/B testing is done above the fold, you likely don't need any async hiding snippets at all. Just applying the defer attribute will do;
- If it's always the same element that is being A/B tested, such as the headline or a hero image, you should just hide those individual elements, instead of preventing the whole page from being rendered.
Server side A/B testing
If you really want to reduce the impact of A/B testing on FCP and thus real user experience to a minimum, server side A/B testing will be a better bet. For example, next to server side Google Optimize, Akamai (EdgeWorkers) and Cloudflare is offering a solution too, where you can load different HTML for the same url, depending on specific conditions.