Core Web Vitals A/B testing best practices

Core Web Vitals A/B testing best practices

A/B testing is often used to come up with better headlines or layout, increasing conversion. However, even today A/B testing isn't always implemented in an optimal way, negatively impacting your business instead.

Creating different A/B variants is often given a lot of thought. But the technical characteristics isn't always given a lot of consideration. Which is a shame, as it can easily impact real user experience. This means Core Web Vitals will be impacted, impacting your revenue and SEO as well.

A/B testing characteristics

To be able to optimally implement A/B testing, it might be convenient to know how browsers work. So let's dive into some A/B testing best practices:

  • A/B testing should never prevent the browser from doing the most important work: parsing. Be sure to use the defer or at least async attribute at all times;
  • If elements above the fold are being A/B tested, then you might want to use the async attribute;
  • And that's where the so-called async hiding snippet comes in: making the page invisible while A/B variants are being applied while the browser can continue parsing-work.

Google Optimize even has a Q&A on which implementation to use. Keep in mind though that it should never block the browser from doing its work as mentioned above.

A/B testing providers

Some common A/B testing providers and their documentation pages regarding the implementation of their A/B testing services:

Other implementation best practices

The implementation mentioned on their guides is called hiding-snippet or anti-flicker script. But these aren't always followed up properly, or just not read at all. Some best practices:

  • Especially when using inlined JS, the script-element should come before any other JS or CSS files;
  • Be sure to not cause any chained requests. It's better to directly embed A/B testing, instead of loading it via Google Tag Manager or Adobe Tag Manager;
  • In case of inlined JS, the browser's look-ahead-parser won't be able to tell that a specific file should be downloaded. Downloading the required JavaScript in advance is a best practice. This can be achieved using a preload resource hint.
    When the exact resource path changes per request or session, you might want to just do a domain name lookup in advance by using the preconnect resource hint.
  • Be sure to set a timeout, in case the required JavaScript could not do its work in time. To determine a proper timeout, you could look at real user monitoring data, especially unique pageview experiences as those will result in higher FCP experiences.

But do you really need a hiding snippet?

Hiding snippet is necessary to prevent users from seeing page flickering. Page flicker could distract users or skew your A/B testing results. A hiding snippet will prevent this, but as it is impacting user engagement and perceived performance, it's just the lesser of two evils.

And maybe you don't even need this. So, also note:

  • if no A/B testing is done above the fold, you likely don't need any async hiding snippets at all. Just applying the defer attribute will do;
  • If it's always the same element that is being A/B tested, such as the headline or a hero image, you should just hide those individual elements, instead of preventing the whole page from being rendered.

Server side A/B testing

Most A/B testing is done using JavaScript, known as client side A/B testing. This has be to executed on the main thread of the browser. It's then not only the download of the JavaScript, but also parsing and execution that is impacting performance.

If you really want to reduce the impact of A/B testing on FCP and thus real user experience to a minimum, server side A/B testing will be a better bet. For example, next to server side Google Optimize, Akamai (EdgeWorkers) and Cloudflare is offering a solution too, where you can load different HTML for the same url, depending on specific conditions.

We've implemented a self made server side A/B testing for a website not running on a CDN. This even works on top of server side caching. On the server, it then only takes 3ms to apply other variants. That's all. You won't find any JavaScript that will be downloaded nor executed in 3ms, illustrating that server side A/B testing will beat client side A/B testing.