Skip to main content

Creating holdout experiments

The purpose of a holdout test is to compare the performance of a webpage or set of pages without Rokt’s integration to the performance of the same page with Rokt. This allows you to accurately measure long term impacts to customer behaviour by assigning a set of users to a holdout group (also known as a control). This group of customers are excluded from seeing Rokt’s integration and held to a baseline experience for an extended period of time.

In addition to measuring customer experience impacts, we can also use this data to demonstrate the effectiveness of Rokt’s integration and optimization efforts, and scientifically prove the net value that Rokt contributes to your bottom line.

Steps to create a holdout test in One Platform#

To create a holdout experiment, follow the following simple steps:

  1. Log in to your account in One Platform.
  2. Go to the Experiments Overview page (Main menu > Experiments).
  3. Click Add experiment.
  4. Enter an experiment name and select experiment type Page Holdout Experiment.
  5. Enter a hypothesis.
  6. Select a primary metric (and optional secondary metrics).
  7. Select the page you’d like to test.
  8. Click Next.

Design variant#

The next step is to define the holdout variant which imitates your original experience and allows you to test your hypothesis. It’s important to ensure there is an accurate baseline in place (known as a control) that will act as a point of comparison for any results you receive from your Rokt integration and optimization on your page.

The baseline experience should closely replicate the existing experience on your website. This includes the design and layout of your page, and the offer(s) that are displayed to your customers (if any).

You can select from one of two options when configuring the holdout variant.

  1. Display the page without Rokt. Select this option if you do not intend to show a Rokt placement to customers assigned to the holdout group.
  2. Display a Rokt placement to replicate an existing experience. Select this option if you want to configure a Rokt placement to replace some content on your page. For example, if Rokt is replacing an existing insurance upsell on your payment page.

It's important that your placement is identical in design, interactions, location, and copy to your existing experience in order to establish the baseline performance of your page. This ensures an accurate point of comparison when measuring impact and value from your Rokt optimized experience.

Set traffic allocation#

The traffic allocation step is used to define how many of your customers you wish to include in your holdout group. Enter a numeric value or use the slider to specify the percentage of all customers visiting the page to include in your experiment cohort.

We recommend a 5% control to measure the impact of Rokt's optimization.

Start your experiment#

Click Start experiment when you’re ready for your customers to start allocating customers to your holdout. When the status field says Live, your experiment is running on the page. Most experiments go live within five minutes.

How long should your experiment run?#

Page holdout experiments are a longer term study. We recommend you keep a holdout experiment running long enough to capture multiple cycles of repeat behaviour. This will differ depending on the average repeat frequency for your site. For example, repeat customers may make one purchase a month on average for a retail site, but once every six months on a travel website.

It could take up to or more than six months to gain enough data to accurately measure customer repeat purchase rates and make a confident decision about the impact.

Was this article helpful?