Last updated Oct 11, 2024

Create a holdout test

The purpose of a holdout test is to compare the performance of a webpage or set of pages without Rokt’s integration to the performance of the same page with Rokt. This allows you to accurately measure long term impacts to customer behaviour by assigning a set of users to a holdout group (also known as a control). This group of customers are excluded from seeing Rokt’s integration and held to a baseline experience for an extended period of time.

In addition to measuring customer experience impacts, we can also use this data to demonstrate the effectiveness of Rokt’s integration and optimization efforts, and scientifically prove the net value that Rokt contributes to your bottom line.

Create a holdout test

1. Log in to my.rokt.com, and click on your account on the Home page.
2. Click Transactions.
3. Click Experiments.
4. Click Add experiment.
5. Enter an experiment name.
6. Select Page Holdout Experiment as the experiment type.
7. Click Continue.
8. Enter a hypothesis.
9. Select a Success metric.
10. Optional Select a Secondary metric.
11. Enter a Minimum uplift percentage.
12. Select an Experiment page.
13. Select an Audience to run your experiment on.
14. Click Next.

Design a variant

The next step is to define the holdout variant which imitates your original experience and allows you to test your hypothesis. It’s important to ensure there is an accurate baseline in place (known as a control) that will act as a point of comparison for any results you receive from your Rokt integration and optimization on your page.

The baseline experience should closely replicate the existing experience on your website. This includes the design and layout of your page, and the offer(s) that are displayed to your customers (if any).

You can select from one of two options when configuring the holdout variant.

1. Display the page without Rokt: Select this option if you do not intend to show a Rokt layout to customers assigned to the holdout group.
2. Display a Rokt layout to replicate an existing experience: Select this option if you want to configure a Rokt layout to replace some content on your page. For example, if Rokt is replacing an existing insurance upsell on your payment page. See How to create a layout for more information.

It's important that your layout is identical in design, interactions, location, and copy to your existing experience in order to establish the baseline performance of your page. This ensures an accurate point of comparison when measuring impact and value from your Rokt optimized experience.

Set traffic allocation

The traffic allocation step is used to define how many of your customers you wish to include in your holdout group. Enter a numeric value or use the slider to specify the percentage of all customers visiting the page to include in your experiment cohort.

Recommendation

We recommend a 5% control to measure the impact of Rokt's optimization.

Start your experiment

Click Start experiment when you’re ready for your customers to start allocating customers to your holdout. When the status field says Live, your experiment is running on the page. Most experiments go live within five minutes.