Skip to main content

Building experiments

To build a creative experiment, follow these steps:

  1. Log in to your account in One Platform.
  2. From the left navigation, click Experiments.
  3. Click Add experiment.
  4. Enter an experiment name.
  5. Select Creative Multivariate Experiment.
  6. Click Continue.

Experiment details#

  1. Enter a hypothesis. A good hypothesis clarifies the problem you are trying to solve, focuses your experiment, and gives a measure of success. Make sure your hypothesis clearly articulates the change you are making (e.g., changing the copy of the creative) and identifies your main success metric (e.g., conversions per referral).
  2. Select a primary metric (and optional secondary metrics). When measuring the success or failure of your experiment, the primary metric determines whether your experiment proves or disproves the hypothesis.
  3. Select a Minimum uplift.
    1. Minimum uplift is the smallest relative percentage change you want to detect in the primary success metric. Rokt uses this value to estimate the expected duration of the test, recommend actions, and ensure you collect enough data to measure your desired uplift.
  4. Click Next.

Design variants#

The next step is to define the experiment variations that will validate or invalidate your hypothesis. It’s important to ensure there is a baseline in place (known as a control) that will act as a point of comparison for any results you receive from your experiment. Below are some rules which guide best practice when designing your variants:

  1. Only test one element at a time so that you understand which element triggered the results you are seeing. We don’t recommend changing both the call to action and the landing page simultaneously.
  2. Set up control and treatment. Retain a version of your original creative in the experiment; this is your control. The new variations you are testing are called treatments. They are challenger variants you’re testing against your control.

To get started:

  1. Select a campaign and creative from the dropdowns. This is an existing creative which will be the baseline for your experiment. Once selected, you can see the audience details for the creative.
  2. Click Next.
  3. Click New Variant + to define your creative variant.
  4. Make changes to the creative variant. Keep in mind the recommendation above to only change one element at a time. For example, change one of CTA, landing page, copy, logo, or promotion value.
  5. You can create up to five variants to test against your control.
  6. When you finish adding variants, click Next.

Traffic allocation & variant distribution#

The traffic allocation step is used to define how many of your customers you wish to include in your experiment.

Enter a numeric value, or use the slider to specify the percentage of all customers viewing the offer to include in your experiment cohort. This is a mechanism to control exposure to your test variants and reduce the risk of harming your overall campaign performance. It is important here to consider how long it will take to generate meaningful results, which can differ based on the scale of your campaign, and how long it takes your typical customer to convert after seeing an offer.

Starting your experiment#

When you’re ready for customers to see the experiment, click Start experiment. When the status field says Live, your experiment is running. Most experiments go live within five minutes. You can view all configured experiments on the Experiment Overview page.

How long should an experiment run?#

We recommend keeping an experiment running for at least two weeks to ensure you collect enough data. Do not end your experiment until at least one variant has a 95% probability to beat baseline. It could take up to 30 days to gain enough data to make a confident decision about the outcome.

Was this article helpful?