To create a page experiment, follow the following simple steps:
- Login to your account in One Platform.
- Go to the Experiments Overview page.
- Click Add experiment.
- Enter a Name and select experiment type "Page Multivariate Experiment".
- Enter a hypothesis.
- Select a Primary metric (and optional secondary metrics).
- Select the Page you’d like to test.
- Click Next.
The next step is to define the experiment variants that will validate or invalidate your hypothesis. It’s important to always ensure there is a baseline in place (known as a control) that will act as a point of comparison for any results you receive from your experiment. Below are some rules which guide best practice when designing your variants:
- Only test one element at a time so that you know what change is responsible for the results you are seeing. This helps you to build an understanding of how different elements influence customer behaviour, once you’ve got a good picture you can focus your experiments on the most impactful elements.
- Set up control and treatment. Retain a version of your original creative in the experiment, this is your control. The new variations you are testing are called treatments. These are challenger variants you want to test against your control.
You can create up to five variants to test against your original page experience. To get started, select a control variant from the list. This is an existing variant on the page which will be involved in the experiment.
Once you’ve selected your control, create test variants by changing one thing about your experience. This can be the placement design, layout, location, or offer combinations. Learn more about the types of elements you can test on your page.
If you’re changing something about the placement design or the types of offers that are allowed to show on your page, you’ll need to create new placements and/or establish different campaign control rules before building your experiment and adding variants.
- To make changes to your placement design, go to the Placements section (Transactions > Placements) and click Add placement. Use the editor panel to make a change (e.g., the button color). When you’re done, click Save. Repeat this process to create additional variations of your placement for the experiment.
- To change your campaign or marketplace control rules, go to the Controls tab (Transactions > Controls), and click Add controls, making sure you link to a placement. Repeat this process to create additional variations of your controls for the experiment.
To start configuring your test variants, click anywhere in the variant section for Variant 1. Enter a Variant name, select one or more placement, and input a Target element (this step is only required for embedded placements). To create multiple variants, click New variant in the variant tab section. Once you’ve finished designing all of your treatment variants, click Next.
The traffic allocation step is used to define how many of your customers you wish to include in your experiment. Enter a numeric value or use the slider to specify the percentage of all customers visiting the page to include in your experiment cohort.
Distribution refers to how the allocated traffic should be split between each of the test variants in the experiment. All variants are weighted equally by default. A customer who is included in your experiment has an equal chance of seeing any of your variants.
Click Start experiment when you’re ready for your customers to see the experiment. When the status field says Live, your experiment is running on the page. Most experiments will be live within 5 minutes.
We recommend you keep an experiment running for at least two weeks to ensure you collect enough data. Do not end your experiment until at least one variant has a 95% probability to beat baseline. It could take up to or more than 30 days to gain enough data to make a confident decision about the outcome.