How long should my experiment run?
Holdout tests
Page holdout experiments are a longer term study. We recommend you keep a holdout experiment running long enough to capture multiple cycles of repeat behaviour. This will differ depending on the average repeat frequency for your site. For example, repeat customers may make one purchase a month on average for a retail site, but once every six months on a travel website.
It could take up to six months or more to gain enough data to accurately measure customer repeat purchase rates and make a confident decision about the impact.
Multivariate tests
We recommend you keep an experiment running for at least two weeks to ensure you collect enough data. Do not end your experiment until at least one variant has a 95% probability to beat baseline.
It could take up to or more than 30 days to gain enough data to make a confident decision about the outcome.