1. Datasets for Online Controlled Experiments
- Author
-
Liu, B, Cardoso, A, Couturier, P, and McCoy, E
- Subjects
Methodology (stat.ME) ,FOS: Computer and information sciences ,Computer Science - Databases ,Applications (stat.AP) ,Databases (cs.DB) ,Statistics - Applications ,Statistics - Methodology - Abstract
Online Controlled Experiments (OCE) are the gold standard to measure impact and guide decisions for digital products and services. Despite many methodological advances in this area, the scarcity of public datasets and the lack of a systematic review and categorization hinder its development. We present the first survey and taxonomy for OCE datasets, which highlight the lack of a public dataset to support the design and running of experiments with adaptive stopping, an increasingly popular approach to enable quickly deploying improvements or rolling back degrading changes. We release the first such dataset, containing daily checkpoints of decision metrics from multiple, real experiments run on a global e-commerce platform. The dataset design is guided by a broader discussion on data requirements for common statistical tests used in digital experimentation. We demonstrate how to use the dataset in the adaptive stopping scenario using sequential and Bayesian hypothesis tests and learn the relevant parameters for each approach., 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks. 17 pages, 2 figures, 2 tables. Dataset available on Open Science Framework: https://osf.io/64jsb/
- Published
- 2021