A Complete Incrementality Analysis

Overview

Incrementality asks what happened because a treatment was applied, beyond what would have happened under control. IncrementalityTEST analyzes a collection of experiment-level treatment and control measurements.

The workflow has four steps:

  1. calculate business metrics where necessary;
  2. validate and pair treatment and control rows;
  3. calculate one effect per experiment;
  4. summarize those effects and quantify uncertainty.

Prepare group-level data

Each experiment needs exactly one row for control and one for treatment.

library(IncrementalityTEST)

results <- data.frame(
  experiment = rep(paste0("test_", 1:6), each = 2),
  group = rep(c("control", "treatment"), 6),
  transactions = c(100, 112, 80, 87, 130, 141, 92, 96, 110, 121, 70, 79),
  users = c(1000, 1005, 800, 804, 1200, 1208, 900, 902, 1050, 1053, 700, 704),
  revenue = c(2500, 2860, 1920, 2140, 3300, 3690, 2200, 2350, 2750, 3100, 1680, 1950),
  buyers = c(90, 99, 72, 78, 115, 124, 83, 86, 101, 109, 63, 70)
)

Calculate the four supported commerce metrics and attach them to the data.

metrics <- calculate_metrics(
  results$transactions,
  results$users,
  results$revenue,
  results$buyers
)
results <- cbind(results, metrics)
head(results)
#>   experiment     group transactions users revenue buyers      RPU         BR
#> 1     test_1   control          100  1000    2500     90 2.500000 0.09000000
#> 2     test_1 treatment          112  1005    2860     99 2.845771 0.09850746
#> 3     test_2   control           80   800    1920     72 2.400000 0.09000000
#> 4     test_2 treatment           87   804    2140     78 2.661692 0.09701493
#> 5     test_3   control          130  1200    3300    115 2.750000 0.09583333
#> 6     test_3 treatment          141  1208    3690    124 3.054636 0.10264901
#>        AOV      TPB
#> 1 25.00000 1.111111
#> 2 25.53571 1.131313
#> 3 24.00000 1.111111
#> 4 24.59770 1.115385
#> 5 25.38462 1.130435
#> 6 26.17021 1.137097

Inspect experiment-level effects

metric_differences() uses treatment minus control by default. A positive number therefore means that the metric was higher under treatment.

effects <- metric_differences(results, metric = "RPU")
effects
#>   experiment  control treatment difference
#> 1     test_1 2.500000  2.845771  0.3457711
#> 2     test_2 2.400000  2.661692  0.2616915
#> 3     test_3 2.750000  3.054636  0.3046358
#> 4     test_4 2.444444  2.605322  0.1608771
#> 5     test_5 2.619048  2.943970  0.3249220
#> 6     test_6 2.400000  2.769886  0.3698864

Missing groups, duplicate groups, and nonnumeric metrics produce explicit errors. Use na_action = "omit" only when dropping incomplete experiments is methodologically defensible.

Estimate the overall effect

analysis <- analyze_incrementality(
  results,
  metric = "RPU",
  conf_level = 0.95,
  bootstrap_times = 2000,
  seed = 2026
)
analysis
#> Incrementality analysis
#> Metric: RPU
#> Direction: treatment-control
#> Experiments: 6
#> 
#>     method      mean standard_error conf_level     lower     upper
#>  Student t 0.2946306     0.03069823       0.95 0.2157183 0.3735430
#>  Bootstrap 0.2946306     0.02819784       0.95 0.2358360 0.3433737

The object contains all intermediate and final results:

analysis$differences
#>   experiment  control treatment difference
#> 1     test_1 2.500000  2.845771  0.3457711
#> 2     test_2 2.400000  2.661692  0.2616915
#> 3     test_3 2.750000  3.054636  0.3046358
#> 4     test_4 2.444444  2.605322  0.1608771
#> 5     test_5 2.619048  2.943970  0.3249220
#> 6     test_6 2.400000  2.769886  0.3698864
analysis$t_interval
#>   n      mean standard_error conf_level     lower    upper
#> 1 6 0.2946306     0.03069823       0.95 0.2157183 0.373543
analysis$bootstrap_interval
#>        mean standard_error conf_level    lower     upper
#> 1 0.2946306     0.02819784       0.95 0.235836 0.3433737

The t interval assumes that experiment-level effects are independent and that their sampling distribution is reasonably approximated by a normal distribution. The percentile bootstrap makes fewer distributional assumptions, but a small or unrepresentative set of experiments still limits inference.

Report the result

A useful report states:

The package estimates an unweighted mean across experiments. If experiments have materially different precision or target populations, a hierarchical model or justified weighting strategy may be more appropriate.

Working with legacy data

For existing datasets containing iabtest_id and numeric abt_group values, the compatibility function remains available:

legacy <- transform(
  results,
  iabtest_id = experiment,
  abt_group = ifelse(group == "control", 0, 1)
)
test_metric(legacy, "RPU")
#>   iabtest_id    inc_RPU
#> 1     test_1 -0.3457711
#> 2     test_2 -0.2616915
#> 3     test_3 -0.3046358
#> 4     test_4 -0.1608771
#> 5     test_5 -0.3249220
#> 6     test_6 -0.3698864

For historical compatibility, test_metric() uses control minus treatment. New analyses should use metric_differences() so the direction is explicit.