---
title: "A Complete Incrementality Analysis"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{A Complete Incrementality Analysis}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

## Overview

Incrementality asks what happened because a treatment was applied, beyond what
would have happened under control. `IncrementalityTEST` analyzes a collection
of experiment-level treatment and control measurements.

The workflow has four steps:

1. calculate business metrics where necessary;
2. validate and pair treatment and control rows;
3. calculate one effect per experiment;
4. summarize those effects and quantify uncertainty.

## Prepare group-level data

Each experiment needs exactly one row for control and one for treatment.

```{r}
library(IncrementalityTEST)

results <- data.frame(
  experiment = rep(paste0("test_", 1:6), each = 2),
  group = rep(c("control", "treatment"), 6),
  transactions = c(100, 112, 80, 87, 130, 141, 92, 96, 110, 121, 70, 79),
  users = c(1000, 1005, 800, 804, 1200, 1208, 900, 902, 1050, 1053, 700, 704),
  revenue = c(2500, 2860, 1920, 2140, 3300, 3690, 2200, 2350, 2750, 3100, 1680, 1950),
  buyers = c(90, 99, 72, 78, 115, 124, 83, 86, 101, 109, 63, 70)
)
```

Calculate the four supported commerce metrics and attach them to the data.

```{r}
metrics <- calculate_metrics(
  results$transactions,
  results$users,
  results$revenue,
  results$buyers
)
results <- cbind(results, metrics)
head(results)
```

## Inspect experiment-level effects

`metric_differences()` uses treatment minus control by default. A positive
number therefore means that the metric was higher under treatment.

```{r}
effects <- metric_differences(results, metric = "RPU")
effects
```

Missing groups, duplicate groups, and nonnumeric metrics produce explicit
errors. Use `na_action = "omit"` only when dropping incomplete experiments is
methodologically defensible.

## Estimate the overall effect

```{r}
analysis <- analyze_incrementality(
  results,
  metric = "RPU",
  conf_level = 0.95,
  bootstrap_times = 2000,
  seed = 2026
)
analysis
```

The object contains all intermediate and final results:

```{r}
analysis$differences
analysis$t_interval
analysis$bootstrap_interval
```

The t interval assumes that experiment-level effects are independent and that
their sampling distribution is reasonably approximated by a normal
distribution. The percentile bootstrap makes fewer distributional assumptions,
but a small or unrepresentative set of experiments still limits inference.

## Report the result

A useful report states:

- the metric and effect direction;
- the number and selection of experiments;
- the mean experiment-level incremental effect;
- the confidence level and interval method;
- the treatment/control unit and any filtering decisions.

The package estimates an unweighted mean across experiments. If experiments
have materially different precision or target populations, a hierarchical
model or justified weighting strategy may be more appropriate.

## Working with legacy data

For existing datasets containing `iabtest_id` and numeric `abt_group` values,
the compatibility function remains available:

```{r}
legacy <- transform(
  results,
  iabtest_id = experiment,
  abt_group = ifelse(group == "control", 0, 1)
)
test_metric(legacy, "RPU")
```

For historical compatibility, `test_metric()` uses control minus treatment.
New analyses should use `metric_differences()` so the direction is explicit.