---
title: "Weighted and survey data"
description: >
  Weights, complex survey designs, and the three levels of margin of error — which one your file deserves, on both producers.
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Weighted and survey data}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
# Messages and warnings are off for every chunk: the teaching notes tabxplor prints (an
# auto-detected family, an over-dispersion caveat) are explained in the prose where they
# matter, and repeated under every table they only clutter it. Re-enable one with
# `message = TRUE` on the chunk that needs it.
knitr::opts_chunk$set(collapse = TRUE, comment = "#>",
                      message = FALSE, warning = FALSE)
```

```{r setup}
library(tabxplor)

# Pin the legend language: it defaults to "auto" = the ambient locale, so building this English
# vignette on a French machine silently renders French legends and captions (the -fr articles pin
# "fr" for the same reason). Output must not depend on where it is built.
options(tabxplor.lang = "en")
Sys.setenv(LANGUAGE = "en")   # the test-summary / model-fit row labels go through gettext, not this option
library(dplyr)

# Tables render as tabxplor's real html tables (the recommended everyday setting); the shared
# stylesheet is emitted once by tab_css() below, and the hover tooltips are kept off here.
options(tabxplor.print = "html")
options(tabxplor.tab_kable_css = FALSE)
options(tabxplor.tab_kable_tooltips = FALSE)

options(cli.num_colors = 256)
set_color_palette(theme = "light")
```

```{r, echo = FALSE, results = "asis"}
# The website carries a light/dark switch and tab_css("auto") follows it; a shipped vignette
# is always read on a light page, so there it stays light.
cat(tab_css(theme = if (Sys.getenv("IN_PKGDOWN") == "true") "auto" else "light"))
```

```{r, echo = FALSE, include = FALSE}
# Colour the console outputs (ANSI -> html, via fansi), but hand as-is results (the html tables,
# marked by knitr with an ASIS token) back to knitr's default hook untouched.
# Escape the three HTML specials before fansi turns the ANSI codes into markup.
esc_html <- function(x) gsub(">", "&gt;", gsub("<", "&lt;", gsub("&", "&amp;", x, fixed = TRUE),
                                               fixed = TRUE), fixed = TRUE)
# fansi is Suggests-only, so the ANSI -> html step degrades: without it the escape codes are
# stripped and the output is handed on uncoloured, which is what a check run with no Suggests gets.
ansi_html <- if (requireNamespace("fansi", quietly = TRUE)) {
  function(x) fansi::sgr_to_html(x = esc_html(x), warn = FALSE)
} else {
  function(x) esc_html(gsub("\033\\[[0-9;]*m", "", x))
}
default_output_hook <- knitr::knit_hooks$get("output")
knitr::knit_hooks$set(output = function(x, options) {
  if (grepl("KNITR_ASIS_OUTPUT_TOKEN", x, fixed = TRUE)) return(default_output_hook(x, options))
  paste0('<pre class="r-output"><code>',
         ansi_html(x),
         '</code></pre>')
})
# A cli message or warning is its own kind of condition, so knitr routes each through its own hook,
# not `output`: without these two it would land in the collapsed source block, ANSI codes and all.
for (hook in c("message", "warning")) {
  knitr::knit_hooks$set(stats::setNames(list(function(x, options) {
    paste0('<pre class="r-output"><code>',
           ansi_html(x),
           '</code></pre>')
  }), hook))
}
```

```{r data, include = FALSE}
gss_simple <- gss_cat_data_formatting()
```

*Une version française de ce document est disponible : [Données pondérées et plans de sondage](https://bricenocenti.github.io/tabxplor/articles/tabxplor-weights-fr.html).*

Most survey files come with a **weight**: a number saying how many people in the population each respondent stands for. The argument `wt =` uses it, and every percentage and mean in the table becomes an estimate of the *population* rather than of the people you happened to interview.

That is the easy half. The harder half is the **margin of error** around those percentages — the confidence intervals, the significance stars, the colours and the tests. There, tabxplor has **three levels**, and a table's footer always tells you which one it is on.

| you pass | level | what the intervals and tests carry |
|---|---|---|
| `wt = w` | 1 | the weighted estimate, on the raw *n* — no design effect at all |
| `wt = w` + `design_effect = TRUE` | 2 | the unequal weighting, exactly — blind to clustering and calibration |
| a `survey` design as `data` | 3 | the full design: strata, clusters, `fpc`, calibration |

This vignette is about choosing between them. `vignette("tabxplor")` covers cross-tables themselves, and `vignette("tabxplor-reg")` the regression framework.

## Level 1 — weighted percentages, plain margins of error (the default)

```{r, echo=TRUE, eval=FALSE}
tab(my_survey, education, job, wt = weight, pct = "row")
#> Weighted by weight; confidence intervals and tests use the unweighted sample size.
```

The percentages are weighted; the margins of error simply count your real respondents. This is the convention almost every teaching text and every point-and-click program uses, and it is why it is the default here. But when the weights are **unequal** it is optimistic: a sample where some people count for four others carries less information than its size suggests, and the intervals come out usually a bit too narrow.

## Level 2 — `design_effect = TRUE`

```{r, echo=TRUE, eval=FALSE}
tab(my_survey, education, job, wt = weight, pct = "row", design_effect = TRUE)
options(tabxplor.design_effect = TRUE)   # or once, for the whole session
#> Weighted by weight; confidence intervals and tests account for the weighting.
```

Now the unequal weighting is taken into account in **every** interval, star, colour threshold and whole-table test. This is not a rule of thumb: a column of weights *is* a survey design — the simplest one, with no clusters and no strata — and tabxplor computes that design's variance exactly, reproducing the reference [survey](https://CRAN.R-project.org/package=survey) package to the last digit.

Here it is on real data. We give `gss_simple` a made-up weight (never-married people are under-covered in most surveys, so they count for more), and build the same table twice:

```{r, echo = TRUE}
gss_w <- dplyr::mutate(gss_simple, w = ifelse(marital %in% "Never married", 2.5, 0.8))

tab(gss_w, race, party3, wt = w, pct = "row", ci = "cell", na = "drop")
```

```{r, echo = TRUE}
tab(gss_w, race, party3, wt = w, pct = "row", ci = "cell", na = "drop",
    design_effect = TRUE)
```

Same percentages, wider brackets, and a footer that says what changed. Behind them, the 16 292 White respondents are now worth about 11 800 — the price of the unequal weights.

If you weight at all, level 2 is the honest reading of your own numbers, and it costs nothing. It is not the default only because switching it on moves every figure in every table anyone has already produced, so it should be a deliberate, visible act rather than a silent one. One more reason to turn it on: a regression table's observed columns are *always* at level 2, so this is what makes a `tab()` percentage and a `tab_reg()` observed percentage on the same data directly comparable.

## Level 3 — a `survey` design

Real surveys are not drawn by picking names at random from the whole country. They are usually **stratified** (drawn separately by region, city size…), often **clustered** (a few neighbourhoods are picked first, then several households inside each), and their weights are usually **calibrated** so the sample matches known population totals for age, sex, region. Those three facts are not in the weight column; they live in extra variables. If your file has them, build a design with the [survey](https://CRAN.R-project.org/package=survey) package and pass it as `data`:

```{r, echo=TRUE, eval=FALSE}
library(survey)
d <- svydesign(ids = ~psu, strata = ~stratum, weights = ~w, data = my_survey, nest = TRUE)
tab(d, race, marital, pct = "row", color = TRUE, test = TRUE)
#> Design-based (survey): weighted estimates, intervals and tests account for the sample design.
```

Everything then follows the design: the estimates, the p-values, and every confidence interval, star and colour threshold. Replicate-weight designs (`svrepdesign()`) and two-phase designs are not supported — tabxplor refuses them rather than approximating.

## What level 2 can and cannot do for you

Level 2 sees the weights, and nothing else. That matters, because the three things it cannot see do **not** pull in the same direction:

| | seen at level 2 | needs a design (level 3) |
|---|---|---|
| unequal weights | **yes, exactly** | |
| strata, calibration, finite population | | would **narrow** your intervals — by a few percent at most |
| **clusters** | | can make them **much wider** |

So the two corrections you are missing at level 2 are wildly unequal. Ignoring the strata and the calibration costs a few percent, and in the *safe* direction — your intervals are slightly too cautious. Ignoring **clusters** goes the other way and can be enormous: on a real clustered school survey, an average that level 2 reports with a margin of error of ±3 points really deserves ±9. On the American NHANES health survey, the distribution of ethnicity comes out **nine times** too precise, because that survey is built around exactly that variable. These do not cancel out: when clustering is present, it wins.

**Which surveys are clustered?** Not the ones drawn from a register, or run by web, telephone or mail — there level 2 is simply the right answer, however unequal the weights. But **face-to-face** household surveys are clustered by construction, because sending interviewers to scattered addresses is unaffordable. The French *Enquête Emploi*, for instance, is drawn as groups of about **twenty neighbouring dwellings**; neighbours resemble each other, so twenty interviews in one street tell you less than twenty interviews spread across the country.

**Does it matter for *your* table?** Usually much less than those numbers suggest, and there is a simple test. The cluster effect largely cancels out in a *comparison*, as long as both compared groups are found inside the same neighbourhoods. So ask one question about your row variable:

> Does it **vary within** a neighbourhood, or does it **define** one?

Sex, age, diploma, occupation, income, opinions — these vary within any neighbourhood, so a comparison between their categories is close to right at level 2. Region, urban/rural, city size, type of neighbourhood, local immigrant density — these *are* the neighbourhood, and there level 2 is as wrong as it is for a single percentage. The measurements back this up: on the same clustered file where an average is three times too precise, the *difference* between two groups is only 13 % too precise, and a regression coefficient 8 % — and differences are exactly what tabxplor's colours and stars test.

**Check what your file actually contains before reaching for level 3.** Look for a stratum, cluster or calibration variable (`strate`, `grappe`, `psu`, `nomen`…). Many released research files ship one calibrated weight and nothing else — the cluster is a small geographic area, so it is withheld for confidentiality — and on such a file a `survey` design would carry only the unequal weighting, which is exactly what level 2 already gives you. The European Social Survey is a useful middle case: it does publish its clusters and strata, but in a *separate* file you have to download and merge.

## Regression on weighted data

There are only **two ways to hand tabxplor your weights**, and they are the same two: either you give `wt =` a weight column, or you build the design once and pass **the design itself** as `data`.

```{r, eval = FALSE}
tab_reg(data, "outcome", c("pred1", "pred2"), wt = "weight")

library(survey)
d <- svydesign(ids = ~psu, strata = ~stratum, weights = ~w, data = my_survey, nest = TRUE)
tab_reg(d, "outcome", c("pred1", "pred2"), empirical = TRUE)
```

Everything then follows that design, in one regime: the `Model_*` coefficients and their intervals (`survey::svyglm()`), the average marginal effects, the per-standard-deviation scaling of numeric predictors, the model-versus-observed gap test — **and the observed `Obs_*` companion columns**, whose intervals are built on the design variance of each cell. That last point is what makes the comparison work: the model column and the observed column beside it are measured the same way, so a difference between them is about *adjustment*, not about two different notions of uncertainty.

Two honest limits. A comparison *between two cells* (an observed odds ratio, a risk difference) ignores the design covariance between them, so it lands a few percent either side of the exact answer — against the 15–25 % it was out by when those columns were computed as if the sample were simple random. And a *nominal* outcome has no observed *column*: its crude value is folded into the model cell as a point, with no interval to make design-based.

### Why a weighted `tab_reg()` is not a weighted `tab()`

`tab_reg()` cannot choose: its observed columns have to be measured like the model column beside them, and that one is design-based by construction (`survey::svyglm()`). `tab()` can, and keeps the descriptive convention — a weighted estimate on the raw sample size — as its default. So `tab_reg()` is always at **level 2** (or level 3 under a design), while `tab()` starts at level 1, and `tab(design_effect = TRUE)` is what brings the two into line. Each table's footer says which it used.

## The fine print

- **Sometimes level 2 makes an interval *narrower*.** That is correct, not a bug. If the weights happen to line up with what you are measuring, unequal selection can carry *more* information than equal selection, and the effective sample size comes out above your number of respondents.
- **Degrees of freedom.** A clustered survey is usually built on few clusters — 15 to 60 is common — and level 3 refers each interval to that number, which widens it again by roughly 8 %. Level 2 has no clusters to count, so it cannot do this: a second, separate reason a face-to-face survey's true margin of error is wider than what level 2 prints.
- **When level 3 cannot be computed.** If a design's variance fails for a table, it falls back to the level-2 correction and says so in its footer, rather than claiming a design its numbers do not carry.
- **Exact for a cell, cautious for a difference.** Level 3 is exact for a single percentage; for a cell-vs-reference *difference* it cannot carry the covariance between the two cells, so the difference comes out slightly too wide rather than too narrow. It will never hand you a star the design does not support.
- **Pre-aggregated counts and cost.** Level 2 needs the individual weights, so a table built with `tab_counts()` from published counts cannot use it — such a table says so in its footer. And a level-3 table costs roughly three times a weighted one.

## Where to go next

- `vignette("tabxplor")` — cross-tables, colours and confidence intervals.
- `vignette("tabxplor-reg")` — the regression framework.
- `?tab` for `wt`, `design_effect` and `ci_method`; `?tabxplor-options` for the session defaults.
- `survey::svydesign()` for complex designs (strata, clusters, calibration, finite-population corrections).