---
title: "Disposition screening"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Disposition screening}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
# Auth/network chunks are shown but not run at build time; the disposition_summary
# chunk below (on an in-memory frame) runs, so its output is real.
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = FALSE)
```

Screen a phone sample against the **disposition dataset** -- one row per `(phone, campaign_id)` for every recipient Survey160 has contacted across closed campaigns -- to drop numbers already completed, refused, or screened out before you field. Assumes you have completed First-time setup (the package README, or `?s160_gcs_init` from R): Cloud Storage access to `s160_disposition_prod`.

## Screen a sample

Pull the dataset once (cached), then annotate your sample in place:

```{r}
library(survey160r)
s160_gcs_init()   # one browser sign-in, cached
dataset <- disposition_pull()            # a few hundred MB first pull, then cached

my_sample <- data.frame(phone = c("2015550101", "2015550102"))   # your phone sample
cleaned <- disposition_screen(my_sample, dataset)

# keep numbers not already finished: not completed (SMS or web), and not a hard
# stop (declined, screened out, or an unsplit terminal)
kept <- subset(
  cleaned,
  !((n_completed > 0 | n_web_complete > 0) %in% TRUE |
      (n_terminated > 0 | n_refused > 0 | n_ineligible > 0) %in% TRUE)
)
```

`disposition_screen()` returns `my_sample` unchanged with the screening columns appended 1:1 (see [What you get back](#what-you-get-back)). The columns are cumulative **counts** across the number's campaigns, so `> 0` means "reached that status at least once". `n_terminated`, `n_refused` (declined), and `n_ineligible` (screened out) are the hard-stop signals; on the prod projection they are set independently -- a number can be flagged `refused` / `ineligible` without a bare `terminated`, and vice versa -- so screen on all three. The `%in% TRUE` guard keeps blank or unparseable phones (they screen to all-`NA`), so they surface for correction rather than vanishing.

## Screen on contact history

The counts say what a number ever did; `columns = "history"` says what its **next** send would carry, so the list-screening rule measured on `disposition_steps()` (below) applies to a sample unchanged. It appends `prior_completes`, `prior_replies`, `prior_unanswered`, the `history_segment` (`first_contact`, `completer_2_plus`, `completer_1`, `replied`, `unanswered_3_plus`, `unanswered_1_2`) and the recency columns `steps_since_reply` / `steps_since_complete` / `steps_since_undeliverable` (campaigns since the last one; `0` when it was the latest campaign) and `days_since_reply` / `days_since_complete` / `days_since_undeliverable` (counted to `as_of`, today by default), `NA` until the event has happened. On the 2026 production rows the segments complete at about 17%, 9%, 1.1%, 0.55% (first contact), 0.33% and 0.17% of sends, and a number the carrier rejected starts replying again only after about two years, so the usual rule is:

```{r}
cleaned <- disposition_screen(my_sample, dataset, columns = c("summary", "history"))

# skip numbers that never answered three or more campaigns, or were rejected by
# the carrier in the last two years; then send the completers first
keep <- subset(
  cleaned,
  !(history_segment == "unanswered_3_plus") %in% TRUE &
    !(days_since_undeliverable < 730) %in% TRUE
)
keep <- keep[order(keep$history_segment != "completer_2_plus",
                   keep$history_segment != "completer_1"), ]
```

The `campaign_ids` / `date_from` / `date_to` scope applies to the history too: a number with no campaign in scope is at its `first_contact`. A no-reply campaign whose delivery error is one of `undeliverable_codes` (the hard carrier rejections by default) counts as undeliverable; every other outcome but a plain non-response is a reply.

## Look up specific numbers

`disposition_summary()` runs the per-phone rollup as a standalone query; `disposition_records()` returns the raw stored rows, one per `(phone, campaign_id)`:

```{r}
disposition_summary(dataset, phones = c("2015550101", "2015550102"))
disposition_records(dataset, campaign_ids = 1234)
```

Both take `phones` / `campaign_ids` / `date_from` / `date_to` filters; `disposition_summary()` also takes `statuses =` to keep only phones whose latest campaign ended a given way (`statuses = c("refused", "ineligible", "terminated")` for every hard stop). Phones match digit-normalized: a leading US `1` is dropped, so an 11-digit number matches a stored 10-digit one. `disposition_records()` hands back the stored schema as-is, one row per `(phone, campaign_id)` -- the funnel flags, the recipient's carrier and any delivery error, the Tracker dimensions, and the disposition date. See [Stored record columns](#stored-record-columns).

## Screen several samples

With `duckdb` installed (see the README), pass the dataset path for each sample: `disposition_screen()` and a `phones`-scoped `disposition_summary()` / `disposition_records()` read only that sample's rows, in seconds and well under 1 GB of RAM. Without `duckdb`, each call loads the whole projection, so read once and summarize in memory per sample instead. `disposition_summary()` accepts either the dataset path or an already-read frame; on a frame it does no I/O, so this runs:

```{r summary_frame, eval = TRUE}
library(survey160r)
records <- data.frame(
  phone          = c("5551234567", "5551234567", "5559876543"),
  campaign_id    = c(101L, 102L, 101L),
  engaged        = c(1L, 1L, 0L), opted_in     = c(1L, 0L, 0L),
  completed      = c(1L, 0L, 0L), web_complete = c(0L, 0L, 0L),
  refused        = c(0L, 1L, 0L), ineligible   = c(0L, 0L, 0L),
  terminated     = c(0L, 1L, 0L),
  disposition_date = as.Date(c("2026-01-10", "2026-01-20", "2026-01-15"))
)
disposition_summary(records, phones = c("5551234567", "5550000000"))
```

Note `5551234567`: its **latest** disposition is `refused` (its most recent campaign, 102, ended in a refusal) while its **best** is `completed` (campaign 101 finished the survey). Without `duckdb`, read once and reuse: `records <- disposition_records(dataset)`, then `disposition_summary(records, phones = ...)` for each list.

## Roll up to funnel counts by dimension

To measure the funnel across recipients (rather than screen a list), `disposition_funnel()` groups the dataset by any dimension and returns the counts plus the standard funnel rates -- the disposition analogue of `campaign_metrics_summary()`. It takes the dataset path or an already-read frame; on a frame it does no I/O, so this runs:

```{r funnel_frame, eval = TRUE}
records <- data.frame(
  phone       = as.character(1:5),
  campaign_id = c(101L, 101L, 102L, 102L, 102L),
  sent        = c(1L, 1L, 1L, 1L, 1L),
  engaged     = c(1L, 1L, 0L, 1L, 0L),
  opted_in    = c(1L, 0L, 0L, 1L, 0L),
  completed   = c(1L, 0L, 0L, 0L, 0L),
  refused     = c(0L, 1L, 0L, 0L, 0L),
  ineligible  = c(0L, 0L, 1L, 0L, 0L),
  carrier     = c("AT&T", "AT&T", "Verizon", "Verizon", NA),
  disposition_date = as.Date("2026-01-10")
)
disposition_funnel(records, by = "carrier", grain = "all")
```

`by` takes any stored dimension -- `carrier`, `survey_mode`, `tracker_client`, `campaign_id`, ... -- or several at once. `grain = "day"` (the default) adds `disposition_date` for a daily series -- equivalently, include `disposition_date` in `by` with `grain = "all"`; `grain = "all"` on its own collapses the date axis. Each group carries the counts (`campaigns`, `n_sent` through `n_refused`) and, unless `rates = FALSE`, the shared `funnel_rates()`: `engagement_rate`, `opted_in_rate` (opt-ins / sent), `opted_in_engaged_rate` (opt-ins / engaged), and `completion_rate` -- proportions, or `0`-`100` with `percent = TRUE`. A carrier the uploaded list did not supply is its own `NA` group; an all-off-channel group's completes stay `NA`, not a false `0`.

Roll up the whole projection with `disposition_funnel(dataset, by = "carrier")` -- with `duckdb` installed the grouping runs inside the Parquet scan, so the full production projection rolls up in under a second and well under 1 GB of RAM -- or pre-filter with `disposition_records()` first (by campaign or date) to scope it.

## Follow numbers over time

The screen and the funnel collapse a number's campaigns into counts. To ask what happened *in order* -- no reply first, then engaged, then completed -- use `disposition_paths()` (one row per phone) and `disposition_steps()` (one row per campaign, with the history up to it). Both need `duckdb`, `DBI` and `dbplyr`, and return a lazy table that DuckDB computes: aggregate with `dplyr` verbs, then `collect()`.

```{r}
library(dplyr)
dataset <- disposition_pull()

# Numbers that did not answer, later engaged, then completed
disposition_paths(dataset, "non_response > engaged|opted_in > completed|web_complete") |>
  filter(matched) |>
  count()

# The commonest path shapes
disposition_paths(dataset) |> count(shape, sort = TRUE) |> head(20) |> collect()

# Reply rate on the next campaign after k non-responses in a row
disposition_steps(dataset, sample_frac = 0.1) |>
  filter(label == "non_response", !is.na(next_label)) |>
  group_by(run = pmin(run_length, 5L)) |>
  summarise(n = n(), reply_next = mean(as.integer(
    !next_label %in% c("non_response", "undeliverable")), na.rm = TRUE)) |>
  arrange(run) |>
  collect()

# Flag a sample's numbers whose last three contacts got no reply
flags <- disposition_paths(dataset, "non_response|undeliverable{3,}$",
                           phones = my_sample$phone) |>
  select(phone, path, matched) |>
  collect()
```

Each campaign gets one label, the furthest step it reached, ordered by `disposition_date` (a missing date counts as the oldest, as in the summary, so a phone's last step is its `latest_disposition`). A path spells them as letters: N non_response, X undeliverable (no reply and a hard carrier rejection), E engaged, O opted_in, T terminated, I ineligible, R refused, C completed, W web_complete -- `"NNEC"` is two non-responses, engaged, then completed, and its `shape` is `"N+E+C+"`. A pattern joins labels (or letters) with `>` ("later") or `>>` ("the very next campaign"), takes `|` alternatives, `+` / `{m,n}` repeats and `^` / `$` anchors; `?disposition_paths` has the full syntax.

- **Scope for speed.** A full-projection query takes 15-25 seconds; `sample_frac = 0.1` (the same 10% of numbers on every run) or `phones =` take seconds.
- **History segments.** `disposition_steps(dataset, columns = "history")` adds `prior_completes`, `prior_replies`, `prior_unanswered` and `history_segment` (`first_contact`, `completer_2_plus`, `completer_1`, `replied`, `unanswered_3_plus`, `unanswered_1_2`): group a year's rows by it to see where sends go against where completes come from, the basis of a list-screening rule. The same group carries the recency columns: `steps_since_reply` / `steps_since_complete` / `steps_since_undeliverable` (campaigns since the last one before this step; `0` when it was the previous campaign) and `days_since_reply` / `days_since_complete` / `days_since_undeliverable`, all `NA` until the event has happened. On the 2026 production rows campaigns, not days, are what predicts: a number completes 20% of the time when its previous campaign was a complete, 13% / 10% / 8% / 5% after one / two / three / four unanswered campaigns in between and 3% after five, whatever the calendar gap; the calendar matters for undeliverables, which start replying again after about two years.
- **History is never cut short.** On `disposition_steps()`, `campaign_ids` / `date_from` / `date_to` drop rows *after* the history is computed, so a 2026 row still counts its 2025 non-responses. On `disposition_paths()` the dates instead choose which campaigns form the path.
- **Undeliverable is separate from non-response.** After a hard rejection (codes 4700, 4701, 4702, 4720, 4750, 4753) the next campaign gets a reply about 0.7% of the time, against 5.9% after a plain non-response. `undeliverable_codes = NULL` folds them back, as the summary does.
- **Text-to-web external hides completes.** `completion_known` (steps) and `n_unknown_completion` (paths) mark campaigns where a complete is never reported, so a `completed` step cannot match there.
- **August 2026 terminals.** Since August 2026 most engaged recipients who did not finish are stored as an unsplit `terminated`. `terminal = "engaged"` labels those by the step they reached instead.
- **Memory.** DuckDB works within 4 GB and spills to disk past it (up to ~20 GB on a full-projection query, freed when it finishes). Change it with `options(survey160r.duckdb_memory = "8GB")` or `options(survey160r.duckdb_temp_dir = "/path")`.

## What you get back

`disposition_screen()` and `disposition_summary()` return one row per phone:

| Column | Meaning |
|---|---|
| `phone` | The digit-normalized number. |
| `n_campaigns` / `campaigns` | How many campaigns contacted it (`0` = never contacted), and the comma-separated id list. |
| `n_engaged` | Campaigns where it replied. |
| `n_opted_in` | Campaigns where it consented. |
| `n_completed` / `n_web_complete` | Campaigns it completed on SMS (reached the close) / on the web. |
| `n_terminated` | Campaigns the producer flagged as a hard stop (the `terminated` flag). |
| `n_refused` / `n_ineligible` | Campaigns where the terminal was a decline / a screen-out (name-classified). On the prod projection these are set independently of `n_terminated` and need not sum to it -- so screen on all three. |
| `n_error` | Campaigns that carried a carrier delivery-error code. |
| `latest_disposition` / `latest_campaign_id` | Outcome of the **most-recent** campaign, and that campaign's id. |
| `best_disposition` / `best_campaign_id` | **Furthest-reached** outcome across all its campaigns, and the id that reached it. |
| `first_disposition_date` / `last_disposition_date` | Earliest and latest disposition date across its campaigns. |

The counts overlap rather than partition `n_campaigns` -- a completed campaign is also engaged -- so read each as "reached status X in this many campaigns".

`disposition_screen(columns = "history")` appends the next-contact history block instead (or as well, with `c("summary", "history")`): `prior_completes`, `prior_replies`, `prior_unanswered`, `history_segment`, `steps_since_reply`, `steps_since_complete`, `steps_since_undeliverable`, `days_since_reply`, `days_since_complete`, `days_since_undeliverable` -- the columns `disposition_steps(columns = "history")` carries per step, with the same names and definitions, as of the campaign that would come next (see [Screen on contact history](#screen-on-contact-history)). A never-contacted number has counts `0`, segment `first_contact` and every recency column `NA`.

`latest_disposition` and `best_disposition`, least to most advanced: `never_contacted` -> `non_response` -> `engaged` -> `opted_in` -> `terminated` -> `ineligible` -> `refused` -> `completed` -> `web_complete`. The terminal band (`terminated` < `ineligible` < `refused`) sits between `opted_in` and `completed`: `terminated` is the unsplit residual, `ineligible` a screen-out, `refused` a decline. `disposition_records()` returns the stored per-campaign columns instead; see [Stored record columns](#stored-record-columns).

### Stored record columns

`disposition_records()` returns the dataset as stored -- one row per `(phone, campaign_id)`, contacted-only -- which `disposition_summary()` and `disposition_screen()` roll up per phone (above). The columns, in funnel order:

| Column | Meaning |
|---|---|
| `phone` / `campaign_id` | The row key: the digit-normalized number and the campaign that contacted it. |
| `sent` | Always `1` -- the dataset holds only contacted numbers. |
| `engaged` | Replied at all. |
| `opted_in` | Consented to take the survey. |
| `completed` | Reached the survey close in-channel. `NA` on text-to-web external (no completion callback). |
| `web_complete` | Completed on the web. Equals `completed` for a `t2w` campaign; `0` for `sms` and `t2w_external`. |
| `terminated` | Hard-stop flag. On the prod projection it is set independently of the split below -- a row can be `terminated` with neither `refused` nor `ineligible`, and either can be set without `terminated` -- so screen on all three. |
| `refused` / `ineligible` | The name-classified terminal split -- a decline / a screen-out. |
| `error` | The raw carrier delivery-error code for the send; `NA` when the export carries no usable code. |
| `carrier` | The recipient's mobile network, from the export's optional `misc` column; `NA` when that column is omitted or blank. |
| `survey_mode` | Detected send channel: `sms`, `t2w`, or `t2w_external`. |
| `tracker_loi` / `tracker_topic` | Length of interview in minutes, and the campaign topic, from the Project Tracker. |
| `tracker_registration_id` | The campaign's 10DLC registration id, from the Project Tracker. |
| `disposition_date` | The number's last-send date, or `NA` when no send survives archival. |

The funnel flags overlap rather than partition -- a `completed` row is also `engaged` -- so read each as a status the row reached, not an exclusive bucket. Provenance of the columns: `sent`, the funnel flags, `survey_mode`, `error`, and `carrier` are derived from the campaign export; the `tracker_*` dimensions (`tracker_loi`, `tracker_topic`, `tracker_registration_id`, and the rest of the Project-Tracker set -- see `?disposition_records`) come from the Tracker, so they are populated only for campaigns it covers and read `NA` on older campaigns that predate it. An un-enriched projection (published before a column existed, or without Tracker enrichment) simply omits that column, and `disposition_records()` returns whatever subset is present -- see `?disposition_records`.

## Notes

- **Contacted-only.** A loaded number that was never texted is not in the dataset; it screens as `never_contacted`.
- **Text-to-web (external) campaigns** have no completion callback, so `completed` is `NA` -- never a false positive, but an external completer reads as `n_completed = 0` and is kept (a false negative you cannot screen out here).
- **`disposition_date` and `error` are populated.** The live producer stamps each phone's last-send date and any carrier delivery-error code, so the `date_from` / `date_to` filters and `n_error` work (prod spans April 2021 to the last pipeline pass). A row with no surviving send has an `NA` `disposition_date`, which any date bound drops.
- **Terminal split rollout.** `refused` / `ineligible` are set for the terminals the routing could name-classify; a small residual stays `terminated`. A projection produced before the split defaults them to `0`, so those rows read as plain `terminated`.
- **Screen on the counts, not on `latest_disposition`.** `latest_disposition` / `best_disposition` are informational; a number whose most-recent campaign only reached `engaged` can still have completed an earlier one, so filter on the `n_*` counts (which see every campaign), not the latest label.
- **Scale.** The projection is over a hundred million rows (about 700 MB on disk). With `duckdb` installed, a screen reads only your sample's rows, a `campaign_ids` / `date_from` / `date_to` filter reads only the matching rows, and `disposition_funnel()` groups inside the scan -- seconds and a GB or two of RAM at most, so all of it runs on a laptop. Without it, every read loads the whole projection (tens of GB), so install `duckdb` or read once and summarize per sample in memory (see [Screen several samples](#screen-several-samples)). Set `options(survey160r.disposition_engine = "nanoparquet")` to force the fallback.
- **Stale cache.** The dataset is re-published periodically; `disposition_pull(refresh = TRUE)` forces a fresh download.
- **`env`.** `disposition_pull()` reads `"prod"` (default) or `"dev"` -- there is no `staging` tier (staging sends fold into the prod bucket), so `env = "staging"` errors; use `env = "prod"` for staging data. Run `s160_datasets()` to see every dataset's environments.

## See also

`?disposition_screen`, `?disposition_summary`, `?disposition_records`, `?disposition_funnel`, `?disposition_steps`, `?disposition_paths`, `?disposition_pull`. For the funnel rates `disposition_funnel()` shares with the latency side, `?funnel_rates` and `?campaign_metrics_summary`. To build the dataset from a raw campaign CSV (the producer side), `?disposition_run` and `?disposition_input_columns`; `?survey160r` lists every function by task. To screen the same sample against the opt-out list, `vignette("opt-out")`. For latency reports, `vignette("latency")`; for the fleet funnel by registration or brand, `vignette("campaign-metrics")`.