Disposition screening

Screen a sample

Pull the dataset once (cached), then annotate your sample in place:

library(survey160r)
s160_gcs_init()   # one browser sign-in, cached
dataset <- disposition_pull()            # a few hundred MB first pull, then cached

my_sample <- data.frame(phone = c("2015550101", "2015550102"))   # your phone sample
cleaned <- disposition_screen(my_sample, dataset)

# keep numbers not already finished: not completed (SMS or web), and not a hard
# stop (declined, screened out, or an unsplit terminal)
kept <- subset(
  cleaned,
  !((n_completed > 0 | n_web_complete > 0) %in% TRUE |
      (n_terminated > 0 | n_refused > 0 | n_ineligible > 0) %in% TRUE)
)

disposition_screen() returns my_sample unchanged with the screening columns appended 1:1 (see What you get back). The columns are cumulative counts across the number’s campaigns, so > 0 means “reached that status at least once”. n_terminated, n_refused (declined), and n_ineligible (screened out) are the hard-stop signals; on the prod projection they are set independently – a number can be flagged refused / ineligible without a bare terminated, and vice versa – so screen on all three. The %in% TRUE guard keeps blank or unparseable phones (they screen to all-NA), so they surface for correction rather than vanishing.

Screen on contact history

The counts say what a number ever did; columns = "history" says what its next send would carry, so the list-screening rule measured on disposition_steps() (below) applies to a sample unchanged. It appends prior_completes, prior_replies, prior_unanswered, the history_segment (first_contact, completer_2_plus, completer_1, replied, unanswered_3_plus, unanswered_1_2) and the recency columns steps_since_reply / steps_since_complete / steps_since_undeliverable (campaigns since the last one; 0 when it was the latest campaign) and days_since_reply / days_since_complete / days_since_undeliverable (counted to as_of, today by default), NA until the event has happened. On the 2026 production rows the segments complete at about 17%, 9%, 1.1%, 0.55% (first contact), 0.33% and 0.17% of sends, and a number the carrier rejected starts replying again only after about two years, so the usual rule is:

cleaned <- disposition_screen(my_sample, dataset, columns = c("summary", "history"))

# skip numbers that never answered three or more campaigns, or were rejected by
# the carrier in the last two years; then send the completers first
keep <- subset(
  cleaned,
  !(history_segment == "unanswered_3_plus") %in% TRUE &
    !(days_since_undeliverable < 730) %in% TRUE
)
keep <- keep[order(keep$history_segment != "completer_2_plus",
                   keep$history_segment != "completer_1"), ]

The campaign_ids / date_from / date_to scope applies to the history too: a number with no campaign in scope is at its first_contact. A no-reply campaign whose delivery error is one of undeliverable_codes (the hard carrier rejections by default) counts as undeliverable; every other outcome but a plain non-response is a reply.

Look up specific numbers

disposition_summary() runs the per-phone rollup as a standalone query; disposition_records() returns the raw stored rows, one per (phone, campaign_id):

disposition_summary(dataset, phones = c("2015550101", "2015550102"))
disposition_records(dataset, campaign_ids = 1234)

Both take phones / campaign_ids / date_from / date_to filters; disposition_summary() also takes statuses = to keep only phones whose latest campaign ended a given way (statuses = c("refused", "ineligible", "terminated") for every hard stop). Phones match digit-normalized: a leading US 1 is dropped, so an 11-digit number matches a stored 10-digit one. disposition_records() hands back the stored schema as-is, one row per (phone, campaign_id) – the funnel flags, the recipient’s carrier and any delivery error, the Tracker dimensions, and the disposition date. See Stored record columns.

Screen several samples

With duckdb installed (see the README), pass the dataset path for each sample: disposition_screen() and a phones-scoped disposition_summary() / disposition_records() read only that sample’s rows, in seconds and well under 1 GB of RAM. Without duckdb, each call loads the whole projection, so read once and summarize in memory per sample instead. disposition_summary() accepts either the dataset path or an already-read frame; on a frame it does no I/O, so this runs:

library(survey160r)
records <- data.frame(
  phone          = c("5551234567", "5551234567", "5559876543"),
  campaign_id    = c(101L, 102L, 101L),
  engaged        = c(1L, 1L, 0L), opted_in     = c(1L, 0L, 0L),
  completed      = c(1L, 0L, 0L), web_complete = c(0L, 0L, 0L),
  refused        = c(0L, 1L, 0L), ineligible   = c(0L, 0L, 0L),
  terminated     = c(0L, 1L, 0L),
  disposition_date = as.Date(c("2026-01-10", "2026-01-20", "2026-01-15"))
)
disposition_summary(records, phones = c("5551234567", "5550000000"))
#>        phone n_campaigns campaigns n_engaged n_opted_in n_completed
#> 1 5550000000           0      <NA>         0          0           0
#> 2 5551234567           2   101,102         2          1           1
#>   n_web_complete n_terminated n_refused n_ineligible n_error latest_disposition
#> 1              0            0         0            0       0    never_contacted
#> 2              0            1         1            0       0            refused
#>   latest_campaign_id best_disposition best_campaign_id first_disposition_date
#> 1               <NA>  never_contacted             <NA>                   <NA>
#> 2                102        completed              101             2026-01-10
#>   last_disposition_date
#> 1                  <NA>
#> 2            2026-01-20

Note 5551234567: its latest disposition is refused (its most recent campaign, 102, ended in a refusal) while its best is completed (campaign 101 finished the survey). Without duckdb, read once and reuse: records <- disposition_records(dataset), then disposition_summary(records, phones = ...) for each list.

Roll up to funnel counts by dimension

To measure the funnel across recipients (rather than screen a list), disposition_funnel() groups the dataset by any dimension and returns the counts plus the standard funnel rates – the disposition analogue of campaign_metrics_summary(). It takes the dataset path or an already-read frame; on a frame it does no I/O, so this runs:

records <- data.frame(
  phone       = as.character(1:5),
  campaign_id = c(101L, 101L, 102L, 102L, 102L),
  sent        = c(1L, 1L, 1L, 1L, 1L),
  engaged     = c(1L, 1L, 0L, 1L, 0L),
  opted_in    = c(1L, 0L, 0L, 1L, 0L),
  completed   = c(1L, 0L, 0L, 0L, 0L),
  refused     = c(0L, 1L, 0L, 0L, 0L),
  ineligible  = c(0L, 0L, 1L, 0L, 0L),
  carrier     = c("AT&T", "AT&T", "Verizon", "Verizon", NA),
  disposition_date = as.Date("2026-01-10")
)
disposition_funnel(records, by = "carrier", grain = "all")
#>   carrier campaigns n_sent n_engaged n_opted_in n_completed n_ineligible
#> 1    AT&T         1      2         2          1           1            0
#> 2 Verizon         1      2         1          1           0            1
#> 3    <NA>         1      1         0          0           0            0
#>   n_refused engagement_rate opted_in_rate opted_in_engaged_rate completion_rate
#> 1         1             1.0           0.5                   0.5             0.5
#> 2         0             0.5           0.5                   1.0             0.0
#> 3         0             0.0           0.0                    NA             0.0

by takes any stored dimension – carrier, survey_mode, tracker_client, campaign_id, … – or several at once. grain = "day" (the default) adds disposition_date for a daily series – equivalently, include disposition_date in by with grain = "all"; grain = "all" on its own collapses the date axis. Each group carries the counts (campaigns, n_sent through n_refused) and, unless rates = FALSE, the shared funnel_rates(): engagement_rate, opted_in_rate (opt-ins / sent), opted_in_engaged_rate (opt-ins / engaged), and completion_rate – proportions, or 0-100 with percent = TRUE. A carrier the uploaded list did not supply is its own NA group; an all-off-channel group’s completes stay NA, not a false 0.

Roll up the whole projection with disposition_funnel(dataset, by = "carrier") – with duckdb installed the grouping runs inside the Parquet scan, so the full production projection rolls up in under a second and well under 1 GB of RAM – or pre-filter with disposition_records() first (by campaign or date) to scope it.

Follow numbers over time

The screen and the funnel collapse a number’s campaigns into counts. To ask what happened in order – no reply first, then engaged, then completed – use disposition_paths() (one row per phone) and disposition_steps() (one row per campaign, with the history up to it). Both need duckdb, DBI and dbplyr, and return a lazy table that DuckDB computes: aggregate with dplyr verbs, then collect().

library(dplyr)
dataset <- disposition_pull()

# Numbers that did not answer, later engaged, then completed
disposition_paths(dataset, "non_response > engaged|opted_in > completed|web_complete") |>
  filter(matched) |>
  count()

# The commonest path shapes
disposition_paths(dataset) |> count(shape, sort = TRUE) |> head(20) |> collect()

# Reply rate on the next campaign after k non-responses in a row
disposition_steps(dataset, sample_frac = 0.1) |>
  filter(label == "non_response", !is.na(next_label)) |>
  group_by(run = pmin(run_length, 5L)) |>
  summarise(n = n(), reply_next = mean(as.integer(
    !next_label %in% c("non_response", "undeliverable")), na.rm = TRUE)) |>
  arrange(run) |>
  collect()

# Flag a sample's numbers whose last three contacts got no reply
flags <- disposition_paths(dataset, "non_response|undeliverable{3,}$",
                           phones = my_sample$phone) |>
  select(phone, path, matched) |>
  collect()

Each campaign gets one label, the furthest step it reached, ordered by disposition_date (a missing date counts as the oldest, as in the summary, so a phone’s last step is its latest_disposition). A path spells them as letters: N non_response, X undeliverable (no reply and a hard carrier rejection), E engaged, O opted_in, T terminated, I ineligible, R refused, C completed, W web_complete – "NNEC" is two non-responses, engaged, then completed, and its shape is "N+E+C+". A pattern joins labels (or letters) with > (“later”) or >> (“the very next campaign”), takes | alternatives, + / {m,n} repeats and ^ / $ anchors; ?disposition_paths has the full syntax.

• Scope for speed. A full-projection query takes 15-25 seconds; sample_frac = 0.1 (the same 10% of numbers on every run) or phones = take seconds.
• History segments. disposition_steps(dataset, columns = "history") adds prior_completes, prior_replies, prior_unanswered and history_segment (first_contact, completer_2_plus, completer_1, replied, unanswered_3_plus, unanswered_1_2): group a year’s rows by it to see where sends go against where completes come from, the basis of a list-screening rule. The same group carries the recency columns: steps_since_reply / steps_since_complete / steps_since_undeliverable (campaigns since the last one before this step; 0 when it was the previous campaign) and days_since_reply / days_since_complete / days_since_undeliverable, all NA until the event has happened. On the 2026 production rows campaigns, not days, are what predicts: a number completes 20% of the time when its previous campaign was a complete, 13% / 10% / 8% / 5% after one / two / three / four unanswered campaigns in between and 3% after five, whatever the calendar gap; the calendar matters for undeliverables, which start replying again after about two years.
• History is never cut short. On disposition_steps(), campaign_ids / date_from / date_to drop rows after the history is computed, so a 2026 row still counts its 2025 non-responses. On disposition_paths() the dates instead choose which campaigns form the path.
• Undeliverable is separate from non-response. After a hard rejection (codes 4700, 4701, 4702, 4720, 4750, 4753) the next campaign gets a reply about 0.7% of the time, against 5.9% after a plain non-response. undeliverable_codes = NULL folds them back, as the summary does.
• Text-to-web external hides completes. completion_known (steps) and n_unknown_completion (paths) mark campaigns where a complete is never reported, so a completed step cannot match there.
• August 2026 terminals. Since August 2026 most engaged recipients who did not finish are stored as an unsplit terminated. terminal = "engaged" labels those by the step they reached instead.
• Memory. DuckDB works within 4 GB and spills to disk past it (up to ~20 GB on a full-projection query, freed when it finishes). Change it with options(survey160r.duckdb_memory = "8GB") or options(survey160r.duckdb_temp_dir = "/path").

What you get back

disposition_screen() and disposition_summary() return one row per phone:

The counts overlap rather than partition n_campaigns – a completed campaign is also engaged – so read each as “reached status X in this many campaigns”.

disposition_screen(columns = "history") appends the next-contact history block instead (or as well, with c("summary", "history")): prior_completes, prior_replies, prior_unanswered, history_segment, steps_since_reply, steps_since_complete, steps_since_undeliverable, days_since_reply, days_since_complete, days_since_undeliverable – the columns disposition_steps(columns = "history") carries per step, with the same names and definitions, as of the campaign that would come next (see Screen on contact history). A never-contacted number has counts 0, segment first_contact and every recency column NA.

latest_disposition and best_disposition, least to most advanced: never_contacted -> non_response -> engaged -> opted_in -> terminated -> ineligible -> refused -> completed -> web_complete. The terminal band (terminated < ineligible < refused) sits between opted_in and completed: terminated is the unsplit residual, ineligible a screen-out, refused a decline. disposition_records() returns the stored per-campaign columns instead; see Stored record columns.

Notes

• Contacted-only. A loaded number that was never texted is not in the dataset; it screens as never_contacted.
• Text-to-web (external) campaigns have no completion callback, so completed is NA – never a false positive, but an external completer reads as n_completed = 0 and is kept (a false negative you cannot screen out here).
• disposition_date and error are populated. The live producer stamps each phone’s last-send date and any carrier delivery-error code, so the date_from / date_to filters and n_error work (prod spans April 2021 to the last pipeline pass). A row with no surviving send has an NA disposition_date, which any date bound drops.
• Terminal split rollout. refused / ineligible are set for the terminals the routing could name-classify; a small residual stays terminated. A projection produced before the split defaults them to 0, so those rows read as plain terminated.
• Screen on the counts, not on latest_disposition. latest_disposition / best_disposition are informational; a number whose most-recent campaign only reached engaged can still have completed an earlier one, so filter on the n_* counts (which see every campaign), not the latest label.
• Scale. The projection is over a hundred million rows (about 700 MB on disk). With duckdb installed, a screen reads only your sample’s rows, a campaign_ids / date_from / date_to filter reads only the matching rows, and disposition_funnel() groups inside the scan – seconds and a GB or two of RAM at most, so all of it runs on a laptop. Without it, every read loads the whole projection (tens of GB), so install duckdb or read once and summarize per sample in memory (see Screen several samples). Set options(survey160r.disposition_engine = "nanoparquet") to force the fallback.
• Stale cache. The dataset is re-published periodically; disposition_pull(refresh = TRUE) forces a fresh download.
• env. disposition_pull() reads "prod" (default) or "dev" – there is no staging tier (staging sends fold into the prod bucket), so env = "staging" errors; use env = "prod" for staging data. Run s160_datasets() to see every dataset’s environments.

See also

?disposition_screen, ?disposition_summary, ?disposition_records, ?disposition_funnel, ?disposition_steps, ?disposition_paths, ?disposition_pull. For the funnel rates disposition_funnel() shares with the latency side, ?funnel_rates and ?campaign_metrics_summary. To build the dataset from a raw campaign CSV (the producer side), ?disposition_run and ?disposition_input_columns; ?survey160r lists every function by task. To screen the same sample against the opt-out list, vignette("opt-out"). For latency reports, vignette("latency"); for the fleet funnel by registration or brand, vignette("campaign-metrics").