disposition_screen(columns = "history") (or c("summary", "history"))
appends what each number's next send would carry: prior_completes,
prior_replies, prior_unanswered, history_segment and the recency
columns steps_since_* / days_since_* -- the columns
disposition_steps(columns = "history") carries per step, with the same
names and definitions, computed per phone as of the campaign that would come
next, so a rule measured on the steps table (drop unanswered_3_plus and a
carrier rejection within two years, send the completers first) applies to a
sample unchanged. as_of (default today) sets the day the days_since_*
columns count to; undeliverable_codes names the delivery errors that make a
no-reply campaign undeliverable, as on disposition_steps(). The
campaign_ids / date_from / date_to scope applies to the history too, and
a never-contacted number is at its first_contact. Computed in R over the
sample's own rows (the scoped read), so it runs on either engine and adds
well under a second per million phones.campaign_id (a dictionary-encoded column, which the nanoparquet
read keeps as a factor) now orders a phone's campaigns by its labels, so
disposition_summary() / disposition_screen() pick the same latest campaign
on a date tie as the sequence readers; before, the factor's level codes
decided (0.78.1).disposition_steps(columns = "history") adds six recency columns next to the
prior_* counts and history_segment: steps_since_reply,
steps_since_complete, steps_since_undeliverable (campaigns since the last
such step before this one; 0 when it was the previous campaign) and
days_since_reply, days_since_complete, days_since_undeliverable
(disposition_date minus that step's); NA until the event has happened.
On the 2026 production rows campaigns, not days, are what predicts the next
outcome (a number completes 20% of the time when its previous campaign was a
complete, 13% / 10% / 8% / 5% after one / two / three / four unanswered
campaigns in between, 3% after five), while the calendar matters mainly for
undeliverables, which start replying again after about two years. The history group now builds the letter path for the
recency columns, which adds about four seconds on the full production projection
(10 s to 14 s for the 2026 rows); a second window layer keyed on step
was tried and cost twice that, the sort being the whole price.steps_since_* / days_since_* columns; no
column or behaviour changed.Twenty rounds over the README, the articles and the function help, written for a Survey Manager who scripts against the exports. No behaviour changed.
DBI / dbplyr for the sequence readers; First-time setup names the
analytics bucket.campaign_metrics_pull() points at campaign_metrics_summary() /
campaign_metrics_records() instead of "your Parquet reader of choice" (the
raw file multi-counts); s160_datasets() documents the campaign_metrics
dataset; the campaign-metrics article is listed on the package site.?s160_gcs_init and ?s160_api_auth gain a "Before
you start" section (what to get from your team lead, the bucket grant behind a
403, a browser that does not open, starting over); every article's intro
points at it, so a vignette() reader is not sent to the README.?disposition_screen and ?opt_out_screen screen a
sample against a tiny temp Parquet, so the appended columns and the keep
filter run from the help page; the *_pull() call stays in \dontrun{}.duckdb / DBI / dbplyr, memory ballooning without
duckdb, a stale cache hit) and a recipe to list and clear the dataset cache;
?survey160r gains a "Local cache" section.latency_run() / the latency article call out that field_timezone defaults
to "UTC"; the disposition article names the statuses / date filters on
the lookups; disposition_pull() says what its path is ready for and what the
first pull costs; writing-style.md records the README and runnable-example
conventions.Outputs are identical to the previous version (checked on synthetic campaigns and on the production projection, ~165M rows); only the work behind them changed.
disposition_summary(path, phones = ), disposition_screen(), disposition_records(path, phones = ))
reads the matching rows in one scan: when every requested number is ten
digits -- the usual screen -- the stored value is matched as-is and only
other-length values are normalized, in the same scan (1.8 s -> 0.8 s for
100k numbers on prod). disposition_steps() / disposition_paths() scope
their phones the same way.campaign_ids on a Parquet path compares the stored INTEGER id as an
integer (when the id is written as one), so DuckDB skips the row groups of
other campaigns: one campaign's records in 0.3 s instead of 1 s on prod.disposition_run() decodes each id.<q>.scriptDate column once (the funnel,
terminal and date logic each parsed it again): ~2x faster on a 200k-row,
12-question export.disposition_summary() on a frame) groups phones from
one radix sort and reads the date span off the ordering it already has
(two fewer passes over the rows); the final sort is skipped when the
rollup already comes out in phone order.latency_report() groups the segment cells by the integer segment index
rather than the label, counts NA reasons in one pass, and computes the cell
quantiles with the type-7 arithmetic directly; na_if_blank() tests string
length instead of comparing strings.disposition_steps() gains a "history" column group: prior_completes,
prior_replies and prior_unanswered (what the number did before this
campaign) and history_segment -- first_contact, completer_2_plus,
completer_1, replied, unanswered_3_plus, unanswered_1_2 -- the
list-screening segments of the 2026 completion analysis
(survey160-notebooks, disposition/2026-10-23-disposition-paths-completion),
so notebooks and dashboards share one definition. Not in the default
columns; it shares the one window sort, so it costs about the same as
"so_far".Two new readers follow each number's campaigns in time order -- "did not
answer, later engaged, then completed" -- which the per-phone counts of
disposition_summary() / disposition_screen() cannot express.
disposition_paths(): one row per phone, its campaigns as a letter path
("NNEC") and run-length shape ("N+E+C+"), plus an optional sequence
pattern ("non_response > engaged|opted_in > completed|web_complete",
"non_response|undeliverable{3,}$") with matched, match positions and
dates. > means later, >> the next campaign; labels or their letters,
|, +, {m,n}, ^ / $.disposition_steps(): one row per (phone, campaign) with the history up
to it -- step / n_campaigns, prev_label / next_label, the gaps in
days, running counts, run_length, path_so_far -- for questions such as
the reply rate after k non-responses in a row.dbplyr table that DuckDB computes, on one
session-wide connection with a 4 GB memory limit that spills to disk past
it (options(survey160r.duckdb_memory, survey160r.duckdb_temp_dir)); each
session spills into its own subdirectory, removed when the session ends. On
the production projection (~165M rows) a full-scan aggregate takes 15-25 s
at 5.5-7.5 GB of process memory; phones = or sample_frac = (a
reproducible md5 sample of numbers) take seconds. They need duckdb, DBI
and dbplyr (dbplyr is a new Suggests); an interactive session is offered
the install.disposition_summary(): by disposition_date, then
campaign id as a number, a missing date or id counting as the oldest -- so
a phone's last step is its latest_disposition.disposition_summary()'s precedence, plus
undeliverable: no reply with a hard carrier rejection (4700, 4701, 4702,
4720, 4750, 4753 by default, undeliverable_codes =). terminal = "engaged" relabels unsplit terminals, which dominate the data since August
2026. completion_known / n_unknown_completion mark text-to-web external
campaigns, whose completes are never reported.Twenty review rounds over every error, warning and message the package emits, so a failure says which function failed, what it received, and what to do next. Message wording changed throughout; code that matched error text should dispatch on the new condition classes instead.
s160_error; the
specific ones add s160_not_initialized (sign in first), s160_not_found
(campaign, export file, local file or dataset), s160_http_error (the API
answered a request with an error; $status holds the HTTP code),
s160_auth_error (the sign-in itself was refused; also carries $status) and
s160_config_error (a latency config, or the export it is checked against,
fails validation). ?survey160r has a "Errors and messages" section with a
tryCatch() example.`timeout` must be a positive number, not -1., `destdir` must be a single directory path, not a character vector of length 2., `timestamps` must be "character" or "POSIXct", not "posix".`sample` has 3 column(s): id, cell, name), and question_timestamps() lists the
questions the export does carry. `filters.population` names the
column(s) it references that data lacks, echoes the expression, and
reports a non-logical result with its value.out of order at intro -> q1 (12.0% of 50 rows)). A non-list config is rejected up front.s160_gcs_campaign_results_files() / s160_api_campaign_results(); a
missing projection at the matching *_pull(); a relative local path names
the working directory it was resolved against.S160_API_USERID, S160_PROD_API_KEY ...) and the URL; a missing
variable in a non-interactive session says what to add to ~/.Renviron;
"not initialized" errors say the next step. API errors carry the HTTP status.gs://.../744_raw_data_download.csv),
progress against timeout, and on timeout how to pick the export up later
(s160_gcs_campaign_results_status()).No files found for campaign 9999 in gs://campaign_results/ (prod).), the
cache hit says how to refresh, and the "no requested column matched" warning
shows what was requested against the header.statuses, the disposition engine option
(with the packages to install), an unknown dataset, and an unaccepted CSV
reader argument names the reader (data.table::fread) to look up.campaign_metrics_summary() / funnel_rates() / disposition_summary()
paging errors carry the function prefix like the rest of the package.campaign_metrics_summary() counted every send twice on the published
projection. campaign_all.parquet carries two grains per (campaign_id, date) -- hour cells and a day-total cell (hour_local = NA) with the same
sends -- and the summary summed both, so every n_* count was doubled (the
rates were unaffected: numerator and denominator doubled alike). It now
counts each (campaign_id, date) once: its day total when present, else its
hour cells. ?campaign_metrics_records documents the two grains for callers
who sum the cells themselves.New article, vignette("campaign-metrics"): summarize the campaign-metrics
funnel by dimension, work with the cells directly, and what you get back.
The package help page (?survey160r) is now listed in the manual's help
index (it was marked internal) and groups every exported function by task:
first-time setup, campaign exports, latency, disposition, opt-out, campaign
metrics, and utilities. A test keeps it complete.
Help-page titles follow one style -- an imperative verb, "Cloud Storage" rather than "GCS", "and" rather than "+", and no parenthetical asides -- so the alphabetical help index reads as a list of tasks.
The package Description covers the whole surface (disposition, opt-out and
campaign metrics), not only latency.
The articles share one structure: task sections, then "What you get back", "Notes" and "See also", each linking the others.
The disposition docs (?disposition_pull, ?disposition_summary) no
longer call disposition_all.parquet phone-sorted: it is a consolidated
concat of the per-campaign files, so a phone lookup scans the whole file.
disposition_run() no longer counts a literal "NA" in an export's
error_code column as a delivery-error code. Some historical exports write
"NA" (quoted) for "no error", and the CSV reader keeps a quoted "NA" as
text in a character column, so those records reported error = "NA" and
inflated n_error. It now maps to NA, like a blank or "None".helper-raw-export.R) and a committed
fixture (fixtures/raw_export.csv) reproduce the byte-level shape of the
historical export corpus: quoted fields, duplicated phone / userid
headers, CRLF files, multilingual text with embedded line breaks, numeric
carrier codes in status, "None" / "NA" tokens, and trimmed fractional
seconds. New end-to-end tests run the readers, latency_run() and
disposition_run() on it.The Parquet-path readers now push their work into the DuckDB scan (when duckdb is installed), so only the result reaches R. Outputs are identical to the previous version (checked on the production projection, ~141M rows):
disposition_funnel(path) groups and counts inside the scan: 12 s / ~16 GB
peak on the production projection -> 0.6 s / ~0.1 GB. It previously could not
run on a 16 GB laptop. An in-memory frame, the nanoparquet engine, or an
unusual file still groups in R.disposition_records(), disposition_summary() and disposition_screen()
apply campaign_ids / date_from / date_to inside the scan as well as the
phone match. One campaign's records went from 189 s / 46 GB to 3 s / 1.6 GB
(disposition_summary(): 115 s / 22 GB -> 3.3 s / 0.8 GB). The R filter still
runs on the returned rows, so the pre-filter only ever drops rows it would.opt_out_screen() narrows the opt-out list to the screened numbers before
de-duplicating it (5M-row list: 3.7 s -> 2.2 s).source_csv_hash provenance digest uses the system sha256sum when it is
on the PATH (~6x faster than hashing in R, same digest), so
s160_read_csv(hash = TRUE) costs little more than the read.latency_report() / latency_run(), disposition_run(), the disposition
readers and utils_fix_double_utf8() were profiled end to end on a 16 GB
laptop budget and tuned in thirty output-identical rounds (every change is
gated on byte-identical results). On a 200k-respondent, 12-question export
latency_run() goes from ~46 s to ~3 s (peak heap 2.1 GB to 0.8 GB) and
disposition_run() from ~6 s to ~0.35 s; a 1M-respondent compact run from
~140 s / 6.5 GB RSS to ~13 s / 3.6 GB:
unique()
on the key columns instead of unique.data.frame() on the whole long frame
(which pasted every row to a string: ~30% of latency_report() wall time and
the single largest transient allocation). The long frame is also converted to
a data.table in place (setDT) rather than deep-copied once per grain.quantile() call, the n_na_* counts) once, and the per-threshold
n_le / n_resp_over as wide columns fanned out afterwards -- instead of
grouping the long frame once per threshold and recomputing the quantiles
four times.parse_campaign_timestamps() parses through lubridate's single-order
parse_date_time2() for the export's Y-m-d H:M:OS format (same C parser,
bit-identical instants, ~70x faster) and only retries the strings that fail
through the lenient multi-order parser. Side effect, and a fix: a non-standard
but parseable string (an ISO T separator, a compact YmdHMS) now parses
the same whether it sits alone or inside a column of standard timestamps --
the multi-order parser's per-call order training made that data-dependent.
parse_timestamps() no longer pre-strips the Z (the parser accepts it) and
returns its columns with an explicit tzone = "UTC".respondent_summary) is a
data.table GForce max over the valid segments instead of a dplyr
group_by(respondent) + summarise(max(na.rm = TRUE)) evaluated per group
in R (it was ~40% of latency_report() after the aggregation fixes); the
compact path's split() + vapply() form shares the same helper.as.POSIXlt() conversion per
timestamp column (.local_date_hour()), replacing the
as.Date(format(x, tz)) + as.integer(format(x, "%H", tz)) pair that
rendered every instant to a string twice -- in the latency frame builders
(both paths), the summary / ineligible / refusal frames, the date filter and
disposition_run()'s disposition_date. Segment deltas are the direct
numeric difference (what difftime(units = "mins") computes) without its
dispatch.latency_report() parses the flow's timestamp columns ONCE, up front on
the pre-filter data, and the summary / ineligible / refusal frames, the
dedupe and the date filter read the parsed POSIXct columns through
.column_timestamps() instead of each re-decoding the strings (the export's
timestamps were decoded three to four times per run). The population mask
is likewise evaluated once, on the raw columns, and shared by the summary
frame's opted_in signal and the population filter. Parse-failure counts
are summed from the row-subset masks after the population filter, so the
diagnostics are unchanged..local_date_hour() converts per distinct UTC minute rather than per
instant (a zone's offset is a whole number of minutes, so every instant in a
UTC minute shares its local date and hour) -- ~20x cheaper on a campaign
column, verified identical across DST edges and half-hour zones.max() (the suppressWarnings() wrapper forced a per-group R
evaluation over hundreds of thousands of groups; an all-NA input yields no
groups, so there was nothing to suppress).na_if_blank() tests each character column with a single == "" pass.disposition_run()'s exact-duplicate-row collapse compares only the rows
whose phone recurs (a duplicate row must share its phone), instead of
duplicated() over the whole wide export -- which pasted every row to a
string and was ~90% of the function's wall time and its largest transient
allocation. ~10x faster end to end on a 200k-row export.disposition_summary() /
disposition_screen()) derives every group's first/last row position from
the group sizes once instead of a neighbour-comparison pass per ordering,
and renders campaign-id labels once per distinct id.latency_report() composes the dedupe and date-filter row selections into
one index and applies it once, projected to the columns the frame builders
read (flow timestamps + campaign id), instead of copying the full-width
input once per filter step.source_csv_hash sha256 is computed with tools::sha256sum() when
available (R >= 4.5; the same digest, ~2x faster on an export-sized file),
falling back to digest on an older R.latency_report() runs the config shape and column checks before the
parse and the flow-order check on the parsed columns, so the flow-order
ratio no longer decodes every timestamp a second time. Same checks, same
precedence; latency_validate_config() is unchanged.is.na() per segment instead of re-scanning every prior batchDate,
O(segments) rather than O(segments^2)), classifies NA reasons over the NA
rows' indices only, and the compact path counts its NA-date rows with one
tabulate() per reason.n_na_* counts are one tabulate() over an integer-coded
na_reason instead of three string comparisons per cell.build_latency_frame() fills each column of the long frame in place,
slice by slice, instead of binding a per-segment list of sub-frames (the
frame was briefly held twice)..normalize_phone()'s ten-digit fast-path test matches bytewise
(useBytes = TRUE): the same decision without per-string encoding
validation, for the 100M+-row projection reads.utils_fix_double_utf8() tests each value with one grepl() for a
Latin-1-supplement code point and runs the run-by-run repair only on the
distinct candidates (~100x faster on a clean export column; a 2M-value column
with repeated mojibake labels goes from ~20 s to ~0.3 s). Output unchanged.latency_funnel() de-duplicates the bucket key with data.table's
duplicated() instead of duplicated.data.frame() (which pastes every row
to a string), ~200x faster on a fleet-wide consolidated..local_date_hour() keys its distinct minutes as integers and derives the
calendar date from the broken-down time by integer arithmetic
(.days_from_civil()) instead of as.Date.POSIXlt() -- ~10x faster on the
distinct minutes, verified against as.Date() across two centuries and the
DST / half-hour zone checks.data.frame() coercion pass over
millions of rows) and the compact path binds plain lists per segment.order() + duplicated() per cell instead of a unique() per
threshold)..question_timestamp() returns a single opener's column directly (no
coalesce() over one input), and parse_campaign_timestamps() skips the
retry mask on a column that parsed completely.%chin% (~6x faster than %in% on the projection's phone column), and the
per-phone error count tests error with one regex instead of
trimws() + nzchar().disposition_funnel()'s grouped counts are GForce-eligible (bare sum() /
uniqueN() / .N on internally aliased columns, the all-NA rule applied
from a per-group NA count computed only for a column that has NAs) instead
of a per-group .SD$ evaluation in R: ~3-4x faster, same output, including
a by column that is also a funnel flag.s160_read_csv() and s160_gcs_campaign_results_read() gain
timestamps = c("character", "POSIXct"). "POSIXct" decodes every
id.<q>.scriptDate / id.<q>.batchDate column to UTC POSIXct at read
time through parse_campaign_timestamps() -- 8 bytes per cell against ~60
for the export's strings, so a wide export takes roughly half the memory.
fread already infers a clean timestamp column as POSIXct; the option
guarantees it for a column fread has to keep as character (one unparseable
cell is enough) and for the read.csv fallback. Every transform accepts
either form and gives the same result: the parse-failure mask travels on the
parsed column (a parse_failed attribute) so latency_report()'s
parse_failures_per_column still counts those cells. Default
"character" is unchanged.build_ineligible_frame() / build_refusal_frame() and their day rollups
share one terminal-frame builder (.terminal_segment_frame() /
.collapse_terminal_to_day()); the full and compact latency_report() paths
share the hour/day aggregation (.two_grain_consolidated()) and the
per-segment column computation (.segment_columns()); output unchanged.make test-file FILE=tests/testthat/test-<x>.R runs one test file with the
package loaded once.tests/testthat/helper-synthetic.R (deterministic synthetic export /
disposition generators), test-synthetic_parity.R (full vs compact, string
vs POSIXct input, and each fast-path helper against its reference form) and
scripts/bench.R / make bench (hot-path wall time + peak heap harness).max() is never probed on an empty table (which warned).loi, topic, and registration_id are renamed
to tracker_loi, tracker_topic, and tracker_registration_id, so the
disposition parquet's Tracker-derived dimensions match the campaign (latency)
parquet's tracker_* convention. disposition_records() now returns the
prefixed names. A projection written before the rename (carrying the bare names)
has those columns omitted by records() until it is re-stamped with the
prefixed names.mode is renamed to survey_mode, and a new tracker_mode
column is recognised, mirroring the latency parquet's split of the DETECTED
mode (survey_mode, from disposition_run() / detect_survey_mode()) from the
DECLARED Tracker mode (tracker_mode, a downstream tracker enrichment).
disposition_records() returns survey_mode in place of mode and carries
tracker_mode when the projection has it.disposition_records() now recognises the full tracker_* dimension set --
adding tracker_client, tracker_project, tracker_brand, tracker_state,
tracker_vendor, tracker_fielding_location, tracker_pricing_structure,
tracker_voter_file_source, and tracker_n_questions alongside the existing
tracker_loi/tracker_topic/tracker_registration_id, so the disposition
projection carries the same Tracker dimensions as the campaign (latency) parquet.
Enrichment columns: returned when the projection has them.duckdb instead of silently loading the whole projection. When a scoped read
(disposition_screen(), or disposition_summary() / disposition_records()
with phones) hits a projection over 100 MB and duckdb is not installed, an
interactive session prompts to install it (via rlang::check_installed()) and a
non-interactive one stops with the same guidance, rather than reading the whole
file into memory (tens of GB). Set
options(survey160r.disposition_engine = "nanoparquet") to take the whole-file
read deliberately (no prompt), or install duckdb. Attaching the package without
duckdb (library(survey160r)) now prints a one-line install hint. Small files,
glob paths, and a pinned engine are unaffected. This replaces the previous
once-per-session message.disposition_funnel() rolls the disposition dataset up to funnel counts +
rates by dimension. The disposition analogue of campaign_metrics_summary():
same by / rates / percent shape, a campaigns count, and the shared
funnel_rates() definition, over the disposition projection instead of the
latency one, plus the disposition-only terminal split (n_ineligible /
n_refused). Groups the per-recipient rows by one or more columns (by, default
carrier) and -- on the day grain -- by disposition_date, so "how many sends /
opt-ins / completes per carrier" is one call over disposition_records() /
disposition_pull() output. A carrier the uploaded list did not supply is kept
as its own NA group; an all-off-channel group's n_completed (and completion
rate) stays NA rather than a false 0.funnel_rates() now also emits opted_in_rate (opted_in / sent, the
funnel-anchored opt-in rate) alongside the existing opted_in_engaged_rate
(opted_in / engaged, opt-in conversion among the engaged). The two answer
different questions -- end-to-end yield per send vs conversion once a recipient
replies -- so both are emitted, and every consumer (campaign_metrics_summary(),
disposition_funnel()) inherits both. Additive: existing columns are unchanged.consolidated frame gains n_refused, the refusal-terminal
count. It is the per-(bucket, segment_index) sibling of n_ineligible,
counting recipients who reached the refusal terminal (id.refusal.scriptDate)
at each segment. This completes the latency funnel's terminal split
(refused + ineligible) and matches the disposition parquet's refused column
name. Emitted identically on both the full-frame and compact = TRUE paths.disposition_screen() runs on a laptop: the sample's phone match now happens
inside the Parquet scan. With the duckdb package installed (new in Suggests,
used automatically), disposition_screen() -- and disposition_summary() /
disposition_records() when given phones -- read only the sample's rows
instead of loading the whole projection. On the production projection (~140M
rows) a 10k-phone screen drops from ~150 s and ~26 GB peak RAM to ~2 s (~4-5 s
on a 4-core laptop) and ~0.6 GB; a 200k-phone screen from ~200 s to ~3 s, still under 1 GB. Results are
identical; a 4-core, 4 GB-capped run screens 200k phones in ~6 s where the old
code was killed for running out of memory. Without duckdb the readers fall back to the whole-file
nanoparquet read (a phone-scoped read of a file over 100 MB says once per
session that duckdb would help);
options(survey160r.disposition_engine = "nanoparquet") forces the fallback.
Whole-file reads (disposition_funnel(), an unscoped disposition_summary() /
disposition_records()) also go through duckdb when installed: about a
third less peak memory and roughly twice as fast (disposition_funnel() by carrier: ~19 GB /
20 s to ~13 GB / 9 s).
The per-phone rollup behind disposition_summary() / disposition_screen()
is vectorized (one group index instead of a tapply() per output column):
~10x faster on large samples, identical results. The first_disposition_date /
last_disposition_date columns no longer carry a stray one-dimensional dim
attribute (a tapply() artifact, present only when every row was contacted).
Phone normalization skips the regex rewrite for numbers that are already a
plain 10 digits, the common case: ~5x faster on large vectors, same result.
Shared by disposition_screen() and opt_out_screen().
latency_report() / latency_run() gain compact = TRUE, a streaming path
that never materialises the long frame. On very wide campaigns the
(n_questions - 1) x N_respondents long frame is almost entirely NA drop-off
(a row per respondent for every configured question, even the deep ones no
respondent reached). With compact = TRUE the report is computed segment by
segment, keeping only rows with a real date and rebuilding the dropped
NA-date rows' (date=NA, hour_local=NA) day-rollup bucket from per-segment
counts, so peak memory is bounded by real respondent engagement rather than by
(n_questions - 1) x N. consolidated and diagnostics match the default
path -- enforced by parity tests on synthetic fixtures and hand-built edge
cases, and additionally checked manually against three real multi-GB exports
(a full byte-identical run on one, 250k-row subsets on the others); only
latency_frame differs, then holding just the kept real-date rows. The
default (compact = FALSE) is unchanged.
latency_report() per-bucket aggregation rewritten on data.table; very
large campaigns now complete in seconds on a modest memory budget. The
three per-bucket aggregations (aggregate_totals(),
aggregate_worst_cascade(), aggregate_segment_cells()) moved from
dplyr::group_by() / summarise() to data.table, and the long frame is
bound with data.table::rbindlist() instead of do.call(rbind, ...). On a
~47M-row long frame (a ~500k-respondent, 95-question campaign) the report
dropped from >15 min and >11 GB (the dplyr path did not finish) to ~11 s,
and comfortably inside a 32 GB budget. Output is identical -- the full test
suite, including the legacy-parity fixtures, passes unchanged; only the
aggregation engine changed. (The package is marked .datatable.aware so
data.table's non-standard j/by evaluation resolves from within the
namespace-qualified calls.)
Lower peak memory in latency_report() on very large campaigns. Two
output-neutral reclaims: the no-op population filter (the common case, when
filters.population is unset) now returns the frame unchanged instead of
taking a whole-frame data[mask, ] copy; and the hour-grain aggregation's
transient grouped frames are reclaimed (an explicit gc()) before the day
pass allocates its own, so R's lazy collection no longer leaves both passes'
intermediates resident at once. Neither changes any output -- they trim the
peak on the biggest campaigns where the latency long-frame dominates memory.
s160_gcs_campaign_results_read() gains a columns_fn argument for
header-derived column projection. The reader already accepted an explicit
columns= set, but a caller that wants only the columns a transform reads
(via latency_input_columns() / disposition_input_columns()) cannot build
that set without the file's header, which lives in GCS. columns_fn closes
the gap: a resolver function(header) -> columns is applied to the header of
the just-downloaded file, and its result becomes the projection. This makes
the GCS reader project as narrowly as the local s160_read_csv() path does,
so a fleet pass over wide exports keeps only the needed columns and reads far
less into memory. columns= still wins when supplied; a resolver (or
header-peek) error emits a warning and falls back to a full read (so a broken
projection is visible rather than silently reading every column); a
non-function columns_fn is rejected; and with neither argument the reader is
unchanged and reads every column.disposition_run() now emits a carrier column, and disposition_records()
returns it. The recipient's mobile carrier ("AT&T", "Verizon", "T-Mobile",
"Metro PCS", "US Cellular", "Other", ...) rides in the campaign export's
optional misc block, present only for campaigns whose uploaded list supplied
it. Surrounding whitespace is trimmed and a blank/whitespace value maps to NA
(the same hygiene as error, and matching the DB producer's trim()); semantic
normalization (MVNO-folding, typo-fixing, grouping) is left to the read layer. It is matched
case-insensitively, and NA for a campaign
whose export omits the column. disposition_input_columns() retains it so a
projected read keeps it. A projection produced before this release simply lacks
the column and the reader omits it (subset-tolerant, like registration_id).
disposition_run() collapses fully-identical duplicate rows instead of
erroring on them. A campaign export can repeat a respondent's row verbatim
(a re-export appended to itself, a batch duplicated in an upstream merge). An
exact-duplicate row carries no information the first copy does not, so it is
now dropped before the one-row-per-(phone, campaign_id) grain guard rather
than failing the whole campaign. The guard is unchanged for a genuine
conflict: a duplicate phone whose rows differ still stops, so a real
grain violation is never silently merged. Source provenance (meta) is
preserved across the collapse.
Articles and README refreshed for the current disposition reader surface.
The disposition guide, the README, and the opt-out guide now document the
n_* status counts (which replaced the removed ever_* booleans), the
refused / ineligible terminal split, best_disposition, and the populated
disposition_date (the "beta / all-NA date" caveats are gone). The
disposition_pull() size note is corrected from "around 150 MB" to "a few
hundred MB" -- the projection has grown well past the old figure. Docs-only;
no behaviour change.
Reference pages corrected. disposition_records()'s documented return
schema now lists registration_id (added to the stored schema in 0.54.0). The
disposition_screen() example and the disposition guide now screen on all
three hard stops (n_terminated / n_refused / n_ineligible) rather than
n_terminated alone: on the production projection the three are set
independently, so a n_terminated-only filter could keep a refuser. A stray
section-sign glyph in two latency reference pages is spelled out ("section").
Docs-only.
utils_fix_double_utf8(): detect or repair double-encoded (mojibake) text in a
campaign export. A raw export can carry non-ASCII characters in its appended
sample columns (treatment labels, vendor-appended employer/title text) doubly
encoded -- an upload that read a UTF-8 file as Latin-1 and re-encoded it to
UTF-8, so an en dash arrives as "â". The function verifies by
default and repairs on request: with apply = FALSE (the default) it is a dry
run that reports how many values it would repair (and, for a data frame, which
columns) and returns the input unchanged; with apply = TRUE it returns the
repaired data. It reverses only maximal Latin-1-supplement runs whose bytes
re-read as valid UTF-8, so genuinely single-encoded text (a real en dash, an
emoji, an accented name) is left byte-for-byte unchanged; a clean input is
returned unchanged, and the repair is idempotent. It works on a character
vector or on the character columns of a data frame, and unless quiet = TRUE
it emits a concise one-line message() summary (a count, the affected columns
capped at three with +N more, and -- on a dry run -- how to apply), so
neither a dry run nor a repair is silent. The result is returned invisibly, so
a bare console check prints only the summary line and not the (often
100-plus-column) data. It is a post-hoc remedy for exports produced before the
double encoding is fixed at its upload source.
campaign_metrics_records() / campaign_metrics_summary() / funnel_rates():
aggregate the campaign-metrics projection. Post-pull transforms in the
campaign_metrics family, parallel to disposition_records() /
disposition_summary(). campaign_metrics_records() reads campaign_all.parquet
into analysis-ready cells -- the tracker_* dimensions renamed to short names,
the funnel counts, and the cell keys -- and, by default, collapses the
latency/threshold replication to one row per cell (so summing the raw pull can
no longer double-count). campaign_metrics_summary(by = ...) rolls those cells
up to summed funnel counts by any project dimension(s) and appends the funnel
rates. funnel_rates() is the single definition of engagement_rate,
opted_in_engaged_rate, and completion_rate (zero denominators return NA).
campaign_metrics_pull(): fetch the consolidated campaign-metrics projection.
A new bare reader in the campaign_metrics family, parallel to
disposition_pull() / opt_out_pull(), that downloads campaign_all.parquet
(the fleet's per-campaign funnel + recipient-latency view carrying the
tracker_* project dimensions) from the environment's analytics bucket and
returns the local path. Available for prod and dev. It is aggregate
metrics, not phone-keyed PII, so it lives in the analytics bucket and has no
screen helper -- read the pulled Parquet directly (e.g. arrow / nanoparquet)
and aggregate.
disposition_records() carries a registration_id column when present. The
per-campaign registration id (an id-like context scalar added upstream during
enrichment) is now part of .DISPOSITION_RECORD_COLS, so a projection enriched
with it passes the value through unchanged (one string per (phone, campaign_id)).
Projections produced before the column existed read exactly as before -- the
reader returns just the subset of canonical columns present, so the addition is
backward compatible and requires no re-enrichment of old data.
disposition_by_phone/disposition_all.parquet and
global_opt_out/global_opt_out.parquet to disposition_all.parquet and
global_opt_out.parquet (bucket root), matching the single-file-at-root
convention. disposition_pull() / opt_out_pull() resolve the new paths via
the bundled inst/config.json; the objects must be republished at the root
before this version is consumed.refused no longer silently drops reply-only refusals. A terminal step is
now counted as reached by the respondent's own reply (batchDate), not only by
a send (scriptDate). Many scripts route a refuser to the refusal step and end
without sending a message, so the refusal is recorded only on the inbound
reply and had no scriptDate -- disposition_run() was scoring those as not
refused, undercounting refused fleet-wide. disposition_input_columns() now
also projects the terminal families' batchDate. ineligible is unchanged
(screen-outs do send their message).The disposition readers surface the refused / ineligible split.
disposition_records() returns the two new columns as stored;
disposition_summary() / disposition_screen() gain n_refused /
n_ineligible counts (alongside n_terminated, kept as the union). The derived
disposition category splits too: latest_disposition / best_disposition now
report refused (declined) or ineligible (screened out) where the terminal was
name-classified, ranked terminated < ineligible < refused. terminated
survives as a category for the unsplit residual: disposition_run() itself never
emits one (it sets terminated = refused | ineligible), but a downstream
projection that sets terminated from an authoritative status independent of the
name-split -- the consumer DB producer keys it on SQL status='terminated' --
can carry a terminal with neither flag (e.g. an in-survey q_<n> screener the
name-match cannot classify), and the readers keep those rows terminated.
Breaking: a consumer that
matched latest_disposition == "terminated" must now also handle "refused" /
"ineligible" (screen on all three via statuses=). A pre-0.51.0 projection
that lacks the two columns still reads -- they default to 0 and those rows stay
terminated.
disposition_run() splits terminated into refused and ineligible.
The per-phone frame gains two columns: refused (the recipient reached a
refusal terminal -- declined) and ineligible (a screen-out / termination
terminal). terminated is kept as their union for back-compat. The terminal
steps are name-matched by regex over ALL discovered question columns (not the
two hardcoded id.refusal / id.ineligible), so non-standard names are caught:
online_refusal / refusal_sp count as refused (matched on the full word
refusal); terminate / terminating / term / screenout / misspellings
(ineligble, ineligable) count as ineligible. A bare refuse is
deliberately NOT a refusal: a survey question's refused-to-answer branch
(q_<name>_refuse) and the post-close panel-recruitment decline (panel_refuse,
which fires after the survey close) both stay participation/continuation steps.
Classification is by name rather than the script's step type because type
alone cannot split refused from ineligible (both are terminating) and mislabels
some real screen-outs (e.g. ineligable ships as type=closing); verified
against the full production script corpus, the name regex classifies 99.6% of
terminating steps with no false positives. The residual is 4 generically-named
in-survey screener steps (q_7 / q_4 / q_2 / q_0_copy_copy, 16 campaigns)
whose name carries no terminal signal -- folding in the step type for those is
a deferred enhancement.
Routing opt-in no longer over-counts a non-standard terminal step as
consent. The continuation set now excludes the full refusal + ineligible
families above (previously only ^refus / ^inelig / ^optout), so reaching
online_refusal, terminate, etc. is correctly a hard stop, not an opt-in.
Opt-in is now measured by routing, not by the intro answer text, so
n_opted_in (latency) and opted_in (disposition) no longer read 0 for a
campaign whose consent answer is not the literal "Yes". The default opt-in
signal was id.<opener>.finalText == "Yes", which silently missed every
campaign whose opener routes forward on a different answer -- a Spanish "en
español" branch, an "im down", a templated value -- undercounting opt-in to 0
for those recipients. Opt-in is now derived from the flow: a recipient opted in
when the opener routed them FORWARD, i.e. they reached a continuation step (any
non-opener, non-terminal scriptDate -- a survey-body question or the close
family; refusal / ineligible / opt-out branches are excluded). This is language-
and phrasing-agnostic and reads no answer text, so bilingual and non-"Yes"
campaigns are counted. Both views derive it from the one shared funnel mask, so
they stay in agreement. A caller that needs opt-in decided by a specific answer
can still pass an explicit population filter to disposition_run(); the
latency view uses the routing default. Because opt-in reads no answer text, the
projection helpers (disposition_input_columns(), latency_input_columns()) no
longer retain the opener finalText column by default. A completion is also
treated as an opt-in (opted_in is OR-ed with completed), so the funnel stays
monotone (opted_in >= completed) in every mode -- this matters for text-to-web
campaigns whose link sits in the intro, where the recipient completes on the web
with no downstream close step and the web completion is the only opt-in evidence.
NA integer values are no longer mis-read from nanoparquet-written
projections. nanoparquet 0.5.1 mis-decodes an NA integer (e.g. completed on
t2w_external rows) under col_select -- returning uninitialized memory instead
of NA -- which could corrupt n_completed. disposition_summary() /
disposition_screen() now col_select only DuckDB-written projections (the
production writer, verified NA-safe) and read any other writer in full, so the
counts are correct regardless of writer while the production read stays
column-projected.
disposition_summary() / disposition_screen() report status COUNTS, not
booleans. The per-phone ever_engaged / ever_opted_in / ever_completed /
ever_terminated flags are replaced by cumulative counts n_engaged /
n_opted_in / n_completed / n_web_complete / n_terminated -- how many of
the phone's campaigns reached each status (0 = never, > 0 = the count; they
overlap, since a completed campaign is also engaged). ever_completed's
completed OR web_complete fold is split into n_completed + n_web_complete.
ever_completed folded completed OR web_complete, so migrate a filter like
ever_completed %in% TRUE to (n_completed > 0 | n_web_complete > 0) -- not
n_completed > 0 alone, which would miss a phone that only web-completed.ever_contacted is removed -- it was fully redundant with n_campaigns. A
never-contacted phone is now marked by n_campaigns == 0 (a blank/unparseable
phone still comes back all-NA). Replace !ever_contacted with
n_campaigns == 0, and ever_contacted %in% TRUE with n_campaigns > 0.n_campaigns, campaigns), the status counts, the latest and best
disposition (each with its campaign id), then the first/last date span.
Read columns by name, not position.disposition_summary() / disposition_screen() gain date span and error
count. New columns first_disposition_date / last_disposition_date (the
earliest and latest disposition_date across the phone's campaigns, NA when
none is dated) and n_error (how many of the phone's campaigns carried a
carrier delivery-error code). The summary now reads the projection's optional
error column; an un-enriched frame without it yields n_error = 0.best_disposition + campaign ids on the summary. latest_disposition is
selected by recency (the phone's most-recent campaign); the new
best_disposition is selected by funnel precedence -- the furthest-reached
category across all the phone's campaigns (completed / web_complete rank
highest, then terminated, opted_in, engaged, non_response). Each pins
its campaign with latest_campaign_id / best_campaign_id (returned as
character); campaigns still lists every id.disposition_run() now derives disposition_date. The per-respondent
frame gains a disposition_date column: the row-wise maximum of every
id.<step>.scriptDate send timestamp (the phone's last outbound message),
bucketed to a calendar day in the new field_timezone argument (default
"America/New_York", matching the latency view's send-date bucketing; an
unknown zone is rejected rather than silently mis-bucketed). It is
NA when no send time survives. disposition_input_columns() now retains every
scriptDate column so the max is computed over the full flow, not just the
opener/closer/terminal sends. Previously the column was left for the consumer to
fill (NA from a CSV source); a disposition read no longer needs a separate date
source to order each phone's latest campaign.bucket = is deprecated on the GCS readers and pulls; select data with
env =. s160_gcs_campaign_results_*() gain an env argument
("prod"/"staging"), and disposition_pull() / opt_out_pull() accept the
unified env enum. A single internal registry now maps (dataset, env) to a
physical bucket, so callers never name a GCS bucket and a bucket reorg is a
registry change rather than an API change. s160_api_auth() sources its
campaign bucket from that same registry. Passing bucket = still works but
warns; an env a dataset lacks (e.g. disposition_pull(env = "staging"))
raises a clear "no
Deprecation warnings now route through lifecycle. The bucket =
warnings (on the campaign readers, the pulls, and s160_gcs_init()) use
lifecycle::deprecate_warn(), so each message names the version the argument
was deprecated in and is shown once per session per call site rather than on
every call (no more warning spam in a loop). The positional-env nudge on
s160_api_auth() is throttled the same way via rlang. The deprecated forms
still work exactly as before; only the warning cadence and wording change.
Passing the environment positionally to s160_api_auth() is deprecated;
name it (s160_api_auth(env = "staging")). The environment is the sole
selector, so naming it keeps call sites self-documenting. s160_api_auth("prod")
still works but warns; the default (s160_api_auth(), prod) is unaffected.
s160_gcs_init() no longer needs a bucket. Authentication is
account-level, so a single s160_gcs_init() covers every bucket the account
can read. Reader functions now resolve their own bucket, defaulting to prod
("campaign_results" for campaign exports; s160_disposition_<env> for the
disposition and opt-out pulls). Passing bucket = still works but now warns
and will be removed in a future release; pass bucket = to an individual
reader when you need a non-default bucket. Readiness is now checked by whether
you have authenticated, not by whether a global bucket was set.
s160_gcs_init(adc = TRUE) authenticates headlessly. With adc = TRUE,
s160_gcs_init() uses Application Default Credentials (gcloud auth application-default login or a service-account environment) instead of the
browser OAuth flow -- no client secret, no browser -- for reports, CI, and
scheduled jobs. Default (adc = FALSE) is unchanged. Errors clearly when no
ADC are available.
parse_campaign_timestamps() is now exported. The parser for the
export's timestamp encoding ("2026-01-26 17:30:16.853688Z" -> POSIXct
UTC) that the pipeline uses internally is now public, so report authors can
decode an export column without re-implementing the format. (Renamed from the
internal parse_s160_timestamps_chr.)
New question_* accessors for per-question analysis.
question_timestamps(df, question, field) resolves and parses an export's
id.<question>.<field> column to POSIXct (UTC).
question_funnel(df, questions) returns the per-question reached-count funnel
(one row per question: n_reached, pct_reached) -- the per-question
companion to latency_funnel()'s send/opt-in/complete anchors.
latency_funnel() reduces a consolidated table to the funnel counts
safely. The consolidated frame from latency_run() / latency_report()
denormalises the funnel anchors (n_sent / n_engaged / n_opted_in /
n_completed) across every (segment, threshold_min) row of a bucket and
carries two grains at once (per-hour rows plus a hour_local = NA day
rollup), so summing an anchor column over the raw frame multi-counts.
latency_funnel(consolidated, grain = "day") returns one row per
(campaign_id, date); grain = "hour" returns one row per hour. Both
collapse the segment x threshold fan-out and read a single grain, so the
anchors are correct and sum()-ming across the "hour" result matches the
"day" value. Consumers no longer hand-roll this reduction.
One bundled config for datasets and environments: s160_config() /
s160_datasets(). survey160r now reads a single bundled config
(inst/config.json) mapping each environment to its API base URL and its
datasets' GCS locations -- the one place physical bucket names and URLs live.
s160_config() returns the full nested config; s160_datasets() gives a tidy
(dataset, env) table of what is available (the values to pass as env =),
with the physical bucket kept hidden. Both s160_api_auth() and the GCS
readers resolve through this config, so a bucket reorg is a config edit rather
than scattered code changes. A later release can refresh the config from a
Survey160 endpoint, falling back to the bundled copy.
s160_api_auth() now supports env = "dev". All three environments
(prod, staging, dev) are configured, each reading its key from
S160_<ENV>_API_KEY (prod also accepts the legacy S160_API_KEY); the key
variable is derived from the environment, never stored in the config. Note
disposition/opt-out have no staging tier -- staging sends fold into prod, so
read them with env = "prod".
Download the opt-out list: opt_out_pull(). Fetches the opt-out Parquet
from the environment's s160_disposition_<env> bucket to a local cache and
returns the path, ready to hand straight to opt_out_screen(). Parallel to
disposition_pull() (same auth, caching, and refresh/dest/progress
arguments); it shares the disposition bucket but caches to a distinct file, so
the two pulls never collide.
Screen a sample against the opt-out list: opt_out_screen(). A new
reader parallel to disposition_screen(). It takes a phone sample and a
path to the opt-out Parquet (a phone column plus an optional date_added)
and returns the sample unchanged with two columns appended, 1:1 with the
input rows: opted_out (logical) and opt_out_date (the list's
date_added, uninterpreted). A valid phone absent from the list is
opted_out = FALSE; a blank/unparseable phone is opted_out = NA (unknown,
not a claim it is contactable). Like disposition_screen() it never drops
rows -- the caller decides what to suppress -- so the two screens chain:
annotate a sample on prior disposition and current opt-out status in one
pass, then filter once on the caller's own rules.
Disposition frame now carries the raw carrier error code.
disposition_run() emits an error column: the per-recipient Bandwidth
delivery-error code from the export's error_code field, passed through as a
string (blank / "None" / a reader-supplied NA normalize to NA, so error
is NA for every cleanly delivered record), and disposition_input_columns() projects
error_code accordingly. The code is a delivery-quality attribute orthogonal
to the funnel flags -- an errored record is almost always a non-response whose
message never landed -- and is emitted uninterpreted; a human-readable category
is left to the read layer. This fills the first of the disposition frame's two
enrichment columns (error); date_closed_on remains a downstream placeholder.
New Opt-out screening guide. vignette("opt-out") (and a matching README
section) walks through pulling the opt-out list and screening a sample with
opt_out_pull() + opt_out_screen(), including chaining with disposition
screening and filtering once.
Documentation consistency pass alongside the opt-out family. Corrected
disposition_pull()'s dest help (a directory saves <bucket>.parquet, not
the previously-documented disposition_all_<env>.parquet), added reciprocal
See also links between the opt-out and disposition screens, and listed
opt-out screening in the package overview (?survey160r).
Phone matching is now a single shared normalizer. The digit-normalize
step (strip non-digits, drop a leading US country code) moved to a private
.normalize_phone() in aaa_utils.R, so disposition_screen() and the new
opt_out_screen() match a sample against their datasets identically. No
behavior change.
disposition_pull() can show a download progress bar. A new progress
argument (default interactive()) surfaces a live bar while the ~150 MB
projection downloads, so an interactive pull no longer reads as a hang; batch
and scheduled runs stay silent. The bar is layered onto the existing
gcs_get_object() transfer via httr::progress(), leaving the auth path
unchanged.
Latency and disposition resolve the opener funnel from one shared module.
The name-agnostic opener-set resolution, the opt-in population, the send/reply
timestamp coalesce, and now the composed per-recipient funnel masks
(sent / engaged / opted_in, via .funnel_masks()) all live in a single
R/opener.R used by both the latency summary (build_summary_frame()) and the
disposition transform (disposition_run()), so the two views compute the same
sent/engaged/opted-in funnel from one place and cannot drift. As part of this
the last definitional gap between them is closed: disposition's engaged is
now gated on sent (a reply presupposes a send), matching the latency view's
n_engaged. This only changes output for anomalous rows that carry a reply
timestamp with no send. (The public output columns were then renamed to the
canonical past-tense vocabulary -- see Breaking changes below.)
A wrong or misspelled reader argument now fails with a clear message.
s160_gcs_campaign_results_read() / s160_read_csv() reject a ...
argument the underlying CSV reader would not accept (e.g. a stray
filter_open =) with "argument(s) not accepted by the CSV reader", naming
the function -- instead of a cryptic fread "unused argument" error.
disposition_pull() reports the cached copy's age. On a cache hit the
message now reads "Using cached disposition projection (3 hr old): ..." so a
stale local projection is obvious; pass refresh = TRUE to re-download.
A date filter against the beta's empty date_closed_on now warns.
disposition_summary() / disposition_records() / disposition_screen()
warn when date_from/date_to is set but every date_closed_on is NA
(the current beta does not populate it), so the empty output is explained
rather than silent.
Disposition date_closed_on column renamed to disposition_date. On the
disposition readers (disposition_records() / disposition_summary() /
disposition_screen()) and the stored projection schema, the date column is
now disposition_date, and the date_from / date_to bounds filter it. The
old name asserted a campaign-close date; the value is a per-row disposition
date (one row per (phone, campaign_id)), so the name now matches what the
readers do with it -- ordering each phone's latest campaign and backing the
date filters. The column is still NA in the beta, so this is a name change
with no data impact; the downstream writer/schema are updated in lockstep. The
date_from / date_to argument names are unchanged.
Latency summary count columns renamed to the canonical funnel vocabulary.
On the latency_report() / latency_run() consolidated output, n_texted is
now n_sent and n_consented is now n_opted_in (n_engaged, n_completed,
and n_ineligible are unchanged). This gives the counts the same past-tense
stems as the disposition flags (sent / engaged / opted_in / completed)
-- n_<signal> is the count of that signal. Values are identical; a rename
only. The consolidated schema_version bumps to 6 (algorithm_version stays
2.2.0). Regenerate the latency Parquet fleet and update consumers (the
survey160-shiny dashboards are updated in lockstep).
Disposition output columns renamed to the canonical funnel vocabulary. The
per-row flags started and opt_in are now sent and opted_in, and
complete is now completed -- past tense, matching sent / engaged /
opted_in and the latency n_completed count (web_complete is unchanged; it
mirrors the fixed export column name). The derived latest_disposition
categories change likewise ("opt_in" -> "opted_in", "complete" ->
"completed"), so e.g. disposition_summary(statuses = "opt_in") becomes
statuses = "opted_in", and the per-phone rollup ever_complete is now
ever_completed. This touches disposition_run()'s frame, the
disposition_records() / disposition_summary() / disposition_screen()
schemas, and the disposition Parquet projection. Regenerate any persisted
disposition Parquet and update consumers (the survey160-shiny disposition
pipeline is updated in lockstep). Other rollup names (ever_contacted,
ever_opted_in, …) are unchanged. Beta surface, so no deprecation shim.
s160_gcs_pull_csv() removed; s160_gcs_campaign_results_read() gains a
hash argument. The two GCS CSV readers are now one:
s160_gcs_campaign_results_read(id, hash = TRUE) stamps the same
source_csv_hash / source_csv_path provenance the wrapper did (and that
latency_run() / latency_report() surface on result$meta); hash = FALSE
(default) returns a plain frame, unchanged from before. Mirrors the local
sibling s160_read_csv(hash =). Migration: s160_gcs_pull_csv(id) ->
s160_gcs_campaign_results_read(id, hash = TRUE).
disposition_rollup() removed; disposition_summary() now takes a path
or an in-memory data frame. The two functions did the same per-phone
rollup, split only by input type. disposition_summary(x) now dispatches on
x: a Parquet path is read and then summarized; a data frame (e.g. from
disposition_records()) is summarized directly with no I/O -- the
read-once/summarize-many pattern. Migration: disposition_rollup(df) ->
disposition_summary(df). Positional disposition_summary(path) is
unchanged, but the first argument was renamed dataset -> x, so a named
call disposition_summary(dataset = path) must become
disposition_summary(path) (or disposition_summary(x = path)).
required_latency_columns() renamed to latency_input_columns() and
required_disposition_columns() renamed to disposition_input_columns() --
the domain now leads the name (grouping with latency_* / disposition_*, and
matching the internal .disposition_input_columns helper). Update call sites.
disposition_run() corrected: started now means contacted (the intro
was sent, id.intro.scriptDate) and engaged now means replied
(id.intro.batchDate). Both intro flags previously keyed on the wrong
column (started on the reply, engaged on the accepted answer):
id.intro.scriptDate is the outbound send and id.intro.batchDate the
inbound reply. A
contacted-only run now includes every contacted recipient, non-responders
included, so it returns far more rows (~20x on real campaigns); regenerate any
persisted disposition data. disposition_input_columns() now returns
id.intro.scriptDate in place of id.intro.finalValue.
Latency summary n_texted corrected, and a new n_engaged count added.
The per-(campaign, date, hour_local) summary metrics on the
latency_report() / latency_run() consolidated output previously keyed
n_texted on the recipient's reply (id.intro.batchDate), so it counted
repliers, not sends -- understating campaign volume ~10-15x. It now keys on
the outbound send (id.intro.scriptDate), the same reply-vs-send correction
already applied to disposition_run(), and the send hour now anchors the
date/hour bucket (the only timestamp a texted-but-never-replied recipient
has). A new n_engaged column carries the reply count (the old n_texted
value). n_consented / n_completed counts are unchanged, but every rate
built on n_texted (opt-in %, completion %) becomes a true of-sent rate.
Bumps algorithm_version to 2.2.0 and the consolidated schema_version to
5; regenerate the campaign Parquet fleet and update consumers that read the
summary columns.
Disposition readers renamed for grain clarity. The per-phone file reader
disposition_query() is now disposition_summary(). The two file readers now
read their grain from the name -- disposition_summary() (one row per phone)
beside disposition_records() (one row per (phone, campaign_id)) -- sharing
a private per-phone rollup core (.disposition_rollup). Beta surface, no
deprecation shims.
Disposition reader surface -- screen a phone sample against the disposition
dataset (one row per (phone, campaign_id), contacted-only):
disposition_summary(x, ...) -- roll the disposition data up to one row per
phone (cross-campaign screening flags ever_contacted / ever_completed /
ever_terminated / ... plus latest_disposition); x is the Parquet
projection path (read then summarize) or an in-memory frame from
disposition_records() (summarize directly, no I/O).disposition_screen(sample, dataset, ...) -- annotates a caller's
sample data frame in place with the disposition columns, preserving the
original rows/columns/formatting (the sample-cleaning
surface).Adds a lightweight nanoparquet dependency for the Parquet read.
disposition_pull() -- download the disposition Parquet projection from
the s160_disposition_<env> GCS bucket to a local (cached) path, ready to hand
straight to disposition_summary() / disposition_screen(). Authenticate once
with s160_gcs_init(); pass refresh = TRUE for a fresh copy. Bare-named (it
fetches the survey160r-derived projection, not a raw source) so it groups with
the rest of the disposition family.
disposition_records() -- read the disposition projection's rows as
stored: one row per (phone, campaign_id) with the full disposition schema
(sent, engaged, opted_in, completed, web_complete, terminated,
error, loi, topic, mode, date_closed_on). The raw level beneath
disposition_summary() (which rolls each phone up to one screening row) -- use
it to inspect, export, or build a custom rollup. Takes the same phones /
campaign_ids / date_from / date_to / page scoping as
disposition_summary(), minus statuses (that selects a per-phone
latest_disposition, which exists only after the rollup). Returns only the
canonical columns present, so it also works on an un-enriched projection
straight from disposition_run().
disposition_summary() / disposition_screen() no longer crash on a
column-short projection path. .disposition_read_parquet() projected the
read with a fixed col_select, so pointing either reader at an un-enriched
projection (the computed columns only, no date_closed_on) raised a raw
nanoparquet "Column ... does not exist" error instead of the graceful
optional-date_closed_on handling the docs promise (and that the in-memory
frame path already had). The read now intersects col_select with the file's
actual schema; a short projection summarizes, and a genuinely missing required
column still returns the clean "missing required column" error.
page / page_size = Inf now errors with "positive integers" instead of
an internal "missing value where TRUE/FALSE needed" (the integer check used
x %% 1, and Inf %% 1 is NaN).
SMS completion now counts every close-family branch, not just id.close.
Both the latency summary (n_completed) and the disposition transform
(complete) keyed SMS completion on a hardcoded id.close.scriptDate, so a
bilingual campaign's Spanish completers (who reach id.close_sp /
id.close_latinos) were dropped -- the close-side analogue of the opener bug.
Completion is now resolved over the close family (grep "^close(_|$)", via
.reached_close()), so the union of every language's completers is counted,
matching the app's phonelist.complete. Verified against production: two
campaigns' close + close_sp scriptDates summed to the exact database
closed count (214 and 61). A single-close campaign is unchanged.
disposition_input_columns() now projects the close family so a pruned read
keeps close_sp.
disposition_run() no longer drops or under-counts campaigns whose opening
question is not named intro. started / engaged / opt_in keyed on
hardcoded id.intro.* columns, so (a) a campaign whose sole opening question
is named otherwise (e.g. FIRSTNET, intro_sp) produced all-zero flags and
vanished under the contacted_only default, and (b) a bilingual campaign that
routes recipients to intro and intro_sp / intro_latinos (via the v2
initialconditionals) counted only the intro branch and dropped the rest.
Verified in prod: an 873-send FIRSTNET campaign emitted zero rows (47 such
single-opener campaigns exist), and a live 39,849-send campaign counted only
17,415 of its recipients (22,434 on intro_latinos were dropped; 72 mixed
campaigns exist). The opening question is now resolved per campaign as a set
-- every intro-family question present (id.intro / id.intro_*), or a single
discovered opener (e.g. FIRSTNET) when there is no intro family -- and a
recipient is contacted if they received any of them; opt_in is the
disjunction over the set's finalText columns. disposition_input_columns()
returns every opener's columns (leading with them). Behaviour is byte-identical
for a pure-intro campaign. Regenerate persisted disposition data to pick up
the previously-dropped / under-counted campaigns.
Latency view: campaigns whose opening question is not named intro are no
longer dropped or under-counted. build_summary_frame() and the config /
validation / funnel-filter surface hardcoded id.intro.*, so a campaign whose
opener is named otherwise (e.g. FIRSTNET) hard-errored at
latency_validate_config() (required id.intro.finalText), and a bilingual
campaign that routes recipients to intro and intro_sp / intro_latinos
(via the v2 initialconditionals) counted only the intro branch. The opening
question is now resolved per campaign as a set -- every intro-family question
present (id.intro / id.intro_*), or a single discovered opener (e.g.
FIRSTNET) -- and n_texted / n_engaged / n_consented / the ineligible
anchor / the dedupe + date filters key on it (latency_build_config() builds
the opt-in population as a disjunction over the openers' finalText;
validate_columns_present() requires the population's columns, not a hardcoded
id.intro.finalText). Behaviour is byte-identical for a pure-intro campaign,
and this also fixes the previous crash on an entirely-absent id.intro.scriptDate.
This aligns the latency volumes with the disposition view and the analytics
overview API (which are name-agnostic). NOTE: the mixed-campaign latency
cascade (parallel openers break the linear-chain delta model) is tracked
separately.
download_with_verify() no longer fails on Content-Encoding: gzip
objects. GCS applies decompressive transcoding on download, so the saved
file is the decompressed size while the object metadata's size is the
compressed byte count -- the byte-for-byte size check therefore always
failed (after retries) for a gzip-stored object. Campaign export CSVs
(campaign_results/<id>/..._raw_data_download.csv) are stored gzip-encoded,
so a fleet run downloading them errored on every campaign. The size check is
now skipped (with a message) when the object carries a Content-Encoding;
unencoded truncation is still caught, and any HTTP failure still errors.
The export poll no longer masks a GCS listing failure as "still running".
s160_api_campaign_results() polls GCS for the export to appear; the poll
helper previously swallowed every gcs_list_objects() error to NULL,
which a persistent failure (auth/permission/bucket -- e.g. a forgotten
s160_gcs_init()) turned into a spin until a misleading "Export timed out"
error. A genuinely-absent file yields an empty listing (not an error), so such
failures now surface immediately with a clear Failed to list campaign export files: <cause> message.
Not-found errors are raised as classed conditions instead of matched by
message text. Control flow that previously grepped error strings
("Bad Request"/"Not Found" in s160_api_campaign_get(), "404" in
s160_gcs_campaign_results_read() and disposition_pull()) now dispatches on
condition class: HTTP failures carry an s160_http_error class with the
numeric status, and a GCS 404 is translated once at the
download_with_verify() boundary into an s160_not_found condition the
callers catch by class. The boundary now matches googleCloudStorageR's
actual missing-object message ("File not found...", which carries no
"404") as well as the http_404 form -- the previous bare "404" string
check silently missed a real missing object, surfacing it as a generic
"Failed to download" instead of a clear not-found. Otherwise messages are
unchanged; control flow is just no longer brittle to an upstream reword.
disposition_pull() now checks GCS readiness before downloading. Because
it always resolves a concrete s160_disposition_<env> bucket, the shared
resolve_bucket() readiness check never fired, so calling it before
s160_gcs_init() failed with a raw googleCloudStorageR error wrapped as
"Failed to download" rather than the standard GCS not initialized. Run s160_gcs_init() first. It now calls check_gcs_ready() on the download path
(a cache hit is still served without auth), matching every other GCS entry
point.
API requests now have a bounded timeout and retry transient failures.
Every call in s160_api_* (authentication, the export trigger, campaign reads)
previously had no httr::timeout(), so a hung server could block the R session
or wedge the scheduled producer indefinitely with no error; and a single
transient failure -- a network/curl error or a retryable HTTP status -- aborted
the run. Requests are now bounded to a per-request timeout
(.http_timeout_seconds, 60s) and retried with exponential backoff (up to
.http_max_retries, 3, on network errors and HTTP 429/500/502/503/504). Every
other status is terminal and fails fast -- 4xx client errors and non-transient
5xx (e.g. 501/505) -- so a not-found (s160_api_campaign_get) still errors
immediately.
Retrying the export-trigger POST is safe -- the server just regenerates the
results CSV.
latency_report() no longer duplicates the day-rollup grain when a segment
drops off mid-flow. A blank or unparseable prior-question timestamp gives a
segment an NA segment_date_local (and therefore NA hour_local), so it
already landed at the (date=NA, hour_local=NA) unknown-time bucket in the
hour pass -- which the day pass then re-emitted with the identical key, and the
rbind double-counted it (one drop-off row produced ~12 duplicate grain keys
and inflated the hour_local IS NULL day partition). The hour pass now drops
its own NA-hour rows before stacking, so the (hour=NULL) unknown bucket
lives in the day partition only and the consolidated grain
(campaign_id, date, hour_local, segment, threshold_min) is unique. The
happy path (no drop-offs) is unchanged.
download_with_verify() now actually verifies download size (affects
s160_gcs_campaign_results_read() and disposition_pull()). The expected
size was read from gcs_list_objects(),
whose size is a human-readable string ("483.3 Kb") at every detail
level, so as.numeric() produced NA and the size check plus its retry loop
never ran against real GCS -- a truncated-but-HTTP 200 download was accepted
silently and fed to the reports. The expected size now comes from
gcs_get_object(meta = TRUE), whose size is the exact byte count; a
mismatch retries with backoff and then errors. When object metadata is
unavailable (permissions / transient error) verification is skipped with a
message rather than silently. Also removes the redundant per-download
gcs_list_objects() round-trip.
Rewrote the two vignettes as task-oriented how-to guides.
vignette("latency") and vignette("disposition") now share one structure
(task -> variations -> what you get back -> notes -> see also), address a
single R-competent Survey Manager, and follow a new documentation style guide
(writing-style.md). The _pkgdown.yml reference index is reordered to match
the README (raw data access -> latency -> disposition).
README onboarding and troubleshooting. Added an access checklist to First-time setup (client secret, bucket grant, API keys), an R-universe version-lag note to Installation, a "try it without credentials" example, a "new here?" signpost, and a symptom -> cause -> fix Troubleshooting table (403, not-initialized, stale cache, missing function, wrong-argument, empty date filter). The disposition guide now states the pull's download size/time and how to refresh a stale cache.
Dropped the "Survey Manager" audience framing. The whole package is for
Survey Managers, so singling out one surface as the Survey-Manager surface
was redundant -- the label is removed from the README, both vignettes, the
package help, and code comments, and the per-section audience tags in
?survey160r are dropped (Disposition screening / Latency analysis, no
parenthetical). The README also reorders the three surfaces to raw data access
-> latency -> disposition and links the two guides as clickable relative paths
so they render on GitHub. Credential prompts now say "your team lead" instead
of "your survey manager." Reference docs (@param / @return) describe the
argument or value rather than naming a person, per the R convention.
Docs restructure: vignettes + a slimmer README. The latency and
disposition walkthroughs moved out of the README into full-length articles --
vignette("latency") and vignette("disposition"), rendered as Articles on
the R-universe package site -- and the README is now an overview + install +
disposition quick start + raw-data-access + setup, pointing at the guides.
Adds knitr / rmarkdown (Suggests) and VignetteBuilder: knitr, plus a
_pkgdown.yml grouping the function reference by family (disposition /
latency / raw access). Along the way the latency "Result shape" table was
corrected -- it had drifted to ~14 of the 30 consolidated columns, omitting
survey_mode, the delta distribution, the NA diagnostics, and the n_texted
/ n_engaged / n_consented / n_completed / n_ineligible summary block --
and the two raw-data-access sections were consolidated. The disposition
vignette's runnable example (disposition_summary() on a frame) executes at
build time.
Package-level help (?survey160r). A new overview page orients a reader to
the three layers -- disposition screening, latency
analysis, and the raw s160_* data-access functions -- with pointers to the
entry point for each. The README intro now names all three surfaces and links
to them, and the "Disposition screening" section is expanded with the
end-to-end screening workflow, a table of the columns
disposition_screen() appends, the latest_disposition funnel categories, a
read-once performance note for screening several samples, and the beta caveats.
Corrected and expanded documentation (no code behavior change). The
CLAUDE.md naming-conventions section now matches the current disposition
surface -- disposition_query was renamed to disposition_summary; the
disposition readers are documented as bare-but-derived and disposition_pull
as the one bare function that reaches GCS -- and its Imports list adds
data.table and nanoparquet. The README gains a "Disposition screening"
section (disposition_pull() / disposition_summary() /
disposition_screen() / disposition_records()).
Runnable @examples were added to the pure functions (latency_report(),
latency_build_config(), latency_validate_config(), latency_config_hash(),
latency_discover_questions()) and usage examples to
the reader functions; s160_csv_header()'s example moved out of the
description into a runnable @examples. Fixed disposition_screen()'s
@return to distinguish an absent-but-valid phone (a never_contacted row)
from an unparseable one (an all-NA block).
The export-timestamp-column resolvers share one core (no behavior change).
question_timestamps(), the opener resolver .question_timestamp(), and the
disposition ineligible/refusal reads now go through a single null-safe
.column_timestamps() (resolve id.<q>.<field>, parse via
parse_campaign_timestamps(), all-NA when the column is absent), replacing the
three copies of that logic.
Refactored the disposition readers (no behavior change).
.disposition_filter() now builds one combined keep-mask and subsets the frame
once instead of allocating an intermediate copy per predicate;
.disposition_collapse() groups by phone through a single by_phone() helper
in place of six near-identical tapply() calls; and the duplicated
phone-request normalization in .disposition_rollup() / disposition_records()
moved into a shared .disposition_request_phones(). On the test side, the two
byte-identical disposition-Parquet fixture writers collapsed into one shared
write_disposition_parquet() helper, and a new test pins that
disposition_screen() appends exactly what disposition_summary() computes.
Standardized and consolidated error messages. Every stop() in the
package now follows one documented convention (see R/conditions.R):
exported, user-facing functions prefix the message with "<fn>: " so a
call. = FALSE error still names the failing call; internal helpers stay
bare; argument and config-key names are backtick-quoted; value references
follow a colon. The recurring shapes -- not-initialized, non-empty-string,
not-found, failed-to, and data-frame checks -- are routed through shared
helpers (stop_s160(), check_nonempty_string(), stop_not_initialized(),
stop_not_found(), stop_failed(), check_data_frame()), removing the
duplicated raise-and-format logic (including three identical
gcs_list_objects() tryCatch blocks and two file-download error mappers).
Message wording is unchanged except where it made a concept consistent
(File not found -> file not found; dropped stray function-name prefixes
from internal helpers). No behavior change beyond the reworded text.
campaign_* to latency_* so each function
name matches the view it computes (the campaign entity is still
campaign_id): campaign_run to latency_run, campaign_report to
latency_report, campaign_build_config to latency_build_config,
campaign_validate_config to latency_validate_config, campaign_config_hash
to latency_config_hash, campaign_discover_questions to
latency_discover_questions, and required_csv_columns to
required_latency_columns.disposition_run() now returns a list (consolidated + meta), mirroring
latency_run(), instead of a bare data frame. Read the frame from
result$consolidated; result$meta carries the source source_csv_hash /
source_csv_path.required_disposition_columns() returns the exact (dot-form) CSV columns
disposition_run() reads, so a caller can project a wide export down to just
those columns and get output identical to a full read -- the disposition
analogue of required_csv_columns(). It includes phone (the row key) and
excludes campaignid (the campaign_id is stamped from the argument). Pass
available (e.g. s160_csv_header()) so the data-dependent close-message
Text columns are retained.disposition_run() now returns contacted records only by default. A new
contacted_only argument (default TRUE) filters the frame to rows where an
intro was dispatched (started == 1) -- the contacted/attempted records a
disposition table represents. Non-responses (contacted but no reply) are kept;
only never-attempted rows are dropped. The uniqueness guard and survey-mode
classification still run on the full data, so the filter never changes mode
or masks a duplicate. Pass contacted_only = FALSE to restore one row per
input respondent.disposition_run(campaign_id, data) turns a
campaign's per-respondent results CSV into the disposition frame: one row per
phone with 0/1 funnel flags (started, engaged, opt_in, complete,
web_complete, terminated) plus the campaign's mode. Grain is one row per
(phone, campaign_id), enforced with a duplicate-phone guard. complete
follows the survey-mode rule (web_complete for t2w, id.close.scriptDate
for sms, NA for t2w_external). Algorithm only -- persistence (enrichment
and Parquet output) lives in consumer projects.Multi-environment API sessions, one entry point. s160_api_auth(env)
now takes an environment name -- "prod" (default) or "staging" -- and
returns a connection: an opaque handle carrying the JWT, credentials, base
URL, environment name, and the paired campaign-results GCS bucket. Addressing
by name resolves the URL, bucket, and key var together, so they cannot be
mismatched (no pointing a prod URL at a staging key/bucket). Credentials come
from ~/.Renviron: S160_API_USERID for the user, and a per-environment key
var -- S160_STAGING_API_KEY for staging, S160_PROD_API_KEY (falling back
to the legacy S160_API_KEY) for prod -- prompted and saved on first
interactive run.
s160_api_auth()
refreshes the package default connection and conn-less
s160_api_campaign_results() / s160_api_campaign_get() calls use it.conn = argument. prod <- s160_api_auth("prod"); stg <- s160_api_auth("staging"), then s160_api_campaign_results(744, conn = stg).
Prod and staging stay live in the same session -- e.g. comparing a campaign
across both. Each connection's paired bucket makes the export trigger,
completion poll, and read all target its own environment (closing the
footgun where a staging export was polled against the production bucket).The in-session JWT refresh reuses the credentials stored on the connection, so
a staging connection held alongside prod keeps refreshing against staging.
Breaking: s160_api_auth() no longer takes a base_url argument (use
env) and now returns a connection object (invisibly) instead of NULL;
existing zero-argument single-environment calls are unaffected.
diagnostics$respondent_summary cascade percentages
(pct_clean_at_5min, pct_worst_in_5_to_10, pct_worst_over_10) are now
computed over the measured respondents (those with at least one valid
Delta), matching respondent_summary$n_respondents. They previously
divided by every observed respondent, including those with no valid
segment, so the buckets were deflated by the no-valid fraction and summed
to less than 100% -- and n_respondents * pct / 100 did not recover a
respondent count. The consolidated cascade and legacy-parity definitions
already used the measured-respondent denominator; the diagnostics summary
now agrees. When no respondent has a valid segment the percentages are
NA (as on the empty-frame path) rather than 0.Faster CSV reads via data.table::fread. s160_read_csv() and
s160_gcs_campaign_results_read() now read through data.table::fread
(multithreaded; ~5-10x faster than utils::read.csv on large exports),
falling back to read.csv if data.table is unavailable. data.table
is now a hard dependency -- on Windows it installs as a precompiled
R-universe/CRAN binary (no Rtools needed) and is usually already present.
Output is pinned to stay close to the old read.csv behaviour
(stringsAsFactors = FALSE, check.names = TRUE, and crucially
integer64 = "character" so large IDs come back as character strings
rather than a bit64::integer64 class). Existing calls are unchanged --
this is a transparent speedup.
Column projection for the latency pipeline. New exported helpers
required_csv_columns(config) and s160_csv_header(path), plus a
columns = argument on s160_read_csv() /
s160_gcs_campaign_results_read() / s160_gcs_pull_csv(). Passing the
algorithm's required column set lets fread parse only those columns,
cutting read time and (importantly for parallel fleet runs) per-worker
memory on very wide survey exports. The projection keeps the
non-flow columns the report depends on (id.intro.finalText,
web_complete, id.ineligible.scriptDate) so output is identical to a
full read.
Optional provenance hashing. s160_read_csv(path, hash = FALSE)
skips the sha256 digest() pass (a full second read of the file),
setting source_csv_hash = NA. Useful for large local backfills where
the per-file hash is not needed.
survey_mode. Adds a third survey_mode
value "t2w_external" -- a personalized survey link in the close
message but no web completes (external platform, no webhook).
Completion is not computable from the export, so n_completed is
NA (n_texted / n_consented remain valid). A "survey link" is
detected as a personalized URL in the close message (one that varies
per respondent); a single static stimulus link (e.g. a shared video
URL) is not.Text-to-Web support + survey_mode column.
consolidated gains a per-campaign survey_mode column, classified
from the source CSV:
"t2w" -- web completes present; n_completed counts the
web_complete callback (not id.close.scriptDate, which for
Text-to-Web is just the link sent to every consenter and overstated
completion 2-7x, making n_completed == n_consented)."sms" -- no web completes and no survey link; live SMS, completes
on id.close.scriptDate (unchanged).The authoritative campaign flag (campaigns.use_web_completes) is not
in the CSV export, so this is a data-only heuristic.
pct_le is now always a numeric (double) column, even when a
campaign has no valid latency cells and every value is NA. The
populated assembly path took pct_le straight from the joined
frame without the as.numeric() coercion its sibling numeric
columns use, so an all-NA join result collapsed to a logical
vector. Downstream the fleet writer casts this column to a
float64 Arrow schema; a logical vector failed that cast
(Invalid: cannot convert) and silently dropped the campaign's
Parquet output. Affected campaigns are valid but degenerate --
every recipient hit a carrier delivery error or sat in limbo, so
none produced a measurable latency delta.consolidated now carries four denormalised summary metrics
columns (Phase 1 PR 4, spec §4): n_texted, n_consented,
n_completed, n_ineligible. Counts are anchored by
id.intro.batchDate -- cohort-by-send-time, matching the latency
view's hour bucketing. n_texted is the count of rows with a
non-NA id.intro.batchDate (intro dispatched); n_consented is
the subset that passes config$filters$population (re-using the
existing consent definition rather than a parallel
finalValue/finalText anchor); n_completed is the subset with a
non-NA id.close.scriptDate. n_ineligible is per-segment: the
count of respondents whose id.ineligible.scriptDate is non-NA
AND whose last reached question lands at the segment's endpoint,
joined to latency cells on (campaign_id, date, hour_local, segment_index). The four counts denormalise across the latency
rows that share their bucket keys; Parquet RLE compresses the
repetition. .algorithm_version bumps to "2.1.0",
.schema_version to "4". Consumers that don't read the new
columns are unaffected; consumers that do can gate on a
has_summary_data probe.
Scaffold-first consolidated seeding. aggregate_consolidated()
now builds output rows from the UNION of latency-frame and
summary-frame bucket keys (cross-joined with segments × thresholds),
not just from latency cells. This preserves summary-only buckets:
hours where every respondent was filtered out (e.g. 100 texted, 0
consented) still appear in the parquet with n_texted populated
and latency cell counts at 0 / NA. Without this, the pre-filter
summary contract would be defeated whenever the population filter
rejected an entire hour.
Symmetric NA → 0 backfill for count columns. Scaffold rows with
no matching summary or ineligible row fill all four summary counts
(n_texted, n_consented, n_completed, n_ineligible) plus the
existing latency count columns (n, n_le, n_resp_over,
n_na_*) with 0L. The previous design left summary counts as NA
and only filled n_ineligible to 0 -- consumers couldn't tell
"no data" from "no respondents", and the asymmetry was a footgun.
date_filter now restricts the summary view too. Previously
date_filter only narrowed the latency frame; the summary
computation ran on the full pre-filter population. Symmetric
semantics ("show me this date's data") matches user intent and
avoids the case where a date_filter that excludes everyone still
emits summary rows for the excluded dates.
The latency pipeline is renamed to the campaign pipeline -- the per-campaign Parquet is becoming a general per-campaign metrics artifact (latency view today, summary metrics view next). All orchestrator exports rename:
| Before | After |
|---|---|
| latency_run() | campaign_run() |
| latency_report() | campaign_report() |
| latency_build_config() | campaign_build_config() |
| latency_validate_config() | campaign_validate_config() |
| latency_config_hash() | campaign_config_hash() |
| latency_discover_questions() | campaign_discover_questions() |
The latency sub-view files (R/latency_aggregate.R,
R/latency_frame.R, R/latency_filter.R,
R/latency_diagnostics.R, R/latency_primitives.R) keep their
names -- they implement latency-specific computations and sit
alongside the new orchestrator files as one named view of the
campaign pipeline. Behaviour is unchanged; output Parquet schema
is byte-identical to 0.12.0. algorithm_version stays "2.0.0"
because the algorithm did not change; the rename is API only.
The algorithm spec doc moves from r-scripts/latency_scripts.md to
r-scripts/campaign_scripts.md (lives in the meta-workspace, not
this repo).
consolidated now carries seven new per-cell columns:
mean_delta_min, p50_delta_min, p90_delta_min, p95_delta_min
(distribution shape, threshold-independent so identical across the
four threshold rows of a cell) and n_na_parse, n_na_missing,
n_na_chain (per-cell NA-reason counts derived from
na_reason). .schema_version bumps to "3". The new columns
unlock per-cell distribution and data-quality visualisations
downstream; existing consumers that read columns by name are
unaffected.run_latency() is renamed to latency_run() and is now
source-agnostic. The signature is
\code{latency_run(campaign_id, data, config = NULL, run_at = NULL,
run_by = NULL, ...)}: data is a caller-supplied data frame, so the
function works equally well for CSVs pulled from GCS via
s160_gcs_pull_csv() and for off-GCS sources (Dropbox, local disk,
S3, etc.). bucket, source_bucket, uploader, field_timezone,
project_id, date_filter, and respondent_id_column are no
longer arguments on latency_run() itself; the build-config knobs
flow through ... to latency_build_config(). Optional config =
lets callers pre-build (and mutate) the config, skipping the
auto-build. The function returns the result list from
latency_report().pull_csv_from_gcs() is renamed to s160_gcs_pull_csv() to match
the s160_gcs_* family; behaviour unchanged.s160_read_csv(path, ...) reads a CSV from a
local path and stamps the same source_csv_hash /
source_csv_path attributes that s160_gcs_pull_csv() does. Use
for backfilling archived campaigns from disk / Dropbox / S3 mounts;
hand the result to latency_run() and provenance flows through to
result$meta like it does for active GCS campaigns.latency_report() now populates result$meta$source_csv_hash and
result$meta$source_csv_path from the input data's attributes (in
addition to stamping consolidated$source_csv_hash per-row). Meta
survives data-frame subsetting and is the contract downstream
persistence layers should read.run_latency_all(), write_to_gcs(), s160_gcs_latency_output_status(),
and read_latency() are removed. The first three move to
a consumer project; read_latency() had no in-tree consumers and is
dropped (reintroduce if a real consumer surfaces).scripts/bulk_reprocess.R is removed; a consumer project's fleet
entry point is now the supported path.future, future.apply, duckdb, and DBI leave Suggests. arrow
leaves Imports (no remaining call sites in this package).parse_s160_timestamps_chr(), added a safe_pct() helper
for the "percent of X, NA if denominator is zero" pattern, encapsulated
the data + parse-failed-mask plumbing behind subset_parsed_input(),
extracted classify_na_reason() from the segment loop, and split
aggregate_consolidated() into per-aggregation helpers
(aggregate_totals(), aggregate_worst_cascade(),
aggregate_segment_cells(), assemble_consolidated()). Numeric output
is unchanged; the refactor only reshapes the call graph.run_latency_all() parallelizes per-campaign processing and skips
campaigns whose source CSV is unchanged since the last run.
(run_latency_all() itself is later removed in 0.11.0, when fleet
orchestration moved to a consumer project.)(campaign_id, date, hour_local, segment, threshold_min)
with hour_local 0-23) for time-of-day analysis, plus day rollup rows
(hour_local = NA) carrying correct day-grain n, pct_le, and
respondent-cascade columns. Downstream consumers filter on
hour_local IS NULL for day rollups, hour_local IS NOT NULL for
time-of-day; both are arithmetically correct without any further
rollup. The time_bucket config knob and the reports config slot
are removed -- latency_build_config() no longer accepts a
time_bucket argument, and validate_config() rejects reports as
an unknown key. Existing Parquets in gs://s160_analytics_*/latency/
(which carried only one grain) must be regenerated via the
a consumer project's fleet runner.n_respondents
over-counts cross-hour respondents (a respondent active in two hours
appears in both hours' distinct-respondent counts). Always read the
day rollup row (hour_local IS NULL) for correct day-grain cascade;
do not attempt to recompute it by aggregating the hour rows.texting_windows config field is removed. The algorithm no longer
filters dispatches by an analyst-declared texting plan; n and
pct_le now count every valid dispatch. The pre-removal feature
excluded out-of-window dispatches from the in-window denominator;
with the cube schema introduced in this release downstream consumers
can see which hours had high volume directly from the hour rows.
Diagnostics field n_out_of_window_dropped and
windows_normalized_utc are dropped along with the feature.
latency_build_config() and latency_run() no longer accept a
texting_windows argument.run_latency() no longer takes a config_path argument. The function is
now stateless: it derives flow.questions from the CSV header (via the
new discover_questions()) and assembles the rest of the config from its
named arguments. Sensible defaults are baked in (
field_timezone = "UTC", project_id = campaign_id,
texting_windows = list()); each is overridable via a named argument.
run_latency() no longer requires s160_api_auth() -- the config is
derived from the CSV alone.read_config() and the YAML config schema are removed entirely. Configs
are now built programmatically via build_config() or as hand-written
lists with the same shape. The yaml package is dropped from Imports.
Existing per-wave YAMLs under latency-scripts/*.yaml must be translated
to run_latency(..., field_timezone=..., project_id=..., texting_windows=..., date_filter=...) calls; the YAML files themselves
are retained outside this repo as historical record.latency_report() actually
reads: project_id, campaign_id, field_timezone, flow, filters,
texting_windows, reports. Previously accepted but never-used keys
(project_name, wave_run, display_timezone,
reports$extra_grouping_columns, input, output) are no longer
recognized; validate_config() rejects them as unknown.date and hour_local columns are now bucketed in UTC by
default. Callers consuming
gs://s160_analytics_*/latency/*_latency.parquet that previously
depended on an America/New_York-bucketed output must pass
field_timezone = "America/New_York" explicitly.discover_questions(data) derives the question flow from CSV column
names (either a data frame or a character vector of header tokens).
Accepts both the raw id[<q>]scriptDate bracket form and the dotted
id.<q>.scriptDate form produced by read.csv(). Terminal flow states
(refusal, ineligible) are dropped.build_config(campaign_id, data, ...) is a pure function that assembles
a validated config from the CSV header alone. Named arguments for every
override (field_timezone, project_id, texting_windows,
date_filter, respondent_id_column, time_bucket). No I/O, no API
call.pull_csv_from_gcs() now stamps a source_csv_path attribute on the
returned data frame (the canonical gs://... URI) alongside the
existing source_csv_hash. Lets downstream callers record provenance
without re-deriving the path.bucket (or source_bucket) argument that defaults to the global set
by s160_gcs_init(). Callers can either keep using s160_gcs_init()
once-per-session or pass bucket = "..." per call and skip the global
entirely. run_latency_all() no longer needs to stash/restore the
global bucket since its inner calls thread source_bucket through
every layer. Affects s160_gcs_campaign_results_read,
s160_gcs_campaign_results_list, s160_gcs_campaign_results_files,
s160_gcs_campaign_results_status, pull_csv_from_gcs, and
run_latency.R/latency_report.R (531 lines) is split into five cohesive files:
latency_report.R keeps the orchestrator and shared constants;
latency_filter.R holds the population / dedupe / date filters;
latency_frame.R holds the per-respondent x per-segment frame builder;
latency_aggregate.R holds the consolidated-table aggregation;
latency_diagnostics.R holds the diagnostics-list assembly. The %||%
operator (used in three files) moves to aaa_utils.R. Pure internal
refactor; no behavior change, verified by the legacy-parity test.latency_* namespace to prevent collisions with other R packages and
signal cohesion: discover_questions -> latency_discover_questions,
build_config -> latency_build_config, validate_config ->
latency_validate_config, config_hash -> latency_config_hash. The
old names are removed without a deprecation period; callers using the
pre-0.8.0 names must update.latency_report() and run_latency() accept an optional run_at
argument (defaults to Sys.time()). run_latency_all() stamps a single
fleet-wide timestamp on every campaign in one pass so the latest fleet
output can be selected with WHERE run_at_utc = (SELECT MAX(run_at_utc) FROM latency).run_latency_all(source_bucket, bucket, ...) runs the latency pipeline
for every campaign with an export CSV under source_bucket and writes the
per-campaign Parquet to bucket. Per-campaign failures are caught by
default (continue_on_error = TRUE) and recorded in the returned status
data frame so one bad CSV does not block the rest of the fleet. Saves
and restores the global GCS bucket so the caller's session state is
untouched. Replaces the bespoke iteration loop in
scripts/bulk_reprocess.R, which is now a thin shell wrapper around
this function.scripts/bulk_reprocess.R is refactored to call run_latency_all();
the inline discover_questions, build_config, and process-one helpers
are removed, and the script no longer needs API auth.download_with_verify() no longer crashes when googleCloudStorageR's
gcs_list_objects() returns a human-readable size string (e.g.
"483.3 Kb"). The previous code did as.numeric(size), got NA, then
hit if (actual_size == NA) and aborted with "missing value where
TRUE/FALSE needed". A non-numeric size is now treated as "unknown" and
the download proceeds without verification. Discovered while running
run_latency against the production campaign_results bucket.R (>= 4.1) in DESCRIPTION to match what the current arrow,
dplyr, and lubridate imports already require.respondent_id_column: ~ instead of
the misleading userid, which in Survey160 v2 CSVs is the agent login
rather than a per-respondent identifier.s160_api_campaign_get() now strips sub-second precision when parsing
ISO-8601 timestamp columns, so values like
"2026-01-15T09:30:00.123456Z" (which PostgreSQL can emit) come back as
POSIXct rather than falling through to the string fallback. Numeric UTC
offsets (+05:30, -0400) are also covered. The httr::GET import is
now declared explicitly to match the other httr imports.s160_api_campaign_get(campaign_id) reads a single campaign's attributes
via GET /campaigns/<id>. Returns a single-row data frame with the
campaigns table columns; enriched API-only fields (listlength,
list, login, exports, has_texting_started, sandbox_configuration,
aggregator, has_assigned_registration) are dropped, and JSON columns
(script, prompt, quotas, ...) come back as length-1 list-columns.
Useful for confirming attributes after a state-changing call without
dropping to direct database access. Per-campaign read; not intended for
tight loops over hundreds of IDs. ISO-8601 timestamp columns
(startdate, archive_scheduled_date, ...) are parsed to POSIXct
in UTC so callers do not have to re-parse them.RELEASING.md clarifies that the release tag must point at the release
PR's merge SHA, not HEAD (#14).Latency analysis pipeline (#13). Supersedes the per-wave inline R scripts that the analytics team used to maintain by hand with a single algorithm, output schema, and YAML config per campaign; existing wave scripts will be migrated client by client. New public functions:
latency_report(data, config) -- pure, deterministic; returns
consolidated, latency_frame, diagnostics, meta.read_config(path) / validate_config(config, data) -- YAML loader
plus fail-fast schema and flow-order validation.pull_csv_from_gcs(campaign_id) -- thin wrapper that also computes a
source CSV sha256 for provenance.write_to_gcs(result, campaign_id, bucket, uploader = upload_object) --
writes one Parquet per campaign to
gs://<bucket>/latency/<campaign_id>_latency.parquet with a pinned
Arrow schema, ZSTD compression, and provenance columns
(algorithm_version, config_hash, source_csv_hash, run_at_utc,
run_by). Accepts a custom uploader for batch jobs and tests.read_latency(bucket) -- returns a DuckDB connection and a latency
view over all per-campaign Parquet files.run_latency(...) -- orchestrator for the manual happy-path flow.Fleet-locked universal latency thresholds (1, 3, 5, 10 minutes). Configs that still carry per-wave thresholds are rejected with a named error.
Per-segment NA classification in diagnostics (parse_failure,
missing_endpoint, chain_break); sum-conserving against
n_segments_na.
Legacy-parity CI gate: a generic re-implementation of the four legacy
primitives (timestamp_diff, texting_hour_by_date,
percent_below_thresholds_data, latency_indicator_vars) asserts
cell-for-cell match against the new pipeline on a synthetic fixture.
New dependencies: arrow, lubridate, yaml, digest, dplyr, rlang.
duckdb and DBI are Suggests (required only for read_latency).
s160_gcs_campaign_results_read() verifies the downloaded CSV size
against the GCS object metadata and retries on truncation (#9).R/, man/, or src/ without bumping Version: in
DESCRIPTION now fail the check workflow (#11).ghcr.io/r-hub/containers/ubuntu-release
image instead of installing R from scratch, cutting workflow time
substantially (#10).s160_api_auth() reads the Survey160 API key from .Renviron
instead of taking it as a function argument, and masks the secret
in error output. README updated with the new setup flow and pak
install instructions (#8).s160_api_* and s160_gcs_campaign_results_* namespaces
let R callers kick off a fresh export and then read it back from
GCS in one workflow (#7).inst/oauth-client.json; on first interactive run,
s160_gcs_init() prompts for the client secret and persists it to
~/.Renviron. bucket is now a required named parameter on the
GCS readers to prevent silent reads from the wrong environment
(#2).s160_gcs_campaign_results_read() gains a destdir parameter for
persistent downloads (default is a tempdir that is cleaned up on
exit) and sanitizes the resolved filename (#4).R CMD check and the testthat suite on every
push and PR; warnings fail the build (#3).lintr runs in CI and fails the build on any lint violation (#5).covr reports test coverage on every CI run; the threshold is
enforced at 100% (#6).Initial release. Converts the previous loose script collection into a
proper R package with a DESCRIPTION, NAMESPACE, exported help
pages, and a testthat suite that runs offline via mocks. Public
surface area:
s160_gcs_init() -- OAuth bootstrap for Google Cloud Storage.s160_gcs_campaign_results_read() -- download and parse a campaign
CSV from the configured bucket.s160_gcs_campaign_results_list() -- list available campaign IDs.s160_gcs_campaign_results_files() -- enumerate files for one
campaign.Internal: validate_campaign_id() is a shared input guard reused by
the GCS readers; not exported.
Published to R-universe at
https://survey160.r-universe.dev (#1).