---
title: "Latency analysis"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Latency analysis}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
# Examples read a real campaign CSV / need GCS auth, so they are shown but not
# run at build time.
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = FALSE)
```

Turn a campaign's raw results CSV into a per-campaign recipient-latency report -- how fast each message followed the previous one, and what share of recipients hit the fleet's target windows -- so you can tune send timing. Assumes you have completed First-time setup (the package README, or `?s160_gcs_init` from R) if you read the export from Cloud Storage; a local CSV needs no sign-in.

## Build a report

Read the campaign's results, then run the report:

```{r}
library(survey160r)
s160_gcs_init()            # one browser sign-in, cached
data   <- s160_gcs_campaign_results_read(1234, hash = TRUE)  # or s160_read_csv("path/to/1234.csv")
result <- latency_run(1234, data, field_timezone = "America/New_York")
```

You already read these CSVs in your own scripts; `s160_gcs_campaign_results_read(hash = TRUE)` and `s160_read_csv()` just add the source-hash provenance that `latency_run()` records on `result$meta`. `field_timezone` sets the local time zone for the hour-of-day buckets; it defaults to `"UTC"`, so pass the zone the campaign fielded in (`"America/New_York"`, `"America/Chicago"`, ...) or `hour_local` is off by the offset. `latency_run()` builds the config from the CSV header and returns a list: `consolidated` (the report), `latency_frame` (per-respondent detail), `diagnostics` (parse / NA counts), and `meta` (provenance).

## Customize the config

`latency_run()` derives the config from the CSV header; override through `...`, or build it explicitly to inspect first, then validate before running:

```{r}
config <- latency_build_config(1234, data,
                               field_timezone = "America/New_York",
                               project_id     = 9999,
                               date_filter    = "2026-01-26")   # keep a single field date
latency_validate_config(config, data)   # fail fast: required columns, flow order, no terminal states
result <- latency_report(data, config)
```

## Run deterministically

`latency_report()` is the pure entry point (no I/O). Pass `run_at` to fix `run_at_utc` -- otherwise it comes from the clock -- for reproducible output and tests:

```{r}
latency_report(data, config,
               run_at = as.POSIXct("2026-01-27 12:00:00", tz = "UTC"))
```

A hand-built data frame works too; `result$meta` provenance is then `NA`, which is correct for that case.

## Reduce a report to funnel counts

`consolidated` is denormalized -- the funnel anchors repeat across every segment and threshold row, and the frame carries both hour rows and a day rollup -- so summing a column over it multi-counts. `latency_funnel()` does the one correct reduction:

```{r}
latency_funnel(result$consolidated)                  # one row per (campaign, date)
latency_funnel(result$consolidated, grain = "hour")  # one row per (campaign, date, hour)
```

Each row carries `n_sent`, `n_engaged`, `n_opted_in`, and `n_completed`; summing an anchor across the `"hour"` result equals the matching `"day"` value.

## Count recipients reaching each question

For the "records reaching each question" view -- which stage sheds respondents -- pass the flow in order to `question_funnel()`. It reads the export directly, not the report:

```{r}
question_funnel(data, c("intro", "q1", "q2", "close"))
```

It returns one row per question with `n_reached` and `pct_reached` (share of the first stage). Underneath, `question_timestamps(data, "close")` reads and parses one question's timestamp column, and `parse_campaign_timestamps()` decodes the export's timestamp strings directly -- both operate on the raw export data frame.

## Run a large export

A wide, high-volume campaign fits a laptop with three levers, all output-identical to the default read and run:

```{r}
path   <- "path/to/1234.csv"
header <- s160_csv_header(path)
config <- latency_build_config(1234, header, field_timezone = "America/New_York")
data <- s160_read_csv(path,
                      columns = latency_input_columns(config, available = header),
                      timestamps = "POSIXct")   # 8 bytes per cell, not ~60
result <- latency_run(1234, data, config = config, compact = TRUE)
```

`columns` projects the read to the timestamp, population and campaign-id columns the report needs. `timestamps = "POSIXct"` guarantees compact timestamp columns even where a stray unparseable cell would otherwise keep a column as text (an unparseable cell is still counted in `diagnostics$parse_failures_per_column`). `compact = TRUE` computes the same `consolidated` and `diagnostics` segment by segment, keeping only the rows with a real send date (it never materializes the full respondent-by-segment frame), so peak memory follows real engagement rather than `(questions - 1) x respondents`; `latency_frame` then holds just those rows. A 1M-respondent, 12-question export runs in about 15 seconds within 4 GB.

## What you get back

`result$consolidated` is one row per `(campaign_id, date, hour_local, segment, threshold_min)` -- the shape a consumer project writes to Parquet:

| Column | Meaning |
|---|---|
| `campaign_id`, `project_id` | Wave identity |
| `survey_mode` | `sms` / `t2w` / `t2w_external` |
| `date`, `hour_local` | Bucket grain (`hour_local` is `NA` for the day rollup) |
| `segment`, `segment_index` | Consecutive flow-question pair (the two names joined by a right-arrow, e.g. `intro` then `q1`) and its position |
| `threshold_min` | Fleet threshold (1, 3, 5, or 10 min) |
| `n`, `pct_le` | Per-segment in-window dispatch metrics |
| `n_respondents`, `pct_resp_hit_gt`, `pct_resp_worst_gt` | Respondent-cascade metrics |
| `mean_delta_min`, `p50_delta_min`, `p90_delta_min`, `p95_delta_min` | Per-segment latency distribution (minutes) |
| `n_na_parse`, `n_na_missing`, `n_na_chain` | Segments with no computable latency, by reason |
| `n_sent`, `n_engaged`, `n_opted_in`, `n_completed`, `n_ineligible`, `n_refused` | Bucket volume: sent, engaged, opted-in, completed, screened out, declined |
| `algorithm_version`, `config_hash`, `source_csv_hash`, `run_at_utc`, `run_by` | Provenance |

`result$meta` carries `algorithm_version`, `schema_version`, `config_hash`, `run_at_utc`, `source_csv_hash`, and `source_csv_path`. See `?latency_run` for the full argument list.

## Notes

- **Run or build.** `latency_run()` composes `latency_build_config()` and `latency_report()`; reach for the pieces only when you need to inspect or override the config.
- **No respondent id by default.** `respondent_id_column` defaults to `NULL` -- `userid` is the agent login, not a per-respondent id.
- **Non-interactive runs.** For a scheduled job or a rendered report, authenticate with `s160_gcs_init(adc = TRUE)` -- Application Default Credentials, no browser.

## See also

`?latency_run`, `?latency_report`, `?latency_funnel`, `?question_funnel`, `?question_timestamps`, `?parse_campaign_timestamps`, `?latency_build_config`, `?latency_validate_config`, and `?survey160r` for every function by task. To screen a phone sample, `vignette("disposition")` and `vignette("opt-out")`. For the funnel across the fleet by registration or brand, `vignette("campaign-metrics")`.