Prospective replay and complete-policy analysis utilities
7 October 2026 Pacific (America/Los_Angeles)

These Python standard-library utilities and authored tests are preparation, not real-model results, scientific adoption, calibrated risk guarantees or a registered protocol. They do not launch models or collect study data.

After extracting, run separately in the two subdirectories:
  cd baselines
  python3 -B -m unittest test_baselines test_full16
  python3 -B -O -m unittest test_baselines test_full16

  cd selected-risk
  python3 -B -m unittest test_selected_risk test_paired
  python3 -B -O -m unittest test_selected_risk test_paired

The baseline tests comprise 16 authored cases; the analysis tests comprise 12. Independent source review adds separate checks not included in these test counts. The sixteen-candidate maximum is prospective capability; this package is not a qualified original collector or serving launcher.

Baseline controls: first valid, first public pass, single judge A/B and fixed consensus. Minimal schedules share a candidate bank and budget ceiling, not equal realized expense/information. matched_all buys identical information for each selector, deliberately including information some selectors ignore. Every revealed action is charged; unknown expenses halt. Real cost provenance and model correctness remain external.

Analysis: complete policy x task x three-seed terminal records; equal task weighting; pooled selection-error risk, coverage and unconditional success; unknown-outcome bounds; paired whole-task bootstrap differences. All-abstain risk is undefined. Unknown cross-policy bounds are conservative because shared candidate truth linkage is absent. Percentile envelopes are descriptive, not finite-sample coverage or shifted-distribution guarantees. Source fitting, complete authenticated policy traces, cost accounting and actual calibration have not run here.

Neither candidate-score calibration nor a learned continuation value establishes complete-policy error risk. Supplied coefficients/threshold examples are scaffolds only. These utilities preserve prior frozen study inputs and require a separate reviewed protocol before scientific adoption.

paired.compare suppresses its risk envelope whenever a draw may have undefined risk; selected_risk.bootstrap instead retains positive-coverage bounds with explicit possibly-undefined counts, so its envelope is conditional on positive coverage.