Verifier Reliability Under Budgeted Code Generation

Cisco Caceres · Proposal v0.8 · 21 October 2026 Pacific · Operational status amendment

Status: protocol draft. Local development diagnostics and a separately frozen engineering pilot have run; the proposed matched joint-verifier hypothesis study, novelty and outside publication acceptance remain unestablished. Preregistration is planned and has not occurred. The reviewed readiness checklist specifies the remaining freeze, cost, missing-outcome and authorization gates; it is not a registered protocol.

Latest serving identity preflight · 21 October 2026 Pacific

The latest operator-provided Warden preflight was reported on 21 October 2026 Pacific; the shell output did not preserve its run time. The named Ollama process executable identity matched. Loaded-library qualification and tokenizer parity remained false, candidate calls remained disabled, and zero model calls occurred. This verifies process executable identity only; it does not establish loaded model weights, complete native-library closure, tokenizer/request-template parity, or any verifier-study result. The frozen 24-task MBPP/BigCodeBench H1 collection remains unrun.

When code-generation systems spend more compute choosing an answer, what happens if their verifiers share mistakes or their confidence stops matching correctness?

Execution evidence

The first response contract retained fourteen formatting failures. A separately frozen structured-module follow-up produced twelve candidates on six development tasks that passed public and hidden execution checks in an owned networkless VM. This all-positive set establishes neither verifier discrimination nor a selection benefit. The separately frozen 120-task engineering pilot is complete: 480 candidates, 480 verifier calls and all executable outcomes accounted for. First-candidate success is 109/120; higher-cap SoftBoN scores 91.39% with a conditional descriptive interval that includes zero gain. Only two candidate pools have mixed outcomes. This token-budget engineering result is not the proposed joint-verifier/natural-shift confirmation study or its dollar endpoint. Read the dated methods, failures and limitations.

Current execution status · 7 October 2026 Pacific

On 7 October 2026 Pacific, an owned, networkless VM built an authored wrapper around the pinned official native Jinja parser and completed 17 smoke cases: five accepted and twelve refused as expected. Configure, build and smoke exited zero; owned process groups, cgroup, guest and temporary files were cleaned up. Synthetic-metadata process/schema checks lack an independent expected-byte oracle. The binary hash is guest-reported; its deleted artifact was not independently rehashed. No model or counter calls ran. Installed renderer/tokenizer parity and H1 collection remain pending; this is an engineering milestone, not a hypothesis result or frozen-study amendment. Sanitized engineering aggregate.

A separate run on 7 October 2026 Pacific completed the unchanged 17 mechanics cases and six independently source-derived expected-byte checks, including four cases through the original default route and two separate template controls. All matched; independent review reconciled the requests, source commitments and cleanup receipts. Rendered bytes were not retained for a second direct observation, and the deleted binary’s hash remains guest-reported. The owned networkless VM was removed. This finite authored-metadata diagnostic strengthens rendering evidence; installed serving/tokenizer parity, real collection and the hypothesis study remain pending. Sanitized byte-check aggregate.

Scoped restoration replay · 6 October 2026 Pacific. A fresh owned, networkless VM completed a scoped restoration replay: eight registered native-file byte checks, 168 package versions, 12 ABI checks, 24 references, nine registered controls and the unchanged retained tree were reconciled. The guest was removed. Five historical source-build proofs were reused, not rebuilt. This establishes the registered finite checks, not exhaustive ELF dependency closure or collector admission. Exact serving model/tokenizer/template identity and the real 576-slot development collection remain pending. Read the scoped restoration aggregate and limits. Source-only fitting, calibration and the powered hypothesis study have not run. The completed 120-task engineering pilot remains separate from this preparation and the proposed joint-verifier study.

The question

The study asks whether modeling cross-verifier error dependence improves candidate selection and stopping under a fixed total inference budget and task-family shift, beyond strong single-judge, fixed-consensus and matched-retention filter baselines. The candidate contribution is a replication/extension study and reproducible decision benchmark, conditional on results that distinguish it from prior work. Adaptive compute, code-judge dependence and imperfect-verifier limits are established topics; this proposal claims no new method.

Correlated judge errors are documented by Kohli, and code-judge dependence has been screened directly by Hossain et al. Appendix K's code results are a feasibility screen: its local pointwise majority failed a competence gate, its pairwise bank had no matched-retention filter headroom, and its six-judge pointwise bank had 12 majority errors below the support gate. No code-routing experiment was run. This does not answer whether a budgeted generator policy can select candidates and stop more reliably on hidden outcomes under task-family shift. Han analyzes correlated conjunctive verifier cascades theoretically; Yin et al. address post-selection risk control, and Kwok et al. study repeated, decomposed and logit-derived verification. Any remaining contribution must survive these overlaps and the baselines below. Replication or a negative diagnostic is an acceptable outcome; a new verifier method is not assumed.

This version records literature findings before any confirmatory study. It changes no already frozen task nomination, data selection, collection, model setting, policy grid, hypothesis, effect-size target or statistical analysis. Baseline and policy-level calibration assessments below are requirements for a future reviewed protocol, not retroactive additions to an existing result or permission to tune on target outcomes.

Read the methods supplement: verifier information and missing-outcome bounds. Elementary authored derivations, with model-assisted proof checks; no novelty or empirical controller claim.

Prospective replay and policy-analysis tools · 7 October 2026 Pacific

A standalone source package implements charged fixed-rule baselines with up to sixteen candidates and complete-policy risk, coverage and success analysis across three seeds per task. It includes conservative unknown-outcome bounds and paired whole-task resampling. All 35 supplied authored tests reproduced in normal and optimized Python; separate AI source reviews add checks, not human peer review. These are prospective tools, not real-model findings, calibrated risk guarantees, a serving launcher or adoption into the frozen study. Reproduction notes and limits · Source and tests (ZIP) · Source hashes.

Version 2 fixes a settlement defect: the prior replay could read a malformed action value and raise ValueError before recording an expense above its quote. The corrected replay retains that known expense and stops before reading the value. It also supports requested source-history prefixes through 64 actions; this does not amend the frozen study. Historical version 1 remains available unchanged.

Eighteen-policy authored replay · 7 October 2026 Pacific

A separate standalone package preserves all eighteen adaptive, fixed-rule, allocation and conditional-SoftBoN methods. It reproduces 216 authored outputs: two tasks, three bank seeds and two abstract budget caps. Two overlapping analysis views of fourteen and seven policies retain all seventeen unique joint contrasts without treating shared records as independent runs. Source fitting uses authored eight-candidate banks; evaluation supports sixteen candidates. These fixtures test implementation and accounting, not actual verifier diversity or selection gains.

The package records exact source commitments, complete configurations and fixtures, six expected output hashes, and reproduction tests. It retains known expense before overquote refusal and leaves unknown outcomes and expenses unknown. Its conditional SoftBoN kernel is not a full published-method reproduction. This prospective source package changes no frozen H1 protocol and establishes no real-model result, calibrated risk guarantee, runtime qualification or confirmatory finding. Independently reviewed source and extracted execution checks are AI-assisted review, not human peer review. Reproduction notes and limits · Complete source and tests (ZIP) · Source hashes.

Prospective power planning · 7 October 2026 Pacific

A reproducible planning utility resamples whole-task differences after averaging three seeds per task. Its percentile-bootstrap method is conditional and requires prospective adoption; it selects no confirmatory sample size. Hypothetical laws and authored tests do not establish actual power, and shared-family dependence still needs a separate design. The source and independent AI review preserve those limits; this is not human peer review. Method and limits · Reproduction notes · Source and tests (ZIP).

Exact synthetic diagnostics · 6 October 2026 Pacific

Two authored binary-classification constructions clarify the proposed controls. They are exact mathematical examples, not measurements of LLMs, code-selection results or evidence of a new method. The frozen H1 study is unchanged.

In the two-judge example, increasing shared errors changes both agreement risk and non-abstaining coverage. Lower error among agreement outputs cannot be interpreted as improved unconditional utility. In the four-judge example, identical marginal error rates and pairwise covariance coexist with different majority-vote accuracy: a small exact spread of 0.10 percentage point. A covariance-based verifier count therefore does not uniquely determine portfolio utility across unrestricted joint laws; it may still be a useful empirical predictor.

Methods appendix (Markdown) · Complete reproduction package (ZIP) · Reproduction script · Figure (PDF) · Figure (SVG) · Artifact hashes and scope. Related literature: Kuncheva and Whitaker (2003); the exact constructions are not claimed as original or reproduced from that article.

To reproduce the exact calculations, run python3 -m unittest discover -s DIRECTORY separately in each of the two diagnostic source directories inside the ZIP; these tests require only Python’s standard library. To regenerate the figures, run python3 build.py from the package root with Matplotlib installed (reviewed here with version 3.11.2). Rendering versions and export metadata can change figure bytes; the rational calculations, rather than byte-identical image exports, define the numerical results.

Three verifiers, identical pairwise errors · 7 October 2026 Pacific

A further exact example makes the distinction explicit with three binary verifiers. Two specified laws have identical 90% class-conditional accuracy and pairwise-independent errors, yet their balanced-prior Bayes error is 3% versus 1%. Both use all three queries at the same three abstract units of cost. Higher-order error structure changes decision value while a covariance-based effective count remains three.

This is an elementary known-law classification example, not a novel theorem, measured verifier improvement, code-selection result or change to the frozen H1 study. It does not show insufficiency of the complete conditional pair law in the original two-judge setting. The majority-optimality proof assumes the balanced prior; the reproduction checks other priors and undefined posterior cells separately. Seven authored tests reproduced in normal and optimized Python, with independent AI checks of the proof and tables; this is not human peer review.

Exact derivation · Reproduction notes and limits · Source, tests and truth tables (ZIP) · Source hashes.

Finite verifier information and pool headroom · 7 October 2026 Pacific

A second reproducible mathematical package enumerates 54 matched-law budget/utility cases and 108 candidate-pool cases. In three authored laws with the same single-judge behavior, known-law full-coverage selection succeeds at 85% using one judge, while using both succeeds at 96%, 94.1% or 85%. The second judge changes information acquisition cost from two to four abstract units; these gains are before any cost penalty, not equal-cost empirical improvements. The pool extension separately permits no correct candidate, one correct candidate or both correct, and distinguishes observable selection value from the hidden-truth headroom ceiling.

These standard finite-decision constructions isolate assumptions; they establish no novelty, calibration guarantee or real-model performance. Class-conditional error marginals match within the compared laws, not across changed truth priors. Conditional independence across candidates is assumed. All computation uses rational arithmetic; the figures use optional Matplotlib. Reproduction notes and limits · Source, tests, maps and figures (ZIP) · Information-gain figure (SVG) · Figure captions · Artifact hashes.

Information ceiling for fixed candidate pools · 7 October 2026 Pacific

A short self-published note proves a ceiling on what any selector can gain from a fixed candidate pool given verifier signals, bounds the gain by conditional mutual information, shows that pairwise correlation and effective vote counts do not determine the ceiling, and lists seven ways the ceiling fails to transfer to real policies. It uses textbook inequalities, claims no novelty, contains no model result and has had only a model-assisted check. The matched 24-task study remains unrun. Read the note.

A falsifiable hypothesis

On a held-out task-family shift, a controller calibrated to joint verifier errors will improve hidden-test pass rate over the same controller with joint-error interaction features removed, at one fixed budget. The practical design target is a three-percentage-point improvement, not a predicted result. A paired confidence interval will determine whether the evidence supports improvement, is inconclusive, or contradicts it.

Agreement is not proof: judges can agree on the same wrong answer. A disagreement-only stopping rule will be an ablation. Synthetic correlated-error interventions will probe mechanisms separately from the primary natural-shift evidence.

Experiment design

• One pinned open-weight generator, two verifier families, public-test execution, and up to sixteen cached candidates per task. Live feedback-conditioned revisions are outside the first replay experiment.
• Source-family-only development and calibration, with separate pilot and confirmatory target-family tasks; deduplication by source and problem family. The separate joint-verifier pilot is planned for 120 tasks; it is not the completed 120-task engineering pilot. Confirmatory size follows a power calculation, rather than an arbitrary sample count.
• Controllers see public tests and requested verifier judgments. Hidden outcomes are used only for offline evaluation; future cached scores are unavailable to the controller.
• Audit augmented hidden tests and report correctness against the declared suite, candidate coverage, false acceptance, discrimination and calibration. Passing tests does not prove general correctness.
• Compare single-pass and first-public-test-pass controls, extra sampling, fixed-budget best-of-N, execution agreement, calibration-only stopping, joint-error-aware selection, fixed majority/supermajority, Hossain-style CorrFilter and bias-cluster filters at matched retention, the strongest development-selected fixed rule, and applicable code-specific pairwise or robust-selection methods.
• Freeze policies, thresholds, task manifests, model revisions, primary comparison and analysis before confirmatory outcomes are inspected.

Before the future confirmatory freeze, evaluate three additional design requirements from the closest work. First, report both single-judge controllers, and choose any strongest-single-judge reference only by a frozen source-development/calibration rule, never target outcomes; a multi-judge method must justify its extra information and cost against that reference. Second, report selected-output error and coverage for the complete frozen search/stopping policy separately from candidate-score calibration. Assess a held-out source-split policy-level calibration or risk-control comparator only if its dependence, finite-sample and distribution assumptions are applicable; image-domain guarantees do not transfer automatically to this code study. Third, assess Kwok et al.'s verifier framework for faithful implementation. If the serving API cannot expose the required scoring-token distributions, label a text-verdict adaptation as an adaptation and do not claim reproduction; account for every repeated/decomposed verification action. Any comparator requiring new data or calls needs its own prospective source, budget and task freeze before dispatch. The current diagnostic collection is not expanded by this paragraph.

Prospective design clarifications · 6 October 2026 Pacific

The narrower question is how much decision value remains in joint verifier information after controlling for marginal verifier quality, candidate coverage, available public-test information and cost. This is a question to test, not a claim that a verifier portfolio must improve correctness. These clarifications preserve the existing H1 and frozen collections; new comparisons require a separate prospective protocol and budget before execution.

• Shared pools and matched information. Primary replay policies use the same frozen, ordered candidate bank and outcome suite. Each policy sees only candidates and judgments it has paid to reveal. The joint-feature ablation receives the same inputs, estimator, source-only calibration and stopping rule; only the registered interaction features differ. Both single judges, fixed voting, execution-based selection and applicable published selection methods remain necessary comparisons. A pure-selection diagnostic would hold the revealed candidate set and generation charge common; live feedback-conditioned generation is a separate future experiment.
• Mechanism versus transfer. Natural task-family shift tests performance under changed operating conditions; it does not causally isolate correlation or calibration drift. A separately frozen synthetic mechanism study would vary feasible error coupling while preserving each verifier's class-conditional error rates, score marginals and declared costs. For Bernoulli errors with marginals p and q, joint error probability must lie between max(0, p + q - 1) and min(p, q). Arbitrary correlation levels are not always attainable. Synthetic results would not establish behavior of real model families.
• Measurable dependence. Report joint false acceptance on incorrect candidates and joint false rejection on correct candidates separately, with task-cluster uncertainty and prospectively defined difficulty strata. Disagreement alone can reflect useless errors. A single observation per candidate cannot identify a candidate-specific conditional error probability without an explicit estimation model. Conditional mutual information or an effective-verifier-count summary needs its own estimand, support checks and out-of-sample validation; neither is assumed to predict selection performance. Classical classifier-ensemble diversity measures are prior work, not a new contribution of this proposal.
• Pool suitability and failure cases. Assess correct-candidate coverage, mixed pools, joint false acceptance and oracle selection headroom on development data before committing to a powered study. AST diversity is a syntactic diagnostic, not semantic correctness or independent verifier evidence. Deliberately edited or adversarial pools would form a labeled stress-test arm; outcome-selected pools cannot support an unqualified natural-generalization claim. Retain cases where all judges accept a wrong candidate, disagreement adds no useful information, or more verification selects a worse answer.
• Fixed-bank headroom accounting. Let B be baseline correctness, S selected-output correctness and O indicate that the shared bank contains at least one correct candidate. Then the paired gain is rescue minus regression: E[S] − E[B] = P(S=1, B=0) − P(S=0, B=1) ≤ E[O] − E[B]. This elementary identity is not a novel theorem or an expected gain. Report rescues and regressions separately; a selector can lose accuracy despite positive headroom. Explicit abstention counts as zero success for this endpoint, while missing or ambiguous execution labels remain unknown and require partial-identification bounds rather than an imputed oracle. A resource-constrained ceiling requires a frozen admissible action set containing both policies and charging failures and retries. Adaptive generation needs a separate bank comparison.
• Missing execution outcomes. A reviewed fixed-bank companion computes bounds on coverage, headroom and paired gain over every binary completion of unknown labels. The intervals are individually sharp under its assumptions, not sampling confidence intervals or a jointly feasible rectangle. Both policies must use fixed selections; duplicate candidate identities need shared constraints. Download code, tests and methods (ZIP) or read the assumptions and reproduction instructions. Exact checks cover 2,550 small-bank configurations; a separate AI agent reviewed the formulas and checked another 8,748 five-candidate configurations. This is an elementary methods diagnostic, not an empirical verifier result.
• Endpoints, costs and ceilings. The three-percentage-point target is a practical design and power-planning target, not the definition of scientific success. Keep paired correctness at the frozen primary cost cap as the primary comparison. Secondary accuracy/cost curves must use predeclared budgets; dollars, tokens, latency, execution resources and offline fitting costs remain separate axes rather than a universal interchangeable compute unit. An offline hidden-outcome oracle provides a declared information-relaxed ceiling and never supplies decisions to the evaluated controller. Fractions of oracle gain are undefined when headroom is zero and unstable when it is small.
• Bounded extensions. Value-of-information allocation, repeated verification, larger verifier portfolios, additional generators and mathematics or formal-reasoning transfer studies are conditional follow-ups. They are not a required full factorial grid for the current two-verifier study, and they cannot consume the shared $100 external budget without a frozen cost envelope. Planned contributions remain measurement and a reproducible matched comparison; a new metric, theorem or algorithm will be claimed only if separately justified by evidence.

Prepared controller: exact computational object

The prepared development implementation is a source-only ridge regression of incremental continuation value. Its training target is hidden-suite correctness of the candidate selected after an action and a fixed continuation, minus correctness of the incumbent before that action. Source outcomes enter this separate training operation; target outcomes and unrevealed cached judgments are excluded from decision inputs. This is a continuation-dependent surrogate, not a calibrated probability that a candidate is correct. Real source fitting and evaluation remain pending.

The additive arm has 31 features: an intercept; active and incumbent candidate summaries containing scores, missingness, attempted flags, validity, public-test status and index; candidate count, remaining-budget fraction, quoted-cost fraction and syntactic diversity; and four action indicators. The joint arm adds one cross-score feature, (score_A − 0.5) × (score_B − 0.5), set to raw zero when either score is missing and standardized using source-history rows. Ridge fitting gives each source task equal total weight; feature standardization uses unweighted history rows. Task and family identifiers are not numeric predictors.

Predicted incremental success is clipped to [−1, 1]. Among affordable eligible actions, the controller chooses the highest predicted success gain per quoted cost, stopping when predicted benefit is nonpositive. The interaction coefficient is shared across actions; it is not an action-specific covariance model. A separately frozen conditional-placebo feature arm helps assess the extra feature's capacity, without establishing a causal effect of error dependence.

Scope of the interaction: it carries observed cross-score information only when both scores are nonmissing. Missing histories receive the standardized raw-zero baseline, which can still affect predicted benefit, stopping and cost-normalized ranking. Neither can then be queried again for that active candidate, so its direct use concerns further generation, public testing or stopping, rather than choosing which of those two judges to query next. A general verifier-portfolio allocation policy or a calibrated posterior such as P(correct | scores, context) would require a distinct prospective specification. Improvement by this implementation would support incremental decision value of its registered cross-score feature under matched conditions; it would not alone isolate correlation from nonlinear calibration, difficulty or other interactions. Many histories from a small number of source tasks do not replace independent-task support.

This method clarification changes no frozen H1, collection, model setting, fitting rule or analysis. Controlled dependence experiments, larger portfolios and value-of-information policies remain separately gated follow-ups.

Prospective selected-output risk specification

Evaluate each complete frozen policy at each registered budget, averaging its three seeds within each task. Coverage is the mean fraction of non-abstaining selections; unconditional success is the mean fraction of correct selections; selected-output risk is incorrect selections divided by non-abstaining selections, with equal task weights. Risk is undefined at zero coverage. Resample whole task traces rather than treating seeds or candidate histories as independent observations. Unknown decisions and outcomes remain unknown and require bounds; they are neither abstentions nor incorrect labels. Candidate-score calibration does not certify selected-output risk.

A future policy-level risk comparator must freeze its risk score, threshold grid, source fitting and disjoint calibration tasks, complete stopping and tie-breaking policy, and threshold-selection rule before target outcomes. The prepared ridge continuation-value score is not a risk probability. Any separate terminal-risk estimator must use source-only completed policy traces; additional inference actions remain charged. Source and shifted-target risk are descriptive unless explicitly applicable assumptions justify a guarantee. These are requirements for a future protocol freeze, not an amendment to the frozen diagnostic or a completed calibration result.

Compute and analysis

The primary cap will be four times median single-pass inference cost on the development split, using a frozen price sheet. Two secondary caps explore sensitivity. Charge generation, verification, repeated context, retries and failures; also report tokens, CPU or GPU time and latency. Billed cost is an operating metric, not a FLOPs measurement.

Replay charges only actions a policy requests. The full cached pool remains a separate actual research expense. Natural shifts can change task difficulty and candidate coverage as well as calibration, so they will not be presented as a causal isolation of drift.

The primary comparison keeps the estimator, other features and stopping rule matched. Freeze outcome suites before viewing controller results. Inference is conditional on one candidate pool and one family shift; pool exhaustion and later audit corrections will be reported. The primary endpoint is paired task-level hidden-test success, with a 95% confidence interval. Abstention counts as failure there and is reported separately in risk/coverage analysis. Secondary tests will account for multiple comparisons. Null findings and inconclusive intervals will be published.

Prior work and overlap

The literature already includes adaptive allocation, reward hacking, imperfect coding verifiers, code-judge dependence screens, distribution-shift-aware judges and execution-based selection. Hossain et al.'s code screen found no matched-retention filter headroom and did not run a code-routing experiment; it does not establish the result of a budgeted generator-policy study. A full-text and code review must establish which comparisons are applicable before this study proceeds.

• Snell et al.: Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. Adaptive inference allocation; conceptual replication only, not reproduction of inaccessible original models.
• Stroebl, Kapoor and Narayanan: Inference Scaling fLaws: The Limits of LLM Resampling with Imperfect Verifiers. Direct overlap: imperfect coding tests limit resampling. This study must go beyond that established finding.
• Huang et al.: Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment. Pessimistic selection under imperfect rewards; candidate strong baseline, assumptions need matching.
• Khalaf et al.: Inference-Time Reward Hacking in Large Language Models. Best-of-Poisson and HedgeTune; mandatory feasibility assessment for robust-selection baselines.
• Montgomery et al.: Budget-aware Test-time Scaling via Discriminative Verification. Compute-efficient verifier/self-consistency hybrid; not a new research direction here.
• Li et al.: S*: Test Time Scaling for Code Generation. Execution-grounded pairwise selection and distinguishing inputs; code-specific comparison.
• Qin et al.: DAJ: Data-Reweighted LLM Judge for Test-Time Scaling in Code Generation. Direct shift overlap: difficulty, task distribution and generator-trajectory mismatch.
• Qu: Adaptive Test-Time Compute Allocation via Learned Heuristics over Categorical Structure. Selective intermediate verification; structured-math assumptions may not transfer to final code selection.
• He et al.: Code Generation by Differential Test Time Scaling. Execution clustering without additional selection-model calls; include CPU costs.
• Sriraman and Block: Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment. Objective matters: win-rate and correctness are different endpoints; assess tuned BoN variant.
• Kohli, Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels (v1). Correlated judge errors are established; code-selection transfer and comparison with single judges require explicit assessment.
• Yin et al., Calibrate What You SHIP: Post-Selection Risk Control for Verifier-Guided Text-to-Image Generation (v1). Distinguish candidate calibration from complete-policy selected-output risk; assess applicability before transferring guarantees.
• Kwok et al., LLM-as-a-Verifier: A General-Purpose Verification Framework (v2). Assess a faithful logit-derived/repeated-verification baseline and charge all verification actions; disclosed text-only adaptations are not exact reproductions.
• Hossain et al., Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus (v1). Appendix K screens pointwise and pairwise code-judge banks at matched retention; its results gate feasibility and do not test budgeted candidate selection or stopping.
• Han, Partially Correlated Verifier Cascades in LLM Harnesses (v1). Theoretical correlated conjunctive gates; assumptions do not automatically transfer to heterogeneous candidate-selection policies.
• Kang, Jeong and Cho, T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Models. Assess a public-tool-filter plus verifier baseline with matched information and full tool/model costs.

Full-text overlap review

ADAP already learns stopping and draws on ground-truth calibration; public-test proxies cannot inherit its guarantee. CAPS already budgets pairwise judging and eliminates finalists; scalar-score selection does not reproduce it. DAJ addresses judge distribution mismatch with learned data reweighting. GRACE studies verification granularity under assumptions not automatically satisfied by correlated code-judge errors. The proposed replication/extension contrast is whether conditional cross-verifier dependence adds useful candidate-selection and stopping headroom under task-family shift after matching marginal calibration, candidate coverage, action information and total cost. Hossain et al.'s code-judge screen has no matched-retention filter headroom in its sampled banks; only a prospective generator-policy experiment could address the separate candidate-selection question. This difference is a question to test, not a proven original contribution.

Additional overlap screen · 7 October 2026 (America/Los_Angeles): He et al., Verification Trap (v1), studies shared generator–verifier blind spots, recoverable mis-selection and fixed-pool evidence controls. Its controlled mitigation holds the gate and augmented candidate pool common. This overlaps our selection and headroom diagnostics; those ideas are not claimed as new here. The paper does not report exact latency, GPU hours, token counts or monetary cost, so its reported gains are not cost-matched baseline results for this study. Applicable selector controls still require assessment. Any false-premise stress arm would be separately specified; the frozen H1 protocol is unchanged. This targeted reading is not a replication or an exhaustive literature review.

Schedule and release

The earlier HumanEval engineering pilot is complete. The prospective H1 collection uses MBPP for source development and calibration, and BigCodeBench for the target task-family diagnostic. Dataset files, revisions and acquisition receipts are frozen; H1 collection and confirmatory results remain unrun. The pinned dataset cards list MBPP revision 4bb6404 under CC-BY-4.0 and BigCodeBench revision b74c0d0 under Apache-2.0. These provenance records distinguish the proposed study from the historical pilot. The remaining gates after the completed scoped restoration replay are: finish serving model/tokenizer/template and candidate qualification, complete the real development collection and source-only calibration, obtain sufficient candidate-pool diversity on new task families, finish applicable strong-method baselines, price the powered joint-verifier study, and archive an immutable preregistration. Confirmatory execution follows only if the question and budget survive those gates.

Planned artifacts include protocol versions, licensed task manifests, controller code, model settings, seeds, sanitized traces, cost records, analysis and failure cases. Model-assisted critiques informed this draft; they are not human peer review. No model-training run or new foundation-model architecture is claimed.

A separate transfer study would be needed to establish relevance across stronger models and other benchmarks. This proposal is a first step toward a broader model-research program.

Citation

If you reference or build on this protocol draft and methodology:

Offline failure-atlas analyzer (source and authored tests): distinguishes shared verifier mistakes, disagreement and matched selection changes while preserving unknown outcomes. Prepared 7 October 2026 Pacific; eight authored tests and model-assisted source review, no real-data atlas or demonstrated selection benefit.