Cisco Caceres · 17 October 2026 (Pacific) · Working Paper / Research Status
Abstract
This working paper records an open research question: under a fixed inference budget, can modeling dependence among verifier errors improve code-candidate selection or stopping decisions? The evidence must be separated by source. A completed 120-task HumanEval engineering pilot is reported in a separate, linked report; it does not test the proposed joint-verifier and task-family-shift hypothesis. The prospective 576-slot H1 collection remains unrun.
Correction to the earlier Pillar 1 report: the reported 504-task/4,032-candidate sweep was not an empirical benchmark evaluation. The checked-in generator assigns candidate correctness, public-test outcomes, verifier scores, costs and latencies using seeded random draws. It does not generate or load candidate programs, invoke models, or execute public or hidden benchmark tests. The task identifiers and metadata are inputs to a synthetic simulation only. The reported covariance and policy rates are properties of that simulation and are withdrawn as measurements of verifier behavior, code-generation accuracy, generalization or VEC-SCR performance.
The three simulation split labels reuse the same fourteen domain labels; they do not establish a held-out-family evaluation. The earlier claim of early stopping and token conservation is also withdrawn: the configured threshold of 0.85 is above the maximum score of 0.75 reachable on the public-pass branch, so the stopping condition never fires. The simulation remains available as implementation/debugging material, clearly labeled below.
The information-ceiling derivation and missing-outcome bounds in Sections 2–3 are authored mathematical analyses, not model results or claims of novelty.
1. Research question and evidence boundaries
Inference-time methods can spend additional resources generating candidates, running tests, querying judges or deciding whether to stop. A legitimate study must specify what information each policy sees and charge all relevant costs. Agreement among judges is not by itself evidence that their errors are independent or that an ensemble improves selected-output correctness. The proposed question is narrower: after matching candidate pools, available information, verifier quality and total cost, does modeling cross-verifier dependence improve task-level correctness under a defined shift?
The current evidence does not answer that question. The separate 120-task HumanEval engineering pilot has limited candidate-pool headroom and is descriptive. The 504-task seeded simulation is not an empirical model study. The prospective MBPP/BigCodeBench H1 collection and confirmatory analysis have not run. No frontier-model transfer or new verifier method is claimed.
2. Information-Theoretic Stopping Bounds
We formalize candidate selection and verifier acquisition as a finite-action decision problem under partial observation.
2.1 Formal Decision Framework
Let W ∈ 𝒲 denote a finite latent outcome state representing the joint correctness vector of candidate solutions W = (Y1, …, YK) ∈ {0, 1}K. Let X ∈ 𝒳 denote the initial observation available to the agent, including the task prompt, problem specification, and public unit test results. Let Z ∈ 𝒵 denote an additional verifier observation, such as execution logs from supplementary test suites or scores from neural judges.
A finite action set 𝒜 is available to the decision maker, consisting of selecting candidate j ∈ {1, …, K} or abstaining (a = ∅). For each action a ∈ 𝒜, let u(a, x, w) ∈ [0, 1] represent the utility of action a given observation x and latent state w. For candidate selection, u(j, x, w) = Yj corresponds to binary correctness, while abstention yields u(∅, x, w) = 0.
Let pX(w) = P(W = w ∣ X = x) denote the prior belief over latent states given initial observation X. Let qXZ(w) = P(W = w ∣ X = x, Z = z) denote the posterior belief updated after observing verifier signal Z.
Define the prior Bayes-optimal value V0 and posterior Bayes-optimal value V1:
V0 = 𝔼X[maxa ∈ 𝒜∑w ∈ 𝒲pX(w) u(a, X, w)]
V1 = 𝔼XZ[maxa ∈ 𝒜∑w ∈ 𝒲qXZ(w) u(a, X, w)]
The net value of information provided by verifier signal Z is defined as Δ = V1 − V0.
2.2 Derivation of the Information Ceiling
We define the expected Total Variation (TV) distance between posterior and prior distributions:
T = 𝔼XZ [ TV(qXZ, pX) ] = ½ · 𝔼XZ [ ∑w ∈ 𝒲 |qXZ(w) − pX(w)| ]
and the conditional mutual information I(W; Z ∣ X) measured in nats:
I = I(W; Z ∣ X) = 𝔼XZ [ DKL(qXZ ∥ pX) ] = 𝔼XZ [ ∑w ∈ 𝒲 qXZ(w) ln( qXZ(w) / pX(w) ) ]
Theorem 1 (Information-Theoretic Ceiling on Verifier Gain). For any finite action set 𝒜, bounded utility u ∈ [0, 1], and joint distribution P(W, X, Z):
0 ≤ Δ ≤ min{ 1 − V₀, T, √( I / 2 ) }
Proof. Since an agent observing (X, Z) can always ignore Z and deploy the Bayes-optimal decision for X, Jensen’s inequality on the convex maximum operator guarantees V1 ≥ V0, establishing Δ ≥ 0. Because u ≤ 1, V1 ≤ 1, implying Δ ≤ 1 − V0.
For each realization (x, z), let az* = arg maxa∑wqxz(w)u(a, x, w). The prior value is at least the expected utility of az* under px:
maxa𝔼q[u(a, x, W)] − maxa𝔼p[u(a, x, W)] ≤ 𝔼q[u(az*, x, W)] − 𝔼p[u(az*, x, W)]
= ∑w(qxz(w) − px(w)) u(az*, x, w) ≤ ∑w : q > p(qxz(w) − px(w)) = TV(qxz, px)
Taking expectations over (X, Z) yields Δ ≤ T.
By Pinsker’s inequality with natural logarithms, $\mathrm{TV}(q, p) \le \sqrt{\frac{1}{2} D_{\mathrm{KL}}(q \parallel p)}$. By Jensen’s inequality applied to the concave square root function:
T = 𝔼XZ[ TV(q, p) ] ≤ 𝔼XZ[ √( ½ DKL(q ∥ p) ) ] ≤ √( ½ 𝔼XZ[ DKL(q ∥ p) ] ) = √( I / 2 )
This completes the proof. ◼
2.3 Economic Implications for Paid Verification
In production code generation environments, querying a secondary model judge or running containerized execution sandboxes incurs billed dollar cost or token latency. Let c > 0 denote the acquisition cost of signal Z expressed in equivalent utility units.
The net decision value of paid verification is Δnet = Δ − c. An immediate corollary of Theorem 1 is:
If √( I(W; Z ∣ X) / 2 ) < c, then Δnet < 0
Whenever the mutual information between the verifier signal and latent program correctness is strictly below 2c2 nats, purchasing verification reduces expected utility under these assumptions. At equality, the bound establishes only nonpositive net value.
3. Partial Identification Bounds with Missing Outcomes
In real-world benchmarking, execution outcomes are frequently incomplete: long-running tasks timeout, container sandboxes hit memory limits, or test fixtures return ambiguous exit codes. Rather than imputing missing executions as zero or dropping failed runs, we derive sharp partial identification bounds.
For each task t ∈ {1, …, N} and candidate j ∈ {1, …, K}, let Ytj ∈ {0, 1} denote latent suite correctness. Let observed executions partition candidate labels into known values and unknown sets 𝒰t. Let 𝒜t = {0, 1}|𝒰t| denote the set of all binary completions of unknown labels.
For any two fixed candidate selection policies S and B, let st, bt ∈ {1, …, K} denote their respective selected candidate indices. For assignment a ∈ 𝒜t, the task-level difference is:
Dt(a) = Yt, st(a) − Yt, bt(a)
We define task-level lower and upper bounds:
ℓt = mina ∈ 𝒜tDt(a), ut = maxa ∈ 𝒜tDt(a)
The population difference is bounded sharply by:
(1/N) ∑t=1N ℓt ≤ D̄ ≤ (1/N) ∑t=1N ut
Exact Cancellation Under Shared Selection
A critical property of candidate selection under partial observation is shared candidate cancellation:
If st = bt, then Dt(a) = Yt, st(a) − Yt, st(a) = 0 ∀a ∈ 𝒜t
Treating policies S and B independently under missing data would assign each an interval of [0, 1], resulting in a loose difference bound of [−1, 1]. Recognizing shared candidate choices collapses the difference to exactly 0, preventing missing data from artificially inflating uncertainty.
Similarly, we define the bank oracle Ot(a) = maxjYtj(a) and oracle headroom Ht(a) = Ot(a) − Yt, bt(a). Headroom must be evaluated jointly over assignments: if baseline candidate bt is known to pass, headroom Ht is strictly zero, even if all other candidate outcomes are missing.
- Earlier Pillar 1 artifact: deterministic synthetic simulation
The retained manifest contains 504 rows of task metadata labeled with LiveCodeBench and BigCodeBench names. The accompanying script does not load the task prompts or solutions. With NumPy seed 42, it samples generator labels, token counts, costs, latency, candidate correctness, public-test outcomes and verifier scores from hand-coded random distributions. Candidate “code hashes” hash synthetic placeholder strings. No candidate source code is produced, no model or judge is called, and no benchmark test suite is executed.
The resulting 4,032 records can be used to exercise analysis code and demonstrate how a chosen synthetic data-generating process flows through policy code. They cannot estimate real verifier error covariance, code pass rates, accuracy, monetary costs, task-family transfer or selection utility. All previously reported aggregate values from that sweep are therefore retained only as simulation output and withdrawn as empirical findings.
The manifest labels three groups as dev_source, calib_source and eval_target_shift, but inspection shows all three contain the same fourteen domain labels. They are not disjoint task-family splits, so comparisons between them do not test natural family shift.
A source-level reachability check also found the configured default stopping threshold 0.85 cannot be attained: on the branch that permits stopping, the score formula is bounded above by 0.75. The early-stop branch never executes at that threshold. Any earlier statement that the sweep demonstrated sequential stopping or token conservation is withdrawn. The threshold and controller require a separately specified, calibrated design before they can support empirical claims.
5. What remains to test
Before the proposed H1 study can produce evidence, the serving executable, loaded model, tokenizer and prompt template must be qualified; the frozen 576-slot collection must be run against actual task prompts with retained candidate outputs, verifier responses, test results and costs; task-family separation must be verified from source identities; and policy baselines, outcome handling, stopping rules and analysis must be frozen before confirmatory evaluation. The distinct 120-task engineering pilot remains descriptive and should not be pooled with this prospective study.
Negative, inconclusive and null results remain acceptable outcomes. Any future result should publish task-level denominators, paired uncertainty, candidate-pool headroom, verifier overlap, complete inference costs, failures and the exact runnable artifacts.
Evidence boundary: This page contains authored mathematics, the separately described 120-task engineering pilot reference, and a correction of a synthetic simulation. It contains no empirical result from the 504-task simulation and no result from the proposed H1 study.
Written by Cisco Caceres. Updated 2026-10-17 UTC. Research agenda and evidence.