A conformance verdict drops the bound a refusal named, so `host.deadline_exceeded` cannot be told from `host.deadline_exceeded` #254

New issue

Closed

opened 2026-09-10 12:12:06 +02:00 by buildagent · 0 comments

buildagent commented

2026-09-10 12:12:06 +02:00

Member

The supervisor deliberately builds a refusal that says which of three enforcement points stopped a guest. The conformance verdict — the artifact CI and operators actually read — throws it away one call before writing it down, keeping only the reason code, which is identical for all three.

The tree promises this discrimination, twice

plugin_supervisor::limits, on the wall-clock backstop:

HONEST ABOUT THE COST: this bound is the one piece of supervision that is not deterministic. On a machine loaded enough, a guest that would have finished inside its fuel can be refused here instead. The mitigation is the ratio … and the disclosure: Refusal::limit names WHICH bound fired, so an operator can tell the reproducible refusal from the timing-dependent one.

supervise::deadline_refusal, which constructs it:

A deadline refusal that NAMES WHICH BOUND FIRED. Fuel is reproducible and a wall clock is not … witness says which of the three enforcement points actually fired: fuel (deterministic), epoch (the worker's own clock), or process (the parent's kill). Only the first is guaranteed to repeat.

fn deadline_refusal(which: &'static str, value: u64, fired: &'static str) -> Refusal {
    Refusal::bound(Reason::HostDeadlineExceeded, which, value, value).with_witness(fired)
}

Where it is lost

extract_with_grammar returns Result<ValidatedFacts, Refusal> — the full Refusal, carrying limit, limit_value, observed, witness. Then:

fn reason_of(r: &Refusal) -> &'static str { r.reason.as_str() }

and both worker-refusal sites in conform::check do FixtureOutcome::ungraded(..., reason_of(&r)). FixtureOutcome::refusal is an Option<String> holding only the code. Everything that distinguishes a reproducible refusal from a timing one is discarded one call short of the record.

What that cost, measured

CI job 34810 on 5131654 failed with:

xaml_large_file_budgets.rs:257  the_fact_buffer_ceiling_fires_and_says_so
FixtureOutcome { …, refusal: Some("host.deadline_exceeded"), facts: NotGraded, … }

host.deadline_exceeded from fuel would be deterministic — a real defect in the fixture or the extractor. From epoch or process it is a loaded runner and means nothing about the product. The verdict could not say which, and the same binary passed locally, so the triage had to be done by re-running under artificial load instead of by reading the record.

For the record, the contention was real and self-inflicted in an ironic way: the fix in 3aea49d un-broke OSS corpus (tier 1), so that job went from failing at ~25 min to succeeding at ~54 min — and cargo test, sharing the runner pool, went 32 min → 68 min.

run	corpus job	`cargo test`
722/724 `bd1c4e2`	25m (failed early)	32m ✓
728 `5131654`	54m (succeeds)	68m ✗

Fix

Three additive, serde(default) fields on FixtureOutcome — refusal_limit, refusal_limit_value, refusal_witness — and a FixtureOutcome::refused(…, &Refusal) constructor used at the two sites that hold a real Refusal.

Not folded into ungraded. The other four sites refuse for reasons that are not bounds (a staging failure, a nondeterministic second extraction); giving them an empty limit would make "this refusal names no bound" and "nobody recorded the bound" the same fact, which is the conflation the fields exist to end. Absent stays a third state.

Additive and back-compatible: old verdict files deserialize with None, new ones add keys old readers ignore.

xaml_large_file_budgets's probe now has an arm that fires on a deadline and prints the bound, its value and the witness, instead of the generic "the expectation was never COMPARED" — which is true but sends the reader after the byte ceiling rather than after the machine. The code it matches is bound to code_index_abi::Reason::HostDeadlineExceeded.as_str() rather than spelled out, because a literal that drifted by one character would turn that arm into dead code whose failure mode is silence.

Mutations, run

Restore reason_of in refused — the exact shape that shipped. RED: left: (Some("host.deadline_exceeded"), None, None, None) against the full discriminator. md5 verified changed, then restored.
The round-trip test alone does NOT catch that: it builds the outcome literally, so it grades the verdict FILE and stays green with the constructor throwing the refusal away. That is why there are two tests, and the doc on each says which mutation it owns.
Anti-vacuity arm: a Refusal::new(HostWorkerUnavailable) — not a bound — must record None, so a constructor stamping a placeholder fails.
The existing wire-skew test's HostDeadlineExceeded control fixture now carries a non-empty discriminator and asserts it survives the verdict file, following this tree's own normalize_graph_reply lesson that a wire test over an empty payload passes against a shim that drops everything.

Not fixed here, deliberately

Whether the 30 s wall-clock backstop has enough headroom over legitimate worst-case input is a separate question and a supervision change, so it is not being decided as a side effect of this. I attempted to quantify the headroom and got it wrong: the whole test takes ~17 s idle, but under 2× CPU oversubscription it took 36 s and still passed, which proves the guest's share is well under the 17 s I first attributed to it. The honest statement today is that CI exceeded the bound and an idle machine does not, and the new witness field is what will settle the rest the next time it happens.

The supervisor deliberately builds a refusal that says **which of three enforcement points** stopped a guest. The conformance verdict — the artifact CI and operators actually read — throws it away one call before writing it down, keeping only the reason code, which is **identical for all three**. ## The tree promises this discrimination, twice `plugin_supervisor::limits`, on the wall-clock backstop: > HONEST ABOUT THE COST: this bound is the one piece of supervision that is not deterministic. On a machine loaded enough, a guest that would have finished inside its fuel can be refused here instead. The mitigation is the ratio … and the disclosure: **`Refusal::limit` names WHICH bound fired, so an operator can tell the reproducible refusal from the timing-dependent one.** `supervise::deadline_refusal`, which constructs it: > A deadline refusal that NAMES WHICH BOUND FIRED. Fuel is reproducible and a wall clock is not … `witness` says which of the three enforcement points actually fired: `fuel` (deterministic), `epoch` (the worker's own clock), or `process` (the parent's kill). **Only the first is guaranteed to repeat.** ```rust fn deadline_refusal(which: &'static str, value: u64, fired: &'static str) -> Refusal { Refusal::bound(Reason::HostDeadlineExceeded, which, value, value).with_witness(fired) } ``` ## Where it is lost `extract_with_grammar` returns `Result<ValidatedFacts, Refusal>` — the full `Refusal`, carrying `limit`, `limit_value`, `observed`, `witness`. Then: ```rust fn reason_of(r: &Refusal) -> &'static str { r.reason.as_str() } ``` and both worker-refusal sites in `conform::check` do `FixtureOutcome::ungraded(..., reason_of(&r))`. `FixtureOutcome::refusal` is an `Option<String>` holding only the code. Everything that distinguishes a reproducible refusal from a timing one is discarded **one call short of the record**. ## What that cost, measured CI job **34810** on `5131654` failed with: ``` xaml_large_file_budgets.rs:257 the_fact_buffer_ceiling_fires_and_says_so FixtureOutcome { …, refusal: Some("host.deadline_exceeded"), facts: NotGraded, … } ``` `host.deadline_exceeded` from **fuel** would be deterministic — a real defect in the fixture or the extractor. From **epoch** or **process** it is a loaded runner and means nothing about the product. The verdict could not say which, and the same binary passed locally, so the triage had to be done by re-running under artificial load instead of by reading the record. For the record, the contention was real and self-inflicted in an ironic way: the fix in `3aea49d` un-broke `OSS corpus (tier 1)`, so that job went from failing at ~25 min to succeeding at ~54 min — and `cargo test`, sharing the runner pool, went 32 min → 68 min. | run | corpus job | `cargo test` | |---|---|---| | 722/724 `bd1c4e2` | 25m (failed early) | 32m ✓ | | 728 `5131654` | 54m (succeeds) | 68m ✗ | ## Fix Three additive, `serde(default)` fields on `FixtureOutcome` — `refusal_limit`, `refusal_limit_value`, `refusal_witness` — and a `FixtureOutcome::refused(…, &Refusal)` constructor used at the two sites that hold a real `Refusal`. **Not folded into `ungraded`.** The other four sites refuse for reasons that are not bounds (a staging failure, a nondeterministic second extraction); giving them an empty `limit` would make *"this refusal names no bound"* and *"nobody recorded the bound"* the same fact, which is the conflation the fields exist to end. Absent stays a third state. Additive and back-compatible: old verdict files deserialize with `None`, new ones add keys old readers ignore. `xaml_large_file_budgets`'s probe now has an arm that fires on a deadline and prints the bound, its value and the witness, instead of the generic *"the expectation was never COMPARED"* — which is true but sends the reader after the byte ceiling rather than after the machine. The code it matches is bound to `code_index_abi::Reason::HostDeadlineExceeded.as_str()` rather than spelled out, because a literal that drifted by one character would turn that arm into dead code whose failure mode is silence. ## Mutations, run * Restore `reason_of` in `refused` — the exact shape that shipped. **RED**: `left: (Some("host.deadline_exceeded"), None, None, None)` against the full discriminator. md5 verified changed, then restored. * The round-trip test alone does NOT catch that: it builds the outcome literally, so it grades the verdict FILE and stays green with the constructor throwing the refusal away. That is why there are two tests, and the doc on each says which mutation it owns. * Anti-vacuity arm: a `Refusal::new(HostWorkerUnavailable)` — not a bound — must record `None`, so a constructor stamping a placeholder fails. * The existing wire-skew test's `HostDeadlineExceeded` **control** fixture now carries a non-empty discriminator and asserts it survives the verdict file, following this tree's own `normalize_graph_reply` lesson that a wire test over an empty payload passes against a shim that drops everything. ## Not fixed here, deliberately Whether the 30 s wall-clock backstop has enough headroom over legitimate worst-case input is a separate question and a supervision change, so it is not being decided as a side effect of this. I attempted to quantify the headroom and **got it wrong**: the whole test takes ~17 s idle, but under 2× CPU oversubscription it took 36 s and still passed, which proves the guest's share is well under the 17 s I first attributed to it. The honest statement today is that CI exceeded the bound and an idle machine does not, and the new witness field is what will settle the rest the next time it happens.