overview_payload_budget_e2e is safe from the unconsulted-package-set race by luck, not by design — 25 tokens of headroom against a ~190-token block #133

Closed
opened 2026-09-04 22:50:16 +02:00 by buildagent · 3 comments
Member

Raised in #131's body as "worth confirming before this is closed", and now checked. #131 is fixed and closed; this is the neighbour it warned about, kept open on its own because the answer is "not currently firing" rather than "cannot fire".

The exposure

  • OVERVIEW_SATURATED_MAX_TOKENS = 4300, against a measured 4275 — 25 tokens of headroom.
  • PACKAGE_SET_UNCONSULTED_SEMANTICS is ~190 tokens, rendered whenever a client beats package discovery.
  • crates/mcp-server/tests/overview_payload_budget_e2e.rs contains zero occurrences of wait — verified by count, not by reading. It never waits for package_set_consulted.

If that block ever lands in this reply, the test is 165 tokens over, not 25 under.

What was measured, and what it does not prove

Six consecutive isolated runs on the daemon leg: 6/6 green, runtimes flat at ~2.2s.

That is weak evidence and should not be read as safety. The sibling agent_task_plugin_bench hit the same window 3 times in 13 isolated runs — so a six-run clean sweep is well within what a bimodal race produces by chance.

What would settle it is structural, and I did not establish it: whether the saturated leg's fixture has a package set whose discovery can lose the race. The file does describe a leg measured "over a store holding one wanted package" (and #100's package_store work added another), so the window is reachable in principle for at least some of its legs.

Why it matters more than the headroom suggests

That ceiling guards the first call an agent makes. Its own doc records that when it was last breached the response was to delete a four-line paragraph rather than raise the number — so the 25 tokens are deliberate, and the file already refuses the easy way out.

A flake here would therefore arrive as a payload failure on the most-read surface, and the tempting fix — raise the ceiling — is the one thing that file argues against. Better to remove the race than to widen the band under pressure later.

The fix

The same one #131 took: wait for package_set_consulted before measuring, so the test grades one payload rather than whichever side of a startup race it landed on. crates/mcp-server/tests/coverage_signature_e2e.rs already does this, and agent_task_plugin_bench.rs now does too — so this would be the third instance of a pattern that should probably be a shared helper rather than a third copy.

Two things worth carrying over from #131's fix:

  1. Poll a format that actually carries the field. Its first version polled with response_format: "concise", which drops plugin_activation entirely — the probe read false forever, every phase paid a 60s deadline, the suite went from 2s to 181s, and it still reported ok. The green result hid it; only the runtime gave it away.
  2. Let the deadline expire loudly. If discovery never completes, the questions should run and the ceiling should fail — not skip.

Also worth doing while there

Establish whether the saturated leg can reach the window at all. If it structurally cannot, say so in the file with the reason, and the 25 tokens become a deliberate margin rather than an accident nobody has audited. If it can, the wait is required.

#131 (same race, same block, fixed there), #126 (the flake as first observed, before the mechanism was known).

Raised in #131's body as *"worth confirming before this is closed"*, and now checked. #131 is fixed and closed; this is the neighbour it warned about, kept open on its own because the answer is "not currently firing" rather than "cannot fire". ## The exposure - `OVERVIEW_SATURATED_MAX_TOKENS = 4300`, against a measured **4275** — **25 tokens of headroom**. - `PACKAGE_SET_UNCONSULTED_SEMANTICS` is **~190 tokens**, rendered whenever a client beats package discovery. - `crates/mcp-server/tests/overview_payload_budget_e2e.rs` contains **zero** occurrences of `wait` — verified by count, not by reading. It never waits for `package_set_consulted`. If that block ever lands in this reply, the test is **165 tokens over**, not 25 under. ## What was measured, and what it does not prove Six consecutive isolated runs on the daemon leg: **6/6 green**, runtimes flat at ~2.2s. That is weak evidence and should not be read as safety. The sibling `agent_task_plugin_bench` hit the same window **3 times in 13 isolated runs** — so a six-run clean sweep is well within what a bimodal race produces by chance. What would settle it is structural, and I did not establish it: whether the *saturated* leg's fixture has a package set whose discovery can lose the race. The file does describe a leg measured "over a store holding one wanted package" (and #100's `package_store` work added another), so the window is reachable in principle for at least some of its legs. ## Why it matters more than the headroom suggests That ceiling guards **the first call an agent makes**. Its own doc records that when it was last breached the response was to delete a four-line paragraph rather than raise the number — so the 25 tokens are deliberate, and the file already refuses the easy way out. A flake here would therefore arrive as a *payload* failure on the most-read surface, and the tempting fix — raise the ceiling — is the one thing that file argues against. Better to remove the race than to widen the band under pressure later. ## The fix The same one #131 took: **wait for `package_set_consulted` before measuring**, so the test grades one payload rather than whichever side of a startup race it landed on. `crates/mcp-server/tests/coverage_signature_e2e.rs` already does this, and `agent_task_plugin_bench.rs` now does too — so this would be the third instance of a pattern that should probably be a shared helper rather than a third copy. Two things worth carrying over from #131's fix: 1. **Poll a format that actually carries the field.** Its first version polled with `response_format: "concise"`, which drops `plugin_activation` entirely — the probe read `false` forever, every phase paid a 60s deadline, the suite went from 2s to 181s, and it still reported `ok`. The green result hid it; only the runtime gave it away. 2. **Let the deadline expire loudly.** If discovery never completes, the questions should run and the ceiling should fail — not skip. ## Also worth doing while there Establish whether the saturated leg can reach the window at all. If it structurally cannot, say so **in the file** with the reason, and the 25 tokens become a deliberate margin rather than an accident nobody has audited. If it can, the wait is required. ## Related #131 (same race, same block, fixed there), #126 (the flake as first observed, before the mechanism was known).
Author
Member

Triage 2026-09-06: LEFT OPEN, nothing has moved — and #98 is a near-duplicate whose arithmetic does not reconcile with this one. That mismatch is itself the first task.

Nothing shipped

This issue asked for one of two things: wait for package_set_consulted before measuring (as #131 did), or establish in-file that the saturated leg structurally cannot reach the window. Neither.

  • whole_word search_text("wait") restricted to *overview_payload_budget_e2e.rs: 0 hits (193 files match repo-wide, so the filter emptied it, not the query). No wait, no polling helper.
  • The file's only package_set_consulted occurrence is prose at crates/mcp-server/tests/overview_payload_budget_e2e.rs:245, about a deleted paragraph — not a probe.
  • OVERVIEW_SATURATED_MAX_TOKENS = 4_300 unchanged at :248, and its doc still reads "HEADROOM, RE-MEASURED: 25 tokens" (:239-241).
  • The block still ships unconditionally on consulted == false: crates/mcp-server/src/server.rs:4386-4391 inserts package_set_unconsulted_semantics (constant at :4852).
  • The existing REPORTED_BLOCKS guard does not catch this. overview_payload_budget_e2e.rs:450, used at :774, requires plugin_activation.availability == "reported" — which stays reported when only the package fields are missing. So the disclosure lands inside a "reported" block and goes straight against the ceiling.
  • search_text("#133") across the tree: 0 hits.

Same file, same constant, same 25-token headroom, same remedy family. But they name different blocks, and the numbers do not line up:

block measured size ever observed?
#98 resolver_degradation — serialised at server.rs:9511-9514, present only when the resolver degraded ~44 tokens / ~175 bytes (4319 vs 4275) yes — fired RED at load ~18
#133 package_set_unconsulted_semantics — server.rs:4386-4391 ~190-230 tokens never observed; would land ~165 over, not 19 over

So either these are two different disclosures riding the same thin margin, or one mechanism measured twice with a discrepancy nobody has reconciled. Closing either as a duplicate of the other today would lose one of the two measurements.

Recommended: merge them into one issue whose first task is to reconcile the two figures — because "budget the disclosure" cannot be specified until it is known which disclosure. #98's option 1 or 2 (budget it, or separate the disclosure from the ceiling) would cover both; #133's "wait for discovery" fix covers only #133.

Why this got more urgent, not less

Two independent measurements taken today put the payload against its ceiling from other directions:

  • #160: the startup payload is at 16,530 of 16,555 — 25 tokens of headroom, and tool descriptions grew +1,950 chars since filing.
  • #137: linked_projects has no cap and no fixture with any links, and each link adds ~106 tokens of semantics prose to this same overview payload.

Three separate ways to overrun a ceiling with under 1% headroom, none of which the gate can currently see. This is the same shape as #158: a gate blind to the population it exists for.

🤖 Triage lane, 2026-09-06, master 45cf6e4

## Triage 2026-09-06: LEFT OPEN, nothing has moved — and **#98 is a near-duplicate whose arithmetic does not reconcile with this one.** That mismatch is itself the first task. ### Nothing shipped This issue asked for one of two things: wait for `package_set_consulted` before measuring (as #131 did), or establish in-file that the saturated leg structurally cannot reach the window. **Neither.** - `whole_word` `search_text("wait")` restricted to `*overview_payload_budget_e2e.rs`: **0 hits** (193 files match repo-wide, so the filter emptied it, not the query). No wait, no polling helper. - The file's only `package_set_consulted` occurrence is prose at `crates/mcp-server/tests/overview_payload_budget_e2e.rs:245`, about a *deleted* paragraph — not a probe. - `OVERVIEW_SATURATED_MAX_TOKENS = 4_300` unchanged at `:248`, and its doc still reads *"HEADROOM, RE-MEASURED: **25** tokens"* (`:239-241`). - The block still ships unconditionally on `consulted == false`: `crates/mcp-server/src/server.rs:4386-4391` inserts `package_set_unconsulted_semantics` (constant at `:4852`). - **The existing `REPORTED_BLOCKS` guard does not catch this.** `overview_payload_budget_e2e.rs:450`, used at `:774`, requires `plugin_activation.availability == "reported"` — which stays `reported` when only the package fields are missing. So the disclosure lands *inside* a "reported" block and goes straight against the ceiling. - `search_text("#133")` across the tree: **0 hits.** ### #133 vs #98 — related, and NOT safe to merge as duplicates yet Same file, same constant, same 25-token headroom, same remedy family. But they name **different blocks**, and the numbers do not line up: | | block | measured size | ever observed? | |---|---|---|---| | **#98** | `resolver_degradation` — serialised at `server.rs:9511-9514`, present only when the resolver degraded | **~44 tokens / ~175 bytes** (4319 vs 4275) | **yes** — fired RED at load ~18 | | **#133** | `package_set_unconsulted_semantics` — `server.rs:4386-4391` | **~190-230 tokens** | **never observed**; would land ~165 *over*, not 19 over | So either these are two different disclosures riding the same thin margin, or one mechanism measured twice with a discrepancy nobody has reconciled. **Closing either as a duplicate of the other today would lose one of the two measurements.** Recommended: **merge them into one issue whose first task is to reconcile the two figures** — because "budget the disclosure" cannot be specified until it is known *which* disclosure. #98's option 1 or 2 (budget it, or separate the disclosure from the ceiling) would cover both; #133's "wait for discovery" fix covers only #133. ### Why this got more urgent, not less Two independent measurements taken today put the payload against its ceiling from other directions: - **#160**: the startup payload is at **16,530 of 16,555 — 25 tokens of headroom**, and tool descriptions grew +1,950 chars since filing. - **#137**: `linked_projects` has **no cap and no fixture with any links**, and each link adds ~106 tokens of semantics prose to this same overview payload. Three separate ways to overrun a ceiling with under 1% headroom, none of which the gate can currently see. This is the same shape as **#158**: a gate blind to the population it exists for. 🤖 Triage lane, 2026-09-06, master `45cf6e4`
Author
Member

FIXED — and the race is no longer hypothetical: it fired on the first attempt, and it put the payload 133 tokens OVER the ceiling.

Lane worktree /tmp/cosi-lane-budget, rebased onto origin/master (87a3fc8). Not pushed. The reconciliation with #98 and the full mutation record are on #98; this comment is the half that belongs here.

The evidence this issue said it did not have

The framing was "not currently firing" rather than "cannot fire", on six green isolated runs — correctly called weak evidence. It is now settled the other way.

Unmodified test, unmodified tree, machine carrying four other cargo lanes, first attempt:

saturated overview: 4433 raw estimated tokens (17756 bytes) over 162 files
  daemon-state disclosures: 378 est. tokens
    state                                                    7
    state_detail                                           102
    resolver_degradation                                    62
    plugin_activation.package_set_unconsulted_semantics    208   ← the block

4,433 against a 4,300 ceiling — 133 over. Two further runs on the same tree gave 4,232 and 4,133. Three verdicts, one tree, nothing changed between them.

Your arithmetic was right and the neighbouring triage's is the one that needed correcting: the block measures 208 wire tokens, inside your ~190–230 estimate. (#98's triage attributed a ~44-token load delta to resolver_degradation; that block is a constant 62 tokens, present in the settled state too. The load-conditional pair is state + state_detail.)

The fix, as this issue asked for it

measure_overview in crates/mcp-server/tests/overview_payload_budget_e2e.rs now waits the race out before measuring — the third instance of #131's pattern, with both carry-overs you asked for:

  1. The poll reads a format that carries the field — project_overview with default args, and the predicate is checked on the parsed body it is about to measure, so there is no second call to get out of step.
  2. The deadline expires LOUDLY. If discovery never completes, a warning names what is being measured and the ceilings decide. It never skips.

One correction to the recipe: the predicate is "the block is gone", not package_set_consulted == true. On a project with no plugin package host that field is absent forever and is itself the measurement, so waiting for true would spin to the deadline on every ordinary project — the same 60-second-per-phase shape #131 hit from the other direction.

After the wait, three consecutive runs under the same load: 4,133 / 4,133 / 4,133, content 3,989 every time.

The question you asked me to settle: CAN the saturated leg reach the window?

Yes, and it did. The issue asked for either the wait or an in-file statement that the leg structurally cannot reach it. The measurement answers it: the wait is required, and the 25 tokens were never a deliberate margin.

Not a third copy — but not a shared helper either

You noted this would be the third copy of the pattern and should probably be a helper. It is placed inside measure_overview, the single point every leg of this file already goes through, so this file has one instance rather than one per test. Extracting a cross-file helper shared with coverage_signature_e2e and agent_task_plugin_bench is still open and still worth doing.

The block stays bounded after the wait removes it from view

The cost of waiting is that this gate can no longer see the block, and leaving a 208-token disclosure ungraded because the gate that used to trip over it stopped meeting it is the #158 shape. So package_set_unconsulted_semantics_fits_the_overview_allowance in server.rs bounds the constant at its source (240-token allowance, with a floor beneath it). MUTATION (RUN): duplicate its last two sentences → RED at 258 tokens.

REPORTED_BLOCKS — your reading was right

plugin_activation.availability stays "reported" when only the package fields are missing, so that guard never could have caught this. Untouched; it grades the opposite direction and still should.

Gates

cargo fmt --all -- --check 0 · cargo clippy --workspace --all-targets -- -D warnings 0 · RUSTDOCFLAGS="-D warnings" cargo doc … 0 · cargo test --workspace --no-fail-fast 0 · COSI_E2E_LEG=daemon on this suite 0 · corpus ratchet executed=7, baselines untouched.

🤖 Payload-budget lane, 2026-09-06

## FIXED — and the race is **no longer hypothetical**: it fired on the first attempt, and it put the payload **133 tokens OVER** the ceiling. Lane worktree `/tmp/cosi-lane-budget`, rebased onto `origin/master` (`87a3fc8`). Not pushed. The reconciliation with #98 and the full mutation record are on **#98**; this comment is the half that belongs here. ### The evidence this issue said it did not have The framing was *"not currently firing" rather than "cannot fire"*, on six green isolated runs — correctly called weak evidence. It is now settled the other way. Unmodified test, unmodified tree, machine carrying four other cargo lanes, **first attempt**: ``` saturated overview: 4433 raw estimated tokens (17756 bytes) over 162 files daemon-state disclosures: 378 est. tokens state 7 state_detail 102 resolver_degradation 62 plugin_activation.package_set_unconsulted_semantics 208 ← the block ``` **4,433 against a 4,300 ceiling — 133 over.** Two further runs on the same tree gave 4,232 and 4,133. Three verdicts, one tree, nothing changed between them. Your arithmetic was right and the neighbouring triage's is the one that needed correcting: the block measures **208 wire tokens**, inside your ~190–230 estimate. (#98's triage attributed a ~44-token load delta to `resolver_degradation`; that block is a constant 62 tokens, present in the settled state too. The load-conditional pair is `state` + `state_detail`.) ### The fix, as this issue asked for it `measure_overview` in `crates/mcp-server/tests/overview_payload_budget_e2e.rs` now waits the race out before measuring — the third instance of #131's pattern, with both carry-overs you asked for: 1. **The poll reads a format that carries the field** — `project_overview` with default args, and the predicate is checked on the parsed body it is about to measure, so there is no second call to get out of step. 2. **The deadline expires LOUDLY.** If discovery never completes, a warning names what is being measured and the ceilings decide. It never skips. One correction to the recipe: **the predicate is "the block is gone", not `package_set_consulted == true`.** On a project with no plugin package host that field is absent forever and is itself the measurement, so waiting for `true` would spin to the deadline on every ordinary project — the same 60-second-per-phase shape #131 hit from the other direction. After the wait, three consecutive runs under the same load: **4,133 / 4,133 / 4,133**, content 3,989 every time. ### The question you asked me to settle: CAN the saturated leg reach the window? **Yes, and it did.** The issue asked for either the wait or an in-file statement that the leg structurally cannot reach it. The measurement answers it: the wait is required, and the 25 tokens were never a deliberate margin. ### Not a third copy — but not a shared helper either You noted this would be the third copy of the pattern and should probably be a helper. It is placed **inside `measure_overview`**, the single point every leg of this file already goes through, so this file has one instance rather than one per test. Extracting a cross-file helper shared with `coverage_signature_e2e` and `agent_task_plugin_bench` is still open and still worth doing. ### The block stays bounded after the wait removes it from view The cost of waiting is that this gate can no longer see the block, and leaving a 208-token disclosure ungraded because the gate that used to trip over it stopped meeting it is the #158 shape. So `package_set_unconsulted_semantics_fits_the_overview_allowance` in `server.rs` bounds the constant at its source (240-token allowance, with a floor beneath it). MUTATION (RUN): duplicate its last two sentences → RED at 258 tokens. ### `REPORTED_BLOCKS` — your reading was right `plugin_activation.availability` stays `"reported"` when only the package fields are missing, so that guard never could have caught this. Untouched; it grades the opposite direction and still should. ### Gates `cargo fmt --all -- --check` 0 · `cargo clippy --workspace --all-targets -- -D warnings` 0 · `RUSTDOCFLAGS="-D warnings" cargo doc …` 0 · `cargo test --workspace --no-fail-fast` 0 · `COSI_E2E_LEG=daemon` on this suite 0 · corpus ratchet `executed=7`, baselines untouched. 🤖 Payload-budget lane, 2026-09-06
Author
Member

CLOSING — verified on merged master fc329a8, both legs RUN

Close-out lane. Reconciled with #98 rather than closed as its duplicate: #98 was the load-dependent disclosure, this is the unconsulted-package-set race. Different mechanism, same file, separate evidence.

The wait exists, inside measure_overview (crates/mcp-server/tests/overview_payload_budget_e2e.rs:616-725) — placed at the single point every leg of the file passes through (overview_of :729, overview_with_store :1523), so this is one instance and not a third copy. The file's wait count is no longer zero.

Both carry-overs from #131 are honoured:

  • The poll reads the parsed body it is about to measure (:661, :678-684) — no second call that can get out of step, no concise format that drops the field.
  • The deadline expires loudly (:698-708): a WARNING naming unconsulted and state, then the measurement runs and the ceilings decide. It never skips. That is the shape #131 asked for.

One correction to the recipe, recorded in the file (:672-677): the predicate is "the block is gone", not package_set_consulted == true, because on a project with no plugin package host that field is absent forever and would spin every ordinary project to the deadline.

The question this issue asked to settle is answered by measurement, and the answer is not the comfortable one. :253-259 records the unmodified test on the unmodified tree measuring 4,433 / 4,232 / 4,133 raw, minutes apart. So the saturated leg does reach the window, and when it did it landed 133 tokens over, not 25 under. The "25 tokens of deliberate margin" reading is retired: it was luck, exactly as you said.

The block stays graded after the wait hides it from the e2e, which is the #158 shape avoided: package_set_unconsulted_semantics_fits_the_overview_allowance (crates/mcp-server/src/server.rs:19915-19942) bounds the 208-token constant at its source with a floor (n >= 120, so a trim-to-nothing cannot pass) as well as a ceiling.

RUN on this tree:

COSI_E2E_LEG=daemon overview_payload_budget_e2e   3 passed, 0 failed   exit 0
package_set_unconsulted_semantics_fits_the_overview_allowance ... ok    exit 0

REPORTED_BLOCKS untouched, per your own reading that it grades the opposite direction.

One item from the body deliberately not done, and it is the one you hedged: the wait is still not a shared helper — coverage_signature_e2e.rs and agent_task_plugin_bench.rs keep independent copies. Worth doing, not worth holding this open for.

## CLOSING — verified on merged master `fc329a8`, both legs RUN Close-out lane. Reconciled with #98 rather than closed as its duplicate: #98 was the load-dependent disclosure, this is the unconsulted-package-set race. Different mechanism, same file, separate evidence. **The wait exists**, inside `measure_overview` (`crates/mcp-server/tests/overview_payload_budget_e2e.rs:616-725`) — placed at the single point **every** leg of the file passes through (`overview_of` `:729`, `overview_with_store` `:1523`), so this is one instance and not a third copy. The file's `wait` count is no longer zero. **Both carry-overs from #131 are honoured:** - The poll reads the parsed body it is **about to measure** (`:661`, `:678-684`) — no second call that can get out of step, no `concise` format that drops the field. - The deadline **expires loudly** (`:698-708`): a `WARNING` naming `unconsulted` and `state`, then the measurement runs and the ceilings decide. It never skips. That is the shape #131 asked for. **One correction to the recipe, recorded in the file** (`:672-677`): the predicate is *"the block is gone"*, not `package_set_consulted == true`, because on a project with no plugin package host that field is absent forever and would spin every ordinary project to the deadline. **The question this issue asked to settle is answered by measurement, and the answer is not the comfortable one.** `:253-259` records the unmodified test on the unmodified tree measuring **4,433 / 4,232 / 4,133** raw, minutes apart. So the saturated leg *does* reach the window, and when it did it landed **133 tokens over**, not 25 under. The "25 tokens of deliberate margin" reading is retired: it was luck, exactly as you said. **The block stays graded after the wait hides it from the e2e**, which is the #158 shape avoided: `package_set_unconsulted_semantics_fits_the_overview_allowance` (`crates/mcp-server/src/server.rs:19915-19942`) bounds the 208-token constant at its source with a **floor** (`n >= 120`, so a trim-to-nothing cannot pass) as well as a ceiling. **RUN on this tree:** ``` COSI_E2E_LEG=daemon overview_payload_budget_e2e 3 passed, 0 failed exit 0 package_set_unconsulted_semantics_fits_the_overview_allowance ... ok exit 0 ``` `REPORTED_BLOCKS` untouched, per your own reading that it grades the opposite direction. **One item from the body deliberately not done**, and it is the one you hedged: the wait is still not a shared helper — `coverage_signature_e2e.rs` and `agent_task_plugin_bench.rs` keep independent copies. Worth doing, not worth holding this open for.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#133
No description provided.