h-dv/code-index

Watch

Fork 0

Package authoring never tells an author to size their own reply buffer — it has now caused two total-loss bugs in two packages #228

New issue

Closed

opened 2026-09-08 15:55:26 +02:00 by buildagent · 2 comments

buildagent commented

2026-09-08 15:55:26 +02:00

Member

The two occurrences

de.h-dv.xaml (ours) — $max_tree_bytes = 320000, which cut real markup from roughly 64-80 KB of source. About fifty files affected on a 14,056-file production repository, including every theme dictionary. Found by a production report, not by us. Fixed in #222/#223 by re-deriving all three budgets from FILE_SIZE_LIMIT_BYTES.

de.h-dv.timeline (external, in development) — OUT_LEN = 2 MiB, justified in a doc comment by "the largest .dataset in the corpus is 233 KB". Their measurement, quoted with permission:

generated 2,097,085 bytes of <Field TableName="tbl_N" Name="col_N"/>
44,146 elements -> 88,292 refs expected

probe      REFUSED
extracted  0 of 1 claimed [0 symbols, 0 refs]
refused    1 × abi.frame_truncated

A legal file under FILE_SIZE_LIMIT_BYTES — so the indexer hands it to the extractor — lost every fact in the file. Not truncated. Zero.

Same defect, same cause, two packages, arrived at independently. That makes it ours, not theirs.

Why the authoring guide is the right place to fix it

_prdoc/guides/80-package-authoring.md has a [limits] section, and it says only that a package may only reduce host limits. That is true and it is not the hazard. The reply buffer is not a manifest limit — it is a constant inside the guest that nothing in the package model can see, check, or refuse. So:

the manifest cannot express it;
plugin check cannot catch it, because C1 runs the package's own fixtures, and an author who sized from their corpus has fixtures from that same corpus;
nothing fails until a real file arrives, and then the failure is total rather than partial.

And the one worked example an author copies, crates/guest/example, uses OUT_LEN = 1 << 16 — 64 KiB. Anyone starting from it and not thinking hard inherits a buffer two orders of magnitude below the largest input they will be handed.

What the fix is

A short, explicit passage in 80-package-authoring.md §1, beside the cargo build chain, naming the two numbers that actually bound an author:

code_index_core::FILE_SIZE_LIMIT_BYTES (2 MiB default) — the largest file the indexer will hand any extractor. Size for this, never for your corpus.
MAX_RESPONSE_BYTES (16 MiB) — a reply larger than this cannot be delivered no matter how big the guest's buffer is.

And the point neither author worked out until after being bitten: a bigger buffer only raises the cliff. The TimeLine author put it precisely — a <Sql> body of :a,:b,:c… spends 3 source bytes per binding and buys a 30-byte ref, so 2 MiB of source can still demand ~20 MB, past both their buffer and our response cap. The guidance must therefore be size the buffer AND carry a budget that discloses, not pick a bigger number.

Their fix, which is the shape to document: OUT_LEN 2 → 8 MiB, walk stops at OUT_LEN - 4096 and emits extract.node_limit_reached. Result on the same adversarial file: extracted 1 of 1 [1 symbol, 235,988 refs] — a disclosed partial instead of a silent total loss.

Also worth stating in the guide: the example's 64 KiB is an example-scale number, not a starting point.

Second, smaller finding from the same report

abi.frame_truncated on a guest that returned zero bytes deliberately (its overflow path, because emitting a prefix would read as a complete answer) is indistinguishable, from the operator's side, from a guest that returned a malformed frame. The first points at a buffer; the second points at a bug.

Note this is adjacent to work just landed: abi.frame_truncated had three producers printing identical text, and each now names itself with a witness (no envelope; the worker exited N (0x…), N of 5 envelope bytes arrived, a complete 5-byte envelope whose payload did not decode). A live worker that deliberately answers zero is a fourth case and does not obviously fall into any of those three cleanly — worth checking whether it currently reads as died_without_answering with a success exit status, which would be misleading in its own way.

Low priority relative to the guide passage, but it is the difference between an operator raising a buffer and an operator filing a bug against the package.

Mutations

Remove the sizing passage → a docs gate pinning the two constant names in that section must go RED. (The section currently names neither, so this gate does not exist.)
Change FILE_SIZE_LIMIT_BYTES without updating the guide → the same gate must go RED, so the passage cannot rot away from the number it cites.
For the fourth-case finding: make a live guest return zero bytes → the witness must not claim the worker died.

Credit

Both the diagnosis and the fix shape come from the TimeLine plugin author, who read our XAML post-mortem, went and measured their own extractor against the rule rather than assuming it did not apply to them, and reported the result. That is the second defect this week found by someone outside this repository looking at real output.

Two packages, written independently, have now shipped a guest reply buffer sized from the corpus in front of the author instead of from what the indexer will actually hand them. Both lost facts on legal input. The second was found only because the first's post-mortem was posted in a chat channel where the second author happened to read it. ## The two occurrences **`de.h-dv.xaml`** (ours) — `$max_tree_bytes = 320000`, which cut real markup from roughly 64-80 KB of source. About fifty files affected on a 14,056-file production repository, including every theme dictionary. Found by a production report, not by us. Fixed in #222/#223 by re-deriving all three budgets from `FILE_SIZE_LIMIT_BYTES`. **`de.h-dv.timeline`** (external, in development) — `OUT_LEN = 2 MiB`, justified in a doc comment by *"the largest `.dataset` in the corpus is 233 KB"*. Their measurement, quoted with permission: ``` generated 2,097,085 bytes of <Field TableName="tbl_N" Name="col_N"/> 44,146 elements -> 88,292 refs expected probe REFUSED extracted 0 of 1 claimed [0 symbols, 0 refs] refused 1 × abi.frame_truncated ``` A legal file **under** `FILE_SIZE_LIMIT_BYTES` — so the indexer hands it to the extractor — lost **every fact in the file**. Not truncated. Zero. Same defect, same cause, two packages, arrived at independently. That makes it ours, not theirs. ## Why the authoring guide is the right place to fix it `_prdoc/guides/80-package-authoring.md` has a `[limits]` section, and it says only that a package **may only reduce** host limits. That is true and it is not the hazard. The reply buffer is **not a manifest limit** — it is a constant inside the guest that nothing in the package model can see, check, or refuse. So: * the manifest cannot express it; * `plugin check` cannot catch it, because C1 runs the package's own fixtures, and an author who sized from their corpus has fixtures from that same corpus; * nothing fails until a real file arrives, and then the failure is total rather than partial. And the one worked example an author copies, `crates/guest/example`, uses `OUT_LEN = 1 << 16` — 64 KiB. Anyone starting from it and not thinking hard inherits a buffer two orders of magnitude below the largest input they will be handed. ## What the fix is A short, explicit passage in `80-package-authoring.md` §1, beside the `cargo build` chain, naming the two numbers that actually bound an author: * **`code_index_core::FILE_SIZE_LIMIT_BYTES`** (2 MiB default) — the largest file the indexer will hand any extractor. Size for this, never for your corpus. * **`MAX_RESPONSE_BYTES`** (16 MiB) — a reply larger than this cannot be delivered no matter how big the guest's buffer is. And the point neither author worked out until after being bitten: **a bigger buffer only raises the cliff.** The TimeLine author put it precisely — a `<Sql>` body of `:a,:b,:c…` spends 3 source bytes per binding and buys a 30-byte ref, so 2 MiB of source can still demand ~20 MB, past both their buffer and our response cap. The guidance must therefore be *size the buffer AND carry a budget that discloses*, not *pick a bigger number*. Their fix, which is the shape to document: `OUT_LEN` 2 → 8 MiB, walk stops at `OUT_LEN - 4096` and emits `extract.node_limit_reached`. Result on the same adversarial file: `extracted 1 of 1 [1 symbol, 235,988 refs]` — a disclosed partial instead of a silent total loss. Also worth stating in the guide: the example's 64 KiB is an example-scale number, not a starting point. ## Second, smaller finding from the same report `abi.frame_truncated` on a guest that returned **zero bytes deliberately** (its overflow path, because emitting a prefix would read as a complete answer) is indistinguishable, from the operator's side, from a guest that returned a malformed frame. The first points at a buffer; the second points at a bug. Note this is adjacent to work just landed: `abi.frame_truncated` had three producers printing identical text, and each now names itself with a witness (`no envelope; the worker exited N (0x…)`, `N of 5 envelope bytes arrived`, `a complete 5-byte envelope whose payload did not decode`). A live worker that deliberately answers zero is a **fourth** case and does not obviously fall into any of those three cleanly — worth checking whether it currently reads as `died_without_answering` with a success exit status, which would be misleading in its own way. Low priority relative to the guide passage, but it is the difference between an operator raising a buffer and an operator filing a bug against the package. ## Mutations * Remove the sizing passage → a docs gate pinning the two constant names in that section must go RED. (The section currently names neither, so this gate does not exist.) * Change `FILE_SIZE_LIMIT_BYTES` without updating the guide → the same gate must go RED, so the passage cannot rot away from the number it cites. * For the fourth-case finding: make a live guest return zero bytes → the witness must not claim the worker died. ## Credit Both the diagnosis and the fix shape come from the TimeLine plugin author, who read our XAML post-mortem, went and measured their own extractor against the rule rather than assuming it did not apply to them, and reported the result. That is the second defect this week found by someone outside this repository looking at real output.

dhoyer referenced this issue from a commit

2026-09-09 12:27:14 +02:00

feat: a package author derives the reply buffer, and a full one says so (#228)

dhoyer referenced this issue from a commit

2026-09-09 12:27:14 +02:00

chore: move the SDK pin to the commit that lands the reply-buffer SDK (#228)

buildagent commented

2026-09-09 12:27:41 +02:00

Author

Member

Fixed on master at 915c850 (40e4602 + 915c850).

What shipped

crates/guest/src/budget.rs — the two bounds an author can actually derive from, and the arithmetic between them:

MAX_INPUT_BYTES (2 MiB) — what the host will HAND you
MAX_REPLY_BYTES (16 MiB) — what it will TAKE BACK
reply_buffer_bytes(density) — MAX_INPUT_BYTES * density, floored at MIN_REPLY_BUFFER_BYTES, clamped to MAX_REPLY_BYTES
clamped(density) — const fn, so an author can const _: () = assert!(…) at build time that their format never admits an undeliverable legal file
fact_budget(buffer) — where the walk must stop so the disclosure still fits

Plus Budgeted and Encoder::finish_or_disclose, so a buffer that fills anyway surfaces as extract.node_limit_reached (a disclosed partial) instead of abi.frame_truncated (which reads as "this producer is broken" and sends the reader after a bug that isn't there).

Both mirrored constants are pinned against the host's own by crates/plugin-host/tests/guest_reply_budget.rs, from a crate that links both sides — the duplication exists because this crate has zero dependencies (that is what makes fork/open/connect inexpressible in a guest), so it does not get to drift.

A correction to the brief I wrote

I told the lane to derive from FILE_SIZE_LIMIT_BYTES. That is the compile-time default, not the enforced value: effective_file_size_limit() (crates/core/src/limits.rs:83) is seeded from max_file_size_mb in .code-index.toml and may go up to FILE_SIZE_CEILING_BYTES = 1 GiB. A guest cannot read it — the extract call carries no configuration.

So on a repo whose operator raised the limit, a buffer derived from MAX_INPUT_BYTES is undersized by construction, and the only correct behaviour left is to say so. That is why sizing alone was never the whole fix, and why finish_or_disclose is the load-bearing half. The doc comment on MAX_INPUT_BYTES states this rather than implying the number is a guarantee.

Mutation results

The lane reported M1b RED; I re-ran it independently on the merged tree rather than taking the report as the criterion.

reply_buffer_bytes → … / 2: RED, by three tests, one of which the lane did not credit:

the_clamp_predicate_and_the_clamp_are_the_same_rule ......... FAILED
  clamped(9) says true but reply_buffer_bytes(9) returned 9437184
  (cap is 16777216) — two predicates about one rule must not disagree
a_reply_buffer_derived_from_the_bound_holds_a_legal_maximum_input .. FAILED
an_adequate_buffer_on_the_same_input_emits_facts_and_no_diagnostic . FAILED
test result: FAILED. 7 passed; 3 failed

Restored by cp from snapshot, md5 verified equal to the original (2a0cd…c2c4), touched, re-run 10 passed; 0 failed.

One survivor, reported as a survivor: M1 (MAX_INPUT_BYTES → 256 KiB) does not go red. The derivation test measures density from the same shrunken input, so it grades the arithmetic, not the constant. The constant is instead pinned by the host-side equality test — which is the right grader for it — but no test fails on the constant alone from inside the guest crate, and there cannot be one: the guest crate has no access to the host value. Recording it here rather than papering over it.

Byte-compare gates

The sharpest engineering call in this lane was making Budgeted a separate type rather than adding fields to Encoder. Fields on Encoder changed Encoder::new's codegen and moved every shipped wasm artifact — rustc_guest.wasm 2965→2976, ruby/extractor.wasm 24337→24353, timeline/extractor.wasm 16428→16460 — which would have forced two signed .cip re-signs for a change that adds no facts. As a separate type, nothing moved:

ruby_guest_wire        -> EXIT=0   5 passed
timeline_guest_rebuild -> EXIT=0   1 passed
guest_globals          -> EXIT=0  14 passed
grammar_provenance     -> EXIT=0   7 passed

Merged-tree verification

cargo fmt --all -- --check clean; guest_reply_budget 10, wire_parity 11, guest_boundary 4, sdk_pin 2, bounding_site_registry 17 — all EXIT=0.

Closing. The two motivating packages (de.h-dv.xaml, fixed in #222/#223; de.h-dv.timeline, OUT_LEN justified by "the largest .dataset in the corpus is 233 KB" and losing every fact on a legal 2,097,085-byte file) are the reason the API makes you state a density instead of a number.

Fixed on `master` at 915c850 (`40e4602` + `915c850`). ## What shipped `crates/guest/src/budget.rs` — the two bounds an author can actually derive from, and the arithmetic between them: - `MAX_INPUT_BYTES` (2 MiB) — what the host will HAND you - `MAX_REPLY_BYTES` (16 MiB) — what it will TAKE BACK - `reply_buffer_bytes(density)` — `MAX_INPUT_BYTES * density`, floored at `MIN_REPLY_BUFFER_BYTES`, clamped to `MAX_REPLY_BYTES` - `clamped(density)` — `const fn`, so an author can `const _: () = assert!(…)` at build time that their format never admits an undeliverable legal file - `fact_budget(buffer)` — where the walk must stop so the disclosure still fits Plus `Budgeted` and `Encoder::finish_or_disclose`, so a buffer that fills **anyway** surfaces as `extract.node_limit_reached` (a disclosed partial) instead of `abi.frame_truncated` (which reads as "this producer is broken" and sends the reader after a bug that isn't there). Both mirrored constants are pinned against the host's own by `crates/plugin-host/tests/guest_reply_budget.rs`, from a crate that links both sides — the duplication exists because this crate has zero dependencies (that is what makes `fork`/`open`/`connect` inexpressible in a guest), so it does not get to drift. ## A correction to the brief I wrote I told the lane to derive from `FILE_SIZE_LIMIT_BYTES`. That is the **compile-time default**, not the enforced value: `effective_file_size_limit()` (`crates/core/src/limits.rs:83`) is seeded from `max_file_size_mb` in `.code-index.toml` and may go up to `FILE_SIZE_CEILING_BYTES` = 1 GiB. **A guest cannot read it** — the `extract` call carries no configuration. So on a repo whose operator raised the limit, a buffer derived from `MAX_INPUT_BYTES` is undersized *by construction*, and the only correct behaviour left is to say so. That is why sizing alone was never the whole fix, and why `finish_or_disclose` is the load-bearing half. The doc comment on `MAX_INPUT_BYTES` states this rather than implying the number is a guarantee. ## Mutation results The lane reported M1b RED; I re-ran it independently on the merged tree rather than taking the report as the criterion. `reply_buffer_bytes` → `… / 2`: **RED, by three tests**, one of which the lane did not credit: ``` the_clamp_predicate_and_the_clamp_are_the_same_rule ......... FAILED clamped(9) says true but reply_buffer_bytes(9) returned 9437184 (cap is 16777216) — two predicates about one rule must not disagree a_reply_buffer_derived_from_the_bound_holds_a_legal_maximum_input .. FAILED an_adequate_buffer_on_the_same_input_emits_facts_and_no_diagnostic . FAILED test result: FAILED. 7 passed; 3 failed ``` Restored by `cp` from snapshot, md5 verified equal to the original (`2a0cd…c2c4`), `touch`ed, re-run `10 passed; 0 failed`. **One survivor, reported as a survivor:** M1 (`MAX_INPUT_BYTES` → 256 KiB) does not go red. The derivation test measures density from the same shrunken input, so it grades the *arithmetic*, not the *constant*. The constant is instead pinned by the host-side equality test — which is the right grader for it — but no test fails on the constant alone from inside the guest crate, and there cannot be one: the guest crate has no access to the host value. Recording it here rather than papering over it. ## Byte-compare gates The sharpest engineering call in this lane was making `Budgeted` a **separate type** rather than adding fields to `Encoder`. Fields on `Encoder` changed `Encoder::new`'s codegen and moved every shipped wasm artifact — `rustc_guest.wasm` 2965→2976, `ruby/extractor.wasm` 24337→24353, `timeline/extractor.wasm` 16428→16460 — which would have forced **two signed `.cip` re-signs** for a change that adds no facts. As a separate type, nothing moved: ``` ruby_guest_wire -> EXIT=0 5 passed timeline_guest_rebuild -> EXIT=0 1 passed guest_globals -> EXIT=0 14 passed grammar_provenance -> EXIT=0 7 passed ``` ## Merged-tree verification `cargo fmt --all -- --check` clean; `guest_reply_budget` 10, `wire_parity` 11, `guest_boundary` 4, `sdk_pin` 2, `bounding_site_registry` 17 — all EXIT=0. Closing. The two motivating packages (`de.h-dv.xaml`, fixed in #222/#223; `de.h-dv.timeline`, `OUT_LEN` justified by "the largest `.dataset` in the corpus is 233 KB" and losing *every* fact on a legal 2,097,085-byte file) are the reason the API makes you state a density instead of a number.

buildagent closed this issue

2026-09-09 12:27:44 +02:00

buildagent commented

2026-09-09 12:28:34 +02:00

Author

Member

On the second, smaller finding — I checked it rather than letting it close silently with the main one.

The question was whether a live guest that returns zero bytes deliberately reads as "died" with a success exit status. It does not, and the case is graded:

Exited cleanly, said nothing → WorkerExit::Code(0) => Reason::AbiDecodeFailed (supervise.rs:598), commented "that is not a death, it is a producer that does not implement the protocol" — which is the right claim.
Still alive, said nothing → witness no envelope; the worker is still running (exit_witness, supervise.rs:556), asserted by every_death_names_its_exit_not_only_the_catch_all (crates/plugin-host/tests/interpret.rs:877), whose second half exists specifically for "the third state the one reason code covers and the only one with no death at all."

So no witness claims a death that did not happen.

What remains true, and is deliberate rather than a gap: the live-zero case still carries the reason code abi.frame_truncated, because died_without_answering is a documented four-into-three mapping. The design decision already recorded in short_envelope's doc comment is that the code is disambiguated by a witness, not by minting a fourth code — and each of the three producers now carries one that names which it is. The live-zero case is inside that scheme, not outside it.

Worth noting the #228 fix also removes the motivating instance: a guest using finish_or_disclose no longer returns zero on overflow — it emits extract.node_limit_reached. The zero-return path is now reached only by a guest that does not implement the protocol, which is exactly what abi.decode_failed says.

Both findings addressed; issue stays closed.

On the **second, smaller finding** — I checked it rather than letting it close silently with the main one. The question was whether a live guest that returns zero bytes *deliberately* reads as "died" with a success exit status. It does not, and the case is graded: - **Exited cleanly, said nothing** → `WorkerExit::Code(0) => Reason::AbiDecodeFailed` (`supervise.rs:598`), commented *"that is not a death, it is a producer that does not implement the protocol"* — which is the right claim. - **Still alive, said nothing** → witness `no envelope; the worker is still running` (`exit_witness`, `supervise.rs:556`), asserted by `every_death_names_its_exit_not_only_the_catch_all` (`crates/plugin-host/tests/interpret.rs:877`), whose second half exists specifically for *"the third state the one reason code covers and the only one with no death at all."* So no witness claims a death that did not happen. What remains true, and is deliberate rather than a gap: the live-zero case still carries the reason **code** `abi.frame_truncated`, because `died_without_answering` is a documented four-into-three mapping. The design decision already recorded in `short_envelope`'s doc comment is that the code is disambiguated by a **witness**, not by minting a fourth code — and each of the three producers now carries one that names which it is. The live-zero case is inside that scheme, not outside it. Worth noting the #228 fix also *removes the motivating instance*: a guest using `finish_or_disclose` no longer returns zero on overflow — it emits `extract.node_limit_reached`. The zero-return path is now reached only by a guest that does not implement the protocol, which is exactly what `abi.decode_failed` says. Both findings addressed; issue stays closed.

buildagent referenced this issue

2026-09-09 12:37:44 +02:00

Every agent lane forfeits symbol search for the code it just wrote: worktrees live under .claude/, which is permanently unindexable #238

buildagent referenced this issue from a commit

2026-09-15 16:58:02 +02:00

chore(guest): move the SDK pin to the commit that shipped the SDK