No `cargo test` grades the Svelte extractor's rules — its fixtures run only in the release workflow, and three of its five rules are asserted nowhere #279

New issue

Open

opened 2026-09-15 19:47:34 +02:00 by buildagent · 0 comments

buildagent commented

2026-09-15 19:47:34 +02:00

Member

Found in a test-vacuity audit of 12bfe8f. This is the structural reason a boundary test could ship vacuous (fixed in e8d4694) and not be caught by any gate.

Measured

1. The .expected fixtures are graded by no cargo test.

tests/packages/svelte/fixtures/*.expected are exercised only by plugin check "${DIGEST}" inside .forgejo/workflows/release.yml's package-plugin-svelte job. ci.yml never runs plugin check; no test file under crates/*/tests/ invokes it on this package.

Contrast crates/daemon/tests/ruby_package_e2e.rs, which does exactly that for de.h-dv.ruby ("all five fixtures must pass under the grant"). The Ruby package has the gate; the Svelte package does not.

2. crates/guest/svelte is outside the workspace (#276), so cargo fmt --all, cargo clippy --workspace and the docs job never reach the extractor source either.

Together: none of the six verification gates CLAUDE.md lists grades the Svelte extractor's rules or its boundary. The audit lane had to replicate the release job by hand — pack, key, sign, trust, install, check — to grade them at all.

3. Three of the five rules have no assertion in any cargo test.

rule	emitted	asserted in a cargo test?
A — `{#snippet}` -> function symbol	yes	yes, `svelte_package_e2e`
B — `{@render row(x)}` -> call ref	yes	no
C — `<Header />` -> type ref	yes	yes, via the bind test
D — `onclick={h}` -> call ref	yes	no
E — `{a.b}` -> read on the head	yes	no

write_corpus in svelte_package_e2e.rs contains {@render legend(doubled)}, onclick={handleClick} and {label} — and nothing asserted any ref from them until e8d4694 added a ref-level reader for the boundary. The refs are produced; no cargo test looks at them.

Why this matters more than the coverage number

The audit ran the mutation that disables the <script> boundary entirely. plugin check went red on two fixtures — the fixtures work. The e2e test stayed green. So the one leg that caught the regression is the leg that runs only at release time, on a tag, after the commit has already landed on master.

That is the wrong end of the loop for a gate whose whole job is to catch a silent behaviour change in a sandboxed extractor that ships to operators.

GRAMMAR_BUILDS presence is not enforced. Deleting the svelte row from crates/plugin-host/tests/grammar_provenance.rs's GRAMMAR_BUILDS leaves all 7 tests green. The table has only !GRAMMAR_BUILDS.is_empty() — a typed-out list with no derivation and no floor. WASM_ARTIFACTS beside it is derived from a repository walk and does catch a deleted row; this one does not. Same shape release_gate.rs documents as how de.h-dv.ruby once shipped in no release.

Ten recorded mutations carry a pointer instead of a verdict. crates/abi/src/record.rs M-#229-a..d and crates/indexer/src/packages.rs M-#229-e..j all read "RESULT: recorded at the run"; svelte_package_e2e.rs said "see the report on this change". The audit ran all ten and every one is RED, each killing exactly the test its doc names — so the tests are sound and only the record is thin. svelte_kind_table.rs is the counter-example done right: its verdicts are quoted inline and all reproduce, including the two survivors.

Suggested shape

A svelte_package_e2e test that runs plugin check on the shipped package under the grant, mirroring ruby_package_e2e. That is the single change that closes the largest gap.
Ref-level assertions for rules B, D and E. e8d4694 adds a refs_in reader to that file; these are three asserts on top of machinery that now exists.
Derive GRAMMAR_BUILDS' population, or give it a floor, so a deleted row is red.
Replace "RESULT: recorded at the run" with the observed verdict in the ten sites above — the values are in the audit and every one reproduced.

Items 1 and 2 are the ones that would have caught the vacuous boundary test at cargo test time rather than at release time.

Found in a test-vacuity audit of 12bfe8f. This is the structural reason a boundary test could ship vacuous (fixed in e8d4694) and not be caught by any gate. ## Measured **1. The `.expected` fixtures are graded by no `cargo test`.** `tests/packages/svelte/fixtures/*.expected` are exercised only by `plugin check "${DIGEST}"` inside `.forgejo/workflows/release.yml`'s `package-plugin-svelte` job. `ci.yml` never runs `plugin check`; no test file under `crates/*/tests/` invokes it on this package. Contrast `crates/daemon/tests/ruby_package_e2e.rs`, which does exactly that for `de.h-dv.ruby` ("all five fixtures must pass under the grant"). The Ruby package has the gate; the Svelte package does not. **2. `crates/guest/svelte` is outside the workspace** (#276), so `cargo fmt --all`, `cargo clippy --workspace` and the `docs` job never reach the extractor source either. **Together:** *none of the six verification gates CLAUDE.md lists grades the Svelte extractor's rules or its boundary.* The audit lane had to replicate the release job by hand — pack, key, sign, trust, install, check — to grade them at all. **3. Three of the five rules have no assertion in any `cargo test`.** | rule | emitted | asserted in a cargo test? | | :-- | :-- | :-- | | A — `{#snippet}` -> function symbol | yes | **yes**, `svelte_package_e2e` | | B — `{@render row(x)}` -> call ref | yes | **no** | | C — `<Header />` -> type ref | yes | **yes**, via the bind test | | D — `onclick={h}` -> call ref | yes | **no** | | E — `{a.b}` -> read on the head | yes | **no** | `write_corpus` in `svelte_package_e2e.rs` contains `{@render legend(doubled)}`, `onclick={handleClick}` and `{label}` — and nothing asserted any ref from them until e8d4694 added a ref-level reader for the boundary. The refs are produced; no cargo test looks at them. ## Why this matters more than the coverage number The audit ran the mutation that disables the `<script>` boundary entirely. `plugin check` went **red** on two fixtures — the fixtures work. The e2e test stayed **green**. So the one leg that caught the regression is the leg that runs only at release time, on a tag, after the commit has already landed on master. That is the wrong end of the loop for a gate whose whole job is to catch a silent behaviour change in a sandboxed extractor that ships to operators. ## Related, same audit **`GRAMMAR_BUILDS` presence is not enforced.** Deleting the svelte row from `crates/plugin-host/tests/grammar_provenance.rs`'s `GRAMMAR_BUILDS` leaves **all 7 tests green**. The table has only `!GRAMMAR_BUILDS.is_empty()` — a typed-out list with no derivation and no floor. `WASM_ARTIFACTS` beside it is derived from a repository walk and does catch a deleted row; this one does not. Same shape `release_gate.rs` documents as how `de.h-dv.ruby` once shipped in no release. **Ten recorded mutations carry a pointer instead of a verdict.** `crates/abi/src/record.rs` M-#229-a..d and `crates/indexer/src/packages.rs` M-#229-e..j all read "RESULT: recorded at the run"; `svelte_package_e2e.rs` said "see the report on this change". The audit ran all ten and **every one is RED**, each killing exactly the test its doc names — so the tests are sound and only the record is thin. `svelte_kind_table.rs` is the counter-example done right: its verdicts are quoted inline and all reproduce, including the two survivors. ## Suggested shape 1. A `svelte_package_e2e` test that runs `plugin check` on the shipped package under the grant, mirroring `ruby_package_e2e`. That is the single change that closes the largest gap. 2. Ref-level assertions for rules B, D and E. e8d4694 adds a `refs_in` reader to that file; these are three asserts on top of machinery that now exists. 3. Derive `GRAMMAR_BUILDS`' population, or give it a floor, so a deleted row is red. 4. Replace "RESULT: recorded at the run" with the observed verdict in the ten sites above — the values are in the audit and every one reproduced. Items 1 and 2 are the ones that would have caught the vacuous boundary test at `cargo test` time rather than at release time.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

h-dv/code-index#279

No description provided.

Rows
Columns

No cargo test grades the Svelte extractor's rules — its fixtures run only in the release workflow, and three of its five rules are asserted nowhere #279

Measured

Why this matters more than the coverage number

Related, same audit

Suggested shape

No `cargo test` grades the Svelte extractor's rules — its fixtures run only in the release workflow, and three of its five rules are asserted nowhere #279