No cargo test grades the Svelte extractor's rules — its fixtures run only in the release workflow, and three of its five rules are asserted nowhere #279

Closed
opened 2026-09-15 19:47:34 +02:00 by buildagent · 3 comments
Member

Found in a test-vacuity audit of 12bfe8f. This is the structural reason a boundary test could ship vacuous (fixed in e8d4694) and not be caught by any gate.

Measured

1. The .expected fixtures are graded by no cargo test.

tests/packages/svelte/fixtures/*.expected are exercised only by plugin check "${DIGEST}" inside .forgejo/workflows/release.yml's package-plugin-svelte job. ci.yml never runs plugin check; no test file under crates/*/tests/ invokes it on this package.

Contrast crates/daemon/tests/ruby_package_e2e.rs, which does exactly that for de.h-dv.ruby ("all five fixtures must pass under the grant"). The Ruby package has the gate; the Svelte package does not.

2. crates/guest/svelte is outside the workspace (#276), so cargo fmt --all, cargo clippy --workspace and the docs job never reach the extractor source either.

Together: none of the six verification gates CLAUDE.md lists grades the Svelte extractor's rules or its boundary. The audit lane had to replicate the release job by hand — pack, key, sign, trust, install, check — to grade them at all.

3. Three of the five rules have no assertion in any cargo test.

rule emitted asserted in a cargo test?
A — {#snippet} -> function symbol yes yes, svelte_package_e2e
B — {@render row(x)} -> call ref yes no
C — <Header /> -> type ref yes yes, via the bind test
D — onclick={h} -> call ref yes no
E — {a.b} -> read on the head yes no

write_corpus in svelte_package_e2e.rs contains {@render legend(doubled)}, onclick={handleClick} and {label} — and nothing asserted any ref from them until e8d4694 added a ref-level reader for the boundary. The refs are produced; no cargo test looks at them.

Why this matters more than the coverage number

The audit ran the mutation that disables the <script> boundary entirely. plugin check went red on two fixtures — the fixtures work. The e2e test stayed green. So the one leg that caught the regression is the leg that runs only at release time, on a tag, after the commit has already landed on master.

That is the wrong end of the loop for a gate whose whole job is to catch a silent behaviour change in a sandboxed extractor that ships to operators.

GRAMMAR_BUILDS presence is not enforced. Deleting the svelte row from crates/plugin-host/tests/grammar_provenance.rs's GRAMMAR_BUILDS leaves all 7 tests green. The table has only !GRAMMAR_BUILDS.is_empty() — a typed-out list with no derivation and no floor. WASM_ARTIFACTS beside it is derived from a repository walk and does catch a deleted row; this one does not. Same shape release_gate.rs documents as how de.h-dv.ruby once shipped in no release.

Ten recorded mutations carry a pointer instead of a verdict. crates/abi/src/record.rs M-#229-a..d and crates/indexer/src/packages.rs M-#229-e..j all read "RESULT: recorded at the run"; svelte_package_e2e.rs said "see the report on this change". The audit ran all ten and every one is RED, each killing exactly the test its doc names — so the tests are sound and only the record is thin. svelte_kind_table.rs is the counter-example done right: its verdicts are quoted inline and all reproduce, including the two survivors.

Suggested shape

  1. A svelte_package_e2e test that runs plugin check on the shipped package under the grant, mirroring ruby_package_e2e. That is the single change that closes the largest gap.
  2. Ref-level assertions for rules B, D and E. e8d4694 adds a refs_in reader to that file; these are three asserts on top of machinery that now exists.
  3. Derive GRAMMAR_BUILDS' population, or give it a floor, so a deleted row is red.
  4. Replace "RESULT: recorded at the run" with the observed verdict in the ten sites above — the values are in the audit and every one reproduced.

Items 1 and 2 are the ones that would have caught the vacuous boundary test at cargo test time rather than at release time.

Found in a test-vacuity audit of 12bfe8f. This is the structural reason a boundary test could ship vacuous (fixed in e8d4694) and not be caught by any gate. ## Measured **1. The `.expected` fixtures are graded by no `cargo test`.** `tests/packages/svelte/fixtures/*.expected` are exercised only by `plugin check "${DIGEST}"` inside `.forgejo/workflows/release.yml`'s `package-plugin-svelte` job. `ci.yml` never runs `plugin check`; no test file under `crates/*/tests/` invokes it on this package. Contrast `crates/daemon/tests/ruby_package_e2e.rs`, which does exactly that for `de.h-dv.ruby` ("all five fixtures must pass under the grant"). The Ruby package has the gate; the Svelte package does not. **2. `crates/guest/svelte` is outside the workspace** (#276), so `cargo fmt --all`, `cargo clippy --workspace` and the `docs` job never reach the extractor source either. **Together:** *none of the six verification gates CLAUDE.md lists grades the Svelte extractor's rules or its boundary.* The audit lane had to replicate the release job by hand — pack, key, sign, trust, install, check — to grade them at all. **3. Three of the five rules have no assertion in any `cargo test`.** | rule | emitted | asserted in a cargo test? | | :-- | :-- | :-- | | A — `{#snippet}` -> function symbol | yes | **yes**, `svelte_package_e2e` | | B — `{@render row(x)}` -> call ref | yes | **no** | | C — `<Header />` -> type ref | yes | **yes**, via the bind test | | D — `onclick={h}` -> call ref | yes | **no** | | E — `{a.b}` -> read on the head | yes | **no** | `write_corpus` in `svelte_package_e2e.rs` contains `{@render legend(doubled)}`, `onclick={handleClick}` and `{label}` — and nothing asserted any ref from them until e8d4694 added a ref-level reader for the boundary. The refs are produced; no cargo test looks at them. ## Why this matters more than the coverage number The audit ran the mutation that disables the `<script>` boundary entirely. `plugin check` went **red** on two fixtures — the fixtures work. The e2e test stayed **green**. So the one leg that caught the regression is the leg that runs only at release time, on a tag, after the commit has already landed on master. That is the wrong end of the loop for a gate whose whole job is to catch a silent behaviour change in a sandboxed extractor that ships to operators. ## Related, same audit **`GRAMMAR_BUILDS` presence is not enforced.** Deleting the svelte row from `crates/plugin-host/tests/grammar_provenance.rs`'s `GRAMMAR_BUILDS` leaves **all 7 tests green**. The table has only `!GRAMMAR_BUILDS.is_empty()` — a typed-out list with no derivation and no floor. `WASM_ARTIFACTS` beside it is derived from a repository walk and does catch a deleted row; this one does not. Same shape `release_gate.rs` documents as how `de.h-dv.ruby` once shipped in no release. **Ten recorded mutations carry a pointer instead of a verdict.** `crates/abi/src/record.rs` M-#229-a..d and `crates/indexer/src/packages.rs` M-#229-e..j all read "RESULT: recorded at the run"; `svelte_package_e2e.rs` said "see the report on this change". The audit ran all ten and **every one is RED**, each killing exactly the test its doc names — so the tests are sound and only the record is thin. `svelte_kind_table.rs` is the counter-example done right: its verdicts are quoted inline and all reproduce, including the two survivors. ## Suggested shape 1. A `svelte_package_e2e` test that runs `plugin check` on the shipped package under the grant, mirroring `ruby_package_e2e`. That is the single change that closes the largest gap. 2. Ref-level assertions for rules B, D and E. e8d4694 adds a `refs_in` reader to that file; these are three asserts on top of machinery that now exists. 3. Derive `GRAMMAR_BUILDS`' population, or give it a floor, so a deleted row is red. 4. Replace "RESULT: recorded at the run" with the observed verdict in the ten sites above — the values are in the audit and every one reproduced. Items 1 and 2 are the ones that would have caught the vacuous boundary test at `cargo test` time rather than at release time.
Author
Member

Partial coverage improvement shipped in PR #432 / 0.41.15, with Svelte 0.2.1. Ordinary package tests now grade fixture expectations, including explicit absent decoy symbols/references and the negative Broken fixture; new cases cover legal text and attachments. The extractor is rebuilt from the package's own pinned sources, and the grammar rebuild applies a checksum-held source patch reproducibly. Actual published runtime acceptance also covers embedded JS/TS, attachment bindings, edits and restart. This does not complete the historical rule-by-rule mutation audit, so this issue remains open. Related remaining semantic gaps: #433 and #434.

Partial coverage improvement shipped in PR #432 / [0.41.15](https://git.h-dv.de/h-dv/code-index/releases/tag/v0.41.15), with [Svelte 0.2.1](https://git.h-dv.de/h-dv/code-index/releases/tag/pkg/de.h-dv.svelte/v0.2.1). Ordinary package tests now grade fixture expectations, including explicit absent decoy symbols/references and the negative Broken fixture; new cases cover legal text and attachments. The extractor is rebuilt from the package's own pinned sources, and the grammar rebuild applies a checksum-held source patch reproducibly. Actual published runtime acceptance also covers embedded JS/TS, attachment bindings, edits and restart. This does not complete the historical rule-by-rule mutation audit, so this issue remains open. Related remaining semantic gaps: #433 and #434.
Author
Member

Current-source audit on published0.41.21 narrows this old report. The largest pre-merge conformance gap is repaired: crates/first-party-tests/tests/svelte_package.rs::every_shipped_fixture_passes_runtime_conformance calls the shared production conform::check path and requires at least7 shipped fixtures. package-ci.yml runs package_leg.sh svelte release on scoped PRs/pushes, the same tier as package-release.yml; source qualification run6706 passed the Svelte package leg before this release. Grammar-population coverage is also repaired: grammar_provenance.rs::every_grammar_build_script_is_a_grammar_builds_row discovers the scripts and explicitly records the deleted-Svelte-row RED mutation. Remaining audit details are still real: the daemon's a_svelte_file_carries_symbols_through_the_real_daemon test directly asserts only ruleA, and several ABI/package mutation comments still say RESULT: recorded at the run. Keeping this issue open for those residual assertions/records; this is not evidence that #433/#434 are fixed.

Current-source audit on published0.41.21 narrows this old report. The largest pre-merge conformance gap is repaired: crates/first-party-tests/tests/svelte_package.rs::every_shipped_fixture_passes_runtime_conformance calls the shared production conform::check path and requires at least7 shipped fixtures. package-ci.yml runs package_leg.sh svelte release on scoped PRs/pushes, the same tier as package-release.yml; source qualification run6706 passed the Svelte package leg before this release. Grammar-population coverage is also repaired: grammar_provenance.rs::every_grammar_build_script_is_a_grammar_builds_row discovers the scripts and explicitly records the deleted-Svelte-row RED mutation. Remaining audit details are still real: the daemon's a_svelte_file_carries_symbols_through_the_real_daemon test directly asserts only ruleA, and several ABI/package mutation comments still say RESULT: recorded at the run. Keeping this issue open for those residual assertions/records; this is not evidence that #433/#434 are fixed.
Author
Member

Completed in PR #491 and released in v0.41.36.

Added a real-daemon Svelte template test for render calls, event-handler calls and member-head reads, including a negative assertion against the member tail. It has no script block, so another producer cannot rescue these assertions. Removing each of the three template constructs independently makes its corresponding assertion fail; the restored fixture passes.

Revalidated the existing shipped-fixture runtime conformance gate (all eight Svelte fixtures), derived grammar-build population fence and standalone Rust guest fmt/clippy/rustdoc gates. Removing the Svelte grammar registration fails the population test. The ten old mutation comments now attribute their RED outcomes to the original audit; this is not a claim that all ten mutations were rerun here.

The complete Svelte daemon suite (10 tests), shipped-fixture conformance and grammar provenance suite (7 tests) pass. Final Linux, Windows, package and pilot CI plus all ten release jobs are green. The installed published binaries passed signed-package upgrade tests and self-dogfood with eleven official packages over snapshot and daemon transports.

Completed in PR #491 and released in [v0.41.36](https://git.h-dv.de/h-dv/code-index/releases/tag/v0.41.36). Added a real-daemon Svelte template test for render calls, event-handler calls and member-head reads, including a negative assertion against the member tail. It has no script block, so another producer cannot rescue these assertions. Removing each of the three template constructs independently makes its corresponding assertion fail; the restored fixture passes. Revalidated the existing shipped-fixture runtime conformance gate (all eight Svelte fixtures), derived grammar-build population fence and standalone Rust guest fmt/clippy/rustdoc gates. Removing the Svelte grammar registration fails the population test. The ten old mutation comments now attribute their RED outcomes to the original audit; this is not a claim that all ten mutations were rerun here. The complete Svelte daemon suite (10 tests), shipped-fixture conformance and grammar provenance suite (7 tests) pass. Final Linux, Windows, package and pilot CI plus all ten release jobs are green. The installed published binaries passed signed-package upgrade tests and self-dogfood with eleven official packages over snapshot and daemon transports.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#279
No description provided.