test: mandate positive controls on every corpus assertion (fifth near-miss of the vacuity class)

buildagent commented

2026-07-28 20:38:45 +02:00

Member

P2 — process rule with a concrete enforcement mechanism.

The pattern

This project has now had five incidents where a test passed without testing anything:

I033 — precision_gate decoys all sat in multi-candidate pools, so the entire tier-1b disjunction could be disabled with every language gate green. Fixed by a structural anti-vacuity guard.
I029b — vacuous e2e mutations; wire-skew tests passed with empty payloads.
I035 — e2e ran --no-daemon, so 5 RPC arms were deletable with CI green.
I034 — the freshness barrier bricked changed_symbols permanently and CI stayed green.
2026-07-28 corpus spike — the content-edit third of a three-part mutation silently no-oped (path relative to the wrong cwd, stderr swallowed by 2>/dev/null). Reported 4/4 green. Caught only by an ad-hoc edited_syms_before == edited_syms_after check. Re-run correctly (syms 30→31, 51→52, 242→243, 151→152) it was still green — but it could have hidden a real defect and nothing in the harness would have said so.

Five times is not bad luck; it is a missing invariant in how tests are written here.

The rule

Every corpus assertion must ship with a positive control that fails if the setup did not take effect.

Not "did the assertion pass" but "did the thing I am asserting about actually happen":

Assertion	Required positive control
mutation → no rebind	symbol/ref count in the mutated file actually changed
watcher replay → cold==incremental	the tree actually differs between commits
corpus ran	coverage manifest lists ≥1 executed repo per language (#40)
decoy not resolved	the decoy's candidate pool is provably size-1 (existing I033 pattern)
skip path	a run where everything skipped FAILS

Enforcement, not just documentation

A convention nobody can check will decay. Options to evaluate:

a shared positive_control! helper that records into the run manifest, plus a harness-level assertion that every corpus test registered at least one control
CI check that fails if a corpus test file contains an assertion without a registered control
never 2>/dev/null on a mutation step in a test; capture and assert the exit status

Acceptance

mechanism chosen and implemented (not merely a CONTRIBUTING.md paragraph)
every test added by #39/#41/#42/#43 registers a control
harness fails a corpus test that registers none
the rule documented in CONTRIBUTING.md with the five incidents as rationale

**P2 — process rule with a concrete enforcement mechanism.** ## The pattern This project has now had **five** incidents where a test passed without testing anything: 1. **I033** — `precision_gate` decoys all sat in multi-candidate pools, so the entire tier-1b disjunction could be disabled with every language gate green. Fixed by a structural anti-vacuity guard. 2. **I029b** — vacuous e2e mutations; wire-skew tests passed with empty payloads. 3. **I035** — e2e ran `--no-daemon`, so **5 RPC arms were deletable with CI green**. 4. **I034** — the freshness barrier bricked `changed_symbols` permanently and CI stayed green. 5. **2026-07-28 corpus spike** — the content-edit third of a three-part mutation silently no-oped (path relative to the wrong cwd, stderr swallowed by `2>/dev/null`). Reported 4/4 green. Caught only by an ad-hoc `edited_syms_before == edited_syms_after` check. Re-run correctly (`syms 30→31, 51→52, 242→243, 151→152`) it was still green — but it *could* have hidden a real defect and nothing in the harness would have said so. Five times is not bad luck; it is a missing invariant in how tests are written here. ## The rule **Every corpus assertion must ship with a positive control that fails if the setup did not take effect.** Not "did the assertion pass" but "did the thing I am asserting about actually happen": | Assertion | Required positive control | |---|---| | mutation → no rebind | symbol/ref count in the mutated file actually changed | | watcher replay → cold==incremental | the tree actually differs between commits | | corpus ran | coverage manifest lists ≥1 executed repo per language (#40) | | decoy not resolved | the decoy's candidate pool is provably size-1 (existing I033 pattern) | | skip path | a run where everything skipped FAILS | ## Enforcement, not just documentation A convention nobody can check will decay. Options to evaluate: - a shared `positive_control!` helper that records into the run manifest, plus a harness-level assertion that every corpus test registered at least one control - CI check that fails if a corpus test file contains an assertion without a registered control - never `2>/dev/null` on a mutation step in a test; capture and assert the exit status ## Acceptance - [ ] mechanism chosen and implemented (not merely a CONTRIBUTING.md paragraph) - [ ] every test added by #39/#41/#42/#43 registers a control - [ ] harness fails a corpus test that registers none - [ ] the rule documented in CONTRIBUTING.md with the five incidents as rationale

buildagent referenced this issue

2026-07-28 20:40:08 +02:00

test: git history as a free oracle for changed_symbols / review_diff / handle resolution #50

buildagent added the

Kind/Testing

Priority

Medium

labels

2026-07-28 20:40:46 +02:00

buildagent referenced this issue

2026-07-28 21:36:11 +02:00

test: OSS corpus infrastructure — corpus.toml manifest, pinned fetch/cache, and a non-vacuous coverage manifest #40

buildagent referenced this issue

2026-07-28 21:36:19 +02:00

test: corpus mutation guard — deletion/rename/injection phantom tripwires at real-repo ambiguity #43

buildagent commented

2026-07-28 21:36:28 +02:00

Author

Member

Landed — enforced in the harness, not just documented

corpus::Controls::require(label, held, detail) records a control and fails immediately if it does not hold. Coverage::finish takes the Controls and refuses a run that graded repos while registering none.

The two rules had to be coupled rather than checked separately: a standalone Controls::finish panicked on a legitimate skip, because with no corpus present every repo skips and no control can be registered. So:

graded ≥1 repo → must have ≥1 positive control;
graded nothing → tolerated locally, fails under COSI_CORPUS_REQUIRE=1.

Documented in CONTRIBUTING.md under "Positive controls are mandatory", with all five incidents as rationale.

It earned its keep immediately — three catches during implementation

Rename that wasn't a rename. The #42 suite renamed victims to *.renamed; the walker selects a language by extension, so the file left the index entirely instead of moving. The assertion under test would have passed. The control (file_present(new) && !file_present(old)) failed on the first run.
Unrepresentative replay window. #39's commit selection initially took the oldest end of history, which contains no deletions and no new directories. Both replay tests refused to run: git name-status shapes seen: {'A', 'M'} and 0 new-directory commit(s) in the replayed range. Without the controls both would have passed while testing far less than they claimed.
Empty-index comparisons. Determinism and path-invariance compare two projections; an empty index makes that trivially true. Controls assert non-empty symbol/ref counts.

Registered per suite as landed: determinism 7, path-invariance 14, cold-vs-incremental 7 each, mutation 14 each, watcher replay 3 each.

Acceptance boxes 1–4 met.

## Landed — enforced in the harness, not just documented `corpus::Controls::require(label, held, detail)` records a control and fails immediately if it does not hold. `Coverage::finish` **takes the `Controls`** and refuses a run that graded repos while registering none. The two rules had to be coupled rather than checked separately: a standalone `Controls::finish` panicked on a legitimate skip, because with no corpus present every repo skips and no control *can* be registered. So: - graded ≥1 repo → must have ≥1 positive control; - graded nothing → tolerated locally, fails under `COSI_CORPUS_REQUIRE=1`. Documented in `CONTRIBUTING.md` under "Positive controls are mandatory", with all five incidents as rationale. ## It earned its keep immediately — three catches during implementation 1. **Rename that wasn't a rename.** The #42 suite renamed victims to `*.renamed`; the walker selects a language by extension, so the file left the index entirely instead of moving. The assertion under test would have passed. The control (`file_present(new) && !file_present(old)`) failed on the first run. 2. **Unrepresentative replay window.** #39's commit selection initially took the oldest end of history, which contains no deletions and no new directories. Both replay tests refused to run: `git name-status shapes seen: {'A', 'M'}` and `0 new-directory commit(s) in the replayed range`. Without the controls both would have passed while testing far less than they claimed. 3. **Empty-index comparisons.** Determinism and path-invariance compare two projections; an empty index makes that trivially true. Controls assert non-empty symbol/ref counts. Registered per suite as landed: determinism 7, path-invariance 14, cold-vs-incremental 7 each, mutation 14 each, watcher replay 3 each. Acceptance boxes 1–4 met.

buildagent referenced this issue

2026-07-29 21:26:30 +02:00

I039: OSS corpus test system + the four defects it found (v0.9.0, schema v25) #58

buildagent referenced this issue from a commit

2026-08-03 14:06:57 +02:00

I039: OSS corpus test system + resolver fixes (v0.9.0, schema v25) (#58)

dhoyer referenced this issue from a commit

2026-08-06 14:37:54 +02:00

feat(mcp): disclose total_resolved on find_references; document ref_count semantics (#44)

dhoyer referenced this issue from a commit

2026-08-06 14:37:54 +02:00

release: v0.10.1 — --version hash freshness + find_references total_resolved disclosure

buildagent commented

2026-08-07 21:26:34 +02:00

Author

Member

Closing — verified in-tree.

The rule is enforced, not documented, which was the explicit bar this issue set:

corpus::Controls::require asserts the setup took effect, with the vacuity rationale in its own doc comment ("any assertion built on it would be vacuous").
Coverage::finish fails a corpus test that registers no control: "corpus[{}]: graded {} repo(s) but registered NO positive controls. Every … controls can pass without testing anything." That is acceptance item 3.
The #39/#41/#42/#43 suites route through Coverage/Controls, so item 2 is structural rather than conventional — a new corpus test cannot opt out without failing.
CONTRIBUTING.md carries the rule.

Independent evidence the mechanism actually bites, observed this session: running the ratchet with COSI_CORPUS_REQUIRE=1 and no corpus produced

corpus[ratchet]: executed=0 skipped=7 controls=0 (require=true)
… COSI_CORPUS_REQUIRE=1 but NOTHING executed — 7 repo(s) skipped.
A corpus run that grades nothing must fail, not pass.

That is the sixth incident of this class being caught by the mechanism built for the first five — and it caught a real one: my own baseline run had COSI_CORPUS_DIR unset, so every corpus gate was grading nothing while reporting green. Worth noting for anyone reading this later: the guard only fires when COSI_CORPUS_REQUIRE=1, so a local run without it still silently skips. CI sets it; humans often won't.

Closing — verified in-tree. The rule is **enforced, not documented**, which was the explicit bar this issue set: - `corpus::Controls::require` asserts the setup took effect, with the vacuity rationale in its own doc comment ("any assertion built on it would be vacuous"). - `Coverage::finish` **fails a corpus test that registers no control**: *"corpus[{}]: graded {} repo(s) but registered NO positive controls. Every … controls can pass without testing anything."* That is acceptance item 3. - The `#39`/`#41`/`#42`/`#43` suites route through `Coverage`/`Controls`, so item 2 is structural rather than conventional — a new corpus test cannot opt out without failing. - `CONTRIBUTING.md` carries the rule. Independent evidence the mechanism actually bites, observed this session: running the ratchet with `COSI_CORPUS_REQUIRE=1` and no corpus produced ``` corpus[ratchet]: executed=0 skipped=7 controls=0 (require=true) … COSI_CORPUS_REQUIRE=1 but NOTHING executed — 7 repo(s) skipped. A corpus run that grades nothing must fail, not pass. ``` That is the sixth incident of this class being caught by the mechanism built for the first five — and it caught a real one: my own baseline run had `COSI_CORPUS_DIR` unset, so every corpus gate was grading nothing while reporting green. Worth noting for anyone reading this later: the guard only fires when `COSI_CORPUS_REQUIRE=1`, so a *local* run without it still silently skips. CI sets it; humans often won't.

buildagent closed this issue

2026-08-07 21:26:38 +02:00

buildagent referenced this issue

2026-08-07 21:27:19 +02:00

test: corpus mutation guard — deletion/rename/injection phantom tripwires at real-repo ambiguity #43

dhoyer referenced this issue from a commit

2026-08-07 22:18:26 +02:00

test(corpus): injection mutation family, PR-gated corpus, fetch integrity

buildagent referenced this issue

2026-08-07 22:19:44 +02:00

test: corpus mutation guard — deletion/rename/injection phantom tripwires at real-repo ambiguity #43