h-dv/code-index

Watch

Fork 0

`COSI_CORPUS_REQUIRE=1` has a floor of one, not of nine — 8 of 9 repos can skip and the run passes green #108

New issue

Closed

opened 2026-09-04 15:18:18 +02:00 by buildagent · 1 comment

buildagent commented

2026-09-04 15:18:18 +02:00

Member

Found by the #80-blocker audit and verified first-hand in source before filing.

The defect

crates/indexer/tests/corpus/mod.rs:911:

let require = std::env::var("COSI_CORPUS_REQUIRE").as_deref() == Ok("1");
eprintln!("corpus[{}]: executed={} skipped={} controls={} (require={require})", ...);
if require && self.executed.is_empty() {
    panic!("corpus[{}]: COSI_CORPUS_REQUIRE=1 but NOTHING executed — {} repo(s) skipped: {:?}. \
            A corpus run that grades nothing must fail, not pass.", ...);
}

tests/corpus/corpus.toml pins 9 repos: rust-ripgrep, python-flask, ts-zod, js-express, php-guzzle, ruby-sinatra, cs-dapper, rust-analyzer, py-django.

The guard fires only when the executed set is completely empty. A run in which 8 of the 9 skip — one repo at the wrong sha, a tracked file an antivirus scanner removed, a partial clone — executes one repo, prints executed=1 skipped=8, and passes. The panic message even says "A corpus run that grades nothing must fail", which is precisely the narrower property it enforces: it defends against nothing, not against less than everything.

corpus/mod.rs:106-140 turns both a sha mismatch and a missing tracked file into a plain SKIP, so the two most likely real-world causes both land in the silent path.

Why this matters more than a missing assert

This is the same class as the I035 finding this guard was written for — "five RPC arms deletable with CI green" — and the same class as the anti-vacuity guards elsewhere in this suite. The ctl.len() > 0 positive-control check immediately above it is the pattern done right: it proves the setup took effect. The REQUIRE check below it does not prove the universe was intact.

The consequence is that the corpus suites — which are the only gates that see real-world scale, the sticky-resolution class of bug, and cross-language recall — can silently shrink to a single repo while reporting green. A ratchet over a shrunken universe is not a ratchet.

It is also the shape this project has already paid for repeatedly: a check that was green while the thing it checked was false, and a one-sided bound blind to the failure it is most likely to have. Every way this harness goes wrong in practice reduces the executed set; the guard only triggers at zero.

The fix

skipped is already tracked and already printed, so the data is in hand. But the naive form — require && !self.skipped.is_empty() — is wrong, because not every suite grades every repo (tier-1 vs tier-3, language-specific suites). The honest form is an expected set per suite:

each suite declares which repos it intends to grade;
under COSI_CORPUS_REQUIRE=1, every repo in that set must appear in executed;
the panic names the missing ones and the reason each was skipped, so a wrong sha and a deleted file are distinguishable without re-running.

Three-state discipline applies to the log line too: executed=1 skipped=8 should not read as a healthy run. Under require, "skipped" is a failure, not an observation.

The mutation that must go red

Point one pinned repo at a wrong sha (or remove one tracked file), run with COSI_CORPUS_REQUIRE=1, and the suite must fail naming that repo. Today it passes. That mutation is also the anti-vacuity check for the fix itself: a test that only ever runs with all 9 present cannot tell the new guard from the old one.

Found while auditing #45. Adjacent to #44 (positive controls), and to the corpus suites tracked in #42/#41/#45. The nightly-cron finding is separate and worse in its own way: event=schedule returns 0 runs of 574, so the nightly leg that carries these suites has never fired — a guard with a floor of one is moot if the job never runs at all.

Found by the #80-blocker audit and **verified first-hand in source** before filing. ## The defect `crates/indexer/tests/corpus/mod.rs:911`: ```rust let require = std::env::var("COSI_CORPUS_REQUIRE").as_deref() == Ok("1"); eprintln!("corpus[{}]: executed={} skipped={} controls={} (require={require})", ...); if require && self.executed.is_empty() { panic!("corpus[{}]: COSI_CORPUS_REQUIRE=1 but NOTHING executed — {} repo(s) skipped: {:?}. \ A corpus run that grades nothing must fail, not pass.", ...); } ``` `tests/corpus/corpus.toml` pins **9** repos: `rust-ripgrep`, `python-flask`, `ts-zod`, `js-express`, `php-guzzle`, `ruby-sinatra`, `cs-dapper`, `rust-analyzer`, `py-django`. **The guard fires only when the executed set is completely empty.** A run in which 8 of the 9 skip — one repo at the wrong sha, a tracked file an antivirus scanner removed, a partial clone — executes one repo, prints `executed=1 skipped=8`, and **passes**. The panic message even says "A corpus run that grades nothing must fail", which is precisely the narrower property it enforces: it defends against *nothing*, not against *less than everything*. `corpus/mod.rs:106-140` turns both a sha mismatch and a missing tracked file into a plain SKIP, so the two most likely real-world causes both land in the silent path. ## Why this matters more than a missing assert This is the same class as the I035 finding this guard was written for — *"five RPC arms deletable with CI green"* — and the same class as the anti-vacuity guards elsewhere in this suite. The `ctl.len() > 0` positive-control check immediately above it is the pattern done right: it proves the setup took effect. The `REQUIRE` check below it does not prove the *universe* was intact. The consequence is that the corpus suites — which are the only gates that see real-world scale, the sticky-resolution class of bug, and cross-language recall — can silently shrink to a single repo while reporting green. A ratchet over a shrunken universe is not a ratchet. It is also the shape this project has already paid for repeatedly: **a check that was green while the thing it checked was false**, and **a one-sided bound blind to the failure it is most likely to have**. Every way this harness goes wrong in practice reduces the executed set; the guard only triggers at zero. ## The fix `skipped` is already tracked and already printed, so the data is in hand. But the naive form — `require && !self.skipped.is_empty()` — is wrong, because not every suite grades every repo (tier-1 vs tier-3, language-specific suites). The honest form is an **expected set per suite**: - each suite declares which repos it intends to grade; - under `COSI_CORPUS_REQUIRE=1`, every repo in that set must appear in `executed`; - the panic names the missing ones **and the reason each was skipped**, so a wrong sha and a deleted file are distinguishable without re-running. Three-state discipline applies to the log line too: `executed=1 skipped=8` should not read as a healthy run. Under `require`, "skipped" is a failure, not an observation. ## The mutation that must go red Point one pinned repo at a wrong sha (or remove one tracked file), run with `COSI_CORPUS_REQUIRE=1`, and the suite must fail **naming that repo**. Today it passes. That mutation is also the anti-vacuity check for the fix itself: a test that only ever runs with all 9 present cannot tell the new guard from the old one. ## Related Found while auditing #45. Adjacent to #44 (positive controls), and to the corpus suites tracked in #42/#41/#45. The nightly-cron finding is separate and worse in its own way: `event=schedule` returns **0 runs of 574**, so the nightly leg that carries these suites has never fired — a guard with a floor of one is moot if the job never runs at all.

buildagent referenced this issue

2026-09-04 15:19:33 +02:00

CI: github.event.schedule is empty on this Forgejo, so every cron-gated job — including tier-1 corpus — skipped on all 36 scheduled runs #109

buildagent referenced this issue

2026-09-04 15:21:20 +02:00

gate: plugin conformance, hostile-runtime containment and production disclosure #80

buildagent referenced this issue

2026-09-04 15:59:31 +02:00

SECURITY DOC: the shipped README claims the plugin worker has "no filesystem and no network"; both are false, and our own threat model says so #114

buildagent referenced this issue

2026-09-04 16:15:05 +02:00

gate: full-language migration proof — migrate the Ruby plugin to a package (#80 step 13) #84

buildagent commented

2026-09-04 16:53:07 +02:00

Author

Member

Fixed — and the generic form found a third suite nobody had mentioned.

The fix

Coverage::finish now fails under require on any SkipClass::Unavailable repo. The discriminator is structural — repo_path(spec) returning Err — so all 12 call sites across 6 suites are unedited, and content-determined skips ("no cross-file refs to mutate") stay legal. That distinction is the whole design: a repo that could not be fetched is a broken universe; a repo with nothing to grade is a measurement.

Log line is three-state: executed=N unavailable=U not_applicable=A.

The mutation, verbatim

corpus[ratchet]: SKIP rust-ripgrep [unavailable] — sha mismatch: want 000000000000 got f9c05a949d1a
corpus[ratchet]: executed=6 unavailable=1 not_applicable=0 controls=6 (require=true)
COSI_CORPUS_REQUIRE=1 but 1 pinned repo(s) were NOT AVAILABLE, so this run graded 6 of 7 …
    rust-ripgrep: sha mismatch: want 000000000000 got f9c05a949d1a
test result: FAILED. 1 passed; 1 failed

And its control — same wrong sha, old floor restored: test result: ok. 2 passed. So the old guard demonstrably passed on a shrunken corpus, on real data, today. Plus 3 unit mutations, all RED.

The third instance, and the answer to a question I asked

I had asked whether this fix would automatically cover the #84 axis-C case (ruby_package_parity in no CI job, grading zero, passing). The answer is no, and the reason matters: the floor is opt-in by env var — deliberately, since cargo test --workspace must not demand a 9-repo cache — so a suite that no COSI_CORPUS_REQUIRE=1 job names has a floor of zero. The two changes compose; neither alone suffices.

So the generic form was built: every_corpus_suite_runs_where_the_require_floor_applies — any crates/indexer/tests/*.rs constructing a Coverage must be named by a job that sets the floor.

It found a third suite nobody had raised: upgrade_equivalence's corpus half, not #[ignore]d, which had never graded a repo. All three are now wired; measured --release under contention: ruby_package_parity 8.4 s (executed=1), ruby_package_cost 27.5 s (executed=1 — 1 is their recorded universe, so the floor is satisfied), upgrade_equivalence 47.4 s (executed=7). 4 mutations RED.

The gate caught itself twice before it caught anything else — first matching its own prose, then its own contains argument. Fixed by stripping comments and assembling the needle from parts. That is the third time today a guard's first version matched its own source.

Verification note

corpus_ratchet with REQUIRE=1: executed=7 unavailable=0, baseline.json md5 unmoved at 534084b856c22566c48e386bc41ed67e, never blessed.

One process finding recorded against the lane's own work: its first workspace-test summary was vacuous — it read the wrapper's echo exit code rather than cargo's, and cargo had exited 101. Caught by reading the output file verbatim, which is why the final report quotes exit codes instead of summaries. That is the same failure I made on this issue's sibling (#109) with a vacuous API filter, twice.

## Fixed — and the generic form found a **third** suite nobody had mentioned. ### The fix `Coverage::finish` now fails under `require` on any **`SkipClass::Unavailable`** repo. The discriminator is **structural** — `repo_path(spec)` returning `Err` — so all 12 call sites across 6 suites are unedited, and content-determined skips ("no cross-file refs to mutate") stay legal. That distinction is the whole design: a repo that could not be *fetched* is a broken universe; a repo with nothing to grade is a measurement. Log line is three-state: `executed=N unavailable=U not_applicable=A`. ### The mutation, verbatim ``` corpus[ratchet]: SKIP rust-ripgrep [unavailable] — sha mismatch: want 000000000000 got f9c05a949d1a corpus[ratchet]: executed=6 unavailable=1 not_applicable=0 controls=6 (require=true) COSI_CORPUS_REQUIRE=1 but 1 pinned repo(s) were NOT AVAILABLE, so this run graded 6 of 7 … rust-ripgrep: sha mismatch: want 000000000000 got f9c05a949d1a test result: FAILED. 1 passed; 1 failed ``` **And its control** — same wrong sha, old floor restored: `test result: ok. 2 passed`. So the old guard demonstrably passed on a shrunken corpus, on real data, today. Plus 3 unit mutations, all RED. ### The third instance, and the answer to a question I asked I had asked whether this fix would automatically cover the #84 axis-C case (`ruby_package_parity` in no CI job, grading zero, passing). **The answer is no, and the reason matters:** the floor is **opt-in by env var** — deliberately, since `cargo test --workspace` must not demand a 9-repo cache — so a suite that no `COSI_CORPUS_REQUIRE=1` job *names* has a floor of **zero**. The two changes compose; neither alone suffices. So the generic form was built: **`every_corpus_suite_runs_where_the_require_floor_applies`** — any `crates/indexer/tests/*.rs` constructing a `Coverage` must be named by a job that sets the floor. It found a third suite nobody had raised: **`upgrade_equivalence`'s corpus half**, not `#[ignore]`d, which had **never graded a repo**. All three are now wired; measured `--release` under contention: `ruby_package_parity` 8.4 s (`executed=1`), `ruby_package_cost` 27.5 s (`executed=1` — 1 *is* their recorded universe, so the floor is satisfied), **`upgrade_equivalence` 47.4 s (`executed=7`)**. 4 mutations RED. **The gate caught itself twice before it caught anything else** — first matching its own prose, then its own `contains` argument. Fixed by stripping comments and assembling the needle from parts. That is the third time today a guard's first version matched its own source. ### Verification note `corpus_ratchet` with `REQUIRE=1`: `executed=7 unavailable=0`, `baseline.json` md5 unmoved at `534084b856c22566c48e386bc41ed67e`, never blessed. One process finding recorded against the lane's own work: its first workspace-test summary was **vacuous** — it read the wrapper's `echo` exit code rather than cargo's, and cargo had exited **101**. Caught by reading the output file verbatim, which is why the final report quotes exit codes instead of summaries. That is the same failure I made on this issue's sibling (#109) with a vacuous API filter, twice.

buildagent closed this issue

2026-09-04 16:53:09 +02:00

buildagent referenced this issue

2026-09-04 17:01:03 +02:00

test: agent-task benchmark across builtin and runtime-plugin workflows #51

buildagent referenced this issue

2026-09-04 17:44:57 +02:00

test: agent-task benchmark across builtin and runtime-plugin workflows #51