test: OSS corpus infrastructure — corpus.toml manifest, pinned fetch/cache, and a non-vacuous coverage manifest #40
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
h-dv/code-index#40
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
P1 — enabling infrastructure for #39, #41, #42, #43.
From
_prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md§3.What
A manifest + fetcher so corpus tests have real repos to run against, deterministically and offline-after-first-fetch.
tests/corpus/corpus.toml, one entry per repo:Fetcher behaviour:
~/.cache/cosi-corpus/<sha>/(or$COSI_CORPUS_DIR); never re-download a cached shahistory = truerepos getgit clone --depth Ninstead of a tarballAnti-vacuity — the load-bearing requirement
Corpus tests self-skip when the corpus is absent (matching the established
ground_truth::have()/real_world_workspace.rsidiom). A skip must not be able to masquerade as a pass. This project has had four near-misses of that class (I033 gate-vacuity, I029b vacuous e2e mutations, I035 "5 RPC arms deletable green", and one during the 2026-07-28 spike itself).Therefore:
Tiers
Decisions to settle here
cosi-corpusrepo (brainstorm recommends fetch+cache, submodules clone the world on every runner)licenceon fetch)xtaskcrate vs a dev-onlycrates/corpusmember — no task runner exists today, so this is a new pattern either wayAcceptance
corpus.tomlwith ≥1 tier-1 repo per supported language, all pinned by sha, all permissiveLanded
tests/corpus/corpus.toml— 7 tier-1 repos, one per language, all sha-pinned, all permissive (MIT / BSD-3 / Apache-2.0 / Unlicense).rust-ripgrepcarrieshistory = truefor #39.tests/corpus/fetch.sh— pinned fetch into$COSI_CORPUS_DIR, cached (a repo already at the right sha is left alone), verifies the sha after checkout and fails loudly on mismatch. Depth 1 by default, deeper forhistory = true. Verified end-to-end from an empty directory: all 7 fetched at their pinned shas.crates/indexer/tests/corpus/mod.rs— shared harness: manifest parsing, staging, the id-independent projection,first_diff,Controls,Coverage.corpusjob in.forgejo/workflows/ci.yml(schedule +workflow_dispatch, not a PR gate — it needs github egress and the replay spends minutes driving real inotify events).Decisions taken
licencefield is recorded so mirroring stays possible later without re-auditing.xtaskcrate. A shell script plus test-crate helpers; nothing here needs to run before the workspace builds.COSI_CORPUS_DIRis opt-in with NO default path. Originally it fell back to~/.cache/cosi-corpus, which would silently bolt minutes onto every developer'scargo testthe moment they populated the cache. Now an unset variable means every repo skips.Anti-vacuity
Coverage::finishwritestarget/corpus/coverage-<suite>.jsonand enforces both halves:COSI_CORPUS_REQUIRE=1, which the CI job sets.Verified in both directions:
cargo test --workspacewith the variable unset skips all 7 cleanly and stays green; with it set,executed=7 skipped=0on every suite.Acceptance boxes 1–5 all met.
Status check — NOT closing. 4 of 5 acceptance items met; one is genuinely missing.
I came here to close this as shipped and the evidence says otherwise, so recording it rather than ticking the box.
Met:
corpus.toml: 7 tier-1 repos, one per supported language, all pinned by sha, all permissive.corpus[ratchet]: executed=N skipped=M controls=K).COSI_CORPUS_REQUIRE=1with no corpus → "COSI_CORPUS_REQUIRE=1 but NOTHING executed — 7 repo(s) skipped. A corpus run that grades nothing must fail, not pass."Not met — acceptance item 2, content-hash verification.
tests/corpus/fetch.shpopulates and caches, but grepping it forsha256|shasum|verify|licence|symlink|submodule|normalisfinds only the cache path and anok <name> (cached at <sha>)line. The pinnedshais the git commit, not a hash of the fetched content, so the issue's own requirement — "verify a content hash after fetch — a swapped tarball must fail loudly, not silently" — has nothing enforcing it. The normalisation report is likewise absent from the fetcher (symlink handling exists at stage time incorpus/mod.rs, which is a different guarantee).Separate finding, worth its own decision. The body's tier table says tier 1 = "every PR", but
.forgejo/workflows/ci.ymlgates the corpus job onworkflow_dispatch || schedule == '0 3 * * *'— nightly, not per-PR. That gap is not theoretical: the cs-dapper ratchet drift from v0.10.1 (symbols +277,resolved +162,edges +127) sat unblessed on master until this session found it. A per-PR tier-1 run would have caught it at the commit that caused it.Suggested resolution: either implement the content-hash check and reconcile the cadence, or scope this issue down to what shipped and re-file the fetcher hardening + the PR-cadence question separately.
Closing — the remaining acceptance item landed in
a6b97c0. All five are now met.I met item 2's INTENT differently from its literal wording, so flagging it explicitly rather than quietly ticking the box.
The item asks for "content-hash verification … a swapped tarball must fail loudly". That was written when the open decision was "fetched tarballs vs git submodules vs a separate cosi-corpus repo". The implementation went with git clone at a pinned sha — and for git, the sha is a content hash: objects are content-addressed, git verifies them on fetch, and
fetch.shalready assertedrev-parse HEAD == sha. Adding a sha256 on top would have duplicated git's own integrity guarantee and left the actual hole untouched.The actual hole was the cache-HIT path, which checked only
HEADand never whether the working tree was clean. A locally-modified cached repo reportedok <name> (cached at <sha>)while every corpus metric was computed from different bytes — and because those metrics feed the ratchet, it would surface as a phantom resolver regression rather than as cache corruption. That is precisely the "swapped content passes silently" failure this item exists to prevent, just one layer in from where the wording pointed.Now: a dirty cache is re-fetched rather than trusted. Verified live —
— and the corpus recovered to 7/7
tree cleanat the pinned shas.Normalisation report — symlinks and submodules are counted and reported, deliberately not stripped: stripping would diverge the working tree from the pinned sha and defeat the dirty-tree check just added. Neither is followed today (the walker refuses symlinks; the fetch does not recurse submodules), so the count exists to make a repo that starts carrying them visible instead of quietly changing what the corpus measures.
The cadence finding from my previous comment is also resolved. Corpus tier-1 is now a PR gate (
if: github.event_name != 'schedule' || …), socorpus.toml's "tier 1 = every PR" is true rather than aspirational, and count drift is attributed to the commit that caused it. Cached oncorpus.toml's hash so the new github-egress dependency is a cold-start cost. Both costs — egress in the PR path, ~5–8 min wall clock — are recorded in the workflow comment along with the ratchet-only fallback, so a future revert can be reasoned about rather than guessed at.If you'd rather have a literal per-repo content hash on top of git's, say so and I'll add it — but I'd be adding ceremony over a guarantee git already provides.