h-dv/code-index

Fork 0

feat: corpus baselines + ratchets — metrics JSON per repo, regression fails, --bless carries a reason #45

New issue

Open

opened 2026-07-28 20:39:01 +02:00 by buildagent · 1 comment

buildagent commented

2026-07-28 20:39:01 +02:00

Member

P2 — turns corpus runs into a trend rather than a pass/fail. Depends on #40.

From _prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md §4.

Three enforcement tiers

Tier A — hard gates: phantom_count == 0, no panics/hangs, parse_error_files == 0 (or a per-repo frozen allowlist with reasons), determinism, cold==incremental, wall-clock budget.

Tier B — ratcheted against a committed baseline.json: resolution metrics, symbol-extraction recall, index wall-clock, peak RSS, DB bytes/source KB, query p50/p99, tokens-per-answer.

Tier C — diagnostics, never asserted: the resolution_gaps histogram by reason code × language × repo. This is the worklist (#47), not a score.

Explicitly NOT a gate: raw resolution %

Per the build-vs-buy verdict, the ~20% rate is a denominator artifact — most refs point at stdlib/third-party and the resolver correctly declines them. Gating on it creates direct pressure to mint phantoms, i.e. to destroy the moat to make a dashboard green. It may be recorded; it may never fail a build.

Spike baseline (v0.8.7 @ 855710d, tier-1) to seed the file:

repo	files	symbols	refs	resolved	%	index
rust-ripgrep	231	4127	36607	13759	37.6	1.5s
php-guzzle	165	3059	35791	11035	30.8	1.1s
js-express	206	1917	19748	4154	21.0	0.4s
ruby-sinatra	285	1260	17574	2768	15.8	0.4s
python-flask	226	1620	14708	2075	14.1	0.4s
ts-zod	558	8489	85027	11246	13.2	3.4s
cs-dapper	223	2945	25155	2796	11.1	0.7s

Anti-gaming

Baselines live in the repo, so a change shows up in the PR diff. --bless requires a reason string written into the baseline — blessing must be visible in review, never a silent green-wash. (This project has already seen metrics that "passed" because a derived value was mutated rather than measured — see I030b's impact_batches finding.)

Acceptance

baseline.json per repo, seeded from the table above
Tier A hard-fails; Tier B fails only on regression with a readable diff; Tier C printed
--bless writes a mandatory reason field
raw resolution % recorded but provably not gating

**P2 — turns corpus runs into a trend rather than a pass/fail. Depends on #40.** From `_prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md` §4. ## Three enforcement tiers **Tier A — hard gates:** `phantom_count == 0`, no panics/hangs, `parse_error_files == 0` (or a per-repo frozen allowlist with reasons), determinism, cold==incremental, wall-clock budget. **Tier B — ratcheted against a committed `baseline.json`:** resolution metrics, symbol-extraction recall, index wall-clock, peak RSS, DB bytes/source KB, query p50/p99, tokens-per-answer. **Tier C — diagnostics, never asserted:** the `resolution_gaps` histogram by reason code × language × repo. This is the worklist (#47), not a score. ## Explicitly NOT a gate: raw resolution % Per the build-vs-buy verdict, the ~20% rate is a **denominator artifact** — most refs point at stdlib/third-party and the resolver correctly declines them. Gating on it creates direct pressure to mint phantoms, i.e. to destroy the moat to make a dashboard green. It may be *recorded*; it may never *fail a build*. Spike baseline (v0.8.7 @ 855710d, tier-1) to seed the file: | repo | files | symbols | refs | resolved | % | index | |---|---|---|---|---|---|---| | rust-ripgrep | 231 | 4127 | 36607 | 13759 | 37.6 | 1.5s | | php-guzzle | 165 | 3059 | 35791 | 11035 | 30.8 | 1.1s | | js-express | 206 | 1917 | 19748 | 4154 | 21.0 | 0.4s | | ruby-sinatra | 285 | 1260 | 17574 | 2768 | 15.8 | 0.4s | | python-flask | 226 | 1620 | 14708 | 2075 | 14.1 | 0.4s | | ts-zod | 558 | 8489 | 85027 | 11246 | 13.2 | 3.4s | | cs-dapper | 223 | 2945 | 25155 | 2796 | 11.1 | 0.7s | ## Anti-gaming Baselines live in the repo, so a change shows up in the PR diff. `--bless` **requires a reason string** written into the baseline — blessing must be visible in review, never a silent green-wash. (This project has already seen metrics that "passed" because a derived value was mutated rather than measured — see I030b's `impact_batches` finding.) ## Acceptance - [ ] `baseline.json` per repo, seeded from the table above - [ ] Tier A hard-fails; Tier B fails only on regression with a readable diff; Tier C printed - [ ] `--bless` writes a mandatory reason field - [ ] raw resolution % recorded but provably not gating

buildagent referenced this issue

2026-07-28 20:39:29 +02:00

feat: gap miner — rank unresolved refs into a mission queue clamped by phantom==0 #47

buildagent referenced this issue

2026-07-28 20:40:20 +02:00

test: agent-task benchmark on the corpus — grade the product, with a token ratchet #51

buildagent added the

Kind/Feature

Priority

Medium

labels

2026-07-28 20:40:47 +02:00

buildagent referenced this issue

2026-07-28 22:06:08 +02:00

perf: cold-index throughput degrades ~5-6x from tier-1 to tier-3 repos (time scales ~refs^1.7) #53

buildagent referenced this issue

2026-07-28 22:06:30 +02:00

test: tier-3 scale ceilings — 20k+ file repos, 1M-edge cap, peak RSS, query p99, FTS growth #41

buildagent referenced this issue

2026-07-29 11:51:58 +02:00

test: wall-clock-bounded resolver tests flake under machine load (10s bound, observed failing at load ~45) #55

buildagent commented

2026-07-29 11:53:34 +02:00

Author

Member

Landed (partially) — `tests/corpus/baseline.json` + `crates/indexer/tests/corpus_ratchet.rs`

Exact structural counts pinned per tier-1 repo, gated on any drift, wired into the nightly corpus job.

cs-dapper    files 214 symbols 2945 refs 25155 resolved  2796 edges 1949 imports  608
js-express   files 198 symbols 1917 refs 19748 resolved  4154 edges 1234 imports    0
php-guzzle   files 161 symbols 3059 refs 35791 resolved 11035 edges 7373 imports  796
python-flask files 216 symbols 1620 refs 14708 resolved  2075 edges 1348 imports  650
ruby-sinatra files 205 symbols 1260 refs 17574 resolved  2768 edges  984 imports  231
rust-ripgrep files 214 symbols 4127 refs 36607 resolved 13759 edges 6910 imports  404
ts-zod       files 493 symbols 8489 refs 85027 resolved 11246 edges 5188 imports 1327

Why this suite exists

Every other corpus suite asks "is this index self-consistent?" — deterministic, coherent with a cold index, free of phantom rebinds. None of them notices a change that is self-consistent but unintended. A resolver tweak that quietly moves 4,000 refs passes all of them. #52 is the worked example: it changed resolution on real repos and no test could see it.

resolved is the dimension this exists for.

Counts only — timings deliberately excluded

The issue's Tier B listed wall clock, RSS, DB bytes and tokens-per-answer as ratchet candidates. I implemented only the count dimensions, and I think the rest should stay out:

counts are deterministic (proven by corpus_determinism), so exact equality is a legitimate gate;
wall clock is not — the same binary indexed rust-analyzer in 25s quiet and 84s under load, a 3.4× spread from the machine alone. Any threshold is either too loose to catch a regression or too tight to survive a busy runner.

That is not theoretical: while implementing this, resolver.rs's existing 10s bounded_work assertion failed under load ~45 and then passed 3/3 in isolation on the same binary. Filed as #55. Timings stay recorded-only in target/corpus/scale-*.json.

Anti-gaming

COSI_CORPUS_BLESS=1 requires COSI_CORPUS_BLESS_REASON (rejected below 8 chars) and writes the reason into the committed baseline, so an intentional change shows up in review. This project has already shipped a metric that "passed" because a derived value was mutated rather than measured (I030b's impact_batches), so the escape hatch is deliberately noisy.

Verified non-vacuous, not assumed

perturbing the baseline by one ref → fails with rust-ripgrep: resolved 13760 -> 13759 (-1)
restoring → passes
COSI_CORPUS_BLESS=1 without a reason → refused
baseline md5 unchanged after the probe

A finding the baseline surfaced immediately

js-express has imports: 0. Express is CommonJS and the javascript plugin does not capture require() as an import. Recorded in the baseline as a known oddity rather than hidden — adjacent to #31 item 2. If it ever becomes non-zero that is an improvement, and blessing it should cite the cause.

Still open on this issue

Tier A hard gates are spread across the existing suites (phantom==0 in precision_gate, determinism/coherence in corpus_metamorphic, ceilings in corpus_scale) rather than centralised here — arguably fine, but not what the issue described.
Tier C (resolution_gaps histogram as a diagnostic artifact) is not implemented; it belongs with #47's gap miner.
Tokens-per-answer needs #51.
Tier-3 repos are not ratcheted — they run weekly and their counts are equally deterministic, so adding them is cheap and probably worth it.

## Landed (partially) — `tests/corpus/baseline.json` + `crates/indexer/tests/corpus_ratchet.rs` Exact structural counts pinned per tier-1 repo, gated on any drift, wired into the nightly `corpus` job. ``` cs-dapper files 214 symbols 2945 refs 25155 resolved 2796 edges 1949 imports 608 js-express files 198 symbols 1917 refs 19748 resolved 4154 edges 1234 imports 0 php-guzzle files 161 symbols 3059 refs 35791 resolved 11035 edges 7373 imports 796 python-flask files 216 symbols 1620 refs 14708 resolved 2075 edges 1348 imports 650 ruby-sinatra files 205 symbols 1260 refs 17574 resolved 2768 edges 984 imports 231 rust-ripgrep files 214 symbols 4127 refs 36607 resolved 13759 edges 6910 imports 404 ts-zod files 493 symbols 8489 refs 85027 resolved 11246 edges 5188 imports 1327 ``` ### Why this suite exists Every other corpus suite asks *"is this index self-consistent?"* — deterministic, coherent with a cold index, free of phantom rebinds. **None of them notices a change that is self-consistent but unintended.** A resolver tweak that quietly moves 4,000 refs passes all of them. #52 is the worked example: it changed resolution on real repos and no test could see it. `resolved` is the dimension this exists for. ### Counts only — timings deliberately excluded The issue's Tier B listed wall clock, RSS, DB bytes and tokens-per-answer as ratchet candidates. **I implemented only the count dimensions**, and I think the rest should stay out: - counts are **deterministic** (proven by `corpus_determinism`), so exact equality is a legitimate gate; - wall clock is not — the same binary indexed rust-analyzer in **25s quiet and 84s under load**, a 3.4× spread from the machine alone. Any threshold is either too loose to catch a regression or too tight to survive a busy runner. That is not theoretical: while implementing this, `resolver.rs`'s existing 10s `bounded_work` assertion **failed under load ~45 and then passed 3/3 in isolation** on the same binary. Filed as #55. Timings stay recorded-only in `target/corpus/scale-*.json`. ### Anti-gaming `COSI_CORPUS_BLESS=1` requires `COSI_CORPUS_BLESS_REASON` (rejected below 8 chars) and writes the reason into the committed baseline, so an intentional change shows up in review. This project has already shipped a metric that "passed" because a derived value was mutated rather than measured (I030b's `impact_batches`), so the escape hatch is deliberately noisy. ### Verified non-vacuous, not assumed - perturbing the baseline by **one ref** → fails with `rust-ripgrep: resolved 13760 -> 13759 (-1)` - restoring → passes - `COSI_CORPUS_BLESS=1` without a reason → refused - baseline md5 unchanged after the probe ### A finding the baseline surfaced immediately **`js-express` has `imports: 0`.** Express is CommonJS and the javascript plugin does not capture `require()` as an import. Recorded in the baseline as a known oddity rather than hidden — adjacent to #31 item 2. If it ever becomes non-zero that is an improvement, and blessing it should cite the cause. ### Still open on this issue - Tier A hard gates are spread across the existing suites (phantom==0 in `precision_gate`, determinism/coherence in `corpus_metamorphic`, ceilings in `corpus_scale`) rather than centralised here — arguably fine, but not what the issue described. - Tier C (`resolution_gaps` histogram as a diagnostic artifact) is not implemented; it belongs with #47's gap miner. - Tokens-per-answer needs #51. - Tier-3 repos are not ratcheted — they run weekly and their counts are equally deterministic, so adding them is cheap and probably worth it.

buildagent referenced this issue

2026-07-29 18:18:34 +02:00

bug: the JS/TS plugin never captures CommonJS require() — express's entire module graph is invisible (410 calls, 0 imports) #56