test: agent-task benchmark on the corpus — grade the product, with a token ratchet #51

New issue

Open

opened 2026-07-28 20:40:20 +02:00 by buildagent · 0 comments

buildagent commented

2026-07-28 20:40:20 +02:00

Member

P3 — the only metric here that measures the actual value proposition. Depends on #40, #45.

From _prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md §5.3.

Why

Every other corpus metric measures internals (refs resolved, spans stable, phantoms absent). None would notice a change that improves resolution while making the tools worse to use — wrong shape, missing disclosure, more round-trips, bloated payloads.

The product claim is "the right answer at roughly 10× less context than Read/grep." That claim is currently only prose.

What

~20 realistic questions per corpus repo, each with a hand-verified answer:

who calls X?
what breaks if I change Y?
is Z safe to delete?
where is the definition behind this call site?
which tests cover W?

Scored on correctness (did it find every true answer, no fabrications) and token cost (bytes/tokens to reach it, round-trips needed).

Compare against a ripgrep+Read baseline on the same question, counting the baseline's false positives — yielding precision-per-token, which is the moat claim expressed as a number.

Replaces a known-vacuous test

crates/mcp-server/tests/agent_workflow_bench.rs is #[ignore]d, has 1 assert against 4 println!, runs one Rust project, and measures self-reported bytes. It cannot catch a regression. This supersedes it — and per #38's framing, that vacuity is precisely why it must be rebuilt with a success oracle rather than extended.

Ratchet

Tokens-per-correct-answer lands in baseline.json (#45). A change that answers the same questions using materially more context fails, even if every internal metric improved.

Acceptance

≥20 questions × ≥3 repos with hand-verified answers
correctness oracle: all true answers found, zero fabricated
token + round-trip cost measured against a ripgrep+Read baseline incl. its false-positive count
tokens-per-correct-answer ratcheted
the old #[ignore]d bench deleted, not left alongside

**P3 — the only metric here that measures the actual value proposition. Depends on #40, #45.** From `_prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md` §5.3. ## Why Every other corpus metric measures internals (refs resolved, spans stable, phantoms absent). None would notice a change that improves resolution while making the **tools** worse to use — wrong shape, missing disclosure, more round-trips, bloated payloads. The product claim is *"the right answer at roughly 10× less context than Read/grep."* That claim is currently only prose. ## What ~20 realistic questions per corpus repo, each with a hand-verified answer: - who calls `X`? - what breaks if I change `Y`? - is `Z` safe to delete? - where is the definition behind this call site? - which tests cover `W`? Scored on **correctness** (did it find every true answer, no fabrications) **and token cost** (bytes/tokens to reach it, round-trips needed). Compare against a ripgrep+Read baseline on the same question, counting the baseline's false positives — yielding **precision-per-token**, which is the moat claim expressed as a number. ## Replaces a known-vacuous test `crates/mcp-server/tests/agent_workflow_bench.rs` is `#[ignore]`d, has 1 assert against 4 `println!`, runs one Rust project, and measures self-reported bytes. It cannot catch a regression. This supersedes it — and per #38's framing, that vacuity is precisely why it must be rebuilt with a success oracle rather than extended. ## Ratchet Tokens-per-correct-answer lands in `baseline.json` (#45). A change that answers the same questions using materially more context fails, even if every internal metric improved. ## Acceptance - [ ] ≥20 questions × ≥3 repos with hand-verified answers - [ ] correctness oracle: all true answers found, zero fabricated - [ ] token + round-trip cost measured against a ripgrep+Read baseline incl. its false-positive count - [ ] tokens-per-correct-answer ratcheted - [ ] the old `#[ignore]`d bench deleted, not left alongside

buildagent added the

labels

2026-07-28 20:40:57 +02:00

buildagent referenced this issue

2026-07-29 11:53:35 +02:00

feat: corpus baselines + ratchets — metrics JSON per repo, regression fails, --bless carries a reason #45