feat: multilingual precision + zero-phantom regression gate (reframed agent benchmark) #38

New issue

Open

opened 2026-07-21 21:19:21 +02:00 by buildagent · 0 comments

buildagent commented

2026-07-21 21:19:21 +02:00

Member

Why (reframed from "multilingual agent benchmark")

Two ultradeep multi-agent studies (2026-07-21, post v0.8.0/I032) reframed this:

Strategy brainstorm (4 lenses + synthesis, unanimous): the moat is honest, zero-phantom, cross-language, agent-shaped evidence — not resolution completeness. The ~20% self-resolution rate is a denominator artifact (~40–47% of refs are std/dep/macro symbols with zero in-repo target — unwrap, assert, String, Vec, Option — which the resolver correctly declines; chasing them mints phantoms). Do not chase it.
Build-vs-buy survey (5 tool families, unanimous): keep the hand-rolled resolver — tree-sitter-stack-graphs was archived 2025-09-09 and never covered Rust/Ruby/PHP/C#; SCIP indexers are all build-required (break zero-build/offline); ctags/ast-grep sit below name resolution and would manufacture phantoms. The resolver is permanent and load-bearing.

Therefore its zero-phantom guarantee is the crown jewel — and today it is defended only by manual per-mission adversarial dogfood (I026 and I030 each shipped fixes for phantoms the fixture tests missed). One resolver refactor can silently reintroduce phantoms with green CI. This issue automates that guard.

The original "efficiency/round-trip benchmark" framing is rejected as vacuous: the current crates/mcp-server/tests/agent_workflow_bench.rs is #[ignore]d, has 1 assert vs 4 println!, runs one Rust project, and measures self-reported bytes — it cannot prove correctness or catch a regression.

What (the deliverable)

An asserting precision + phantom==0 regression gate across all 6 languages, built on machine-readable edge oracles, wired into CI. Extend the existing harness (crates/daemon/tests/correctness.rs, lang_e2e.rs, the ground_truth/ dir, tests/fixtures/{lang}/project/EXPECTATIONS.md) — do not reinvent.

Tier 1 — precision / phantom gate (primary, CI-gating)

Promote each language's EXPECTATIONS.md into a machine-readable edge oracle: positive edges (caller→callee, ref→def) plus weighted DECOY / negative cases:
- stdlib namesakes that must not bind (.get() / .unwrap() / .new() where a same-named project symbol exists),
- visibility twins (e.g. php Util::helper private vs Greeter::helper public — already in fixtures),
- same-name cross-file symbols.
Assert (not print):
- phantom_rate == 0 — a resolution=resolved edge bound to the wrong internal target. Load-bearing, non-gameable: a "resolve-everything" resolver must score worse.
- precision of resolved call/method_call/type/qualified edges (target ~100%).
- internal_recall over internal-eligible edges only (denominator excludes no_candidate externals — so it is meaningful, not the misleading 15–20%).
- name_fallback false-identity rate (currently unmeasured — the real confidently-wrong-answer risk).
- per-(language × ref_kind) parity floors (a plugin regressing to 0 must fail; today's loose >=1 bounds let an empty graph pass).
- determinism (two cold indexes of the same tree produce identical edges).

Tier 2 — efficiency vs baseline (secondary)

Tokens + round-trips to a correct answer via code-index vs a ripgrep+Read baseline on scripted tasks (find-all-callers, safe-rename, impact-of-change), with a task-success oracle (did it find every true caller?) and the baseline's false-positive count → precision-per-token (the moat claim, currently only prose).

CI

Wire Tier 1 into .forgejo/workflows/ci.yml so every resolver change is gated. This replaces the manual dogfood safety net with an automated one.

Non-goals

Not an efficiency-only demo. Not chasing the 20% resolution rate. No new resolution capability (a separate, phantom-safe receiver-binding recall pass is future work, guarded by this gate).

Acceptance

Machine-readable oracle per language with positive + decoy cases.
phantom_rate == 0 asserted across all 6 languages, in CI.
precision / internal-recall / name_fallback-false-identity / per-(lang×kind) parity floors / determinism asserted.
Tier-2 efficiency-vs-grep with a success oracle (bytes+round-trips, precision-per-token).
Full gates green; deterministic; no flakiness.

## Why (reframed from "multilingual agent benchmark") Two ultradeep multi-agent studies (2026-07-21, post v0.8.0/I032) reframed this: - **Strategy brainstorm** (4 lenses + synthesis, unanimous): the moat is *honest, zero-phantom, cross-language, agent-shaped* evidence — **not** resolution completeness. The ~20% self-resolution rate is a **denominator artifact** (~40–47% of refs are std/dep/macro symbols with zero in-repo target — `unwrap`, `assert`, `String`, `Vec`, `Option` — which the resolver *correctly declines*; chasing them mints phantoms). Do not chase it. - **Build-vs-buy survey** (5 tool families, unanimous): keep the hand-rolled resolver — `tree-sitter-stack-graphs` was archived 2025-09-09 and never covered Rust/Ruby/PHP/C#; SCIP indexers are all build-required (break zero-build/offline); ctags/ast-grep sit below name resolution and would manufacture phantoms. **The resolver is permanent and load-bearing.** Therefore its **zero-phantom guarantee is the crown jewel** — and today it is defended *only by manual per-mission adversarial dogfood* (I026 and I030 each shipped fixes for phantoms the fixture tests missed). One resolver refactor can silently reintroduce phantoms with green CI. **This issue automates that guard.** The original "efficiency/round-trip benchmark" framing is rejected as vacuous: the current `crates/mcp-server/tests/agent_workflow_bench.rs` is `#[ignore]`d, has 1 assert vs 4 `println!`, runs one Rust project, and measures self-reported bytes — it cannot prove correctness or catch a regression. ## What (the deliverable) An **asserting** precision + `phantom==0` regression gate across all 6 languages, built on machine-readable edge oracles, wired into CI. Extend the existing harness (`crates/daemon/tests/correctness.rs`, `lang_e2e.rs`, the `ground_truth/` dir, `tests/fixtures/{lang}/project/EXPECTATIONS.md`) — do **not** reinvent. ### Tier 1 — precision / phantom gate (primary, CI-gating) - Promote each language's `EXPECTATIONS.md` into a **machine-readable edge oracle**: positive edges (caller→callee, ref→def) **plus weighted DECOY / negative cases**: - stdlib namesakes that must **not** bind (`.get()` / `.unwrap()` / `.new()` where a same-named *project* symbol exists), - visibility twins (e.g. php `Util::helper` private vs `Greeter::helper` public — already in fixtures), - same-name cross-file symbols. - **Assert (not print):** - `phantom_rate == 0` — a `resolution=resolved` edge bound to the **wrong** internal target. Load-bearing, non-gameable: a "resolve-everything" resolver must score **worse**. - `precision` of resolved call/method_call/type/qualified edges (target ~100%). - `internal_recall` over internal-eligible edges **only** (denominator **excludes** `no_candidate` externals — so it is meaningful, not the misleading 15–20%). - `name_fallback` false-identity rate (currently unmeasured — the real confidently-wrong-answer risk). - per-(language × ref_kind) **parity floors** (a plugin regressing to 0 must fail; today's loose `>=1` bounds let an empty graph pass). - determinism (two cold indexes of the same tree produce identical edges). ### Tier 2 — efficiency vs baseline (secondary) - Tokens + round-trips to a **correct** answer via code-index vs a ripgrep+Read baseline on scripted tasks (find-all-callers, safe-rename, impact-of-change), with a **task-success oracle** (did it find every true caller?) and the baseline's false-positive count → **precision-per-token** (the moat claim, currently only prose). ### CI - Wire Tier 1 into `.forgejo/workflows/ci.yml` so every resolver change is gated. This replaces the manual dogfood safety net with an automated one. ## Non-goals - Not an efficiency-only demo. Not chasing the 20% resolution rate. No new resolution capability (a separate, phantom-safe receiver-binding recall pass is future work, guarded by *this* gate). ## Acceptance - [ ] Machine-readable oracle per language with positive + decoy cases. - [ ] `phantom_rate == 0` asserted across all 6 languages, in CI. - [ ] precision / internal-recall / name_fallback-false-identity / per-(lang×kind) parity floors / determinism asserted. - [ ] Tier-2 efficiency-vs-grep with a success oracle (bytes+round-trips, precision-per-token). - [ ] Full gates green; deterministic; no flakiness.

buildagent referenced this issue

2026-07-21 21:19:36 +02:00

feat: quick wins umbrella — 7 single-query capabilities on existing data #30

buildagent referenced this issue from a commit

2026-07-21 22:30:03 +02:00

feat: I033 — multilingual precision/zero-phantom gate (#38) + per-file index_health (#30.6) (v0.8.1)

buildagent referenced this issue

2026-07-28 20:40:20 +02:00

test: agent-task benchmark on the corpus — grade the product, with a token ratchet #51