feat: frozen SCIP oracle — true precision/recall and the HONEST DENOMINATOR (build-required grader, zero-build product) #49

New issue

Open

opened 2026-07-28 20:39:57 +02:00 by buildagent · 0 comments

buildagent commented

2026-07-28 20:39:57 +02:00

Member

P3 — biggest lift, biggest measurement unlock. Justification narrowed by the 2026-07-28 spike.

From _prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md §2.3.

The reframe

The build-vs-buy verdict archived SCIP because it is build-required and the moat is zero-build. That verdict was about ingest. As a test oracle the objection evaporates:

The product must be zero-build. The grader may be arbitrarily expensive, slow and build-required — it runs offline, once, and its output is frozen and committed.

Pipeline: build the repo once → run the authoritative indexer → convert to a compact frozen oracle ({file,line,col} → symbol identity) → commit the artifact → grade forever at zero runtime cost. Refresh quarterly; review the oracle diff as a first-class change, version-stamped with the grader version.

Candidate graders (all need verification before commitment)

Lang	Grader	Confidence
Rust	`rust-analyzer scip`	high — first-party
TS / JS	`scip-typescript` (tsc-backed)	high
Python	`scip-python` (pyright-backed)	high
C#	`scip-dotnet` (Roslyn)	medium
Ruby	`scip-ruby` (Sorbet)	low — may not run on untyped repos
PHP	community `scip-php`	low — verify it exists and works

Start with Rust + TypeScript + Python only. Seven toolchains in CI is not worth it; three prove the concept and cover the two lowest-scoring languages in #46.

The two payoffs

Proven phantoms on real code. We claim identity only when resolution == "resolved", so the oracle is needed only over our resolved set — a far smaller problem than grading all refs. Any confident disagreement is a proven defect.
The honest denominator. The oracle knows which refs are external. Subtract them and we can report, for the first time, "of the refs that were resolvable in-project, we got N%" — replacing the misleading ~11–38% spread (#46) with a number that is honest, comparable, and safe to optimise.

Narrowed justification (post-spike)

The 2026-07-28 mutation testing found 0 phantoms / 1413 resolved cross-file refs. So the oracle's role is measurement, not discovery — whatever precision defects exist did not surface under mutation. Budget it accordingly; it is not the fastest route to a bug.

Risk: the oracle is wrong too

rust-analyzer and pyright have bugs. Count a disagreement as a phantom only where the oracle is confident (a real definition symbol, not a local/unknown). Keep known_oracle_disagreements.toml with a written reason per entry — reusing the name_fallback_ceiling device.

Acceptance

frozen oracle artifacts for rust/ts/python tier-1 repos, version-stamped
phantom detection against the oracle, confidence-gated
honest-denominator internal-resolution metric per language
known_oracle_disagreements.toml with mandatory reasons
refresh procedure documented; oracle diffs reviewed, never auto-accepted

**P3 — biggest lift, biggest measurement unlock. Justification narrowed by the 2026-07-28 spike.** From `_prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md` §2.3. ## The reframe The build-vs-buy verdict archived SCIP because it is **build-required** and the moat is **zero-build**. That verdict was about **ingest**. As a **test oracle** the objection evaporates: > The product must be zero-build. The **grader** may be arbitrarily expensive, slow and build-required — it runs offline, once, and its output is **frozen and committed**. Pipeline: build the repo once → run the authoritative indexer → convert to a compact frozen oracle (`{file,line,col} → symbol identity`) → commit the artifact → grade forever at zero runtime cost. Refresh quarterly; review the **oracle diff** as a first-class change, version-stamped with the grader version. ## Candidate graders (all need verification before commitment) | Lang | Grader | Confidence | |---|---|---| | Rust | `rust-analyzer scip` | high — first-party | | TS / JS | `scip-typescript` (tsc-backed) | high | | Python | `scip-python` (pyright-backed) | high | | C# | `scip-dotnet` (Roslyn) | medium | | Ruby | `scip-ruby` (Sorbet) | low — may not run on untyped repos | | PHP | community `scip-php` | low — verify it exists and works | **Start with Rust + TypeScript + Python only.** Seven toolchains in CI is not worth it; three prove the concept and cover the two lowest-scoring languages in #46. ## The two payoffs 1. **Proven phantoms on real code.** We claim identity only when `resolution == "resolved"`, so the oracle is needed only over *our resolved set* — a far smaller problem than grading all refs. Any confident disagreement is a proven defect. 2. **The honest denominator.** The oracle knows which refs are external. Subtract them and we can report, for the first time, *"of the refs that were resolvable in-project, we got N%"* — replacing the misleading ~11–38% spread (#46) with a number that is honest, comparable, and safe to optimise. ## Narrowed justification (post-spike) The 2026-07-28 mutation testing found **0 phantoms / 1413 resolved cross-file refs**. So the oracle's role is **measurement, not discovery** — whatever precision defects exist did not surface under mutation. Budget it accordingly; it is not the fastest route to a bug. ## Risk: the oracle is wrong too rust-analyzer and pyright have bugs. Count a disagreement as a phantom only where the oracle is *confident* (a real definition symbol, not a local/unknown). Keep `known_oracle_disagreements.toml` with a written reason per entry — reusing the `name_fallback_ceiling` device. ## Acceptance - [ ] frozen oracle artifacts for rust/ts/python tier-1 repos, version-stamped - [ ] phantom detection against the oracle, confidence-gated - [ ] honest-denominator internal-resolution metric per language - [ ] `known_oracle_disagreements.toml` with mandatory reasons - [ ] refresh procedure documented; oracle diffs reviewed, never auto-accepted

buildagent added the

Kind/Feature

Priority

Low

labels

2026-07-28 20:40:55 +02:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

h-dv/code-index#49

No description provided.

Rows
Columns