test: git history as a free oracle for changed_symbols / review_diff / handle resolution #50

New issue

Open

opened 2026-07-28 20:40:08 +02:00 by buildagent · 0 comments

buildagent commented

2026-07-28 20:40:08 +02:00

Member

P3 — a labelled dataset that already exists. Depends on #40 (history=true repos).

From _prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md §2.4.

Why

Every commit in a real repo is a labelled change-set. The diff tools have a free, large, realistic test set and currently use none of it.

This matters because I034's freshness barrier permanently bricked changed_symbols and review_diff on any repo with a tracked file under dist//bin//obj/ — and CI stayed green. A history-driven oracle is the direct guard against that class.

Three oracles

1. Span oracle (HARD assertion).
changed_symbols(C~1..C) must exactly equal the set of symbols whose spans intersect the diff hunks — computable from git plus the index alone, no external tooling. Run it across hundreds of commits.

2. Rename/move torture (HARD).
Real history contains file renames, mode changes, and merges — exactly the inputs cih2_ handle resolution claims to survive (exact / moved_or_changed / ambiguous / missing). Replay them and count misclassifications. Today handles are tested only on synthetic renames.

3. Fix-commit impact recall (DIAGNOSTIC ONLY).
For a bug-fix commit, seed change_impact from the parent's touched symbols and ask whether the files the fix actually touched appear. Measures impact recall against reality — but must not gate: a fix touching unrelated files is not our defect. Report as a trend.

Interaction with #39

#39 replays history through the watcher; this issue replays it through the diff tools. Same fetched repos (history = true), different assertion surface — build the checkout-driver once and share it.

Acceptance

span oracle asserted across ≥200 commits of ≥2 repos
rename/move handle classification measured, misclassifications enumerated
fix-commit recall reported as a diagnostic trend, explicitly non-gating
merge commits and mode changes included, not filtered out
positive control: assert each replayed commit actually changed tracked, indexable files (#44)

**P3 — a labelled dataset that already exists. Depends on #40 (history=true repos).** From `_prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md` §2.4. ## Why Every commit in a real repo is a labelled change-set. The diff tools have a free, large, realistic test set and currently use none of it. This matters because **I034's freshness barrier permanently bricked `changed_symbols` and `review_diff`** on any repo with a tracked file under `dist/`/`bin/`/`obj/` — and CI stayed green. A history-driven oracle is the direct guard against that class. ## Three oracles **1. Span oracle (HARD assertion).** `changed_symbols(C~1..C)` must exactly equal the set of symbols whose spans intersect the diff hunks — computable from git plus the index alone, no external tooling. Run it across hundreds of commits. **2. Rename/move torture (HARD).** Real history contains file renames, mode changes, and merges — exactly the inputs `cih2_` handle resolution claims to survive (`exact` / `moved_or_changed` / `ambiguous` / `missing`). Replay them and count misclassifications. Today handles are tested only on synthetic renames. **3. Fix-commit impact recall (DIAGNOSTIC ONLY).** For a bug-fix commit, seed `change_impact` from the parent's touched symbols and ask whether the files the fix actually touched appear. Measures impact recall against reality — but **must not gate**: a fix touching unrelated files is not our defect. Report as a trend. ## Interaction with #39 #39 replays history through the *watcher*; this issue replays it through the *diff tools*. Same fetched repos (`history = true`), different assertion surface — build the checkout-driver once and share it. ## Acceptance - [ ] span oracle asserted across ≥200 commits of ≥2 repos - [ ] rename/move handle classification measured, misclassifications enumerated - [ ] fix-commit recall reported as a diagnostic trend, explicitly non-gating - [ ] merge commits and mode changes included, not filtered out - [ ] positive control: assert each replayed commit actually changed tracked, indexable files (#44)

buildagent added the

labels

2026-07-28 20:40:56 +02:00