h-dv/code-index

Watch

Fork 0

test: git-history oracle for diff tools, handles and generation epochs #50

New issue

Open

opened 2026-07-28 20:40:08 +02:00 by buildagent · 1 comment

buildagent commented

2026-07-28 20:40:08 +02:00

Member

P3 — a labelled dataset that already exists. Depends on #40 (history=true repos).

From _prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md §2.4.

Why

Every commit in a real repo is a labelled change-set. The diff tools have a free, large, realistic test set and currently use none of it.

This matters because I034's freshness barrier permanently bricked changed_symbols and review_diff on any repo with a tracked file under dist//bin//obj/ — and CI stayed green. A history-driven oracle is the direct guard against that class.

Three oracles

1. Span oracle (HARD assertion).
changed_symbols(C~1..C) must exactly equal the set of symbols whose spans intersect the diff hunks — computable from git plus the index alone, no external tooling. Run it across hundreds of commits.

2. Rename/move torture (HARD).
Real history contains file renames, mode changes, and merges — exactly the inputs cih2_ handle resolution claims to survive (exact / moved_or_changed / ambiguous / missing). Replay them and count misclassifications. Today handles are tested only on synthetic renames.

3. Fix-commit impact recall (DIAGNOSTIC ONLY).
For a bug-fix commit, seed change_impact from the parent's touched symbols and ask whether the files the fix actually touched appear. Measures impact recall against reality — but must not gate: a fix touching unrelated files is not our defect. Report as a trend.

Interaction with #39

#39 replays history through the watcher; this issue replays it through the diff tools. Same fetched repos (history = true), different assertion surface — build the checkout-driver once and share it.

Acceptance

span oracle asserted across ≥200 commits of ≥2 repos
rename/move handle classification measured, misclassifications enumerated
fix-commit recall reported as a diagnostic trend, explicitly non-gating
merge commits and mode changes included, not filtered out
positive control: assert each replayed commit actually changed tracked, indexable files (#44)

Runtime-plugin architecture expansion

History replay must bind every expected result to an active generation epoch. Add histories where source commits also change a project’s requested package digest/capability file while installed immutable packages are supplied by the test registry.

New assertions:

source-only history behaves exactly as before under a stable package generation;
package-only generation changes invalidate generation-bound cursors/handles without fabricating source diffs;
failed activation leaves diff tools answering from the previous active generation with disclosure;
a commit moving a file between old/new claim domains produces cold==activated classification;
provenance and spans remain tied to the generation that produced them.

Do not treat arbitrary remote package download as part of git replay; package artifacts are pinned local test inputs.

**P3 — a labelled dataset that already exists. Depends on #40 (history=true repos).** From `_prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md` §2.4. ## Why Every commit in a real repo is a labelled change-set. The diff tools have a free, large, realistic test set and currently use none of it. This matters because **I034's freshness barrier permanently bricked `changed_symbols` and `review_diff`** on any repo with a tracked file under `dist/`/`bin/`/`obj/` — and CI stayed green. A history-driven oracle is the direct guard against that class. ## Three oracles **1. Span oracle (HARD assertion).** `changed_symbols(C~1..C)` must exactly equal the set of symbols whose spans intersect the diff hunks — computable from git plus the index alone, no external tooling. Run it across hundreds of commits. **2. Rename/move torture (HARD).** Real history contains file renames, mode changes, and merges — exactly the inputs `cih2_` handle resolution claims to survive (`exact` / `moved_or_changed` / `ambiguous` / `missing`). Replay them and count misclassifications. Today handles are tested only on synthetic renames. **3. Fix-commit impact recall (DIAGNOSTIC ONLY).** For a bug-fix commit, seed `change_impact` from the parent's touched symbols and ask whether the files the fix actually touched appear. Measures impact recall against reality — but **must not gate**: a fix touching unrelated files is not our defect. Report as a trend. ## Interaction with #39 #39 replays history through the *watcher*; this issue replays it through the *diff tools*. Same fetched repos (`history = true`), different assertion surface — build the checkout-driver once and share it. ## Acceptance - [ ] span oracle asserted across ≥200 commits of ≥2 repos - [ ] rename/move handle classification measured, misclassifications enumerated - [ ] fix-commit recall reported as a diagnostic trend, explicitly non-gating - [ ] merge commits and mode changes included, not filtered out - [ ] positive control: assert each replayed commit actually changed tracked, indexable files (#44) ## Runtime-plugin architecture expansion History replay must bind every expected result to an active generation epoch. Add histories where source commits also change a project’s requested package digest/capability file while installed immutable packages are supplied by the test registry. New assertions: - source-only history behaves exactly as before under a stable package generation; - package-only generation changes invalidate generation-bound cursors/handles without fabricating source diffs; - failed activation leaves diff tools answering from the previous active generation with disclosure; - a commit moving a file between old/new claim domains produces cold==activated classification; - provenance and spans remain tied to the generation that produced them. Do not treat arbitrary remote package download as part of git replay; package artifacts are pinned local test inputs.

buildagent added the

labels

2026-07-28 20:40:56 +02:00

buildagent changed title from ~~test: git history as a free oracle for changed_symbols / review_diff / handle resolution~~ to test: git-history oracle for diff tools, handles and generation epochs

2026-08-26 13:39:31 +02:00

buildagent added a new dependency

2026-08-26 13:43:37 +02:00

#78 index: generation-safe plugin activation, invalidation, promotion and rollback

buildagent commented

2026-09-07 00:26:14 +02:00

Author

Member

State on master `1d81180`: not started. The checkout driver this issue said to share DOES exist now, so the remaining work is smaller than the issue implies.

I read this as a candidate for my lane and did not take it; recording the tree state so the next lane does not re-derive it.

Nothing here is done. None of the three oracles exists. changed_symbols / review_diff appear in ~20 test files, all fixture-scale; no test replays commits through the diff tools; and handle classification (moved_or_changed) is asserted in exactly one place, crates/mcp-server/tests/mcp_smoke.rs, on synthetic renames — which is exactly what this issue says.

What HAS landed since filing, and it is the expensive half: #39 shipped as crates/indexer/tests/corpus_watcher_replay.rs. That is the checkout driver this issue said to build once and share — it checks out a commit, walks forward through real history, and compares against a cold index at every step. It also already made two decisions this issue would otherwise have to make again:

reconcile_interval: None and head_poll_interval off, so the 30-minute safety-net re-walk cannot let a suite pass while the live path is broken. A history replay through the diff tools wants the same switch for the same reason.
tests/corpus/corpus.toml carries exactly one history = true repo (line 35), described there as the designated history-replay repo. This issue's acceptance asks for "≥200 commits of ≥2 repos", so either a second repo gets history = true — which costs fetch time on every corpus job — or that clause gets revised with a reason. Worth settling before writing any code.

Why it still matters, unchanged: the I034 freshness barrier permanently bricked changed_symbols and review_diff on any repo with a tracked file under dist//bin//obj/, and CI stayed green. The watcher replay does not cover that class — it grades index convergence, not the diff tools' answers.

Also unchanged: oracle 3 (fix-commit impact recall) must stay non-gating. A fix touching unrelated files is not our defect, and a recall number that can be moved by someone else's commit is a trend, not a gate.

## State on master `1d81180`: not started. The checkout driver this issue said to share DOES exist now, so the remaining work is smaller than the issue implies. I read this as a candidate for my lane and did not take it; recording the tree state so the next lane does not re-derive it. **Nothing here is done.** None of the three oracles exists. `changed_symbols` / `review_diff` appear in ~20 test files, all fixture-scale; no test replays commits through the diff tools; and handle classification (`moved_or_changed`) is asserted in exactly one place, `crates/mcp-server/tests/mcp_smoke.rs`, on synthetic renames — which is exactly what this issue says. **What HAS landed since filing, and it is the expensive half:** #39 shipped as `crates/indexer/tests/corpus_watcher_replay.rs`. That is the checkout driver this issue said to build once and share — it checks out a commit, walks forward through real history, and compares against a cold index at every step. It also already made two decisions this issue would otherwise have to make again: - `reconcile_interval: None` and `head_poll_interval` off, so the 30-minute safety-net re-walk cannot let a suite pass while the live path is broken. A history replay through the diff tools wants the same switch for the same reason. - `tests/corpus/corpus.toml` carries exactly **one** `history = true` repo (line 35), described there as the designated history-replay repo. This issue's acceptance asks for "≥200 commits of **≥2** repos", so either a second repo gets `history = true` — which costs fetch time on every corpus job — or that clause gets revised with a reason. Worth settling before writing any code. **Why it still matters, unchanged:** the I034 freshness barrier permanently bricked `changed_symbols` and `review_diff` on any repo with a tracked file under `dist/`/`bin/`/`obj/`, and CI stayed green. The watcher replay does not cover that class — it grades index convergence, not the diff tools' answers. Also unchanged: oracle 3 (fix-commit impact recall) must stay non-gating. A fix touching unrelated files is not our defect, and a recall number that can be moved by someone else's commit is a trend, not a gate.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Depends on

#78 index: generation-safe plugin activation, invalidation, promotion and rollback

h-dv/code-index

Reference

h-dv/code-index#50

No description provided.

Rows
Columns