h-dv/code-index

Fork 0

test: watcher history replay — drive the LIVE notify path across real git history, assert cold==incremental at every commit #39

New issue

Closed

opened 2026-07-28 20:37:34 +02:00 by buildagent · 1 comment

buildagent commented

2026-07-28 20:37:34 +02:00

Member

P1 — the single highest-expected-yield item in the corpus plan.

From _prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md §0 + revised §8.

Why this is #1

The 2026-07-28 spike indexed 7 real OSS repos (ripgrep, flask, zod, express, guzzle, sinatra, Dapper; 170–580 files each) and cleared every axis it could reach:

parse/crash: 7/7 rc=0, parse_errors=0, no panics
determinism: 7/7 identical on an id-independent projection
cold==incremental: 4/4 identical under prepend-40-lines + append-symbol + delete-file + rename-file
deletion mutation: 0 rebinds / 1413 resolved cross-file refs (rust/php/ruby)

But every one of those used one-shot code-index index — the reconciler, not the live notify watcher.

Every historical staleness defect in this project lived in the watcher:

I028 — watcher relative_path drift (single-file-root watch matching)
I034 — the freshness barrier that PERMANENTLY BRICKED changed_symbols/review_diff
still open — Linux dynamic re-watch loss on directory recreate (see project_index_staleness_no_reconcile)

So the corpus's value is concentrated precisely where the spike could not look.

What to build

A harness that, for a pinned repo with real history:

checks out commit N, cold-indexes it, starts code-index watch (live notify, not index);
for each of the next K commits: git checkout the commit while the watcher is running, wait for quiescence, then
compare the watcher-updated DB against a fresh cold index of the same tree, on the id-independent projection:
- symbols: path,name,kind,lang,start_line,start_col,end_line,end_col,qualified_name,visibility
- refs: path,name,kind,start_line,start_col,roles,qualified,qualifier, target rendered as tf.path||'#'||ts.name||'@'||ts.start_line
  (rendering the target by path+name+line, never target_id, so rowid ordering cannot mask a difference)
fail with the first differing row, not just a hash mismatch.

Coverage requirements

Must exercise file create, modify, delete, rename, directory create, and directory delete+recreate — the last is the open Linux bug's exact shape. A commit range that does not contain all six shapes must be reported as such, not silently pass.
Must run on Linux and Windows (per the standing cross-platform rule) — the historical bugs are platform-specific.
Quiescence must be detected, not slept on; a fixed sleep will make this flaky and it will get disabled.

Anti-vacuity (mandatory)

Per the §0 method note — a vacuity nearly produced a false green during the spike itself. Every assertion ships with a positive control: assert the tree actually changed between commits (file count / symbol count delta non-zero) before comparing. A replay where nothing changed must FAIL, not pass. See #44.

Acceptance

Replays ≥50 consecutive commits of ≥1 real repo through the live watcher
cold==incremental asserted per commit on both projections, first-diff reported
all six event shapes covered, coverage asserted
positive control per step
green on Linux and Windows
deterministic, non-flaky, quiescence-based not sleep-based

**P1 — the single highest-expected-yield item in the corpus plan.** From `_prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md` §0 + revised §8. ## Why this is #1 The 2026-07-28 spike indexed 7 real OSS repos (ripgrep, flask, zod, express, guzzle, sinatra, Dapper; 170–580 files each) and **cleared every axis it could reach**: - parse/crash: 7/7 `rc=0`, `parse_errors=0`, no panics - determinism: 7/7 identical on an id-independent projection - cold==incremental: 4/4 identical under prepend-40-lines + append-symbol + delete-file + rename-file - deletion mutation: **0 rebinds / 1413 resolved cross-file refs** (rust/php/ruby) **But every one of those used one-shot `code-index index` — the reconciler, not the live `notify` watcher.** Every historical staleness defect in this project lived in the *watcher*: - I028 — watcher `relative_path` drift (single-file-root watch matching) - I034 — the freshness barrier that PERMANENTLY BRICKED `changed_symbols`/`review_diff` - **still open** — Linux dynamic re-watch loss on directory recreate (see `project_index_staleness_no_reconcile`) So the corpus's value is concentrated precisely where the spike could not look. ## What to build A harness that, for a pinned repo with real history: 1. checks out commit `N`, cold-indexes it, starts `code-index watch` (live `notify`, not `index`); 2. for each of the next `K` commits: `git checkout` the commit **while the watcher is running**, wait for quiescence, then 3. compare the watcher-updated DB against a **fresh cold index** of the same tree, on the id-independent projection: - symbols: `path,name,kind,lang,start_line,start_col,end_line,end_col,qualified_name,visibility` - refs: `path,name,kind,start_line,start_col,roles,qualified,qualifier, target rendered as tf.path||'#'||ts.name||'@'||ts.start_line` (rendering the target by path+name+line, never `target_id`, so rowid ordering cannot mask a difference) 4. fail with the first differing row, not just a hash mismatch. ## Coverage requirements - Must exercise **file create, modify, delete, rename, directory create, and directory delete+recreate** — the last is the open Linux bug's exact shape. A commit range that does not contain all six shapes must be reported as such, not silently pass. - Must run on **Linux and Windows** (per the standing cross-platform rule) — the historical bugs are platform-specific. - Quiescence must be detected, not slept on; a fixed sleep will make this flaky and it will get disabled. ## Anti-vacuity (mandatory) Per the §0 method note — a vacuity nearly produced a false green *during the spike itself*. Every assertion ships with a **positive control**: assert the tree actually changed between commits (file count / symbol count delta non-zero) before comparing. A replay where nothing changed must FAIL, not pass. See #44. ## Acceptance - [ ] Replays ≥50 consecutive commits of ≥1 real repo through the live watcher - [ ] cold==incremental asserted per commit on both projections, first-diff reported - [ ] all six event shapes covered, coverage asserted - [ ] positive control per step - [ ] green on Linux and Windows - [ ] deterministic, non-flaky, quiescence-based not sleep-based

buildagent referenced this issue

2026-07-28 20:37:49 +02:00

test: OSS corpus infrastructure — corpus.toml manifest, pinned fetch/cache, and a non-vacuous coverage manifest #40

buildagent referenced this issue

2026-07-28 20:40:08 +02:00

test: git history as a free oracle for changed_symbols / review_diff / handle resolution #50

buildagent added the

labels

2026-07-28 20:40:41 +02:00

buildagent referenced this issue

2026-07-28 21:36:11 +02:00

test: OSS corpus infrastructure — corpus.toml manifest, pinned fetch/cache, and a non-vacuous coverage manifest #40

buildagent referenced this issue

2026-07-28 21:36:28 +02:00

test: mandate positive controls on every corpus assertion (fifth near-miss of the vacuity class) #44

buildagent commented

2026-07-28 21:40:01 +02:00

Author

Member

Landed — `crates/indexer/tests/corpus_watcher_replay.rs`

Two tests, because the live path and the shipped contract are different promises and conflating them would have hidden the result.

watcher-live-only[rust-ripgrep]:      converged=3 inert=18 new_dir_commits=3
                                      diverged_on_new_dir=3 shapes={'A','D','M'}
watcher-with-reconcile[rust-ripgrep]: converged=8 inert=16 new_dir_commits=3
                                      slowest=3.18s   diverged=0

corpus_watcher_live_path_only_misses_new_directories — reconcile_interval: None, so the safety net cannot paper over a dropped event. Asserts that the only commits the live path fails to absorb are those creating a new directory. A dropped modify, delete, or same-directory add fails.

corpus_watcher_with_reconcile_converges_on_every_commit — the shipped configuration. Asserts every commit converges, including the new-directory ones. 0 divergences.

What this measured

The long-open Linux dynamic re-watch gap is now reproduced on real data and quantified. Commit be739c7 index: add grep-index crate creates crates/index/ (6 files, directory absent in the parent); the live watcher misses all of it — inotify watches are per-directory and the files land before the new watch is armed. All 3 divergences were new-directory commits and nothing else, and the periodic reconcile converges exactly those 3.

That is a much sharper statement than "the watcher sometimes misses things on Linux."

Design points that carry the test

Convergence, not quiescence. The expected projection is cold-indexed FIRST and the watcher DB polled until it matches. Polling for a merely stable projection would declare success before the watcher had begun reacting.
WatchMode::Native, not Auto — a silent fallback to polling would hide the very inotify defects this exists to catch.
Restart after divergence. A live-only run cannot self-heal, so without a restart every later commit would also "diverge" and bury the cause.
Commit selection. Replaying the last N commits verbatim gave terrible coverage: 24 of 29 ripgrep pairs touch only docs, CI and Cargo.lock. interesting_commits now walks history keeping only commits that change an indexable extension, taken from the newest end.

Deviations from the acceptance criteria — stated plainly

25 commits by default, not ≥50 (COSI_CORPUS_REPLAY_COMMITS raises it). Each step cold-indexes the tree; 25 already costs ~200s for the pair of tests. The effective number is what matters and it is lower still — 3 and 8 non-inert steps respectively.
Rename, dir-create and dir-delete+recreate are not separately asserted. shapes proves add/modify/delete were exercised; the pinned ripgrep window contains no R and no directory deletion. Rather than fake them with synthetic edits in a test that claims to replay real history, they are left uncovered and named here.
Linux only so far. The cross-platform half of the acceptance is unmet; the nightly job is linux-hosted.
inert differs between the two tests (18 vs 16) purely as sampling noise — with the reconcile armed the DB is occasionally mid-update when sampled, so a step counts as converging rather than inert. It does not affect either assertion.

Acceptance: boxes 2 and 4 met, 1 and 3 partially, 5 unmet (Linux only), 6 met (quiescence-based polling, no fixed sleeps; green on repeated runs).

## Landed — `crates/indexer/tests/corpus_watcher_replay.rs` Two tests, because the live path and the shipped contract are different promises and conflating them would have hidden the result. ``` watcher-live-only[rust-ripgrep]: converged=3 inert=18 new_dir_commits=3 diverged_on_new_dir=3 shapes={'A','D','M'} watcher-with-reconcile[rust-ripgrep]: converged=8 inert=16 new_dir_commits=3 slowest=3.18s diverged=0 ``` **`corpus_watcher_live_path_only_misses_new_directories`** — `reconcile_interval: None`, so the safety net cannot paper over a dropped event. Asserts that the **only** commits the live path fails to absorb are those creating a new directory. A dropped modify, delete, or same-directory add fails. **`corpus_watcher_with_reconcile_converges_on_every_commit`** — the shipped configuration. Asserts every commit converges, including the new-directory ones. **0 divergences.** ## What this measured The long-open Linux dynamic re-watch gap is now **reproduced on real data and quantified**. Commit `be739c7 index: add grep-index crate` creates `crates/index/` (6 files, directory absent in the parent); the live watcher misses all of it — inotify watches are per-directory and the files land before the new watch is armed. All 3 divergences were new-directory commits and nothing else, and the periodic reconcile converges exactly those 3. That is a much sharper statement than "the watcher sometimes misses things on Linux." ## Design points that carry the test - **Convergence, not quiescence.** The expected projection is cold-indexed FIRST and the watcher DB polled until it matches. Polling for a merely *stable* projection would declare success before the watcher had begun reacting. - **`WatchMode::Native`, not `Auto`** — a silent fallback to polling would hide the very inotify defects this exists to catch. - **Restart after divergence.** A live-only run cannot self-heal, so without a restart every later commit would also "diverge" and bury the cause. - **Commit selection.** Replaying the last N commits verbatim gave terrible coverage: 24 of 29 ripgrep pairs touch only docs, CI and `Cargo.lock`. `interesting_commits` now walks history keeping only commits that change an indexable extension, taken from the **newest** end. ## Deviations from the acceptance criteria — stated plainly - **25 commits by default, not ≥50** (`COSI_CORPUS_REPLAY_COMMITS` raises it). Each step cold-indexes the tree; 25 already costs ~200s for the pair of tests. The effective number is what matters and it is lower still — 3 and 8 non-inert steps respectively. - **Rename, dir-create and dir-delete+recreate are not separately asserted.** `shapes` proves add/modify/delete were exercised; the pinned ripgrep window contains no `R` and no directory *deletion*. Rather than fake them with synthetic edits in a test that claims to replay real history, they are left uncovered and named here. - **Linux only so far.** The cross-platform half of the acceptance is unmet; the nightly job is linux-hosted. - `inert` differs between the two tests (18 vs 16) purely as sampling noise — with the reconcile armed the DB is occasionally mid-update when sampled, so a step counts as converging rather than inert. It does not affect either assertion. Acceptance: boxes 2 and 4 met, 1 and 3 partially, 5 unmet (Linux only), 6 met (quiescence-based polling, no fixed sleeps; green on repeated runs).

buildagent referenced this issue from a pull request that will close it,

2026-07-29 21:26:30 +02:00

I039: OSS corpus test system + the four defects it found (v0.9.0, schema v25) #58

buildagent closed this issue

2026-08-03 14:06:57 +02:00

buildagent referenced this issue from a commit

2026-08-03 14:06:57 +02:00

test: OSS corpus test system — metamorphic, mutation, live-watcher replay (#39-#44)

buildagent referenced this issue from a commit

2026-08-03 14:06:57 +02:00

I039: OSS corpus test system + resolver fixes (v0.9.0, schema v25) (#58)