test: corpus metamorphic gate — determinism, cold==incremental, permutation and path invariance on real repos #42

New issue

Open

opened 2026-07-28 20:38:18 +02:00 by buildagent · 1 comment

buildagent commented

2026-07-28 20:38:18 +02:00

Member

P2 — regression protection with a measured 0-defect starting point.

From _prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md §2.1. Depends on #40.

Honest framing

The 2026-07-28 spike already ran determinism and cold==incremental against 7 real repos and they were clean. So this is explicitly not a bug hunt — it is a guard that locks in a property currently verified only by an ad-hoc shell script that lives in a scratchpad. Do not oversell it in the mission record.

Verified clean on 2026-07-28 (v0.8.7 @ 855710d):

determinism 7/7 (ripgrep, guzzle, express, sinatra, flask, zod, Dapper)
cold==incremental 4/4 under prepend-40-lines + append-symbol + delete + rename

Invariants to encode

Invariant	Status
Determinism — two independent cold indexes identical	verified clean, encode as gate
Cold == incremental — one-shot reconcile after edit/delete/rename	verified clean, encode as gate
Permutation invariance — shuffled file walk order → same result	NOT yet tested; real risk given same-dir/locality resolver tiers
Path invariance — same tree at `/a/b` vs `/x/y/z/w`	NOT yet tested; guards absolute-path leakage into handles/roots
Split invariance — one repo as single project vs primary+link	NOT yet tested; guards cross-project fabrication in fan-out
Encoding robustness — CRLF, BOM, UTF-16, non-UTF-8, huge minified, deep nesting	partially covered at fixture scale only

The three untested ones are the actual value of this issue.

Comparison method (reuse from the spike)

Compare on an id-independent projection — refs must render their target as tf.path||'#'||ts.name||'@'||ts.start_line, never target_id, or rowid ordering masks real differences. This is how the spike avoided a false green.

Acceptance

all six invariants encoded as asserting tests over tier-1 corpus
id-independent projection used throughout
first differing row reported, not just a hash mismatch
positive control per assertion (#44)
mission record states plainly that determinism + cold==incremental were already clean when encoded

**P2 — regression protection with a measured 0-defect starting point.** From `_prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md` §2.1. Depends on #40. ## Honest framing The 2026-07-28 spike **already ran determinism and cold==incremental against 7 real repos and they were clean**. So this is explicitly **not** a bug hunt — it is a guard that locks in a property currently verified only by an ad-hoc shell script that lives in a scratchpad. Do not oversell it in the mission record. Verified clean on 2026-07-28 (v0.8.7 @ 855710d): - determinism 7/7 (ripgrep, guzzle, express, sinatra, flask, zod, Dapper) - cold==incremental 4/4 under prepend-40-lines + append-symbol + delete + rename ## Invariants to encode | Invariant | Status | |---|---| | **Determinism** — two independent cold indexes identical | verified clean, encode as gate | | **Cold == incremental** — one-shot reconcile after edit/delete/rename | verified clean, encode as gate | | **Permutation invariance** — shuffled file walk order → same result | NOT yet tested; real risk given same-dir/locality resolver tiers | | **Path invariance** — same tree at `/a/b` vs `/x/y/z/w` | NOT yet tested; guards absolute-path leakage into handles/roots | | **Split invariance** — one repo as single project vs primary+link | NOT yet tested; guards cross-project fabrication in fan-out | | **Encoding robustness** — CRLF, BOM, UTF-16, non-UTF-8, huge minified, deep nesting | partially covered at fixture scale only | The three untested ones are the actual value of this issue. ## Comparison method (reuse from the spike) Compare on an **id-independent projection** — refs must render their target as `tf.path||'#'||ts.name||'@'||ts.start_line`, never `target_id`, or rowid ordering masks real differences. This is how the spike avoided a false green. ## Acceptance - [ ] all six invariants encoded as asserting tests over tier-1 corpus - [ ] id-independent projection used throughout - [ ] first differing row reported, not just a hash mismatch - [ ] positive control per assertion (#44) - [ ] mission record states plainly that determinism + cold==incremental were already clean when encoded

buildagent added the

labels

2026-07-28 20:40:44 +02:00

buildagent referenced this issue

2026-07-28 20:52:17 +02:00

bug: resolution is STICKY across incremental runs — renaming an anchor file leaves refs resolved that a cold index leaves unresolved #52

buildagent referenced this issue

2026-07-28 21:36:19 +02:00

test: corpus mutation guard — deletion/rename/injection phantom tripwires at real-repo ambiguity #43

buildagent referenced this issue

2026-07-28 21:36:28 +02:00

test: mandate positive controls on every corpus assertion (fifth near-miss of the vacuity class) #44

buildagent commented

2026-07-28 21:36:48 +02:00

Author

Member

Landed — `crates/indexer/tests/corpus_metamorphic.rs`

Five asserting tests over all 7 tier-1 repos, all comparing on the id-independent projection (a ref's target rendered path#name@line, never target_id):

test	result
`corpus_determinism`	7/7 identical
`corpus_path_invariance`	7/7 identical (`/s` vs `/d/e/e/p/e/r/still`)
`corpus_cold_equals_incremental_edit`	7/7 identical
`corpus_cold_equals_incremental_delete`	7/7 identical
`corpus_cold_equals_incremental_rename`	6/7 clean, cs-dapper diverges → #52

The three mutation kinds are split one per test rather than applied together, so a failure names the responsible operation instead of leaving three suspects. That split is what localised #52.

It found a real defect: #52

The combined test failed on cs-dapper. Isolating the three mutations showed the rename alone was responsible: renaming Dapper/SqlMapper.cs — semantically a no-op in C# — leaves 3 refs in CommandDefinition.cs resolved to SqlMapper.Settings.cs that a cold index of the identical tree declines to resolve. Root-caused to sticky resolution (target_id IS NULL gating) plus name-only invalidation. Filed as #52, deliberately not fixed here.

Those 3 sites are enumerated in known_incoherence() with a written reason and an issue reference — the oracle.toml name_fallback_ceiling device. assert_same_except tolerates only those exact (path, name, kind, line, col) sites; symbol differences are never tolerated, and any other ref difference still fails. Verified non-vacuous: the run reports 6 known-incoherence ref difference(s) tolerated (3 sites × both sides of the symmetric difference), so a stale list cannot pass silently. The list must shrink to empty when #52 lands.

Deviation from the issue's plan

Permutation invariance is NOT implemented. The walk order isn't externally controllable, so there is no honest way to test it without a walker seed knob. I'd rather say so than ship something that looks like the invariant but isn't. Split invariance (single project vs primary+link) is also not covered yet. Both remain open on this issue.

Path invariance — the one invariant here the spike never tested — is clean on all 7.

Acceptance: boxes 2, 3, 4, 5 met; box 1 partially (4 of 6 invariants).

## Landed — `crates/indexer/tests/corpus_metamorphic.rs` Five asserting tests over all 7 tier-1 repos, all comparing on the **id-independent projection** (a ref's target rendered `path#name@line`, never `target_id`): | test | result | |---|---| | `corpus_determinism` | 7/7 identical | | `corpus_path_invariance` | 7/7 identical (`/s` vs `/d/e/e/p/e/r/still`) | | `corpus_cold_equals_incremental_edit` | 7/7 identical | | `corpus_cold_equals_incremental_delete` | 7/7 identical | | `corpus_cold_equals_incremental_rename` | 6/7 clean, **cs-dapper diverges → #52** | The three mutation kinds are **split one per test** rather than applied together, so a failure names the responsible operation instead of leaving three suspects. That split is what localised #52. ## It found a real defect: #52 The combined test failed on cs-dapper. Isolating the three mutations showed the rename alone was responsible: renaming `Dapper/SqlMapper.cs` — semantically a no-op in C# — leaves 3 refs in `CommandDefinition.cs` resolved to `SqlMapper.Settings.cs` that a cold index of the identical tree declines to resolve. Root-caused to sticky resolution (`target_id IS NULL` gating) plus name-only invalidation. Filed as **#52**, deliberately not fixed here. Those 3 sites are enumerated in `known_incoherence()` with a written reason and an issue reference — the `oracle.toml` `name_fallback_ceiling` device. `assert_same_except` tolerates **only** those exact `(path, name, kind, line, col)` sites; symbol differences are never tolerated, and any other ref difference still fails. Verified non-vacuous: the run reports `6 known-incoherence ref difference(s) tolerated` (3 sites × both sides of the symmetric difference), so a stale list cannot pass silently. **The list must shrink to empty when #52 lands.** ## Deviation from the issue's plan **Permutation invariance is NOT implemented.** The walk order isn't externally controllable, so there is no honest way to test it without a walker seed knob. I'd rather say so than ship something that looks like the invariant but isn't. Split invariance (single project vs primary+link) is also not covered yet. Both remain open on this issue. Path invariance — the one invariant here the spike never tested — is clean on all 7. Acceptance: boxes 2, 3, 4, 5 met; box 1 partially (4 of 6 invariants).