I039: OSS corpus test system + the four defects it found (v0.9.0, schema v25) #58
No reviewers
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
h-dv/code-index!58
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "corpus-test-system"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #39, #40, #43, #44, #54, #55, #56, #57. Advances #41, #42, #45. #53 stays open — see below.
13 commits, 24 files, +5,485 lines. Version 0.8.7 → 0.9.0; schema v21 → v25.
What this is
Seven real OSS repositories pinned by sha, graded by oracle-free invariants, self-oracling mutations, a live-watcher history replay, and an exact-count ratchet. Plus the four resolver defects the corpus found and three of them fixed.
The spike falsified four of its own predictions first
Before building anything, I measured the plan's premises against the 7 repos at v0.8.7:
parse_errors=0The indexer path at tier-1 scale had nothing left to say, so the plan's early phases were demoted and the watcher and scale promoted. Every defect below came from those two axes.
Defects found and fixed
#52 sticky anchored resolution. Tier 1Q anchors qualified refs on any code file's stem — third-party evidence a stem-blind refill can't see, made permanent by
target_id IS NULLgating. Fixed withstale_path_evidence+ aninstr()relevance test that NULLs matching resolutions and forces a Full pass; the two halves are inseparable, because NULLing under a Scoped pass loses the resolution permanently.#53/#54 superlinear resolve cost. Not algorithmic — three missing indexes (one of them
temp.symbol_buckets' name-join, I025's failure mode recurring) plus a correlated enclosing-symbol subquery inrebuild_symbol_edges, now set-wise: 5.79s → 1.35s, edge set byte-identical on all 9 repos.#56 CommonJS invisibility. express: 410
require()calls, zero imports — its whole module graph missing. Both halves shipped: export-surface analysis (precision) and import capture (0 → 388 imports).#57 relative specifiers matched by bare stem.
./utilsfromlib/reachedtest/utils.js. Now path-resolved; 155/155, 509/509, 173/176 specifiers resolve.Things that went other than planned — please read these
Adversarial review: 8 findings, all in the tests or the claims
Neither core fix could be broken (1,036,264 ref rows compared with zero differences; injected mid-transaction errors; an 8-round two-process race; 8 mutants). What the reviewers did break:
write_delete's arm had zero isolated coverage — deletable with the suite green;corpus_cold_equals_incremental_addwas vacuous — passed with the entire fix disabled;Verification
cargo test --workspacegreen · fmt ·clippy -D warnings· all 7precision_gatelanguages (phantom == 0) · corpus metamorphic 6/6 · mutation guard 0 rebinds / 6,981 sites · ratchet green · scale within ceilings · watcher replay 18 converged / 3 inert, all divergences exactly the new-directory commits.Live upgrade dogfooded before tagging: this repo's own v21 index migrated to v25 in 5s (full reparse, 0/269 parse errors), and the migrated DB is byte-identical to a cold index on symbols, refs and edges.
doctorclean. That path is the I034 failure class and had never run against a real in-use database.Reviewer notes
corpusandcorpus-scaleCI jobs are nightly/weekly, not PR gates — they need GitHub egress from the runner, which is unverified. I'll dispatch them manually against this branch; if egress is blocked that's the thing to fix before merge.tests/corpus/baseline.jsoncarries a mandatory_blessed.reason; every count change in this PR has one.Seven real open-source repositories, one per language, pinned by commit sha in tests/corpus/corpus.toml, graded by oracle-free invariants and self-oracling mutations. Unlike tests/fixtures nothing here is hand-authored, so the corpus reaches ambiguity the 4-14 file fixtures cannot: 7056 cross-file bindings under test vs candidate pools of ~1. A spike ran first and FALSIFIED four predictions: parse/crash (7/7 clean), determinism (7/7), cold==incremental (4/4) and deletion mutation (0 rebinds / 1413 refs) were already clean at v0.8.7. The indexer path had nothing left to say, so the watcher — where I028, I034 and the open Linux re-watch issue all lived — was promoted. #40/#44 harness: sha-pinned fetch, id-independent projection (path#name@line, never target_id), coverage manifest, positive controls enforced by Coverage::finish #42 metamorphic: determinism 7/7, path-invariance 7/7, cold==incremental split per mutation kind #43 mutation guard: 0 rebinds, 0 stray bindings / 7056 sites #39 watcher replay: live path (safety net OFF) vs the shipped configuration, over real ripgrep history Found #52: resolution is sticky. Every resolve tier is gated on `target_id IS NULL` and the only un-resolver keys on symbol NAMES, so renaming a file destroys the stem-derived qualifier-anchoring evidence without invalidating anything. Coherence defect, 0 dangling targets, NOT fixed here — resolver changes need adversarial review. The three affected sites are enumerated with a written reason; anything new still fails. Quantifies the open Linux gap: with reconcile_interval None, ALL live divergences were new-directory commits and nothing else; the periodic reconcile converges exactly those. Corpus is opt-in via COSI_CORPUS_DIR (no default path — it would silently slow every cargo test). Nightly CI job sets COSI_CORPUS_REQUIRE=1 so a run that grades nothing FAILS. Not covered: permutation and split invariance (#42), tier-3 scale (#41), Windows replay, rename/dir-delete event shapes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KExm1ysuBrhWPytdSrnUKd`from_id` was a CORRELATED subquery — "smallest symbol covering this line", `ORDER BY ... LIMIT 1` — evaluated once per resolved ref. The query plan showed `USE TEMP B-TREE FOR ORDER BY` *inside* it: one sort per ref, 116k of them on rust-analyzer. Cost 0.27s at 36k refs vs 5.79s at 340k (24.6x for 9.3x refs). Now the enclosing symbol is resolved once per distinct (file_id, line) — 89k rather than 116k probes on rust-analyzer — via a single window-function sort into an indexed temp table. Measured on a quiet box, medians of 3: isolated SQL 5.79s -> 1.35s (4.3x) end-to-end rust-analyzer 25.7s -> 21.4s wall Output equality PROVEN, not inferred from counts: symmetric difference of the full edge set (from_id, to_id, weight) between old and new SQL is ZERO on all nine corpus repos; end-to-end, the fully-rendered edge projection (both endpoints as path#name@line plus weight) is md5- identical on rust-analyzer, py-django, rust-ripgrep, cs-dapper, ts-zod. Two things deliberately NOT done, both measured: * NOT a single self-contained statement. Expressing it with MATERIALIZED CTEs measured 19.9s — 3.4x SLOWER than the correlated form it replaces — because an unindexed CTE join is probed once per ref, the very shape that made the original slow. The temp index is load-bearing; re-measure before folding this back for tidiness. * m0013's inline copy is left untouched. Migrations are frozen history, it is a one-time seed, and runtime rebuilds overwrite it from this function — with provably identical output. Since the SQL is no longer expressible as one SELECT, the exported SYMBOL_EDGES_SELECT const is gone and the daemon's test helper now calls the production `rebuild_symbol_edges` directly. One implementation, so the two cannot drift — the same reasoning behind sharing TEST_MODULE_CONTAINS_REF. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KExm1ysuBrhWPytdSrnUKdTier 1b's and tier 3's reachability arms matched an import against a candidate file's bare STEM, which discards the directory a RELATIVE specifier is relative to. Measured on express: `require("./utils")` in `lib/application.js` matched both `lib/utils.js` and `test/utils.js`. Relative specifiers are now resolved to an actual file (in Rust — SQLite cannot normalise `../`) and match only through the resolved `temp.import_rel` table; non-relative specifiers keep the stem/segment behaviour, having no directory to resolve against. A strict narrowing: it can remove candidates, never invent one. Tier 1R is deliberately NOT narrowed. Its arm additionally requires the BOUND TYPE to appear in the module string, which a relative specifier never satisfies (the type is on the import's alias) — so it cannot produce the same-stem confusion, and excluding relative specifiers there would cost recall for no precision gain. ## Two dialects, both found by measuring rather than reasoning The first cut resolved paths naively and cost ts-zod 2 659 resolutions — a 23% recall collapse. Cause: TypeScript NodeNext writes the EMITTED extension, `import "./api.js"` where the file is `api.ts`, so nothing resolved and the narrowing removed reachability wholesale. Fixed with extension substitution. Then python-flask resolved only 14 of 176 specifiers (8%). Cause: the python plugin stores the IMPORTED SYMBOL in the module string, so `from ..config import Config` arrives as `..config.Config` and the last segment is a symbol, not a path component. Both readings are now offered, longest first. 173/176 (98%). That resolution RATE is the metric that says whether the narrowing is sound, so it is now logged (`relative_imports=N resolved=M`): an unresolved specifier contributes no reachability at all, so a low rate means lost recall rather than removed phantoms. ## Measured effect ts-zod 11246 -> 10931 (gained 125, lost 440) python-flask 2075 -> 2076 (gained 3, lost 2) js-express 4152 -> 4152 unchanged php-guzzle / ruby-sinatra / rust-ripgrep / cs-dapper unchanged ts-zod's 440 losses are stem-coincidence phantoms, not resolution failures — all 509 of its relative specifiers resolve to a file. Of the losses, 81 crossed the v3/v4 version boundary and 20 bound into test/bench/fixture directories; a hand-traced case had `init` in `mini/iso.ts` bound to `core/core.ts` purely because the specifier `../core/index.js` contains the segment `core`, and `core.ts` does not export `init` at all. The 125 gains cross no version boundary. Three hermetic tests, each with a positive control: the same-stem decoy (with a control call into the file the specifier really names, since an import binding alone lands in `imports` not `refs` and would make the control vacuous), NodeNext substitution, and the Python symbol-suffix form. Baseline re-blessed with the reasoning recorded in the file; the ratchet caught all four count changes first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KExm1ysuBrhWPytdSrnUKdThe JS/TS plugin emitted imports only from `import_statement`, so `require()` — a plain `call_expression` — was invisible. express is 100% CommonJS: 410 `require()` calls across 131 files and ZERO imports, i.e. its entire module graph was missing, starving every import-dependent resolution path. Now captured in the SAME row shape as ESM (one row per in-scope binding with `alias`, or a module-only row for a side-effect `require`), covering plain bindings, shorthand and renamed destructuring, array patterns, `require("m").member`, bare statements, dynamic `import()`, and static template specifiers. A computed `require(name)` or an interpolated template is deliberately NOT captured: there is no statically-known module and inventing one is the same dishonesty as inventing a resolution. js-express imports 0 -> 388, ts-zod +1. ## Why this is a re-land This was implemented, measured, and REVERTED a few commits ago because it produced six phantoms — `lib/application.js` binding a local `fn` into `test/utils.js#fn`. The import rows were correct; the fault was downstream, in tier 1b matching `./utils` against any file whose STEM is `utils`. #57 fixed that, and with it underneath the same capture now lands with: imports 0 -> 388, resolved 4154 -> 4152, gained=0, lost=2 lib/* -> test/utils.js bindings: 0 (was 6) The two losses are #56 part 1's already-blessed phantom removals, not new. Zero resolutions gained — which also corrects my original estimate on the issue: the dominant CommonJS shape is `util.method()`, a member call through a module object, and that needs receiver typing rather than an import edge. The value here is the module graph itself (get_dependencies, importers_of, change_impact), not a resolution jump. One bug found by measuring the resolution RATE rather than assuming it: express's `require("../..")` walks to the repo root, leaving an empty base, and the directory-entry candidate builder produced "/index.js" — an absolute path matching nothing. Fixed; relative-specifier resolution is now 155/155 on express, 509/509 on zod, 173/176 on flask. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KExm1ysuBrhWPytdSrnUKd