h-dv/code-index
Watch
4
0
Fork
You've already forked code-index
 0
Code Issues 46 Pull requests Projects Releases 98 Packages Wiki Activity Actions

Two LineIndex phantoms survive on rust-analyzer because tier 1Q anchors on a file STEM — directions 1 and 2 (package_root_of/lib) measured and REFUSED, direction 3 is what remains #168

New issue
Closed
opened 2026-09-06 01:08:01 +02:00 by buildagent · 4 comments
buildagent commented 2026-09-06 01:08:01 +02:00
Member
Copy link

Found while bounding #165's phantom risk, with an oracle independent of the resolver: each ref crate's own Cargo.toml.

The two binds

Of the 16 717 binds #165's fold newly admits on rust-analyzer, 16 444 cross a crate boundary Cargo declares, 42 are same-crate, and 229 of the remaining 231 travel a real re-export (crates/ide-db/src/lib.rs:75, pub use base_db::{self, ..}). Two do not:

crates/ide/src/lib.rs:142:27            LineIndex (import)  -> crates/rust-analyzer/src/line_index.rs:18  #tier1q_pass1
crates/ide/src/view_syntax_tree.rs:4:27 LineIndex (import)  -> crates/rust-analyzer/src/line_index.rs:18  #tier1q_pass1

Both sites are use ide_db::{..., line_index::{LineCol, LineIndex}, ...}. The correct target is lib/line-index/src/lib.rs:85, re-exported by ide-db. crates/ide does not depend on crates/rust-analyzer — that edge runs the other way — so these are phantoms by the same argument #134 uses.

The mechanism, and it is not the fold

index::package_root_of returns the prefix before the first conventional source directory:

const SRC_DIRS: &[&str] = &["src", "lib", "tests", "test", "app", "spec", "specs", "source"];

lib is in that list. rust-analyzer keeps four real packages under a top-level lib/ (lib/line-index, lib/la-arena, lib/lsp-server, lib/ra-ap-rustc_*-style vendored crates), so for lib/line-index/src/lib.rs the loop matches lib at index 0 and returns "". pkg_tail is empty, temp.pkg_lang skips it (WHERE fp.pkg_tail != ''), and the package cannot anchor a qualifier.

What is left for ide_db::line_index::LineIndex is tier 1Q's rule (b) file-key anchor on q_last: the stem line_index names exactly one file in the whole tree — crates/rust-analyzer/src/line_index.rs. One candidate, so it resolves. This is #134's family verbatim (a basename stem carries no package), reached through the qualified path instead of the import path.

#165 did not create either half. Before the fold, q_first = ide_db was not a known root at all (ide-db is the directory, ide_db the identifier), so qual_known refused the qualifier outright and no anchor ran. The fold correctly admits the qualifier; the stem anchor then mis-aims it. Reverting the fold would hide these two by refusing 16 715 correct binds with them.

Bound

  • 2 phantoms out of 16 717 new binds = 0.012 %.
  • The undeclared-cross-crate rate over the FULL bind set improves: 3.383 % -> 2.534 % (739/21 845 -> 958/37 811, and 229 of the increase is the ide_db::base_db re-export).
  • Recorded in the tests/corpus/tier3-baseline.json bless reason and in index::fold_pkg_sep's doc, so it is not lost.

Directions (each needs a measured before/after, bind-for-bind)

  1. Drop lib from SRC_DIRS, or accept it only when it is not the FIRST component. lib/ as a package container (lib/line-index/Cargo.toml) and lib/ as a source dir (mypkg/lib/foo.rb) are different things, and only the second is what the list is for. Blast radius: every Ruby/PHP layout that uses lib/ as a source root — measurable on ruby-sinatra and php-guzzle.
  2. Prefer the manifest. manifest_package_dirs/nearest_package_dir already know lib/line-index holds a Cargo.toml; pkg_tail is still derived from the lexical package_root_of. Deriving the tail from pkg_dir when there IS a manifest would fix this class without touching the lexical fallback. This is the same "manifests do not guess" argument that made #134 compare pkg_dir rather than pkg_root.
  3. Refuse a tier-1Q anchor whose sole evidence is a file STEM (#134 direction 1, applied to 1Q). Costs recall wherever a stem is the only evidence.

Direction 2 looks like the one clause: it is structural, it changes no lexical behaviour where no manifest exists, and it makes pkg_tail and pkg_dir agree instead of disagreeing.

🤖 Generated with Claude Code

https://claude.ai/code/session_01K1zj5VcFJvJt3pQxe9259K

Found while bounding #165's phantom risk, with an oracle independent of the resolver: each ref crate's own `Cargo.toml`. ## The two binds Of the 16 717 binds #165's fold newly admits on rust-analyzer, **16 444 cross a crate boundary Cargo declares**, 42 are same-crate, and 229 of the remaining 231 travel a real re-export (`crates/ide-db/src/lib.rs:75`, `pub use base_db::{self, ..}`). **Two do not:** ``` crates/ide/src/lib.rs:142:27 LineIndex (import) -> crates/rust-analyzer/src/line_index.rs:18 #tier1q_pass1 crates/ide/src/view_syntax_tree.rs:4:27 LineIndex (import) -> crates/rust-analyzer/src/line_index.rs:18 #tier1q_pass1 ``` Both sites are `use ide_db::{..., line_index::{LineCol, LineIndex}, ...}`. The correct target is `lib/line-index/src/lib.rs:85`, re-exported by `ide-db`. `crates/ide` does not depend on `crates/rust-analyzer` — that edge runs the other way — so these are phantoms by the same argument #134 uses. ## The mechanism, and it is not the fold `index::package_root_of` returns the prefix before the first *conventional source directory*: ```rust const SRC_DIRS: &[&str] = &["src", "lib", "tests", "test", "app", "spec", "specs", "source"]; ``` `lib` is in that list. rust-analyzer keeps four real packages under a top-level `lib/` (`lib/line-index`, `lib/la-arena`, `lib/lsp-server`, `lib/ra-ap-rustc_*`-style vendored crates), so for `lib/line-index/src/lib.rs` the loop matches `lib` at index 0 and returns `""`. **`pkg_tail` is empty**, `temp.pkg_lang` skips it (`WHERE fp.pkg_tail != ''`), and the package cannot anchor a qualifier. What is left for `ide_db::line_index::LineIndex` is tier 1Q's rule (b) file-key anchor on `q_last`: the stem `line_index` names exactly one file in the whole tree — `crates/rust-analyzer/src/line_index.rs`. One candidate, so it resolves. This is #134's family verbatim (a basename stem carries no package), reached through the qualified path instead of the import path. **#165 did not create either half.** Before the fold, `q_first = ide_db` was not a known root at all (`ide-db` is the directory, `ide_db` the identifier), so `qual_known` refused the qualifier outright and no anchor ran. The fold correctly admits the qualifier; the stem anchor then mis-aims it. Reverting the fold would hide these two by refusing 16 715 correct binds with them. ## Bound - 2 phantoms out of 16 717 new binds = **0.012 %**. - The undeclared-cross-crate rate over the FULL bind set improves: **3.383 % -> 2.534 %** (739/21 845 -> 958/37 811, and 229 of the increase is the `ide_db::base_db` re-export). - Recorded in the `tests/corpus/tier3-baseline.json` bless reason and in `index::fold_pkg_sep`'s doc, so it is not lost. ## Directions (each needs a measured before/after, bind-for-bind) 1. **Drop `lib` from `SRC_DIRS`, or accept it only when it is not the FIRST component.** `lib/` as a *package container* (`lib/line-index/Cargo.toml`) and `lib/` as a *source dir* (`mypkg/lib/foo.rb`) are different things, and only the second is what the list is for. Blast radius: every Ruby/PHP layout that uses `lib/` as a source root — measurable on `ruby-sinatra` and `php-guzzle`. 2. **Prefer the manifest.** `manifest_package_dirs`/`nearest_package_dir` already know `lib/line-index` holds a `Cargo.toml`; `pkg_tail` is still derived from the lexical `package_root_of`. Deriving the tail from `pkg_dir` when there IS a manifest would fix this class without touching the lexical fallback. This is the same "manifests do not guess" argument that made #134 compare `pkg_dir` rather than `pkg_root`. 3. **Refuse a tier-1Q anchor whose sole evidence is a file STEM** (#134 direction 1, applied to 1Q). Costs recall wherever a stem is the only evidence. Direction 2 looks like the one clause: it is structural, it changes no lexical behaviour where no manifest exists, and it makes `pkg_tail` and `pkg_dir` agree instead of disagreeing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01K1zj5VcFJvJt3pQxe9259K
buildagent commented 2026-09-06 09:15:15 +02:00
Author
Member
Copy link

REFUSED: directions 1 and 2 both measured, both cost recall, and NEITHER fixes the two phantoms

Measured in /tmp/cosi-lane-precision off origin/master (f6a878a), against a private copy of the pinned corpus. Baseline reproduces the record exactly: rust-analyzer 143 113 resolved, and every other repo's figure matches baseline.json. Bind sets compared id-independently, ref path:line:col kind name -> target path#name@line by <rule>, sorted in the C locale.

Direction 2 — derive pkg_tail from pkg_dir when a manifest exists

The one the issue calls "the one clause".

repo before after lost gained net
rust-analyzer 143 113 141 869 1 620 376 −1 244
cs-dapper 3 744 3 720 24 0 −24
python-flask 3 026 3 019 7 0 −7
rust-ripgrep 15 709 15 708 1 0 −1
ruby-sinatra, ts-zod, php-guzzle, js-express, py-django — — 0 0 0

The two phantoms survive. They do not go away; they change arm:

before:  crates/ide/src/lib.rs:142:27  import LineIndex -> crates/rust-analyzer/src/line_index.rs#LineIndex@18  by 40   (tier1q_pass1)
after:   crates/ide/src/lib.rs:142:27  import LineIndex -> crates/rust-analyzer/src/line_index.rs#LineIndex@18  by 42   (tier1q_tiebreak)

(crates/ide/src/view_syntax_tree.rs:4:27 likewise. One genuine gain beside them: LineCol at 142:18 now binds lib/line-index/src/lib.rs#LineCol@14 correctly.)

Direction 1 — drop lib from SRC_DIRS when it is the FIRST component

repo before after lost gained net
rust-analyzer 143 113 141 962 1 519 368 −1 151
ruby-sinatra 2 850 2 735 117 2 −115 (−4.0 %)
php-guzzle 12 031 12 097 227 293 +66
rust-ripgrep 15 709 15 705 4 0 −4

The two phantoms survive here too, and again as rule 42 — tier1q_tiebreak.

The mechanism, read at source rather than inferred from the delta

Both directions do the same structural thing: they give lib/<pkg> a non-empty pkg_tail. An empty tail is currently an EXEMPTION from #134's origin gate (AND (fp.pkg_tail = '' OR …)). Filling it in subjects those files to the gate, and the gate then refuses every bind an import does not vouch for — which on rust-analyzer means the re-export binds.

Where the 1 620 rust-analyzer losses land, by target package: lib/text-size 1 071, lib/smol_str 263, crates/syntax 102, lib/line-index 84, lib/ungrammar 54, lib/lsp-server 21, lib/la-arena 14.

Checked against the oracle #168 itself used — each referencing crate's own Cargo.toml. crates/base-db/Cargo.toml declares syntax, la-arena, vfs, span, intern, cfg — and not text-size, not smol_str. Same for crates/hir-def. So crates/base-db/src/lib.rs:312:49 type TextSize -> lib/text-size/src/size.rs#TextSize@24 travels syntax's re-export, exactly like the 229 binds through crates/ide-db/src/lib.rs:75 that this issue's own bound counted as legitimate. These are the good kind of bind, and both directions withdraw ~1 300 of them to fix 2 phantoms.

That trade — 67 legitimate resolutions against 2 phantoms — is the one index::manifest_package_dirs' doc already records as "Measured, then rejected." This is the same trade at 20× the scale.

Direction 1 also does what the issue predicted for Ruby/PHP, and the Ruby number is the one that decides it: lib/sinatra/base.rb goes from root "" to root lib/sinatra, tail sinatra, and ruby-sinatra loses 115 of 2 850 binds (4.0 %). php-guzzle nets +66 but churns 520 rows in both directions, which is not a result to accept without reading every one.

Where that leaves the issue

Direction 3 is the only one that touches the mechanism, and both other directions demonstrate it by relocating the phantom into tier1q_tiebreak — which is the stem-anchored arm. The issue's own analysis said so ("the file-stem anchor is then the only candidate"); the measurement now says the package tail is not what mis-aims these two, and giving lib/line-index a tail changes which stem-anchored arm answers, not whether one does.

Not implemented here. Direction 3 costs recall wherever a stem is the only evidence, so it needs the same bind-for-bind before/after over the nine repos plus a reading of the losses, and it is a change inside tier 1Q rather than at the pkg_tail funnel.

The residual stands as filed

2 phantoms in 16 444 admitted cross-crate binds (0.012 %), bounded by an oracle outside the resolver, recorded in fold_pkg_sep's doc and in the tier3-baseline.json bless reason. Nothing in this measurement changes that bound; what it changes is which of the three directions can be expected to close it.

No source change from this lane on #168. The two experiments were applied, measured and reverted; crates/indexer/src/index.rs is md5-identical to its pre-experiment snapshot.

## REFUSED: directions 1 and 2 both measured, both cost recall, and NEITHER fixes the two phantoms Measured in `/tmp/cosi-lane-precision` off `origin/master` (`f6a878a`), against a private copy of the pinned corpus. Baseline reproduces the record exactly: rust-analyzer **143 113** resolved, and every other repo's figure matches `baseline.json`. Bind sets compared id-independently, `ref path:line:col kind name -> target path#name@line by <rule>`, sorted in the C locale. ### Direction 2 — derive `pkg_tail` from `pkg_dir` when a manifest exists The one the issue calls "the one clause". | repo | before | after | lost | gained | net | |---|---:|---:|---:|---:|---:| | rust-analyzer | 143 113 | 141 869 | 1 620 | 376 | **−1 244** | | cs-dapper | 3 744 | 3 720 | 24 | 0 | −24 | | python-flask | 3 026 | 3 019 | 7 | 0 | −7 | | rust-ripgrep | 15 709 | 15 708 | 1 | 0 | −1 | | ruby-sinatra, ts-zod, php-guzzle, js-express, py-django | — | — | 0 | 0 | 0 | **The two phantoms survive.** They do not go away; they change arm: ``` before: crates/ide/src/lib.rs:142:27 import LineIndex -> crates/rust-analyzer/src/line_index.rs#LineIndex@18 by 40 (tier1q_pass1) after: crates/ide/src/lib.rs:142:27 import LineIndex -> crates/rust-analyzer/src/line_index.rs#LineIndex@18 by 42 (tier1q_tiebreak) ``` (`crates/ide/src/view_syntax_tree.rs:4:27` likewise. One genuine gain beside them: `LineCol` at `142:18` now binds `lib/line-index/src/lib.rs#LineCol@14` correctly.) ### Direction 1 — drop `lib` from `SRC_DIRS` when it is the FIRST component | repo | before | after | lost | gained | net | |---|---:|---:|---:|---:|---:| | rust-analyzer | 143 113 | 141 962 | 1 519 | 368 | **−1 151** | | ruby-sinatra | 2 850 | 2 735 | 117 | 2 | **−115** (−4.0 %) | | php-guzzle | 12 031 | 12 097 | 227 | 293 | +66 | | rust-ripgrep | 15 709 | 15 705 | 4 | 0 | −4 | **The two phantoms survive here too, and again as rule 42** — `tier1q_tiebreak`. ### The mechanism, read at source rather than inferred from the delta Both directions do the same structural thing: they give `lib/<pkg>` a **non-empty `pkg_tail`**. An empty tail is currently an EXEMPTION from #134's origin gate (`AND (fp.pkg_tail = '' OR …)`). Filling it in subjects those files to the gate, and the gate then refuses every bind an import does not vouch for — which on rust-analyzer means the **re-export** binds. Where the 1 620 rust-analyzer losses land, by target package: `lib/text-size` 1 071, `lib/smol_str` 263, `crates/syntax` 102, `lib/line-index` 84, `lib/ungrammar` 54, `lib/lsp-server` 21, `lib/la-arena` 14. Checked against the oracle #168 itself used — each referencing crate's own `Cargo.toml`. `crates/base-db/Cargo.toml` declares `syntax`, `la-arena`, `vfs`, `span`, `intern`, `cfg` — and **not** `text-size`, **not** `smol_str`. Same for `crates/hir-def`. So `crates/base-db/src/lib.rs:312:49 type TextSize -> lib/text-size/src/size.rs#TextSize@24` travels `syntax`'s re-export, exactly like the 229 binds through `crates/ide-db/src/lib.rs:75` that this issue's own bound counted as legitimate. These are the good kind of bind, and both directions withdraw ~1 300 of them to fix 2 phantoms. That trade — 67 legitimate resolutions against 2 phantoms — is the one `index::manifest_package_dirs`' doc already records as *"Measured, then rejected."* This is the same trade at 20× the scale. Direction 1 also does what the issue predicted for Ruby/PHP, and the Ruby number is the one that decides it: `lib/sinatra/base.rb` goes from root `""` to root `lib/sinatra`, tail `sinatra`, and ruby-sinatra loses **115 of 2 850** binds (4.0 %). php-guzzle nets +66 but churns 520 rows in both directions, which is not a result to accept without reading every one. ### Where that leaves the issue **Direction 3 is the only one that touches the mechanism**, and both other directions demonstrate it by relocating the phantom **into** `tier1q_tiebreak` — which *is* the stem-anchored arm. The issue's own analysis said so ("the file-stem anchor is then the only candidate"); the measurement now says the package tail is not what mis-aims these two, and giving `lib/line-index` a tail changes which stem-anchored arm answers, not whether one does. Not implemented here. Direction 3 costs recall wherever a stem is the only evidence, so it needs the same bind-for-bind before/after over the nine repos plus a reading of the losses, and it is a change inside tier 1Q rather than at the `pkg_tail` funnel. ### The residual stands as filed 2 phantoms in 16 444 admitted cross-crate binds (0.012 %), bounded by an oracle outside the resolver, recorded in `fold_pkg_sep`'s doc and in the `tier3-baseline.json` bless reason. Nothing in this measurement changes that bound; what it changes is which of the three directions can be expected to close it. **No source change from this lane on #168.** The two experiments were applied, measured and reverted; `crates/indexer/src/index.rs` is md5-identical to its pre-experiment snapshot.
buildagent commented 2026-09-06 12:47:48 +02:00
Author
Member
Copy link

STAYING OPEN — the defect is live on master; two of three directions are now measured and refused

Close-out lane, master 552e3a2. Title corrected to name the mechanism the measurement actually found.

Why this is not a "refused, recorded" close

A refusal backed by numbers is a legitimate close when the refusal disposes of the defect. Here it does not: the two LineIndex phantoms still bind on master, and package_root_of's SRC_DIRS is unchanged —

crates/indexer/src/index.rs:2044
    const SRC_DIRS: &[&str] = &["src", "lib", "tests", "test", "app", "spec", "specs", "source"];

— so lib/line-index/src/lib.rs still yields pkg_tail == '' and still cannot anchor a qualifier. Nothing about the reported behaviour changed.

What the measurement did settle — the issue's own diagnosis was wrong

Both directions were applied, measured bind-for-bind against the nine pinned repos, and reverted (index.rs md5-identical to its pre-experiment snapshot). Neither fixes the two phantoms; both relocate them from rule 40 tier1q_pass1 to rule 42 tier1q_tiebreak — which is still a stem-anchored arm. And both cost real recall:

direction rust-analyzer net other repos
2 — derive pkg_tail from pkg_dir when a manifest exists −1 244 (1 620 lost / 376 gained) cs-dapper −24, python-flask −7, rust-ripgrep −1
1 — drop lib from SRC_DIRS when first −1 151 (1 519 / 368) ruby-sinatra −115 (−4.0 %), php-guzzle +66 on 520 churned rows

The mechanism, read at source rather than inferred from the delta: both directions give lib/<pkg> a non-empty pkg_tail, and an empty tail is currently an exemption from #134's origin gate (AND (fp.pkg_tail = '' OR …)). Filling it in subjects those files to the gate, which then refuses every bind an import does not vouch for — i.e. the re-export binds. The rust-analyzer losses land on lib/text-size (1 071), lib/smol_str (263), crates/syntax (102), lib/line-index (84). Checked against the same oracle this issue used — each referencing crate's own Cargo.toml — those travel syntax's re-export exactly as the 229 binds through crates/ide-db/src/lib.rs:75 do, which this issue's own bound already counted as legitimate.

That is the trade index::manifest_package_dirs' doc already records as "Measured, then rejected" — here at 20× the scale: ~1 300 correct resolutions for 2 phantoms.

Corrected scope — what remains

Direction 3 only: refuse a tier-1Q anchor whose sole evidence is a file STEM (#134 direction 1, applied to 1Q). It is the only direction that touches the mechanism, and the other two demonstrate that by relocating the phantom into the stem-anchored arm. It needs the same bind-for-bind before/after over the nine repos plus a reading of the losses, because it costs recall wherever a stem is the only evidence.

Directions 1 and 2 should not be re-attempted without new evidence; the numbers above are the record.

The residual stands as filed

2 phantoms in 16 444 admitted cross-crate binds (0.012 %), bounded by an oracle outside the resolver, recorded in index::fold_pkg_sep's doc and in the tests/corpus/tier3-baseline.json bless reason.

## STAYING OPEN — the defect is live on master; two of three directions are now measured and refused Close-out lane, master `552e3a2`. **Title corrected** to name the mechanism the measurement actually found. ### Why this is not a "refused, recorded" close A refusal backed by numbers is a legitimate close when the refusal disposes of the *defect*. Here it does not: the two `LineIndex` phantoms **still bind on master**, and `package_root_of`'s `SRC_DIRS` is unchanged — ``` crates/indexer/src/index.rs:2044 const SRC_DIRS: &[&str] = &["src", "lib", "tests", "test", "app", "spec", "specs", "source"]; ``` — so `lib/line-index/src/lib.rs` still yields `pkg_tail == ''` and still cannot anchor a qualifier. Nothing about the reported behaviour changed. ### What the measurement did settle — the issue's own diagnosis was wrong Both directions were applied, measured bind-for-bind against the nine pinned repos, and reverted (`index.rs` md5-identical to its pre-experiment snapshot). **Neither fixes the two phantoms; both relocate them from rule 40 `tier1q_pass1` to rule 42 `tier1q_tiebreak`** — which is still a stem-anchored arm. And both cost real recall: | direction | rust-analyzer net | other repos | |---|---:|---| | 2 — derive `pkg_tail` from `pkg_dir` when a manifest exists | **−1 244** (1 620 lost / 376 gained) | cs-dapper −24, python-flask −7, rust-ripgrep −1 | | 1 — drop `lib` from `SRC_DIRS` when first | **−1 151** (1 519 / 368) | **ruby-sinatra −115 (−4.0 %)**, php-guzzle +66 on 520 churned rows | The mechanism, read at source rather than inferred from the delta: both directions give `lib/<pkg>` a non-empty `pkg_tail`, and an empty tail is currently an **exemption** from #134's origin gate (`AND (fp.pkg_tail = '' OR …)`). Filling it in subjects those files to the gate, which then refuses every bind an import does not vouch for — i.e. the re-export binds. The rust-analyzer losses land on `lib/text-size` (1 071), `lib/smol_str` (263), `crates/syntax` (102), `lib/line-index` (84). Checked against the same oracle this issue used — each referencing crate's own `Cargo.toml` — those travel `syntax`'s re-export exactly as the 229 binds through `crates/ide-db/src/lib.rs:75` do, which this issue's own bound already counted as legitimate. That is the trade `index::manifest_package_dirs`' doc already records as *"Measured, then rejected"* — here at 20× the scale: ~1 300 correct resolutions for 2 phantoms. ### Corrected scope — what remains **Direction 3 only**: refuse a tier-1Q anchor whose sole evidence is a file STEM (#134 direction 1, applied to 1Q). It is the only direction that touches the mechanism, and the other two demonstrate that by relocating the phantom *into* the stem-anchored arm. It needs the same bind-for-bind before/after over the nine repos plus a reading of the losses, because it costs recall wherever a stem is the only evidence. Directions 1 and 2 should not be re-attempted without new evidence; the numbers above are the record. ### The residual stands as filed 2 phantoms in 16 444 admitted cross-crate binds (**0.012 %**), bounded by an oracle outside the resolver, recorded in `index::fold_pkg_sep`'s doc and in the `tests/corpus/tier3-baseline.json` bless reason.
buildagent changed title from package_root_of counts a top-level lib/ as a source dir, so lib/line-index has an empty package tail and a stem anchor binds the wrong LineIndex to Two LineIndex phantoms survive on rust-analyzer because tier 1Q anchors on a file STEM — directions 1 and 2 (package_root_of/lib) measured and REFUSED, direction 3 is what remains 2026-09-06 12:47:49 +02:00
buildagent commented 2026-09-06 17:35:05 +02:00
Author
Member
Copy link

DIRECTION 3 MEASURED AND REFUSED — it costs 5 840 correct binds to remove the two phantoms, and its narrow form does not remove them at all

Resolver-recall lane, worktree off master fc329a8. All three directions are now measured. Two experiments, both applied, measured bind-for-bind over the nine pinned repos, and reverted — crates/indexer/src/index.rs is md5-identical to its pre-experiment snapshot (43634ff127df8fad4cea1f9a09bb99d0).

How direction 3 was made measurable

temp.file_keys is the union of module symbol names and file BASENAME STEMS, inserted by two different statements, so "the anchor's sole evidence is a stem" is decidable: add from_stem to that table and gate the tier-1Q arms on it. The stem tie-break (temp.qual_stem_files, rule 42) is a stem anchor by construction and has no other evidence at all.

Experiment 3a — refuse the stem in tier 1Q's rule-(b) q_last file-key anchor

The arm that produced the two filed binds.

repo lost new net
rust-analyzer 727 743 +16
py-django 4 459 4 467 +8
python-flask 71 71 0
rust-ripgrep 64 64 0
ts-zod, js-express 8 / 3 8 / 3 0

Nearly a no-op on recall — and the two phantoms SURVIVE:

crates/ide/src/lib.rs:142:27            import LineIndex -> crates/rust-analyzer/src/line_index.rs#LineIndex@18  by 42
crates/ide/src/view_syntax_tree.rs:4:27 import LineIndex -> crates/rust-analyzer/src/line_index.rs#LineIndex@18  by 42

Rule 40 -> 42, exactly as directions 1 and 2 did. rust-analyzer's movement is 647 binds relocating 40 -> 42 with identical targets. The tie-break arm catches everything the pass-1 anchor drops, because it anchors on the same stem.

That is the third independent demonstration of one fact: the phantom does not belong to any single arm. Every direction so far has moved it between arms that share the stem.

Experiment 3b — the COMPLETE form: no stem-only anchor anywhere in tier 1Q

Rule-(b) q_last gated on from_stem = 0 and qual_stem_files emptied, so rule 42 cannot fire.

The two phantoms are gone. And so is this:

repo lost new net
py-django 4 599 89 −4 510
rust-analyzer 1 078 103 −975
rust-ripgrep 71 1 −70
python-flask 71 0 −71
ts-zod 16 0 −16
js-express 5 0 −5
total 5 840 193 −5 647

Roughly 2 800 correct binds per phantom, against ~650 for direction 2 and ~575 for direction 1.

The losses read at source, and they are the ordinary case

django/__init__.py:1:34        import get_version  -> django/utils/version.py#get_version@24        by 40
django/apps/config.py:5:36     import ImproperlyConfigured -> django/core/exceptions.py#…@123       by 40
crates/hir-def/src/expr_store.rs:31:84  import Path -> crates/hir-def/src/expr_store/path.rs#Path@16 by 40

The Cargo oracle over rust-analyzer's 1 078 losses: 896 same-crate, 166 cross-crate, 46 undeclared (ide-assists -> hir-ty, travelling hir's re-export). These are from django.utils.version import get_version and use crate::expr_store::path::Path — the bread-and-butter qualified import.

Why direction 3 is MIS-FRAMED, and this is the finding

The issue treats "the anchor's sole evidence is a file STEM" as weak evidence. In a module-per-file language it is not evidence at all — it is the language's own naming rule. django.utils.version's last segment IS the file stem of django/utils/version.py, by Python's import semantics; crate::expr_store::path IS expr_store/path.rs, by Rust's. Refusing the stem anchor does not remove a heuristic, it removes the primary mechanism by which a qualified import resolves in two of the seven languages — which is why py-django alone loses 4 510 binds.

The LineIndex case is not "a stem was trusted". It is a stem was trusted GLOBALLY: the key line_index was matched against every file in the tree rather than against files the qualifier's own root could reach. crates/ide reaching crates/rust-analyzer/src/line_index.rs is wrong because ide_db does not name that crate — not because a stem was involved.

So the direction worth measuring next is not a refusal but a scoping: require the stem-anchored file to be reachable from the qualifier's root (q_first's package/root), instead of accepting any file in the project whose basename matches. That is the shape #134's origin gate already has for tier 3, applied to tier 1Q's anchor, and it would leave django.utils.version untouched (the stem sits under the django root the qualifier names) while refusing ide_db::line_index::LineIndex -> crates/rust-analyzer/….

Where the issue stands

direction verdict cost fixes the 2?
1 — drop lib from SRC_DIRS refused −1 151 ra, −115 ruby-sinatra no (relocates to rule 42)
2 — derive pkg_tail from pkg_dir refused −1 244 ra no (relocates to rule 42)
3a — refuse the stem in rule (b) q_last refused ~0 no (relocates to rule 42)
3b — refuse every stem-only tier-1Q anchor refused −5 647 yes

All three directions as filed are now closed with numbers. The residual stands exactly as recorded: 2 phantoms in 16 444 admitted cross-crate binds (0.012 %), bounded by an oracle outside the resolver, written into index::fold_pkg_sep's doc and the tests/corpus/tier3-baseline.json bless reason. What should change in this issue is the DIRECTIONS list — replace it with the root-scoped anchor above, which is the only remaining shape that has not been shown to trade thousands of correct binds for two.

Note on the numbers' baseline

These experiments were run on top of this lane's #69 change (tier 1R serving the member pool), so the absolute totals differ from the earlier comment's. The deltas are all measured against that same build, and #69 touches tier 1R only — no tier-1Q arm — so the tier-1Q conclusions are unaffected. The two LineIndex binds are byte-identical in both baselines.

No source change from this lane on #168.

## DIRECTION 3 MEASURED AND REFUSED — it costs **5 840 correct binds** to remove the two phantoms, and its narrow form does not remove them at all Resolver-recall lane, worktree off master `fc329a8`. All three directions are now measured. Two experiments, both applied, measured bind-for-bind over the nine pinned repos, and reverted — `crates/indexer/src/index.rs` is md5-identical to its pre-experiment snapshot (`43634ff127df8fad4cea1f9a09bb99d0`). ### How direction 3 was made measurable `temp.file_keys` is the union of `module` symbol names and file BASENAME STEMS, inserted by two different statements, so "the anchor's sole evidence is a stem" is decidable: add `from_stem` to that table and gate the tier-1Q arms on it. The stem tie-break (`temp.qual_stem_files`, rule 42) is a stem anchor by construction and has no other evidence at all. ### Experiment 3a — refuse the stem in tier 1Q's rule-(b) `q_last` file-key anchor The arm that produced the two filed binds. | repo | lost | new | net | |---|---:|---:|---:| | rust-analyzer | 727 | 743 | +16 | | py-django | 4 459 | 4 467 | +8 | | python-flask | 71 | 71 | 0 | | rust-ripgrep | 64 | 64 | 0 | | ts-zod, js-express | 8 / 3 | 8 / 3 | 0 | Nearly a no-op on recall — **and the two phantoms SURVIVE**: ``` crates/ide/src/lib.rs:142:27 import LineIndex -> crates/rust-analyzer/src/line_index.rs#LineIndex@18 by 42 crates/ide/src/view_syntax_tree.rs:4:27 import LineIndex -> crates/rust-analyzer/src/line_index.rs#LineIndex@18 by 42 ``` Rule **40 -> 42**, exactly as directions 1 and 2 did. rust-analyzer's movement is 647 binds relocating 40 -> 42 with identical targets. The tie-break arm catches everything the pass-1 anchor drops, because it anchors on the same stem. That is the third independent demonstration of one fact: **the phantom does not belong to any single arm.** Every direction so far has moved it between arms that share the stem. ### Experiment 3b — the COMPLETE form: no stem-only anchor anywhere in tier 1Q Rule-(b) `q_last` gated on `from_stem = 0` **and** `qual_stem_files` emptied, so rule 42 cannot fire. **The two phantoms are gone.** And so is this: | repo | lost | new | net | |---|---:|---:|---:| | py-django | 4 599 | 89 | **−4 510** | | rust-analyzer | 1 078 | 103 | **−975** | | rust-ripgrep | 71 | 1 | −70 | | python-flask | 71 | 0 | −71 | | ts-zod | 16 | 0 | −16 | | js-express | 5 | 0 | −5 | | **total** | **5 840** | **193** | **−5 647** | Roughly **2 800 correct binds per phantom**, against ~650 for direction 2 and ~575 for direction 1. ### The losses read at source, and they are the ordinary case ``` django/__init__.py:1:34 import get_version -> django/utils/version.py#get_version@24 by 40 django/apps/config.py:5:36 import ImproperlyConfigured -> django/core/exceptions.py#…@123 by 40 crates/hir-def/src/expr_store.rs:31:84 import Path -> crates/hir-def/src/expr_store/path.rs#Path@16 by 40 ``` The Cargo oracle over rust-analyzer's 1 078 losses: 896 same-crate, 166 cross-crate, 46 undeclared (`ide-assists -> hir-ty`, travelling `hir`'s re-export). These are `from django.utils.version import get_version` and `use crate::expr_store::path::Path` — the bread-and-butter qualified import. ### Why direction 3 is MIS-FRAMED, and this is the finding The issue treats "the anchor's sole evidence is a file STEM" as weak evidence. In a module-per-file language it is not evidence at all — **it is the language's own naming rule.** `django.utils.version`'s last segment IS the file stem of `django/utils/version.py`, by Python's import semantics; `crate::expr_store::path` IS `expr_store/path.rs`, by Rust's. Refusing the stem anchor does not remove a heuristic, it removes the primary mechanism by which a qualified import resolves in two of the seven languages — which is why py-django alone loses 4 510 binds. The `LineIndex` case is not "a stem was trusted". It is **a stem was trusted GLOBALLY**: the key `line_index` was matched against every file in the tree rather than against files the qualifier's own root could reach. `crates/ide` reaching `crates/rust-analyzer/src/line_index.rs` is wrong because `ide_db` does not name that crate — not because a stem was involved. So the direction worth measuring next is not a refusal but a **scoping**: require the stem-anchored file to be reachable from the qualifier's root (`q_first`'s package/root), instead of accepting any file in the project whose basename matches. That is the shape #134's origin gate already has for tier 3, applied to tier 1Q's anchor, and it would leave `django.utils.version` untouched (the stem sits under the `django` root the qualifier names) while refusing `ide_db::line_index::LineIndex -> crates/rust-analyzer/…`. ### Where the issue stands | direction | verdict | cost | fixes the 2? | |---|---|---:|---| | 1 — drop `lib` from `SRC_DIRS` | refused | −1 151 ra, −115 ruby-sinatra | **no** (relocates to rule 42) | | 2 — derive `pkg_tail` from `pkg_dir` | refused | −1 244 ra | **no** (relocates to rule 42) | | 3a — refuse the stem in rule (b) `q_last` | refused | ~0 | **no** (relocates to rule 42) | | 3b — refuse every stem-only tier-1Q anchor | refused | **−5 647** | yes | **All three directions as filed are now closed with numbers.** The residual stands exactly as recorded: 2 phantoms in 16 444 admitted cross-crate binds (0.012 %), bounded by an oracle outside the resolver, written into `index::fold_pkg_sep`'s doc and the `tests/corpus/tier3-baseline.json` bless reason. What should change in this issue is the DIRECTIONS list — replace it with the root-scoped anchor above, which is the only remaining shape that has not been shown to trade thousands of correct binds for two. ### Note on the numbers' baseline These experiments were run on top of this lane's #69 change (tier 1R serving the member pool), so the absolute totals differ from the earlier comment's. The deltas are all measured against that same build, and #69 touches tier 1R only — no tier-1Q arm — so the tier-1Q conclusions are unaffected. The two `LineIndex` binds are byte-identical in both baselines. **No source change from this lane on #168.**
buildagent commented 2026-09-10 02:24:19 +02:00
Author
Member
Copy link

Fixed on master at 6f8cddf (merged bb214f1). And this issue undercounts the family by more than 3×.

Two corrections to the measurement

"Two LineIndex phantoms" is three. crates/rust-analyzer/src/cli/parse.rs:3:25 binds the same wrong target and predates #165's fold, so this issue only ever counted the new binds.

The family is 10, not 3. Seven unfiled hir_def::layout::{TagEncoding,Variants} → crates/hir-ty/src/layout.rs binds of the identical shape — two of them self-referential, where layout.rs:76:41's pub type TagEncoding = hir_def::layout::TagEncoding<…> bound the alias on its own line to itself. crates/hir-def does not depend on hir-ty, checked in its Cargo.toml.

So the "0.012 % of 16,444" bound was computed on the wrong denominator.

The proposed fix is right in direction and insufficient as stated

index::qual_root_scope_sql is one predicate spliced into both stem-anchored tier-1Q arms (the q_last file-key arm and the qual_stem_files tie-break). A stem-anchored candidate is admitted only if the qualifier's first segment names no package, or the candidate has no package root, or it sits under a root that segment names, or it sits in a package that root's own files import.

Two findings this issue does not contain:

1. Containment alone withdraws 35 CORRECT binds. 27 of them are hir::db::HirDatabase reaching crates/hir-ty/src/db.rs across a one-line pub use at crates/hir/src/db.rs:7. The re-export arm (temp.root_pkg_reach, one hop from the root's own imports) is what separates those from the phantoms: crates/ide-db imports nothing named rust_analyzer; crates/hir-def imports nothing named hir_ty. This issue's "root-scoped anchor" has no such arm.

2. A root may not be read off qseg_pkg's SUFFIX relation. view_hir ends with the tail hir; rooting it at crates/hir refused view_hir::view_hir() inside crates/ide. Fixed with LENGTH(sp.pkg_tail) = LENGTH(sp.seg). That hazard is mentioned nowhere in the issue.

Bind census, read at source, against a de98a4f binary

The prior binary reproduces baseline.json exactly on all 7 pinned repos and reproduces both filed phantoms at rule 40 — the harness's own control.

#168 alone: six repos bind-for-bind identical. rust-analyzer −10 / +20:

  • All 10 losses are phantoms — the 3 LineIndex (including the unfiled cli/parse.rs) and the 7 hir_def::layout family.
  • All 20 gains verified correct at source — 10 × hir_expand::proc_macro::ProcMacroKind (decoy hir-def/src/nameres/proc_macro.rs, same stem), 4 × ide_db::rename::RenameConfig (decoy ide/src/rename.rs:30, including two sites inside that very file that literally write ide_db::rename::RenameConfig), 6 × hir::term_search::term_search.

Mutations

  • I1 (ungate the q_last arm) → RED on the REFUSAL, target Some("crates/scope-user/src/gadget.rs") by rule 40 — the filed rule.
  • I2 (ungate only qual_stem_files) → RED, same target by rule 42, reproducing the relocation each of this issue's three prior experiments hit.
  • I3 (delete root_pkg_reach) → RED: "the re-export arm is not optional…"
  • I5 (drop the length equality) → RED: "view_gauge merely ENDS WITH the tail gauge…"
  • I4 SURVIVED first. Deleting the file_root_anc disjunct fired nothing, because anchor rule (c) reached the file anyway via a shared package tail. Reported as a survivor, then a third test (a_stem_anchor_inside_a_nested_package_of_the_qualifiers_root_is_admitted, the django.utils.version shape) was written that the same mutation takes RED.

precision_gate re-run by me post-merge: 7/7, phantom_count == 0, recall = 1.000, with the denominator GROWN — rust forbid_sites 6 → 7, python 7 → 8, probes 13 → 16 and 14 → 16.

Three of this repo's own gates caught real defects in the work: doc_citation_gate (a cited test that was never written), generation_build's ungated-imports census (root_pkg_reach read imports without a generation gate — a pending generation could have let a package under construction vouch for a root), and pool_capability_registry (three new temp relations with no declared stance).

Closing. The absolute-module stem family this fixes is the half #203 explicitly left open; the remaining sibling is #246.

Fixed on `master` at `6f8cddf` (merged `bb214f1`). **And this issue undercounts the family by more than 3×.** ## Two corrections to the measurement **"Two `LineIndex` phantoms" is three.** `crates/rust-analyzer/src/cli/parse.rs:3:25` binds the same wrong target and **predates #165's fold**, so this issue only ever counted the *new* binds. **The family is 10, not 3.** Seven unfiled `hir_def::layout::{TagEncoding,Variants} → crates/hir-ty/src/layout.rs` binds of the identical shape — **two of them self-referential**, where `layout.rs:76:41`'s `pub type TagEncoding = hir_def::layout::TagEncoding<…>` bound the alias on its own line to itself. `crates/hir-def` does not depend on `hir-ty`, checked in its `Cargo.toml`. So the "0.012 % of 16,444" bound was computed on the wrong denominator. ## The proposed fix is right in direction and insufficient as stated `index::qual_root_scope_sql` is one predicate spliced into **both** stem-anchored tier-1Q arms (the `q_last` file-key arm and the `qual_stem_files` tie-break). A stem-anchored candidate is admitted only if the qualifier's first segment names no package, **or** the candidate has no package root, **or** it sits under a root that segment names, **or** it sits in a package that root's own files import. Two findings this issue does not contain: **1. Containment alone withdraws 35 CORRECT binds.** 27 of them are `hir::db::HirDatabase` reaching `crates/hir-ty/src/db.rs` across a one-line `pub use` at `crates/hir/src/db.rs:7`. The **re-export arm** (`temp.root_pkg_reach`, one hop from the root's own imports) is what separates those from the phantoms: `crates/ide-db` imports nothing named `rust_analyzer`; `crates/hir-def` imports nothing named `hir_ty`. This issue's "root-scoped anchor" has no such arm. **2. A root may not be read off `qseg_pkg`'s SUFFIX relation.** `view_hir` *ends with* the tail `hir`; rooting it at `crates/hir` refused `view_hir::view_hir()` inside `crates/ide`. Fixed with `LENGTH(sp.pkg_tail) = LENGTH(sp.seg)`. That hazard is mentioned nowhere in the issue. ## Bind census, read at source, against a `de98a4f` binary The prior binary reproduces `baseline.json` **exactly** on all 7 pinned repos and reproduces both filed phantoms at rule 40 — the harness's own control. **#168 alone: six repos bind-for-bind identical.** rust-analyzer **−10 / +20**: - **All 10 losses are phantoms** — the 3 `LineIndex` (including the unfiled `cli/parse.rs`) and the 7 `hir_def::layout` family. - **All 20 gains verified correct at source** — 10 × `hir_expand::proc_macro::ProcMacroKind` (decoy `hir-def/src/nameres/proc_macro.rs`, same stem), 4 × `ide_db::rename::RenameConfig` (decoy `ide/src/rename.rs:30`, **including two sites inside that very file that literally write `ide_db::rename::RenameConfig`**), 6 × `hir::term_search::term_search`. ## Mutations - **I1** (ungate the `q_last` arm) → **RED on the REFUSAL**, target `Some("crates/scope-user/src/gadget.rs")` **by rule 40** — the filed rule. - **I2** (ungate only `qual_stem_files`) → **RED**, same target **by rule 42**, reproducing the relocation each of this issue's three prior experiments hit. - **I3** (delete `root_pkg_reach`) → **RED**: *"the re-export arm is not optional…"* - **I5** (drop the length equality) → **RED**: *"`view_gauge` merely ENDS WITH the tail `gauge`…"* - **I4 SURVIVED first.** Deleting the `file_root_anc` disjunct fired nothing, because anchor rule (c) reached the file anyway via a shared package tail. Reported as a survivor, then a third test (`a_stem_anchor_inside_a_nested_package_of_the_qualifiers_root_is_admitted`, the `django.utils.version` shape) was written that the **same mutation takes RED**. `precision_gate` re-run by me post-merge: **7/7, `phantom_count == 0`, `recall = 1.000`**, with the denominator GROWN — rust `forbid_sites` 6 → 7, python 7 → 8, probes 13 → 16 and 14 → 16. Three of this repo's own gates caught real defects in the work: `doc_citation_gate` (a cited test that was never written), `generation_build`'s ungated-`imports` census (**`root_pkg_reach` read `imports` without a generation gate — a pending generation could have let a package under construction vouch for a root**), and `pool_capability_registry` (three new temp relations with no declared stance). Closing. The absolute-module stem family this fixes is the half #203 explicitly left open; the remaining sibling is #246.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
p1-integration
p0-integration
release/v0.32.2
release/v0.32.1
epic-i072-release-032
windows-installer
i269b-registry-truth
probe-win-baseline
fix/246-shadowed-imports
release/0.28.1
fix/review-followthrough-20260911
iwin-msvc-archive
v0.32.2
v0.32.1
v0.32.0
v0.31.2
v0.31.1
v0.31.0
v0.30.1
v0.30.0
v0.29.0
v0.28.3
v0.28.2
v0.28.1
v0.28.0
v0.27.1
v0.27.0
v0.26.1
v0.26.0
v0.25.0
v0.24.1
v0.24.0
v0.23.1
v0.23.0
v0.22.3
v0.22.2
v0.22.1
v0.22.0
v0.21.0
v0.20.3
v0.20.2
v0.20.1
v0.20.0
v0.19.5
v0.19.4
v0.19.3
v0.19.2
v0.19.1
v0.19.0
v0.18.1
v0.18.0
v0.17.0
v0.16.0
v0.15.3
v0.14.0
v0.13.3
v0.13.2
v0.13.1
v0.13.0
v0.12.0
v0.11.7
v0.11.6
v0.11.5
v0.11.4
v0.11.3
v0.11.2
v0.11.1
v0.11.0
v0.10.2
v0.10.1
v0.10.0
v0.9.0
v0.8.7
v0.8.6
v0.8.5
v0.8.4
v0.8.3
v0.8.2
v0.8.1
v0.8.0
v0.7.1
v0.7.0
v0.6.1
v0.6.0
v0.5.21
v0.5.20
v0.5.19
v0.5.18
v0.5.17
v0.5.16
v0.5.15
v0.5.14
v0.5.13
v0.5.12
v0.5.11
v0.5.10
v0.5.9
v0.5.8
v0.5.7
v0.5.6
v0.5.5
v0.5.4
v0.5.3
v0.5.2
v0.5.1
v0.5.0
v0.4.2
v0.4.1
v0.4.0
v0.3.3
v0.3.2
v0.3.1
v0.3.0
v0.2.0
Labels
Clear labels   
code-review

   
correctness

   
dos

   
performance

   
security

   
severity/high

   
severity/low

   
severity/medium

   
tech-debt
  
Kind/Breaking
Breaking change that won't be backward compatible

  
Kind/Bug
Something is not working

  
Kind/Documentation
Documentation changes

  
Kind/Enhancement
Improve existing functionality

  
Kind/Feature
New functionality

  
Kind/Security
This is security issue

  
Kind/Testing
Issue or pull request related to testing

  
Priority
Critical
 The priority is critical

  
Priority
High
 The priority is high

  
Priority
Low
 The priority is low

  
Priority
Medium
 The priority is medium

  
Reviewed
Confirmed
 Issue has been confirmed

  
Reviewed
Duplicate
 This issue or pull request already exists

  
Reviewed
Invalid
 Invalid issue

  
Reviewed
Won't Fix
 This issue won't be fixed

  
Status
Abandoned
 Somebody has started to work on this but abandoned work

  
Status
Blocked
 Something is blocking this issue or pull request

  
Status
Need More Info
 Feedback is required to reproduce issue or to continue work

No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#168
No description provided.