h-dv/code-index

Watch

Fork 0

`Visibility::Unknown` has no wire slot, and it costs a packaged JavaScript 19% of ALL its resolutions — MEASURED #166

New issue

Closed

opened 2026-09-06 00:53:27 +02:00 by buildagent · 2 comments

buildagent commented

2026-09-06 00:53:27 +02:00

Member

Split out of #86 as its own comment asked: "if it is near zero the gap is theoretical and TypeScript can port; if it is not, it is a blocker of the same kind as is_extension is for C#, and it should be split out rather than left as a paragraph here."

It is not near zero. The measurement that comment specified has now been run.

The gap

core::raw::Visibility::Unknown is a THIRD state — "I looked and could not tell" — and the fact ABI has only "exported" and the four it enumerates. code_index_abi::VISIBILITIES omits unknown on purpose: registry.rs states the reason at length, "'not extracted, so treat as widely visible' is precisely the authority a third party may not assert", and a_packaged_extractor_cannot_claim_unknown_visibility grades it. DEFAULT_VISIBILITY is VisibilityId(2) = file.

So the asymmetry core::raw::Visibility's own note names is real and it spans the whole lattice: Unknown pools as Exported for a compiled plugin and as File for a packaged one.

typescript.rs's CommonJS arm returns Unknown for a file whose exports the plugin could not detect, and its comment says exactly what must not happen to it:

A file that exports NOTHING detectable keeps the old conservative Unknown — it may be a global script or use a re-export idiom we do not model, and mis-marking those File would silently drop real cross-file resolutions.

A packaged TypeScript takes precisely that loss. The mapping is not a bug — it is the correct least-authority default — but the wire cannot express the state.

The population

Two pinned corpora, indexed with the shipped binary:

corpus	`unknown` symbols	of total	refs resolving TO them	of those, CROSS-FILE
`js-express` (CommonJS)	1,860	97.0% of 1,917	4,077	833
`ts-zod`	4,387	45.4% of 9,667	3,373	130

js-express is nearly all of it, which is the shape you would predict: the arm fires on CommonJS files with no detectable module.exports assignment, and that is most of express.

The cost, measured causally rather than bounded

The table above is an upper bound. To get the actual number, typescript.rs's Unknown arm was changed to return Visibility::File — the exact mapping facts_to_extract applies to a packaged extractor — and both corpora were re-indexed from scratch with the same binary otherwise:

corpus	shipped	simulated packaged	delta
`js-express`	4,153 resolved	3,348	−805 (−19.4%)
`ts-zod`	10,781 resolved	10,828	+47 (+0.4%)

A packaged JavaScript loses one resolution in five. The 805 tracks the 833 cross-file upper bound closely, which is what says the mechanism is the one named: a file-visible symbol is in no cross-file candidate pool.

The ts-zod +47 is worth stating rather than rounding away, and it is not a win: demoting candidates NARROWS pools, so some refs that were ambiguous now resolve uniquely. A gain earned by hiding candidates is a precision question, not a recall one, and it was not inspected bind-for-bind here.

Why this is a port blocker and not a tuning knob

Same shape as is_extension for C#, and #86 says so: no manifest and no host change can close it. The host's unknown → file is the least-authority default and must stay; what is missing is a way for a producer to say "I could not tell", which is a capability question — believing it means admitting the symbol to cross-file pools on the producer's word.

php, python, rust and ruby are unaffected (they emit no Unknown). typescript and javascript are blocked, and for javascript the number is large.

Method, so it can be re-run

# population
sqlite3 <db> "SELECT f.lang, s.visibility, COUNT(*) FROM symbols s
              JOIN files f ON f.id = s.file_id
              WHERE f.lang IN ('javascript','typescript') GROUP BY 1,2"
sqlite3 <db> "SELECT COUNT(*) FROM refs r JOIN symbols s ON s.id = r.target_id
              WHERE s.visibility = 'unknown' AND r.file_id <> s.file_id"

# cost: crates/plugins/src/typescript.rs, the CommonJS arm's
# `return Visibility::Unknown` -> `return Visibility::File`, rebuild, reindex
sqlite3 <db> "SELECT COUNT(*) FROM refs WHERE target_id IS NOT NULL"

Both corpora are $COSI_CORPUS_DIR/js-express and $COSI_CORPUS_DIR/ts-zod. The plugin edit was reverted and md5-verified; nothing in this measurement is checked in.

#86 (where this was a paragraph), #77 (the capability decision this needs), #75.

🤖 Generated with Claude Code

https://claude.ai/code/session_01K1zj5VcFJvJt3pQxe9259K

Split out of #86 as its own comment asked: *"if it is near zero the gap is theoretical and TypeScript can port; if it is not, it is a blocker of the same kind as `is_extension` is for C#, and it should be split out rather than left as a paragraph here."* **It is not near zero.** The measurement that comment specified has now been run. ## The gap `core::raw::Visibility::Unknown` is a THIRD state — "I looked and could not tell" — and the fact ABI has only "exported" and the four it enumerates. `code_index_abi::VISIBILITIES` omits `unknown` **on purpose**: `registry.rs` states the reason at length, *"'not extracted, so treat as widely visible' is precisely the authority a third party may not assert"*, and `a_packaged_extractor_cannot_claim_unknown_visibility` grades it. `DEFAULT_VISIBILITY` is `VisibilityId(2)` = `file`. So the asymmetry `core::raw::Visibility`'s own note names is real and it spans the whole lattice: **`Unknown` pools as `Exported` for a compiled plugin and as `File` for a packaged one.** `typescript.rs`'s CommonJS arm returns `Unknown` for a file whose exports the plugin could not detect, and its comment says exactly what must not happen to it: > A file that exports NOTHING detectable keeps the old conservative Unknown — it may be a global script or use a re-export idiom we do not model, and **mis-marking those File would silently drop real cross-file resolutions**. A packaged TypeScript takes precisely that loss. The mapping is not a bug — it is the correct least-authority default — but the wire cannot express the state. ## The population Two pinned corpora, indexed with the shipped binary: | corpus | `unknown` symbols | of total | refs resolving TO them | of those, CROSS-FILE | |---|---:|---:|---:|---:| | `js-express` (CommonJS) | **1,860** | 97.0% of 1,917 | 4,077 | **833** | | `ts-zod` | **4,387** | 45.4% of 9,667 | 3,373 | **130** | `js-express` is nearly all of it, which is the shape you would predict: the arm fires on CommonJS files with no detectable `module.exports` assignment, and that is most of express. ## The cost, measured causally rather than bounded The table above is an upper bound. To get the actual number, `typescript.rs`'s `Unknown` arm was changed to return `Visibility::File` — the exact mapping `facts_to_extract` applies to a packaged extractor — and both corpora were re-indexed from scratch with the same binary otherwise: | corpus | shipped | simulated packaged | delta | |---|---:|---:|---:| | `js-express` | 4,153 resolved | **3,348** | **−805 (−19.4%)** | | `ts-zod` | 10,781 resolved | **10,828** | **+47 (+0.4%)** | **A packaged JavaScript loses one resolution in five.** The 805 tracks the 833 cross-file upper bound closely, which is what says the mechanism is the one named: a `file`-visible symbol is in no cross-file candidate pool. The `ts-zod` `+47` is worth stating rather than rounding away, and it is not a win: demoting candidates NARROWS pools, so some refs that were ambiguous now resolve uniquely. A gain earned by hiding candidates is a precision question, not a recall one, and it was not inspected bind-for-bind here. ## Why this is a port blocker and not a tuning knob Same shape as `is_extension` for C#, and #86 says so: **no manifest and no host change can close it.** The host's `unknown → file` is the least-authority default and must stay; what is missing is a way for a producer to say "I could not tell", which is a capability question — believing it means admitting the symbol to cross-file pools on the producer's word. `php`, `python`, `rust` and `ruby` are unaffected (they emit no `Unknown`). **typescript and javascript are blocked**, and for javascript the number is large. ## Method, so it can be re-run ``` # population sqlite3 <db> "SELECT f.lang, s.visibility, COUNT(*) FROM symbols s JOIN files f ON f.id = s.file_id WHERE f.lang IN ('javascript','typescript') GROUP BY 1,2" sqlite3 <db> "SELECT COUNT(*) FROM refs r JOIN symbols s ON s.id = r.target_id WHERE s.visibility = 'unknown' AND r.file_id <> s.file_id" # cost: crates/plugins/src/typescript.rs, the CommonJS arm's # `return Visibility::Unknown` -> `return Visibility::File`, rebuild, reindex sqlite3 <db> "SELECT COUNT(*) FROM refs WHERE target_id IS NOT NULL" ``` Both corpora are `$COSI_CORPUS_DIR/js-express` and `$COSI_CORPUS_DIR/ts-zod`. The plugin edit was reverted and md5-verified; nothing in this measurement is checked in. ## Related #86 (where this was a paragraph), #77 (the capability decision this needs), #75. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01K1zj5VcFJvJt3pQxe9259K

buildagent referenced this issue

2026-09-06 00:55:25 +02:00

plugin ABI: the qualifier-TEXT gap (gaps 1 and 2 CLOSED; gap 3 is a #77 capability decision, not an ABI gap) #86

buildagent commented

2026-09-06 17:35:29 +02:00

Author

Member

CONFIRMED — the population reproduces byte-for-byte on an independent build, and the gap is NOT reachable from the resolver side

Resolver-recall lane, worktree off master fc329a8. I own resolution this round, so the question I was asked to settle is whether any resolver change can close this. It cannot, and the reason is worth stating precisely rather than restating the issue.

Population, re-measured with a binary I built from `fc329a8`

corpus	`unknown` symbols	of total	refs resolving TO them	of those, CROSS-FILE
`js-express`	1 860	97.0 % of 1 917	4 077	833
`ts-zod`	4 387	45.4 % of 9 667	3 373	130

Identical to the filed numbers in every cell. The js-express breakdown is worth seeing whole, because it is what makes this a blocker rather than a rounding error:

javascript | unknown  | 1860
javascript | file     |   41
javascript | exported |    9
javascript | local    |    7

Nine symbols in the whole of express are detectably exported. Everything else is the CommonJS arm's honest "I looked and could not tell". Under the packaged mapping all 1 860 become file, and a file-visible symbol is in no cross-file candidate pool — symbol_buckets splits exported_n from private_n on exactly visibility NOT IN ('file','local'), and exported_pools, which is what every cross-file tier reads, is built from exported_n alone. So the mechanism the causal measurement found is the one the code has.

Why no resolver change reaches this, which is the part I can add

I spent this round widening a resolution tier (#69: tier 1R now serves the member pool, +4 922 binds over nine repos, 0 lost). It moved js-express by 0 binds, and that is not a coincidence of the change — it is the same wall. Every tier that could rescue a demoted symbol reads exported_pools; a producer that cannot say "I could not tell" has already been decided against before any tier runs. The resolver's inputs do not contain the distinction, so no tier can restore it.

That is the strongest form of "no manifest and no host change can close it": it is not that the host's unknown -> file default is tunable, it is that the state is gone by the time the resolver sees the row.

What I did NOT re-run, and why that is stated rather than glossed

The causal −805 / −19.4 % on js-express and +47 on ts-zod needed the plugin edit rebuilt, and I did not reproduce it — the population above already corroborates the mechanism closely (805 against an 833 cross-file upper bound), and this lane's build capacity went to #69's and #168's bind-for-bind passes. The ts-zod +47 is still uninspected bind-for-bind, and the issue is right to flag it as a precision question rather than a win: demoting candidates NARROWS pools, so a previously ambiguous group can go unique. In this lane the same effect appeared on py-django — refusing one wrong candidate produced 70 NEW correct binds — so the shape is real and the sign is not determinable without reading the rows. It should be read before anyone cites +47 as evidence for anything.

Verdict

Confirmed, blocked, and correctly filed as a port blocker rather than a defect. The host's unknown -> file is the least-authority default and must stay; what is missing is a producer-side way to say "not determined", which is a capability decision (#77) because believing it means admitting a symbol to cross-file pools on a third party's word. a_packaged_extractor_cannot_claim_unknown_visibility is the current spec and should not be relaxed without that decision.

One thing worth adding to the issue's own framing: the third state is already representable in the DATABASE (s.visibility = 'unknown' is what both queries above select on) and is only missing from the WIRE. So the change, when #77 admits it, is an ABI addition plus a capability gate — not a schema change and not a resolver change.

## CONFIRMED — the population reproduces byte-for-byte on an independent build, and the gap is NOT reachable from the resolver side Resolver-recall lane, worktree off master `fc329a8`. I own resolution this round, so the question I was asked to settle is whether any resolver change can close this. It cannot, and the reason is worth stating precisely rather than restating the issue. ### Population, re-measured with a binary I built from `fc329a8` | corpus | `unknown` symbols | of total | refs resolving TO them | of those, CROSS-FILE | |---|---:|---:|---:|---:| | `js-express` | **1 860** | 97.0 % of 1 917 | **4 077** | **833** | | `ts-zod` | **4 387** | 45.4 % of 9 667 | **3 373** | **130** | Identical to the filed numbers in every cell. The `js-express` breakdown is worth seeing whole, because it is what makes this a blocker rather than a rounding error: ``` javascript | unknown | 1860 javascript | file | 41 javascript | exported | 9 javascript | local | 7 ``` Nine symbols in the whole of express are detectably exported. Everything else is the CommonJS arm's honest "I looked and could not tell". Under the packaged mapping all 1 860 become `file`, and a `file`-visible symbol is in no cross-file candidate pool — `symbol_buckets` splits `exported_n` from `private_n` on exactly `visibility NOT IN ('file','local')`, and `exported_pools`, which is what every cross-file tier reads, is built from `exported_n` alone. So the mechanism the causal measurement found is the one the code has. ### Why no resolver change reaches this, which is the part I can add I spent this round widening a resolution tier (#69: tier 1R now serves the member pool, +4 922 binds over nine repos, 0 lost). It moved **js-express by 0 binds**, and that is not a coincidence of the change — it is the same wall. Every tier that could rescue a demoted symbol reads `exported_pools`; a producer that cannot say "I could not tell" has already been decided against before any tier runs. The resolver's inputs do not contain the distinction, so no tier can restore it. That is the strongest form of "no manifest and no host change can close it": it is not that the host's `unknown -> file` default is tunable, it is that the state is gone by the time the resolver sees the row. ### What I did NOT re-run, and why that is stated rather than glossed The causal −805 / −19.4 % on `js-express` and +47 on `ts-zod` needed the plugin edit rebuilt, and I did not reproduce it — the population above already corroborates the mechanism closely (805 against an 833 cross-file upper bound), and this lane's build capacity went to #69's and #168's bind-for-bind passes. The `ts-zod` **+47** is still uninspected bind-for-bind, and the issue is right to flag it as a precision question rather than a win: demoting candidates NARROWS pools, so a previously ambiguous group can go unique. In this lane the same effect appeared on py-django — refusing one wrong candidate produced 70 NEW correct binds — so the shape is real and the sign is not determinable without reading the rows. It should be read before anyone cites +47 as evidence for anything. ### Verdict **Confirmed, blocked, and correctly filed as a port blocker rather than a defect.** The host's `unknown -> file` is the least-authority default and must stay; what is missing is a producer-side way to say "not determined", which is a capability decision (#77) because believing it means admitting a symbol to cross-file pools on a third party's word. `a_packaged_extractor_cannot_claim_unknown_visibility` is the current spec and should not be relaxed without that decision. One thing worth adding to the issue's own framing: the third state is already representable in the DATABASE (`s.visibility = 'unknown'` is what both queries above select on) and is only missing from the WIRE. So the change, when #77 admits it, is an ABI addition plus a capability gate — not a schema change and not a resolver change.

buildagent referenced this issue

2026-09-06 18:40:05 +02:00

plugin ABI: the qualifier-TEXT gap (gaps 1 and 2 CLOSED; gap 3 is a #77 capability decision, not an ABI gap) #86

buildagent referenced this issue

2026-09-06 18:42:16 +02:00

gate: full-language migration proof — migrate the Ruby plugin to a package (#80 step 13) #84

dhoyer referenced this issue from a commit

2026-09-10 03:24:29 +02:00

feat: the guest ABI is bracketed, and `unknown` visibility has a wire slot (#153, #166, #154, #229)

dhoyer referenced this issue from a commit

2026-09-10 03:24:29 +02:00

merge lane/E: the guest ABI is bracketed, and unknown visibility has a wire slot (#166 #153 #154 #229)

buildagent commented

2026-09-10 07:43:54 +02:00

Author

Member

Fixed by 59c658a, merged at 110199d. Verified in the tree rather than from the commit subject:

// crates/abi/src/registry.rs:120
pub const VISIBILITIES: &[&str] = &["exported", "module", "file", "local", "unknown"];

and registry.rs:564 asserts VISIBILITIES.contains(&"unknown"), with the comment at :544 recording that the previous test asserted the opposite — so the claim this issue made is now graded in the direction that would catch a revert.

crates/indexer/tests/visibility_pool_split.rs (296 lines, added by the same commit) covers the pool-split consequence that made the missing slot cost a packaged JavaScript 19% of its resolutions.

Fixed by `59c658a`, merged at `110199d`. Verified in the tree rather than from the commit subject: ```rust // crates/abi/src/registry.rs:120 pub const VISIBILITIES: &[&str] = &["exported", "module", "file", "local", "unknown"]; ``` and `registry.rs:564` asserts `VISIBILITIES.contains(&"unknown")`, with the comment at :544 recording that the previous test asserted the opposite — so the claim this issue made is now graded in the direction that would catch a revert. `crates/indexer/tests/visibility_pool_split.rs` (296 lines, added by the same commit) covers the pool-split consequence that made the missing slot cost a packaged JavaScript 19% of its resolutions.

buildagent closed this issue

2026-09-10 07:43:58 +02:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

h-dv/code-index#166

No description provided.

Rows
Columns

Visibility::Unknown has no wire slot, and it costs a packaged JavaScript 19% of ALL its resolutions — MEASURED #166

The gap

The population

The cost, measured causally rather than bounded

Why this is a port blocker and not a tuning knob

Method, so it can be re-run

Related

CONFIRMED — the population reproduces byte-for-byte on an independent build, and the gap is NOT reachable from the resolver side

Population, re-measured with a binary I built from fc329a8

Why no resolver change reaches this, which is the part I can add

What I did NOT re-run, and why that is stated rather than glossed

Verdict

`Visibility::Unknown` has no wire slot, and it costs a packaged JavaScript 19% of ALL its resolutions — MEASURED #166

Population, re-measured with a binary I built from `fc329a8`