investigate: resolution spread across languages on real repos — zod 13.2% and Dapper 11.1% vs ripgrep 37.6% #46

New issue

Open

opened 2026-07-28 20:39:17 +02:00 by buildagent · 0 comments

buildagent commented

2026-07-28 20:39:17 +02:00

Member

P2 — diagnostic, explain before anyone optimises.

The observation

The 2026-07-28 corpus spike produced the first real-world per-language resolution figures this project has had:

repo	lang	files	symbols	refs	resolved	%	refs/file
rust-ripgrep	rust	231	4127	36607	13759	37.6	158
php-guzzle	php	165	3059	35791	11035	30.8	217
js-express	js	206	1917	19748	4154	21.0	96
ruby-sinatra	ruby	285	1260	17574	2768	15.8	62
python-flask	python	226	1620	14708	2075	14.1	65
ts-zod	ts	558	8489	85027	11246	13.2	172
cs-dapper	c#	223	2945	25155	2796	11.1	113

A 3.4× spread between Rust and C#.

Why this needs explaining, not fixing

The standing verdict is that the aggregate rate is a denominator artifact and must not be chased — chasing it mints phantoms. That verdict is almost certainly the explanation for much of this spread too:

ts-zod is a type-system library: an enormous share of refs are type-level and point at TS/lib built-ins. 172 refs/file is the highest in the set.
cs-dapper leans on BCL types (IDbConnection, Task<T>, LINQ) that have no in-project definition.
php-guzzle scoring 30.8% despite PHP's dynamism is interesting in the other direction — PSR interfaces are in-project, and I025 qualifier capture plus the importless same-dir class resolution do real work.

So the likely honest conclusion is "these numbers are mostly denominator composition, not resolver quality."

But that is a hypothesis, and it should be measured rather than assumed — precisely because it is the comfortable answer. If some of the gap is a genuine plugin gap (e.g. C# generics or TS conditional types dropping refs that do have in-project targets), that is a real recall bug hiding behind a convenient narrative.

What to do

Break each repo's refs down by resolution_gaps reason code (no_candidate / internal_missed / unresolved_qualified / unresolved_bare) — the split already exists in project_overview.file_health
For each language, sample ~50 internal_missed refs and classify by hand: genuinely external, or a real miss?
Report the internal-eligible rate per language (denominator excluding provably-external refs) — this is what #49's oracle would eventually give exactly, and a rough hand-sampled version is available now for far less effort
Feed any genuine gap classes into #47 as ranked mission candidates

Acceptance

per-language reason-code breakdown for all 7 repos
hand-classified sample per language with the external-vs-real-miss split
a written verdict: how much of the 3.4× spread is denominator composition vs plugin gap
any real gap class filed as its own issue with measured frequency

**P2 — diagnostic, explain before anyone optimises.** ## The observation The 2026-07-28 corpus spike produced the first real-world per-language resolution figures this project has had: | repo | lang | files | symbols | refs | resolved | % | refs/file | |---|---|---|---|---|---|---|---| | rust-ripgrep | rust | 231 | 4127 | 36607 | 13759 | **37.6** | 158 | | php-guzzle | php | 165 | 3059 | 35791 | 11035 | **30.8** | 217 | | js-express | js | 206 | 1917 | 19748 | 4154 | **21.0** | 96 | | ruby-sinatra | ruby | 285 | 1260 | 17574 | 2768 | **15.8** | 62 | | python-flask | python | 226 | 1620 | 14708 | 2075 | **14.1** | 65 | | ts-zod | ts | 558 | 8489 | 85027 | 11246 | **13.2** | 172 | | cs-dapper | c# | 223 | 2945 | 25155 | 2796 | **11.1** | 113 | A 3.4× spread between Rust and C#. ## Why this needs explaining, not fixing The standing verdict is that the aggregate rate is a **denominator artifact** and must not be chased — chasing it mints phantoms. That verdict is almost certainly the explanation for much of this spread too: - **ts-zod** is a type-system library: an enormous share of refs are type-level and point at TS/lib built-ins. 172 refs/file is the highest in the set. - **cs-dapper** leans on BCL types (`IDbConnection`, `Task<T>`, LINQ) that have no in-project definition. - **php-guzzle** scoring 30.8% despite PHP's dynamism is interesting in the other direction — PSR interfaces are in-project, and I025 qualifier capture plus the importless same-dir class resolution do real work. So the likely honest conclusion is "these numbers are mostly denominator composition, not resolver quality." **But that is a hypothesis, and it should be measured rather than assumed** — precisely because it is the comfortable answer. If some of the gap is a genuine plugin gap (e.g. C# generics or TS conditional types dropping refs that *do* have in-project targets), that is a real recall bug hiding behind a convenient narrative. ## What to do - Break each repo's refs down by `resolution_gaps` reason code (`no_candidate` / `internal_missed` / `unresolved_qualified` / `unresolved_bare`) — the split already exists in `project_overview.file_health` - For each language, sample ~50 `internal_missed` refs and classify by hand: genuinely external, or a real miss? - Report the **internal-eligible** rate per language (denominator excluding provably-external refs) — this is what #49's oracle would eventually give exactly, and a rough hand-sampled version is available now for far less effort - Feed any genuine gap classes into #47 as ranked mission candidates ## Acceptance - [ ] per-language reason-code breakdown for all 7 repos - [ ] hand-classified sample per language with the external-vs-real-miss split - [ ] a written verdict: how much of the 3.4× spread is denominator composition vs plugin gap - [ ] any real gap class filed as its own issue with measured frequency

buildagent referenced this issue

2026-07-28 20:39:57 +02:00

feat: frozen SCIP oracle — true precision/recall and the HONEST DENOMINATOR (build-required grader, zero-build product) #49

buildagent added the

correctness

Priority

Medium

labels

2026-07-28 20:40:47 +02:00

buildagent referenced this issue

2026-07-29 18:18:34 +02:00

bug: the JS/TS plugin never captures CommonJS require() — express's entire module graph is invisible (410 calls, 0 imports) #56