Packages cannot see inside a multi-language file: one grammar per package, no tree-sitter injection #275
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
h-dv/code-index#275
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
STATUS: DESIGN SETTLED AND CORRECTED — not yet implemented
The original report is kept verbatim below the divider; the comments cite it by section. What changed, and where:
file_contributionscarriesregion_start/region_endwith the region in its UNIQUE key, and its own comment says single-owner "today". MEASURED: 0 files with >1 contribution, all 888 rows atregion_start = 0.refs.langandimports.langDO exist since m0044 — I asserted the opposite from one call site. This makes the design simpler, not harder.injections.scmengine. A new optional fact tag, followingTAG_SYMBOL_BASENAME(#229/#268): the guest says "bytes [a,b) are language L", and the language is named by INDEX into a host-owned table, never a string..svelte. Delegating to another package's sandboxed extractor is OUT of v1 — and that exclusion is enforced by the TYPE, not by discipline.THE FOUR SOCKETS THIS FEATURE FILLS
This is not a feature being bolted on. The tree was shaped for it, in four places, and three of them are currently pinned by tests asserting they are UNUSED:
file_contributions.region_start/region_end, region in the UNIQUE key, "single-owner today".fact.language_not_owned,fact.region_out_of_range,fact.region_depth_exceeded— withreason_producers.rs::PRODUCERLESSpinning that they have none.EMBEDDED_DISPATCH_SEMANTICS_VERSION, sitting at0, whose doc is a specification for this exact change: "The day a<script>block inside HTML is dispatched to a second producer, the same bytes extract differently and every index built under the old semantics is stale."symbols.langsince m0001,refs.lang/imports.langsince m0044.None of that licenses "it mostly works already". Every socket is untested by construction. A change filling them must edit the tests that pin them empty, in the same commit.
WHAT THE REVIEW CHANGED — do not implement the first spec as written
Four items are BLOCK-grade:
builtin. Routing them underbuiltin/<lang>reaches the outcomepackages.rs:1898explicitly refuses ("it gains the whole authority model"), through a door that refusal does not watch. Minting underde.h-dv.svelte/typescriptkeeps the refusal holding, gives the rows the package's OWN operator-granted capabilities, closes the "indistinguishable from real TypeScript" hazard for free, and dissolves three of the four open questions..svelteedits onto a Full resolve — measured at 400x the median Scoped pass on this repository. Use a PER-CONTRIBUTION outline hash instead.span_extent_amplification.rs. Charge the padded buffer, and bound the host parse: the builtin arm has no fuel, no epoch and no timeout.<script>with noexportyields Unknown-visibility symbols that enter cross-file candidate pools;precision_gatestructurally cannot see it (JavaScriptforbid_sitesis 1). The record needs a third field: what SCOPE the region is.And one prerequisite: nothing in this repository can measure this feature's cost. The pinned corpus holds zero
.sveltefiles andcost-baseline.jsoninstalls no packages. Asvelte_package_cost.rsis a prerequisite, not a follow-up.BLOCKED ON, and blocking
read_file_claim'sgroup_concatalready takes a pathological plan and this feature would hand it more rows. Fixing it first is what makes any cost number here attributable (the #189 lesson).Original report (superseded 2026-09-15)
Kept verbatim: the comments on this issue cite it by section.
Found while reviewing #268 (Svelte package). Recorded as a HOST limit rather than a Svelte problem, because it is one.
The limit, measured
One grammar per package.
crates/package/src/manifest.rs:142declares:Not
Option<Grammar>, notVec<Grammar>. A package names exactly one[grammar]block with oneartifactand oneexported_name, so a claimed file is parsed by one grammar and produces one tree.No tree-sitter language injection anywhere in the tree. A
search_textforinjectionacross the workspace returns 37 hits and every one is fault injection, flag injection or SQL injection. There is no concept of a secondary grammar parsing a sub-range of a file.Why that is a real gap
A large and growing class of source files is multi-language BY CONSTRUCTION — the embedded language is not an edge case, it is where the code lives:
.svelte<script>, CSS in<style>.vue<script>, CSS in<style>.astro.mdx.razor/.cshtml@{ }For all of these, the community tree-sitter grammar parses the template and exposes the embedded block as raw text; editors recover the inner structure with an INJECTION query that runs a second grammar over that range. We have no equivalent, so an extractor receives an opaque token where the functions, imports and declarations are.
Concretely for #268: four of its eight planned extraction targets — props (
export let,let { x } = $props()), runes ($state,$derived), script functions, and imports — are all inside<script>.Why the XAML precedent does not cover it
tests/packages/xamlbridges into C# withscope = "paired_file", and that works because the C# is a SEPARATE FILE (MainWindow.xaml+MainWindow.xaml.cs), already indexed by the compiled-in C# plugin. The bridge joins two files.In an SFC the second language is in the SAME file. There is no paired file to bridge to and no row to point at, so
paired_file,same_directoryandsame_fileall have nothing on the other end.crates/package/src/bridge.rs'sevidence_for_scopeadmits only those three scopes.What this does NOT claim
That injection is the only possible answer, or that it is cheap. The host runs guests in a sandboxed WASM worker with a per-package grammar loaded by
plugin-host; a second grammar per file is a real cost in memory, in the kind-id tables each extractor compiles against (ruby_kind_table.rspins 351 kind ids and 33 field ids for ONE grammar), and in the fact-ABI, which currently has no way to say "this fact came from a sub-range parsed by another grammar".It also does not claim any file is currently WRONG. Nothing mis-indexes today; these formats are simply text-only, which is honest. This is about what a package is ABLE to express.
Options, none obviously right
plugin-host, the fact ABI and the digest (a second grammar is a second artifact to pin reproducibly).same_filebridge against facts the outer grammar emits as opaque. Cheapest, and weakest: the package emits a coarse ref for the script block and bridges within the file. Buys little — there is nothing structured on the other side._prdoc/guides/80-package-authoring.mdgains a section stating the limit so the next package author designs around it instead of into it. Defensible; #268 can ship a useful v0.1.0 on this basis.Option 4 is worth doing REGARDLESS of which of 1–3 is chosen, because the guide's silence is what let #268 be specified with script-block extraction in its milestones.
Scope note
Pre-existing; not introduced by #268, which is only the first package to run into it. No package in the tree is affected today —
xaml(paired file),timeline(single language) andruby(single language) all sit inside the limit.Confirmed from the grammar side, independently
The issue above argues this limit from the HOST: one
pub grammar: Grammarper package, no injection anywhere in the tree. That is an argument about what we cannot consume.The grammars themselves make the same statement about what they PRODUCE, and they were written by people who have never seen this codebase. Measured on
tree-sitter-svelte-ng1.0.2 (crate sha256ef0a71f9cf5e94373cc86c64893630c8a29bb25d3390a248268d08af2165fa37),queries/injections.scm:The entire
<script>body is ONEraw_textnode. The grammar ships an injection query because that is the only way anything sees inside it.tree-sitter-svelte-next0.1.1 is a separate fork by a different author and does the same thing, mapping script totypescriptand style tocss.Corroborating measurement from
node-types.json, which separates what the template grammar really models from what it defers:{#snippet …}{@render …}$props$state$derivedSvelte 5's runes score zero not because the grammar is behind — it models Svelte 5's template additions fully — but because runes are JavaScript expressions and the grammar correctly declines to re-implement a JavaScript parser.
Why this strengthens the case rather than just restating it
Two independent sources agreeing matters more than either alone:
Both say it, so neither is available. Every maintained tree-sitter grammar for a multi-language format draws this boundary the same way, because an injection query IS the ecosystem's answer to embedded languages. A host that cannot run one cannot reach the embedded half of ANY of these formats, no matter which grammar is pinned.
That also gives option 2 (sub-range delegation) a concrete shape it did not have when this issue was filed: the grammars already SHIP the range queries, as
queries/injections.scm, naming both the capture range and the target language. The mapping frominjection.languageto an installed plugin is a lookup we could already perform againstBUILTIN_CLAIMS. What is missing is a way to run the query and to attribute the resulting facts to a sub-range of the outer file — not a way to know which range or which language.Scope note, unchanged
Nothing mis-indexes today;
.svelteand its siblings are text-only, which is honest. This remains about what a package is ABLE to express.The substrate for this already exists, and has never been exercised
Grounding before implementation turned up something the issue as filed did not know: the schema was designed for sub-range contributions and is waiting for a caller.
file_contributions(m0043,producer_identity):region_startis IN THE UNIQUENESS KEY, and the comment says single-owner "today". Several other pieces line up the same way:symbols.langis per ROW, not per file — a.sveltefile can already holdtypescriptsymbols.refshas nolangcolumn and does not need one. A ref's language arrives throughrefs.contribution_id → file_contributions.component_id → extraction_components.lang;fill_bridge_refsalready joins exactly that way. So a second contribution in another language stamps its refs correctly for free.contribution_id, so a second contribution inherits the whole lifecycle.And it is unexercised, which is the part to be careful about
Measured against this repository's own live index:
Every contribution in a real database sits at
[0, size). So this is a designed-for capability with no caller, no test and no production evidence — the exact shape where latent defects live. Nothing here should be read as "it already works"; the claim is only that the storage model does not have to change.Decided design: the guest emits the range, the host owns the language
Rather than the issue's option 1 (a second grammar per package) or a host-side query engine, the range comes from a new OPTIONAL FACT TAG, following the pattern
TAG_SYMBOL_BASENAMEjust established in #229/#268:The Svelte extractor already knows the script range — its grammar parsed the
<script>boundary and hands it back as oneraw_textnode. It does not need to parse the contents, and the host does not need to runinjections.scm. The guest says "bytes [a, b) are language L"; the host routes that region to whichever component claims L, offsets the spans, and records a secondfile_contributionover exactly that region.The language is named by INDEX into a host-owned table, not by string. This is the rule
de.h-dv.ruby0.5.0 records forREL_QUALIFIERS: "The wire carries a u16 INDEX into a table the HOST owns, so this package cannot express a qualifier the host did not write down, and there is no authority left for an operator to grant. A value that cannot be minted needs no permission to mint." A package must not be able to delegate a region to an arbitrary language it names itself.Scope of the first change
Foundation plus one real consumer. The delegated region goes to the existing COMPILED-IN typescript/javascript extractors — mature, already trusted, already the correct answer for a
.sveltescript block — and it is proven end to end on.svelte. Delegating to another PACKAGE's sandboxed extractor is deferred: grants, sandbox budgets and digests would all have to compose across two packages, and none of that is needed to make the capability real.Building the foundation with synthetic tests and no consumer was considered and rejected on this repository's own rule — a gate with no demand behind it is the shape that shipped 15k lines with zero production callers.
What this does NOT promise
That the cross-directory component bind from #268 comes back automatically. It should: the import becomes a fact, and
a_component_tag_does_not_bind_across_directories_and_that_is_the_ceilingis written as an ASSERTION so it goes red the day that changes. But the resolver tiers have not been examined for it, and #268's ceiling comment will be re-earned by measurement rather than assumed.Correction, and the extent of the anticipation
One claim in my previous comment is WRONG
I wrote: "
refshas nolangcolumn and does not need one. A ref's language arrives throughrefs.contribution_id → file_contributions.component_id → extraction_components.lang."refs.langexists. So doesimports.lang. Both since m0044 (crates/indexer/src/migrations/m0044_row_language.rs). Measured:The resolver splices
refs.langdirectly (crates/indexer/src/index.rs~4245,row_lang), with thefilesjoin only as a pre-m0044 fallback. The component→lang path I described is the bridge path specifically (fill_bridge_refs), which is where I read it from and over-generalised.This makes the design simpler, not harder: the delegated pass passes its language straight to
writer::insert_refs, which already takeslang: &str. No new column, no join change, no inference. But the correction matters on its own terms — I asserted a schema fact from one call site, and one call site is not the schema.The codebase anticipated this feature in FOUR places, not one
My previous comment named the
file_contributionsregions. There are three more, and together they change how this should be read — this is not a feature being bolted on, it is a socket being filled.1. The region columns, as reported:
region_start/region_end, region in the UNIQUE key, comment says single-owner "today".2. THREE REASON CODES ALREADY EXIST FOR THIS AND HAVE NEVER HAD A PRODUCER (
crates/abi/src/reason.rs:211-215):And
crates/abi/tests/reason_producers.rs::PRODUCERLESSpins that they have none — so that test goes red on this change and is edited as part of it, which is the gate confirming the socket is being filled rather than a new one cut.3.
EMBEDDED_DISPATCH_SEMANTICS_VERSION(crates/indexer/src/activation.rs:94) is already folded into the activation identity, sitting at0, and its doc is a specification for this exact change:That constant exists to be spent on this, and spending it is what makes every existing index correctly re-activate.
4. Per-row languages —
symbols.langsince m0001,refs.lang/imports.langsince m0044 — so a file holding two languages needs no schema change at all.What that does and does not license
It does NOT license "this mostly works already". The measurement stands: 0 files with more than one contribution, all 888 rows at
region_start = 0. Every one of these sockets is untested by construction, and three of them are pinned by tests asserting they are unused.What it does license is confidence that the design is the one the tree was shaped for, rather than one imposed on it — and a concrete obligation: a change that fills these sockets must edit the tests that pin them empty, in the same commit, or those tests are the ones that quietly stop meaning anything.
One hazard the spec surfaced that is worth stating here
m0044's backfill isUPDATE refs SET lang = (SELECT f.lang FROM files f WHERE f.id = refs.file_id)— unconditional, idempotent only under the very invariant delegation breaks. It cannot re-run in normal operation, butupgrade_equivalenceandcontribution_generation_key::strip_to_v50deliberately roll versions BACK. A rollback reaching 44 would relanguage every delegated row to the file's route language, and the fresh-vs-migrated projection comparison would then differ. The fix is a repair migration ordered after 44, on the same "a rollback that reaches 51 always reaches 52" argument m0052 already makes — not an edit to a shipped migration.Six-lane adversarial review — the design changes before implementation
The mechanism is right. Three things must change, one listed hazard is false, and one prerequisite is missing entirely.
1. Attribute delegated rows to the DELEGATING PACKAGE, not to
builtincrates/indexer/src/packages.rs:1898-1926already refuses, asIndexerError::Integrity, any package claim resolving to a reserved key:Routing delegated rows under
builtin/typescriptreaches that exact outcome through a door the refusal does not watch. Verified:is_reserved_keyhas three production call sites (packages.rs:1921,packages.rs:2171,activation.rs:499) and all three are on the grant/identity side. The three contribution writers (writer.rs:306,writer.rs:463,build.rs:1377) callensure_componentwith no check at all. The only thing standing between a package andgrant_alltoday is that the component is derived from the file's own language — an accident of single-owner routing, not a guard, and delegation is precisely what removes it.Mint the component under the delegating package's key —
de.h-dv.svelte/typescript,lang = "typescript".extraction_componentsis alreadyUNIQUE (package_id, component)with its ownlangcolumn, so:symbols.lang/refs.lang/imports.langstill readtypescript; the resolver splices exactly as designedis_reserved_keyis false,grant_allnever fires, and the existing refusal keeps holdingresolverlist — a package withresolver = []gets searchable-but-inert delegated rows, which is the documented correct defaultextraction_identitymove, no approval-store field, noschema.v1.jsonchangecomponent_idbecomes the discriminator. Under the original design a delegated row joins the samebuiltin/typescriptcomponent as real.tsfiles and onlyregion_start <> 0tells them apart.Correction to my earlier comment while I am here: it is eight pool capabilities, not four (
RESOLVER_CAPABILITIES,manifest.rs:64-98).2. HAZARD #1 IS FALSE, and the proposed fix would be expensive
The spec says delegated symbols MUST be folded into
compute_outline_hashor "a function added to a<script>is never resolvable from another file."Measured, that does not follow for ADD.
writer.rs:409-412puts the file inresolve_scope_filesgated onoutline_hash.is_some()— not onoutline_changed— and the Scoped predicate (index.rs:3967-3974) carries a project-wide NAME arm, with the candidate pool built unscoped from the wholesymbolstable. A new symbol is a candidate on a Scoped pass exactly as on a Full one.The real hazard is REMOVE / RENAME, and it is not in the spec. Delete a function from a
<script>: hash unchanged ⇒ Scoped;write_upsertdeletes thefilesrow;refs.target_id … ON DELETE SET NULLNULLs every cross-file ref that pointed at it; the scope name arm asks whether that name is a symbol in the scope file, which it no longer is. Those refs sit outside the Scoped population and stay unresolved, with a latched staleresolved_by, until an unrelated change forces a Full pass. This silently falsifiesinfluence.rs:157-168("any symbol added or removed … forcesResolveScope::Full").And folding in would cost.
typescript.rs'semit_varputs the initializer into the hashed signature viasignature_until_body, so$state(0)→$state(1)and any$derived(expr)tweak would move the hash. Measured on this repository — 888 files, 226,116 refs — a Scoped pass touches a median of 565 refs (0.250%), max 38,765 (17.1%): the firewall is worth 400x on the median edit and 5.8x on the worst.index.rs:724-728records 17 ms → 17.9 s (~1,050x) on rust-analyzer when one delete forced a full re-decide. Folding delegations into the file-level hash would push roughly half to two-thirds of.svelteedits onto the Full path.Do this instead: a PER-CONTRIBUTION
outline_hashandoutline_changed.file_contributionsalready keys on(file_id, component_id, generation_id, region_start)andm0054_promotion_classification.rs:92-99already carriesnew_outline_hash/prior_outline_hash. That restores theinfluence.rsinvariant without making an initializer edit global.3. The charge is on the wrong quantity
The spec charges
region_end - region_start; the allocation is the padded buffer, bounded byregion_end— i.e. by the FILE. 64 one-byte regions at EOF: ~640 wire bytes, 64 x file_size allocated and parsed. At the 2 MiB default that is 128 MiB per file, ~1.0 s of pure tree-sitter; at the 1 GiB operator ceiling, 64 GiB.This is the identical mistake
crates/abi/tests/span_extent_amplification.rs:88-94already records as a RUN mutation ("charge the span's START instead of its extent — every span starts at 0, so the charge is 0").It also relocates attacker-steered work to the unbounded arm.
index.rs:10769-10783states the asymmetry: the package arm is bounded by fuel, epoch, memory ceiling and the parent's kill; the builtin arm "parses in this process, on this thread". Measured:set_timeout_microsandset_cancellation_flaghave zero hits tree-wide. Charge the padded buffer length, and add a per-filemax_delegated_bytesceiling.4. Delegated symbol VISIBILITY is a phantom factory — not in the ten hazards
TypeScriptPlugin::extractdecides module semantics from the bytes handed to it:is_esm= any top-levelexport_statement, elsecollect_cjs_exports, else the plugin's own comment — "those stay Unknown so the resolver never gates them". Unknown passes the cross-file visibility gate.A Svelte
<script>with noexport— an ordinary component with no props — therefore yields top-level symbols with Unknown visibility, in a language the resolver treats as ordinary TypeScript. A helper namedformatinFoo.sveltebecomes a cross-file candidate for every unresolvedformatin every.tsfile in the project. Svelte component scripts are not importable by name, except<script module>— exactly the distinction a(start, end, lang)triple cannot express.No gate here can see this.
precision_gatenever indexes a corpus repo and scores phantoms only against DECLARED decoys; JavaScript'sforbid_sitesis 1.The record needs a third field: what SCOPE the region is (module vs local).
.vue<script>vs<script setup>,.astrofrontmatter and.razor@codeall ask the same question, so it belongs in the record rather than in the Svelte consumer.5. Prerequisite: nothing in this repository can measure the cost
Measured: the pinned corpus is 9 repos and contains zero
.sveltefiles;cost-baseline.jsoninstalls no packages (plugin_path_cost.rssays so: "the pinned corpus installs no packages, so the guest path never executes under it"). Ship #275 as specified and it is cost-unmeasured by construction.A
.sveltefixture tree plus asvelte_package_cost.rsmodelled onruby_package_cost.rs— both legs,wall_ratio_pctwith a ceiling AND a floor — is a prerequisite, not a follow-up.Smaller corrections
FactRegionDepthExceededstays producerless and its rationale inreason_producers.rsmust be REWRITTEN rather than deleted.AND afc.region_start = fc.region_startis not a contribution's identity; addAND afc.component_id = fc.component_id. It works today only because a<script>can never start at byte 0 — an accident, not a structural fact.write_upsert's container rule becomes a cross-contribution write.CONTAINER_SCOPE_FILEisrefs.file_id = ?1 AND s.file_id = ?1; its siblingwrite_pending_contributionalready documents why that is wrong with two contributions. The svelte module symbol spans the whole file, so a component intest.sveltewould mark every delegated TypeScript ref as test code. Move toCONTAINER_SCOPE_CONTRIBUTION.FactLanguageNotOwned's declared meaning is claim-level ("a fact claimed a language the dispatched component does not own"). TheFactRelQualifierUnknownprecedent cuts FOR a new code: an index outside a closed registry gets its own refusal, because "degrading there would let a producer choose between two host behaviours by picking an index". MintFactRegionLanguageUnknown.conform.rsis a separate door and the spec never mentions it.conform.rs:107-115already says where this lands: "When embedded dispatch ships, its comparison belongs inexpect::grade."host.extract(appears in exactly two places, soplugin check --repodoes not pass throughextract_forand gets no delegation unless someone puts it there.DELEGABLE_LANGUAGESis the feature's entire population and nothing grades its size. Adding"css"later would silently change extraction semantics for every existing.svelteindex. Pin the table's contents beside the version so widening it without bumping is red.PosMap; measured,typescript.rshas FOUR sites computingnode.start_position().row + 1directly, bypassingPosMap(484, 1165, 1290, 1408), plus one each in five other plugins. It works because tree-sitter itself counts the synthetic rows. Record that, or the obvious "simplify to a PosMap origin offset" refactor silently breaks four fields.(C-1)spaces putregion_startat codepoint columnC. The design is correct as written.Strength worth recording
"Delegating to another package is out of v1" is enforced by the type, not by discipline:
select_plugin(path, &[Arc<dyn LanguagePlugin>])can only return a compiled-in plugin, and packages reach the indexer throughPackageHost::extract, a different trait with one call site. A synthetic.tspath cannot reach a package claiming.ts. That is the structural clause this repo asks for and it is already true — say so in the spec.One alternative the spec should record and reject in writing
Eight of the ten listed hazards are consequences of ONE shape change — two contributions per file. That suggests a fifth option nobody named: declare the region as a virtual sub-file (
Foo.svelte#script.ts) routed through ordinaryclassify_walked/select_plugin.files.langstays honest, contributions stay one-per-file, and the outline hash, container rule, coverage DTO andread_file_claimall keep their single-producer shapes. It is probably disqualified — afilesrow with no bytes on disk, no mtime, and anindex_coverageanswer for a path nobody can open, against freshness machinery that keys on real stat triples — but "eight hazards are one shape change" is the strongest argument an implementer will meet, and the spec currently has no written answer to it.read_file_claim's diagnosticsgroup_concattakes a different index from its siblings, and its comment says it does not #278read_file_claim's diagnosticsgroup_concattakes a different index from its siblings, and its comment says it does not #278