epic: every builtin language becomes a first-party plugin package #309

New issue

Open

opened 2026-09-27 15:03:26 +02:00 by buildagent · 0 comments

buildagent commented 2026-09-27 15:03:26 +02:00

Member

Copy link

Follows #75 (runtime plugin architecture). Goal: the base system is infrastructure only, and each of the seven languages (Ruby, PHP, JS, TS, C#, Rust, Python) ships as a first-party plugin package. A language fix can then be released without a core release.

Status as of 2026-09-27: Phase 0 and Phase 1 are merged (master 4ecf930). No language has flipped yet; the builtin parsers still index everything.

Operator decisions (2026-09-26)

1. Ids: first-party packages use RESERVED PLAIN ids (ruby, php, …). No migration of rows, configs or baselines. Third-party packages may not claim them.
2. Enablement: first-party packages ship in the release archive and are OPT-IN.
   • code-index init, or the first start with no decision, enables the languages the repository contains and says so.
   • An automatic start records the decision in .code-index/languages.toml, never in the project root.
   • Upgrades keep every language the index already holds.
3. Perf budget per port, against the builtin on the pinned corpus: at most 10% slower warm and 20% slower cold. No regression for users without packages.
4. Fallback: the builtin stays for ONE release after a flip, switchable with [languages] <lang> = builtin|package, and is deleted in N+1.
5. Port order, one per release: Ruby → PHP → JS → TS → C# → Rust → Python.

Premise corrections from planning

• Most language logic is in the resolver, not the extractors. 23 of the 26 language fixes since v0.32.0 touched the resolver. The per-language resolver rules have to become package-declared data: profiles, a module_map hook and new facts.
• The real cost blocker is the general influence pass, which adds about 80% SQLite work once any package holds capabilities. First-party packages are now exempt.
• The earlier Ruby overhead figures (1.44× vs about 3.4×) are both correct: they were measured before and after #113. See _prdoc/records/P1-ruby-package-overhead.md.

Phase 0 — DONE (#306, master 0bcdac1)

• Profile routing: every inline builtin language id is routed through lang_profile. There were 57 literals at 41 sites; now 0, enforced by lang_literal_gate with no allowlist. The resolver SQL reads temp.lang_profile.
• Language filters: a lang filter given as a profile name (lang: "ruby") also matches packaged languages.
• Real-grammar costs: _prdoc/records/P0-real-grammar-costs.md. Workers set oom_score_adj=1000, precompile runs in parallel, idle workers retire after 60 s.
• Ruby fixes: the W0 local-decisions fixture, and a transient package refusal now keeps the file's facts (W7).
• Gates: a parameterized grammar build (with the PHP wchar.h shim), a generic package_parity LangSpec harness, and COSI_CORPUS_LANGS.
• Packaged C# isolation: packaged C# is isolated from builtin C# in the partial-class and reclassification tiers, which now key on lang and take their capability gates.

Phase 1 — DONE (#308, master 4ecf930)

• Reserved plain ids and language_reserved: first-party rows are stamped with the plain id.
• Owner setting: [languages] <lang> = auto|builtin|package. A switch re-extracts only the files that change hands and is disclosed.
• Opt-in enablement: code-index languages list|enable|owner, the language_not_enabled disclosure, and the upgrade rule.
• Declared profiles: [languages.profile] with extends and closed enums. The first_party_profiles gate checks each builtin's declaration against its compiled answers.
• ABI tags: doc span 0x8005, qualifier segments 0x8006, exports 0x8007, grantable lexical_local/pytest_fixture, extension_methods, and multi-grammar [[grammars]] lanes.
• module_map hook: host side, plus Composer PSR-4 in code-index-guest-kit. It agrees with the builtin on 129 of 129 comparable php-guzzle files.
• code-index-guest-kit: a native Ruby differential over ruby-sinatra (152 files) finds 0 differences.
• Influence-pass exemption for compiled-in first-party keys: SQLite work drops from 1.76× to 1.00× the builtin. An operator's --first-party anchor grants reserved-id permission only, not the exemption.
• HOST_EXTRACTION_EPOCH (m0072/m0073), now 2, with a fingerprint gate over every shipped package's derived rows.
• Verified caches: grammar and extractor caches, keyed per lane by the wasm's content hash, with garbage collection. Loading the Ruby grammar drops from 215 ms to 24 ms, and the pool grows lazily.

Remaining before the FIRST flip (Ruby)

• #301 resumable resolve. It also causes the nightly query_cli flake on master.
• #307 promotion lock at 100k files: 13.9 s, 89.9% of it re-stamping carried rows.
• Move the declared-profile table (created on first use) into a numbered migration; the package baseline must then be re-measured.
• Re-measure the cold budget on a quiet machine: it is met on wall time, measured on a contended machine.
• Named grammar lanes do not use the grammar cache yet.

Phase 2 — distribution (not started)

• Bundle the first-party packages in the release archive, with an enabled language mapped to its bundled package.
• Independent package releases: package tags, a rolling signed catalog, plugin update <id>, and a requires_host range. The reusable release workflow from Phase 0 is designed but not built, and must be verified by dispatching it.

Phase 3 — flips, one per release

• Ruby: first a shadow release in which the package ships and the builtin still owns the language; then the flip; then delete the builtin one release later.
• PHP: needs the resolver to consume module_map (PSR-4).
• JS, TS: need multi-grammar lanes to use the grammar cache.
• C#.
• Rust.
• Python: needs the resolver to consume the exports fact (__all__) and module_map (src layouts).

Known and not fixed

• pc_calls chooses the enclosing class from ungated symbols. This only matters with two producers in one file, which no shipped path creates yet.
• bounding_site_registry matches bounds by name across the workspace, so a reused name inherits another bound's disclosure.
• corpus_require_floor greps --test <name> without checking which package it belongs to.
• Lane worktrees are only indexed when COSI_WORKTREES_DIR is set (#238), and our own .mcp.json did not set it until now.

Follows #75 (runtime plugin architecture). **Goal:** the base system is infrastructure only, and each of the seven languages (Ruby, PHP, JS, TS, C#, Rust, Python) ships as a first-party plugin package. A language fix can then be released without a core release. **Status as of 2026-09-27:** Phase 0 and Phase 1 are merged (master `4ecf930`). No language has flipped yet; the builtin parsers still index everything. ## Operator decisions (2026-09-26) 1. **Ids:** first-party packages use RESERVED PLAIN ids (`ruby`, `php`, …). No migration of rows, configs or baselines. Third-party packages may not claim them. 2. **Enablement:** first-party packages ship in the release archive and are OPT-IN. - `code-index init`, or the first start with no decision, enables the languages the repository contains and says so. - An automatic start records the decision in `.code-index/languages.toml`, never in the project root. - Upgrades keep every language the index already holds. 3. **Perf budget per port**, against the builtin on the pinned corpus: at most 10% slower warm and 20% slower cold. No regression for users without packages. 4. **Fallback:** the builtin stays for ONE release after a flip, switchable with `[languages] <lang> = builtin|package`, and is deleted in N+1. 5. **Port order,** one per release: Ruby → PHP → JS → TS → C# → Rust → Python. ## Premise corrections from planning - **Most language logic is in the resolver, not the extractors.** 23 of the 26 language fixes since v0.32.0 touched the resolver. The per-language resolver rules have to become package-declared data: profiles, a `module_map` hook and new facts. - **The real cost blocker is the general influence pass,** which adds about 80% SQLite work once any package holds capabilities. First-party packages are now exempt. - **The earlier Ruby overhead figures (1.44× vs about 3.4×) are both correct:** they were measured before and after #113. See `_prdoc/records/P1-ruby-package-overhead.md`. ## Phase 0 — DONE (#306, master `0bcdac1`) - [x] **Profile routing:** every inline builtin language id is routed through `lang_profile`. There were 57 literals at 41 sites; now 0, enforced by `lang_literal_gate` with no allowlist. The resolver SQL reads `temp.lang_profile`. - [x] **Language filters:** a `lang` filter given as a profile name (`lang: "ruby"`) also matches packaged languages. - [x] **Real-grammar costs:** `_prdoc/records/P0-real-grammar-costs.md`. Workers set `oom_score_adj=1000`, precompile runs in parallel, idle workers retire after 60 s. - [x] **Ruby fixes:** the W0 local-decisions fixture, and a transient package refusal now keeps the file's facts (W7). - [x] **Gates:** a parameterized grammar build (with the PHP `wchar.h` shim), a generic `package_parity` `LangSpec` harness, and `COSI_CORPUS_LANGS`. - [x] **Packaged C# isolation:** packaged C# is isolated from builtin C# in the partial-class and reclassification tiers, which now key on `lang` and take their capability gates. ## Phase 1 — DONE (#308, master `4ecf930`) - [x] **Reserved plain ids and `language_reserved`:** first-party rows are stamped with the plain id. - [x] **Owner setting:** `[languages] <lang> = auto|builtin|package`. A switch re-extracts only the files that change hands and is disclosed. - [x] **Opt-in enablement:** `code-index languages list|enable|owner`, the `language_not_enabled` disclosure, and the upgrade rule. - [x] **Declared profiles:** `[languages.profile]` with `extends` and closed enums. The `first_party_profiles` gate checks each builtin's declaration against its compiled answers. - [x] **ABI tags:** doc span `0x8005`, qualifier segments `0x8006`, exports `0x8007`, grantable `lexical_local`/`pytest_fixture`, `extension_methods`, and multi-grammar `[[grammars]]` lanes. - [x] **`module_map` hook:** host side, plus Composer PSR-4 in `code-index-guest-kit`. It agrees with the builtin on 129 of 129 comparable `php-guzzle` files. - [x] **`code-index-guest-kit`:** a native Ruby differential over ruby-sinatra (152 files) finds 0 differences. - [x] **Influence-pass exemption** for compiled-in first-party keys: SQLite work drops from 1.76× to 1.00× the builtin. An operator's `--first-party` anchor grants reserved-id permission only, not the exemption. - [x] **`HOST_EXTRACTION_EPOCH`** (m0072/m0073), now 2, with a fingerprint gate over every shipped package's derived rows. - [x] **Verified caches:** grammar and extractor caches, keyed per lane by the wasm's content hash, with garbage collection. Loading the Ruby grammar drops from 215 ms to 24 ms, and the pool grows lazily. ## Remaining before the FIRST flip (Ruby) - [ ] #301 resumable resolve. It also causes the nightly `query_cli` flake on master. - [ ] #307 promotion lock at 100k files: 13.9 s, 89.9% of it re-stamping carried rows. - [ ] Move the declared-profile table (created on first use) into a numbered migration; the package baseline must then be re-measured. - [ ] Re-measure the cold budget on a quiet machine: it is met on wall time, measured on a contended machine. - [ ] Named grammar lanes do not use the grammar cache yet. ## Phase 2 — distribution (not started) - [ ] Bundle the first-party packages in the release archive, with an enabled language mapped to its bundled package. - [ ] Independent package releases: package tags, a rolling signed catalog, `plugin update <id>`, and a `requires_host` range. The reusable release workflow from Phase 0 is designed but not built, and must be verified by dispatching it. ## Phase 3 — flips, one per release - [ ] **Ruby:** first a shadow release in which the package ships and the builtin still owns the language; then the flip; then delete the builtin one release later. - [ ] **PHP:** needs the resolver to consume `module_map` (PSR-4). - [ ] **JS, TS:** need multi-grammar lanes to use the grammar cache. - [ ] **C#.** - [ ] **Rust.** - [ ] **Python:** needs the resolver to consume the exports fact (`__all__`) and `module_map` (src layouts). ## Known and not fixed - `pc_calls` chooses the enclosing class from ungated symbols. This only matters with two producers in one file, which no shipped path creates yet. - `bounding_site_registry` matches bounds by name across the workspace, so a reused name inherits another bound's disclosure. - `corpus_require_floor` greps `--test <name>` without checking which package it belongs to. - Lane worktrees are only indexed when `COSI_WORKTREES_DIR` is set (#238), and our own `.mcp.json` did not set it until now.

buildagent referenced this issue 2026-09-27 15:03:30 +02:00

epic: runtime plugin architecture — dynamic grammar, extraction and resolution packages #75

buildagent added a new dependency 2026-09-27 15:03:32 +02:00

#301 Reference resolution is one non-resumable transaction, so one-shot queries after an upgrade can never converge

buildagent added a new dependency 2026-09-27 15:03:33 +02:00

#307 Promotion lock at 100k files: stop re-stamping carried rows (13.9 s, 89.9% is the carry)

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

master

p1-integration

p0-integration

release/v0.32.2

release/v0.32.1

epic-i072-release-032

windows-installer

i269b-registry-truth

probe-win-baseline

fix/246-shadowed-imports

release/0.28.1

fix/review-followthrough-20260911

iwin-msvc-archive

v0.32.2

v0.32.1

v0.32.0

v0.31.2

v0.31.1

v0.31.0

v0.30.1

v0.30.0

v0.29.0

v0.28.3

v0.28.2

v0.28.1

v0.28.0

v0.27.1

v0.27.0

v0.26.1

v0.26.0

v0.25.0

v0.24.1

v0.24.0

v0.23.1

v0.23.0

v0.22.3

v0.22.2

v0.22.1

v0.22.0

v0.21.0

v0.20.3

v0.20.2

v0.20.1

v0.20.0

v0.19.5

v0.19.4

v0.19.3

v0.19.2

v0.19.1

v0.19.0

v0.18.1

v0.18.0

v0.17.0

v0.16.0

v0.15.3

v0.14.0

v0.13.3

v0.13.2

v0.13.1

v0.13.0

v0.12.0

v0.11.7

v0.11.6

v0.11.5

v0.11.4

v0.11.3

v0.11.2

v0.11.1

v0.11.0

v0.10.2

v0.10.1

v0.10.0

v0.9.0

v0.8.7

v0.8.6

v0.8.5

v0.8.4

v0.8.3

v0.8.2

v0.8.1

v0.8.0

v0.7.1

v0.7.0

v0.6.1

v0.6.0

v0.5.21

v0.5.20

v0.5.19

v0.5.18

v0.5.17

v0.5.16

v0.5.15

v0.5.14

v0.5.13

v0.5.12

v0.5.11

v0.5.10

v0.5.9

v0.5.8

v0.5.7

v0.5.6

v0.5.5

v0.5.4

v0.5.3

v0.5.2

v0.5.1

v0.5.0

v0.4.2

v0.4.1

v0.4.0

v0.3.3

v0.3.2

v0.3.1

v0.3.0

v0.2.0

Labels

Clear labels   

code-review

  

correctness

  

dos

  

performance

  

security

  

severity/high

  

severity/low

  

severity/medium

  

tech-debt

  

Kind/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No labels

code-review

correctness

dos

performance

security

severity/high

severity/low

severity/medium

tech-debt

Kind/Breaking

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Depends on

#301 Reference resolution is one non-resumable transaction, so one-shot queries after an upgrade can never converge

h-dv/code-index

#307 Promotion lock at 100k files: stop re-stamping carried rows (13.9 s, 89.9% is the carry)

h-dv/code-index

Reference

h-dv/code-index#309

No description provided.