Phase 1 of pluggable languages: plain ids, owner setting, opt-in enablement, ABI and profiles, cost fixes #308
No reviewers
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
h-dv/code-index!308
Loading…
Reference in a new issue
No description provided.
Delete branch "p1-integration"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Phase 1 builds the foundation for moving languages into plugin packages. No language is switched to a package yet: the builtin parsers still index everything, and enabling a language chooses who owns it only under
auto.Language ids, owner setting and enablement (lane P)
ruby), byte-identical to the builtin's, so filters, baselines and bridges don't move. A third-party package that declares a reserved id is refused withlanguage_reserved, at install and at activation.plugin trust add --first-partylets an operator anchor their own key for reserved ids.[languages] <lang> = auto | builtin | package. A switch re-extracts only the files that change hands, andproject_overviewandindex_freshnessreport it. This is the one-release fallback.code-index initenables the languages the repository contains and writes.code-index.toml..code-index/languages.toml, so the user's git tree stays clean.language_not_enabled, with the command to fix it.code-index languages list | enable | owner.Plugin interface, profiles and guest kit (lane Q)
[languages.profile](extendsplus 17 fields, as closed enums). Unknown values are refused at install. Thefirst_party_profilesgate checks that each builtin's canonical declaration matches its compiled answers.0x8005);0x8006);0x8007);lexical_localandpytest_fixtureroles;extension_methodscapability;[[grammars]]), with one worker lane per grammar.module_maphook. Sandboxed and cached. The Composer PSR-4 mapping agrees with the builtin on all 129 comparable files ofphp-guzzle. The resolver does not read module maps or the exports fact yet; that happens before PHP and Python switch to packages.code-index-guest-kit. The shared guest helpers, used by the Ruby, Svelte and TimeLine packages, whose wasm is unchanged. A native Ruby differential runs over ruby-sinatra: 152 files, 0 differences from the builtin.Cost and robustness (lane R)
HOST_EXTRACTION_EPOCH(m0073). Package rows are keyed by the extraction epoch, not the host release. A gate fingerprints every shipped package's derived rows and fails if they change without an epoch bump. The epoch is now 2, because the plain-id stamp changed Ruby's rows._prdoc/records/P1-ruby-package-overhead.md): the earlier 1.44× vs 3.4× disagreement is settled, and cold start is attributed per phase.Integration
.cwasm.--first-partyanchor.ruby-package-cost. Re-measured under schema 73. It stays within its band; the −1.9% drift predates Phase 1.bounding_site_registry. Two indexer constants that reused package-crate names were renamed.project_overviewis back under its token budget (4,046 of 4,050).Validation
0397b0c):corpus_ratchet,corpus_mutation,corpus_costandcorpus_stage;agent_task_bench;first_party_influencered.tests/corpus/baseline.json,cost-baseline.jsonandstage-baseline.jsonare unchanged. Onlypackage-baseline.jsonandruby-package-cost.jsonmoved, for the reasons above.Not in this PR
🤖 Generated with Claude Code
https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu
7ebf88e, the commit that lands #112 phase 1's encoder methods 2c38fb668a- `project_overview.language_ownership` carries only what an agent can act on: `not_enabled` and owner rows (with the enabled list and the reading beside them), an owner switch, and on the one start that made it the enablement decision. `{}` is the reported steady state; `availability: unavailable` the unreported one. The same renderer serves `code-index://stats` (disclosure_derivation_registry G2). - Paid for under overview_payload_budget_e2e's pinned content reserve by removing the one sentence of `activation::SEMANTICS` that described the block's own absence: it rode only inside the block, and the absent state already renders as `availability: unavailable` with its own sentence. Saturated content 4047 -> 4028 with the new block included. - The withheld-grant census joins the producer only when a package has ever produced (reader_epoch_e2e: a never-activated index runs no generation predicate). - Fixtures that share one project between two starts, or that assert a clean git tree, now carry an explicit `[languages]` decision or the enablement marker: the first-start pass is an event of whichever start meets an undecided project first (mcp_smoke daemon parity, link_payload_scaling_e2e, answer_provenance_e2e); the settle-barrier population gains language_ownership_e2e. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmum0072 moved CURRENT_VERSION to 72, and package_baseline records the schema version it was measured under as a CONDITION: the run refused the blessed states ('written under 71, this run is 72'). That baseline may not be re-blessed by this lane, so the table is created on the first declaration by profiles::ensure_table and every reader probes for it (absent = the builtin answer, which is what such an index resolved under). A project whose packages declare nothing never has the table and its schema is unchanged. generation_policy_registry could see neither a lazily-created table (its source scan reads migrations, its live scan a freshly migrated database) — the hole it exists to close. It now carries LAZY_OBJECTS: each entry's creator is EXERCISED before sqlite_master is read, must spell CREATE TABLE IF NOT EXISTS, and the object is registered like any other (declared_profiles: PromotionRestamped by write_declared, with its promotion window recorded in PROMOTION_DOES_NOT_RESTAMP_YET). Corpus suites (ratchet, mutation, stage, cost, ruby_package_parity, package_baseline) green with COSI_CORPUS_REQUIRE=1 and no baseline moved; precision_gate 7/7, phantoms 0. Also: the guest kit's docs say it depends on the SDK alone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmuEvery discovery now runs `collect_stale_caches` after it has warmed or verified its own entries. Removed: a `<package digest>` directory whose package is no longer INSTALLED (nothing can look it up again); an entry of ANOTHER host build whose manifest was last used more than STALE_CACHE_AFTER (7 days) ago; any entry with no manifest. A verified use stamps the manifest's mtime, so age means "unused", not "old". Not approval-keyed, deliberately: the caches serve every project on the machine and an approval is one project's. Test cache_gc.rs, mutations RUN: the uninstalled arm disabled -> RED; the age threshold multiplied by 1000 -> RED; the call removed from discovery -> RED ("an uninstalled package's cache survived"). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmuThe integration made `get_classified_bytes` return `anchor.first_party || is_first_party_key(key)`. Lane P's tests anchor their fixture key `--first-party` so a package may claim the reserved id `ruby`, and that flag then exempted lane R's THIRD-party legs from the general influence pass: first_party_influence recorded a third-party container builtin-equivalent, and package_cost_attribution's positive control saw 0 general-pass executions. Restore lane R's rule: the exemption is `Store::is_first_party_key` (compiled-in FIRST_PARTY_ANCHORS, widened only by `with_first_party_keys`). An operator's `--first-party` anchor keeps meaning what lane P built it for - permission to claim a reserved id - and never switches off the pass. Both meanings are now documented at `is_first_party_key` and `TrustAnchor::first_party`. MUTATION (RUN): put `anchor.first_party ||` back. RESULT: RED in both first_party_influence ("a third-party container was recorded builtin-equivalent: [1]") and package_cost_attribution (POSITIVE CONTROL FAILED, 0 general-pass executions). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmuhost_extraction_epoch fired: Ruby's host-derived fingerprint moved (05316132.. -> 3e6e49ac..) under epoch 1 with unchanged package digests. CAUSE, VERIFIED: lane P stamps a first-party package's rows with the reserved plain id (`langid::row_lang`, `de.h-dv.ruby/ruby` -> `ruby`). With `row_lang` mutated back to the identity, the gate passes against the epoch-1 record unchanged, i.e. that is the ONLY moved input. Row diff of the two dumps: 137 of 142 rows differ, and after substituting `de.h-dv.ruby/ruby` -> `ruby` in the old dump the diff is empty — every change is the `lang` column. This is the gate's designed path: rows on operators' disks were written with the wire id, so the epoch moves and `reconcile_extraction_epoch` re-derives them. Re-recorded with COSI_BLESS_HOST_EPOCH=1; before the bless the epoch-only bump was RED ("HOST_EXTRACTION_EPOCH is 2 and the record was taken at 1"). The re-extraction of files derived under epoch 1 is graded by `a_package_file_derived_under_another_epoch_is_re_extracted_and_nothing_else`, which rewinds the stamp to HOST_EXTRACTION_EPOCH - 1 (now 1) and to NULL and requires exactly the three package files to re-parse. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmuThe band was refused on a CONDITION: measured under schema 71, binary at 73 (m0072, m0073). Re-measured, not edited, through the suite's bless (COSI_RUBY_PKG_COST_BLESS=1, reason 'Phase 1: re-measured under schema 73'). NUMBERS (release, ruby-sinatra, 153 files): record(s71)0bcdac1this tree (4 runs) package vm_step 30,656,396 30,093,721 30,083,210..30,092,091 package fullscan 630,321 630,864 625,672..626,467 sort / autoindex 396/22833 396/22833 396/22833 builtin vm_step 17,053,958 17,090,476 17,068,543..17,072,396 builtin fullscan — 228,796 222,859 wall ratio 144 141 141 144 142 134(written) Every dimension is inside the old band (+5% / -12%). ATTRIBUTED: - vm_step -1.9% against the record is NOT Phase 1: an isolated release run of0bcdac1(Phase 0 tip) already reads 30.09M. Phase 1 moves it by ~-0.02%, and reverting lane P's plain-id stamp moves it by nothing (30,092,091). The drift happened within band over Phase 0's 84 commits, where this gate passed it. - fullscan -0.8% IS Phase 1, and it is shared: the builtin leg dropped by the same ~5.9k (228,796 -> 222,859), so it is a common-path saving, not package-path work. - The first-party exemption does NOT apply here: this leg's key is not first party, and it pays the influence pass (package_cost_attribution: 4 general-pass executions). Load 3.6-6.6 through the runs; one full gate run was in progress in the main checkout (not isolated). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmuMerged: master was fast-forwarded to
4ecf930after CI on that exact commit passed on Linux (run 5648) and native Windows (run 5647). The two fixes after the PR opened:ffb91ad: rustfmt on the Ruby guest crate. The rebuiltextractor.wasmis byte-identical, so the package digest did not move.4ecf930:psr4_vs_builtinpasses/-separated paths to the hook, as the host does. Production was unaffected.I'm closing this PR by hand because Forgejo does not detect a fast-forward merge.
Pull request closed