#75 - epic: runtime plugin architecture — dynamic grammar, extraction and resolution packages - h-dv/code-index

buildagent commented

2026-08-26 12:30:04 +02:00

Member

Decision

code-index will become a runtime-extensible indexing platform. A format is supported by installing a plugin package, not by linking another Rust crate and publishing another code-index release.

This deliberately reverses the v1 non-goal “Dynamic plugin loading from disk” in ARCHITECTURE.md and _prdoc/vision/00-mission.md. The implementation must update those decisions and record the new trust model before the first plugin is enabled.

Definition of pluggable

This epic is complete only when an already-released code-index binary can:

install a previously unknown plugin package;
load its grammar dynamically;
run its declarative or executable extractor;
index symbols, references and imports;
apply only explicitly approved resolver semantics and cross-language bridges;
upgrade, disable, roll back and remove the plugin without stale rows;
survive malformed, adversarial and non-terminating plugin code without wedging the daemon; and
disclose exactly which package generation produced every result.

A runtime rule file over a grammar compiled into the binary does not meet this definition. Grammar loading is foundational, not a deferred enhancement.

Governing architecture

plugin package
  manifest + content digest + ABI requirements
  grammar.wasm
  declarative queries/rules
  optional extractor.wasm
  language resolution profile
  explicit cross-language bridge declarations
  fixtures and conformance expectations
          |
          v
supervised plugin-host process
  no project filesystem or network access
  bounded CPU, memory, output and parse depth
  hard deadline; kill and restart on timeout
          |
          v
validated extraction-fact protocol
  symbols, refs, imports, diagnostics
  no database ids and no asserted target edges
          |
          v
core indexer
  row-level producer/generation provenance
  capability-gated candidate pools
  core-owned resolution
  generation-safe activation and rollback

The core resolver remains the only component allowed to declare an edge. A plugin may emit a reference site and evidence, never a target symbol id. That rule is necessary but not sufficient: candidate-pool membership is itself authority and is therefore capability-gated by #77.

Package model

A package is content-addressed and immutable. Its identity covers every byte and every interpretation input:

package id and semantic version;
package-format and host ABI versions;
tree-sitter grammar artifact and grammar ABI;
declarative extraction rules and optional extractor component;
language profile and bridge declarations;
ordered path claims and precedence;
declared resource budgets;
conformance fixtures;
rule-engine and plugin-host compatibility requirements.

An edited package is a new digest even when its author forgot to bump the version. Projects activate an exact digest. Mutable “latest” identifiers may be resolved by an install command but are never stored as the active identity.

Project checkouts are untrusted. Merely cloning a repository must never execute a grammar or extractor. Packages are installed into a user-controlled registry and explicitly approved for a project. Project configuration may request a package digest but cannot silently grant executable or resolver capabilities.

Two extractor tiers

A real ecosystem needs both:

Declarative extraction for formats whose semantics can be expressed as bounded tree queries, captures, literal transforms and container rules.
Optional extractor WASM for stateful semantics such as scope tracking, visibility sections, binding inference, macro/token handling and framework conventions.

The second tier is required because the existing plugins demonstrate that node-kind tables alone are not expressive enough. Both tiers emit the same validated fact protocol and receive the same budgets. Executable extractors do not gain database, filesystem, network or resolver access.

Compiled plugins may remain as a trusted fast path during migration, but the external package path must be able to express and pass the corpus of at least one complete existing language plugin. “Markup only” is not acceptance for this epic.

Language and bridge model

Every package-defined language has its own stable namespaced language id. It must not borrow csharp, rust or another host language merely to enter existing pools.

Resolution-relevant behavior moves out of hardcoded language allowlists into a validated language profile, including:

case and identifier normalization;
qualified-name separator and canonicalization;
symbol kinds eligible as types, members and containers;
import/module normalization and relative-import behavior;
visibility defaults;
type-position and member-only role semantics;
permitted proximity and reachability rules.

Cross-language behavior is explicit. XAML to C#, Razor to C#, Vue to TypeScript and similar relationships are bridge capabilities with direction, source ref classes, destination symbol classes and scope constraints. A bridge is never inferred from two languages sharing a string.

Default capability is searchable-but-inert: plugin-produced symbols appear in search and outlines but participate in no cross-file candidate pool. Capabilities are promoted deliberately and are visible in payloads.

Isolation decision

The current tree-sitter WASM integration cannot be the production security boundary: an infinite lexer is not interruptible through its public API, stores may share memory/table state across grammars, and store lifetime conflicts with the current rayon parse path.

Dynamic grammars and extractor components therefore run in a supervised helper process defined by #79. The parent reads the already-eligible file and sends bytes plus bounded metadata; the plugin receives no path-based filesystem capability. A timeout kills the worker, not a rayon thread or the daemon. One plugin package never shares a WASM store with another package.

An in-process trusted fast path may be considered later, but it is not the correctness path and cannot change observable extraction output.

Data ownership and provenance

Provenance is row-level, not file-level. Embedded-language files can contain facts from the wrapper package, a host-language extractor and bridge rules simultaneously. Symbols, refs, imports and diagnostics must identify package digest, activation generation, language id and extraction component.

The existing files.lang value may remain as a compatibility summary, but it cannot be the source of truth for mixed-language resolution. #77 defines the schema and pool changes.

Activation and recovery

Plugin changes are index inputs just like source bytes. #78 replaces extension-only invalidation with generation-based activation:

build a pending generation without exposing partial results;
re-evaluate every file in the package claim domain, including formerly text-only and newly ineligible files;
validate extraction and resolver gates;
atomically promote the generation;
retain the previous generation for rollback;
recover deterministically after kill -9 at every transition;
garbage-collect inactive generations only after promotion is durable.

Cold indexing and incremental activation must converge to the same active database projection.

Child issues and order

#76 — immutable package format, versioned host ABI, language profile schema and validated extraction-fact protocol.
#79 — supervised, killable runtime grammar and extractor host. Depends on #76.
#77 — row-level provenance, dynamic language identities, capability-gated resolver participation and explicit bridges. Depends on #76; integrates with #79.
#78 — content-digest detection, claim-domain re-evaluation, pending/active generations, atomic promotion and rollback. Depends on #76, #77 and #79.
#80 — package conformance kit, hostile-plugin tests, disclosures, operational CLI and end-to-end production proof. Depends on all preceding issues.

#81 remains independent and should ship immediately: honest disclosure of symbol-blind extensions is required before and after runtime plugins exist.

Reference implementation

XAML/C# is the first end-to-end bridge because it exercises all hard parts:

a dynamically loaded XML-family grammar;
XAML symbols and event/type references;
code-behind C# as a distinct destination language;
searchable but initially inert x:Name declarations;
explicit Click and x:Class bridge rules;
ambiguous Binding paths remaining unresolved unless a declared, graded rule proves a destination;
generated Designer.cs coverage measured separately rather than hidden by the plugin system.

A second acceptance package must migrate one complete existing compiled language plugin and produce an equivalent fact projection on its fixtures and corpus probes. This prevents an architecture that is only nominally generic.

Production gates

The epic does not waive existing production blockers or scale contracts. Plugin activation must degrade with disclosure rather than hang, OOM or poison the active index.

Required end-to-end mutations include:

infinite grammar lexer;
infinite extractor loop;
memory and output bombs;
malformed spans, parent ordering, kinds and UTF-8;
package edit without version bump;
claim precedence reorder;
text to code and code to text transitions;
package removal and rollback;
daemon crash during every activation phase;
mixed-language file with multiple producers;
attempted candidate-pool contamination;
undeclared cross-language bridge;
old client/new daemon and new client/old daemon skew.

Final acceptance

Using only released binaries and plugin-management commands:

install an unknown package by pinned digest;
activate it for a project;
index its new grammar with useful symbols, refs and imports;
resolve only explicitly permitted edges;
observe package/generation/capability provenance;
upgrade it and prove cold equals incremental;
kill the daemon during upgrade and recover the old or new complete generation, never a mixture;
roll back and remove it with no surviving rows;
demonstrate that a hanging or malicious package is terminated while the daemon and other projects remain usable.

Anything less is extensible configuration, not a pluggable indexer.

## Decision code-index will become a runtime-extensible indexing platform. A format is supported by installing a plugin package, not by linking another Rust crate and publishing another code-index release. This deliberately reverses the v1 non-goal “Dynamic plugin loading from disk” in ARCHITECTURE.md and _prdoc/vision/00-mission.md. The implementation must update those decisions and record the new trust model before the first plugin is enabled. ## Definition of pluggable This epic is complete only when an already-released code-index binary can: 1. install a previously unknown plugin package; 2. load its grammar dynamically; 3. run its declarative or executable extractor; 4. index symbols, references and imports; 5. apply only explicitly approved resolver semantics and cross-language bridges; 6. upgrade, disable, roll back and remove the plugin without stale rows; 7. survive malformed, adversarial and non-terminating plugin code without wedging the daemon; and 8. disclose exactly which package generation produced every result. A runtime rule file over a grammar compiled into the binary does not meet this definition. Grammar loading is foundational, not a deferred enhancement. ## Governing architecture plugin package manifest + content digest + ABI requirements grammar.wasm declarative queries/rules optional extractor.wasm language resolution profile explicit cross-language bridge declarations fixtures and conformance expectations | v supervised plugin-host process no project filesystem or network access bounded CPU, memory, output and parse depth hard deadline; kill and restart on timeout | v validated extraction-fact protocol symbols, refs, imports, diagnostics no database ids and no asserted target edges | v core indexer row-level producer/generation provenance capability-gated candidate pools core-owned resolution generation-safe activation and rollback The core resolver remains the only component allowed to declare an edge. A plugin may emit a reference site and evidence, never a target symbol id. That rule is necessary but not sufficient: candidate-pool membership is itself authority and is therefore capability-gated by #77. ## Package model A package is content-addressed and immutable. Its identity covers every byte and every interpretation input: - package id and semantic version; - package-format and host ABI versions; - tree-sitter grammar artifact and grammar ABI; - declarative extraction rules and optional extractor component; - language profile and bridge declarations; - ordered path claims and precedence; - declared resource budgets; - conformance fixtures; - rule-engine and plugin-host compatibility requirements. An edited package is a new digest even when its author forgot to bump the version. Projects activate an exact digest. Mutable “latest” identifiers may be resolved by an install command but are never stored as the active identity. Project checkouts are untrusted. Merely cloning a repository must never execute a grammar or extractor. Packages are installed into a user-controlled registry and explicitly approved for a project. Project configuration may request a package digest but cannot silently grant executable or resolver capabilities. ## Two extractor tiers A real ecosystem needs both: 1. Declarative extraction for formats whose semantics can be expressed as bounded tree queries, captures, literal transforms and container rules. 2. Optional extractor WASM for stateful semantics such as scope tracking, visibility sections, binding inference, macro/token handling and framework conventions. The second tier is required because the existing plugins demonstrate that node-kind tables alone are not expressive enough. Both tiers emit the same validated fact protocol and receive the same budgets. Executable extractors do not gain database, filesystem, network or resolver access. Compiled plugins may remain as a trusted fast path during migration, but the external package path must be able to express and pass the corpus of at least one complete existing language plugin. “Markup only” is not acceptance for this epic. ## Language and bridge model Every package-defined language has its own stable namespaced language id. It must not borrow csharp, rust or another host language merely to enter existing pools. Resolution-relevant behavior moves out of hardcoded language allowlists into a validated language profile, including: - case and identifier normalization; - qualified-name separator and canonicalization; - symbol kinds eligible as types, members and containers; - import/module normalization and relative-import behavior; - visibility defaults; - type-position and member-only role semantics; - permitted proximity and reachability rules. Cross-language behavior is explicit. XAML to C#, Razor to C#, Vue to TypeScript and similar relationships are bridge capabilities with direction, source ref classes, destination symbol classes and scope constraints. A bridge is never inferred from two languages sharing a string. Default capability is searchable-but-inert: plugin-produced symbols appear in search and outlines but participate in no cross-file candidate pool. Capabilities are promoted deliberately and are visible in payloads. ## Isolation decision The current tree-sitter WASM integration cannot be the production security boundary: an infinite lexer is not interruptible through its public API, stores may share memory/table state across grammars, and store lifetime conflicts with the current rayon parse path. Dynamic grammars and extractor components therefore run in a supervised helper process defined by #79. The parent reads the already-eligible file and sends bytes plus bounded metadata; the plugin receives no path-based filesystem capability. A timeout kills the worker, not a rayon thread or the daemon. One plugin package never shares a WASM store with another package. An in-process trusted fast path may be considered later, but it is not the correctness path and cannot change observable extraction output. ## Data ownership and provenance Provenance is row-level, not file-level. Embedded-language files can contain facts from the wrapper package, a host-language extractor and bridge rules simultaneously. Symbols, refs, imports and diagnostics must identify package digest, activation generation, language id and extraction component. The existing files.lang value may remain as a compatibility summary, but it cannot be the source of truth for mixed-language resolution. #77 defines the schema and pool changes. ## Activation and recovery Plugin changes are index inputs just like source bytes. #78 replaces extension-only invalidation with generation-based activation: - build a pending generation without exposing partial results; - re-evaluate every file in the package claim domain, including formerly text-only and newly ineligible files; - validate extraction and resolver gates; - atomically promote the generation; - retain the previous generation for rollback; - recover deterministically after kill -9 at every transition; - garbage-collect inactive generations only after promotion is durable. Cold indexing and incremental activation must converge to the same active database projection. ## Child issues and order 1. #76 — immutable package format, versioned host ABI, language profile schema and validated extraction-fact protocol. 2. #79 — supervised, killable runtime grammar and extractor host. Depends on #76. 3. #77 — row-level provenance, dynamic language identities, capability-gated resolver participation and explicit bridges. Depends on #76; integrates with #79. 4. #78 — content-digest detection, claim-domain re-evaluation, pending/active generations, atomic promotion and rollback. Depends on #76, #77 and #79. 5. #80 — package conformance kit, hostile-plugin tests, disclosures, operational CLI and end-to-end production proof. Depends on all preceding issues. #81 remains independent and should ship immediately: honest disclosure of symbol-blind extensions is required before and after runtime plugins exist. ## Reference implementation XAML/C# is the first end-to-end bridge because it exercises all hard parts: - a dynamically loaded XML-family grammar; - XAML symbols and event/type references; - code-behind C# as a distinct destination language; - searchable but initially inert x:Name declarations; - explicit Click and x:Class bridge rules; - ambiguous Binding paths remaining unresolved unless a declared, graded rule proves a destination; - generated Designer.cs coverage measured separately rather than hidden by the plugin system. A second acceptance package must migrate one complete existing compiled language plugin and produce an equivalent fact projection on its fixtures and corpus probes. This prevents an architecture that is only nominally generic. ## Production gates The epic does not waive existing production blockers or scale contracts. Plugin activation must degrade with disclosure rather than hang, OOM or poison the active index. Required end-to-end mutations include: - infinite grammar lexer; - infinite extractor loop; - memory and output bombs; - malformed spans, parent ordering, kinds and UTF-8; - package edit without version bump; - claim precedence reorder; - text to code and code to text transitions; - package removal and rollback; - daemon crash during every activation phase; - mixed-language file with multiple producers; - attempted candidate-pool contamination; - undeclared cross-language bridge; - old client/new daemon and new client/old daemon skew. ## Final acceptance Using only released binaries and plugin-management commands: 1. install an unknown package by pinned digest; 2. activate it for a project; 3. index its new grammar with useful symbols, refs and imports; 4. resolve only explicitly permitted edges; 5. observe package/generation/capability provenance; 6. upgrade it and prove cold equals incremental; 7. kill the daemon during upgrade and recover the old or new complete generation, never a mixture; 8. roll back and remove it with no surviving rows; 9. demonstrate that a hanging or malicious package is terminated while the daemon and other projects remain usable. Anything less is extensible configuration, not a pluggable indexer.

phase	issues	state
0	#65	done — resolver fan-out bounded, per-stage budgets, live progress + cancellation
1	#76, #79	done — fact ABI, `.cip` format, supervised killable worker
2	#77	done — schema v42→v48: producer identity, row provenance, capability-gated pools, directional bridges
3	#78	done — schema v49→v54: 8-state durable machine, atomic promotion + rollback, `kill -9` at all 12 transitions, reader epoch
4	#80	in progress
—	#81	closed

phase	state
#65 resolver fan-out budgets	closed (v0.23.0)
#76 static package validation	closed
#77 bridges / resolution packages	closed
#78 lifecycle, generations, rollback	closed
#79 isolated runtime validation	closed
#80 conformance / containment / disclosure gate	OPEN
#84 full-language migration proof (#80 step 13)	OPEN
#86 three ABI gaps blocking any language but Ruby/PHP	OPEN

child	state
#76 — package format, host ABI, profile schema, fact protocol	closed
#79 — supervised killable grammar/extractor host	closed
#77 — provenance, dynamic identities, capability-gated resolution	closed
#78 — content-digest detection, generations, promotion, rollback	closed
#80 — conformance, hostile containment, disclosure, E2E gate	OPEN
#81 — symbol-blind coverage disclosure (independent)	closed

Rows
Columns

epic: runtime plugin architecture — dynamic grammar, extraction and resolution packages #75

Decision

Definition of pluggable

Governing architecture

Package model

Two extractor tiers

Language and bridge model

Isolation decision

Data ownership and provenance

Activation and recovery

Child issues and order

Reference implementation

Production gates

Final acceptance

Status as of v0.26.1 — measured, not remembered

Phases

The three production-wiring gaps from #80 S15 are now ALL CLOSED

What still stands between here and final acceptance

Carried forward, measured and named

Close-out sweep: the epic's true remaining count is 63 open issues, and the phase chain is DONE

The epic's own structure

Closed by this sweep (7)

Kept open with corrected text (10)

Two facts that bear directly on this epic's readiness

The 63, grouped

Close-out round 2 — the true open count, 2026-09-06, master fc329a8

This epic's children

Closed in this round

Left open, with corrected residuals

One thing this epic should know

Close-out round 2 — the true open count, 2026-09-06, master `fc329a8`