h-dv/code-index

Watch

Fork 0

runtime: supervised, killable dynamic grammar and extractor host #79

New issue

Closed

opened 2026-08-26 12:33:21 +02:00 by buildagent · 3 comments

buildagent commented

2026-08-26 12:33:21 +02:00

Member

Child of #75. Depends on #76.

Decision

Dynamic grammar loading is no longer a deferred experiment. It is the runtime foundation of the plugin architecture.

The verified tree-sitter WASM work remains valuable, but its unsafe lifecycle properties change the deployment design: untrusted grammar and extractor execution runs in a supervised helper process, not inside the daemon or rayon parse pool.

Why a process boundary is mandatory

Verified constraints in tree-sitter 0.22.6:

a WASM lex function can loop forever; parser timeout checks occur outside the lex call and the public store API exposes no fuel/epoch control;
catch_unwind cannot catch a hang or stack exhaustion;
multiple grammars in one store share linear memory and indirect-function-table state;
WasmStore is not Send and must outlive parser use;
the current indexer creates parsers inside rayon work, while LanguagePlugin returns a Language by value;
ABI-incompatible grammar loading can appear successful until set_language;
Windows GNU and musl have unresolved runtime/support risks.

A long-lived daemon cannot accept “one malformed file or grammar permanently consumes a worker” as degradation. OS-process termination is the initial correctness boundary even if future tree-sitter/wasmtime APIs make in-process interruption possible.

Runtime topology

Ship a dedicated code-index-plugin-host executable with the same release.

The daemon/indexer owns a bounded supervisor:

parent indexer
  eligibility and size checks
  package/capability lookup
  bounded request queue
         |
         v
supervisor
  worker keyed by package digest
  deadline + memory/output accounting
  crash/restart/quarantine policy
         |
         v
plugin-host child
  one package digest / one WASM store
  no inherited project descriptors
  no filesystem/network/environment capability
  grammar + declarative engine + optional extractor
         |
         v
#76 fact response
  validated again in parent

A worker never loads two package digests into one store. Package upgrade starts a new worker; it does not mutate a running worker.

The parent sends file bytes and a project-relative logical path. The child does not open project paths. Standard input/output framing must not share the MCP server protocol or logs. Stderr is bounded diagnostic output and cannot block the child.

Process containment

Cross-platform requirements:

close or explicitly whitelist inherited handles/file descriptors;
sanitized environment with no credentials or project variables;
private working directory with no project checkout;
no network capability where the platform can enforce it;
memory ceiling and child-tree containment;
hard wall-clock deadline per request;
kill the entire worker process tree on timeout;
bounded startup and shutdown deadlines;
Windows Job Object or equivalent child-tree ownership;
Unix process-group and resource-limit ownership;
worker death never kills the daemon or corrupts active generation state.

WASM still supplies deterministic guest isolation inside the child. The process boundary supplies killability and package-to-package isolation.

Security claims must be stated per platform. If filesystem/network denial cannot be strongly enforced on a supported platform, the payload and documentation say so; do not call a sanitized environment a sandbox.

Grammar loading

The host validates before first parse:

grammar artifact digest matches the package;
tree-sitter ABI is within both package declaration and host-supported range;
language.version is checked explicitly before set_language;
expected exported language name exists;
grammar initialization, query compilation and empty-input parse complete within startup budgets;
a package with an external scanner passes a non-trivial fixture probe.

Legacy dylink vs dylink.0, ABI 12/15 behavior and scanner-built artifacts each receive fixtures with stable rejection codes.

Package tooling must provide a documented reproducible grammar build path. Requiring an undocumented Docker/emscripten ritual is not an ecosystem. The project may ship a builder image, but package format and runtime cannot depend on Docker being installed on the user machine.

Extractor execution

The host supports the #76 tiers:

Declarative query/rule engine.
Optional extractor.wasm using the stable fact ABI.

Grammar and extractor may execute in the same per-package worker because the process is already the package trust boundary. They do not share state with another package.

The exact tree representation supplied to extractor.wasm must be benchmarked before ABI freeze:

serialized bounded event stream;
host calls over opaque node handles; or
extractor linked into the same component/runtime.

Raw C pointers, Rust layout and tree-sitter internal struct layout are forbidden ABI.

Region extraction for embedded languages is orchestrated by the parent/supervisor. A wrapper package returns validated regions and source maps; the supervisor dispatches region bytes to the selected installed host-language component and remaps validated spans. Recursive embedding has a strict depth and total-byte budget.

Scheduling and backpressure

global cap on live plugin-host processes;
per-project and per-package request concurrency caps;
bounded queue with cancellation;
package startup coalescing so N files do not start N workers;
idle worker retirement;
no plugin request executes on Tokio core workers;
slow plugin cannot starve compiled plugins or another project;
parse result ordering remains deterministic despite concurrent workers.

The parent distinguishes timeout, worker crash, ABI rejection, resource refusal, invalid facts and user cancellation. Each has a stable reason code used by #80.

Failure policy

Per-file failure:

reject the complete component result;
emit a bounded diagnostic;
leave previous active generation visible;
continue other files up to an anomaly threshold.

Per-package failure:

exponential restart backoff;
quarantine after a bounded crash/timeout count;
stop dispatching new work;
fail activation under #78;
keep previous generation active;
require explicit retry after package/config change or operator action.

A one-off malformed project file must not permanently quarantine a sound grammar unless it repeatedly crashes the worker; parse errors returned normally are not crashes.

Trust and installation

A repository may request a digest but cannot install or execute it. Installation and capability approval occur in user-controlled state.

Minimum flow:

plugin inspect/validate without execution (#76);
plugin install --sha256 ;
plugin check in isolated host (#80);
project enable exact digest with explicit capabilities;
#78 builds and promotes a generation.

No automatic remote download during indexing. Registry/network distribution can be layered above an explicit install command; the indexer operates on local immutable packages.

Performance contract

The earlier measurements remain baselines, not acceptance:

WASM parsing around 1.6–1.7x native;
substantial runtime dependency/build-size cost;
per-store startup/JIT and RSS costs.

Measure:

helper cold and warm startup;
IPC cost at 1 KiB, 100 KiB and max file size;
grammar/extractor CPU;
peak RSS per worker and globally;
throughput under mixed native/dynamic load;
watcher latency;
crash/restart and timeout recovery latency.

Hot built-ins may remain native, but shadow mode must prove the package path produces equivalent facts. Performance fast paths cannot use different extraction semantics.

Tests

Hostile components:

infinite grammar lexer;
infinite external scanner;
infinite extractor loop;
memory.grow bomb;
output/framing bomb;
stderr flood;
deep recursion/stack exhaustion;
abort/trap;
fork/child attempt where applicable;
malformed response and mid-frame exit.

Lifecycle:

worker crash with queued requests;
timeout during watcher indexing;
daemon shutdown with live workers;
package upgrade while old worker is busy;
project A’s plugin hangs while project B remains usable;
process descriptor/handle inheritance audit;
Windows and musl runtime smoke tests, not link-only gates.

ABI:

supported and unsupported grammar versions;
legacy dylink artifact;
external-scanner grammar;
declarative and executable extractor packages;
embedded-region dispatch and span remapping.

Acceptance

An unknown grammar package loads and parses without rebuilding code-index.
A malicious infinite lexer is terminated inside the deadline; the daemon answers a health/tool request afterward.
Package A cannot read/write package B’s memory because they never share a worker/store.
The child has no project filesystem or network access under every platform claim we publish.
Windows and musl artifacts execute real dynamic grammar and trap/timeout smoke tests.
Bounded mixed-load performance stays within committed CPU/RSS/latency ceilings.
#78 can treat host failure as a failed pending generation while the prior generation stays queryable.

Child of #75. Depends on #76. ## Decision Dynamic grammar loading is no longer a deferred experiment. It is the runtime foundation of the plugin architecture. The verified tree-sitter WASM work remains valuable, but its unsafe lifecycle properties change the deployment design: untrusted grammar and extractor execution runs in a supervised helper process, not inside the daemon or rayon parse pool. ## Why a process boundary is mandatory Verified constraints in tree-sitter 0.22.6: - a WASM lex function can loop forever; parser timeout checks occur outside the lex call and the public store API exposes no fuel/epoch control; - catch_unwind cannot catch a hang or stack exhaustion; - multiple grammars in one store share linear memory and indirect-function-table state; - WasmStore is not Send and must outlive parser use; - the current indexer creates parsers inside rayon work, while LanguagePlugin returns a Language by value; - ABI-incompatible grammar loading can appear successful until set_language; - Windows GNU and musl have unresolved runtime/support risks. A long-lived daemon cannot accept “one malformed file or grammar permanently consumes a worker” as degradation. OS-process termination is the initial correctness boundary even if future tree-sitter/wasmtime APIs make in-process interruption possible. ## Runtime topology Ship a dedicated code-index-plugin-host executable with the same release. The daemon/indexer owns a bounded supervisor: parent indexer eligibility and size checks package/capability lookup bounded request queue | v supervisor worker keyed by package digest deadline + memory/output accounting crash/restart/quarantine policy | v plugin-host child one package digest / one WASM store no inherited project descriptors no filesystem/network/environment capability grammar + declarative engine + optional extractor | v #76 fact response validated again in parent A worker never loads two package digests into one store. Package upgrade starts a new worker; it does not mutate a running worker. The parent sends file bytes and a project-relative logical path. The child does not open project paths. Standard input/output framing must not share the MCP server protocol or logs. Stderr is bounded diagnostic output and cannot block the child. ## Process containment Cross-platform requirements: - close or explicitly whitelist inherited handles/file descriptors; - sanitized environment with no credentials or project variables; - private working directory with no project checkout; - no network capability where the platform can enforce it; - memory ceiling and child-tree containment; - hard wall-clock deadline per request; - kill the entire worker process tree on timeout; - bounded startup and shutdown deadlines; - Windows Job Object or equivalent child-tree ownership; - Unix process-group and resource-limit ownership; - worker death never kills the daemon or corrupts active generation state. WASM still supplies deterministic guest isolation inside the child. The process boundary supplies killability and package-to-package isolation. Security claims must be stated per platform. If filesystem/network denial cannot be strongly enforced on a supported platform, the payload and documentation say so; do not call a sanitized environment a sandbox. ## Grammar loading The host validates before first parse: - grammar artifact digest matches the package; - tree-sitter ABI is within both package declaration and host-supported range; - language.version is checked explicitly before set_language; - expected exported language name exists; - grammar initialization, query compilation and empty-input parse complete within startup budgets; - a package with an external scanner passes a non-trivial fixture probe. Legacy dylink vs dylink.0, ABI 12/15 behavior and scanner-built artifacts each receive fixtures with stable rejection codes. Package tooling must provide a documented reproducible grammar build path. Requiring an undocumented Docker/emscripten ritual is not an ecosystem. The project may ship a builder image, but package format and runtime cannot depend on Docker being installed on the user machine. ## Extractor execution The host supports the #76 tiers: 1. Declarative query/rule engine. 2. Optional extractor.wasm using the stable fact ABI. Grammar and extractor may execute in the same per-package worker because the process is already the package trust boundary. They do not share state with another package. The exact tree representation supplied to extractor.wasm must be benchmarked before ABI freeze: - serialized bounded event stream; - host calls over opaque node handles; or - extractor linked into the same component/runtime. Raw C pointers, Rust layout and tree-sitter internal struct layout are forbidden ABI. Region extraction for embedded languages is orchestrated by the parent/supervisor. A wrapper package returns validated regions and source maps; the supervisor dispatches region bytes to the selected installed host-language component and remaps validated spans. Recursive embedding has a strict depth and total-byte budget. ## Scheduling and backpressure - global cap on live plugin-host processes; - per-project and per-package request concurrency caps; - bounded queue with cancellation; - package startup coalescing so N files do not start N workers; - idle worker retirement; - no plugin request executes on Tokio core workers; - slow plugin cannot starve compiled plugins or another project; - parse result ordering remains deterministic despite concurrent workers. The parent distinguishes timeout, worker crash, ABI rejection, resource refusal, invalid facts and user cancellation. Each has a stable reason code used by #80. ## Failure policy Per-file failure: - reject the complete component result; - emit a bounded diagnostic; - leave previous active generation visible; - continue other files up to an anomaly threshold. Per-package failure: - exponential restart backoff; - quarantine after a bounded crash/timeout count; - stop dispatching new work; - fail activation under #78; - keep previous generation active; - require explicit retry after package/config change or operator action. A one-off malformed project file must not permanently quarantine a sound grammar unless it repeatedly crashes the worker; parse errors returned normally are not crashes. ## Trust and installation A repository may request a digest but cannot install or execute it. Installation and capability approval occur in user-controlled state. Minimum flow: 1. plugin inspect/validate without execution (#76); 2. plugin install --sha256 <digest>; 3. plugin check in isolated host (#80); 4. project enable exact digest with explicit capabilities; 5. #78 builds and promotes a generation. No automatic remote download during indexing. Registry/network distribution can be layered above an explicit install command; the indexer operates on local immutable packages. ## Performance contract The earlier measurements remain baselines, not acceptance: - WASM parsing around 1.6–1.7x native; - substantial runtime dependency/build-size cost; - per-store startup/JIT and RSS costs. Measure: - helper cold and warm startup; - IPC cost at 1 KiB, 100 KiB and max file size; - grammar/extractor CPU; - peak RSS per worker and globally; - throughput under mixed native/dynamic load; - watcher latency; - crash/restart and timeout recovery latency. Hot built-ins may remain native, but shadow mode must prove the package path produces equivalent facts. Performance fast paths cannot use different extraction semantics. ## Tests Hostile components: - infinite grammar lexer; - infinite external scanner; - infinite extractor loop; - memory.grow bomb; - output/framing bomb; - stderr flood; - deep recursion/stack exhaustion; - abort/trap; - fork/child attempt where applicable; - malformed response and mid-frame exit. Lifecycle: - worker crash with queued requests; - timeout during watcher indexing; - daemon shutdown with live workers; - package upgrade while old worker is busy; - project A’s plugin hangs while project B remains usable; - process descriptor/handle inheritance audit; - Windows and musl runtime smoke tests, not link-only gates. ABI: - supported and unsupported grammar versions; - legacy dylink artifact; - external-scanner grammar; - declarative and executable extractor packages; - embedded-region dispatch and span remapping. ## Acceptance 1. An unknown grammar package loads and parses without rebuilding code-index. 2. A malicious infinite lexer is terminated inside the deadline; the daemon answers a health/tool request afterward. 3. Package A cannot read/write package B’s memory because they never share a worker/store. 4. The child has no project filesystem or network access under every platform claim we publish. 5. Windows and musl artifacts execute real dynamic grammar and trap/timeout smoke tests. 6. Bounded mixed-load performance stays within committed CPU/RSS/latency ceilings. 7. #78 can treat host failure as a failed pending generation while the prior generation stays queryable.

#	criterion	evidence
1	unknown grammar loads without rebuilding code-index	the XAML package is installed from a `.cip` by pinned digest and indexes `.xaml` with symbols; no code-index rebuild
2	infinite lexer terminated inside the deadline, daemon answers afterward	containment suite + the release job's own timeout/trap test
3	package A cannot touch package B's memory — never share a worker/store	one host process per package; `plugin-supervisor` owns the split
4	child has no project filesystem or network access on every platform we publish	`crates/plugin-supervisor/src/contain.rs` + `crates/plugin-host/tests/containment.rs`
5	Windows and musl execute real dynamic grammar and trap/timeout smoke, not link-only	met, and exceeded — see below
6	bounded mixed-load performance within committed ceilings	`crates/plugin-host/tests/mixed_load_bench.rs`
7	#78 can treat host failure as a failed pending generation while the prior stays queryable	generations + `packages_rejected_by_failed_generations` / `runtime_quarantined_packages`, which are deliberately separate fields

Rows
Columns

runtime: supervised, killable dynamic grammar and extractor host #79

Decision

Why a process boundary is mandatory

Runtime topology

Process containment

Grammar loading

Extractor execution

Scheduling and backpressure

Failure policy

Trust and installation

Performance contract

Tests

Acceptance

All seven acceptance criteria met — closing

Criterion 5 is stronger than the issue asked for

The isolation decision held up under adversarial use

Correction: this did NOT close, and the tracker was right to refuse

All seven criteria met. Closing on 7b3fc7c, CI runs 587 and 586 green.

Criterion 5 — Windows and musl execute a real grammar plus trap/timeout smoke

Criterion 4 — no filesystem or network access "under every platform claim we publish"

Two mutations survived, and both produced better tests

All seven criteria met. Closing on `7b3fc7c`, CI runs 587 and 586 green.