h-dv/code-index

Watch

Fork 0

plugin ABI: immutable package format and validated extraction-fact protocol #76

New issue

Closed

opened 2026-08-26 12:31:25 +02:00 by buildagent · 3 comments

buildagent commented

2026-08-26 12:31:25 +02:00

Member

Child of #75. This issue defines the contract between an immutable plugin package, the supervised runtime host, and the trusted indexer core.

It replaces the earlier proposal for “one node-kind/role table per language.” Declarative rules remain a supported extractor tier, but they are not the plugin architecture: complete language support may require a bounded executable extractor component.

Outcome

A third party can build a content-addressed package for a grammar and extractor code that did not exist when code-index was released. The core can validate the package without executing it, negotiate a versioned ABI with #79, and validate every emitted fact before it reaches the database.

Package layout

The canonical archive is deterministic and safe to unpack:

plugin.toml
grammar.wasm
queries/*.scm                 optional
rules/*.toml                  optional
extractor.wasm                optional
fixtures/**                   required for publish/conformance
LICENSE
checksums                     derived, not trusted input

Requirements:

normalized relative UTF-8 paths only; no absolute paths, dot-dot, symlinks or duplicate case-folded names;
deterministic file order, timestamps and compression settings;
maximum archive, expanded, per-file and file-count bounds;
content digest computed over a canonical length-prefixed encoding, not archive container bytes;
package id is reverse-domain/namespaced and language ids are scoped beneath it;
semantic version is metadata; the digest is identity;
unknown required fields fail closed, while explicitly optional extension blocks can be ignored by declared version rules.

Manifest contract

The manifest declares, at minimum:

package-format version;
required host ABI range;
package id/version/digest algorithm;
grammar artifact, exported language name and supported tree-sitter ABI;
one or more namespaced language ids;
ordered path claims: extensions, exact basenames and bounded globs;
extractor components and which claims invoke them;
language resolution profiles;
requested resolver and bridge capabilities;
resource ceilings at or below host maximums;
fixtures and expected fact projections;
package dependencies, if supported, pinned by digest.

No package may request ambient filesystem, environment, clock, randomness, subprocess or network access. If a future ABI adds a capability, absence remains the default.

Claim precedence is explicit and deterministic. A tie between packages is a configuration error, never “first filesystem listing wins.” Reordering claims changes package activation identity and is handled by #78.

Host ABI

Use a versioned, length-delimited binary protocol shared by declarative and executable extractors. The package never links to internal Rust types or SQLite schema.

The request contains only bounded data:

ABI version and request id;
package digest, component id and language id;
project-relative logical path;
file bytes already accepted by the parent eligibility and size gates;
selected claim id;
optional outer-region/source-map information for embedded extraction;
deterministic configuration values explicitly declared by the ABI.

The response contains extraction facts and diagnostics. It cannot contain database ids, resolved target ids, SQL, filesystem paths outside the logical file, or instructions to load another component.

The ABI must support cancellation by process termination. It must not require cleanup messages for correctness.

Extraction facts

The fact protocol covers:

Symbols

local fact id, unique within the response;
name, kind, signature and documentation under byte limits;
language id and component id;
byte and line/column spans;
visibility, defaulting to file-local;
optional parent local fact id, required to precede the child;
qualified-name evidence as structured components, not an unchecked prejoined string;
bounded roles/capability class from a closed ABI registry.

References

name and ref kind;
language id and component id;
source span;
qualified flag and structured qualifier evidence;
role bits such as type-position/data-member-only;
binding/receiver evidence where available;
no target symbol id and no “resolved” assertion.

Imports

source language/component;
structured module segments;
alias/binding information;
source span and import class.

Diagnostics

stable reason code;
bounded human detail;
component and source span when applicable;
severity limited to the host-defined taxonomy.

Parent-side validator

Every response is hostile input. Before database insertion, the core validates:

response byte and row budgets;
exact request/package/component identity;
UTF-8 and string limits;
spans within the supplied file or declared source-map region;
line/column consistency recomputed from bytes rather than trusted;
parent-before-child ordering, no cycles and valid local ids;
known kinds, ref kinds, roles and visibility;
language/component ownership;
capability use is a subset of the installed and project-approved grant;
declarative and executable extractors satisfy identical rules.

One invalid fact rejects the component result for that file. Partial insertion is forbidden. The file receives a stable plugin diagnostic and the previous active generation remains visible under #78.

Validation code belongs in a dependency-light crate fuzzable without starting a daemon, tree-sitter or SQLite.

Declarative extractor tier

Declarative extraction supports bounded operations only:

compiled tree queries/captures;
field/ancestor/descendant relations with depth limits;
literal and identifier selection;
fixed normalization operations from a closed registry;
container construction;
qualifier/receiver capture;
region extraction with a validated source map;
emission of facts defined above.

No arbitrary regex with unbounded backtracking, loops, recursion, user code or host callbacks. Query compilation and match counts have hard budgets.

The format must express the fields that affect resolver safety: visibility, qualified-vs-qualifier, roles, parent ordering and source language. Defaults are least authority: file visibility, searchable-only symbols, unresolved refs and no bridge capability.

Executable extractor tier

extractor.wasm exists for semantics a data table cannot honestly model: scope state, rebinding, framework conventions, macro/token handling or language-specific normalization.

It uses the same fact ABI. It does not receive raw tree-sitter pointers or host addresses. The host exposes a bounded tree/event representation or runs grammar plus extractor within the isolated worker. ABI representation and serialization costs must be measured before freezing v1.

An extractor cannot mint new language ids, kinds or capabilities at runtime; all are manifest-declared and install-time validated.

Language profile schema

The package format defines data consumed by #77:

identifier and case normalization;
qualified-name representation/separators;
type/member/container kind classes;
visibility semantics;
import and relative-module rules;
type-position and data-member roles;
allowed resolver pool/tier capabilities;
cross-language bridge requests.

Hardcoded language allowlists must be inventory-tested and migrated to either a core invariant or a profile field. A new dynamic language with an omitted required semantic must fail activation loudly, not silently lose a resolver tier.

Compatibility

Host and package negotiate an integer ABI plus feature bits.
New optional response fields are length-delimited and safely skippable.
Semantic changes require a new ABI or engine version included in #78’s generation identity.
The core supports at least the current and immediately previous stable ABI during a documented migration window.
Reason-code strings and fact enums are wire contracts from first release.
An old daemon that cannot report plugin state must produce an explicit unavailable disclosure through #80.

CLI surface owned here

code-index plugin inspect
code-index plugin digest
code-index plugin validate

These commands do not execute grammar or extractor code. Runtime checking belongs to #80 through the supervised host.

Tests

canonical archive reproducibility;
archive traversal/symlink/case-collision bombs;
unknown required/optional fields;
every malformed fact shape;
oversized strings, rows, trees and diagnostics;
span/source-map forgery;
parent cycles and forward parents;
capability escalation;
package/component/request identity mismatch;
ABI previous/current/next negotiation;
mutation gate ensuring every manifest field that changes interpretation changes the digest;
fuzz targets for archive parsing, manifest parsing, ABI decode and fact validation.

Acceptance

A package unknown to the compiled binary validates without code changes.
Both a declarative extractor and executable extractor emit the same fact protocol.
No malformed or over-capability response can reach database insertion.
Package identity changes for every grammar, extractor, rule, profile, claim-order or engine-input change.
The contract contains everything #79, #77 and #78 require without importing internal indexer structs.
One complete existing plugin’s fixture projection can be represented through this ABI, proving it is not markup-only.

Child of #75. This issue defines the contract between an immutable plugin package, the supervised runtime host, and the trusted indexer core. It replaces the earlier proposal for “one node-kind/role table per language.” Declarative rules remain a supported extractor tier, but they are not the plugin architecture: complete language support may require a bounded executable extractor component. ## Outcome A third party can build a content-addressed package for a grammar and extractor code that did not exist when code-index was released. The core can validate the package without executing it, negotiate a versioned ABI with #79, and validate every emitted fact before it reaches the database. ## Package layout The canonical archive is deterministic and safe to unpack: plugin.toml grammar.wasm queries/*.scm optional rules/*.toml optional extractor.wasm optional fixtures/** required for publish/conformance LICENSE checksums derived, not trusted input Requirements: - normalized relative UTF-8 paths only; no absolute paths, dot-dot, symlinks or duplicate case-folded names; - deterministic file order, timestamps and compression settings; - maximum archive, expanded, per-file and file-count bounds; - content digest computed over a canonical length-prefixed encoding, not archive container bytes; - package id is reverse-domain/namespaced and language ids are scoped beneath it; - semantic version is metadata; the digest is identity; - unknown required fields fail closed, while explicitly optional extension blocks can be ignored by declared version rules. ## Manifest contract The manifest declares, at minimum: - package-format version; - required host ABI range; - package id/version/digest algorithm; - grammar artifact, exported language name and supported tree-sitter ABI; - one or more namespaced language ids; - ordered path claims: extensions, exact basenames and bounded globs; - extractor components and which claims invoke them; - language resolution profiles; - requested resolver and bridge capabilities; - resource ceilings at or below host maximums; - fixtures and expected fact projections; - package dependencies, if supported, pinned by digest. No package may request ambient filesystem, environment, clock, randomness, subprocess or network access. If a future ABI adds a capability, absence remains the default. Claim precedence is explicit and deterministic. A tie between packages is a configuration error, never “first filesystem listing wins.” Reordering claims changes package activation identity and is handled by #78. ## Host ABI Use a versioned, length-delimited binary protocol shared by declarative and executable extractors. The package never links to internal Rust types or SQLite schema. The request contains only bounded data: - ABI version and request id; - package digest, component id and language id; - project-relative logical path; - file bytes already accepted by the parent eligibility and size gates; - selected claim id; - optional outer-region/source-map information for embedded extraction; - deterministic configuration values explicitly declared by the ABI. The response contains extraction facts and diagnostics. It cannot contain database ids, resolved target ids, SQL, filesystem paths outside the logical file, or instructions to load another component. The ABI must support cancellation by process termination. It must not require cleanup messages for correctness. ## Extraction facts The fact protocol covers: ### Symbols - local fact id, unique within the response; - name, kind, signature and documentation under byte limits; - language id and component id; - byte and line/column spans; - visibility, defaulting to file-local; - optional parent local fact id, required to precede the child; - qualified-name evidence as structured components, not an unchecked prejoined string; - bounded roles/capability class from a closed ABI registry. ### References - name and ref kind; - language id and component id; - source span; - qualified flag and structured qualifier evidence; - role bits such as type-position/data-member-only; - binding/receiver evidence where available; - no target symbol id and no “resolved” assertion. ### Imports - source language/component; - structured module segments; - alias/binding information; - source span and import class. ### Diagnostics - stable reason code; - bounded human detail; - component and source span when applicable; - severity limited to the host-defined taxonomy. ## Parent-side validator Every response is hostile input. Before database insertion, the core validates: - response byte and row budgets; - exact request/package/component identity; - UTF-8 and string limits; - spans within the supplied file or declared source-map region; - line/column consistency recomputed from bytes rather than trusted; - parent-before-child ordering, no cycles and valid local ids; - known kinds, ref kinds, roles and visibility; - language/component ownership; - capability use is a subset of the installed and project-approved grant; - declarative and executable extractors satisfy identical rules. One invalid fact rejects the component result for that file. Partial insertion is forbidden. The file receives a stable plugin diagnostic and the previous active generation remains visible under #78. Validation code belongs in a dependency-light crate fuzzable without starting a daemon, tree-sitter or SQLite. ## Declarative extractor tier Declarative extraction supports bounded operations only: - compiled tree queries/captures; - field/ancestor/descendant relations with depth limits; - literal and identifier selection; - fixed normalization operations from a closed registry; - container construction; - qualifier/receiver capture; - region extraction with a validated source map; - emission of facts defined above. No arbitrary regex with unbounded backtracking, loops, recursion, user code or host callbacks. Query compilation and match counts have hard budgets. The format must express the fields that affect resolver safety: visibility, qualified-vs-qualifier, roles, parent ordering and source language. Defaults are least authority: file visibility, searchable-only symbols, unresolved refs and no bridge capability. ## Executable extractor tier extractor.wasm exists for semantics a data table cannot honestly model: scope state, rebinding, framework conventions, macro/token handling or language-specific normalization. It uses the same fact ABI. It does not receive raw tree-sitter pointers or host addresses. The host exposes a bounded tree/event representation or runs grammar plus extractor within the isolated worker. ABI representation and serialization costs must be measured before freezing v1. An extractor cannot mint new language ids, kinds or capabilities at runtime; all are manifest-declared and install-time validated. ## Language profile schema The package format defines data consumed by #77: - identifier and case normalization; - qualified-name representation/separators; - type/member/container kind classes; - visibility semantics; - import and relative-module rules; - type-position and data-member roles; - allowed resolver pool/tier capabilities; - cross-language bridge requests. Hardcoded language allowlists must be inventory-tested and migrated to either a core invariant or a profile field. A new dynamic language with an omitted required semantic must fail activation loudly, not silently lose a resolver tier. ## Compatibility - Host and package negotiate an integer ABI plus feature bits. - New optional response fields are length-delimited and safely skippable. - Semantic changes require a new ABI or engine version included in #78’s generation identity. - The core supports at least the current and immediately previous stable ABI during a documented migration window. - Reason-code strings and fact enums are wire contracts from first release. - An old daemon that cannot report plugin state must produce an explicit unavailable disclosure through #80. ## CLI surface owned here - code-index plugin inspect <archive> - code-index plugin digest <archive> - code-index plugin validate <archive> These commands do not execute grammar or extractor code. Runtime checking belongs to #80 through the supervised host. ## Tests - canonical archive reproducibility; - archive traversal/symlink/case-collision bombs; - unknown required/optional fields; - every malformed fact shape; - oversized strings, rows, trees and diagnostics; - span/source-map forgery; - parent cycles and forward parents; - capability escalation; - package/component/request identity mismatch; - ABI previous/current/next negotiation; - mutation gate ensuring every manifest field that changes interpretation changes the digest; - fuzz targets for archive parsing, manifest parsing, ABI decode and fact validation. ## Acceptance 1. A package unknown to the compiled binary validates without code changes. 2. Both a declarative extractor and executable extractor emit the same fact protocol. 3. No malformed or over-capability response can reach database insertion. 4. Package identity changes for every grammar, extractor, rule, profile, claim-order or engine-input change. 5. The contract contains everything #79, #77 and #78 require without importing internal indexer structs. 6. One complete existing plugin’s fixture projection can be represented through this ABI, proving it is not markup-only.

#	criterion	verdict	evidence
1	an unknown package validates without code changes	MET	`crates/cli/src/plugin.rs:140/147/149` (`Inspect`/`Validate`/`Digest`); XAML and Ruby both install by pinned digest
2	both a declarative and an executable extractor emit the same fact protocol	NOT MET	`crates/indexer/src/packages.rs:1537` refuses `Tier::Declarative` outright
3	no malformed or over-capability response reaches DB insertion	MET	`crates/abi/src/record.rs:575 validate()`, all-or-nothing; `Grants::default` denies; `crates/abi/tests/abi_fuzz.rs` exhaustive corruption
4	identity changes for every interpretation input	MET	`extraction_identity.rs::every_manifest_field_moves_exactly_the_digests_it_should`, `claim_order_is_identity`; `digest_stability.rs::every_byte_of_every_entry_moves_the_digest`
5	contract complete without importing indexer structs	MET	`crates/abi/src/lib.rs:161 dependency_gate::this_crate_takes_no_dependencies`
6	one complete existing plugin, proving not markup-only	MET IN THE WORKING TREE ONLY	`tests/packages/ruby/` + parity on all three axes — all uncommitted

Rows
Columns

plugin ABI: immutable package format and validated extraction-fact protocol #76

Outcome

Package layout

Manifest contract

Host ABI

Extraction facts

Symbols

References

Imports

Diagnostics

Parent-side validator

Declarative extractor tier

Executable extractor tier

Language profile schema

Compatibility

CLI surface owned here

Tests

Acceptance

Audit: 4 of 6 criteria met, and this issue is now the single formal blocker for #77.

C2 — unmet by construction, and the tree is honest about it

C6 — the part that changed today

Also unmet, from the manifest contract

Status

C2 is retired by decision, not left unmet. Amending this issue by comment rather than editing it away.

The retirement is NOT what this issue's audit comment literally proposed, and there is a measurement for why

Where the rule could not go, and why that matters

Two doors, not three — and the third was measured, not assumed

Compatibility proved from real pre-change digests

The retired reason code

What went wrong in the doing, recorded

Remaining on this issue

C6 is met on master, and CI is green. Closing.

What C6 actually rests on now

One thing that was true when this issue was last updated and is not true now

Remaining, and split out rather than swept