feat(distribution): unified in-tree registry and Forgejo distribution for binaries and plugin packages #269

New issue

Open

opened 2026-09-12 18:16:04 +02:00 by buildagent · 0 comments

buildagent commented

2026-09-12 18:16:04 +02:00

Member

Unified In-Tree Registry and Forgejo Distribution for Binaries and Plugin Packages

1. Problem Statement & Motivation

Today, code-index has two decoupled distribution channels that share the same release runner and Forgejo forge (git.h-dv.de/h-dv/code-index):

Core Binaries: Distributed via release archives (code-index-${TAG}-${slug}.tar.gz) and managed by install.sh using GitHub/Forgejo release API lookups.
Plugin Packages: Distributed as standalone .cip containers and .cips detached signature sidecars, configured manually via code-index plugin add <url> --sha256 <hex> or copied to .code-index/plugins/.

The Friction for Operators and AI Agents

When an operator or AI agent opens a project containing unindexed source files (e.g. .svelte, .xaml):

project_overview correctly measures and reports symbol_blind_extensions: [".svelte"].
If the package is not already present in the local machine store (Store::installed), activation_available has no offer because the store is local-only.
The agent cannot programmatically resolve:
1. What plugin package exists for .svelte?
2. Where is it hosted on Forgejo?
3. What is its current SHA-256 digest, publisher fingerprint, and ABI compatibility?
Furthermore, configuring [update."<package_id>"] in .code-index.toml currently requires manually pasting raw release URLs that may rot across tag rotations.

By bringing a unified distribution catalog directly in-tree and leveraging Forgejo's built-in Releases and Generic Package Registry, code-index can publish, discover, install, and update both core binaries and plugin packages in one cohesive, cryptographically verified mechanism.

2. Technical Architecture

2.1 In-Tree Registry Catalog Schema (`distribution/registry.v1.json`)

A versioned, machine-readable catalog generated and signed during the CI release workflow and published to Forgejo:

{
  "$schema": "https://git.h-dv.de/h-dv/code-index/raw/branch/master/distribution/schema.v1.json",
  "schema_version": 1,
  "generated_at_epoch_seconds": 1789230000,
  "code_index": {
    "version": "0.28.2",
    "tag": "v0.28.2",
    "binaries": [
      "code-index",
      "code-index-daemon",
      "code-index-mcp",
      "code-index-plugin-host"
    ],
    "platforms": {
      "linux-x86_64": {
        "archive": "code-index-v0.28.2-linux-x86_64.tar.gz",
        "sha256": "4f8a2b...",
        "libc": "glibc",
        "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-linux-x86_64.tar.gz"
      },
      "linux-aarch64": {
        "archive": "code-index-v0.28.2-linux-aarch64.tar.gz",
        "sha256": "8b2c1d...",
        "libc": "glibc",
        "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-linux-aarch64.tar.gz"
      },
      "linux-x86_64-musl": {
        "archive": "code-index-v0.28.2-linux-x86_64-musl.tar.gz",
        "sha256": "9c3d4e...",
        "libc": "musl",
        "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-linux-x86_64-musl.tar.gz"
      },
      "windows-x86_64": {
        "archive": "code-index-v0.28.2-windows-x86_64.zip",
        "sha256": "5e6f7a...",
        "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-windows-x86_64.zip"
      }
    }
  },
  "plugins": {
    "de.h-dv.xaml": {
      "id": "de.h-dv.xaml",
      "name": "XAML Extractor",
      "claims": [".xaml"],
      "publisher": {
        "name": "h-dv (first party)",
        "fingerprint": "10a480436ce..."
      },
      "latest_version": "0.1.0",
      "versions": {
        "0.1.0": {
          "package_digest": "sha256:7a8b9c...",
          "extraction_identity": "sha256:1a2b3c...",
          "min_code_index_version": "0.28.0",
          "archive": "de.h-dv.xaml-0.1.0.cip",
          "signature": "de.h-dv.xaml-0.1.0.cips",
          "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/de.h-dv.xaml-0.1.0.cip",
          "signature_url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/de.h-dv.xaml-0.1.0.cips",
          "capabilities": {
            "resolver": ["bridge_source"],
            "bridges": ["de.h-dv.xaml/xaml:type->csharp:class"]
          }
        }
      }
    }
  }
}

3. Forgejo CI/CD Release Pipeline Integration (`.forgejo/workflows/release.yml`)

The existing release.yml workflow already builds all target binary archives and packages/signs in-tree plugins. The workflow will be extended in the final release job:

Manifest Synthesis:
- Aggregate checksums from the 4 platform builds (shipped/code-index-${TAG}-*.tar.gz.sha256).
- Read digests and identities from packed plugins (*.cip.digest.txt).
- Assemble distribution-manifest.json for the specific release tag and update the cumulative registry.json.
Cryptographic Signing of the Manifest:
- Sign registry.json using CODE_INDEX_PUBLISHER_KEY to produce registry.json.sig (Ed25519 detached signature).
- This ensures the catalog metadata cannot be forged or tampered with in-flight.
Multi-Destination Publication on Forgejo:
- Attach registry.json and registry.json.sig to the Forgejo Release assets.
- Publish to Forgejo Generic Package Registry:
  https://git.h-dv.de/api/packages/h-dv/generic/code-index-distribution/latest/registry.json
- Commit the updated distribution/registry.json back to master (or publish via Forgejo Pages).

4. Tooling & Client Integration

4.1 Enhanced `install.sh`

Reads registry.json directly from the release or Forgejo generic package endpoint.
Verifies registry.json.sig against the compiled-in first-party public key.
Validates binary archive checksums without separate HTTP requests.

Adds support for plugin bundling:

sh install.sh --with-plugin de.h-dv.xaml

4.2 CLI `code-index plugin` Enhancements

Direct Add by ID:
code-index plugin add de.h-dv.xaml queries the registry catalog, automatically resolves the URL, sha256, and detached signature, and proceeds through standard verification and operator elicitation.
Streamlined Updates:
In .code-index.toml:
```
[update."de.h-dv.xaml"]
source     = "registry"
auto_apply = true
```
plugin update automatically resolves the newest release from the registry, enforcing all 7 safety gates (strict semver ordering #255, subset grants, unchanged extraction identity).

4.3 Agent & MCP Server Integration

Expose Registry as MCP Resource: Expose cosi://registry/plugins to allow AI agents to browse available plugins.

Intelligent Overview Warnings: When project_overview detects symbol_blind_extensions: [".xaml"], it checks the cached registry and includes:

"registry_available": [
  {
    "package_id": "de.h-dv.xaml",
    "claims": [".xaml"],
    "latest_version": "0.1.0",
    "action_hint": "Call `plugin_add` with `package: \"de.h-dv.xaml\"`"
  }
]

One-Step MCP Install: Allow plugin_add to accept package: "<id>", fetching the .cip and .cips from the registry and prompting the human operator via MCP elicitation.

5. Security & Threat Model Compliance

Transport Carries No Trust:
- As established in _prdoc/guides/80-threat-model.md, HTTP carries no trust.
- Binaries are verified via published SHA-256 checksums before unpacking.
- Packages are verified via detached Ed25519 signatures before ingestion.
- The registry catalog itself is signed with the publisher key.
First-Grant Rule Maintained:
- Even with a central registry, the first grant of a package on any machine remains a human decision requiring interactive confirmation.
- Unattended auto-apply continues to strictly enforce capability parity (C_{\text{new}} \subseteq C_{\text{in\_force}}).
Air-gapped & Mirror Support:
- install.sh --base-url <url|path> and code-index --registry-url <url|path> allow air-gapped environments or local disk mirrors to function identically without external network access.

6. Required Quality Gates & Tests

Per IXT V2 and code-index standards:

installer_targets.rs extension: Grade that all published binary targets in .forgejo/workflows/release.yml appear in registry.json.
registry_schema_gate.rs: Validate distribution/registry.v1.json against its JSON schema.
plugin_registry_e2e.rs: End-to-end integration test over loopback mock HTTP server testing registry resolution, signature checking, tampered catalog rejection, and unattended update.
Mutation testing: Grade every rejection branch (tampered signature, invalid checksum, unorderable semver).

# Unified In-Tree Registry and Forgejo Distribution for Binaries and Plugin Packages ## 1. Problem Statement & Motivation Today, `code-index` has two decoupled distribution channels that share the same release runner and Forgejo forge (`git.h-dv.de/h-dv/code-index`): 1. **Core Binaries**: Distributed via release archives (`code-index-${TAG}-${slug}.tar.gz`) and managed by `install.sh` using GitHub/Forgejo release API lookups. 2. **Plugin Packages**: Distributed as standalone `.cip` containers and `.cips` detached signature sidecars, configured manually via `code-index plugin add <url> --sha256 <hex>` or copied to `.code-index/plugins/`. ### The Friction for Operators and AI Agents When an operator or AI agent opens a project containing unindexed source files (e.g. `.svelte`, `.xaml`): * `project_overview` correctly measures and reports `symbol_blind_extensions: [".svelte"]`. * If the package is not already present in the local machine store (`Store::installed`), `activation_available` has no offer because the store is local-only. * The agent cannot programmatically resolve: 1. What plugin package exists for `.svelte`? 2. Where is it hosted on Forgejo? 3. What is its current SHA-256 digest, publisher fingerprint, and ABI compatibility? * Furthermore, configuring `[update."<package_id>"]` in `.code-index.toml` currently requires manually pasting raw release URLs that may rot across tag rotations. By bringing a **unified distribution catalog directly in-tree** and leveraging Forgejo's built-in **Releases and Generic Package Registry**, `code-index` can publish, discover, install, and update both core binaries and plugin packages in one cohesive, cryptographically verified mechanism. --- ## 2. Technical Architecture ### 2.1 In-Tree Registry Catalog Schema (`distribution/registry.v1.json`) A versioned, machine-readable catalog generated and signed during the CI release workflow and published to Forgejo: ```json { "$schema": "https://git.h-dv.de/h-dv/code-index/raw/branch/master/distribution/schema.v1.json", "schema_version": 1, "generated_at_epoch_seconds": 1789230000, "code_index": { "version": "0.28.2", "tag": "v0.28.2", "binaries": [ "code-index", "code-index-daemon", "code-index-mcp", "code-index-plugin-host" ], "platforms": { "linux-x86_64": { "archive": "code-index-v0.28.2-linux-x86_64.tar.gz", "sha256": "4f8a2b...", "libc": "glibc", "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-linux-x86_64.tar.gz" }, "linux-aarch64": { "archive": "code-index-v0.28.2-linux-aarch64.tar.gz", "sha256": "8b2c1d...", "libc": "glibc", "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-linux-aarch64.tar.gz" }, "linux-x86_64-musl": { "archive": "code-index-v0.28.2-linux-x86_64-musl.tar.gz", "sha256": "9c3d4e...", "libc": "musl", "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-linux-x86_64-musl.tar.gz" }, "windows-x86_64": { "archive": "code-index-v0.28.2-windows-x86_64.zip", "sha256": "5e6f7a...", "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-windows-x86_64.zip" } } }, "plugins": { "de.h-dv.xaml": { "id": "de.h-dv.xaml", "name": "XAML Extractor", "claims": [".xaml"], "publisher": { "name": "h-dv (first party)", "fingerprint": "10a480436ce..." }, "latest_version": "0.1.0", "versions": { "0.1.0": { "package_digest": "sha256:7a8b9c...", "extraction_identity": "sha256:1a2b3c...", "min_code_index_version": "0.28.0", "archive": "de.h-dv.xaml-0.1.0.cip", "signature": "de.h-dv.xaml-0.1.0.cips", "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/de.h-dv.xaml-0.1.0.cip", "signature_url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/de.h-dv.xaml-0.1.0.cips", "capabilities": { "resolver": ["bridge_source"], "bridges": ["de.h-dv.xaml/xaml:type->csharp:class"] } } } } } } ``` --- ## 3. Forgejo CI/CD Release Pipeline Integration (`.forgejo/workflows/release.yml`) The existing `release.yml` workflow already builds all target binary archives and packages/signs in-tree plugins. The workflow will be extended in the final `release` job: 1. **Manifest Synthesis**: - Aggregate checksums from the 4 platform builds (`shipped/code-index-${TAG}-*.tar.gz.sha256`). - Read digests and identities from packed plugins (`*.cip.digest.txt`). - Assemble `distribution-manifest.json` for the specific release tag and update the cumulative `registry.json`. 2. **Cryptographic Signing of the Manifest**: - Sign `registry.json` using `CODE_INDEX_PUBLISHER_KEY` to produce `registry.json.sig` (Ed25519 detached signature). - This ensures the catalog metadata cannot be forged or tampered with in-flight. 3. **Multi-Destination Publication on Forgejo**: - Attach `registry.json` and `registry.json.sig` to the Forgejo Release assets. - Publish to Forgejo Generic Package Registry: `https://git.h-dv.de/api/packages/h-dv/generic/code-index-distribution/latest/registry.json` - Commit the updated `distribution/registry.json` back to `master` (or publish via Forgejo Pages). --- ## 4. Tooling & Client Integration ### 4.1 Enhanced `install.sh` * Reads `registry.json` directly from the release or Forgejo generic package endpoint. * Verifies `registry.json.sig` against the compiled-in first-party public key. * Validates binary archive checksums without separate HTTP requests. * Adds support for plugin bundling: ```bash sh install.sh --with-plugin de.h-dv.xaml ``` ### 4.2 CLI `code-index plugin` Enhancements * **Direct Add by ID**: `code-index plugin add de.h-dv.xaml` queries the registry catalog, automatically resolves the URL, sha256, and detached signature, and proceeds through standard verification and operator elicitation. * **Streamlined Updates**: In `.code-index.toml`: ```toml [update."de.h-dv.xaml"] source = "registry" auto_apply = true ``` `plugin update` automatically resolves the newest release from the registry, enforcing all 7 safety gates (strict semver ordering #255, subset grants, unchanged extraction identity). ### 4.3 Agent & MCP Server Integration * **Expose Registry as MCP Resource**: Expose `cosi://registry/plugins` to allow AI agents to browse available plugins. * **Intelligent Overview Warnings**: When `project_overview` detects `symbol_blind_extensions: [".xaml"]`, it checks the cached registry and includes: ```json "registry_available": [ { "package_id": "de.h-dv.xaml", "claims": [".xaml"], "latest_version": "0.1.0", "action_hint": "Call `plugin_add` with `package: \"de.h-dv.xaml\"`" } ] ``` * **One-Step MCP Install**: Allow `plugin_add` to accept `package: "<id>"`, fetching the `.cip` and `.cips` from the registry and prompting the human operator via MCP elicitation. --- ## 5. Security & Threat Model Compliance 1. **Transport Carries No Trust**: - As established in `_prdoc/guides/80-threat-model.md`, HTTP carries no trust. - Binaries are verified via published SHA-256 checksums before unpacking. - Packages are verified via detached Ed25519 signatures before ingestion. - The registry catalog itself is signed with the publisher key. 2. **First-Grant Rule Maintained**: - Even with a central registry, the first grant of a package on any machine **remains a human decision** requiring interactive confirmation. - Unattended auto-apply continues to strictly enforce capability parity ($C_{\text{new}} \subseteq C_{\text{in\_force}}$). 3. **Air-gapped & Mirror Support**: - `install.sh --base-url <url|path>` and `code-index --registry-url <url|path>` allow air-gapped environments or local disk mirrors to function identically without external network access. --- ## 6. Required Quality Gates & Tests Per IXT V2 and `code-index` standards: 1. `installer_targets.rs` extension: Grade that all published binary targets in `.forgejo/workflows/release.yml` appear in `registry.json`. 2. `registry_schema_gate.rs`: Validate `distribution/registry.v1.json` against its JSON schema. 3. `plugin_registry_e2e.rs`: End-to-end integration test over loopback mock HTTP server testing registry resolution, signature checking, tampered catalog rejection, and unattended update. 4. Mutation testing: Grade every rejection branch (tampered signature, invalid checksum, unorderable semver).

buildagent referenced this issue from a commit

2026-09-13 11:16:36 +02:00

release: prepare v0.28.3 with distribution registry (#269)

buildagent referenced this issue

2026-09-13 17:32:35 +02:00

Receiver-binding phantom: a read of pkg.digest binds a same-named pub field in another crate, and localising the receiver does not move it #270

buildagent referenced this issue

2026-09-13 20:14:28 +02:00

Fork-lock gates cannot reach the third executable-writing door (File::create + write! + chmod), and plugin-host/tests/kill_cost.rs is a live member #271

buildagent referenced this issue from a commit

2026-09-13 20:45:40 +02:00

fix(distribution): make the catalog derive every value it states (#269)