feat(distribution): unified in-tree registry and Forgejo distribution for binaries and plugin packages #269

Closed
opened 2026-09-12 18:16:04 +02:00 by buildagent · 1 comment
Member

Unified In-Tree Registry and Forgejo Distribution for Binaries and Plugin Packages

1. Problem Statement & Motivation

Today, code-index has two decoupled distribution channels that share the same release runner and Forgejo forge (git.h-dv.de/h-dv/code-index):

  1. Core Binaries: Distributed via release archives (code-index-${TAG}-${slug}.tar.gz) and managed by install.sh using GitHub/Forgejo release API lookups.
  2. Plugin Packages: Distributed as standalone .cip containers and .cips detached signature sidecars, configured manually via code-index plugin add <url> --sha256 <hex> or copied to .code-index/plugins/.

The Friction for Operators and AI Agents

When an operator or AI agent opens a project containing unindexed source files (e.g. .svelte, .xaml):

  • project_overview correctly measures and reports symbol_blind_extensions: [".svelte"].
  • If the package is not already present in the local machine store (Store::installed), activation_available has no offer because the store is local-only.
  • The agent cannot programmatically resolve:
    1. What plugin package exists for .svelte?
    2. Where is it hosted on Forgejo?
    3. What is its current SHA-256 digest, publisher fingerprint, and ABI compatibility?
  • Furthermore, configuring [update."<package_id>"] in .code-index.toml currently requires manually pasting raw release URLs that may rot across tag rotations.

By bringing a unified distribution catalog directly in-tree and leveraging Forgejo's built-in Releases and Generic Package Registry, code-index can publish, discover, install, and update both core binaries and plugin packages in one cohesive, cryptographically verified mechanism.


2. Technical Architecture

2.1 In-Tree Registry Catalog Schema (distribution/registry.v1.json)

A versioned, machine-readable catalog generated and signed during the CI release workflow and published to Forgejo:

{
  "$schema": "https://git.h-dv.de/h-dv/code-index/raw/branch/master/distribution/schema.v1.json",
  "schema_version": 1,
  "generated_at_epoch_seconds": 1789230000,
  "code_index": {
    "version": "0.28.2",
    "tag": "v0.28.2",
    "binaries": [
      "code-index",
      "code-index-daemon",
      "code-index-mcp",
      "code-index-plugin-host"
    ],
    "platforms": {
      "linux-x86_64": {
        "archive": "code-index-v0.28.2-linux-x86_64.tar.gz",
        "sha256": "4f8a2b...",
        "libc": "glibc",
        "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-linux-x86_64.tar.gz"
      },
      "linux-aarch64": {
        "archive": "code-index-v0.28.2-linux-aarch64.tar.gz",
        "sha256": "8b2c1d...",
        "libc": "glibc",
        "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-linux-aarch64.tar.gz"
      },
      "linux-x86_64-musl": {
        "archive": "code-index-v0.28.2-linux-x86_64-musl.tar.gz",
        "sha256": "9c3d4e...",
        "libc": "musl",
        "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-linux-x86_64-musl.tar.gz"
      },
      "windows-x86_64": {
        "archive": "code-index-v0.28.2-windows-x86_64.zip",
        "sha256": "5e6f7a...",
        "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-windows-x86_64.zip"
      }
    }
  },
  "plugins": {
    "de.h-dv.xaml": {
      "id": "de.h-dv.xaml",
      "name": "XAML Extractor",
      "claims": [".xaml"],
      "publisher": {
        "name": "h-dv (first party)",
        "fingerprint": "10a480436ce..."
      },
      "latest_version": "0.1.0",
      "versions": {
        "0.1.0": {
          "package_digest": "sha256:7a8b9c...",
          "extraction_identity": "sha256:1a2b3c...",
          "min_code_index_version": "0.28.0",
          "archive": "de.h-dv.xaml-0.1.0.cip",
          "signature": "de.h-dv.xaml-0.1.0.cips",
          "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/de.h-dv.xaml-0.1.0.cip",
          "signature_url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/de.h-dv.xaml-0.1.0.cips",
          "capabilities": {
            "resolver": ["bridge_source"],
            "bridges": ["de.h-dv.xaml/xaml:type->csharp:class"]
          }
        }
      }
    }
  }
}

3. Forgejo CI/CD Release Pipeline Integration (.forgejo/workflows/release.yml)

The existing release.yml workflow already builds all target binary archives and packages/signs in-tree plugins. The workflow will be extended in the final release job:

  1. Manifest Synthesis:
    • Aggregate checksums from the 4 platform builds (shipped/code-index-${TAG}-*.tar.gz.sha256).
    • Read digests and identities from packed plugins (*.cip.digest.txt).
    • Assemble distribution-manifest.json for the specific release tag and update the cumulative registry.json.
  2. Cryptographic Signing of the Manifest:
    • Sign registry.json using CODE_INDEX_PUBLISHER_KEY to produce registry.json.sig (Ed25519 detached signature).
    • This ensures the catalog metadata cannot be forged or tampered with in-flight.
  3. Multi-Destination Publication on Forgejo:
    • Attach registry.json and registry.json.sig to the Forgejo Release assets.
    • Publish to Forgejo Generic Package Registry:
      https://git.h-dv.de/api/packages/h-dv/generic/code-index-distribution/latest/registry.json
    • Commit the updated distribution/registry.json back to master (or publish via Forgejo Pages).

4. Tooling & Client Integration

4.1 Enhanced install.sh

  • Reads registry.json directly from the release or Forgejo generic package endpoint.
  • Verifies registry.json.sig against the compiled-in first-party public key.
  • Validates binary archive checksums without separate HTTP requests.
  • Adds support for plugin bundling:
    sh install.sh --with-plugin de.h-dv.xaml
    

4.2 CLI code-index plugin Enhancements

  • Direct Add by ID:
    code-index plugin add de.h-dv.xaml queries the registry catalog, automatically resolves the URL, sha256, and detached signature, and proceeds through standard verification and operator elicitation.
  • Streamlined Updates:
    In .code-index.toml:
    [update."de.h-dv.xaml"]
    source     = "registry"
    auto_apply = true
    
    plugin update automatically resolves the newest release from the registry, enforcing all 7 safety gates (strict semver ordering #255, subset grants, unchanged extraction identity).

4.3 Agent & MCP Server Integration

  • Expose Registry as MCP Resource: Expose cosi://registry/plugins to allow AI agents to browse available plugins.
  • Intelligent Overview Warnings: When project_overview detects symbol_blind_extensions: [".xaml"], it checks the cached registry and includes:
    "registry_available": [
      {
        "package_id": "de.h-dv.xaml",
        "claims": [".xaml"],
        "latest_version": "0.1.0",
        "action_hint": "Call `plugin_add` with `package: \"de.h-dv.xaml\"`"
      }
    ]
    
  • One-Step MCP Install: Allow plugin_add to accept package: "<id>", fetching the .cip and .cips from the registry and prompting the human operator via MCP elicitation.

5. Security & Threat Model Compliance

  1. Transport Carries No Trust:
    • As established in _prdoc/guides/80-threat-model.md, HTTP carries no trust.
    • Binaries are verified via published SHA-256 checksums before unpacking.
    • Packages are verified via detached Ed25519 signatures before ingestion.
    • The registry catalog itself is signed with the publisher key.
  2. First-Grant Rule Maintained:
    • Even with a central registry, the first grant of a package on any machine remains a human decision requiring interactive confirmation.
    • Unattended auto-apply continues to strictly enforce capability parity (C_{\text{new}} \subseteq C_{\text{in\_force}}).
  3. Air-gapped & Mirror Support:
    • install.sh --base-url <url|path> and code-index --registry-url <url|path> allow air-gapped environments or local disk mirrors to function identically without external network access.

6. Required Quality Gates & Tests

Per IXT V2 and code-index standards:

  1. installer_targets.rs extension: Grade that all published binary targets in .forgejo/workflows/release.yml appear in registry.json.
  2. registry_schema_gate.rs: Validate distribution/registry.v1.json against its JSON schema.
  3. plugin_registry_e2e.rs: End-to-end integration test over loopback mock HTTP server testing registry resolution, signature checking, tampered catalog rejection, and unattended update.
  4. Mutation testing: Grade every rejection branch (tampered signature, invalid checksum, unorderable semver).
# Unified In-Tree Registry and Forgejo Distribution for Binaries and Plugin Packages ## 1. Problem Statement & Motivation Today, `code-index` has two decoupled distribution channels that share the same release runner and Forgejo forge (`git.h-dv.de/h-dv/code-index`): 1. **Core Binaries**: Distributed via release archives (`code-index-${TAG}-${slug}.tar.gz`) and managed by `install.sh` using GitHub/Forgejo release API lookups. 2. **Plugin Packages**: Distributed as standalone `.cip` containers and `.cips` detached signature sidecars, configured manually via `code-index plugin add <url> --sha256 <hex>` or copied to `.code-index/plugins/`. ### The Friction for Operators and AI Agents When an operator or AI agent opens a project containing unindexed source files (e.g. `.svelte`, `.xaml`): * `project_overview` correctly measures and reports `symbol_blind_extensions: [".svelte"]`. * If the package is not already present in the local machine store (`Store::installed`), `activation_available` has no offer because the store is local-only. * The agent cannot programmatically resolve: 1. What plugin package exists for `.svelte`? 2. Where is it hosted on Forgejo? 3. What is its current SHA-256 digest, publisher fingerprint, and ABI compatibility? * Furthermore, configuring `[update."<package_id>"]` in `.code-index.toml` currently requires manually pasting raw release URLs that may rot across tag rotations. By bringing a **unified distribution catalog directly in-tree** and leveraging Forgejo's built-in **Releases and Generic Package Registry**, `code-index` can publish, discover, install, and update both core binaries and plugin packages in one cohesive, cryptographically verified mechanism. --- ## 2. Technical Architecture ### 2.1 In-Tree Registry Catalog Schema (`distribution/registry.v1.json`) A versioned, machine-readable catalog generated and signed during the CI release workflow and published to Forgejo: ```json { "$schema": "https://git.h-dv.de/h-dv/code-index/raw/branch/master/distribution/schema.v1.json", "schema_version": 1, "generated_at_epoch_seconds": 1789230000, "code_index": { "version": "0.28.2", "tag": "v0.28.2", "binaries": [ "code-index", "code-index-daemon", "code-index-mcp", "code-index-plugin-host" ], "platforms": { "linux-x86_64": { "archive": "code-index-v0.28.2-linux-x86_64.tar.gz", "sha256": "4f8a2b...", "libc": "glibc", "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-linux-x86_64.tar.gz" }, "linux-aarch64": { "archive": "code-index-v0.28.2-linux-aarch64.tar.gz", "sha256": "8b2c1d...", "libc": "glibc", "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-linux-aarch64.tar.gz" }, "linux-x86_64-musl": { "archive": "code-index-v0.28.2-linux-x86_64-musl.tar.gz", "sha256": "9c3d4e...", "libc": "musl", "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-linux-x86_64-musl.tar.gz" }, "windows-x86_64": { "archive": "code-index-v0.28.2-windows-x86_64.zip", "sha256": "5e6f7a...", "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/code-index-v0.28.2-windows-x86_64.zip" } } }, "plugins": { "de.h-dv.xaml": { "id": "de.h-dv.xaml", "name": "XAML Extractor", "claims": [".xaml"], "publisher": { "name": "h-dv (first party)", "fingerprint": "10a480436ce..." }, "latest_version": "0.1.0", "versions": { "0.1.0": { "package_digest": "sha256:7a8b9c...", "extraction_identity": "sha256:1a2b3c...", "min_code_index_version": "0.28.0", "archive": "de.h-dv.xaml-0.1.0.cip", "signature": "de.h-dv.xaml-0.1.0.cips", "url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/de.h-dv.xaml-0.1.0.cip", "signature_url": "https://git.h-dv.de/h-dv/code-index/releases/download/v0.28.2/de.h-dv.xaml-0.1.0.cips", "capabilities": { "resolver": ["bridge_source"], "bridges": ["de.h-dv.xaml/xaml:type->csharp:class"] } } } } } } ``` --- ## 3. Forgejo CI/CD Release Pipeline Integration (`.forgejo/workflows/release.yml`) The existing `release.yml` workflow already builds all target binary archives and packages/signs in-tree plugins. The workflow will be extended in the final `release` job: 1. **Manifest Synthesis**: - Aggregate checksums from the 4 platform builds (`shipped/code-index-${TAG}-*.tar.gz.sha256`). - Read digests and identities from packed plugins (`*.cip.digest.txt`). - Assemble `distribution-manifest.json` for the specific release tag and update the cumulative `registry.json`. 2. **Cryptographic Signing of the Manifest**: - Sign `registry.json` using `CODE_INDEX_PUBLISHER_KEY` to produce `registry.json.sig` (Ed25519 detached signature). - This ensures the catalog metadata cannot be forged or tampered with in-flight. 3. **Multi-Destination Publication on Forgejo**: - Attach `registry.json` and `registry.json.sig` to the Forgejo Release assets. - Publish to Forgejo Generic Package Registry: `https://git.h-dv.de/api/packages/h-dv/generic/code-index-distribution/latest/registry.json` - Commit the updated `distribution/registry.json` back to `master` (or publish via Forgejo Pages). --- ## 4. Tooling & Client Integration ### 4.1 Enhanced `install.sh` * Reads `registry.json` directly from the release or Forgejo generic package endpoint. * Verifies `registry.json.sig` against the compiled-in first-party public key. * Validates binary archive checksums without separate HTTP requests. * Adds support for plugin bundling: ```bash sh install.sh --with-plugin de.h-dv.xaml ``` ### 4.2 CLI `code-index plugin` Enhancements * **Direct Add by ID**: `code-index plugin add de.h-dv.xaml` queries the registry catalog, automatically resolves the URL, sha256, and detached signature, and proceeds through standard verification and operator elicitation. * **Streamlined Updates**: In `.code-index.toml`: ```toml [update."de.h-dv.xaml"] source = "registry" auto_apply = true ``` `plugin update` automatically resolves the newest release from the registry, enforcing all 7 safety gates (strict semver ordering #255, subset grants, unchanged extraction identity). ### 4.3 Agent & MCP Server Integration * **Expose Registry as MCP Resource**: Expose `cosi://registry/plugins` to allow AI agents to browse available plugins. * **Intelligent Overview Warnings**: When `project_overview` detects `symbol_blind_extensions: [".xaml"]`, it checks the cached registry and includes: ```json "registry_available": [ { "package_id": "de.h-dv.xaml", "claims": [".xaml"], "latest_version": "0.1.0", "action_hint": "Call `plugin_add` with `package: \"de.h-dv.xaml\"`" } ] ``` * **One-Step MCP Install**: Allow `plugin_add` to accept `package: "<id>"`, fetching the `.cip` and `.cips` from the registry and prompting the human operator via MCP elicitation. --- ## 5. Security & Threat Model Compliance 1. **Transport Carries No Trust**: - As established in `_prdoc/guides/80-threat-model.md`, HTTP carries no trust. - Binaries are verified via published SHA-256 checksums before unpacking. - Packages are verified via detached Ed25519 signatures before ingestion. - The registry catalog itself is signed with the publisher key. 2. **First-Grant Rule Maintained**: - Even with a central registry, the first grant of a package on any machine **remains a human decision** requiring interactive confirmation. - Unattended auto-apply continues to strictly enforce capability parity ($C_{\text{new}} \subseteq C_{\text{in\_force}}$). 3. **Air-gapped & Mirror Support**: - `install.sh --base-url <url|path>` and `code-index --registry-url <url|path>` allow air-gapped environments or local disk mirrors to function identically without external network access. --- ## 6. Required Quality Gates & Tests Per IXT V2 and `code-index` standards: 1. `installer_targets.rs` extension: Grade that all published binary targets in `.forgejo/workflows/release.yml` appear in `registry.json`. 2. `registry_schema_gate.rs`: Validate `distribution/registry.v1.json` against its JSON schema. 3. `plugin_registry_e2e.rs`: End-to-end integration test over loopback mock HTTP server testing registry resolution, signature checking, tampered catalog rejection, and unattended update. 4. Mutation testing: Grade every rejection branch (tampered signature, invalid checksum, unorderable semver).
Author
Member

Reviewed against current production rather than implementing this older proposal again. The distribution feature has shipped.

Current contract: distribution/registry.v1.json and its schema describe core archives and package identities; releases publish a signed catalog binding all four archive hashes. Official packages are published independently and selected through the signed package channel. CLI add/install/update resolve package IDs, verify catalog/container signatures and digests, support pinned versions and local/URL catalog overrides, and preserve project authority. MCP exposes code-index://registry/plugins and the package-install door. The installers support archive mirrors and --with-plugin; installation alone grants nothing. Later operator-authorized automatic first-party policy supersedes the proposal's original manual-only scheduling assumption.

Published0.41.43 independently verified: all15 release assets, catalog signature, four signed-catalog archive hashes, exact installer source, all four bundles, installed four-binary hashes, and complete signed CLI/MCP/watch/restart acceptance. Receipt: /tmp/code-index-v0.41.43-published/production-verification.json. The real unchanged-core signed Svelte0.2.3→0.2.4 channel upgrade is also proven on #354. Fresh current Svelte/TypeScript registry-ID installs were repeated during #433's audit.

Current local regression checks pass:38 plugin_registry_e2e tests (including tampered/missing/untrusted/replayed catalog, digest mismatch, mirror and unattended-update refusals),8 registry_schema_gate tests,9 version-site checks and the MCP registry-resource read test. No claim is made that a new mutation campaign was run for every historical rejection branch.

Closing the delivered distribution feature, not every later architecture or language follow-up. Removing compiled-in package rows and the remaining host/package qualification coupling remain #318; bounded recovery is #410; unresolved component imports remain #433. The draft API/endpoint spellings in the original proposal should not drive another parallel distribution mechanism.

Reviewed against current production rather than implementing this older proposal again. The distribution feature has shipped. Current contract: distribution/registry.v1.json and its schema describe core archives and package identities; releases publish a signed catalog binding all four archive hashes. Official packages are published independently and selected through the signed package channel. CLI add/install/update resolve package IDs, verify catalog/container signatures and digests, support pinned versions and local/URL catalog overrides, and preserve project authority. MCP exposes code-index://registry/plugins and the package-install door. The installers support archive mirrors and --with-plugin; installation alone grants nothing. Later operator-authorized automatic first-party policy supersedes the proposal's original manual-only scheduling assumption. Published0.41.43 independently verified: all15 release assets, catalog signature, four signed-catalog archive hashes, exact installer source, all four bundles, installed four-binary hashes, and complete signed CLI/MCP/watch/restart acceptance. Receipt: /tmp/code-index-v0.41.43-published/production-verification.json. The real unchanged-core signed Svelte0.2.3→0.2.4 channel upgrade is also proven on #354. Fresh current Svelte/TypeScript registry-ID installs were repeated during #433's audit. Current local regression checks pass:38 plugin_registry_e2e tests (including tampered/missing/untrusted/replayed catalog, digest mismatch, mirror and unattended-update refusals),8 registry_schema_gate tests,9 version-site checks and the MCP registry-resource read test. No claim is made that a new mutation campaign was run for every historical rejection branch. Closing the delivered distribution feature, not every later architecture or language follow-up. Removing compiled-in package rows and the remaining host/package qualification coupling remain #318; bounded recovery is #410; unresolved component imports remain #433. The draft API/endpoint spellings in the original proposal should not drive another parallel distribution mechanism.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#269
No description provided.