h-dv/code-index

Watch

Fork 0

feat: `plugin add <url>` — fetch a package over HTTP, with the digest pin mandatory #236

New issue

Closed

opened 2026-09-09 10:35:29 +02:00 by buildagent · 1 comment

buildagent commented

2026-09-09 10:35:29 +02:00

Member

Today plugin add [SOURCE] takes a .cip path, an sha256: identity already in the store, or the single .cip under <root>/.code-index/plugins/. Getting a package from a release means curl twice — the .cip and its .cips — and knowing that the second is not optional. That is the first thing an operator does and the only step with no support.

What to build

code-index plugin add https://git.h-dv.de/h-dv/code-index/releases/download/v0.27.1/de.h-dv.timeline-0.1.0.cip \
  --sha256 sha256:150ceb22…

SOURCE accepts an http(s):// URL alongside the existing forms.
The .cips is fetched too, from <url>.cips by default, overridable with --signature-url. A missing signature is a REFUSAL naming the URL it tried — a .cip without its .cips is not undocumented, it is uninstallable, which is what v0.24.0 shipped.
--sha256 is REQUIRED when SOURCE is a URL. See the threat note below; this is the clause that matters.
Everything after the bytes land is unchanged: parse, compute the install identity, verify the detached signature against this operator's trust store before a single value is rendered, resolve the grant, measure the reindex domain, render, one affirmative act, then install/check/enable in that order.

Why the digest pin is mandatory for a URL and not for a file

The signature already gives authenticity: a hostile URL or a MITM cannot forge a package without the publisher key, and verification happens before anything is rendered. So HTTP does not weaken that.

What a URL adds is substitution. The bytes at a URL are chosen by whoever controls the URL and the network, not by the operator — so a validly signed but different package can be served: an older version with a known defect, or a different package by the same publisher. With a local file the operator at least chose the bytes off disk.

--sha256 closes exactly that gap and nothing else. It should be stated in those terms in the help, not as generic caution.

What a fix must prove

A URL serving the expected bytes installs, and the resulting store entry is byte-identical to installing the same file locally.
A URL whose bytes do not match --sha256 REFUSES, naming both digests. Must go RED against a build that skips the check.
A URL with no .cips beside it REFUSES naming the signature URL it tried.
A URL serving a package signed by a key this operator does not anchor REFUSES with signature_untrusted — i.e. the existing trust check is reached, not bypassed by the fetch path.
--sha256 omitted with a URL SOURCE is a refusal, not a warning. Mutation: make it a warning → that test goes RED.
Anti-vacuity: a LOCAL file without --sha256 still installs. Otherwise "require the pin everywhere" passes and the distinction this issue rests on is untested.
Redirects, a non-200, a truncated body and a body larger than any plausible package each fail with a message naming which one happened. A partial download that happens to hash correctly is impossible; one that does not must not be left in the store.

Not in scope

Registry protocols, search, version resolution. This is one URL to one file. Version resolution is where auto-update lives — filed separately.

Today `plugin add [SOURCE]` takes a `.cip` path, an `sha256:` identity already in the store, or the single `.cip` under `<root>/.code-index/plugins/`. Getting a package from a release means `curl` twice — the `.cip` and its `.cips` — and knowing that the second is not optional. That is the first thing an operator does and the only step with no support. ## What to build ``` code-index plugin add https://git.h-dv.de/h-dv/code-index/releases/download/v0.27.1/de.h-dv.timeline-0.1.0.cip \ --sha256 sha256:150ceb22… ``` * **`SOURCE` accepts an `http(s)://` URL** alongside the existing forms. * **The `.cips` is fetched too**, from `<url>.cips` by default, overridable with `--signature-url`. A missing signature is a REFUSAL naming the URL it tried — a `.cip` without its `.cips` is not undocumented, it is uninstallable, which is what v0.24.0 shipped. * **`--sha256` is REQUIRED when SOURCE is a URL.** See the threat note below; this is the clause that matters. * Everything after the bytes land is unchanged: parse, compute the install identity, **verify the detached signature against this operator's trust store before a single value is rendered**, resolve the grant, measure the reindex domain, render, one affirmative act, then install/check/enable in that order. ## Why the digest pin is mandatory for a URL and not for a file The signature already gives **authenticity**: a hostile URL or a MITM cannot forge a package without the publisher key, and verification happens before anything is rendered. So HTTP does not weaken that. What a URL adds is **substitution**. The bytes at a URL are chosen by whoever controls the URL and the network, not by the operator — so a *validly signed* but *different* package can be served: an older version with a known defect, or a different package by the same publisher. With a local file the operator at least chose the bytes off disk. `--sha256` closes exactly that gap and nothing else. It should be stated in those terms in the help, not as generic caution. ## What a fix must prove * A URL serving the expected bytes installs, and the resulting store entry is byte-identical to installing the same file locally. * **A URL whose bytes do not match `--sha256` REFUSES**, naming both digests. Must go RED against a build that skips the check. * A URL with no `.cips` beside it REFUSES naming the signature URL it tried. * A URL serving a package signed by a key this operator does not anchor REFUSES with `signature_untrusted` — i.e. the existing trust check is reached, not bypassed by the fetch path. * **`--sha256` omitted with a URL SOURCE is a refusal**, not a warning. Mutation: make it a warning → that test goes RED. * **Anti-vacuity**: a LOCAL file without `--sha256` still installs. Otherwise "require the pin everywhere" passes and the distinction this issue rests on is untested. * Redirects, a non-200, a truncated body and a body larger than any plausible package each fail with a message naming which one happened. A partial download that happens to hash correctly is impossible; one that does not must not be left in the store. ## Not in scope Registry protocols, search, version resolution. This is one URL to one file. Version resolution is where auto-update lives — filed separately.

buildagent referenced this issue

2026-09-09 10:35:58 +02:00

feat: automatic package update — apply silently ONLY when nothing about the authority changed, refuse otherwise #237

dhoyer referenced this issue from a commit

2026-09-09 12:12:25 +02:00

feat: `plugin add <url>`, with the digest pin mandatory (#236)

buildagent commented

2026-09-09 12:13:08 +02:00

Author

Member

Implemented in 4b02fb5, on master.

code-index plugin add https://…/de.h-dv.timeline-0.1.0.cip --sha256 sha256:150ceb22…

18 integration arms, all green. Every item in "what a fix must prove" is implemented and tested, including the anti-vacuity one — a_local_file_without_the_pin_still_installs, without which "require the pin everywhere" would pass and the distinction this issue rests on would go untested.

The change is step 1 only, with one deliberate exception

locate_package grew a URL arm that ends with a .cip and its .cips on local disk in a staging directory. Everything from parse onward is reached by identical code with identical arguments.

The exception is the pin comparison, and it cannot live in step 1: the install identity is package_digest of the parsed container — domain-separated and structured — not a hash of the file. So "are these the bytes you named" is a question only a parsed package can answer. It sits where cmd_install already puts it, before the anchor, before the grant, before anything is rendered or written.

Verified independently before merge, not taken from the report: inverting that comparison to if &entry.digest == want && false makes both pin arms fail —

test a_local_file_with_a_wrong_pin_is_refused ... FAILED
test served_bytes_that_are_not_the_pinned_ones_refuse_naming_both ... FAILED

i.e. the substituted package installs. Restored by cp, md5 identical.

Locating the signature

<url> plus one appended byte, derived from SIGNATURE_EXT.strip_prefix(PACKAGE_EXT) — the URL spelling of packages::signature_path's append, never replace rule, held against the file rule by a unit test over four names. --signature-url overrides and is refused for a non-URL source.

Note …/download/latest → …/download/latests, which is correct and surprising, so it is covered. That case is also why one of the survivors below mattered.

Both size ceilings are derived, and enforced twice

Declared Content-Length and bytes that actually arrived — the argument read_bounded already makes about lying metadata, applied to a server:

.cip — MAX_FETCH_BYTES is MAX_PACKAGE_BYTES: a byte past what read_package accepts off disk could never be installed anyway.
.cips — SIG_HEADER_LEN + SIG_RECORD_LEN * MAX_SIGNATURES = 16 + 8×96 = 784 bytes exactly, the largest file SigFile::parse can accept.
MAX_REDIRECTS = 5 is not derivable and its doc says so. What makes it reviewable instead is that the hops used and the final URL are disclosed on every fetch line.

Redirects are followed, and the issue was wrong to imply otherwise

The issue listed redirects among the things that must fail. They are followed, up to the bound, and disclosed — because refusing the first hop would refuse the release-download case this issue exists for. --proto/--proto-redir are pinned to http,https.

The transport is the operator's `curl`, and that follows from this issue's own threat model

The transport is trusted for nothing: the signature gives authenticity, the pin gives exactness. Adding TLS to get confidentiality of which package is fetched would mean a crypto crate, a cert path, ~30 crates in a binary that executes untrusted wasm, a cargo deny licence question (webpki-roots is MPL-2.0), and C builds on four release targets.

Verified rather than asserted: zero TLS crates in Cargo.lock, and shelling out is already this tree's pattern — git for diffs, ps/kill for process facts. A missing curl refuses by name and points at the pre-existing workflow, which performs the same two checks over the same bytes.

Three mutations SURVIVED and are recorded in the tests rather than hidden

Faking the anchor in cmd_add survived — the trust check has two doors; Store::install re-verifies. The mutation that does grade it (disabling verify_detached inside Store::verifying_anchor) went RED: the unanchored package installed over HTTP.
Deleting the post-transfer size check survived on curl 8.5.0, which aborts mid-transfer at --max-filesize (exit 63). Kept anyway, because that is a property of the fetcher's version, not of the contract — and removing --max-filesize reddens the declared-size arm while leaving this one green, which is how the two were told apart.
default_signature_url replacing the extension survived its first test, because for a name ending in .cip append and replace agree. Strengthened with non-.cip names; then RED.

Also worth recording: making fetch_package return anyhow::Result survived until the result was bound as let e: FetchError — ? still converts, so the type was lost with the compiler silent. It is now a compile error.

Reuse for #237, and a door deliberately left shut

fetch_package returns a typed FetchError { what, url, failure }, so auto-update can report unreachable rather than up to date without matching on prose. The fetch report goes to stderr, because plugin add's stdout must still begin with the confirmation (nothing_is_written_before_the_answer asserts it).

No URL door over MCP. plugin_add still takes digest/file only — an agent must not choose which bytes the machine fetches. no_url_door_exists_over_mcp now grades that, because three NotAgentVisible registry rows rest on it.

Not established

curl on Windows. It ships in Windows 10 1803+/Server 2019+ and Command::new("curl") gets the binary rather than a shell alias, but the native CI job is the only place this gets proven. If it is absent there the new suite goes RED rather than skipping — deliberate.
Real HTTPS. Every test is loopback HTTP; TLS is curl's and untested here by construction.

One more: this repo's own prose_spacing_gate caught a real defect in the work — a lost \ continuation was shipping 18 spaces of indent inside an operator-facing refusal.

_prdoc/guides/80-operator-recovery.md claimed "plugin add has no --sha256". Now false; corrected.

Gates, isolated: fmt, clippy (host and windows-gnu), cargo test --workspace --no-fail-fast (327 suites, 0 FAILED, counted over the whole log), rustdoc -D warnings — all exit 0.

Implemented in `4b02fb5`, on `master`. ``` code-index plugin add https://…/de.h-dv.timeline-0.1.0.cip --sha256 sha256:150ceb22… ``` 18 integration arms, all green. Every item in "what a fix must prove" is implemented and tested, including the anti-vacuity one — `a_local_file_without_the_pin_still_installs`, without which "require the pin everywhere" would pass and the distinction this issue rests on would go untested. ## The change is step 1 only, with one deliberate exception `locate_package` grew a URL arm that ends with a `.cip` and its `.cips` on local disk in a staging directory. Everything from parse onward is reached by identical code with identical arguments. The exception is the pin comparison, and it **cannot** live in step 1: the install identity is `package_digest` of the **parsed container** — domain-separated and structured — not a hash of the file. So *"are these the bytes you named"* is a question only a parsed package can answer. It sits where `cmd_install` already puts it, before the anchor, before the grant, before anything is rendered or written. **Verified independently before merge**, not taken from the report: inverting that comparison to `if &entry.digest == want && false` makes **both** pin arms fail — ``` test a_local_file_with_a_wrong_pin_is_refused ... FAILED test served_bytes_that_are_not_the_pinned_ones_refuse_naming_both ... FAILED ``` i.e. the substituted package installs. Restored by `cp`, md5 identical. ## Locating the signature `<url>` plus one appended byte, derived from `SIGNATURE_EXT.strip_prefix(PACKAGE_EXT)` — the URL spelling of `packages::signature_path`'s **append, never replace** rule, held against the file rule by a unit test over four names. `--signature-url` overrides and is refused for a non-URL source. Note `…/download/latest` → `…/download/latests`, which is correct and surprising, so it is covered. That case is also why one of the survivors below mattered. ## Both size ceilings are derived, and enforced twice Declared `Content-Length` **and** bytes that actually arrived — the argument `read_bounded` already makes about lying metadata, applied to a server: * `.cip` — `MAX_FETCH_BYTES` **is** `MAX_PACKAGE_BYTES`: a byte past what `read_package` accepts off disk could never be installed anyway. * `.cips` — `SIG_HEADER_LEN + SIG_RECORD_LEN * MAX_SIGNATURES` = 16 + 8×96 = **784 bytes exactly**, the largest file `SigFile::parse` can accept. * `MAX_REDIRECTS = 5` is **not** derivable and its doc says so. What makes it reviewable instead is that the hops used and the final URL are disclosed on every fetch line. ## Redirects are followed, and the issue was wrong to imply otherwise The issue listed redirects among the things that must fail. They are followed, up to the bound, and disclosed — because refusing the first hop would refuse the release-download case this issue exists for. `--proto`/`--proto-redir` are pinned to `http,https`. ## The transport is the operator's `curl`, and that follows from this issue's own threat model The transport is trusted for **nothing**: the signature gives authenticity, the pin gives exactness. Adding TLS to get confidentiality of *which* package is fetched would mean a crypto crate, a cert path, ~30 crates in a binary that executes untrusted wasm, a `cargo deny` licence question (`webpki-roots` is MPL-2.0), and C builds on four release targets. Verified rather than asserted: **zero TLS crates in `Cargo.lock`**, and shelling out is already this tree's pattern — `git` for diffs, `ps`/`kill` for process facts. A missing `curl` refuses by name and points at the pre-existing workflow, which performs the same two checks over the same bytes. ## Three mutations SURVIVED and are recorded in the tests rather than hidden 1. **Faking the anchor in `cmd_add` survived** — the trust check has two doors; `Store::install` re-verifies. The mutation that *does* grade it (disabling `verify_detached` inside `Store::verifying_anchor`) went RED: the unanchored package installed over HTTP. 2. **Deleting the post-transfer size check survived** on curl 8.5.0, which aborts mid-transfer at `--max-filesize` (exit 63). Kept anyway, because that is a property of the fetcher's *version*, not of the contract — and removing `--max-filesize` reddens the declared-size arm while leaving this one green, which is how the two were told apart. 3. **`default_signature_url` replacing the extension survived** its first test, because for a name ending in `.cip` append and replace agree. Strengthened with non-`.cip` names; then RED. Also worth recording: making `fetch_package` return `anyhow::Result` **survived** until the result was bound as `let e: FetchError` — `?` still converts, so the type was lost with the compiler silent. It is now a compile error. ## Reuse for #237, and a door deliberately left shut `fetch_package` returns a typed `FetchError { what, url, failure }`, so auto-update can report **unreachable** rather than **up to date** without matching on prose. The fetch report goes to stderr, because `plugin add`'s stdout must still begin with the confirmation (`nothing_is_written_before_the_answer` asserts it). **No URL door over MCP.** `plugin_add` still takes `digest`/`file` only — an agent must not choose which bytes the machine fetches. `no_url_door_exists_over_mcp` now grades that, because three `NotAgentVisible` registry rows rest on it. ## Not established * **`curl` on Windows.** It ships in Windows 10 1803+/Server 2019+ and `Command::new("curl")` gets the binary rather than a shell alias, but the native CI job is the only place this gets proven. If it is absent there the new suite goes RED rather than skipping — deliberate. * **Real HTTPS.** Every test is loopback HTTP; TLS is curl's and untested here by construction. One more: this repo's own `prose_spacing_gate` caught a real defect in the work — a lost `\` continuation was shipping 18 spaces of indent inside an operator-facing refusal. `_prdoc/guides/80-operator-recovery.md` claimed *"`plugin add` has no `--sha256`"*. Now false; corrected. Gates, isolated: fmt, clippy (host **and** windows-gnu), `cargo test --workspace --no-fail-fast` (327 suites, 0 FAILED, counted over the whole log), rustdoc `-D warnings` — all exit 0.

buildagent closed this issue

2026-09-09 12:13:13 +02:00

buildagent referenced this issue

2026-09-09 14:33:53 +02:00

feat: automatic package update — apply silently ONLY when nothing about the authority changed, refuse otherwise #237

buildagent referenced this issue

2026-09-10 16:13:22 +02:00

plugin add <url> holds the id AND the source URL but offers no [update] stanza — the one moment "easy update" is free, and it is dropped #257

dhoyer referenced this issue from a commit

2026-09-10 18:36:36 +02:00

docs+test: the update path is documented where an operator looks, and its capability clause is graded (#255, #256, #257)