I066: make the distribution catalog derive every value it states, and make the signature matter (#269) #272
No reviewers
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
h-dv/code-index!272
Loading…
Reference in a new issue
No description provided.
Delete branch "i269b-registry-truth"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What was wrong
v0.28.3 published a distribution catalog whose four platform checksums were fabricated, signed with the publisher key, and wrong. The real
linux-x86_64archive hashes0dce077f…; the catalog saidd95f0cb7…. All four disagreed with CI's own.sha256sidecars, and the publishedregistry.v1.jsonis byte-identical to the in-tree file, with a.sigbeside it.It broke nothing only because nothing consumed it yet —
install.shstill used the per-archive sidecars. The first consumer to trust the catalog would have rejected genuine artifacts.Root cause was structural, not carelessness:
registry_schema_gatepinned the catalog's version toCARGO_PKG_VERSION, so every release forced a human to edit that file and invent four hashes that cannot exist until CI has built the archives. The gate demanded the edit and could not grade it. Mutation: replacing a checksum with 64as left every gate green.The one rule
No value in
distribution/registry.v1.jsonis authored. Each is derived from the artifact it describes, by the step holding that artifact.Plugin facts are derivable in-tree (
tests/packages/*/plugin.toml+ the packed.cip); platform facts only at release time (artifacts/**/*.sha256). A document that cannot derive a fact now declares it unmeasured rather than inventing one —PlatformSetis a taggedMeasured/Unmeasuredenum, so an empty archive map can never stand in for "not measured here", and a source form carrying agenerated_attimestamp is refused outright.min_code_index_versionwas dropped rather than authored: no artifact in the tree states a minimum host version and no code read it. The package's own[abi]bracket is the real statement, andplugin installalready enforces it.The catalog now covers all three published packages, not one.
The signature now matters
verify_catalog_signaturewas#[allow(dead_code)]with zero callers, while the client fetched catalogs over HTTP and used them. It is now required on every door but the embedded one, verified againstTrustSet::load(store)rather thanFIRST_PARTY_ANCHORS— which is what makesplugin trust removereach the catalog door exactly as it reaches an installed package.Probed end to end through the shipped binary, not a harness:
.sigverified_by= the signer's fingerprintsignature_invalid.sigbeside itsignature_missing, naming the path it looked forsignature_untrustedplugin trust remove, then re-read the same catalogsignature_untrusted, trusted keys 2 → 1On the producing side,
release_registry.shbecomessynthesize \| stage \| audit. It measures the archive, not the sidecar; refuses a non-regular one through a single predicate used at all three sites; verifies its own signature through the customer's code path; and every silent skip became a loud failure — an absent publisher key used to exit 0 with no signature at all.Gates that could not fail
Two adversarial rounds, 28 confirmed defects, all fixed, each with an executed mutation. The recurring shape was gates grading text instead of behaviour:
run:and never the script it invoked. Now every step inlines the scripts it invokes, transitively and extension-agnostically, matches verbs on normalised shell (five whitespace spellings evadedcontains), counts unresolved.forgejo/scriptsmentions as a remainder that must be zero, and derives the forbidden verb set from the call graph —plugin addreachesrequire_host,cmd_check_fixturesandactivation::activate, a superset of all three hand-listed verbs.const, orstarts_witheach evaded it. It now probes the live server.distribution/schema.v1.jsonwas executed by nothing and already disagreed withvalidate(). It is executed, and an agreement table asserts the two engines differ only where registered.#[path]modules, so two installing targets contributed zero bytes while its message named a file it never reached. Replaced with a population assertion, not a bigger number.fs::write+ chmod was outside both fork-lock gates — in a file this branch had just grown by 313 lines.Client and installer
[update."<id>"]getsfrom = "registry"as its own key, mutually exclusive withsource. A sentinel insidesourcewas indistinguishable from a hostname, andregistry.example.com/pkg.cipwas swallowed by the prefix test and served the embedded catalog instead of that server, silently. The registry arm still falls through toassess_auto_apply, so #255's ordering clause refuses a stale catalog offering an older signed version.install.sh --with-plugininstalls into the machine store and grants nothing, printing theplugin addline for the human. It previously made the first capability grant on a fresh machine non-interactively, inside acurl \| sh, for whichever project the working directory happened to detect.project_overview'sregistry_availableblock is gone — it collapsed four different absences into one and read an unconsultable package store as "nothing installed". The workspace join moved tocode-index://registry/plugins, which is not token-ratcheted and can afford an honest three-state disclosure. Measured: the overview surface has 3 tokens of headroom, and an honest block there costs 7.Verification
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspace --no-fail-fastCOSI_E2E_LEG=daemon cargo test -p code-index-mcpCOSI_CORPUS_REQUIRE=1baseline.jsonunmovedprecision_gate -- --nocapturephantom_count == 0cargo docunderRUSTDOCFLAGS="-D warnings"The workspace run earned its keep: it caught a bare
fs::copyof a built executable that every-p-scoped lane was structurally blind to. A scoped green is not a workspace green.Filed rather than folded in
pubfield in a crate that isn't a dependency, and localising the receiver doesn't move it.File::create+write!+ chmod) that neither fork-lock gate can reach, with a live member inplugin-host.Not included
The version bump. Per this repo's convention that is its own commit, and the change is more than a patch — the catalog document's shape moved,
[update]gained a key,plugin installlearned the registry,plugin catalogis new, and--with-pluginchanged behaviour.🤖 Generated with Claude Code
https://claude.ai/code/session_012CsUniH3NNfPrBiHXaibvp
v0.28.3 published a distribution catalog whose four platform checksums were fabricated, signed with the publisher key, and wrong. None was the checksum of the archive it named — the real linux-x86_64 archive hashes 0dce077f…, the catalog said d95f0cb7…. Nothing broke only because no consumer read them yet. THE ONE RULE, applied everywhere: no value in distribution/registry.v1.json is authored. Each is derived from the artifact it describes, by the step holding that artifact. Plugin facts are derivable in-tree, from tests/packages/*/ plugin.toml and the packed .cip; platform facts only at release time, from artifacts/**/*.sha256. A document that cannot derive a fact now declares it unmeasured instead of inventing one. * PlatformSet is a tagged Measured/Unmeasured enum, so an empty archive map can never stand in for "not measured here". A source form carrying a generated_at timestamp is refused — v0.28.3's was the issue body's sample value, and it read as provenance. * The catalog covers all three published packages, not one; registry_schema_ gate re-derives every plugin value and refuses a drift. * release_registry.sh gains synthesize|stage|audit. It measures the ARCHIVE, not the sidecar, refuses a non-regular one through a single predicate used at all three sites, and every silent skip became a loud failure — an absent publisher key used to exit 0 with no signature at all. * min_code_index_version is DROPPED rather than authored: no artifact states a minimum host version and no code read it. The package's own [abi] bracket is the real statement and plugin install already enforces it. THE SIGNATURE NOW MATTERS. verify_catalog_signature was #[allow(dead_code)] with zero callers while the client fetched catalogs over HTTP and used them. It is required on every door but the embedded one, verified against TrustSet::load (store) rather than FIRST_PARTY_ANCHORS — so plugin trust remove reaches the catalog door exactly as it reaches an installed package. Probed end to end through the shipped binary: valid ⇒ exit 0 naming the signer; byte-flip ⇒ signature_invalid; no .sig ⇒ signature_missing; unanchored signer ⇒ signature_untrusted; trust remove then re-read ⇒ signature_untrusted, trusted keys 2 → 1. GATES THAT COULD NOT FAIL. Two adversarial rounds found 28 defects, all fixed, each with an executed mutation: * "must not execute packages" read a one-line run: and never the script it invoked. Now every step inlines the scripts it invokes, transitively and extension-agnostically, matches verbs on normalised shell, counts unresolved mentions as a remainder that must be zero, and DERIVES the forbidden set from the call graph — plugin add reaches require_host, cmd_check_fixtures and activation::activate, a superset of all three verbs that were listed by hand. * The URI-scheme gate grepped its own source; a helper fn, a const or starts_with all evaded it. It now probes the live server. * distribution/schema.v1.json was executed by nothing and already disagreed with validate(). It is executed, and an agreement table asserts the two engines differ only where registered. * A count floor passed on the wrong population: the fork-lock closure ignored #[path] modules, so two installing targets contributed zero bytes while its message named a file it never reached. Replaced with a population check. * fs::write + chmod was outside both fork-lock gates, in a file this branch had just grown by 313 lines. CLIENT AND INSTALLER. [update."<id>"] gets from = "registry" as its own key, mutually exclusive with source — a sentinel inside source was indistinguishable from a hostname, and registry.example.com/pkg.cip was swallowed by the prefix test and served the embedded catalog instead. The registry arm still falls through to assess_auto_apply, so #255's ordering clause refuses a stale catalog offering an older signed version. install.sh --with-plugin installs into the machine store and GRANTS NOTHING, printing the plugin add line for the human: the first grant on a machine stays a human decision, and it used to be made non-interactively inside a curl | sh for whichever project the cwd detected. project_overview's registry_available block is gone — it collapsed four absences into one and read an unconsultable store as "nothing installed"; the join moved to code-index://registry/plugins, which is not token-ratcheted. Gates: fmt, clippy -D warnings, cargo test --workspace, the daemon leg, the corpus ratchet with tests/corpus/baseline.json UNMOVED, and precision_gate 7/7 with phantom_count == 0. Filed rather than folded in: #270 (a receiver-binding phantom that binds a field across an undeclared crate boundary) and #271 (a third executable-writing door neither fork-lock gate can reach). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012CsUniH3NNfPrBiHXaibvpTWO LIVE FAILURES, BOTH FROM ONE BAD MEASUREMENT. I066 decision 2 chose `jq` for the catalog synthesiser and called it settled by measurement: "jq appears 20x across release.yml and .forgejo/scripts/; python3 appears 0x". That counted TEXT IN YAML, which says nothing about a container. Probed for real: $ docker run --rm ci-base:1.92.0-v2 sh -c 'command -v jq; command -v python3' jq MISSING python3 /usr/bin/python3 $ ... find / -xdev -name jq -type f ; dpkg -l | grep -w jq (nothing) no jq package The image ships no jq at all; the workflow apt-installs it where it needs it, and the ordering was fatal for us: 4177 - name: Stage distribution registry <- requires jq 4522 apt-get install -y -qq jq <- inside "Create Release" So `cargo test` was RED in CI (the test job installs no jq and our tests execute the script) and THE RELEASE WOULD HAVE DIED AT LINE 4177, 345 lines before jq exists. The script's own `command -v` refusal would have fired correctly, which is the only consolation: a loud refusal rather than an unsigned catalog. * All 14 jq invocations become `python3 -c` over the `json` module. `RETAG_PY` stays ONE definition used twice — a shell preamble concatenated ahead of whichever program needs it, so synthesize and audit cannot drift. Every refusal arm, CRLF tolerance, `refuse_irregular`, EXTRACTDIR/KEYDIR and the audit re-verification are unchanged. Verified INSIDE the image, which is the only proof that counts: 3 + 27 + 36 tests, zero failures. AND THE GATE THAT WOULD HAVE CAUGHT IT, derived from both sides rather than checking for jq: every tool a `.forgejo/scripts/*.sh` declares with `command -v` must exist in every job that invokes that script. Invocation comes from the existing `walk_script_parts`, so there is one parser. `IMAGE_BASELINES` is per-image and carries the `docker run` probes VERBATIM with tag and digest — a list nobody can re-derive is the same defect one level up — and a tag bump goes red because the row set must equal the workflows' declared images. Installs are matched POSITIONALLY, which is what makes line 4522 not count for line 4177. A job with no container gets no baseline and FAILS the join: "nothing probed this" must not read like "the tool is there". Also: `locate_package` asked `var_os(..).is_some()` for the registry variables while the loader's `resolve_source` treats empty or whitespace-only as UNSET. An exported-but-empty COSI_REGISTRY_URL= — what a CI wrapper leaves behind — made `plugin add <digest>` report the operator's argument as "a package the catalog does not list", an error about a registry nobody configured. Both doors now ask `catalog_source`. Mutations run, including two that corrected my expectations: deleting the python3 refusal is NOT a silent skip (set -eu plus `|| die` still refuses) — it costs the DIAGNOSIS, reporting a missing interpreter as "source catalog is not readable JSON". And routing the step walker through the new slicer moved `inlined` 17 -> 79 as a uniform +6, because `step_script` drops six leading spaces: a constant, not a measurement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012CsUniH3NNfPrBiHXaibvpNOT A CODE DEFECT, AND NOT THE RUNNER. `cargo fmt --all` puts EVERY target's source path on ONE rustfmt command line, and Windows caps a command line at 32767 characters. The runner said so and the message pointed nowhere near the cause: Der Dateiname oder die Erweiterung ist zu lang. (os error 206) This utility formats all bin and lib files of the current crate ... `os error 206` is ERROR_FILENAME_EXCED_RANGE. Measured from `cargo metadata --no-deps`, projected onto the runner's real workspace prefix `C:\ci-work\work\ffa0b1e7f5b61314\hostexecutor\`: targets 353 projected 32019 limit 32767 headroom 748 and rustfmt's own arguments take the rest. Master sat just under the wall; this branch's four new test targets pushed it over. That is the whole story of runs 774 and 777 failing in ~30s while master's content ON A BRANCH passed the same runner in 51 minutes. * ci-windows.yml formats PER PACKAGE, aggregating so every offender is named rather than just the first. Its anti-vacuity is a count floor AND a named required population, because a bare count floor cannot check its own population. * A LINUX-SIDE gate now measures this, so the next person to add four test files learns in two seconds on their laptop instead of in thirty on a runner whose logs are hard to reach. It prints its numbers every run, derives targets from cargo metadata, and knows the two regimes: under `--all` the bound is the whole workspace, under the per-package loop it is the largest single package (code-index-indexer, 97 targets, 8798 chars, 23969 of headroom). THE NUMBER THE GATE VOLUNTEERED IS THE ONE WORTH KEEPING: with the prefix set to `C:\` the same targets project to 16893 rather than 32115. Nearly half the budget is the RUNNER'S PATH PREFIX, not the tree — the one constant the gate cannot derive is what its verdict leans on hardest, and the file says so rather than implying precision it has not got. Measured, not reasoned: `clippy --workspace`, `build --workspace` and `test --workspace` are NOT exposed — cargo spawns one rustc per target with exactly one `.rs` argument (child command lines of 1380 and 1133 chars), so only `cargo fmt --all`'s argv grows with the target count. ci.yml keeps `--all` deliberately; Linux has no such limit. Two self-corrections recorded rather than tidied: the gate's first run flagged the REPLACEMENT step as whole-workspace, because the loop's own error message is a string containing `cargo fmt --check` (the #178 shape) — the predicate now requires command position and a third test grades both arms. And the gate's first cut stripped the workspace root with a `/`-joined prefix, which would have matched nothing on the very runner it protects, since `cargo metadata` reports `src_path` with the platform separator. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012CsUniH3NNfPrBiHXaibvp