Epic: split CI per language package — fix and release one package fast, without running every other package's pipeline #354

Closed
opened 2026-10-11 23:58:28 +02:00 by buildagent · 4 comments
Member

Goal (operator, 2026-10-11)

When a language has a problem, we must be able to fix it and release an update of that package only, quickly, without running the CI of every other package or a core release.

What already exists

  • CI-FAST (#344). A package-only commit turns off the host jobs, and package-ci runs only the legs whose inputs moved (routed by .provenance).
  • package-release.yml (#309). A pkg/<id>/v<version> tag publishes a package without a core release, then merges it into the signed pkg-channel.
  • #345. de.h-dv.ruby owns its extractor source; the builtin is a frozen snapshot with declared one-way deltas.

What the first real use showed (de.h-dv.ruby 0.10.0, 2026-10-11)

Getting one package out took five workflow runs (1089, 1090, 1094, 1102, and the core release 1088), about six hours, two operator approvals for dispatches, and a hotfix release (v0.36.1, PR #353). Zammad saw a 404 in between and needed a manual plugin add workaround.

  1. The gate demands a leg for the TAGGED commit (run 1089). package_ci_gate.sh requires the package's own package-ci leg, in scope, for the exact tagged commit.
    • A push run on a commit that does not touch tests/packages/<name> leaves that leg out, so the gate polls absent until its deadline.
    • The only ways out were a hand-dispatched full package-ci run (1090) or re-tagging.
  2. The deadline is shorter than the job it waits for. require_ci_green.sh uses DEADLINE_MINUTES=50, but the ruby leg takes 74 min and the others 63–65 min (run 1090). Even a timely dispatch cannot beat it.
  3. The legs are slow. A "fast update" cannot cost more than an hour of CI per package.
  4. Only Ruby owns its source. php, python, csharp, javascript, typescript and rust still build from shared sources (crates/guest/languages, host plugin files). A fix to one moves several packages and runs all their legs; #348 moved six packages at once.
  5. The core release pre-lists package rows. v0.36.0's catalog listed ruby 0.10.0 at its own release before that release existed. Its URL was also spelled with literal slashes, which the host answers 404.
    • Fixed in v0.36.1: encoded pkg%2F… segment, merge_relocated, and a download check.
    • Still open: the core release should not list a package version the channel has not published.
  6. Workflow and scripts come from different commits (run 1102). The dispatched workflow came from master, but the channel job ran the scripts from the tagged commit's checkout. That commit predated package_channel_check.sh --downloadable, so the last check failed with a usage error after the channel was already published. The published channel itself is correct.
  7. Never exercised end to end. package-release had only run locally (DRY_RUN and a fake API) until 2026-10-11; runs 1094 and 1102 were its first real runs.

Lanes (proposal; plan to be agreed before work starts)

  • A — Gate.
    • Grade the package leg on the commit that last changed the package's inputs, not on the tagged commit.
    • Or: package-release runs that package's leg itself, inline.
    • Wait for a RUNNING leg instead of a fixed deadline; keep a bound for absent.
    • Test both shapes, with mutations run.
  • B — Fast leg. Measure where the 63–74 min go, per package. Keep in the release path only what grades the bytes being shipped (pack, conformance, fixtures, parity on the package's own population). Move the expensive corpus and overhead legs to nightly. Target: minutes, measured.
  • C — Ownership. Extend #345 to the other six languages, one at a time: each package owns its extractor source, and the builtin becomes a frozen snapshot with declared deltas. Order by field demand.
  • D — Catalog. The package channel is the only place that lists package versions. A core release seeds the channel only with versions already published, and never with rows pointing at a package release that does not exist yet.
  • E — One version of the pipeline per run. Workflow and scripts must come from the same commit. Either run the scripts from the workflow's ref, or check out the tag's workflow. Then a mismatch fails BEFORE anything is published. Grade this in release_gate.
  • F — Drill. A real package-only fix, from commit through PR, package-ci, tag and channel, to plugin update on a project. Timed end to end, with no host jobs and no other package's legs. It is the acceptance test for this epic.

Acceptance

  • A one-line fix in one language's package is published to the channel with only that package's CI, in a bounded time, with no hand dispatch.
  • No other package's legs run, and no core release is needed.
  • plugin update <id> installs it on a real project (zammad-dev is the field reference).

Related: #309, #318, #344, #345, #353.

🤖 Generated with Claude Code

## Goal (operator, 2026-10-11) When a language has a problem, we must be able to fix it and release an update of **that package only**, quickly, without running the CI of every other package or a core release. ## What already exists - **CI-FAST (#344).** A package-only commit turns off the host jobs, and package-ci runs only the legs whose inputs moved (routed by `.provenance`). - **package-release.yml (#309).** A `pkg/<id>/v<version>` tag publishes a package without a core release, then merges it into the signed `pkg-channel`. - **#345.** de.h-dv.ruby owns its extractor source; the builtin is a frozen snapshot with declared one-way deltas. ## What the first real use showed (de.h-dv.ruby 0.10.0, 2026-10-11) Getting one package out took five workflow runs (1089, 1090, 1094, 1102, and the core release 1088), about six hours, two operator approvals for dispatches, and a hotfix release (v0.36.1, PR #353). Zammad saw a 404 in between and needed a manual `plugin add` workaround. 1. **The gate demands a leg for the TAGGED commit (run 1089).** `package_ci_gate.sh` requires the package's own package-ci leg, in scope, for the exact tagged commit. - A push run on a commit that does not touch `tests/packages/<name>` leaves that leg out, so the gate polls `absent` until its deadline. - The only ways out were a hand-dispatched full package-ci run (1090) or re-tagging. 2. **The deadline is shorter than the job it waits for.** `require_ci_green.sh` uses `DEADLINE_MINUTES=50`, but the ruby leg takes 74 min and the others 63–65 min (run 1090). Even a timely dispatch cannot beat it. 3. **The legs are slow.** A "fast update" cannot cost more than an hour of CI per package. 4. **Only Ruby owns its source.** php, python, csharp, javascript, typescript and rust still build from shared sources (`crates/guest/languages`, host plugin files). A fix to one moves several packages and runs all their legs; #348 moved six packages at once. 5. **The core release pre-lists package rows.** v0.36.0's catalog listed ruby 0.10.0 at its own release before that release existed. Its URL was also spelled with literal slashes, which the host answers 404. - Fixed in v0.36.1: encoded `pkg%2F…` segment, `merge_relocated`, and a download check. - Still open: the core release should not list a package version the channel has not published. 6. **Workflow and scripts come from different commits (run 1102).** The dispatched workflow came from master, but the channel job ran the scripts from the tagged commit's checkout. That commit predated `package_channel_check.sh --downloadable`, so the last check failed with a usage error **after** the channel was already published. The published channel itself is correct. 7. **Never exercised end to end.** package-release had only run locally (DRY_RUN and a fake API) until 2026-10-11; runs 1094 and 1102 were its first real runs. ## Lanes (proposal; plan to be agreed before work starts) - **A — Gate.** - Grade the package leg on the commit that last changed the package's inputs, not on the tagged commit. - Or: package-release runs that package's leg itself, inline. - Wait for a RUNNING leg instead of a fixed deadline; keep a bound for `absent`. - Test both shapes, with mutations run. - **B — Fast leg.** Measure where the 63–74 min go, per package. Keep in the release path only what grades the bytes being shipped (pack, conformance, fixtures, parity on the package's own population). Move the expensive corpus and overhead legs to nightly. Target: minutes, measured. - **C — Ownership.** Extend #345 to the other six languages, one at a time: each package owns its extractor source, and the builtin becomes a frozen snapshot with declared deltas. Order by field demand. - **D — Catalog.** The package channel is the only place that lists package versions. A core release seeds the channel only with versions already published, and never with rows pointing at a package release that does not exist yet. - **E — One version of the pipeline per run.** Workflow and scripts must come from the same commit. Either run the scripts from the workflow's ref, or check out the tag's workflow. Then a mismatch fails BEFORE anything is published. Grade this in `release_gate`. - **F — Drill.** A real package-only fix, from commit through PR, package-ci, tag and channel, to `plugin update` on a project. Timed end to end, with no host jobs and no other package's legs. It is the acceptance test for this epic. ## Acceptance - [ ] A one-line fix in one language's package is published to the channel with only that package's CI, in a bounded time, with no hand dispatch. - [ ] No other package's legs run, and no core release is needed. - [ ] `plugin update <id>` installs it on a real project (zammad-dev is the field reference). Related: #309, #318, #344, #345, #353. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Author
Member

Plan for the v0.37 train (operator decision, 2026-10-12)

Scope: all of this epic, including lane C for all six languages, plus #355, #356 and the release-note fix (languages owner ruby package, not enable). Lane A uses the inline leg. One full run, one PR, one CI round, one tag set.

Measured before planning (run 1090, ruby leg, 73.9 min)

  • Only about 15 minutes of the leg grades the bytes being shipped: the reproducible guest rebuild, the package's own suites, the release scripts and the release gates. About 52 minutes is compile:
    • test-profile compile, 15 min;
    • a nested dev build inside ruby_guest_rebuild, 12 min;
    • the --release compile for corpus parity and cost, 30 min.
  • Every builtin walker (crates/plugins/src/*.rs) is compiled into every package lane. That is why one walker edit moves all packages.
  • The core release lists package rows by URL shape only. It never checks that the asset exists, and its channel seed does not use --downloadable.
  • Nothing grades that a workflow and the scripts it runs come from one commit (run 1102).

Wave 1 (running)

Lane Content
pipeline (A+B+E) One package_leg.sh <name> <tier> used by package-ci and package-release. Tier release grades only the shipped bytes. Corpus and overhead legs run as a parallel job, plus nightly. package-release runs the leg inline with no deadline wait. Scripts come from the workflow's own commit, or the dispatch is refused, graded in release_gate. Compile cost is cut and measured.
catalog (D) Rows on pkg%2F… belong to package-release. A core release takes them from the published channel, never from the in-tree bump. Its channel seed checks that each URL downloads. Release-note text fix.
own-c0 (C foundation) The Ruby-only ownership, frozen-builtin and walker-parity tests become one list-driven set. PHP is the pilot (row 2). Measure whether moving a walker or its gating changes any package's bytes. Write a recipe for the remaining languages, including JS and TS each owning a copy of typescript.rs.
vue-label #355
changed-paging #356

Wave 2 (after own-c0)

python, csharp, rust, javascript and typescript, one lane each, following own-c0's recipe.

Then

  • Integrate on train-037, one full gate run, version bump, PR and CI.
  • Merge, tag and release, with operator approval as usual.
  • F drill: a real package-only fix, from commit through the inline package-release to the channel and plugin update, timed. Any package whose bytes the ownership move changes gets its own package release through the new pipeline, which is the drill run for real.

🤖 Generated with Claude Code

## Plan for the v0.37 train (operator decision, 2026-10-12) Scope: **all of this epic, including lane C for all six languages**, plus #355, #356 and the release-note fix (`languages owner ruby package`, not `enable`). Lane A uses the **inline leg**. One full run, one PR, one CI round, one tag set. ### Measured before planning (run 1090, ruby leg, 73.9 min) - Only about 15 minutes of the leg grades the bytes being shipped: the reproducible guest rebuild, the package's own suites, the release scripts and the release gates. About 52 minutes is compile: - test-profile compile, 15 min; - a nested dev build inside `ruby_guest_rebuild`, 12 min; - the `--release` compile for corpus parity and cost, 30 min. - Every builtin walker (`crates/plugins/src/*.rs`) is compiled into every package lane. That is why one walker edit moves all packages. - The core release lists package rows by URL *shape* only. It never checks that the asset exists, and its channel seed does not use `--downloadable`. - Nothing grades that a workflow and the scripts it runs come from one commit (run 1102). ### Wave 1 (running) | Lane | Content | |---|---| | pipeline (A+B+E) | One `package_leg.sh <name> <tier>` used by package-ci and package-release. Tier `release` grades only the shipped bytes. Corpus and overhead legs run as a parallel job, plus nightly. package-release runs the leg inline with no deadline wait. Scripts come from the workflow's own commit, or the dispatch is refused, graded in `release_gate`. Compile cost is cut and measured. | | catalog (D) | Rows on `pkg%2F…` belong to package-release. A core release takes them from the published channel, never from the in-tree bump. Its channel seed checks that each URL downloads. Release-note text fix. | | own-c0 (C foundation) | The Ruby-only ownership, frozen-builtin and walker-parity tests become one list-driven set. PHP is the pilot (row 2). Measure whether moving a walker or its gating changes any package's bytes. Write a recipe for the remaining languages, including JS and TS each owning a copy of `typescript.rs`. | | vue-label | #355 | | changed-paging | #356 | ### Wave 2 (after own-c0) python, csharp, rust, javascript and typescript, one lane each, following own-c0's recipe. ### Then - Integrate on `train-037`, one full gate run, version bump, PR and CI. - Merge, tag and release, with operator approval as usual. - **F drill:** a real package-only fix, from commit through the inline package-release to the channel and `plugin update`, timed. Any package whose bytes the ownership move changes gets its own package release through the new pipeline, which is the drill run for real. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Author
Member

v0.37.0 released (release run 1128 green). It ships the per-package extractors for php, python, csharp, rust, javascript and typescript as 0.1.3, plus the catalog rule. de.h-dv.ruby 0.10.1 is withheld from the catalog until its own package release, and the release says so.

Dogfood on the release binary, this repo, 1355 files:

  • Builtin vs 0.36.1: all 409,777 binds are identical, including targets and resolved_by.
  • rust 0.1.3 package vs builtin: every symbol and bind is identical across all 879 .rs files.
  • ruby 0.10.1 package vs builtin: 2 binds changed resolving rule, with the same targets. The package also adds 308 refs in the .rbx fixtures.

Still open in this epic:

  • F drill: the first package-only release (pkg/de.h-dv.ruby/v0.10.1). It is not done yet and needs operator approval.

Found while dogfooding, being fixed for the next patch release:

  • An owner flip (languages owner <lang> package) re-extracted the whole tree. It also left the rollback generation holding the wrong files: the carried ones instead of the switched ones.
  • A rollback after an edit left the package's rows live.
  • lang=vue and lang=svelte matched nothing.
**v0.37.0 released** (release run 1128 green). It ships the per-package extractors for php, python, csharp, rust, javascript and typescript as 0.1.3, plus the catalog rule. de.h-dv.ruby 0.10.1 is withheld from the catalog until its own package release, and the release says so. **Dogfood on the release binary, this repo, 1355 files:** - **Builtin vs 0.36.1:** all 409,777 binds are identical, including targets and `resolved_by`. - **rust 0.1.3 package vs builtin:** every symbol and bind is identical across all 879 .rs files. - **ruby 0.10.1 package vs builtin:** 2 binds changed resolving rule, with the same targets. The package also adds 308 refs in the `.rbx` fixtures. **Still open in this epic:** - **F drill:** the first package-only release (`pkg/de.h-dv.ruby/v0.10.1`). It is not done yet and needs operator approval. **Found while dogfooding, being fixed for the next patch release:** - An owner flip (`languages owner <lang> package`) re-extracted the whole tree. It also left the rollback generation holding the wrong files: the carried ones instead of the switched ones. - A rollback after an edit left the package's rows live. - `lang=vue` and `lang=svelte` matched nothing.
Author
Member

Independent package-release evidence from Python 0.1.4 (2026-10-22): run 1476 completed its four normal stages in 15m20s: own-package qualification, pack/sign/operator install and negative controls, immutable publication, signed-channel merge and published read-back. No host CI job or other language package leg ran in that release workflow, and no core release was needed.

Independent operator proof used the published core 0.41.8 (4734318a) with its original signed Python 0.1.3 bundle in an isolated project. The reported Hatch wheel layout initially returned abi.frame_truncated. plugin update de.h-dv.python --grant requested --yes, using the default signed package channel and no custom trust, selected official Python 0.1.4. Re-indexing produced an active-generation layout of ready, withheld=0, no reason. The separately downloaded official artifact also mapped the expected modules and excluded files outside the wheel package.

This does not establish the whole package-only PR acceptance: PR #407 combines host diagnostics with a Python layout helper change under guest-kit, so its PR qualification correctly selected all packages. That shared-input fan-out remains a distinct optimization/ownership question. This evidence establishes the independent release and operator-update path, not completion of every acceptance item.

Independent package-release evidence from Python 0.1.4 (2026-10-22): [run 1476](https://git.h-dv.de/h-dv/code-index/actions/runs/1476) completed its four normal stages in 15m20s: own-package qualification, pack/sign/operator install and negative controls, immutable publication, signed-channel merge and published read-back. No host CI job or other language package leg ran in that release workflow, and no core release was needed. Independent operator proof used the published core `0.41.8 (4734318a)` with its original signed Python 0.1.3 bundle in an isolated project. The reported Hatch wheel layout initially returned `abi.frame_truncated`. `plugin update de.h-dv.python --grant requested --yes`, using the default signed package channel and no custom trust, selected official Python 0.1.4. Re-indexing produced an active-generation layout of `ready`, `withheld=0`, no reason. The separately downloaded official artifact also mapped the expected modules and excluded files outside the wheel package. This does **not** establish the whole package-only PR acceptance: PR #407 combines host diagnostics with a Python layout helper change under `guest-kit`, so its PR qualification correctly selected all packages. That shared-input fan-out remains a distinct optimization/ownership question. This evidence establishes the independent release and operator-update path, not completion of every acceptance item.
Author
Member

Closing after auditing the previously missing F drill against work already shipped, rather than reimplementing the old plan.

Real package-only fix: PR #489 (Svelte0.2.4), head fd19c6e500, merged as f349d01bdf. Its changed files are the Svelte guest, package bytes/fixtures/catalog row and package-owned tests; no runtime or core-version edit. The classifier explicitly reported mode=packages, packages=svelte and every host group false. Linux run2182 skipped host qualification jobs; Windows run2181 skipped its native build/test job; pilot run2184 skipped both unrelated pilots. Other package matrix entries were scope no-ops. The Svelte PR qualification run2183 completed in4m55s.

The independent package-tag release run2189 completed all four stages in4m18s: own-byte qualification, pack/sign/operator checks, immutable publication and signed-channel update. No hand dispatch or core release was needed. Published assets: https://git.h-dv.de/h-dv/code-index/releases/tag/pkg%2Fde.h-dv.svelte%2Fv0.2.4 . Earlier comments already reconciled lanesA–E and left only this end-to-end evidence outstanding.

Fresh operator verification today used unchanged published core0.41.43 in a new isolated real project/store. Installed official signed Svelte0.2.3, indexed a keyed each block, then ran plugin update de.h-dv.svelte --grant requested --yes against the default signed channel. Before: wrapper1, missing body read0, parse errors0. After official0.2.4: wrapper1, body read1, parse errors0. Core binary/version unchanged throughout. The store contained only the old package before update. Complete fresh-store drill1.206s; named update0.587s. Raw command outputs and receipt: /tmp/cosi-044-svelte-drill-timed/commands.json and verification.json.

This establishes the bounded package-only PR→release/channel→operator-update path. General host/package isolation refinements remain independently tracked in #318; broader startup and resolver gaps remain separate.

Closing after auditing the previously missing F drill against work already shipped, rather than reimplementing the old plan. Real package-only fix: PR #489 (Svelte0.2.4), head fd19c6e500be09b075eeaf5cfd1c9560541cdf1c, merged as f349d01bdf96ffcc29ed4bfff03e42f1f01ae5c4. Its changed files are the Svelte guest, package bytes/fixtures/catalog row and package-owned tests; no runtime or core-version edit. The classifier explicitly reported mode=packages, packages=svelte and every host group false. Linux run2182 skipped host qualification jobs; Windows run2181 skipped its native build/test job; pilot run2184 skipped both unrelated pilots. Other package matrix entries were scope no-ops. The Svelte PR qualification run2183 completed in4m55s. The independent package-tag release run2189 completed all four stages in4m18s: own-byte qualification, pack/sign/operator checks, immutable publication and signed-channel update. No hand dispatch or core release was needed. Published assets: https://git.h-dv.de/h-dv/code-index/releases/tag/pkg%2Fde.h-dv.svelte%2Fv0.2.4 . Earlier comments already reconciled lanesA–E and left only this end-to-end evidence outstanding. Fresh operator verification today used unchanged published core0.41.43 in a new isolated real project/store. Installed official signed Svelte0.2.3, indexed a keyed each block, then ran `plugin update de.h-dv.svelte --grant requested --yes` against the default signed channel. Before: wrapper1, missing body read0, parse errors0. After official0.2.4: wrapper1, body read1, parse errors0. Core binary/version unchanged throughout. The store contained only the old package before update. Complete fresh-store drill1.206s; named update0.587s. Raw command outputs and receipt: /tmp/cosi-044-svelte-drill-timed/commands.json and verification.json. This establishes the bounded package-only PR→release/channel→operator-update path. General host/package isolation refinements remain independently tracked in #318; broader startup and resolver gaps remain separate.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#354
No description provided.