Per-package CI: each package tested in its own leg, never in host CI (#318 phase 2, #314) #325

Merged
buildagent merged 3 commits from p2-per-package-ci into master 2026-10-07 11:30:44 +02:00
Member

Phase 2 of #318, plus the #314 fix behind nightly run 5754's failure.

Phase 2

  • New crate code-index-first-party-tests. It holds every package-self suite: provenance, digest equals pack, grammar copy, name section, builder paths, rebuild with byte compare, declared profile. It also holds one <lang>_language_overhead per language. Host CI excludes the crate, on both Linux and Windows.
  • New .forgejo/workflows/package-ci.yml.
    • A static 10-leg matrix, package (<name>). Each leg runs only when the changes job puts it in scope; an empty scope fails open and runs every leg.
    • Each leg runs its package's suites, guest gates, release scripts (COSI_PACKAGE), corpus suites and builtin parity. The parity step checks semantics only on each push and enforces both budgets nightly.
    • The nightly cron reports and never blocks.
  • ci_changes.sh routes package paths to package legs:
    • a change under tests/packages/<name>/**, tables/<name>.rs or crates/guest/<name> runs only that package's leg, and no host jobs;
    • a shared guest change reaches the 7 or 9 legs that consume it;
    • a host change runs no package leg.
    • ci_path_gating checks each of these cases on a temporary repo.
  • host_reads_no_live_package: the allowlist shrinks to 1 Phase 3 row, 3 Phase 4 rows and 3 data rows.

#314

  • tests/phase2_acceptance.py builds every child process environment through one helper, child_env, which removes COSI_CORPUS_DIR. The ci.yml step also unsets it.
  • code_index_test_support::product::command is now the only way a test spawns a binary that can reach the MCP server. 52 call sites in 14 files use it. A registry test, mcp_spawn_registry, enforces this.

Verified

  • fmt and clippy are clean.
  • Targeted guards, re-run independently: ci_path_gating 33/33, host_reads_no_live_package 5/5, version_site_registry 8/8, grammar_provenance 7/7, mcp_spawn_registry 3/3, frozen_pins 3/3, release_notes_quoting 2/2.
  • Mutations were run for every guard. I re-ran one myself: removing the scrub from the spawn helper turns the_door_removes_the_variable_the_server_reads red.
  • No .code-index directory is left in the corpus.

Expected red

package (python) fails its builtin-parity step on python-flask with the #315 delta. That gap was never measured before, because nightly CI died earlier at the #314 contamination guard.

#315 is not fixable package-only:

  • Extraction is identical between the builtin and the package.
  • The difference comes from host code, python_imports.rs (the denied-refs list and filter_layout_targets).
  • Most of the 91 "losses" are wrong binds made by the builtin.

Known gap (Phase 4)

package-release.yml's REQUIRED_JOBS still lists host jobs. So a package-only commit, whose host jobs are all skipped, cannot be released without a full dispatch.

🤖 Generated with Claude Code

https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

Phase 2 of #318, plus the #314 fix behind nightly run 5754's failure. ## Phase 2 - **New crate `code-index-first-party-tests`.** It holds every package-self suite: provenance, digest equals pack, grammar copy, name section, builder paths, rebuild with byte compare, declared profile. It also holds one `<lang>_language_overhead` per language. Host CI excludes the crate, on both Linux and Windows. - **New `.forgejo/workflows/package-ci.yml`.** - A static 10-leg matrix, `package (<name>)`. Each leg runs only when the `changes` job puts it in scope; an empty scope fails open and runs every leg. - Each leg runs its package's suites, guest gates, release scripts (`COSI_PACKAGE`), corpus suites and builtin parity. The parity step checks semantics only on each push and enforces both budgets nightly. - The nightly cron reports and never blocks. - **`ci_changes.sh` routes package paths to package legs:** - a change under `tests/packages/<name>/**`, `tables/<name>.rs` or `crates/guest/<name>` runs only that package's leg, and no host jobs; - a shared guest change reaches the 7 or 9 legs that consume it; - a host change runs no package leg. - `ci_path_gating` checks each of these cases on a temporary repo. - **`host_reads_no_live_package`:** the allowlist shrinks to 1 Phase 3 row, 3 Phase 4 rows and 3 data rows. ## #314 - `tests/phase2_acceptance.py` builds every child process environment through one helper, `child_env`, which removes `COSI_CORPUS_DIR`. The ci.yml step also unsets it. - `code_index_test_support::product::command` is now the only way a test spawns a binary that can reach the MCP server. 52 call sites in 14 files use it. A registry test, `mcp_spawn_registry`, enforces this. ## Verified - fmt and clippy are clean. - Targeted guards, re-run independently: `ci_path_gating` 33/33, `host_reads_no_live_package` 5/5, `version_site_registry` 8/8, `grammar_provenance` 7/7, `mcp_spawn_registry` 3/3, `frozen_pins` 3/3, `release_notes_quoting` 2/2. - Mutations were run for every guard. I re-ran one myself: removing the scrub from the spawn helper turns `the_door_removes_the_variable_the_server_reads` red. - No `.code-index` directory is left in the corpus. ## Expected red `package (python)` fails its builtin-parity step on python-flask with the #315 delta. That gap was never measured before, because nightly CI died earlier at the #314 contamination guard. #315 is **not** fixable package-only: - Extraction is identical between the builtin and the package. - The difference comes from host code, `python_imports.rs` (the denied-refs list and `filter_layout_targets`). - Most of the 91 "losses" are wrong binds made by the builtin. ## Known gap (Phase 4) `package-release.yml`'s `REQUIRED_JOBS` still lists host jobs. So a package-only commit, whose host jobs are all skipped, cannot be released without a full dispatch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu
`crates/first-party-tests` (a workspace member) holds every suite that
grades a first-party package's OWN bytes, named `<pkg>_*.rs`:

* moved whole: ruby_builtin_expectations, ruby_claim_parity,
  ruby_native_differential, ruby_package_cost/overhead/parity,
  timeline_guest_rebuild, timeline_package_e2e;
* split out of host suites: the xaml/ruby/svelte "artifact that was
  recorded" tests (`<pkg>_package_record.rs`), ruby_guest_wire's rebuild
  (`ruby_guest_rebuild.rs`), the package half of first_party_profiles,
  every package row of grammar_provenance's WASM_ARTIFACTS, its .wat
  recompile and build.py rebuild (`support/package_self.rs`, one
  `<pkg>_package.rs` per package), and the per-language legs of
  language_package_overhead (`<lang>_language_overhead.rs`; shared
  machinery in crates/indexer/tests/package_overhead/).

Each package records what its executables are in
`tests/packages/<pkg>.provenance` (beside the .digest), and what its
extractor is built from as `input` rows plus a `source_inputs` aggregate
in the .digest. The rebuild-and-compare runs while those match; a moved
BUILTIN or other shared input is REPORTED, never failed.

`.forgejo/workflows/package-ci.yml`: a static 10-leg matrix,
`package (<name>)`, Linux only, each leg exiting early when out of
scope; a nightly compatibility run on its one cron, report-only.
`ci_changes.sh` emits `packages` alongside the host groups: a
package-only change turns every host group off; a shared-guest change
turns on every leg that records an input in that crate plus everything;
a host module a package target includes by #[path] reaches its legs.
Host `cargo test --workspace` (ci.yml, ci-windows.yml) excludes the
crate; the corpus job no longer names the moved Ruby suites.

host_reads_no_live_package: the allowlist holds only the phase 3
(catalog) and phase 4 (release pipeline) rows and data rows, typed by
`Home`; the package home is skipped whole. ci_path_gating grades the
matrix against the package census, target ownership, the host
exclusion, the phase 3/4 readers in every leg and the routing arms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu
No test reaches code-index-mcp with COSI_CORPUS_DIR; the acceptance runner scrubs it (#314)
Some checks failed
package-ci.yml / No test reaches code-index-mcp with COSI_CORPUS_DIR; the acceptance runner scrubs it (#314) (pull_request) Failing after 0s
CI (Windows) / changes (what this run must test) (pull_request) Successful in 20s
CI / changes (what this run must test) (pull_request) Successful in 30s
CI / cargo fmt (pull_request) Successful in 1m8s
CI / OSS corpus tier-3 scale (nightly) (pull_request) Has been skipped
CI / Grammar rebuild from source (nightly) (pull_request) Has been skipped
CI / CI lane wall-clock headroom (pull_request) Successful in 51s
CI / guest crates (fmt, clippy, doc) (pull_request) Successful in 1m31s
CI / cargo test (pull_request) Has been cancelled
CI / cargo test (daemon transport) (pull_request) Has been cancelled
CI / Plugin path cost + pool throughput (nightly) (pull_request) Has been cancelled
CI / cargo check (MSRV 1.98) (pull_request) Has been cancelled
CI / cargo deny (pull_request) Has been cancelled
CI / OSS corpus (tier 1) (pull_request) Has been cancelled
CI / cargo test (abi, 32-bit + wasm32) (pull_request) Has been cancelled
CI / cargo clippy (pull_request) Has been cancelled
CI / cargo doc (intra-doc links) (pull_request) Has been cancelled
CI / cargo check (windows-gnu) (pull_request) Has been cancelled
CI (Windows) / fmt + clippy + build + test (windows) (pull_request) Has been cancelled
dd13d3e382
Nightly run 5754: `plugin-path-cost` exports COSI_CORPUS_DIR job-wide,
`tests/phase2_acceptance.py` spawned `code-index-mcp` with it, and the
server links every pinned checkout under it (#191) and indexes it IN
PLACE. `/tmp/cosi-corpus/rust-ripgrep` was left carrying `.code-index/`
and all four `language_package_overhead` tests failed the corpus
contamination guard.

* `tests/phase2_acceptance.py` builds every child environment through
  one `child_env`, which removes COSI_CORPUS_DIR (and its two siblings)
  with the settings it already scrubbed; the ci.yml step also unsets
  them, so neither half alone is load-bearing.
* `code_index_test_support::product::command` is the one door a test
  spawns a product binary through when its file can reach the server
  (directly, or through `code-index query` / the blast-radius hook,
  which start it with what they inherit). It removes the variable.
  The 52 spawn sites in 14 test files across cli, daemon, indexer and
  mcp-server now go through it.
* `crates/test-support/tests/mcp_spawn_registry.rs` fails when a test
  that names the server builds a Command over a product binary any
  other way, and reads the removal back off a real Command against the
  name `crates/mcp-server/src/main.rs` declares.

Keeping a linked corpus project's index state OUTSIDE the checkout is
not part of this change: linked projects keep their state under their
own root, and moving it touches the daemon's project-home resolution.
Left on #314.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu
package-ci: continue-on-error is a literal; Forgejo rejected the whole workflow (#318)
Some checks failed
CI / Plugin path cost + pool throughput (nightly) (pull_request) Has been skipped
CI (Windows) / changes (what this run must test) (push) Successful in 27s
CI (Windows) / fmt + clippy + build + test (windows) (push) Has been skipped
Package CI / changes (which packages this run must test) (push) Successful in 23s
CI / changes (what this run must test) (push) Successful in 34s
CI / cargo fmt (push) Successful in 1m10s
CI / cargo clippy (push) Has been skipped
CI / cargo check (MSRV 1.98) (push) Has been skipped
CI / cargo check (windows-gnu) (push) Has been skipped
CI / cargo test (abi, 32-bit + wasm32) (push) Has been skipped
CI / cargo deny (push) Has been skipped
CI / CI lane wall-clock headroom (push) Has been skipped
CI / guest crates (fmt, clippy, doc) (push) Has been skipped
CI / cargo doc (intra-doc links) (push) Has been skipped
CI / OSS corpus (tier 1) (push) Has been skipped
CI / OSS corpus tier-3 scale (nightly) (push) Has been skipped
CI / Grammar rebuild from source (nightly) (push) Has been skipped
CI / cargo test (push) Has been skipped
CI / cargo test (daemon transport) (push) Has been skipped
CI / Plugin path cost + pool throughput (nightly) (push) Has been skipped
Package CI / package (xaml) (push) Successful in 23m53s
Package CI / package (timeline) (push) Successful in 23m55s
Package CI / package (svelte) (push) Successful in 24m2s
Package CI / package (rust) (push) Successful in 43m52s
Package CI / package (php) (push) Successful in 44m43s
Package CI / package (csharp) (push) Successful in 44m45s
Package CI / package (python) (push) Failing after 45m10s
Package CI / package (javascript) (push) Successful in 45m13s
Package CI / package (typescript) (push) Successful in 45m20s
Package CI / package (ruby) (push) Successful in 51m14s
04b5b96e3f
The first PR run of package-ci.yml failed in 0 s with no job: Forgejo
parses a step's continue-on-error as a bool, and a `${{ }}` there is a
string ("cannot unmarshal !!str into bool"). The nightly needs no
continue-on-error: no gate reads a package leg (require_ci_green.sh
grades named REQUIRED_JOBS only), so a red nightly leg is the report.

ci_path_gating::continue_on_error_is_a_literal_in_every_workflow guards
every workflow; mutation run (an expression back on one step -> red).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu
buildagent deleted branch p2-per-package-ci 2026-10-07 11:30:44 +02:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index!325
No description provided.