cargo test under load: tests that spawn the plugin-host worker fail with host.worker_unavailable #363

Closed
opened 2026-10-13 06:06:23 +02:00 by buildagent · 2 comments
Member

Under a loaded runner, tests that build and spawn code-index-plugin-host mid-run fail with host.worker_unavailable. Unloaded runs of the same tree pass.

Observed

ci.yml 1115 (push, master 85b29151; tree d3285eba, identical to PR head 17506152, whose ci.yml 1112 was all green in 28.6 min): cargo test took 7302 s and 3/4873 failed:

  • code-index-cli::plugin_update_cli::doctor_keeps_its_remediation_after_a_check_has_run: seeding the approved version failed … conformance_failed … C1 — staging: host.worker_unavailable (plugin_update_cli.rs:390)
  • code-index-daemon::xaml_package_e2e::malformed_markup_and_oversized_input_hit_named_budgets: NotGraded, refusal: Some("host.w…") (xaml_package_e2e.rs:1789)
  • code-index-indexer::grammar_cache::a_grammar_is_compiled_once_per_install_and_the_worker_uses_it: the worker did not compile into the cache it was handed, left: [] (grammar_cache.rs:161). Its stderr shows Blocking waiting for file lock on build directory, then Compiling code-index-plugin-host.

ci.yml 1104 (nightly, 876b78b5, the previous release): cargo test took 6614 s, and grammar_cache::a_grammar_is_compiled_once… plus host_extraction_epoch::the_host_derivation_of_the_shipped_packages_is_pinned_to_the_epoch failed.

Both slow runs overlapped with a package-ci run of 22+ jobs on the same runner host. In 1115's case that was nightly 1117.

Hypothesis (unverified)

Some tests build the worker binary themselves (cargo build -p code-index-plugin-host under a shared target lock), or spawn it with a startup deadline. Under load, the build or the spawn overruns, and the refusal is a deadline, not a defect. The test then reports a product refusal. NotGraded is at least honest about that, but CI goes red.

Next

  • Reproduce: run the three tests under CPU contention (e.g. stress-ng or a parallel corpus leg), and make the failure print the deadline and elapsed time.
  • Fix candidates: prebuild the worker once per run (COSI_PLUGIN_HOST_PREBUILT, already used by package_leg.sh); give the startup deadline a load-independent basis; or serialize the nightly package-ci against ci.yml.

Cost so far: master CI red after the #360 merge, and a manual job re-run before the release tag.

Related: package overhead budget (filed alongside).

Under a loaded runner, tests that build and spawn `code-index-plugin-host` mid-run fail with `host.worker_unavailable`. Unloaded runs of the **same tree** pass. ## Observed **ci.yml 1115** (push, master 85b29151; tree d3285eba, identical to PR head 17506152, whose ci.yml 1112 was all green in 28.6 min): `cargo test` took **7302 s** and 3/4873 failed: - `code-index-cli::plugin_update_cli::doctor_keeps_its_remediation_after_a_check_has_run`: `seeding the approved version failed … conformance_failed … C1 — staging: host.worker_unavailable` (plugin_update_cli.rs:390) - `code-index-daemon::xaml_package_e2e::malformed_markup_and_oversized_input_hit_named_budgets`: `NotGraded`, `refusal: Some("host.w…")` (xaml_package_e2e.rs:1789) - `code-index-indexer::grammar_cache::a_grammar_is_compiled_once_per_install_and_the_worker_uses_it`: `the worker did not compile into the cache it was handed`, `left: []` (grammar_cache.rs:161). Its stderr shows `Blocking waiting for file lock on build directory`, then `Compiling code-index-plugin-host`. **ci.yml 1104** (nightly, 876b78b5, the previous release): `cargo test` took **6614 s**, and `grammar_cache::a_grammar_is_compiled_once…` plus `host_extraction_epoch::the_host_derivation_of_the_shipped_packages_is_pinned_to_the_epoch` failed. Both slow runs overlapped with a package-ci run of 22+ jobs on the same runner host. In 1115's case that was nightly 1117. ## Hypothesis (unverified) Some tests build the worker binary themselves (`cargo build -p code-index-plugin-host` under a shared target lock), or spawn it with a startup deadline. Under load, the build or the spawn overruns, and the refusal is a deadline, not a defect. The test then reports a product refusal. `NotGraded` is at least honest about that, but CI goes red. ## Next - Reproduce: run the three tests under CPU contention (e.g. `stress-ng` or a parallel corpus leg), and make the failure print the deadline and elapsed time. - Fix candidates: prebuild the worker once per run (`COSI_PLUGIN_HOST_PREBUILT`, already used by `package_leg.sh`); give the startup deadline a load-independent basis; or serialize the nightly package-ci against ci.yml. Cost so far: master CI red after the #360 merge, and a manual job re-run before the release tag. Related: package overhead budget (filed alongside).
Author
Member

More evidence. It weakens the "load from a parallel package-ci run" half of the hypothesis.

  • ci.yml 1118 (nightly 05:00, same commit 85b29151, no package-ci run of this repo in parallel): cargo test ran 6800 s, with 4 failures, all host.worker_unavailable. The fourth is plugin_registry_e2e::a_package_absent_from_the_catalog_is_unreachable_by_name.
  • Same runner, same config, 10× apart. PR run 1112 and push run 1115 ran on the same runner (989ccda6), with identical changes outputs (mode=full, guest=true). The suite took 747 s vs 7302 s. The initial build is comparable (10m39 vs 11m14 for 1118), so the cache isn't the difference.
  • Where the time goes. 250 tests are SLOW on push/schedule against 1 on the PR. The worst are tests that shell out to cargo or build binaries themselves: package_release_scripts::every_first_party_package_releases_through_the_shared_scripts 619 s, plugin_activation_cli::* about 480 s each, query_cli::* 140–260 s. The log shows Blocking waiting for file lock on build directory.
  • The failures don't look like timeouts. Several worker_unavailable failures happen in 0.2–0.6 s, which looks like the worker binary being absent or mid-rebuild when spawned.

Revised hypothesis, still unverified: in-test cargo build of code-index-plugin-host and the guest packages races the suite's other in-test builds on one target dir. A spawn that lands while the binary is being replaced fails at once. Why only push/schedule is 10× slower is not explained yet. Nightly ci.yml also runs tier-3 and the plugin-path-cost jobs alongside, but 1115 (push) didn't.

The first step is to diff the nextest per-test timings of 1112 and 1115 by test binary, and to find which test triggers the rebuild mid-suite.

More evidence. It weakens the "load from a parallel package-ci run" half of the hypothesis. - **ci.yml 1118** (nightly 05:00, same commit 85b29151, no package-ci run of this repo in parallel): `cargo test` ran **6800 s**, with 4 failures, all `host.worker_unavailable`. The fourth is `plugin_registry_e2e::a_package_absent_from_the_catalog_is_unreachable_by_name`. - **Same runner, same config, 10× apart.** PR run 1112 and push run 1115 ran on the same runner (989ccda6), with identical `changes` outputs (`mode=full`, `guest=true`). The suite took 747 s vs 7302 s. The initial build is comparable (10m39 vs 11m14 for 1118), so the cache isn't the difference. - **Where the time goes.** 250 tests are SLOW on push/schedule against 1 on the PR. The worst are tests that shell out to cargo or build binaries themselves: `package_release_scripts::every_first_party_package_releases_through_the_shared_scripts` 619 s, `plugin_activation_cli::*` about 480 s each, `query_cli::*` 140–260 s. The log shows `Blocking waiting for file lock on build directory`. - **The failures don't look like timeouts.** Several `worker_unavailable` failures happen in 0.2–0.6 s, which looks like the worker binary being absent or mid-rebuild when spawned. Revised hypothesis, still unverified: in-test `cargo build` of `code-index-plugin-host` and the guest packages races the suite's other in-test builds on one target dir. A spawn that lands while the binary is being replaced fails at once. Why only push/schedule is 10× slower is **not explained yet**. Nightly ci.yml also runs tier-3 and the plugin-path-cost jobs alongside, but 1115 (push) didn't. The first step is to diff the nextest per-test timings of 1112 and 1115 by test binary, and to find which test triggers the rebuild mid-suite.
Author
Member

Released in v0.37.0. The fix is 17fa0ff2, merged in #364.

crates/build_version.rs now watches only paths that exist. Before, it always watched .git/packed-refs, which a push or schedule checkout does not have, and cargo treats a missing watched path as always stale.

  • Locally: I used a fresh clone in CI's layout (no packed-refs). Three rebuilds compiled nothing (0.3 s each). A new commit rebuilt and stamped the new hash.
  • CI: the master push run on e182fccb (run 1126) took 28.9 min for cargo test, down from 7302 s on run 1115. No worker_unavailable failures.

Closing.

Released in **v0.37.0**. The fix is 17fa0ff2, merged in #364. `crates/build_version.rs` now watches only paths that exist. Before, it always watched `.git/packed-refs`, which a push or schedule checkout does not have, and cargo treats a missing watched path as always stale. - **Locally:** I used a fresh clone in CI's layout (no `packed-refs`). Three rebuilds compiled nothing (0.3 s each). A new commit rebuilt and stamped the new hash. - **CI:** the master push run on e182fccb (run 1126) took 28.9 min for `cargo test`, down from 7302 s on run 1115. No `worker_unavailable` failures. Closing.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#363
No description provided.