cargo test under load: tests that spawn the plugin-host worker fail with host.worker_unavailable #363
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
h-dv/code-index#363
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Under a loaded runner, tests that build and spawn
code-index-plugin-hostmid-run fail withhost.worker_unavailable. Unloaded runs of the same tree pass.Observed
ci.yml 1115 (push, master
85b29151; treed3285eba, identical to PR head17506152, whose ci.yml 1112 was all green in 28.6 min):cargo testtook 7302 s and 3/4873 failed:code-index-cli::plugin_update_cli::doctor_keeps_its_remediation_after_a_check_has_run:seeding the approved version failed … conformance_failed … C1 — staging: host.worker_unavailable(plugin_update_cli.rs:390)code-index-daemon::xaml_package_e2e::malformed_markup_and_oversized_input_hit_named_budgets:NotGraded,refusal: Some("host.w…")(xaml_package_e2e.rs:1789)code-index-indexer::grammar_cache::a_grammar_is_compiled_once_per_install_and_the_worker_uses_it:the worker did not compile into the cache it was handed,left: [](grammar_cache.rs:161). Its stderr showsBlocking waiting for file lock on build directory, thenCompiling code-index-plugin-host.ci.yml 1104 (nightly,
876b78b5, the previous release):cargo testtook 6614 s, andgrammar_cache::a_grammar_is_compiled_once…plushost_extraction_epoch::the_host_derivation_of_the_shipped_packages_is_pinned_to_the_epochfailed.Both slow runs overlapped with a package-ci run of 22+ jobs on the same runner host. In 1115's case that was nightly 1117.
Hypothesis (unverified)
Some tests build the worker binary themselves (
cargo build -p code-index-plugin-hostunder a shared target lock), or spawn it with a startup deadline. Under load, the build or the spawn overruns, and the refusal is a deadline, not a defect. The test then reports a product refusal.NotGradedis at least honest about that, but CI goes red.Next
stress-ngor a parallel corpus leg), and make the failure print the deadline and elapsed time.COSI_PLUGIN_HOST_PREBUILT, already used bypackage_leg.sh); give the startup deadline a load-independent basis; or serialize the nightly package-ci against ci.yml.Cost so far: master CI red after the #360 merge, and a manual job re-run before the release tag.
Related: package overhead budget (filed alongside).
More evidence. It weakens the "load from a parallel package-ci run" half of the hypothesis.
85b29151, no package-ci run of this repo in parallel):cargo testran 6800 s, with 4 failures, allhost.worker_unavailable. The fourth isplugin_registry_e2e::a_package_absent_from_the_catalog_is_unreachable_by_name.changesoutputs (mode=full,guest=true). The suite took 747 s vs 7302 s. The initial build is comparable (10m39 vs 11m14 for 1118), so the cache isn't the difference.package_release_scripts::every_first_party_package_releases_through_the_shared_scripts619 s,plugin_activation_cli::*about 480 s each,query_cli::*140–260 s. The log showsBlocking waiting for file lock on build directory.worker_unavailablefailures happen in 0.2–0.6 s, which looks like the worker binary being absent or mid-rebuild when spawned.Revised hypothesis, still unverified: in-test
cargo buildofcode-index-plugin-hostand the guest packages races the suite's other in-test builds on one target dir. A spawn that lands while the binary is being replaced fails at once. Why only push/schedule is 10× slower is not explained yet. Nightly ci.yml also runs tier-3 and the plugin-path-cost jobs alongside, but 1115 (push) didn't.The first step is to diff the nextest per-test timings of 1112 and 1115 by test binary, and to find which test triggers the rebuild mid-suite.
Released in v0.37.0. The fix is
17fa0ff2, merged in #364.crates/build_version.rsnow watches only paths that exist. Before, it always watched.git/packed-refs, which a push or schedule checkout does not have, and cargo treats a missing watched path as always stale.packed-refs). Three rebuilds compiled nothing (0.3 s each). A new commit rebuilt and stamped the new hash.e182fccb(run 1126) took 28.9 min forcargo test, down from 7302 s on run 1115. Noworker_unavailablefailures.Closing.