h-dv/code-index
Watch
4
0
Fork
You've already forked code-index
 0
Code Issues 21 Pull requests Projects Releases 194 Packages Wiki Activity Actions

The nightly runs have been red on 11 of 13 completed runs since 2026-11-06 and nothing reports it: overhead budgets, two intermittent startup failures, a step-guard cascade #582

New issue
Open
opened 2026-11-12 22:21:52 +01:00 by buildagent · 10 comments
buildagent commented 2026-11-12 22:21:52 +01:00
Member
Copy link

Scope

The scheduled runs are the only place where the shipped-shape checks, the package overhead budgets, the worker-pool throughput and the mixed-load ceilings are graded. Between 2026-11-06 and 2026-11-12 package-ci.yml failed on every completed night and ci.yml on four of six. Nobody was told, and no ticket covered any of it. A nightly that is red for unrelated reasons every night cannot report a new regression: the startup regression #581 landed in exactly this period.

Attributed from the job logs on 2026-11-12 (read-only; no suite was re-run). The full attribution with run links, log lines and commit ranges is kept by the lead; the causes are listed here.

Priority: High · Risk: none-internal today, but it hides regressions (cost, hang) · Size: M
Merged into this ticket: #288 (mixed_load_ceilings flaky at ladder x64)

Open causes

# Cause Nights Runs on PR/push? What is established Not verified
1 Package overhead budgets: its_package_matches_native_semantics_and_fits_cold_and_warm_budgets (crates/first-party-tests/tests/support/language_overhead.rs:432) every completed night 11-06 … 11-12 no (schedule and dispatch only) Timing only, semantic_failure: null. Warm ratio: ruby-sinatra 1.10–1.14 against 1.10 on six nights, cs-dapper 1.33–1.34 against 1.15 on two, python-flask and ts-zod once each. On 11-12, for the first time, three COLD budgets failed (js-express, python-flask, ruby-sinatra), first red in the commit range eea2214a..a7ce9603, the same range in which #581 appeared. Whether the warm misses are noise near a tight bound or a real regression; whether the cold misses share #581's cause. No profile, no bisection.
2 script_daemon_imports::named_script_imports_follow_configuration_exports_and_daemon_restart: "the daemon EXITED before it became ready" (JavaScript and TypeScript package legs) 11-11, once yes (package legs, scoped by changed package) The daemon process died during its first start, after 19–24 s; it is not a timeout. The test starts it with stdout and stderr on /dev/null, so the reason is lost. Exit status, signal, and cause.
3 tests/phase2_acceptance.py, check_missing_embedded: plugin add of the TypeScript package into a project with a live daemon exhausted its 180 s deadline over plugin_mutation_busy retries 11-12, once yes (step "Check signed generic runtime acceptance" of cargo test) The nextest suites passed (5571/5571). First red in eea2214a..a7ce9603. What held the mutation lock, how many retries ran, and whether the last child was working or blocked. Related trackers: #520 (one writer), #581 (startup cost).
4 "no CLI at target/release/code-index" in the seven-language parity step 11-08, 11-09 no A step-guard mismatch: "Build this tree's release CLI" runs only on success, its consumer runs on `success()
5 Runner capacity admission: require_idle_runner.py timed out, "runner did not become quiet; measurement not started" 11-08, once no Load 8.1 on 8 CPUs against a limit of 2.0 for 90 minutes. The following "Mixed-load metrics" step then fails on the missing report: a refused measurement is reported as a failed one. What produced the load.
6 #288: mixed_load_ceilings at ladder x64, host.memory_ceiling_exceeded last documented 2026-09-21 (run 856) no Did not occur in this period: the test passed on every night it ran (11-05, 06, 07, 09, 11). Its hardening is still missing: the ladder input is the largest *.json anywhere under the corpus directory (mixed_load_bench.rs:771-807), so a polluted checkout changes the measurement, and the refusal does not report the requested bytes. Older failed logs have expired, so a later occurrence cannot be excluded.

Already repaired (recorded so they are not re-investigated)

  • PHP native/package diagnostic compared two different host contexts (11-05, 11-06); replaced by pinned_php_package_cli_matches_in_process_index in 33d9c8ce.
  • Worker-pool speedup 1.9x under the 4.0x floor (11-05, 11-06); passes since the workflow prebuilds the worker and unsets RAYON_NUM_THREADS (33d9c8ce). Which of the adjustments mattered was not isolated.
  • Shipped-shape Vue symbol callee_vue missing (11-08, 11-09); passes since 37ac4101 (package languages added later take part in activation).
  • cargo deny: RUSTSEC-2026-0462 in rmcp 3.7.0 (11-10); rmcp 3.9.0 since f7ee339e.

Wanted

  1. Cause 1: measure before changing a bound. Run the overhead job by dispatch on the tree with the #581 fix; if the cold misses are gone, say so in #581. For the warm ratios, compare an isolated run against an isolated run and decide per package: regression (attribute it) or a bound tighter than the measurement's noise (state the measured spread and set the bound from it).
  2. Causes 2 and 3: make the failure print its state. A test daemon's stderr goes to a file that the panic prints; the acceptance script prints each retry, the lock holder if known, and the child's output on timeout. Then attribute on the next occurrence.
  3. Cause 4: the consumer step requires the build step's outcome, not only the job state.
  4. Cause 5: a refused measurement ends as "not measured" and is visible as such, without failing the following steps.
  5. Cause 6 (#288): pin the ladder input to a tracked file; report requested bytes and the cap in the refusal.
  6. Reporting: a red scheduled run reaches the operator or the tracker on the same day (for example a job that comments on a standing ticket with the failed jobs). Without it this ticket repeats.

Acceptance

Seven consecutive nights in which both scheduled workflows are green or every red job is attributed in this ticket within a day.

## Scope The scheduled runs are the only place where the shipped-shape checks, the package overhead budgets, the worker-pool throughput and the mixed-load ceilings are graded. Between 2026-11-06 and 2026-11-12 `package-ci.yml` failed on every completed night and `ci.yml` on four of six. Nobody was told, and no ticket covered any of it. A nightly that is red for unrelated reasons every night cannot report a new regression: the startup regression #581 landed in exactly this period. Attributed from the job logs on 2026-11-12 (read-only; no suite was re-run). The full attribution with run links, log lines and commit ranges is kept by the lead; the causes are listed here. **Priority:** High · **Risk:** none-internal today, but it hides regressions (cost, hang) · **Size:** M **Merged into this ticket:** #288 (`mixed_load_ceilings` flaky at `ladder x64`) ## Open causes | # | Cause | Nights | Runs on PR/push? | What is established | Not verified | |---|---|---|---|---|---| | 1 | Package overhead budgets: `its_package_matches_native_semantics_and_fits_cold_and_warm_budgets` (`crates/first-party-tests/tests/support/language_overhead.rs:432`) | every completed night 11-06 … 11-12 | no (schedule and dispatch only) | Timing only, `semantic_failure: null`. Warm ratio: `ruby-sinatra` 1.10–1.14 against 1.10 on six nights, `cs-dapper` 1.33–1.34 against 1.15 on two, `python-flask` and `ts-zod` once each. On 11-12, for the first time, three COLD budgets failed (`js-express`, `python-flask`, `ruby-sinatra`), first red in the commit range `eea2214a..a7ce9603`, the same range in which #581 appeared. | Whether the warm misses are noise near a tight bound or a real regression; whether the cold misses share #581's cause. No profile, no bisection. | | 2 | `script_daemon_imports::named_script_imports_follow_configuration_exports_and_daemon_restart`: "the daemon EXITED before it became ready" (JavaScript and TypeScript package legs) | 11-11, once | yes (package legs, scoped by changed package) | The daemon process died during its first start, after 19–24 s; it is not a timeout. The test starts it with stdout and stderr on `/dev/null`, so the reason is lost. | Exit status, signal, and cause. | | 3 | `tests/phase2_acceptance.py`, `check_missing_embedded`: `plugin add` of the TypeScript package into a project with a live daemon exhausted its 180 s deadline over `plugin_mutation_busy` retries | 11-12, once | yes (step "Check signed generic runtime acceptance" of `cargo test`) | The nextest suites passed (5571/5571). First red in `eea2214a..a7ce9603`. | What held the mutation lock, how many retries ran, and whether the last child was working or blocked. Related trackers: #520 (one writer), #581 (startup cost). | | 4 | "no CLI at target/release/code-index" in the seven-language parity step | 11-08, 11-09 | no | A step-guard mismatch: "Build this tree's release CLI" runs only on success, its consumer runs on `success() || failure()`. Any earlier failed step in the job turns into two further failures that say nothing about the CLI. | — | | 5 | Runner capacity admission: `require_idle_runner.py` timed out, "runner did not become quiet; measurement not started" | 11-08, once | no | Load 8.1 on 8 CPUs against a limit of 2.0 for 90 minutes. The following "Mixed-load metrics" step then fails on the missing report: a refused measurement is reported as a failed one. | What produced the load. | | 6 | #288: `mixed_load_ceilings` at `ladder x64`, `host.memory_ceiling_exceeded` | last documented 2026-09-21 (run 856) | no | Did not occur in this period: the test passed on every night it ran (11-05, 06, 07, 09, 11). Its hardening is still missing: the ladder input is the largest `*.json` anywhere under the corpus directory (`mixed_load_bench.rs:771-807`), so a polluted checkout changes the measurement, and the refusal does not report the requested bytes. | Older failed logs have expired, so a later occurrence cannot be excluded. | ## Already repaired (recorded so they are not re-investigated) - PHP native/package diagnostic compared two different host contexts (11-05, 11-06); replaced by `pinned_php_package_cli_matches_in_process_index` in `33d9c8ce`. - Worker-pool speedup 1.9x under the 4.0x floor (11-05, 11-06); passes since the workflow prebuilds the worker and unsets `RAYON_NUM_THREADS` (`33d9c8ce`). Which of the adjustments mattered was not isolated. - Shipped-shape Vue symbol `callee_vue` missing (11-08, 11-09); passes since `37ac4101` (package languages added later take part in activation). - `cargo deny`: RUSTSEC-2026-0462 in rmcp 3.7.0 (11-10); rmcp 3.9.0 since `f7ee339e`. ## Wanted 1. Cause 1: measure before changing a bound. Run the overhead job by dispatch on the tree with the #581 fix; if the cold misses are gone, say so in #581. For the warm ratios, compare an isolated run against an isolated run and decide per package: regression (attribute it) or a bound tighter than the measurement's noise (state the measured spread and set the bound from it). 2. Causes 2 and 3: make the failure print its state. A test daemon's stderr goes to a file that the panic prints; the acceptance script prints each retry, the lock holder if known, and the child's output on timeout. Then attribute on the next occurrence. 3. Cause 4: the consumer step requires the build step's outcome, not only the job state. 4. Cause 5: a refused measurement ends as "not measured" and is visible as such, without failing the following steps. 5. Cause 6 (#288): pin the ladder input to a tracked file; report requested bytes and the cap in the refusal. 6. Reporting: a red scheduled run reaches the operator or the tracker on the same day (for example a job that comments on a standing ticket with the failed jobs). Without it this ticket repeats. ## Acceptance Seven consecutive nights in which both scheduled workflows are green or every red job is attributed in this ticket within a day.
buildagent commented 2026-11-12 22:23:31 +01:00
Author
Member
Copy link

Attribution of the scheduled runs 2026-11-06 to 2026-11-12, as written by the attribution worker on 2026-11-12 (read-only; the linked commits-*.txt and log-*.txt files are local evidence and are not attached).

Full attribution

Nightly failure attribution — 2026-11-06 through 2026-11-12

Read-only investigation of master 3c8742bf5499ee2145e0238517e6c1f6df5a11fb. No suites run, fixes attempted, repository files changed, or Forgejo writes made. Dates below are the run's created date in the API's Europe/Berlin offset; package jobs often execute on the previous UTC date. The requested seven-day window includes November 6, in addition to the six dates supplied in the question.

There are 11 failed scheduled runs and 13 failed jobs, grouped into nine failure signatures below. A shared symptom family is not proof of a shared low-level defect. Cancelled, skipped, and admission-refused measurements are distinguished from a benchmark result.

Evidence: runs-*.json, scheduled-all.json, jobs-<API run id>.json, and log-<API run id>-<zero-based job index>.txt in this directory. The API run id differs from the number in html_url. Each log was obtained through <html_url>/jobs/<index>/attempt/1/logs. Source was read with code-index, including read_code(rev=...) at historical commits; selected replies are retained in source-evidence.json. Code-index requires read_code.response_format=json and offers no format argument on index_coverage/project_overview; those were the only necessary exceptions to concise. All calls used envelope=minimal. No shell source-reading fallback was used. Absolute-path coverage confirmed the workflow and shared daemon fixture belong to this worktree.

1. Package cold/warm overhead budgets — size M

Jobs and nights: package-ci.yml, index 24, package overhead budgets (nightly, one package at a time); step Both overhead budgets, one package at a time. It fails on every completed nightly in this window: November 6, 7, 9, 10, 11, 12. November 8 was cancelled and is an absent measurement. Individual packages vary, so this is persistent job redness with intermittent package-specific failures.

Every assertion below is test its_package_matches_native_semantics_and_fits_cold_and_warm_budgets, at crates/first-party-tests/tests/support/language_overhead.rs:432. These are the decisive assertion messages, not merely the final shell aggregate:

Night UI run / API id First decisive messages (one per failed package)
Nov 6 2252 / 7934 cs-dapper warm ratio 1.334 exceeds 1.15; ruby-sinatra warm ratio 1.114 exceeds 1.10
Nov 7 2321 / 8045 cs-dapper warm ratio 1.342 exceeds 1.15; python-flask warm ratio 1.126 exceeds 1.10; ruby-sinatra warm ratio 1.100 exceeds 1.10 (unrounded ratio 1.1003902840663624)
Nov 9 2472 / 8275 ruby-sinatra warm ratio 1.112 exceeds 1.10
Nov 10 2511 / 8319 ruby-sinatra warm ratio 1.137 exceeds 1.10
Nov 11 2535 / 8348 ruby-sinatra warm ratio 1.135 exceeds 1.10; ts-zod warm ratio 1.101 exceeds 1.10
Nov 12 2560 / 8513 js-express cold: package 1.431 s exceeds the limit 1.430 s (= 1.20 x builtin 0.775 s + allowance 0.50 s); ratio 1.846; python-flask cold: package 1.550 s exceeds the limit 1.501 s (= 1.20 x builtin 0.834 s + allowance 0.50 s); ratio 1.857; ruby-sinatra cold: package 1.633 s exceeds the limit 1.608 s (= 1.20 x builtin 0.923 s + allowance 0.50 s); ratio 1.768

First appearance / range: the current streak starts November 6. The preceding November 5 overhead job, API 7849 / UI 2199, is green at fb4eb790b25a3cb0411bb4b782346feebe2940cb; first red is 0a6599d6b0fcb178e2c6b9e18620e71c848d3878. Earlier October runs also had red overhead jobs, but their logs are unavailable, so this is not a claim of first-ever occurrence. git log --oneline fb4eb790..0a6599d6 -- <measurement/package paths> is empty for the helper, guest code, package manifests and workflow (command and output). Expanding to all first-party tests and indexer runtime finds f54a8758 (cold constant lookup), 11bf0021 (mixed-language invalidation/scoped constants), 50726eb4 (Svelte tests), and 56dbb24a (basename fact ABI conformance): expanded log. These are candidates, not a bisection.

Package-specific transitions: C#/Ruby first red Nov 6 after Nov 5 green; Python first red Nov 7 after Nov 6 green (0a6599d6..b2405a4e); TypeScript first red Nov 11 after Nov 10 green (e3ec6ead..eea2214a); JavaScript and the cold failures first red Nov 12 after Nov 11 green (eea2214a..a7ce9603). Other reds in those workflows do not invalidate their successful measurements.

Schedule-only: yes, plus manual dispatch. .forgejo/workflows/package-ci.yml:400-407 gates package-overhead to schedule/dispatch; PR and push runs do not assert these timings. Its needs: [package, package-corpus] and success() || failure() deliberately permit measurements after a failed package matrix leg; Nov 11 overhead is an independent failure, not a needs cascade. Package suites on PR/push grade semantic parity separately.

Issue: no issue matching these budget failures among the 40 open issue titles. #581 concerns a different, shipped-process startup slowdown, not this paired in-process index measurement; its body does not establish the same cause. #534's stale-pin problem and #515's duplicate extractors are related measurement concerns, not this assertion signature.

Established: the helper takes medians of paired native/package rounds, alternates execution order, and separately asserts semantics and budgets (language_overhead.rs:367-440). Cold bound is 1.20 * native + allowance; warm bounds are 1.15 for C#, 1.20 for PHP, 1.10 otherwise (:25-103). The logged failed rows have semantic_failure:null; these are timing failures. Some misses are extremely small: Nov 12 JavaScript is 0.000751 s over its cold limit. Load readings are present, e.g. Nov 7 C# ended at 3.62 7.98 33.63, and Nov 9 Ruby at 1.15 7.03 27.15. The measurements are serial within this job, but cannot exclude other workflows or host activity.

Not verified: which costs account for the extra time, whether the small misses are noise or a runtime regression, whether all affected packages share a mechanism, and the first-ever failing night. Load alone proves neither contention nor causation. No profiling or rerun was performed.

2. PHP native/package diagnostic compared different host semantics — size S

Jobs and nights: ci.yml, index 14, Plugin path cost + pool throughput (nightly), step Seven-language parity, no-package CLI against builtins and package overhead. November 6 2253, API 7936; also directly verified November 5 2200, API 7854, just outside the window.

Error: thread 'pinned_php_binding_diagnostic' ... panicked at crates/indexer/tests/language_package_overhead.rs:274:5: followed by Err("references and targets: 38733 vs 38733 rows; first 69: ..."). The first difference is src/Auth/DigestAuth.php, RuntimeException, type reference at 296:28, flags 65536, differing in the following 0/1 field. Subsequent Utils rows bind to src/Utils.php#Utils@18 on one leg and are Psr7/unresolved on the other. Full literal message: log-7936-14.txt:1705.

First appearance / range: earliest directly verified November 5 (fb4eb790b25a3cb0411bb4b782346feebe2940cb); still red Nov 6, green Nov 7. Last preceding whole job green was October 25, API 6671, e47526105f9c2096db2ac7248dc896f892446376. Thus conservative observed bracket e4752610..fb4eb790, not an exact first-red bracket: October 26's lane was red and its log returns 404; October 27–Nov 4 were blocked/cancelled. Relevant path log includes 0aea4f69 (PHP language-rule resolution), 716be0f5 (type operand normalization), b6715cc0 (package facts), and 97a1bef5 (plugin proofs). No culprit inferred from their titles.

Schedule-only: yes; the enclosing plugin-path-cost job runs on schedule/manual dispatch only (ci.yml:1691). The diagnostic itself was ignored in ordinary cargo test.

Issue: no issue for this exact parity diagnostic. #531's PHP wrong-bind report, #515's two extractor implementations, and #534's corpus population issue are related but describe different failures.

Established: historical source at 0a6599d6 shows an ignored persistent diagnostic comparing native and package databases with exact parity. By b2405a4e, commit 33d9c8ce replaced it with pinned_php_package_cli_matches_in_process_index, explicitly discovers the host with the operator-vouched store, asserts packages are not builtin_equivalent, and compares matching CLI/in-process package contexts. This explains a real test-setup mismatch, and the Nov 7 log passes the replacement. Recovery path log.

Not verified: a source-level proof that every old row delta is explained by publisher classification, or that no genuine PHP extractor difference existed. Exact first failing night is unavailable. The replacement's later Nov 8/9 NotFound failures have a separate cause below.

3. Worker-pool speedup below its two-sweep floor — size M

Jobs and nights: same CI job index 14; step Worker-pool throughput against one lane. Nov 6 API 7936 / UI 2253; also Nov 5 API 7854 / UI 2200.

Error: thread 'throughput_before_and_after' ... panicked at crates/indexer/tests/package_pool.rs:843:9: then 8 lanes gave 1.93x over one lane, under the 4.00x floor ... ON TWO INDEPENDENT SWEEPS (Nov 6). Sweeps were 1.93x and 1.87x; it grades the better. Nov 5 was 1.98x. Nov 6 end load was 3.54 2.95 3.33.

First appearance / range: earliest directly verified Nov 5, repeats Nov 6; Nov 7, 8, 9, 11 measurements pass. Nov 10/12 lane skipped, not a pass. Same conservative October 25 green e47526105f9c2096db2ac7248dc896f892446376 to Nov 5 first verified red fb4eb790b25a3cb0411bb4b782346feebe2940cb bracket as cause 2, with censored intermediate runs. Path log and recovery log. The package_pool.rs-only log in that introduction bracket is empty: the assertion need not have changed for its measurement to fail.

Schedule-only: yes, plus dispatch; ignored test under plugin-path-cost, absent from normal PR/push execution.

Issue: no issue matching this two-sweep throughput failure. #288 is a different test and different failure mode.

Established: the source computes one-lane/wide-lane elapsed ratios, remeasures if below floor_for, and asserts the better of two against half-linear speedup capped at 4 (package_pool.rs:752-856). The runner reports eight cores, at the strictest point of that floor. In 33d9c8ce, the workflow adds an explicit test prebuild, fresh release worker build, idle admission, COSI_PLUGIN_HOST_PREBUILT=1, and unsets RAYON_NUM_THREADS; Nov 7 passes. Historical workflow reads before/after are retained in source-evidence.json.

Not verified: contention, CPU quota/topology, a worker/build-shape mismatch, and a real throughput regression remain competing explanations. The change and recovery are correlated, not a measured isolation of which adjustment mattered. First-ever failing night cannot be recovered from expired logs.

4. Runner capacity admission timed out before mixed-load test — size S

Job/night: ci.yml, Nov 8 2425, API 8214, index 14; step Plugin-host mixed-load ceilings (#79 acceptance 6).

First decisive error: log-8214-14.txt:1062: {"benchmark_capacity": "timeout", "last_sample": {"load_1m": 8.1240234375, "visible_cpus": 8, "load_limit": 2.0}, "reason": "runner did not become quiet; measurement not started"}. python3 .forgejo/scripts/require_idle_runner.py exits 1. Next step Mixed-load metrics also exits 1: cat: target/corpus/plugin-host-mixed-load.json: No such file or directory. That missing report is a downstream effect, not a second mixed-load test failure.

First appearance / range: Nov 8, intermittent (Nov 7 and 9 admission succeeds). Last green Nov 7 b2405a4e9c715c2eea77e55c6e748e009d69f20c, first red Nov 8 e7510767f7cd33a9d741a16673fb68e6742e4cfd. git log --oneline b2405a4e..e7510767 -- .forgejo/scripts/require_idle_runner.py .forgejo/workflows/ci.yml is empty: transcript. This is an environment/precondition transition, not evidence of a source regression in that range.

Schedule-only: yes, plus dispatch, inside plugin-path-cost. PR/push does not invoke this precondition/test step.

Issue: no issue matching this capacity timeout. It is not #288: no mixed_load_ceilings sample ran.

Established: the historical gate requires load <= 0.25 per visible CPU continuously for 10 seconds within 90 minutes; quota can reduce the CPU count. Logs show it waited and never obtained admission. A cancelled package-overhead job overlapped this interval and logged its own long waits; cancellation remains absent measurement.

Not verified: which process/workflow produced the load, whether the load was CPU or I/O pressure, or whether a different runner would admit it. No benchmark verdict can be inferred from this refusal.

5. Shipped-shape Vue script symbol absent after package installation — size M

Jobs/nights: ci.yml index 14, step Shipped shape through CLI, MCP, bundle and changed_symbols; Nov 8 API 8214 / UI 2425 and Nov 9 2494, API 8298. Command: python3 tests/shipped_shape_acceptance.py --bin-dir target/shipped-shape/release --out target/corpus/shipped-shape-acceptance.json.

Error: AssertionError: package/watch did not produce Comp.vue:callee_vue: {'error': 'symbol_not_found', ...} (log-8214-14.txt:1470, log-8298-14.txt:1645). Traceback originates in shipped_shape_acceptance.py:330 -> check_queries:172 -> phase2_acceptance.py:319. This is the shipped-shape script importing a helper, not the earlier generic package-only script.

First appearance / range: first Nov 8 after Nov 7 green: b2405a4e9c715c2eea77e55c6e748e009d69f20c..e7510767f7cd33a9d741a16673fb68e6742e4cfd. Repeats Nov 9, passes Nov 11; Nov 10/12 enclosing job skipped. Introduction path log includes 0fb55728 (deactivate excluded languages), ed1d0146 (builtin ownership), and package approval/recovery changes. The fixture file's own log in the introduction interval is empty. Recovery log includes 37ac4101 (shipped package-language activation), 4a971b16 (delegated script languages), 072e4125 (activation census).

Schedule-only: this precise shipped-shape check is inside the nightly/dispatch job. PR/push does run generic candidate runtime acceptance in cargo test (ci.yml:986), but that is a different build shape and script; it does not execute this shipped-default check.

Issue: no issue matching this exact missing-symbol nightly. #548 describes template/import and framework recall gaps, rather than a missing declaration inside a script after language/package activation. #515 discusses build-shape coverage; neither proves this cause. Not #288.

Established: the fixture enables the seven ordinary languages, then adds Vue separately; it expects callee_vue from the .vue TypeScript region. The log's final search says no such symbol and freshness metadata_agrees, incomplete:false. Recovery commit 37ac4101 explicitly brings newly added package languages into activation preview and recorded language decisions and adds a packages-only Vue regression test. The Nov 11 nightly passes. This strongly implicates package language activation/delegation, rather than a mere unresolved caller.

Not verified: which introduction commit caused it; whether the missing row on each failed runner was specifically a language-decision exclusion, embedded producer refusal, or another activation path. The search answer alone does not expose Comp.vue's contribution/refusal rows. No bisect or reproduction was run.

6. Later CLI parity tests run after the CLI build is skipped — size S

Jobs/nights: CI index 14, Nov 8 and 9 (same runs as cause 5), step Seven-language parity, no-package CLI against builtins and package overhead.

First errors: no_package_cli_index_matches_the_in_process_builtins panics at language_package_overhead.rs:99:5: no CLI at /workspace/h-dv/code-index/target/release/code-index. pinned_php_package_cli_matches_in_process_index then panics at :293:10: called Result::unwrap() on an Err value: Os { code: 2, kind: NotFound, message: "No such file or directory" }. See Nov 8 log lines 1567–1574; Nov 9 lines 1742–1749.

First appearance / range: first Nov 8, repeats Nov 9, passes Nov 7 and Nov 11. Same last green b2405a4e9c715c2eea77e55c6e748e009d69f20c to first red e7510767f7cd33a9d741a16673fb68e6742e4cfd bracket and path log. No workflow change appears in this interval. This is a newly exposed conditional-step cascade, not demonstrated CLI source breakage.

Schedule-only: yes, plus dispatch. These ignored parity invocations live in plugin-path-cost.

Issue: no issue for the missing-build dependency/step guard mismatch.

Established: Build this tree's release CLI has the default success-only condition, while its consumer has if: success() || failure(). On Nov 8 earlier admission and shipped-shape steps fail; on Nov 9 shipped-shape fails. Consequently the CLI build does not run, yet parity is graded using the hardcoded target/release/code-index environment path. Other release CLI builds target separate directories. Both logs lack the CLI build invocation before these tests. The source explicitly asserts that file exists. Follow this failure back to causes 4/5; it is not independent evidence of PHP parity or CLI indexing failure.

Not verified: no test of CLI behavior occurred in these failures. No change to step guards was tried.

7. Newly reported rmcp advisory fails cargo-deny — size S

Job/night: CI Nov 10 2512, API 8321, index 6 cargo deny. Step Check advisories, bans, licenses, sources; failing command cargo deny check advisories bans licenses sources.

First decisive error: log-8321-6.txt:393-400 (ANSI stripped): error[vulnerability]: Unbounded response body buffering in Streamable HTTP clients, identifies rmcp 3.7.0 and RUSTSEC-2026-0462. All three attempts report advisories FAILED; this is not the generic GitHub advisory-fetch warning printed by the wrapper.

First appearance / range: this advisory first observed Nov 10; Nov 9 and Nov 11 deny jobs green. Last green f8571742109abe565c4560e812bb99220812a91e, first red e3ec6ead8dd3463a7af2636f43d16138ff8d2b0e. git log --oneline f8571742..e3ec6ead -- Cargo.lock Cargo.toml deny.toml .forgejo/workflows/ci.yml is empty: transcript. Thus the advisory database's calendar-time change, not a dependency introduction in that interval, is the supported explanation. Recovery range e3ec6ead..eea2214a contains f7ee339e deps: rmcp 3.9.0 (RUSTSEC-2026-0462) and version commit 81ff0158: log.

Schedule-only: no. deny is a normal push/PR CI job (subject to change scoping); the workflow does not reserve it for schedule.

Issue: no issue among open titles. Already addressed by the dependency update above; no new issue was posted.

Established: dependency/advisory failure in all three attempts, repaired dependency version by next night. cargo test and cargo test (daemon transport) are skipped because test needs deny, and the daemon leg needs test; plugin-path-cost is then skipped because it needs the daemon leg. Those are absent measurements, not additional test failures.

Not verified: exact RustSec publication timestamp, exploitability in this product's enabled transport features, or any runtime security claim. Attribution relies on the advisory printed in CI, not a fresh audit.

8. Shared JS/TS daemon-import test child exits before readiness — size M

Jobs/night: package CI Nov 11 2535, API 8348, indices 3 package (javascript) and 10 package (typescript). Both fail step The package grades its own bytes, phase own_suites, test script_daemon_imports::named_script_imports_follow_configuration_exports_and_daemon_restart. Commands originate in .forgejo/scripts/package_leg.sh <package> release and run the respective daemon-import test binary.

Errors: both panic at crates/first-party-tests/tests/support/../../../daemon/tests/common/mod.rs:555:17:

  • JavaScript, log 8348-3 line 636: the daemon EXITED before it became ready (pid 7943 is a zombie: it exited and was not reaped), after 24.317365912s.
  • TypeScript, log 8348-10 line 636: same message, pid 8115, after 19.552017358s.

First appearance / range: first verified Nov 11, intermittent; both exact tests are ok in Nov 10 logs 8319-3/10 and Nov 12 logs 8513-3/10. Last green e3ec6ead8dd3463a7af2636f43d16138ff8d2b0e, first red eea2214a21a369840f2b510ac5fb538f8eec8f3e. Relevant log, shared-fixture log. Includes c598b555 (isolate seam daemon builds), 6f878569 (Unix detach), d20f4226 (read-pool health), 9bb16ab6 (failed COMMIT cleanup), 37ac4101 (package language activation), and activation changes. The shared test source itself has no change in that interval. No single commit is implicated by this broad interval.

Schedule-only: no. package and package-corpus matrices explicitly permit PR, push, schedule and manual events (package-ci.yml:239,306), with changed-package scoping on PR/push and all packages on schedule. These package own suites can run on ordinary CI.

Issue: no issue for this exact dead-daemon startup signature. #520 concerns multiwriter approval/activation concurrency, but the logs do not prove that mechanism. #581's slow one-shot startup is also a different signature: these processes are dead, not merely late.

Established: both wrappers include the same shared import/restart test. It installs/approves JS and TS, starts a daemon, and waits for readiness; the helper checks the process's death and deliberately distinguishes it from a live timeout. The stack points to script_daemon_imports.rs:93:49, the initial start(&operator, &root) call, before the import assertions or restart. Stdout/stderr are /dev/null, exactly as the panic warns. Thus two failed jobs represent one shared initial-startup signature. Their later red overhead measurement is independently attributed to cause 1, since overhead runs after failed matrix legs by design.

Not verified: exit status/signal and daemon diagnostic output, OOM, package activation failure, concurrent build effects, or another reason for process exit. The source does not justify choosing one.

9. Generic candidate acceptance exhausts plugin-add deadline — size M

Job/night: CI Nov 12 2561, API 8517, index 10 cargo test. The failing step is Check signed generic runtime acceptance, command python3 tests/phase2_acceptance.py --bin-dir target/debug --out candidate-pilot-results/signed-package-acceptance.json (ci.yml:984-986). It is not a failed Rust test: the ordinary nextest profile passes 5571/5571, and timing profile later passes 55/55.

First decisive error: log-8517-10.txt:1808:

subprocess.TimeoutExpired: Command '['/tmp/phase2-acceptance-m4d16tsa/missing-inner-install/code-index', '--root', '/tmp/phase2-acceptance-m4d16tsa/missing-inner-project', 'plugin', 'add', '/tmp/phase2-acceptance-m4d16tsa/typescript.cip', '--grant', 'requested', '--yes']' timed out after 40.8586009349674 seconds

Traceback: main -> check_missing_embedded:178 -> run:240 -> subprocess.run. The command installs the missing TypeScript producer while the Svelte-containing project has a live MCP/daemon.

First appearance / range: first verified Nov 12, one occurrence; previous Nov 11 same candidate-acceptance check passes and emits its missing-producer repair report. Last green eea2214a21a369840f2b510ac5fb538f8eec8f3e, first red a7ce9603e6c70d2af3b551684ac82d58016c5bea. Relevant commit log includes 3755ecd1 (bound grant reconciliation), bcd65745 (startup snapshot safety), 85ef5503 (reader publication), 7c23f9e5 (startup negatives), d01bb339 (stats readers), d4c4fb28 (startup state), and script-import work. This is a large integration interval, not an isolated culprit.

Schedule-only: no. The candidate build and this generic acceptance step run in ordinary cargo test on scoped PR/push runs; only the separate release-shaped checks in cause 5 are in the nightly lane. Green PR/push checks therefore coexist with an intermittent failure in an ordinary step.

Issue: no issue for this exact deadline failure. #520 is a plausible related tracker, because the operation races a live daemon's approval/activation writer; its body proposes a single writer, but does not report this timeout. #581 is related startup-performance context only, not established attribution: it measures shipped release one-shot startup; this run uses a debug packages-only candidate build and a plugin mutation with retry. #563 is a conformance verdict/grant mismatch, a different error.

Established: run(timeout=180, retry_busy=True) shares one deadline over retries; only a nonzero result whose stderr starts error: plugin_mutation_busy: is retried. The 40.8586 seconds in the exception is the remaining subprocess allowance, not the total command budget; about 139 seconds had already been spent in earlier attempts/loop overhead. The helper's source specifically documents concurrent cold-start approval writing here. The final subprocess did not finish within its remaining allowance. cargo test (daemon transport) is skipped because it needs the failed test job; plugin-path-cost is skipped because it needs that daemon leg. These are not new nightly timing failures.

Not verified: captured output from the final timed-out child, how many busy retries occurred, what held the mutation lock, whether the child was doing useful work or deadlocked, whether this is a performance regression, and which commit introduced it. No rerun, lock tracing, or fix was attempted.

#288: current failures versus the historical mixed-load failure

None of the seven-day failed jobs is the failure described in #288. The directly read CI logs show mixed_load_ceilings ... ok on Nov 5, 6, 7, 9, 11. All also print ladder x64 ... host.memory_ceiling_exceeded; this is an observed refusal row and crossover diagnostic, not itself a panic. Nov 8 admission refuses to start the test; Nov 10/12 its enclosing lane is skipped. No result is imputed for those nights.

Current source mixed_load_bench.rs:2580-2613 deliberately finds and prints the largest successful and smallest refused input and excludes refused rows from timing comparisons. worker.rs:552-590 returns MemoryCeiling when the guest limiter's denial flag is set. Therefore seeing the same memory-ceiling text on a green night cannot identify #288 or establish host memory pressure.

Last positively documented occurrence found: 2026-09-21, scheduled UI run 856, API id 5378, commit bea410f11a3e9236190cbd046f382c4ee23caae3, at 05:00:50 CEST / 03:00:50 UTC, corpus-scale index 11. Its test failure is quoted in #288's body; the API confirms schedule, date, SHA and failed corpus-scale job. The issue has no comments adding a later occurrence. UI run 725 is only suspected by the issue, not confirmed.

Limit on the word “last”: the September 21 original log, later September 23 corpus-scale failure (UI 885), October 28 corpus-scale failure (UI 1911), and other older failed-job logs return HTTP 404. The last retained logs found begin Nov 5. Later failed corpus-scale jobs cannot be assigned to #288 from job status alone. Thus Sept 21 is the last verified/documented occurrence in available evidence, not a proven last-ever recurrence. The exact last nightly occurrence cannot be established with the retained logs. This limitation is material; no newer red run was relabelled as #288 merely because of its duration, job name, or refusal text.

Run inventory

Night Workflow UI / API run Status Failed job indices
2026-11-06 package-ci.yml 2252 / 7934 failure 24
2026-11-06 ci.yml 2253 / 7936 failure 14
2026-11-07 package-ci.yml 2321 / 8045 failure 24
2026-11-07 ci.yml 2342 / 8067 success —
2026-11-08 package-ci.yml 2412 / 8186 cancelled —
2026-11-08 ci.yml 2425 / 8214 failure 14
2026-11-09 package-ci.yml 2472 / 8275 failure 24
2026-11-09 ci.yml 2494 / 8298 failure 14
2026-11-10 package-ci.yml 2511 / 8319 failure 24
2026-11-10 ci.yml 2512 / 8321 failure 6
2026-11-11 package-ci.yml 2535 / 8348 failure 3, 10, 24
2026-11-11 ci.yml 2536 / 8350 success —
2026-11-12 package-ci.yml 2560 / 8513 failure 24
2026-11-12 ci.yml 2561 / 8517 failure 10

Workflow coverage and dependency accounting

At the checked-out master, the schedule/manual-only jobs are corpus-scale, plugin-path-cost, grammar-rebuild in ci.yml, and package-overhead in package-ci.yml. The nightly-only wording includes manual dispatch. Tier-one corpus is an ordinary PR/push job. deny, test, test-daemon-leg and the package/package-corpus matrices are also available on PR/push, subject to path/package change scoping. The exact failing step/build shape matters; availability of a different acceptance script does not cover the shipped-shape check.

Nov 10 needs chain: deny failure -> test skipped -> test-daemon-leg skipped -> plugin-path-cost skipped. Nov 12: test failure -> test-daemon-leg skipped -> plugin-path-cost skipped. Nov 8/9 missing release CLI is a within-job step-condition cascade. All remaining jobs in the failed runs succeeded; the only extra failed jobs beyond the usual one per run are the JavaScript/TypeScript package suites on Nov 11.

Evidence limits and size convention

S/M/L are rough investigation/remediation scope guesses, not estimates from an implementation: S = localized CI/dependency/test setup; M = runtime/measurement attribution needing focused evidence; L would require a broader redesign. No cause here has enough evidence to demand an L redesign. “No issue” means no matching open issue after examining all 40 returned titles and the bodies of plausible candidates (#288, #515, #520, #530, #531, #534, #548, #563, #581); it does not claim closed issues were searched exhaustively.

The API history was paginated without the broken event=schedule filter and selected on trigger_event. Older successful job statuses establish only that those job invocations passed, not that today's test population was present. The old PHP/pool introduction brackets are explicitly conservative for that reason. Relevant git log commands and their complete outputs are retained in commits-*.txt; an empty log is recorded as empty rather than filled with an assumed culprit.

Attribution of the scheduled runs 2026-11-06 to 2026-11-12, as written by the attribution worker on 2026-11-12 (read-only; the linked `commits-*.txt` and `log-*.txt` files are local evidence and are not attached). <details><summary>Full attribution</summary> # Nightly failure attribution — 2026-11-06 through 2026-11-12 Read-only investigation of master `3c8742bf5499ee2145e0238517e6c1f6df5a11fb`. No suites run, fixes attempted, repository files changed, or Forgejo writes made. Dates below are the run's `created` date in the API's Europe/Berlin offset; package jobs often execute on the previous UTC date. The requested seven-day window includes November 6, in addition to the six dates supplied in the question. There are **11 failed scheduled runs and 13 failed jobs**, grouped into nine failure signatures below. A shared symptom family is not proof of a shared low-level defect. Cancelled, skipped, and admission-refused measurements are distinguished from a benchmark result. Evidence: `runs-*.json`, `scheduled-all.json`, `jobs-<API run id>.json`, and `log-<API run id>-<zero-based job index>.txt` in this directory. The API run id differs from the number in `html_url`. Each log was obtained through `<html_url>/jobs/<index>/attempt/1/logs`. Source was read with code-index, including `read_code(rev=...)` at historical commits; selected replies are retained in `source-evidence.json`. Code-index requires `read_code.response_format=json` and offers no format argument on `index_coverage`/`project_overview`; those were the only necessary exceptions to `concise`. All calls used `envelope=minimal`. No shell source-reading fallback was used. Absolute-path coverage confirmed the workflow and shared daemon fixture belong to this worktree. ## 1. Package cold/warm overhead budgets — size M **Jobs and nights:** `package-ci.yml`, index 24, **package overhead budgets (nightly, one package at a time)**; step **Both overhead budgets, one package at a time**. It fails on every completed nightly in this window: November 6, 7, 9, 10, 11, 12. November 8 was cancelled and is an absent measurement. Individual packages vary, so this is persistent job redness with intermittent package-specific failures. Every assertion below is test `its_package_matches_native_semantics_and_fits_cold_and_warm_budgets`, at `crates/first-party-tests/tests/support/language_overhead.rs:432`. These are the decisive assertion messages, not merely the final shell aggregate: | Night | UI run / API id | First decisive messages (one per failed package) | |---|---|---| | Nov 6 | [2252](https://git.h-dv.de/h-dv/code-index/actions/runs/2252) / 7934 | `cs-dapper warm ratio 1.334 exceeds 1.15`; `ruby-sinatra warm ratio 1.114 exceeds 1.10` | | Nov 7 | [2321](https://git.h-dv.de/h-dv/code-index/actions/runs/2321) / 8045 | `cs-dapper warm ratio 1.342 exceeds 1.15`; `python-flask warm ratio 1.126 exceeds 1.10`; `ruby-sinatra warm ratio 1.100 exceeds 1.10` (unrounded ratio 1.1003902840663624) | | Nov 9 | [2472](https://git.h-dv.de/h-dv/code-index/actions/runs/2472) / 8275 | `ruby-sinatra warm ratio 1.112 exceeds 1.10` | | Nov 10 | [2511](https://git.h-dv.de/h-dv/code-index/actions/runs/2511) / 8319 | `ruby-sinatra warm ratio 1.137 exceeds 1.10` | | Nov 11 | [2535](https://git.h-dv.de/h-dv/code-index/actions/runs/2535) / 8348 | `ruby-sinatra warm ratio 1.135 exceeds 1.10`; `ts-zod warm ratio 1.101 exceeds 1.10` | | Nov 12 | [2560](https://git.h-dv.de/h-dv/code-index/actions/runs/2560) / 8513 | `js-express cold: package 1.431 s exceeds the limit 1.430 s (= 1.20 x builtin 0.775 s + allowance 0.50 s); ratio 1.846`; `python-flask cold: package 1.550 s exceeds the limit 1.501 s (= 1.20 x builtin 0.834 s + allowance 0.50 s); ratio 1.857`; `ruby-sinatra cold: package 1.633 s exceeds the limit 1.608 s (= 1.20 x builtin 0.923 s + allowance 0.50 s); ratio 1.768` | **First appearance / range:** the current streak starts November 6. The preceding November 5 overhead job, API 7849 / UI 2199, is green at `fb4eb790b25a3cb0411bb4b782346feebe2940cb`; first red is `0a6599d6b0fcb178e2c6b9e18620e71c848d3878`. Earlier October runs also had red overhead jobs, but their logs are unavailable, so this is not a claim of first-ever occurrence. `git log --oneline fb4eb790..0a6599d6 -- <measurement/package paths>` is empty for the helper, guest code, package manifests and workflow ([command and output](commits-overhead.txt)). Expanding to all first-party tests and indexer runtime finds `f54a8758` (cold constant lookup), `11bf0021` (mixed-language invalidation/scoped constants), `50726eb4` (Svelte tests), and `56dbb24a` (basename fact ABI conformance): [expanded log](commits-overhead-expanded.txt). These are candidates, not a bisection. Package-specific transitions: C#/Ruby first red Nov 6 after Nov 5 green; Python first red Nov 7 after Nov 6 green (`0a6599d6..b2405a4e`); TypeScript first red Nov 11 after Nov 10 green (`e3ec6ead..eea2214a`); JavaScript and the cold failures first red Nov 12 after Nov 11 green (`eea2214a..a7ce9603`). Other reds in those workflows do not invalidate their successful measurements. **Schedule-only:** yes, plus manual dispatch. `.forgejo/workflows/package-ci.yml:400-407` gates `package-overhead` to schedule/dispatch; PR and push runs do not assert these timings. Its `needs: [package, package-corpus]` and `success() || failure()` deliberately permit measurements after a failed package matrix leg; Nov 11 overhead is an independent failure, not a `needs` cascade. Package suites on PR/push grade semantic parity separately. **Issue:** **no issue** matching these budget failures among the 40 open issue titles. #581 concerns a different, shipped-process startup slowdown, not this paired in-process index measurement; its body does not establish the same cause. #534's stale-pin problem and #515's duplicate extractors are related measurement concerns, not this assertion signature. **Established:** the helper takes medians of paired native/package rounds, alternates execution order, and separately asserts semantics and budgets (`language_overhead.rs:367-440`). Cold bound is `1.20 * native + allowance`; warm bounds are 1.15 for C#, 1.20 for PHP, 1.10 otherwise (`:25-103`). The logged failed rows have `semantic_failure:null`; these are timing failures. Some misses are extremely small: Nov 12 JavaScript is 0.000751 s over its cold limit. Load readings are present, e.g. Nov 7 C# ended at `3.62 7.98 33.63`, and Nov 9 Ruby at `1.15 7.03 27.15`. The measurements are serial within this job, but cannot exclude other workflows or host activity. **Not verified:** which costs account for the extra time, whether the small misses are noise or a runtime regression, whether all affected packages share a mechanism, and the first-ever failing night. Load alone proves neither contention nor causation. No profiling or rerun was performed. ## 2. PHP native/package diagnostic compared different host semantics — size S **Jobs and nights:** `ci.yml`, index 14, **Plugin path cost + pool throughput (nightly)**, step **Seven-language parity, no-package CLI against builtins and package overhead**. November 6 [2253](https://git.h-dv.de/h-dv/code-index/actions/runs/2253), API 7936; also directly verified November 5 [2200](https://git.h-dv.de/h-dv/code-index/actions/runs/2200), API 7854, just outside the window. **Error:** `thread 'pinned_php_binding_diagnostic' ... panicked at crates/indexer/tests/language_package_overhead.rs:274:5:` followed by `Err("references and targets: 38733 vs 38733 rows; first 69: ...")`. The first difference is `src/Auth/DigestAuth.php`, `RuntimeException`, type reference at 296:28, flags `65536`, differing in the following `0`/`1` field. Subsequent `Utils` rows bind to `src/Utils.php#Utils@18` on one leg and are `Psr7`/unresolved on the other. Full literal message: `log-7936-14.txt:1705`. **First appearance / range:** earliest directly verified November 5 (`fb4eb790b25a3cb0411bb4b782346feebe2940cb`); still red Nov 6, green Nov 7. Last preceding *whole job* green was October 25, API 6671, `e47526105f9c2096db2ac7248dc896f892446376`. Thus conservative observed bracket `e4752610..fb4eb790`, not an exact first-red bracket: October 26's lane was red and its log returns 404; October 27–Nov 4 were blocked/cancelled. [Relevant path log](commits-php-and-pool.txt) includes `0aea4f69` (PHP language-rule resolution), `716be0f5` (type operand normalization), `b6715cc0` (package facts), and `97a1bef5` (plugin proofs). No culprit inferred from their titles. **Schedule-only:** yes; the enclosing `plugin-path-cost` job runs on schedule/manual dispatch only (`ci.yml:1691`). The diagnostic itself was ignored in ordinary `cargo test`. **Issue:** **no issue** for this exact parity diagnostic. #531's PHP wrong-bind report, #515's two extractor implementations, and #534's corpus population issue are related but describe different failures. **Established:** historical source at `0a6599d6` shows an ignored persistent diagnostic comparing native and package databases with exact parity. By `b2405a4e`, commit `33d9c8ce` replaced it with `pinned_php_package_cli_matches_in_process_index`, explicitly discovers the host with the operator-vouched store, asserts packages are not `builtin_equivalent`, and compares matching CLI/in-process package contexts. This explains a real test-setup mismatch, and the Nov 7 log passes the replacement. [Recovery path log](commits-php-pool-recovery.txt). **Not verified:** a source-level proof that every old row delta is explained by publisher classification, or that no genuine PHP extractor difference existed. Exact first failing night is unavailable. The replacement's later Nov 8/9 NotFound failures have a separate cause below. ## 3. Worker-pool speedup below its two-sweep floor — size M **Jobs and nights:** same CI job index 14; step **Worker-pool throughput against one lane**. Nov 6 API 7936 / UI 2253; also Nov 5 API 7854 / UI 2200. **Error:** `thread 'throughput_before_and_after' ... panicked at crates/indexer/tests/package_pool.rs:843:9:` then **`8 lanes gave 1.93x over one lane, under the 4.00x floor ... ON TWO INDEPENDENT SWEEPS`** (Nov 6). Sweeps were 1.93x and 1.87x; it grades the better. Nov 5 was 1.98x. Nov 6 end load was `3.54 2.95 3.33`. **First appearance / range:** earliest directly verified Nov 5, repeats Nov 6; Nov 7, 8, 9, 11 measurements pass. Nov 10/12 lane skipped, not a pass. Same conservative October 25 green `e47526105f9c2096db2ac7248dc896f892446376` to Nov 5 first verified red `fb4eb790b25a3cb0411bb4b782346feebe2940cb` bracket as cause 2, with censored intermediate runs. [Path log](commits-php-and-pool.txt) and [recovery log](commits-php-pool-recovery.txt). The `package_pool.rs`-only log in that introduction bracket is empty: the assertion need not have changed for its measurement to fail. **Schedule-only:** yes, plus dispatch; ignored test under `plugin-path-cost`, absent from normal PR/push execution. **Issue:** **no issue** matching this two-sweep throughput failure. #288 is a different test and different failure mode. **Established:** the source computes one-lane/wide-lane elapsed ratios, remeasures if below `floor_for`, and asserts the better of two against half-linear speedup capped at 4 (`package_pool.rs:752-856`). The runner reports eight cores, at the strictest point of that floor. In `33d9c8ce`, the workflow adds an explicit test prebuild, fresh release worker build, idle admission, `COSI_PLUGIN_HOST_PREBUILT=1`, and unsets `RAYON_NUM_THREADS`; Nov 7 passes. Historical workflow reads before/after are retained in `source-evidence.json`. **Not verified:** contention, CPU quota/topology, a worker/build-shape mismatch, and a real throughput regression remain competing explanations. The change and recovery are correlated, not a measured isolation of which adjustment mattered. First-ever failing night cannot be recovered from expired logs. ## 4. Runner capacity admission timed out before mixed-load test — size S **Job/night:** `ci.yml`, Nov 8 [2425](https://git.h-dv.de/h-dv/code-index/actions/runs/2425), API 8214, index 14; step **Plugin-host mixed-load ceilings (#79 acceptance 6)**. **First decisive error:** `log-8214-14.txt:1062`: `{"benchmark_capacity": "timeout", "last_sample": {"load_1m": 8.1240234375, "visible_cpus": 8, "load_limit": 2.0}, "reason": "runner did not become quiet; measurement not started"}`. `python3 .forgejo/scripts/require_idle_runner.py` exits 1. Next step **Mixed-load metrics** also exits 1: `cat: target/corpus/plugin-host-mixed-load.json: No such file or directory`. That missing report is a downstream effect, not a second mixed-load test failure. **First appearance / range:** Nov 8, intermittent (Nov 7 and 9 admission succeeds). Last green Nov 7 `b2405a4e9c715c2eea77e55c6e748e009d69f20c`, first red Nov 8 `e7510767f7cd33a9d741a16673fb68e6742e4cfd`. `git log --oneline b2405a4e..e7510767 -- .forgejo/scripts/require_idle_runner.py .forgejo/workflows/ci.yml` is empty: [transcript](commits-idle.txt). This is an environment/precondition transition, not evidence of a source regression in that range. **Schedule-only:** yes, plus dispatch, inside `plugin-path-cost`. PR/push does not invoke this precondition/test step. **Issue:** **no issue** matching this capacity timeout. It is **not #288**: no `mixed_load_ceilings` sample ran. **Established:** the historical gate requires load <= 0.25 per visible CPU continuously for 10 seconds within 90 minutes; quota can reduce the CPU count. Logs show it waited and never obtained admission. A cancelled package-overhead job overlapped this interval and logged its own long waits; cancellation remains absent measurement. **Not verified:** which process/workflow produced the load, whether the load was CPU or I/O pressure, or whether a different runner would admit it. No benchmark verdict can be inferred from this refusal. ## 5. Shipped-shape Vue script symbol absent after package installation — size M **Jobs/nights:** `ci.yml` index 14, step **Shipped shape through CLI, MCP, bundle and changed_symbols**; Nov 8 API 8214 / UI 2425 and Nov 9 [2494](https://git.h-dv.de/h-dv/code-index/actions/runs/2494), API 8298. Command: `python3 tests/shipped_shape_acceptance.py --bin-dir target/shipped-shape/release --out target/corpus/shipped-shape-acceptance.json`. **Error:** `AssertionError: package/watch did not produce Comp.vue:callee_vue: {'error': 'symbol_not_found', ...}` (`log-8214-14.txt:1470`, `log-8298-14.txt:1645`). Traceback originates in `shipped_shape_acceptance.py:330 -> check_queries:172 -> phase2_acceptance.py:319`. This is the **shipped-shape** script importing a helper, not the earlier generic package-only script. **First appearance / range:** first Nov 8 after Nov 7 green: `b2405a4e9c715c2eea77e55c6e748e009d69f20c..e7510767f7cd33a9d741a16673fb68e6742e4cfd`. Repeats Nov 9, passes Nov 11; Nov 10/12 enclosing job skipped. [Introduction path log](commits-vue-and-cli.txt) includes `0fb55728` (deactivate excluded languages), `ed1d0146` (builtin ownership), and package approval/recovery changes. The fixture file's own log in the introduction interval is empty. [Recovery log](commits-vue-recovery-runtime.txt) includes `37ac4101` (shipped package-language activation), `4a971b16` (delegated script languages), `072e4125` (activation census). **Schedule-only:** this precise shipped-shape check is inside the nightly/dispatch job. PR/push does run generic candidate runtime acceptance in `cargo test` (`ci.yml:986`), but that is a different build shape and script; it does not execute this shipped-default check. **Issue:** **no issue** matching this exact missing-symbol nightly. #548 describes template/import and framework recall gaps, rather than a missing declaration inside a script after language/package activation. #515 discusses build-shape coverage; neither proves this cause. Not #288. **Established:** the fixture enables the seven ordinary languages, then adds Vue separately; it expects `callee_vue` from the `.vue` TypeScript region. The log's final search says no such symbol and freshness `metadata_agrees`, `incomplete:false`. Recovery commit `37ac4101` explicitly brings newly added package languages into activation preview and recorded language decisions and adds a packages-only Vue regression test. The Nov 11 nightly passes. This strongly implicates package language activation/delegation, rather than a mere unresolved caller. **Not verified:** which introduction commit caused it; whether the missing row on each failed runner was specifically a language-decision exclusion, embedded producer refusal, or another activation path. The search answer alone does not expose `Comp.vue`'s contribution/refusal rows. No bisect or reproduction was run. ## 6. Later CLI parity tests run after the CLI build is skipped — size S **Jobs/nights:** CI index 14, Nov 8 and 9 (same runs as cause 5), step **Seven-language parity, no-package CLI against builtins and package overhead**. **First errors:** `no_package_cli_index_matches_the_in_process_builtins` panics at `language_package_overhead.rs:99:5`: **`no CLI at /workspace/h-dv/code-index/target/release/code-index`**. `pinned_php_package_cli_matches_in_process_index` then panics at `:293:10`: **`called Result::unwrap() on an Err value: Os { code: 2, kind: NotFound, message: "No such file or directory" }`**. See Nov 8 log lines 1567–1574; Nov 9 lines 1742–1749. **First appearance / range:** first Nov 8, repeats Nov 9, passes Nov 7 and Nov 11. Same last green `b2405a4e9c715c2eea77e55c6e748e009d69f20c` to first red `e7510767f7cd33a9d741a16673fb68e6742e4cfd` bracket and [path log](commits-vue-and-cli.txt). No workflow change appears in this interval. This is a newly exposed conditional-step cascade, not demonstrated CLI source breakage. **Schedule-only:** yes, plus dispatch. These ignored parity invocations live in `plugin-path-cost`. **Issue:** **no issue** for the missing-build dependency/step guard mismatch. **Established:** **Build this tree's release CLI** has the default success-only condition, while its consumer has `if: success() || failure()`. On Nov 8 earlier admission and shipped-shape steps fail; on Nov 9 shipped-shape fails. Consequently the CLI build does not run, yet parity is graded using the hardcoded `target/release/code-index` environment path. Other release CLI builds target separate directories. Both logs lack the CLI build invocation before these tests. The source explicitly asserts that file exists. Follow this failure back to causes 4/5; it is not independent evidence of PHP parity or CLI indexing failure. **Not verified:** no test of CLI behavior occurred in these failures. No change to step guards was tried. ## 7. Newly reported rmcp advisory fails cargo-deny — size S **Job/night:** CI Nov 10 [2512](https://git.h-dv.de/h-dv/code-index/actions/runs/2512), API 8321, index 6 **cargo deny**. Step **Check advisories, bans, licenses, sources**; failing command `cargo deny check advisories bans licenses sources`. **First decisive error:** `log-8321-6.txt:393-400` (ANSI stripped): **`error[vulnerability]: Unbounded response body buffering in Streamable HTTP clients`**, identifies **`rmcp 3.7.0`** and **`RUSTSEC-2026-0462`**. All three attempts report `advisories FAILED`; this is not the generic GitHub advisory-fetch warning printed by the wrapper. **First appearance / range:** this advisory first observed Nov 10; Nov 9 and Nov 11 deny jobs green. Last green `f8571742109abe565c4560e812bb99220812a91e`, first red `e3ec6ead8dd3463a7af2636f43d16138ff8d2b0e`. `git log --oneline f8571742..e3ec6ead -- Cargo.lock Cargo.toml deny.toml .forgejo/workflows/ci.yml` is empty: [transcript](commits-deny.txt). Thus the advisory database's calendar-time change, not a dependency introduction in that interval, is the supported explanation. Recovery range `e3ec6ead..eea2214a` contains **`f7ee339e deps: rmcp 3.9.0 (RUSTSEC-2026-0462)`** and version commit `81ff0158`: [log](commits-deny-recovery.txt). **Schedule-only:** **no**. `deny` is a normal push/PR CI job (subject to change scoping); the workflow does not reserve it for schedule. **Issue:** **no issue** among open titles. Already addressed by the dependency update above; no new issue was posted. **Established:** dependency/advisory failure in all three attempts, repaired dependency version by next night. `cargo test` and `cargo test (daemon transport)` are **skipped** because `test` needs `deny`, and the daemon leg needs `test`; `plugin-path-cost` is then **skipped** because it needs the daemon leg. Those are absent measurements, not additional test failures. **Not verified:** exact RustSec publication timestamp, exploitability in this product's enabled transport features, or any runtime security claim. Attribution relies on the advisory printed in CI, not a fresh audit. ## 8. Shared JS/TS daemon-import test child exits before readiness — size M **Jobs/night:** package CI Nov 11 [2535](https://git.h-dv.de/h-dv/code-index/actions/runs/2535), API 8348, indices **3 package (javascript)** and **10 package (typescript)**. Both fail step **The package grades its own bytes**, phase **own_suites**, test **`script_daemon_imports::named_script_imports_follow_configuration_exports_and_daemon_restart`**. Commands originate in `.forgejo/scripts/package_leg.sh <package> release` and run the respective daemon-import test binary. **Errors:** both panic at `crates/first-party-tests/tests/support/../../../daemon/tests/common/mod.rs:555:17`: - JavaScript, log 8348-3 line 636: **`the daemon EXITED before it became ready (pid 7943 is a zombie: it exited and was not reaped), after 24.317365912s.`** - TypeScript, log 8348-10 line 636: same message, **pid 8115**, after **19.552017358s**. **First appearance / range:** first verified Nov 11, intermittent; both exact tests are `ok` in Nov 10 logs 8319-3/10 and Nov 12 logs 8513-3/10. Last green `e3ec6ead8dd3463a7af2636f43d16138ff8d2b0e`, first red `eea2214a21a369840f2b510ac5fb538f8eec8f3e`. [Relevant log](commits-daemon.txt), [shared-fixture log](commits-daemon-shared-fixture.txt). Includes `c598b555` (isolate seam daemon builds), `6f878569` (Unix detach), `d20f4226` (read-pool health), `9bb16ab6` (failed COMMIT cleanup), `37ac4101` (package language activation), and activation changes. The shared test source itself has no change in that interval. No single commit is implicated by this broad interval. **Schedule-only:** **no**. `package` and `package-corpus` matrices explicitly permit PR, push, schedule and manual events (`package-ci.yml:239,306`), with changed-package scoping on PR/push and all packages on schedule. These package own suites can run on ordinary CI. **Issue:** **no issue** for this exact dead-daemon startup signature. #520 concerns multiwriter approval/activation concurrency, but the logs do not prove that mechanism. #581's slow one-shot startup is also a different signature: these processes are dead, not merely late. **Established:** both wrappers include the same shared import/restart test. It installs/approves JS and TS, starts a daemon, and waits for readiness; the helper checks the process's death and deliberately distinguishes it from a live timeout. The stack points to `script_daemon_imports.rs:93:49`, the initial `start(&operator, &root)` call, before the import assertions or restart. Stdout/stderr are `/dev/null`, exactly as the panic warns. Thus two failed jobs represent one shared initial-startup signature. Their later red overhead measurement is independently attributed to cause 1, since overhead runs after failed matrix legs by design. **Not verified:** exit status/signal and daemon diagnostic output, OOM, package activation failure, concurrent build effects, or another reason for process exit. The source does not justify choosing one. ## 9. Generic candidate acceptance exhausts plugin-add deadline — size M **Job/night:** CI Nov 12 [2561](https://git.h-dv.de/h-dv/code-index/actions/runs/2561), API 8517, index 10 **cargo test**. The failing step is **Check signed generic runtime acceptance**, command `python3 tests/phase2_acceptance.py --bin-dir target/debug --out candidate-pilot-results/signed-package-acceptance.json` (`ci.yml:984-986`). It is **not a failed Rust test**: the ordinary nextest profile passes **5571/5571**, and timing profile later passes **55/55**. **First decisive error:** `log-8517-10.txt:1808`: `subprocess.TimeoutExpired: Command '['/tmp/phase2-acceptance-m4d16tsa/missing-inner-install/code-index', '--root', '/tmp/phase2-acceptance-m4d16tsa/missing-inner-project', 'plugin', 'add', '/tmp/phase2-acceptance-m4d16tsa/typescript.cip', '--grant', 'requested', '--yes']' timed out after 40.8586009349674 seconds` Traceback: `main -> check_missing_embedded:178 -> run:240 -> subprocess.run`. The command installs the missing TypeScript producer while the Svelte-containing project has a live MCP/daemon. **First appearance / range:** first verified Nov 12, one occurrence; previous Nov 11 same candidate-acceptance check passes and emits its missing-producer repair report. Last green `eea2214a21a369840f2b510ac5fb538f8eec8f3e`, first red `a7ce9603e6c70d2af3b551684ac82d58016c5bea`. [Relevant commit log](commits-acceptance-timeout.txt) includes `3755ecd1` (bound grant reconciliation), `bcd65745` (startup snapshot safety), `85ef5503` (reader publication), `7c23f9e5` (startup negatives), `d01bb339` (stats readers), `d4c4fb28` (startup state), and script-import work. This is a large integration interval, not an isolated culprit. **Schedule-only:** **no**. The candidate build and this generic acceptance step run in ordinary `cargo test` on scoped PR/push runs; only the separate release-shaped checks in cause 5 are in the nightly lane. Green PR/push checks therefore coexist with an intermittent failure in an ordinary step. **Issue:** **no issue** for this exact deadline failure. **#520 is a plausible related tracker**, because the operation races a live daemon's approval/activation writer; its body proposes a single writer, but does not report this timeout. **#581 is related startup-performance context only**, not established attribution: it measures shipped release one-shot startup; this run uses a debug packages-only candidate build and a plugin mutation with retry. #563 is a conformance verdict/grant mismatch, a different error. **Established:** `run(timeout=180, retry_busy=True)` shares one deadline over retries; only a nonzero result whose stderr starts `error: plugin_mutation_busy:` is retried. The 40.8586 seconds in the exception is the remaining subprocess allowance, **not the total command budget**; about 139 seconds had already been spent in earlier attempts/loop overhead. The helper's source specifically documents concurrent cold-start approval writing here. The final subprocess did not finish within its remaining allowance. `cargo test (daemon transport)` is skipped because it needs the failed test job; `plugin-path-cost` is skipped because it needs that daemon leg. These are not new nightly timing failures. **Not verified:** captured output from the final timed-out child, how many busy retries occurred, what held the mutation lock, whether the child was doing useful work or deadlocked, whether this is a performance regression, and which commit introduced it. No rerun, lock tracing, or fix was attempted. ## #288: current failures versus the historical mixed-load failure **None of the seven-day failed jobs is the failure described in [#288](https://git.h-dv.de/h-dv/code-index/issues/288).** The directly read CI logs show `mixed_load_ceilings ... ok` on Nov 5, 6, 7, 9, 11. All also print `ladder x64 ... host.memory_ceiling_exceeded`; this is an observed refusal row and crossover diagnostic, not itself a panic. Nov 8 admission refuses to start the test; Nov 10/12 its enclosing lane is skipped. No result is imputed for those nights. Current source `mixed_load_bench.rs:2580-2613` deliberately finds and prints the largest successful and smallest refused input and excludes refused rows from timing comparisons. `worker.rs:552-590` returns `MemoryCeiling` when the guest limiter's denial flag is set. Therefore seeing the same memory-ceiling text on a green night cannot identify #288 or establish host memory pressure. **Last positively documented occurrence found:** **2026-09-21, scheduled UI run [856](https://git.h-dv.de/h-dv/code-index/actions/runs/856), API id 5378, commit `bea410f11a3e9236190cbd046f382c4ee23caae3`**, at 05:00:50 CEST / 03:00:50 UTC, `corpus-scale` index 11. Its test failure is quoted in #288's body; the API confirms schedule, date, SHA and failed corpus-scale job. The issue has no comments adding a later occurrence. UI run 725 is only suspected by the issue, not confirmed. **Limit on the word “last”:** the September 21 original log, later September 23 corpus-scale failure (UI 885), October 28 corpus-scale failure (UI 1911), and other older failed-job logs return HTTP 404. The last retained logs found begin Nov 5. Later failed corpus-scale jobs cannot be assigned to #288 from job status alone. Thus **Sept 21 is the last verified/documented occurrence in available evidence, not a proven last-ever recurrence**. The exact last nightly occurrence cannot be established with the retained logs. This limitation is material; no newer red run was relabelled as #288 merely because of its duration, job name, or refusal text. ## Run inventory | Night | Workflow | UI / API run | Status | Failed job indices | |---|---|---|---|---| | 2026-11-06 | package-ci.yml | [2252](https://git.h-dv.de/h-dv/code-index/actions/runs/2252) / 7934 | failure | 24 | | 2026-11-06 | ci.yml | [2253](https://git.h-dv.de/h-dv/code-index/actions/runs/2253) / 7936 | failure | 14 | | 2026-11-07 | package-ci.yml | [2321](https://git.h-dv.de/h-dv/code-index/actions/runs/2321) / 8045 | failure | 24 | | 2026-11-07 | ci.yml | [2342](https://git.h-dv.de/h-dv/code-index/actions/runs/2342) / 8067 | success | — | | 2026-11-08 | package-ci.yml | [2412](https://git.h-dv.de/h-dv/code-index/actions/runs/2412) / 8186 | cancelled | — | | 2026-11-08 | ci.yml | [2425](https://git.h-dv.de/h-dv/code-index/actions/runs/2425) / 8214 | failure | 14 | | 2026-11-09 | package-ci.yml | [2472](https://git.h-dv.de/h-dv/code-index/actions/runs/2472) / 8275 | failure | 24 | | 2026-11-09 | ci.yml | [2494](https://git.h-dv.de/h-dv/code-index/actions/runs/2494) / 8298 | failure | 14 | | 2026-11-10 | package-ci.yml | [2511](https://git.h-dv.de/h-dv/code-index/actions/runs/2511) / 8319 | failure | 24 | | 2026-11-10 | ci.yml | [2512](https://git.h-dv.de/h-dv/code-index/actions/runs/2512) / 8321 | failure | 6 | | 2026-11-11 | package-ci.yml | [2535](https://git.h-dv.de/h-dv/code-index/actions/runs/2535) / 8348 | failure | 3, 10, 24 | | 2026-11-11 | ci.yml | [2536](https://git.h-dv.de/h-dv/code-index/actions/runs/2536) / 8350 | success | — | | 2026-11-12 | package-ci.yml | [2560](https://git.h-dv.de/h-dv/code-index/actions/runs/2560) / 8513 | failure | 24 | | 2026-11-12 | ci.yml | [2561](https://git.h-dv.de/h-dv/code-index/actions/runs/2561) / 8517 | failure | 10 | ## Workflow coverage and dependency accounting At the checked-out master, the schedule/manual-only jobs are **`corpus-scale`, `plugin-path-cost`, `grammar-rebuild`** in `ci.yml`, and **`package-overhead`** in `package-ci.yml`. The nightly-only wording includes manual dispatch. Tier-one `corpus` is an ordinary PR/push job. `deny`, `test`, `test-daemon-leg` and the package/package-corpus matrices are also available on PR/push, subject to path/package change scoping. The exact failing step/build shape matters; availability of a different acceptance script does not cover the shipped-shape check. Nov 10 needs chain: `deny failure -> test skipped -> test-daemon-leg skipped -> plugin-path-cost skipped`. Nov 12: `test failure -> test-daemon-leg skipped -> plugin-path-cost skipped`. Nov 8/9 missing release CLI is a within-job step-condition cascade. All remaining jobs in the failed runs succeeded; the only extra failed jobs beyond the usual one per run are the JavaScript/TypeScript package suites on Nov 11. ## Evidence limits and size convention S/M/L are rough investigation/remediation scope guesses, not estimates from an implementation: S = localized CI/dependency/test setup; M = runtime/measurement attribution needing focused evidence; L would require a broader redesign. No cause here has enough evidence to demand an L redesign. “No issue” means no matching open issue after examining all 40 returned titles and the bodies of plausible candidates (#288, #515, #520, #530, #531, #534, #548, #563, #581); it does not claim closed issues were searched exhaustively. The API history was paginated without the broken `event=schedule` filter and selected on `trigger_event`. Older successful job statuses establish only that those job invocations passed, not that today's test population was present. The old PHP/pool introduction brackets are explicitly conservative for that reason. Relevant `git log` commands and their complete outputs are retained in `commits-*.txt`; an empty log is recorded as empty rather than filled with an assumed culprit. </details>
buildagent commented 2026-11-13 01:06:00 +01:00
Author
Member
Copy link

Night of 2026-11-13: a new cause (runner DNS), not a product failure

Scheduled package-ci.yml run 8599 (master 3c8742bf) failed in 18 jobs: package suites (clippy), all eleven package (<language>) jobs and six package-corpus jobs.

Cause 10, new: name resolution failed on the runners. Every failed job first fails in the step "Install the guest target" (rustup target add wasm32-unknown-unknown) with component download failed for rust-std-wasm32-unknown-unknown … dns error: failed to lookup address information: Temporary failure in name resolution (23:01–23:02 UTC). The Python leg additionally could not download a crate (Could not resolve host: static.crates.io).

  • Everything red after that step is a consequence: COSI_GUEST_REBUILD=1 was set but the wasm32-unknown-unknown target is not installed, and can't find crate for core in the Svelte, Timeline and Vue guest clippy.
  • No test assertion about the product failed. The six corpus jobs report passing suites and are red only because of the install step.
  • Not seen in the two preceding scheduled runs (8513, 8348).
  • The overhead budget job of this run was still running when the logs were read, so cause 1 has no verdict for this night.

What this adds to the ticket: the scheduled package jobs download a toolchain component and crates at run time, so one resolver outage on the runner reddens 18 jobs. Options: retry the download step, or keep the wasm target and the crates in the runner image or cache. Whether the outage was the runner's resolver or the upstream is not established.

## Night of 2026-11-13: a new cause (runner DNS), not a product failure Scheduled `package-ci.yml` run 8599 (master `3c8742bf`) failed in 18 jobs: `package suites (clippy)`, all eleven `package (<language>)` jobs and six `package-corpus` jobs. **Cause 10, new: name resolution failed on the runners.** Every failed job first fails in the step "Install the guest target" (`rustup target add wasm32-unknown-unknown`) with `component download failed for rust-std-wasm32-unknown-unknown` … `dns error: failed to lookup address information: Temporary failure in name resolution` (23:01–23:02 UTC). The Python leg additionally could not download a crate (`Could not resolve host: static.crates.io`). - Everything red after that step is a consequence: `COSI_GUEST_REBUILD=1 was set but the wasm32-unknown-unknown target is not installed`, and `can't find crate for core` in the Svelte, Timeline and Vue guest clippy. - No test assertion about the product failed. The six corpus jobs report passing suites and are red only because of the install step. - Not seen in the two preceding scheduled runs (8513, 8348). - The overhead budget job of this run was still running when the logs were read, so cause 1 has no verdict for this night. **What this adds to the ticket:** the scheduled package jobs download a toolchain component and crates at run time, so one resolver outage on the runner reddens 18 jobs. Options: retry the download step, or keep the wasm target and the crates in the runner image or cache. Whether the outage was the runner's resolver or the upstream is not established.
buildagent commented 2026-11-13 06:16:41 +01:00
Author
Member
Copy link

Cause 3 reproduced locally under load, and the outage of this night (2026-11-13)

Cause 3 (plugin_mutation_busy) is not specific to the runner. In a local run of cargo test -p code-index-indexer on a machine that was building in two other worktrees at the same time, a_withdrawn_or_explicitly_owned_language_is_not_adopted (crates/indexer/tests/bundled_adopt.rs:320) failed once with NotAdopted { lang: "ruby", reason: "plugin_mutation_busy", witness: "another plugin writer is changing this project; …" }. The same test passed in the targeted rerun and in the full rerun on the same tree. The test has one project and one writer of its own, so under load the adoption meets a writer it does not expect or gives up waiting too early; that is the same refusal code as in the scheduled failure in tests/phase2_acceptance.py on 11-12 and belongs to #520 (one writer for activation state).

In the same contended run a_thousand_overriding_users_cost_one_decision_not_one_per_site (crates/indexer/tests/member_lookup.rs:2312) failed its wall-clock bound with 72.9 s. That is a timing bound measured under contention, not a regression report.

Cause 10 (runner DNS) lasted the whole night. Between 23:01 UTC and at least 05:12 UTC the runners could not resolve static.rust-lang.org, index.crates.io, static.crates.io and finally code.forgejo.org (the checkout action). Nine runs of ci.yml on the release pull request #583 (one from the pull request, eight dispatched) failed for that reason and none of them reached the cargo test jobs; cargo test is skipped whenever cargo deny fails, and cargo deny needs the crates index. Two properties of the workflow made one resolver outage block a release: the test jobs depend on a job that needs the network, and nothing distinguishes "could not fetch" from "found an advisory" in the job's result.

## Cause 3 reproduced locally under load, and the outage of this night (2026-11-13) **Cause 3 (`plugin_mutation_busy`) is not specific to the runner.** In a local run of `cargo test -p code-index-indexer` on a machine that was building in two other worktrees at the same time, `a_withdrawn_or_explicitly_owned_language_is_not_adopted` (crates/indexer/tests/bundled_adopt.rs:320) failed once with `NotAdopted { lang: "ruby", reason: "plugin_mutation_busy", witness: "another plugin writer is changing this project; …" }`. The same test passed in the targeted rerun and in the full rerun on the same tree. The test has one project and one writer of its own, so under load the adoption meets a writer it does not expect or gives up waiting too early; that is the same refusal code as in the scheduled failure in `tests/phase2_acceptance.py` on 11-12 and belongs to #520 (one writer for activation state). In the same contended run `a_thousand_overriding_users_cost_one_decision_not_one_per_site` (crates/indexer/tests/member_lookup.rs:2312) failed its wall-clock bound with 72.9 s. That is a timing bound measured under contention, not a regression report. **Cause 10 (runner DNS) lasted the whole night.** Between 23:01 UTC and at least 05:12 UTC the runners could not resolve `static.rust-lang.org`, `index.crates.io`, `static.crates.io` and finally `code.forgejo.org` (the checkout action). Nine runs of `ci.yml` on the release pull request #583 (one from the pull request, eight dispatched) failed for that reason and none of them reached the `cargo test` jobs; `cargo test` is skipped whenever `cargo deny` fails, and `cargo deny` needs the crates index. Two properties of the workflow made one resolver outage block a release: the test jobs depend on a job that needs the network, and nothing distinguishes "could not fetch" from "found an advisory" in the job's result.
buildagent commented 2026-11-13 08:43:59 +01:00
Author
Member
Copy link

Workflow changes on the train branch after v0.41.52 (2026-11-13), and what they still owe

Merged into integration-1114; none of it is in v0.41.52. Checked locally (48 CI script unit tests, the registry gates, a YAML parse of all six workflows, targeted Rust tests with mutations). A dispatched run has to prove each item, because conditions and step outcomes are evaluated by the runner.

Cause Change Still to prove by a run
tests skipped when cargo deny cannot fetch (night of 11-13) cargo test no longer needs deny; deny stays a required job and require_ci_green.sh still refuses a release whose deny job failed scheduling of the test jobs when deny is red
deny: fetch failure against finding the job stays red in both cases and its summary quotes the tool's own lines as "policy findings", "could not fetch" or "unclassified" the wording against real output
10, downloads one helper retries rustup target add, rustup toolchain install, cargo install and the rustup bootstrap: three attempts of at most 600 s. It cannot tell a transient from a permanent error and retries both; the last status is kept. Linux jobs only: no step on the Windows runner uses Python today, so its bootstrap is unchanged a real outage
4, step guards a consumer step requires the OUTCOME of the step that builds what it reads (CLI and worker for parity, worker for fleet cost, runtimes for both acceptance runs, the scale build for ceilings/ratchet/2M overview, the guest target for the package suites) that no producer is skipped for another reason, which would skip its consumer silently
5, refused measurement the step stays red with the line NOT MEASURED: runner capacity admission refused; the step that reads the report is skipped instead of failing a second time —
2, daemon exited before ready the shared test helper keeps the daemon's stderr in a file and prints its last 8 KiB and the exit status or signal in the panic the next occurrence
6 (#288), ladder input pinned to rust-analyzer/crates/project-model/test_data/ripgrep-metadata.json; path and size are printed; a missing file is a clear failure timing against the ceilings
3, plugin_mutation_busy attributed and fixed, see #520 the scheduled acceptance run

Found on the way and not changed: nine other e2e helpers discard a child's stderr the same way (spawn_authed, the fleet spawn_daemon, spawn_initialized_snapshot, doctor attach, the release/workspace spawn, spawn_daemon_mode_with_env, spawn_installed_sibling, spawn_via_daemon_with_env).

Open: cause 1 (the overhead budgets; the job is dispatched on the fixed tree once v0.41.52 is merged) and a way for a red scheduled run to reach the operator on the same day. The second needs a decision, because it would let CI post to the tracker by itself.

The runners resolved names again at about 07:13 UTC on 11-13; the outage lasted roughly eight hours.

## Workflow changes on the train branch after v0.41.52 (2026-11-13), and what they still owe Merged into `integration-1114`; none of it is in v0.41.52. Checked locally (48 CI script unit tests, the registry gates, a YAML parse of all six workflows, targeted Rust tests with mutations). **A dispatched run has to prove each item**, because conditions and step outcomes are evaluated by the runner. | Cause | Change | Still to prove by a run | |---|---|---| | tests skipped when `cargo deny` cannot fetch (night of 11-13) | `cargo test` no longer needs `deny`; `deny` stays a required job and `require_ci_green.sh` still refuses a release whose deny job failed | scheduling of the test jobs when deny is red | | deny: fetch failure against finding | the job stays red in both cases and its summary quotes the tool's own lines as "policy findings", "could not fetch" or "unclassified" | the wording against real output | | 10, downloads | one helper retries `rustup target add`, `rustup toolchain install`, `cargo install` and the rustup bootstrap: three attempts of at most 600 s. It cannot tell a transient from a permanent error and retries both; the last status is kept. Linux jobs only: no step on the Windows runner uses Python today, so its bootstrap is unchanged | a real outage | | 4, step guards | a consumer step requires the OUTCOME of the step that builds what it reads (CLI and worker for parity, worker for fleet cost, runtimes for both acceptance runs, the scale build for ceilings/ratchet/2M overview, the guest target for the package suites) | that no producer is skipped for another reason, which would skip its consumer silently | | 5, refused measurement | the step stays red with the line `NOT MEASURED: runner capacity admission refused`; the step that reads the report is skipped instead of failing a second time | — | | 2, daemon exited before ready | the shared test helper keeps the daemon's stderr in a file and prints its last 8 KiB and the exit status or signal in the panic | the next occurrence | | 6 (#288), ladder input | pinned to `rust-analyzer/crates/project-model/test_data/ripgrep-metadata.json`; path and size are printed; a missing file is a clear failure | timing against the ceilings | | 3, `plugin_mutation_busy` | attributed and fixed, see #520 | the scheduled acceptance run | Found on the way and not changed: nine other e2e helpers discard a child's stderr the same way (`spawn_authed`, the fleet `spawn_daemon`, `spawn_initialized_snapshot`, doctor `attach`, the release/workspace `spawn`, `spawn_daemon_mode_with_env`, `spawn_installed_sibling`, `spawn_via_daemon_with_env`). Open: cause 1 (the overhead budgets; the job is dispatched on the fixed tree once v0.41.52 is merged) and a way for a red scheduled run to reach the operator on the same day. The second needs a decision, because it would let CI post to the tracker by itself. The runners resolved names again at about 07:13 UTC on 11-13; the outage lasted roughly eight hours.
buildagent commented 2026-11-13 12:23:40 +01:00
Author
Member
Copy link

One unattributed test failure on the v0.41.52 head (172bcb20), 2026-11-13

After the runners' DNS came back, ci.yml was dispatched twice on the same commit:

run cargo test note
8631 5649 of 5650 passed code-index-mcp::activation_offer_e2e sfc_first_contact_delegates_script_languages_and_refuses_only_unavailable_regions panicked at crates/mcp-server/tests/support/mcp.rs:895: "the index never reached a steady state in 60s", last state reconciling; the test ran 129 s
8637 all passed all 16 jobs green, including the nightly-only ones

What I measured locally for that test (same commit unless stated):

  • alone: 36 s, passed (twice);
  • with 24 busy loops on 12 CPUs: 102 s, passed;
  • previous master 3c8742bf: 144 s alone, 381 s under the same load, passed both times.

So the release head is about four times faster in this test than the tree before it, plain CPU load does not reproduce the failure, and run 8631 as a whole was not slower than a green run (2236 s against 2640 s for the job). I found no earlier record of this test failing. The cause is not known. A dispatched run executes the nightly-only jobs at the same time as cargo test, which a pull-request run does not; that is a difference, not an explanation.

What would decide it next time: the panic names only the last index state. The helper should also print what the daemon was doing (state detail and the end of the daemon log), as the daemon test helper does since the CI hardening lane. That goes into the next train together with the other test helpers that discard stderr.

Two more facts from run 8637

  • The plugin path cost job passed completely on the fixed tree: language_package_overhead 3 passed (47 s), plugin_path_cost 3 passed, bench_promotion_lock, bench_generation_gc (1494 s), bench_read_epoch and the ignored library benches passed. The last scheduled run in which this job passed was 8350 on 2026-11-11.
  • My local full gate did not contain five steps of that job (claim_domain_scale, bench_promotion_lock, bench_generation_gc, bench_read_epoch, the ignored --lib benches). For this release they are covered by run 8637 only. They are added to the gate script for the next train.
**One unattributed test failure on the v0.41.52 head (`172bcb20`), 2026-11-13** After the runners' DNS came back, `ci.yml` was dispatched twice on the same commit: | run | `cargo test` | note | | :-- | :-- | :-- | | 8631 | 5649 of 5650 passed | `code-index-mcp::activation_offer_e2e sfc_first_contact_delegates_script_languages_and_refuses_only_unavailable_regions` panicked at `crates/mcp-server/tests/support/mcp.rs:895`: "the index never reached a steady state in 60s", last state `reconciling`; the test ran 129 s | | 8637 | all passed | all 16 jobs green, including the nightly-only ones | What I measured locally for that test (same commit unless stated): - alone: 36 s, passed (twice); - with 24 busy loops on 12 CPUs: 102 s, passed; - previous master `3c8742bf`: 144 s alone, 381 s under the same load, passed both times. So the release head is about four times faster in this test than the tree before it, plain CPU load does not reproduce the failure, and run 8631 as a whole was not slower than a green run (2236 s against 2640 s for the job). I found no earlier record of this test failing. **The cause is not known.** A dispatched run executes the nightly-only jobs at the same time as `cargo test`, which a pull-request run does not; that is a difference, not an explanation. What would decide it next time: the panic names only the last index state. The helper should also print what the daemon was doing (state detail and the end of the daemon log), as the daemon test helper does since the CI hardening lane. That goes into the next train together with the other test helpers that discard stderr. **Two more facts from run 8637** - The plugin path cost job passed completely on the fixed tree: `language_package_overhead` 3 passed (47 s), `plugin_path_cost` 3 passed, `bench_promotion_lock`, `bench_generation_gc` (1494 s), `bench_read_epoch` and the ignored library benches passed. The last scheduled run in which this job passed was 8350 on 2026-11-11. - My local full gate did not contain five steps of that job (`claim_domain_scale`, `bench_promotion_lock`, `bench_generation_gc`, `bench_read_epoch`, the ignored `--lib` benches). For this release they are covered by run 8637 only. They are added to the gate script for the next train.
buildagent commented 2026-11-13 12:37:31 +01:00
Author
Member
Copy link

Follow-up on the sfc_first_contact… failure of run 8631: investigated, still undecided

A worker read the complete captured output and the startup path, and tried to reproduce it. Result:

  • The failing job was the snapshot leg (--no-daemon). The path makes no network access (a traced run opened no network socket), no updater runs on it, and the project and store locks are taken without waiting in the release. So the night's DNS outage and the update lease of #520 are ruled out as causes.
  • During startup the process answers reconciling for the whole of parity enablement, bundled adoption, package discovery and compilation, indexing and the census. The failure output does not say which of these it was in.
  • Not reproduced by: eight writers doing random overwrites with fdatasync beside the test (50 s, passed), all eleven tests of the binary in parallel (36 s, passed), eight simultaneous isolated copies of the test (54–58 s for the whole test, all passed).
  • Reproduced only artificially: the debug delay hook for package discovery set to 65 s gives exactly the CI panic. That shows the panic cannot tell the phases apart; it does not show the CI cause.

Verdict: neither "runner timing" nor "product stays unsettled for a minute" is established. The deadline is not raised.

Next step, started as a lane for the next train: the settle wait prints the state history, the progress fields, the child's PID and the end of its stderr when it gives up, and the startup state_detail names the phase it is in, so that a waiting user or agent can read it as well.

**Follow-up on the `sfc_first_contact…` failure of run 8631: investigated, still undecided** A worker read the complete captured output and the startup path, and tried to reproduce it. Result: - The failing job was the snapshot leg (`--no-daemon`). The path makes no network access (a traced run opened no network socket), no updater runs on it, and the project and store locks are taken without waiting in the release. So the night's DNS outage and the update lease of #520 are ruled out as causes. - During startup the process answers `reconciling` for the whole of parity enablement, bundled adoption, package discovery and compilation, indexing and the census. The failure output does not say which of these it was in. - Not reproduced by: eight writers doing random overwrites with `fdatasync` beside the test (50 s, passed), all eleven tests of the binary in parallel (36 s, passed), eight simultaneous isolated copies of the test (54–58 s for the whole test, all passed). - Reproduced only artificially: the debug delay hook for package discovery set to 65 s gives exactly the CI panic. That shows the panic cannot tell the phases apart; it does not show the CI cause. Verdict: neither "runner timing" nor "product stays unsettled for a minute" is established. The deadline is not raised. Next step, started as a lane for the next train: the settle wait prints the state history, the progress fields, the child's PID and the end of its stderr when it gives up, and the startup `state_detail` names the phase it is in, so that a waiting user or agent can read it as well.
buildagent commented 2026-11-13 15:02:41 +01:00
Author
Member
Copy link

Cause 1 (overhead budgets) on the fixed tree: package-ci.yml dispatched on master 7fc7fb20 (v0.41.52), run 8657

All package and package-corpus jobs passed. The nightly-only overhead job measured seven budgets; five pass, two are red. Cold start, seconds, median of five; the limit is 1.2 × builtin + 0.6 s:

repository builtin package limit the five package samples runner load (1 min) before → after
cs-dapper 1.18 1.81 2.42 3.57, 2.11, 1.67, 1.81, 1.66 1.62 → 2.69
js-express 0.77 1.16 1.43 3.00, 1.16, 1.08, 1.24, 1.16 1.70 → 1.94
php-guzzle 1.69 3.20 2.63 4.03, 2.05, 2.18, 4.86, 3.20 1.72 → 9.52
python-flask 0.88 1.33 1.56 3.12, 1.10, 1.16, 1.33, 1.41 1.92 → 2.39
ruby-sinatra 0.93 1.34 1.62 3.33, 1.24, 1.45, 1.26, 1.34 1.80 → 1.92
rust-ripgrep 2.21 2.68 3.15 4.73, 2.62, 2.80, 2.68, 2.63 1.70 → 1.88
ts-zod 5.59 7.24 7.21 6.71, 4.92, 8.65, 7.24, 7.93 1.85 → 17.80

Reading:

  • The startup fix of #581 took the budgets back inside their limits where the runner stayed quiet: in the five passing rows the load after the measurement is 1.9 to 2.7, and after the first sample (which includes the package compile) the samples are steady.
  • Both red rows are the two measurements during which the runner's load rose far above the admission limit of 2.0 (to 9.5 and to 17.8 on 8 CPUs). The passing measurements do not raise the load like that, so the load came from outside the measurement. The PHP samples split into two groups (2.05 and 2.18 against 3.2 to 4.9); the two quiet samples are inside the limit. TypeScript misses its limit by 0.03 s.
  • So for these two rows this run decides nothing about the product: they are contended measurements. The admission check looks at the load only before the measurement starts.

What follows from it:

  1. The measurement should check the load again when it ends. A measurement that ended above the limit is "not measured (contended)", is repeated a bounded number of times, and stays red with that reason if no quiet repetition succeeds. Today it is reported as a budget failure, which names the wrong cause.
  2. The local full gate never ran these overhead budgets (package_leg.sh <package> full does not include the overhead phase). A mechanical comparison of the gate script with all workflow jobs found this and a few other steps that had run only in CI (deny_check.py, the three wasm32 checks, the candidate pilot of the test job, all of tests/ci/test_*.py instead of one file). They are in the gate script for the next train; PHP and TypeScript get an uncontended local measurement there.
  3. What the load came from on the runner host at 13:5x UTC is not known from here; the runners are not on the development machine.
**Cause 1 (overhead budgets) on the fixed tree: `package-ci.yml` dispatched on master `7fc7fb20` (v0.41.52), run 8657** All package and package-corpus jobs passed. The nightly-only overhead job measured seven budgets; five pass, two are red. Cold start, seconds, median of five; the limit is 1.2 × builtin + 0.6 s: | repository | builtin | package | limit | the five package samples | runner load (1 min) before → after | | :-- | --: | --: | --: | :-- | :-- | | cs-dapper | 1.18 | 1.81 | 2.42 | 3.57, 2.11, 1.67, 1.81, 1.66 | 1.62 → 2.69 | | js-express | 0.77 | 1.16 | 1.43 | 3.00, 1.16, 1.08, 1.24, 1.16 | 1.70 → 1.94 | | **php-guzzle** | 1.69 | **3.20** | 2.63 | 4.03, 2.05, 2.18, 4.86, 3.20 | 1.72 → **9.52** | | python-flask | 0.88 | 1.33 | 1.56 | 3.12, 1.10, 1.16, 1.33, 1.41 | 1.92 → 2.39 | | ruby-sinatra | 0.93 | 1.34 | 1.62 | 3.33, 1.24, 1.45, 1.26, 1.34 | 1.80 → 1.92 | | rust-ripgrep | 2.21 | 2.68 | 3.15 | 4.73, 2.62, 2.80, 2.68, 2.63 | 1.70 → 1.88 | | **ts-zod** | 5.59 | **7.24** | 7.21 | 6.71, 4.92, 8.65, 7.24, 7.93 | 1.85 → **17.80** | Reading: - The startup fix of #581 took the budgets back inside their limits where the runner stayed quiet: in the five passing rows the load after the measurement is 1.9 to 2.7, and after the first sample (which includes the package compile) the samples are steady. - Both red rows are the two measurements during which the runner's load rose far above the admission limit of 2.0 (to 9.5 and to 17.8 on 8 CPUs). The passing measurements do not raise the load like that, so the load came from outside the measurement. The PHP samples split into two groups (2.05 and 2.18 against 3.2 to 4.9); the two quiet samples are inside the limit. TypeScript misses its limit by 0.03 s. - So for these two rows this run decides nothing about the product: they are contended measurements. The admission check looks at the load only before the measurement starts. What follows from it: 1. The measurement should check the load again when it ends. A measurement that ended above the limit is "not measured (contended)", is repeated a bounded number of times, and stays red with that reason if no quiet repetition succeeds. Today it is reported as a budget failure, which names the wrong cause. 2. The local full gate never ran these overhead budgets (`package_leg.sh <package> full` does not include the `overhead` phase). A mechanical comparison of the gate script with all workflow jobs found this and a few other steps that had run only in CI (`deny_check.py`, the three wasm32 checks, the candidate pilot of the test job, all of `tests/ci/test_*.py` instead of one file). They are in the gate script for the next train; PHP and TypeScript get an uncontended local measurement there. 3. What the load came from on the runner host at 13:5x UTC is not known from here; the runners are not on the development machine.
buildagent commented 2026-11-13 15:36:30 +01:00
Author
Member
Copy link

Contended overhead measurements: correction of my proposal above, and what is now on the next-train branch (lane r39, merged locally as 77fbf1df; not released, and not yet proven by a dispatched run).

Correction. I proposed above to call a measurement contended when the load at its end is above the admission limit (2.0 on the 8-CPU runner). That rule is wrong: the measurement itself loads the machine. In run 8657 two passing measurements ended at 2.69 (cs-dapper) and 2.39 (python-flask); they would have been repeated for about 24 minutes each and probably reported as not measured.

Data. 56 overhead records from the eight most recent runs whose logs are still available (older logs are gone), all on 8 CPUs:

records load (1 min) at the end
passed 39 0.99 to 3.01
failed 17 1.06 to 17.80

The two groups overlap: most of the failures ended on a quiet machine. Several of them are from nights on trees with the startup regression of #581 (run 8513 for example); I have not attributed each of the 14 one by one.

Rule implemented. A measurement is contended when the load at its end is above four times the admission limit, which is one runnable task per visible CPU (8.0 on this runner): more work was runnable than the machine has processors. The end limit is derived from the admission limit that the wrapper passes in, and the record states it. Applied to the 56 records: no pass is reclassified; 3 of the 17 failures are contended (php-guzzle 9.52 and ts-zod 17.80 in run 8657, cs-dapper 11.71 in run 8599); the other 14 failures stay budget failures. This is a saturation check, not a fitted threshold: it will not catch moderate contention below saturation, and it says so in docs/releasing.md.

Behaviour.

  • A contended measurement is repeated once after the admission condition holds again, within 70 minutes per package including waits. A quiet repetition decides as before (pass, or the budget failure with its usual text).
  • If no attempt was quiet: exit status 75, red, NOT MEASURED (contended) with every attempt's samples and loads, and no budget sentence.
  • A semantic mismatch between package and builtin is reported at once and never repeated.
  • A developer run without the idle requirement behaves as before; its record says contention_check: "off".
  • The measurement's output is streamed while it runs.

Checked. The seven records of run 8657 are a table in the unit tests (five quiet, two contended); using the admission limit as the end limit again fails that test on exactly cs-dapper and python-flask (mutation run by the worker). In my run the 54 Python CI tests and the five Rust contention tests pass.

Open.

  • The overhead job declares no timeout-minutes. With repetitions the formal upper bound is 70 minutes per package; a night on a busy runner can take hours. The job needs a job-wide bound; not changed here.
  • Proof needs a dispatched run of package-ci.yml on the branch, like every workflow item of the CI hardening lane.
  • Local measurement of PHP and TypeScript on a quiet machine follows with the full gate run.
**Contended overhead measurements: correction of my proposal above, and what is now on the next-train branch** (lane r39, merged locally as `77fbf1df`; not released, and not yet proven by a dispatched run). **Correction.** I proposed above to call a measurement contended when the load at its end is above the admission limit (2.0 on the 8-CPU runner). That rule is wrong: the measurement itself loads the machine. In run 8657 two passing measurements ended at 2.69 (cs-dapper) and 2.39 (python-flask); they would have been repeated for about 24 minutes each and probably reported as not measured. **Data.** 56 overhead records from the eight most recent runs whose logs are still available (older logs are gone), all on 8 CPUs: | | records | load (1 min) at the end | | :-- | --: | :-- | | passed | 39 | 0.99 to 3.01 | | failed | 17 | 1.06 to 17.80 | The two groups overlap: most of the failures ended on a quiet machine. Several of them are from nights on trees with the startup regression of #581 (run 8513 for example); I have not attributed each of the 14 one by one. **Rule implemented.** A measurement is contended when the load at its end is above four times the admission limit, which is one runnable task per visible CPU (8.0 on this runner): more work was runnable than the machine has processors. The end limit is derived from the admission limit that the wrapper passes in, and the record states it. Applied to the 56 records: no pass is reclassified; 3 of the 17 failures are contended (php-guzzle 9.52 and ts-zod 17.80 in run 8657, cs-dapper 11.71 in run 8599); the other 14 failures stay budget failures. This is a saturation check, not a fitted threshold: it will not catch moderate contention below saturation, and it says so in `docs/releasing.md`. **Behaviour.** - A contended measurement is repeated once after the admission condition holds again, within 70 minutes per package including waits. A quiet repetition decides as before (pass, or the budget failure with its usual text). - If no attempt was quiet: exit status 75, red, `NOT MEASURED (contended)` with every attempt's samples and loads, and no budget sentence. - A semantic mismatch between package and builtin is reported at once and never repeated. - A developer run without the idle requirement behaves as before; its record says `contention_check: "off"`. - The measurement's output is streamed while it runs. **Checked.** The seven records of run 8657 are a table in the unit tests (five quiet, two contended); using the admission limit as the end limit again fails that test on exactly cs-dapper and python-flask (mutation run by the worker). In my run the 54 Python CI tests and the five Rust contention tests pass. **Open.** - The overhead job declares no `timeout-minutes`. With repetitions the formal upper bound is 70 minutes per package; a night on a busy runner can take hours. The job needs a job-wide bound; not changed here. - Proof needs a dispatched run of `package-ci.yml` on the branch, like every workflow item of the CI hardening lane. - Local measurement of PHP and TypeScript on a quiet machine follows with the full gate run.
buildagent commented 2026-11-13 16:50:48 +01:00
Author
Member
Copy link

Operator decision, 2026-11-13: when a scheduled run is red, CI itself posts a comment on one standing tracker issue, naming the failed jobs and linking the run. A lane implements it now; this issue is the target until a dedicated "nightly status" issue is wanted.

**Operator decision, 2026-11-13:** when a scheduled run is red, CI itself posts a comment on **one standing tracker issue**, naming the failed jobs and linking the run. A lane implements it now; this issue is the target until a dedicated "nightly status" issue is wanted.
buildagent commented 2026-11-14 12:28:46 +01:00
Author
Member
Copy link

v0.41.53 reports red scheduled ci.yml and package-ci.yml runs on #582, folding repeated failures into the existing comment. Contended overhead measurements retry once and otherwise report NOT MEASURED (contended); CPU limits are read from the process's cgroup hierarchy, downloads retry, cargo test no longer waits for cargo deny, and settle timeouts include diagnostics.

The 2026-11-14 scheduled ci.yml run 8711 on v0.41.52 had one red job, cargo test (daemon transport): two previews saw different live index counts because the test did not pin index state. The nightly throughput job was skipped; 14 other jobs were green.

The scheduled package-ci.yml run 8697 of the same night on v0.41.52 was red in the overhead job only; every package and corpus job was green. In that job the C# measurement never started (the runner did not become quiet within 90 minutes while a pull-request round ran on it), python-flask measured a warm ratio of 1.100 and ruby-sinatra 1.161, each against 1.10.

The dispatched package-ci.yml proof on release head 8ff34387 had 69 jobs green; the overhead job was red on Ruby at 1.100 against 1.10, and the dry-run report printed. The dispatched ci.yml proof: run 8726 on release head 8ff34387 had all 17 jobs green, including tier-3 scale, grammar rebuild from source, path cost and pool throughput, and the report job.

Ruby measured 1.115 / 1.041 / 1.116 / 1.063 in four quiet local runs against 1.10; seven of eight earlier nightly runs inspected were red for ruby-sinatra. The decision is to release v0.41.53 with this recorded as a known flapping budget and make the measurement stable next. Nothing in this release touches Ruby's warm path; the overhead job has no job-wide time limit.

On slow runners in PR #587, a test hit its 60 s settle deadline; the 600 s per-attempt install wrapper timed out three times and was fixed in 8ff34387; and the 30 s precompile wall-clock budget refused a valid package (#589). The #588 and #589 fixes ship in the next release. The preview test still needs its index state pinned; the fresh-checkout CLI tests also need to build the plugin host or name the missing binary.

Released as v0.41.53 at tagged commit 59b8bc3d; release run 8731 was green, 15 assets were published, and installed binaries report 0.41.53 (59b8bc3d).

v0.41.53 reports red scheduled `ci.yml` and `package-ci.yml` runs on #582, folding repeated failures into the existing comment. Contended overhead measurements retry once and otherwise report `NOT MEASURED (contended)`; CPU limits are read from the process's cgroup hierarchy, downloads retry, `cargo test` no longer waits for `cargo deny`, and settle timeouts include diagnostics. The 2026-11-14 scheduled `ci.yml` run 8711 on v0.41.52 had one red job, `cargo test (daemon transport)`: two previews saw different live index counts because the test did not pin index state. The nightly throughput job was skipped; 14 other jobs were green. The scheduled `package-ci.yml` run 8697 of the same night on v0.41.52 was red in the overhead job only; every package and corpus job was green. In that job the C# measurement never started (the runner did not become quiet within 90 minutes while a pull-request round ran on it), `python-flask` measured a warm ratio of 1.100 and `ruby-sinatra` 1.161, each against 1.10. The dispatched `package-ci.yml` proof on release head `8ff34387` had 69 jobs green; the overhead job was red on Ruby at 1.100 against 1.10, and the dry-run report printed. The dispatched `ci.yml` proof: run 8726 on release head `8ff34387` had all 17 jobs green, including tier-3 scale, grammar rebuild from source, path cost and pool throughput, and the report job. Ruby measured 1.115 / 1.041 / 1.116 / 1.063 in four quiet local runs against 1.10; seven of eight earlier nightly runs inspected were red for `ruby-sinatra`. The decision is to release v0.41.53 with this recorded as a known flapping budget and make the measurement stable next. Nothing in this release touches Ruby's warm path; the overhead job has no job-wide time limit. On slow runners in PR #587, a test hit its 60 s settle deadline; the 600 s per-attempt install wrapper timed out three times and was fixed in `8ff34387`; and the 30 s precompile wall-clock budget refused a valid package (#589). The #588 and #589 fixes ship in the next release. The preview test still needs its index state pinned; the fresh-checkout CLI tests also need to build the plugin host or name the missing binary. Released as v0.41.53 at tagged commit `59b8bc3d`; release run 8731 was green, 15 assets were published, and installed binaries report `0.41.53 (59b8bc3d)`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
integration-1114
integration-1113
fix/production-batch-046
fix/production-followups-044
fix/automatic-project-reconciliation
fix/automatic-first-party-grants
fix/release-state-settlement
diagnostic/runner-capacity-04123
diag/runner-resources-6992
diag/runner-capacity-7013
codex/release-v0.41.21
codex/csharp-method-group-uses
codex/release-v0.41.20
codex/csharp-generic-invocation-types
codex/pilot-semantic-contract
codex/candidate-runtime-pilots
codex/csharp-provider
codex/sdk-boundary
v0.41.53
v0.41.52
v0.41.51
v0.41.50
pkg/de.h-dv.php/v0.1.6
v0.41.49
v0.41.48
pkg/de.h-dv.rust/v0.1.8
pkg/de.h-dv.xaml/v0.3.4
pkg/de.h-dv.typescript/v0.1.7
pkg/de.h-dv.javascript/v0.1.6
v0.41.47
v0.41.46
pkg/de.h-dv.vue/v0.1.2
pkg/de.h-dv.svelte/v0.2.5
pkg/de.h-dv.typescript/v0.1.6
pkg/de.h-dv.javascript/v0.1.5
v0.41.45
v0.41.44
v0.41.43
pkg/de.h-dv.xaml/v0.3.3
pkg/de.h-dv.csharp/v0.1.11
v0.41.42
pkg/de.h-dv.timeline/v0.1.2
v0.41.41
v0.41.40
v0.41.39
v0.41.38
v0.41.37
v0.41.36
v0.41.35
pkg/de.h-dv.svelte/v0.2.4
v0.41.34
v0.41.33
v0.41.32
pkg/de.h-dv.svelte/v0.2.3
v0.41.31
v0.41.30
v0.41.29
pkg/de.h-dv.rust/v0.1.7
v0.41.28
v0.41.27
v0.41.26
v0.41.25
v0.41.24
v0.41.23
pkg/de.h-dv.vue/v0.1.1
pkg/de.h-dv.timeline/v0.1.1
pkg/de.h-dv.svelte/v0.2.2
pkg/de.h-dv.rust/v0.1.6
pkg/de.h-dv.python/v0.1.5
pkg/de.h-dv.php/v0.1.4
pkg/de.h-dv.ruby/v0.10.2
pkg/de.h-dv.typescript/v0.1.5
pkg/de.h-dv.javascript/v0.1.4
pkg/de.h-dv.csharp/v0.1.10
v0.41.22
v0.41.21
pkg/de.h-dv.csharp/v0.1.9
v0.41.20
v0.41.19
v0.41.18
pkg/de.h-dv.xaml/v0.3.2
pkg/de.h-dv.csharp/v0.1.8
v0.41.17
v0.41.16
pkg/de.h-dv.csharp/v0.1.7
v0.41.15
pkg/de.h-dv.xaml/v0.3.1
pkg/de.h-dv.svelte/v0.2.1
v0.41.14
v0.41.13
v0.41.12
v0.41.11
v0.41.10
v0.41.9
pkg/de.h-dv.python/v0.1.4
v0.41.8
v0.41.7
pkg/de.h-dv.rust/v0.1.5
v0.41.6
v0.41.5
pkg/de.h-dv.csharp/v0.1.6
v0.41.4
v0.41.3
v0.41.2
pkg/de.h-dv.csharp/v0.1.5
v0.41.1
v0.41.0
v0.40.0
v0.39.0
pkg/de.h-dv.ruby/v0.10.1
v0.38.0
v0.37.2
v0.37.1
v0.37.0
v0.36.1
pkg/de.h-dv.ruby/v0.10.0
v0.36.0
v0.35.0
v0.34.1
v0.34.0
v0.33.1
pkg-channel
v0.33.0
v0.32.2
v0.32.1
v0.32.0
v0.31.2
v0.31.1
v0.31.0
v0.30.1
v0.30.0
v0.29.0
v0.28.3
v0.28.2
v0.28.1
v0.28.0
v0.27.1
v0.27.0
v0.26.1
v0.26.0
v0.25.0
v0.24.1
v0.24.0
v0.23.1
v0.23.0
v0.22.3
v0.22.2
v0.22.1
v0.22.0
v0.21.0
v0.20.3
v0.20.2
v0.20.1
v0.20.0
v0.19.5
v0.19.4
v0.19.3
v0.19.2
v0.19.1
v0.19.0
v0.18.1
v0.18.0
v0.17.0
v0.16.0
v0.15.3
v0.14.0
v0.13.3
v0.13.2
v0.13.1
v0.13.0
v0.12.0
v0.11.7
v0.11.6
v0.11.5
v0.11.4
v0.11.3
v0.11.2
v0.11.1
v0.11.0
v0.10.2
v0.10.1
v0.10.0
v0.9.0
v0.8.7
v0.8.6
v0.8.5
v0.8.4
v0.8.3
v0.8.2
v0.8.1
v0.8.0
v0.7.1
v0.7.0
v0.6.1
v0.6.0
v0.5.21
v0.5.20
v0.5.19
v0.5.18
v0.5.17
v0.5.16
v0.5.15
v0.5.14
v0.5.13
v0.5.12
v0.5.11
v0.5.10
v0.5.9
v0.5.8
v0.5.7
v0.5.6
v0.5.5
v0.5.4
v0.5.3
v0.5.2
v0.5.1
v0.5.0
v0.4.2
v0.4.1
v0.4.0
v0.3.3
v0.3.2
v0.3.1
v0.3.0
v0.2.0
Labels
Clear labels   
code-review

   
correctness

   
dos

   
performance

   
security

   
severity/high

   
severity/low

   
severity/medium

   
tech-debt
  
Kind/Breaking
Breaking change that won't be backward compatible

  
Kind/Bug
Something is not working

  
Kind/Documentation
Documentation changes

  
Kind/Enhancement
Improve existing functionality

  
Kind/Feature
New functionality

  
Kind/Security
This is security issue

  
Kind/Testing
Issue or pull request related to testing

  
Priority
Critical
 The priority is critical

  
Priority
High
 The priority is high

  
Priority
Low
 The priority is low

  
Priority
Medium
 The priority is medium

  
Reviewed
Confirmed
 Issue has been confirmed

  
Reviewed
Duplicate
 This issue or pull request already exists

  
Reviewed
Invalid
 Invalid issue

  
Reviewed
Won't Fix
 This issue won't be fixed

  
Status
Abandoned
 Somebody has started to work on this but abandoned work

  
Status
Blocked
 Something is blocking this issue or pull request

  
Status
Need More Info
 Feedback is required to reproduce issue or to continue work

No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#582
No description provided.