CI: Windows step advances about 7.5 minutes after native nextest returns #435
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
h-dv/code-index#435
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Windows CI run 6516 (UI run 1611), job 47985, head
9898aa94712c86b8d9d2cbb0b7a8099305a18cac:Root cause is not yet measured. Separate the nextest process exit time from PowerShell/runner pipe draining and any background-process shutdown. Record process creation times when identifying descendants: Windows ParentProcessId alone can falsely attach an unrelated process after PID reuse.
Acceptance: identify the owner of the post-summary delay, reproduce the mechanism narrowly, and remove the delay while preserving nextest's actual exit status, complete test set, and runtime deadlines. Do not turn a printed summary into a forced green exit or kill processes by name. Keep timing, doctest, installer and shipped-profile smoke steps mandatory.
Related #408 concerns a different stage (cache compression after tests). This report concerns the workspace-nextest step before timing begins.
Measured boundary, 2026-11-07
PR #502 run 8058 (Windows run 2334), job 62861, independently records the native return:
2026-11-07T01:33:27.1789666Z(5,155 tests, 15 marked leaky).2026-11-07T01:33:27.2257396Z, exit code0, PowerShell PID21620.2026-11-07T01:40:56.9631876Z, PowerShell PID30424.This locates the unexplained interval after native nextest has returned to PowerShell. It does not yet distinguish PowerShell process exit, inherited pipe handles, runner cleanup, or scheduling. Kernel process-exit and pipe-owner observations are still missing; leak counts do not identify the owner. Keep the issue open and preserve the actual native exit status and all downstream gates.
New correlated witness on PR#439 head4390bb131f490bb7e423cd801621e7ead2b8f50d, Windows run6548/UI1634, job48306, task39376. Workspace nextest printed its final summary at2026-10-24 09:45:29.645Z: all4836 passed,11 leaky. The ci-timing nextest run began09:52:59.593Z (about450s later). Its preceding cargo test build reports2.00s at09:52:50.716Z. Exact next-step start/runner finalization timestamps remain to be extracted; this is the observed summary→timing-run interval, not a claim that nextest stayed alive throughout it.
Three partners independently inspected processes read-only at09:49:07–09:49:22Z: forgejo-runner.exe pid9800 had no children, and no live cargo-nextest/cargo/rustc was found. Runner identity was then correlated explicitly: UUID52610585-3169-471c-8854-68cc9c5c653e in pid9800's command line, ownerW100/ci-runner, and C:/ci-work/work/af2a0a61ef253305/hostexecutor created/written in this job's09:25–09:26Z startup window. Process creation times were included to avoid PID reuse. The claim 'no compiler processes created in the prior2h' was corrected by the reporting partner: Win32_Process snapshots only establish what is still alive, not prior lifetimes.
Thus a matched-runner snapshot during this interval already showed nextest absent. Further measurement should locate process exit, output forwarding, task reporting and next-step scheduling; the snapshots do not identify one of those as the cause. No process/runner/workflow state was modified, no summary was promoted to a green exit, and all remaining Windows checks continue normally.
Visibility caveat from E3's follow-up: observers run as another Windows account. Process names and parent links are visible, while ci-runner command lines can be unavailable. Host-wide command-line searches alone therefore cannot prove process absence. E4 explicitly reported process-name absence, and the matched-runner child snapshots are retained with their limited observation times. Capture the runner account's own step/process lifecycle for a definitive boundary measurement.
Reproduced on v0.41.19 qualification head9854283a, Windows run6659/job49596: ordinary nextest prints its passing4846/4846 summary at2026-10-24 21:52:49 UTC (9 marked leaky). The ci-timing build finishes at22:00:33 and its35-test run begins at22:00:41. The approximately7.5-minute post-summary delay is still present. The next phase did start normally; no service restart, forced exit or interpretation of the printed summary as process success was used. Root cause remains unmeasured; retaining this as a distinct performance follow-up.
Another bounded observation on PR465's native Windows run6683 / UI1758, job49862, exact head
2e269bebe48cb4804b5c0386ab303ebed4f65af8: all 4,847 ordinary tests passed at 2026-10-25 09:20:23.698 UTC (9 marked leaky). The first subsequent job-log output is at 09:28:04.378 UTC, about 461 seconds later; the serial ci-timing run starts at 09:28:12.608. It progressed normally and reached doctests by 09:34. This is additional evidence of the same post-summary delay, not its cause or a failed test.Upstream nextest #656/#668 describes an older Windows spawned-process/Tokio shutdown hang, fixed in 0.9.44; this repository already pins 0.9.144, so that historical fix alone does not justify changing our pin or claiming a repair. Source check also shows the Windows workflow does not invoke the Linux compiler-cache setup changed in PR464. Separate process-exit, inherited-pipe and runner-drain timestamps are still needed. No process-name kill, deadline increase or summary-to-success shortcut was introduced. Primary upstream reference: https://github.com/nextest-rs/nextest/issues/656.
PR#468 final-head instrumentation now locates the delay more precisely on native Windows run6704 (UI1779), job50112:
NEXTEST phase=ci boundary=returned utc=2026-10-25T12:12:47.7591161Z elapsed_ms=831645 exit_code=0 powershell_pid=53900(the log received it at12:12:47.7647194Z).NEXTEST phase=ci-timing boundary=invoke utc=2026-10-25T12:20:26.4289473Z powershell_pid=53096.Measured return→next invocation gap:458.6698312seconds (7m38.67s). This run therefore establishes that cargo/nextest returned to PowerShell promptly after its summary; the long interval is AFTER that return and BEFORE the next step's invocation. The script has only logging and
exitafter the return marker, preserving the native status. This does not independently measure PowerShell process exit or identify runner/stream-drain behavior; those remain the next boundary to observe. PIDs are shell identities at the recorded instant, not permission to act on them later without creation-time checks. No test was skipped, deadline raised, process killed, or summary substituted for workflow success. #435 remains open.Fresh boundary evidence from the successful v0.41.38 source qualification, Windows run 2235 / job 61303, head
f54a875877:NEXTEST phase=ci boundary=returned ... exit_code=0(PowerShell PID 6048).This narrows the observed gap: the native nextest command had already returned to PowerShell successfully before the delay. The owner of the subsequent step/runner delay remains unmeasured. No process-name kills, summary-based forced exits, deadline changes, or skipped gates were used. The ticket remains open pending the narrow runner/process-lifetime reproduction and fix.
CI: Windows nextest returns about 7.5 minutes after its passing summaryto CI: Windows step advances about 7.5 minutes after native nextest returnsThe final 0.41.42 source head
401f32a6passed native Windows qualification, but this delay remains. Job 62913 records the passing 5,155-test summary at2026-11-07T02:50:11.1905009Z, native return with exit 0 at02:50:11.2559585Z(PowerShell PID 17680), and the timing invocation at02:58:01.2483332Z(PID 41252): 469.992375 seconds after native return. There were 16 leaky tests; that count does not identify the delay owner.The subsequent timing, doctest, installer and shipped-profile smoke checks passed. Kernel PowerShell-exit and runner pipe-owner observations remain unmeasured, so this is still open and is not claimed fixed in 0.41.42. Local receipt:
/tmp/cosi-435-final-401f32a-boundaries.json.Measured the native-return gap on the last green Windows run (job 64795): nextest returned exit 0 at 10:48:31.5102981Z, while the next step began 466.623 seconds later.
The dispatch-only, build-free diagnostic isolates the mechanism. Run 8255: inherited and merely redirected descendants each delayed the next step ~8.53s after PowerShell exited; ShellExecute and CreateProcess(bInheritHandles=false) controls took ~0.41s while their descendants were still running. This proves an inherited-handle delay; it does not identify the particular descendant in the historical field run.
Candidate fix
8519eb52launches nextest through ShellExecute, streams the output file as raw bytes, and propagates the native exit code. The actual helper passed Windows run 8257/job 64903: native exits 0 and 37, exact arguments, inherited environment, working directory, paths containing spaces/%/& and split UTF-8 output. The next step began in 0.291s with both background descendants still alive. Both completed independently; no process was killed. Local CI path/release contracts also pass (59+44 tests).Keeping this open until the combined release qualification runs the real full native suite with the helper. No gate, timeout or success criterion was weakened.
Fixed and merged in PR #509, merge93e46879 (tree-identical to qualified source49edd903).
The actual Windows helper now starts nextest without inheriting the runner's handles, forwards raw output and preserves the native process status. Existing timing and downstream checks remain mandatory. Final qualification8262/job64953 passed5,185 workspace tests,43 timing tests and all downstream checks. Nextest returned18:18:04.0806143Z; the next timing step began18:18:04.4545611Z:0.3739468s, versus466.623s on the previous green job. An independent preceding full run measured0.2837696s and also completed successfully.
The diagnostic exercised native exits0/37, arguments/environment/cwd, metacharacter paths, split UTF-8, and still-running background descendants; removing failure/timing/status propagation is rejected by executed contract mutations. The measured step-delay defect is closed.