CI: Windows step advances about 7.5 minutes after native nextest returns #435

Closed
opened 2026-10-24 00:19:00 +02:00 by buildagent · 8 comments
Member

Windows CI run 6516 (UI run 1611), job 47985, head 9898aa94712c86b8d9d2cbb0b7a8099305a18cac:

  • Workspace nextest prints its final summary at 2026-10-23 22:09:05 UTC: all 4,834 tests passed, 9 marked leaky.
  • The next timing step begins about 22:16:42 UTC, approximately 457 seconds later. The public job state confirms the workspace step succeeded (21m22s) and ci-timing is running. This is a slow return, not evidence of failed assertions or a permanent hang.
  • A partner independently inspected the active runner read-only and confirmed the new timing-step PowerShell process and its test child. The partner observed the same shape on an earlier job 46115.

Root cause is not yet measured. Separate the nextest process exit time from PowerShell/runner pipe draining and any background-process shutdown. Record process creation times when identifying descendants: Windows ParentProcessId alone can falsely attach an unrelated process after PID reuse.

Acceptance: identify the owner of the post-summary delay, reproduce the mechanism narrowly, and remove the delay while preserving nextest's actual exit status, complete test set, and runtime deadlines. Do not turn a printed summary into a forced green exit or kill processes by name. Keep timing, doctest, installer and shipped-profile smoke steps mandatory.

Related #408 concerns a different stage (cache compression after tests). This report concerns the workspace-nextest step before timing begins.

Measured boundary, 2026-11-07

PR #502 run 8058 (Windows run 2334), job 62861, independently records the native return:

  • Passing workspace summary: 2026-11-07T01:33:27.1789666Z (5,155 tests, 15 marked leaky).
  • PowerShell records native nextest return: 2026-11-07T01:33:27.2257396Z, exit code 0, PowerShell PID 21620.
  • Next timing invocation: 2026-11-07T01:40:56.9631876Z, PowerShell PID 30424.
  • Native return to next invocation: 449.737448 seconds. The complete Windows job subsequently passed.

This locates the unexplained interval after native nextest has returned to PowerShell. It does not yet distinguish PowerShell process exit, inherited pipe handles, runner cleanup, or scheduling. Kernel process-exit and pipe-owner observations are still missing; leak counts do not identify the owner. Keep the issue open and preserve the actual native exit status and all downstream gates.

Windows CI run 6516 ([UI run 1611](https://git.h-dv.de/h-dv/code-index/actions/runs/1611/jobs/1)), job 47985, head `9898aa94712c86b8d9d2cbb0b7a8099305a18cac`: - Workspace nextest prints its final summary at 2026-10-23 22:09:05 UTC: all 4,834 tests passed, 9 marked leaky. - The next timing step begins about 22:16:42 UTC, approximately 457 seconds later. The public job state confirms the workspace step succeeded (21m22s) and ci-timing is running. This is a slow return, not evidence of failed assertions or a permanent hang. - A partner independently inspected the active runner read-only and confirmed the new timing-step PowerShell process and its test child. The partner observed the same shape on an earlier job 46115. Root cause is not yet measured. Separate the nextest process exit time from PowerShell/runner pipe draining and any background-process shutdown. Record process creation times when identifying descendants: Windows ParentProcessId alone can falsely attach an unrelated process after PID reuse. Acceptance: identify the owner of the post-summary delay, reproduce the mechanism narrowly, and remove the delay while preserving nextest's actual exit status, complete test set, and runtime deadlines. Do not turn a printed summary into a forced green exit or kill processes by name. Keep timing, doctest, installer and shipped-profile smoke steps mandatory. Related #408 concerns a different stage (cache compression after tests). This report concerns the workspace-nextest step before timing begins. ## Measured boundary, 2026-11-07 PR #502 run 8058 ([Windows run 2334](https://git.h-dv.de/h-dv/code-index/actions/runs/2334)), job 62861, independently records the native return: - Passing workspace summary: `2026-11-07T01:33:27.1789666Z` (5,155 tests, 15 marked leaky). - PowerShell records native nextest return: `2026-11-07T01:33:27.2257396Z`, exit code `0`, PowerShell PID `21620`. - Next timing invocation: `2026-11-07T01:40:56.9631876Z`, PowerShell PID `30424`. - Native return to next invocation: **449.737448 seconds**. The complete Windows job subsequently passed. This locates the unexplained interval **after native nextest has returned to PowerShell**. It does not yet distinguish PowerShell process exit, inherited pipe handles, runner cleanup, or scheduling. Kernel process-exit and pipe-owner observations are still missing; leak counts do not identify the owner. Keep the issue open and preserve the actual native exit status and all downstream gates.
Author
Member

New correlated witness on PR#439 head4390bb131f490bb7e423cd801621e7ead2b8f50d, Windows run6548/UI1634, job48306, task39376. Workspace nextest printed its final summary at2026-10-24 09:45:29.645Z: all4836 passed,11 leaky. The ci-timing nextest run began09:52:59.593Z (about450s later). Its preceding cargo test build reports2.00s at09:52:50.716Z. Exact next-step start/runner finalization timestamps remain to be extracted; this is the observed summary→timing-run interval, not a claim that nextest stayed alive throughout it.

Three partners independently inspected processes read-only at09:49:07–09:49:22Z: forgejo-runner.exe pid9800 had no children, and no live cargo-nextest/cargo/rustc was found. Runner identity was then correlated explicitly: UUID52610585-3169-471c-8854-68cc9c5c653e in pid9800's command line, ownerW100/ci-runner, and C:/ci-work/work/af2a0a61ef253305/hostexecutor created/written in this job's09:25–09:26Z startup window. Process creation times were included to avoid PID reuse. The claim 'no compiler processes created in the prior2h' was corrected by the reporting partner: Win32_Process snapshots only establish what is still alive, not prior lifetimes.

Thus a matched-runner snapshot during this interval already showed nextest absent. Further measurement should locate process exit, output forwarding, task reporting and next-step scheduling; the snapshots do not identify one of those as the cause. No process/runner/workflow state was modified, no summary was promoted to a green exit, and all remaining Windows checks continue normally.

Visibility caveat from E3's follow-up: observers run as another Windows account. Process names and parent links are visible, while ci-runner command lines can be unavailable. Host-wide command-line searches alone therefore cannot prove process absence. E4 explicitly reported process-name absence, and the matched-runner child snapshots are retained with their limited observation times. Capture the runner account's own step/process lifecycle for a definitive boundary measurement.

New correlated witness on PR#439 head4390bb131f490bb7e423cd801621e7ead2b8f50d, Windows run6548/UI1634, job48306, task39376. Workspace nextest printed its final summary at2026-10-24 09:45:29.645Z: all4836 passed,11 leaky. The ci-timing nextest run began09:52:59.593Z (about450s later). Its preceding cargo test build reports2.00s at09:52:50.716Z. Exact next-step start/runner finalization timestamps remain to be extracted; this is the observed summary→timing-run interval, not a claim that nextest stayed alive throughout it. Three partners independently inspected processes read-only at09:49:07–09:49:22Z: forgejo-runner.exe pid9800 had no children, and no live cargo-nextest/cargo/rustc was found. Runner identity was then correlated explicitly: UUID52610585-3169-471c-8854-68cc9c5c653e in pid9800's command line, ownerW100/ci-runner, and C:/ci-work/work/af2a0a61ef253305/hostexecutor created/written in this job's09:25–09:26Z startup window. Process creation times were included to avoid PID reuse. The claim 'no compiler processes created in the prior2h' was corrected by the reporting partner: Win32_Process snapshots only establish what is still alive, not prior lifetimes. Thus a matched-runner snapshot during this interval already showed nextest absent. Further measurement should locate process exit, output forwarding, task reporting and next-step scheduling; the snapshots do not identify one of those as the cause. No process/runner/workflow state was modified, no summary was promoted to a green exit, and all remaining Windows checks continue normally. Visibility caveat from E3's follow-up: observers run as another Windows account. Process names and parent links are visible, while ci-runner command lines can be unavailable. Host-wide command-line searches alone therefore cannot prove process absence. E4 explicitly reported process-name absence, and the matched-runner child snapshots are retained with their limited observation times. Capture the runner account's own step/process lifecycle for a definitive boundary measurement.
Author
Member

Reproduced on v0.41.19 qualification head9854283a, Windows run6659/job49596: ordinary nextest prints its passing4846/4846 summary at2026-10-24 21:52:49 UTC (9 marked leaky). The ci-timing build finishes at22:00:33 and its35-test run begins at22:00:41. The approximately7.5-minute post-summary delay is still present. The next phase did start normally; no service restart, forced exit or interpretation of the printed summary as process success was used. Root cause remains unmeasured; retaining this as a distinct performance follow-up.

Reproduced on v0.41.19 qualification head9854283a, Windows run6659/job49596: ordinary nextest prints its passing4846/4846 summary at2026-10-24 21:52:49 UTC (9 marked leaky). The ci-timing build finishes at22:00:33 and its35-test run begins at22:00:41. The approximately7.5-minute post-summary delay is still present. The next phase did start normally; no service restart, forced exit or interpretation of the printed summary as process success was used. Root cause remains unmeasured; retaining this as a distinct performance follow-up.
Author
Member

Another bounded observation on PR465's native Windows run6683 / UI1758, job49862, exact head 2e269bebe48cb4804b5c0386ab303ebed4f65af8: all 4,847 ordinary tests passed at 2026-10-25 09:20:23.698 UTC (9 marked leaky). The first subsequent job-log output is at 09:28:04.378 UTC, about 461 seconds later; the serial ci-timing run starts at 09:28:12.608. It progressed normally and reached doctests by 09:34. This is additional evidence of the same post-summary delay, not its cause or a failed test.

Upstream nextest #656/#668 describes an older Windows spawned-process/Tokio shutdown hang, fixed in 0.9.44; this repository already pins 0.9.144, so that historical fix alone does not justify changing our pin or claiming a repair. Source check also shows the Windows workflow does not invoke the Linux compiler-cache setup changed in PR464. Separate process-exit, inherited-pipe and runner-drain timestamps are still needed. No process-name kill, deadline increase or summary-to-success shortcut was introduced. Primary upstream reference: https://github.com/nextest-rs/nextest/issues/656.

Another bounded observation on PR465's native Windows run6683 / UI1758, job49862, exact head `2e269bebe48cb4804b5c0386ab303ebed4f65af8`: all 4,847 ordinary tests passed at 2026-10-25 09:20:23.698 UTC (9 marked leaky). The first subsequent job-log output is at 09:28:04.378 UTC, about 461 seconds later; the serial ci-timing run starts at 09:28:12.608. It progressed normally and reached doctests by 09:34. This is additional evidence of the same post-summary delay, not its cause or a failed test. Upstream nextest #656/#668 describes an older Windows spawned-process/Tokio shutdown hang, fixed in 0.9.44; this repository already pins 0.9.144, so that historical fix alone does not justify changing our pin or claiming a repair. Source check also shows the Windows workflow does not invoke the Linux compiler-cache setup changed in PR464. Separate process-exit, inherited-pipe and runner-drain timestamps are still needed. No process-name kill, deadline increase or summary-to-success shortcut was introduced. Primary upstream reference: https://github.com/nextest-rs/nextest/issues/656.
Author
Member

PR#468 final-head instrumentation now locates the delay more precisely on native Windows run6704 (UI1779), job50112:

  • Ordinary summary at12:12:47.7355712Z: all4,848 tests pass.
  • PowerShell immediately records native-command return: NEXTEST phase=ci boundary=returned utc=2026-10-25T12:12:47.7591161Z elapsed_ms=831645 exit_code=0 powershell_pid=53900 (the log received it at12:12:47.7647194Z).
  • Next step records NEXTEST phase=ci-timing boundary=invoke utc=2026-10-25T12:20:26.4289473Z powershell_pid=53096.
  • The timing Cargo build reports2.33s at12:20:29.5859382Z; its nextest run begins12:20:37.2002664Z.

Measured return→next invocation gap:458.6698312seconds (7m38.67s). This run therefore establishes that cargo/nextest returned to PowerShell promptly after its summary; the long interval is AFTER that return and BEFORE the next step's invocation. The script has only logging and exit after the return marker, preserving the native status. This does not independently measure PowerShell process exit or identify runner/stream-drain behavior; those remain the next boundary to observe. PIDs are shell identities at the recorded instant, not permission to act on them later without creation-time checks. No test was skipped, deadline raised, process killed, or summary substituted for workflow success. #435 remains open.

PR#468 final-head instrumentation now locates the delay more precisely on native Windows run6704 (UI1779), job50112: - Ordinary summary at12:12:47.7355712Z: all4,848 tests pass. - PowerShell immediately records native-command return: `NEXTEST phase=ci boundary=returned utc=2026-10-25T12:12:47.7591161Z elapsed_ms=831645 exit_code=0 powershell_pid=53900` (the log received it at12:12:47.7647194Z). - Next step records `NEXTEST phase=ci-timing boundary=invoke utc=2026-10-25T12:20:26.4289473Z powershell_pid=53096`. - The timing Cargo build reports2.33s at12:20:29.5859382Z; its nextest run begins12:20:37.2002664Z. Measured return→next invocation gap:458.6698312seconds (7m38.67s). This run therefore establishes that cargo/nextest returned to PowerShell promptly after its summary; the long interval is AFTER that return and BEFORE the next step's invocation. The script has only logging and `exit` after the return marker, preserving the native status. This does not independently measure PowerShell process exit or identify runner/stream-drain behavior; those remain the next boundary to observe. PIDs are shell identities at the recorded instant, not permission to act on them later without creation-time checks. No test was skipped, deadline raised, process killed, or summary substituted for workflow success. #435 remains open.
Author
Member

Fresh boundary evidence from the successful v0.41.38 source qualification, Windows run 2235 / job 61303, head f54a875877:

  • 19:52:52.948 UTC: nextest summary, 5,135 passed.
  • 19:52:52.975 UTC: the existing PowerShell instrumentation records NEXTEST phase=ci boundary=returned ... exit_code=0 (PowerShell PID 6048).
  • 20:00:24.022 UTC: the timing step records its invocation (PowerShell PID 39368), about 451 seconds later.
  • Both test phases and the complete job ultimately succeeded; timing ran all 43 selected tests.

This narrows the observed gap: the native nextest command had already returned to PowerShell successfully before the delay. The owner of the subsequent step/runner delay remains unmeasured. No process-name kills, summary-based forced exits, deadline changes, or skipped gates were used. The ticket remains open pending the narrow runner/process-lifetime reproduction and fix.

Fresh boundary evidence from the successful v0.41.38 source qualification, Windows run 2235 / job 61303, head f54a875877f3ac2623015cc9176f5b143a89db5c: - 19:52:52.948 UTC: nextest summary, 5,135 passed. - 19:52:52.975 UTC: the existing PowerShell instrumentation records `NEXTEST phase=ci boundary=returned ... exit_code=0` (PowerShell PID 6048). - 20:00:24.022 UTC: the timing step records its invocation (PowerShell PID 39368), about 451 seconds later. - Both test phases and the complete job ultimately succeeded; timing ran all 43 selected tests. This narrows the observed gap: the native nextest command had already returned to PowerShell successfully before the delay. The owner of the subsequent step/runner delay remains unmeasured. No process-name kills, summary-based forced exits, deadline changes, or skipped gates were used. The ticket remains open pending the narrow runner/process-lifetime reproduction and fix.
buildagent changed title from CI: Windows nextest returns about 7.5 minutes after its passing summary to CI: Windows step advances about 7.5 minutes after native nextest returns 2026-11-07 05:03:12 +01:00
Author
Member

The final 0.41.42 source head 401f32a6 passed native Windows qualification, but this delay remains. Job 62913 records the passing 5,155-test summary at 2026-11-07T02:50:11.1905009Z, native return with exit 0 at 02:50:11.2559585Z (PowerShell PID 17680), and the timing invocation at 02:58:01.2483332Z (PID 41252): 469.992375 seconds after native return. There were 16 leaky tests; that count does not identify the delay owner.

The subsequent timing, doctest, installer and shipped-profile smoke checks passed. Kernel PowerShell-exit and runner pipe-owner observations remain unmeasured, so this is still open and is not claimed fixed in 0.41.42. Local receipt: /tmp/cosi-435-final-401f32a-boundaries.json.

The final 0.41.42 source head `401f32a6` passed [native Windows qualification](https://git.h-dv.de/h-dv/code-index/actions/runs/2338), but this delay remains. Job 62913 records the passing 5,155-test summary at `2026-11-07T02:50:11.1905009Z`, native return with exit 0 at `02:50:11.2559585Z` (PowerShell PID 17680), and the timing invocation at `02:58:01.2483332Z` (PID 41252): **469.992375 seconds after native return**. There were 16 leaky tests; that count does not identify the delay owner. The subsequent timing, doctest, installer and shipped-profile smoke checks passed. Kernel PowerShell-exit and runner pipe-owner observations remain unmeasured, so this is still open and is not claimed fixed in 0.41.42. Local receipt: `/tmp/cosi-435-final-401f32a-boundaries.json`.
Author
Member

Measured the native-return gap on the last green Windows run (job 64795): nextest returned exit 0 at 10:48:31.5102981Z, while the next step began 466.623 seconds later.

The dispatch-only, build-free diagnostic isolates the mechanism. Run 8255: inherited and merely redirected descendants each delayed the next step ~8.53s after PowerShell exited; ShellExecute and CreateProcess(bInheritHandles=false) controls took ~0.41s while their descendants were still running. This proves an inherited-handle delay; it does not identify the particular descendant in the historical field run.

Candidate fix 8519eb52 launches nextest through ShellExecute, streams the output file as raw bytes, and propagates the native exit code. The actual helper passed Windows run 8257/job 64903: native exits 0 and 37, exact arguments, inherited environment, working directory, paths containing spaces/%/& and split UTF-8 output. The next step began in 0.291s with both background descendants still alive. Both completed independently; no process was killed. Local CI path/release contracts also pass (59+44 tests).

Keeping this open until the combined release qualification runs the real full native suite with the helper. No gate, timeout or success criterion was weakened.

Measured the native-return gap on the last green Windows run (job 64795): nextest returned exit 0 at 10:48:31.5102981Z, while the next step began 466.623 seconds later. The dispatch-only, build-free diagnostic isolates the mechanism. Run 8255: inherited and merely redirected descendants each delayed the next step ~8.53s after PowerShell exited; ShellExecute and CreateProcess(bInheritHandles=false) controls took ~0.41s while their descendants were still running. This proves an inherited-handle delay; it does not identify the particular descendant in the historical field run. Candidate fix 8519eb52 launches nextest through ShellExecute, streams the output file as raw bytes, and propagates the native exit code. The actual helper passed Windows run 8257/job 64903: native exits 0 and 37, exact arguments, inherited environment, working directory, paths containing spaces/%/& and split UTF-8 output. The next step began in 0.291s with both background descendants still alive. Both completed independently; no process was killed. Local CI path/release contracts also pass (59+44 tests). Keeping this open until the combined release qualification runs the real full native suite with the helper. No gate, timeout or success criterion was weakened.
Author
Member

Fixed and merged in PR #509, merge93e46879 (tree-identical to qualified source49edd903).

The actual Windows helper now starts nextest without inheriting the runner's handles, forwards raw output and preserves the native process status. Existing timing and downstream checks remain mandatory. Final qualification8262/job64953 passed5,185 workspace tests,43 timing tests and all downstream checks. Nextest returned18:18:04.0806143Z; the next timing step began18:18:04.4545611Z:0.3739468s, versus466.623s on the previous green job. An independent preceding full run measured0.2837696s and also completed successfully.

The diagnostic exercised native exits0/37, arguments/environment/cwd, metacharacter paths, split UTF-8, and still-running background descendants; removing failure/timing/status propagation is rejected by executed contract mutations. The measured step-delay defect is closed.

Fixed and merged in PR #509, merge93e46879 (tree-identical to qualified source49edd903). The actual Windows helper now starts nextest without inheriting the runner's handles, forwards raw output and preserves the native process status. Existing timing and downstream checks remain mandatory. Final qualification8262/job64953 passed5,185 workspace tests,43 timing tests and all downstream checks. Nextest returned18:18:04.0806143Z; the next timing step began18:18:04.4545611Z:0.3739468s, versus466.623s on the previous green job. An independent preceding full run measured0.2837696s and also completed successfully. The diagnostic exercised native exits0/37, arguments/environment/cwd, metacharacter paths, split UTF-8, and still-running background descendants; removing failure/timing/status propagation is rejected by executed contract mutations. The measured step-delay defect is closed.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#435
No description provided.