CI cache: source-only save/restore verified; failed-upload reporting remains #408

Closed
opened 2026-10-22 20:31:13 +02:00 by buildagent · 9 comments
Member

Audit on v0.41.45 (2026-11-08): source-only cache mitigation and actual save-to-subsequent-restore were already verified (44,012,327-byte archive; see 2026-10-25 evidence). Do not schedule another multi-GB cache-splitting fix. The remaining historical concern is misleading saved-key output after failed cache persistence; that backend/action behavior was not proved fixed. The separate post-nextest delay remains #435 and is not caused by a cache save between the test steps.


Measured in Windows run 1472, job 45745, for PR #407.

All test steps finished before the final cache save. At 17:50:40 UTC, actions/cache invoked Git's tar.exe ... -z. Compression finished at 18:17:31 UTC: 26m51s for 9,909,546,189 bytes (~9,450 MB). At 18:17:51, commitCache reported HTTP 500. The action then printed Cache saved with key, despite that failure. The job stayed green, as cache failure is nonfatal, but spent over 27 minutes on a cache whose persistence was not established.

The workspace report measured ~45.8 GB under target. Three agents on the Windows runner machine independently checked read-only: standalone zstd was absent from their checked user/machine PATHs and common locations. The runner account's own user PATH was not measured. The actual job command establishes gzip use regardless of that remaining environment uncertainty. No machine or process state was changed.

Fix the cache path with measured evidence: make an appropriate compressor available to the runner or reduce the cache population without discarding reusable compiled units blindly; verify successful upload and a subsequent restore. Keep cache persistence failures distinguishable from a successful save. Acceptance should measure total Windows lane time, cache size/compression time and actual restore success. Do not weaken test gates or delete unrelated workspace/runner data.

Audit on v0.41.45 (2026-11-08): source-only cache mitigation and actual save-to-subsequent-restore were already verified (44,012,327-byte archive; see 2026-10-25 evidence). Do not schedule another multi-GB cache-splitting fix. The remaining historical concern is misleading saved-key output after failed cache persistence; that backend/action behavior was not proved fixed. The separate post-nextest delay remains #435 and is not caused by a cache save between the test steps. --- Measured in [Windows run 1472](https://git.h-dv.de/h-dv/code-index/actions/runs/1472), job 45745, for PR #407. All test steps finished before the final cache save. At 17:50:40 UTC, actions/cache invoked Git's `tar.exe ... -z`. Compression finished at 18:17:31 UTC: 26m51s for 9,909,546,189 bytes (~9,450 MB). At 18:17:51, `commitCache` reported HTTP 500. The action then printed `Cache saved with key`, despite that failure. The job stayed green, as cache failure is nonfatal, but spent over 27 minutes on a cache whose persistence was not established. The workspace report measured ~45.8 GB under target. Three agents on the Windows runner machine independently checked read-only: standalone zstd was absent from their checked user/machine PATHs and common locations. The runner account's own user PATH was not measured. The actual job command establishes gzip use regardless of that remaining environment uncertainty. No machine or process state was changed. Fix the cache path with measured evidence: make an appropriate compressor available to the runner or reduce the cache population without discarding reusable compiled units blindly; verify successful upload and a subsequent restore. Keep cache persistence failures distinguishable from a successful save. Acceptance should measure total Windows lane time, cache size/compression time and actual restore success. Do not weaken test gates or delete unrelated workspace/runner data.
Author
Member

Current final-head Windows qualification for PR #411 confirms the CI runner itself uses gzip, independently of the partners' interactive PATHs. Run https://git.h-dv.de/h-dv/code-index/actions/runs/1496, job 46019 logs: 20:49:33 UTC CACHE COMPRESSION: PASS - ...Git/usr/bin/tar.exe and gzip round-tripped the payload; 20:49:34 UTC Cache not found for the exact windows-cargo-v2 key and both fallback prefixes. Compilation then starts cold. This supports addressing the runner's compression tool/cache-save path rather than repeatedly asking the partners to inspect their shell PATHs. No runner changes or CI gate changes were made as part of PR #411.

Current final-head Windows qualification for PR #411 confirms the CI runner itself uses gzip, independently of the partners' interactive PATHs. Run https://git.h-dv.de/h-dv/code-index/actions/runs/1496, job 46019 logs: 20:49:33 UTC `CACHE COMPRESSION: PASS - ...Git/usr/bin/tar.exe and gzip round-tripped the payload`; 20:49:34 UTC `Cache not found` for the exact windows-cargo-v2 key and both fallback prefixes. Compilation then starts cold. This supports addressing the runner's compression tool/cache-save path rather than repeatedly asking the partners to inspect their shell PATHs. No runner changes or CI gate changes were made as part of PR #411.
Author
Member

Mitigation is now included in PR #411, head 8623dd47: Windows CI and release builds cache only registry/index, registry/cache (.crate archives) and git/db. The new shared source-only prefix has no fallback to the old target-containing caches. Removing target/ also removes the restored-build-script repair step. Every non-cache step and job-level gate is unchanged by parsed YAML comparison; 48 CI-routing tests and the release build-group check pass locally.

All three partners independently observed run 1496 after validation in the actions/cache post-save subtree using Git tar -z, started at 21:32 UTC; no build/test process remained under the runner. The obsolete save is superseded by normal PR-update concurrency, without manual process changes.

The final-head rerun will measure the new cache cost. This mitigates the large archive I/O/compression overhead and avoids restoring copied compiler outputs. It does not claim to repair the cache service's earlier CommitCache HTTP 500 or its misleading saved message. Existing host-local Cargo state can still help; remote-cache absence never changes qualification.

Mitigation is now included in PR #411, head 8623dd47: Windows CI and release builds cache only registry/index, registry/cache (.crate archives) and git/db. The new shared source-only prefix has no fallback to the old target-containing caches. Removing target/ also removes the restored-build-script repair step. Every non-cache step and job-level gate is unchanged by parsed YAML comparison; 48 CI-routing tests and the release build-group check pass locally. All three partners independently observed run 1496 after validation in the actions/cache post-save subtree using Git tar -z, started at 21:32 UTC; no build/test process remained under the runner. The obsolete save is superseded by normal PR-update concurrency, without manual process changes. The final-head rerun will measure the new cache cost. This mitigates the large archive I/O/compression overhead and avoids restoring copied compiler outputs. It does not claim to repair the cache service's earlier CommitCache HTTP 500 or its misleading saved message. Existing host-local Cargo state can still help; remote-cache absence never changes qualification.
Author
Member

The source-only cache mitigation in #411 is merged. Final native Windows qualification run1504/job46115 passed all main4,806 tests,27 timing tests and remaining validation. Its source cache saved successfully at1,609,774bytes: post-save began22:42:01.806UTC and completed22:42:02.743UTC (<1second), compared with the earlier9.9GB/26m51s target archive. Native setup confirmed the custom effective CARGO_HOME; the new namespace cannot restore old target archives. The cache backend's earlier HTTP500/misleading success behavior remains a separate issue; this passing save does not establish that backend defect is fixed.

The source-only cache mitigation in #411 is merged. Final native Windows qualification run1504/job46115 passed all main4,806 tests,27 timing tests and remaining validation. Its source cache saved successfully at1,609,774bytes: post-save began22:42:01.806UTC and completed22:42:02.743UTC (<1second), compared with the earlier9.9GB/26m51s target archive. Native setup confirmed the custom effective CARGO_HOME; the new namespace cannot restore old target archives. The cache backend's earlier HTTP500/misleading success behavior remains a separate issue; this passing save does not establish that backend defect is fixed.
Author
Member

Additional observation from release run 1520, Windows job 46311: effective CARGO_HOME is correctly C:\ci\cargo, but at 22:52:56 UTC actions/cache reported no match for either its new version key or the windows-cargo-sources-v1- fallback, despite the earlier PR job's successful-save message. This may reflect cache ref visibility/scoping or the backend behavior; it is not yet diagnosed. The build proceeds normally without a cache. The <1-second/1.6-MB save measurement establishes reduced archive overhead, not successful reuse across PR/tag refs. Keep the backend/reuse portion of this issue open.

Additional observation from release run 1520, Windows job 46311: effective CARGO_HOME is correctly C:\ci\cargo, but at 22:52:56 UTC actions/cache reported no match for either its new version key or the windows-cargo-sources-v1- fallback, despite the earlier PR job's successful-save message. This may reflect cache ref visibility/scoping or the backend behavior; it is not yet diagnosed. The build proceeds normally without a cache. The <1-second/1.6-MB save measurement establishes reduced archive overhead, not successful reuse across PR/tag refs. Keep the backend/reuse portion of this issue open.
Author
Member

Final release measurement: v0.41.10 run 1520/job 46311 completed successfully. Its post-cache save began 23:12:35.794 UTC and completed 23:12:37.965 UTC (about 2.2 seconds), archive 39,737,683 bytes. No HTTP500 appeared in that bounded post-cache log. Thus both qualified CI (1.6 MB, <1 s) and release (39.7 MB, ~2.2 s) avoid the former multi-gigabyte/tens-of-minutes save cost. The earlier cross-ref restore miss remains undiagnosed; backend/reuse is still open. Stable release and independent artifact/upgrade verification are complete.

Final release measurement: v0.41.10 run 1520/job 46311 completed successfully. Its post-cache save began 23:12:35.794 UTC and completed 23:12:37.965 UTC (about 2.2 seconds), archive 39,737,683 bytes. No HTTP500 appeared in that bounded post-cache log. Thus both qualified CI (1.6 MB, <1 s) and release (39.7 MB, ~2.2 s) avoid the former multi-gigabyte/tens-of-minutes save cost. The earlier cross-ref restore miss remains undiagnosed; backend/reuse is still open. Stable release and independent artifact/upgrade verification are complete.
Author
Member

Additional native Windows qualification evidence from PR #420, separate from the archive-upload fix already shipped:

  • Job 46716 (head a3c9f56b) passed 4,823 normal tests at 05:11:30.913 UTC, then produced no new log line until 05:19:15.694 UTC: a 7m44.8s gap. Compiler/linker warning output resumed before the separate timing run, and the whole native workflow eventually passed.
  • Current native run, job 46764 / head 49068746, passed all 4,824 normal tests at 05:59:51.420 UTC. Its log still ended at that summary at 06:08 UTC, while Forgejo reported the job running.

The workflow invokes cargo nextest run separately for ci and ci-timing, with no source change between them. The elapsed gap is directly measured; whether it is process teardown, Cargo/metadata work, target reuse or runner/logging behavior is not established. Investigate that transition before claiming Windows compiler/cache reuse is solved. This does not justify skipping timing tests or changing their production bounds; #408 remains open.

Additional native Windows qualification evidence from PR #420, separate from the archive-upload fix already shipped: - Job 46716 (head a3c9f56b) passed 4,823 normal tests at 05:11:30.913 UTC, then produced no new log line until 05:19:15.694 UTC: a 7m44.8s gap. Compiler/linker warning output resumed before the separate timing run, and the whole native workflow eventually passed. - [Current native run](https://git.h-dv.de/h-dv/code-index/actions/runs/1556), job 46764 / head 49068746, passed all 4,824 normal tests at 05:59:51.420 UTC. Its log still ended at that summary at 06:08 UTC, while Forgejo reported the job running. The workflow invokes `cargo nextest run` separately for `ci` and `ci-timing`, with no source change between them. The elapsed gap is directly measured; whether it is process teardown, Cargo/metadata work, target reuse or runner/logging behavior is not established. Investigate that transition before claiming Windows compiler/cache reuse is solved. This does not justify skipping timing tests or changing their production bounds; #408 remains open.
Author
Member

A second measured Windows inter-phase delay on PR #426, final head 36b492c91a, run 6407/job 46935:

  • Main nextest summary: 10:17:53.859 UTC, all 4,829 tests passed (498.683s).
  • Next timing invocation reports Finished test profile ... in 2.27s at 10:25:40.497 UTC.
  • Timing nextest starts at 10:25:48.867 UTC, 27 tests.

Summary-to-next-start gap: 475.008s (7m55s), consistent with the earlier 7m45s observation. The two nextest commands are adjacent workflow steps in ci-windows.yml (513–532); there is no cache save between them. This is not evidence that cache compression causes this particular gap. Instrument command-exit / PowerShell / runner boundaries to separate time after nextest's summary from time before the next command starts. The timing invocation's reported 2.27s compile time does not account for the gap. No checks were waived or rerun to hide it.

A second measured Windows inter-phase delay on PR #426, final head 36b492c91abae5c67e3a693326650171fc140a53, run 6407/job 46935: - Main nextest summary: 10:17:53.859 UTC, all 4,829 tests passed (498.683s). - Next timing invocation reports `Finished test profile ... in 2.27s` at 10:25:40.497 UTC. - Timing nextest starts at 10:25:48.867 UTC, 27 tests. Summary-to-next-start gap: 475.008s (7m55s), consistent with the earlier 7m45s observation. The two nextest commands are adjacent workflow steps in ci-windows.yml (513–532); there is no cache save between them. This is not evidence that cache compression causes this particular gap. Instrument command-exit / PowerShell / runner boundaries to separate time after nextest's summary from time before the next command starts. The timing invocation's reported 2.27s compile time does not account for the gap. No checks were waived or rerun to hide it.
Author
Member

The missing save→restore evidence is now measured, including across release-tag and PR refs. v0.41.20 release run6699 (UI1774), Windows job50058, created the source-only archive at10:16:43.498 UTC, measured44,012,327bytes, reported Cache saved successfully at10:16:45.391, and saved key windows-cargo-sources-v1-47d586f501347e7261acfa6175815428315bb5d51531f0c48f5fe858c2b509b0. PR#468 run6700 (UI1775), job50064, later restored that exact key and exact44,012,327-byte archive: extraction11:03:37.587→11:03:38.846 UTC, then Cache restored successfully/Cache restored from key. This establishes persistence and actual subsequent reuse for the source-only mitigation, not just an optimistic save message. The earlier multi-GB archive/compression overhead is avoided. No new source change or CI waiver was needed for this audit. The backend/action behavior of misleadingly printing a saved key after its historical HTTP500 has not been repaired or independently re-exercised; keep that remaining behavior distinct from the now-verified archive reduction and reuse.

The missing save→restore evidence is now measured, including across release-tag and PR refs. v0.41.20 release run6699 (UI1774), Windows job50058, created the source-only archive at10:16:43.498 UTC, measured44,012,327bytes, reported `Cache saved successfully` at10:16:45.391, and saved key `windows-cargo-sources-v1-47d586f501347e7261acfa6175815428315bb5d51531f0c48f5fe858c2b509b0`. PR#468 run6700 (UI1775), job50064, later restored that exact key and exact44,012,327-byte archive: extraction11:03:37.587→11:03:38.846 UTC, then `Cache restored successfully`/`Cache restored from key`. This establishes persistence and actual subsequent reuse for the source-only mitigation, not just an optimistic save message. The earlier multi-GB archive/compression overhead is avoided. No new source change or CI waiver was needed for this audit. The backend/action behavior of misleadingly printing a saved key after its historical HTTP500 has not been repaired or independently re-exercised; keep that remaining behavior distinct from the now-verified archive reduction and reuse.
buildagent changed title from CI: Windows cache save spends 27 minutes compressing 9.9 GB, then fails with HTTP 500 to CI cache: source-only save/restore verified; failed-upload reporting remains 2026-11-08 13:54:47 +01:00
Author
Member

Triage 2026-11-09: Done: the Windows cache holds sources only, on a v1 key, with continue-on-error; save and restore were verified on 2026-10-25. The remaining 'Cache saved after HTTP 500' is upstream action behaviour costing about 2 s and a cold restore at worst. Closing.

Triage 2026-11-09: Done: the Windows cache holds sources only, on a v1 key, with continue-on-error; save and restore were verified on 2026-10-25. The remaining 'Cache saved after HTTP 500' is upstream action behaviour costing about 2 s and a cold restore at worst. Closing.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#408
No description provided.