CI cache: source-only save/restore verified; failed-upload reporting remains #408
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
h-dv/code-index#408
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Audit on v0.41.45 (2026-11-08): source-only cache mitigation and actual save-to-subsequent-restore were already verified (44,012,327-byte archive; see 2026-10-25 evidence). Do not schedule another multi-GB cache-splitting fix. The remaining historical concern is misleading saved-key output after failed cache persistence; that backend/action behavior was not proved fixed. The separate post-nextest delay remains #435 and is not caused by a cache save between the test steps.
Measured in Windows run 1472, job 45745, for PR #407.
All test steps finished before the final cache save. At 17:50:40 UTC, actions/cache invoked Git's
tar.exe ... -z. Compression finished at 18:17:31 UTC: 26m51s for 9,909,546,189 bytes (~9,450 MB). At 18:17:51,commitCachereported HTTP 500. The action then printedCache saved with key, despite that failure. The job stayed green, as cache failure is nonfatal, but spent over 27 minutes on a cache whose persistence was not established.The workspace report measured ~45.8 GB under target. Three agents on the Windows runner machine independently checked read-only: standalone zstd was absent from their checked user/machine PATHs and common locations. The runner account's own user PATH was not measured. The actual job command establishes gzip use regardless of that remaining environment uncertainty. No machine or process state was changed.
Fix the cache path with measured evidence: make an appropriate compressor available to the runner or reduce the cache population without discarding reusable compiled units blindly; verify successful upload and a subsequent restore. Keep cache persistence failures distinguishable from a successful save. Acceptance should measure total Windows lane time, cache size/compression time and actual restore success. Do not weaken test gates or delete unrelated workspace/runner data.
Current final-head Windows qualification for PR #411 confirms the CI runner itself uses gzip, independently of the partners' interactive PATHs. Run https://git.h-dv.de/h-dv/code-index/actions/runs/1496, job 46019 logs: 20:49:33 UTC
CACHE COMPRESSION: PASS - ...Git/usr/bin/tar.exe and gzip round-tripped the payload; 20:49:34 UTCCache not foundfor the exact windows-cargo-v2 key and both fallback prefixes. Compilation then starts cold. This supports addressing the runner's compression tool/cache-save path rather than repeatedly asking the partners to inspect their shell PATHs. No runner changes or CI gate changes were made as part of PR #411.Mitigation is now included in PR #411, head
8623dd47: Windows CI and release builds cache only registry/index, registry/cache (.crate archives) and git/db. The new shared source-only prefix has no fallback to the old target-containing caches. Removing target/ also removes the restored-build-script repair step. Every non-cache step and job-level gate is unchanged by parsed YAML comparison; 48 CI-routing tests and the release build-group check pass locally.All three partners independently observed run 1496 after validation in the actions/cache post-save subtree using Git tar -z, started at 21:32 UTC; no build/test process remained under the runner. The obsolete save is superseded by normal PR-update concurrency, without manual process changes.
The final-head rerun will measure the new cache cost. This mitigates the large archive I/O/compression overhead and avoids restoring copied compiler outputs. It does not claim to repair the cache service's earlier CommitCache HTTP 500 or its misleading saved message. Existing host-local Cargo state can still help; remote-cache absence never changes qualification.
The source-only cache mitigation in #411 is merged. Final native Windows qualification run1504/job46115 passed all main4,806 tests,27 timing tests and remaining validation. Its source cache saved successfully at1,609,774bytes: post-save began22:42:01.806UTC and completed22:42:02.743UTC (<1second), compared with the earlier9.9GB/26m51s target archive. Native setup confirmed the custom effective CARGO_HOME; the new namespace cannot restore old target archives. The cache backend's earlier HTTP500/misleading success behavior remains a separate issue; this passing save does not establish that backend defect is fixed.
Additional observation from release run 1520, Windows job 46311: effective CARGO_HOME is correctly C:\ci\cargo, but at 22:52:56 UTC actions/cache reported no match for either its new version key or the windows-cargo-sources-v1- fallback, despite the earlier PR job's successful-save message. This may reflect cache ref visibility/scoping or the backend behavior; it is not yet diagnosed. The build proceeds normally without a cache. The <1-second/1.6-MB save measurement establishes reduced archive overhead, not successful reuse across PR/tag refs. Keep the backend/reuse portion of this issue open.
Final release measurement: v0.41.10 run 1520/job 46311 completed successfully. Its post-cache save began 23:12:35.794 UTC and completed 23:12:37.965 UTC (about 2.2 seconds), archive 39,737,683 bytes. No HTTP500 appeared in that bounded post-cache log. Thus both qualified CI (1.6 MB, <1 s) and release (39.7 MB, ~2.2 s) avoid the former multi-gigabyte/tens-of-minutes save cost. The earlier cross-ref restore miss remains undiagnosed; backend/reuse is still open. Stable release and independent artifact/upgrade verification are complete.
Additional native Windows qualification evidence from PR #420, separate from the archive-upload fix already shipped:
a3c9f56b) passed 4,823 normal tests at 05:11:30.913 UTC, then produced no new log line until 05:19:15.694 UTC: a 7m44.8s gap. Compiler/linker warning output resumed before the separate timing run, and the whole native workflow eventually passed.49068746, passed all 4,824 normal tests at 05:59:51.420 UTC. Its log still ended at that summary at 06:08 UTC, while Forgejo reported the job running.The workflow invokes
cargo nextest runseparately forciandci-timing, with no source change between them. The elapsed gap is directly measured; whether it is process teardown, Cargo/metadata work, target reuse or runner/logging behavior is not established. Investigate that transition before claiming Windows compiler/cache reuse is solved. This does not justify skipping timing tests or changing their production bounds; #408 remains open.A second measured Windows inter-phase delay on PR #426, final head
36b492c91a, run 6407/job 46935:Finished test profile ... in 2.27sat 10:25:40.497 UTC.Summary-to-next-start gap: 475.008s (7m55s), consistent with the earlier 7m45s observation. The two nextest commands are adjacent workflow steps in ci-windows.yml (513–532); there is no cache save between them. This is not evidence that cache compression causes this particular gap. Instrument command-exit / PowerShell / runner boundaries to separate time after nextest's summary from time before the next command starts. The timing invocation's reported 2.27s compile time does not account for the gap. No checks were waived or rerun to hide it.
The missing save→restore evidence is now measured, including across release-tag and PR refs. v0.41.20 release run6699 (UI1774), Windows job50058, created the source-only archive at10:16:43.498 UTC, measured44,012,327bytes, reported
Cache saved successfullyat10:16:45.391, and saved keywindows-cargo-sources-v1-47d586f501347e7261acfa6175815428315bb5d51531f0c48f5fe858c2b509b0. PR#468 run6700 (UI1775), job50064, later restored that exact key and exact44,012,327-byte archive: extraction11:03:37.587→11:03:38.846 UTC, thenCache restored successfully/Cache restored from key. This establishes persistence and actual subsequent reuse for the source-only mitigation, not just an optimistic save message. The earlier multi-GB archive/compression overhead is avoided. No new source change or CI waiver was needed for this audit. The backend/action behavior of misleadingly printing a saved key after its historical HTTP500 has not been repaired or independently re-exercised; keep that remaining behavior distinct from the now-verified archive reduction and reuse.CI: Windows cache save spends 27 minutes compressing 9.9 GB, then fails with HTTP 500to CI cache: source-only save/restore verified; failed-upload reporting remainsTriage 2026-11-09: Done: the Windows cache holds sources only, on a v1 key, with continue-on-error; save and restore were verified on 2026-10-25. The remaining 'Cache saved after HTTP 500' is upstream action behaviour costing about 2 s and a cold restore at worst. Closing.