Concurrent daemonless plugin updates race activation and restore stale approval records #438
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
h-dv/code-index#438
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
W100 and a second partner independently report this on published core 0.41.15 (
065c8864), Windows, using C:/tools/code-index.exe. No daemon was running. Two overlappingplugin update <package> --grant requested --yes --root <same-root>commands updated C# and XAML.The C# command passed C1, then failed with
building the generation: sqlite error: UNIQUE constraint failed: index 'idx_plugin_generations_one_in_flight', followed byupdate_reverted. The XAML command waited for the SQLite writer and continued a foreground full rebuild. While XAML still ran, status showed the prior C#/XAML approvals, an active generation, an obsolete deleting candidate, and a new ready candidate. The approval rollback rewrites the entire record from a snapshot, so concurrent independent commands may also overwrite another command's approval; this consequence is a source-supported risk pending a controlled reproduction.Current build::request already uses BEGIN IMMEDIATE and a final in-transaction census; do not assume the old allocation race explains this report. Investigate the complete approval/read-plan/write/build/promote/rollback operation and different desired activation identities, including replacement of another live candidate. Existing same-identity two-builder coverage does not grade two different package updates.
Expected: serialize project approval mutations or refuse a competing command clearly, before it reads/mutates a stale record. Preserve all successful unrelated approvals, active facts, grants, candidate attribution and rollback safety. Keep independent projects and read-only status available. Use a process-scoped OS lock with crash release if an existing lock cannot cover this boundary; avoid holding a SQLite transaction throughout extraction. Grade two real CLI processes, failure restoration, and crash/retry. Also disclose before the expensive work that daemonless activation runs a foreground reindex and may take time. Do not weaken admission, runtime deadlines or generation constraints.
Temporary operator workaround: run updates for a project sequentially; no forced GC, DB deletion or interruption of the existing update was requested.
Additional field evidence from W100: both CLI processes were confirmed 0.41.15 (
065c8864). XAML logged retirement of the live candidate at 08:21:25Z (activation_identity_changed). The C# process later acquired the generation-state-change lock and hit the uniqueness constraint at 08:22:28Z. Itsupdate_revertedrewrote the whole approval record at 08:22:38Z, removing the newer XAML approval already written by the still-running updater. Old C#/XAML digests remained; neither new digest remained. Thus stale-record restoration is observed, not only a source-derived risk. Partners are leaving the current process untouched and will report its final promotion outcome.A per-project OS mutation guard is implemented locally across the complete CLI snapshot/approve/build/promote/restore operation. It refuses a competing mutator before any approval read, keeps status and check-only commands available, uses no long SQLite transaction, and releases on process exit. A deterministic test with two different package updaters pauses the first after its approval write behind an external SQLite writer: the second refuses without fetching its catalog or changing approvals, status still works, the first then completes and retrying the second preserves both upgraded approvals. Initial targeted run passed (2.94 s). Not yet merged or released; broader validation and review remain.
W100 reports the final outcome at08:53Z: the concurrent XAML updater exited0 and promoted generation5 after the C# rollback had erased its approval. The active generation reported22,131 contributions,393,555 symbols,2,148,236 refs and0 refusals; the approval record remained the older XAML0.3.0/C#0.1.6 snapshot (mtime08:22:38Z), with neither new digest. Thus the stale rollback led to active-generation/approval divergence. A subsequent startup reverting to the recorded activation is a source-supported expectation, not yet a field observation.
Recovery guidance uses public commands only: once no updater/candidate is running, retry the operator-authorized updates strictly sequentially and verify each final approval/active identity. No forced collection or manual DB edits requested. Original logs retained by the partner.
PR#439(head20ce41d131fcc5f7d8a147e1fe75715a201b423b) prevents concurrent CLI mutators from entering the snapshot/approval/build/promotion/restore operation. All73 relevant local tests, all-target CLI Clippy, formatting and whitespace checks pass; bypassing the guard made the bounded two-process regression fail. Package CI6546 and pilot6547 are green; normal core6545 and Windows6544 qualification are still running. No merge/release yet.
Released and independently verified in stable v0.41.16, tagged at qualified version head18151656 after normal merges #439/#440. All ten release jobs passed, including native Windows archive smoke and signed channel publication.
Independent verification used published assets and compiled trust anchors: both catalog projections/signatures, all four archive hashes, all seven bundled packages in each archive, and all35 prior channel package versions across11 packages were retained. Package versions are unchanged.
The shipped .15→.16 upgrade retained the same database inode, exact approvals/package digests, and active generation4/digest while preserving the C#/XAML private-handler bridge, plain Name fields, binding/unrelated-file decoys and Svelte diagnostics. A published-binary concurrency witness held SQLite's writer, observed
plugin_mutation_busyfor a second package command via a root alias/different TMP settings, preserved the first command's exact approval snapshot, kept status available, and allowed another project sharing the same installation/store to update. After releasing the writer and retrying sequentially, C#0.1.7 and XAML0.3.1 were both in the ACTIVE generation. Killing/reaping an updater released the OS lock and the next update succeeded.Runtime MCP checks used isolated
--no-daemon; normal daemon/Windows paths are separately covered by CI. The first operational witness waited on digest membership although that digest already existed as shadowed; it was corrected to wait for selectedapproved_seqand rerun in a fresh project. No production change or assertion weakening was needed.Local receipts:
/tmp/cosi-shipped-04116-final/bundle-verification.jsonand/tmp/cosi-04116-published-runtime-proof-v2/report.json. Upgrade the core binaries together and let any older updater finish before another command.