Decouple plugin extraction invalidation from core release and worker build labels #412
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
h-dv/code-index#412
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
A core-only upgrade currently reparses unchanged plugin-owned files, even when every signed package byte and ABI remains unchanged. This undermines the runtime/plugin separation and makes small operational fixes costly on large projects.
Reproduced with shipped binaries in the same isolated project/operator home: core 0.41.8 (
4734318a) -> 0.41.9 (44fec715). For the isolation fixture, the 0.41.8 executable directory was composed with the identical signed 0.41.9 package bundle used by the second phase, so Python was already 0.1.4 in BOTH phases. This was a composed test fixture, not a claim about the original 0.41.8 archive. The Python digest remained sha256:bf7ed47e1ce6f70331edd8213c2038f825ee7b4ade7bfec678cf33f8cd18f992, layouts stayed ready/withheld=0, and no project configuration or explicit package-update command was involved. Active generation moved 2 -> 3; all three unchanged files were parsed again. Activation digest moved 3b2dc3b1603333ca1a6135400707da811d67004ef9810ebcad995c5a874b40dd -> 83bd814f067488b160ed1a4f2d6e2ce28d8152326f73f71b117f3116a834e815.Source: activation::identity_of stores CARGO_PKG_VERSION as HostIdentity::host_version; activation_digest hashes it unconditionally. The worker identity_line also includes its release and build identifier. dirty_domain maps host identity changes to total Work::Reparse, and startup cannot recover an older host identity so it falls back to a full build. The host-version rationale is carrying the compiled-in extractor set; shipped plugin-only configurations need a deliberate independent invalidation contract.
Investigate a stable extraction-semantics identity covering the actual fact ABI, parser/engine versions and semantic engine flags, embedded dispatch/source-map semantics, and any remaining builtin extractor identity. Keep release/build labels for diagnostics. Do not simply remove identity inputs: changes that alter emitted facts must still invalidate extraction; resolver-only changes must receive the required resolve; incompatible old identities need a conservative transition. Verify both unchanged-package core upgrades and real parser/ABI/extraction-semantic changes.
Related #410 handles proving the old package activation for bounded package upgrades. The Windows field case upgraded both core and Python, so a Python-only cause is not established. PR #411 fixes reconnect behavior and intentionally does not change this invalidation policy.
Independent read-only design review identifies why simply dropping release/build labels would be unsafe. I verified the worker's native tree-sitter dependency, concrete fuel/memory/response budgets, and host fact-to-row mapping anchors. The runtime identity must account for native parser semantics, engine/version/flags, fact ABI, validation and extraction limits, embedded/source-map semantics, host fact mapping and writer behavior, and remaining builtin feature/grammar identity. Several of those currently ride implicitly on a release/build change rather than appearing explicitly in the activation identity.
A small candidate design is an explicit host fact-semantics epoch plus a stable worker extraction fingerprint, with release/build labels kept for diagnostics. A meaningful fixed facts-to-stored-rows fixture should guard the epoch: cover tag handling, positions, qualifiers, test/member roles, visibility/grant gating, embedded processing and relevant refusals; fact-changing mutations should force an intentional epoch decision. Parser/engine dependency and budget changes must also change the fingerprint automatically or fail a guard. Unknown legacy identity remains conservative.
For the initial migration, prefer one explicit conservative rebuild over maintaining two digest formulas across multiple releases unless measurements justify that extra machinery. Future equivalent core/worker builds should then retain facts, while real extraction-semantic changes still rebuild. The review also flagged builtin feature-set differences and UNIVERSAL_SKIP withdrawal behavior for verification; those observations are not yet established by a dedicated reproduction. No implementation is included in PR #411.
Field confirmation after the operator upgraded all three Windows partners to published 0.41.10 (
d2597cd3), 2026-10-22 23:38–23:41 UTC. Each independently confirmed its actual runtime version. Both large projects (20,495 and 22,124 files) kept active generation 4 serving while generation 5 rebuilt; logs explicitly report unrecoverable recorded activation due to host/worker identity movement and full re-extraction. Package bytes were unchanged. The smaller Python project reached steady state after initial reconcile.Bounded read-only observations found one daemon per root, no redundant daemon, no reconnect timeout/internal_error, and correct representative symbol/text results. During rebuilding, large-project symbol queries were 2–179 ms and text 15–19 ms; health 0–20 ms. Aggregate stats on the 22,124-file project still took 24,587 ms. These are partner observations, not a controlled benchmark; steady-state completion for both large projects is still pending. Existing Python 0.1.3 shadowed grants were left untouched. This confirms the recovery release works in this sample while the core-upgrade rebuild cost remains unresolved.
Post-release field reports for v0.41.13: E3, E4, 16.0 and local-ai-server observed a full re-extraction after upgrading from v0.41.12 with project configuration unchanged. The local-ai-server report measured 1,626 reparsed files and zero carried files. These are partner observations, not a controlled equivalence benchmark.
Source review confirms the current fingerprint conservatively includes every file below the Host role's indexer/core/plugins/abi/package/plugin-supervisor source directories; Worker also includes package sources (crates/extraction_fingerprint.rs). This release changed approval.rs, confirm.rs and package/builtin.rs, so a changed identity is expected under that boundary even though package bytes are unchanged. This does not demonstrate that version-only normalization regressed. It does demonstrate that ordinary runtime fixes can still trigger expensive rebuilds.
Any follow-up narrowing must preserve fact-validation, grants, mapping/writer behavior, parser/engine and budget invalidation. Simply excluding these files or retaining facts across every patch release would be unsafe. A version-only published upgrade remains a separate field witness; no such result is claimed here. E4 independently confirms the v0.41.13 refusal-disclosure correction without database deletion.
Follow-up field readiness checks after the v0.41.13 rebuilds: all four reporting roots reached ready without database deletion or new reported errors. E3: 23,991 reparsed / 0 carried, ~25.3 minutes startup-to-ready, zero parse errors. 16.0: 24,920 reparsed / 0 carried, ~25.8 minutes, zero parse errors; reported reference resolution alone took about 12.5 minutes. E4: 12,221 reparsed / 0 carried, ~1.8 minutes, zero parse errors (one pre-existing partial source remains). local-ai-server: 1,626 reparsed / 0 carried, ~59 seconds. These are independent partner log observations rather than controlled performance measurements. Both large startups completed under the 30-minute idle threshold, so neither establishes a field witness for #415. The conservative host-source invalidation cost remains as described above.
Published v0.41.15→v0.41.16 field witness from W100/E3/E4: four real project roots upgraded without re-extraction, schema reset, or approval/digest changes. All partners verified matching core binaries at
18151656.8e73d1ffretained. Bound at 12:44:42Z, opened at 12:44:56Z, index complete at 12:45:02Z. C#38194755and XAML0c65e958are live.bOk_Clickbridge still resolves from XAML:77 to C# target_id888980, unchanged from before the upgrade.These are field receipts, not a general proof for future source changes. Independent published-binary isolated upgrade also retained the exact active generation/digest and DB inode: #438 (comment)