Decouple plugin extraction invalidation from core release and worker build labels #412

Closed
opened 2026-10-22 23:01:55 +02:00 by buildagent · 5 comments
Member

A core-only upgrade currently reparses unchanged plugin-owned files, even when every signed package byte and ABI remains unchanged. This undermines the runtime/plugin separation and makes small operational fixes costly on large projects.

Reproduced with shipped binaries in the same isolated project/operator home: core 0.41.8 (4734318a) -> 0.41.9 (44fec715). For the isolation fixture, the 0.41.8 executable directory was composed with the identical signed 0.41.9 package bundle used by the second phase, so Python was already 0.1.4 in BOTH phases. This was a composed test fixture, not a claim about the original 0.41.8 archive. The Python digest remained sha256:bf7ed47e1ce6f70331edd8213c2038f825ee7b4ade7bfec678cf33f8cd18f992, layouts stayed ready/withheld=0, and no project configuration or explicit package-update command was involved. Active generation moved 2 -> 3; all three unchanged files were parsed again. Activation digest moved 3b2dc3b1603333ca1a6135400707da811d67004ef9810ebcad995c5a874b40dd -> 83bd814f067488b160ed1a4f2d6e2ce28d8152326f73f71b117f3116a834e815.

Source: activation::identity_of stores CARGO_PKG_VERSION as HostIdentity::host_version; activation_digest hashes it unconditionally. The worker identity_line also includes its release and build identifier. dirty_domain maps host identity changes to total Work::Reparse, and startup cannot recover an older host identity so it falls back to a full build. The host-version rationale is carrying the compiled-in extractor set; shipped plugin-only configurations need a deliberate independent invalidation contract.

Investigate a stable extraction-semantics identity covering the actual fact ABI, parser/engine versions and semantic engine flags, embedded dispatch/source-map semantics, and any remaining builtin extractor identity. Keep release/build labels for diagnostics. Do not simply remove identity inputs: changes that alter emitted facts must still invalidate extraction; resolver-only changes must receive the required resolve; incompatible old identities need a conservative transition. Verify both unchanged-package core upgrades and real parser/ABI/extraction-semantic changes.

Related #410 handles proving the old package activation for bounded package upgrades. The Windows field case upgraded both core and Python, so a Python-only cause is not established. PR #411 fixes reconnect behavior and intentionally does not change this invalidation policy.

A core-only upgrade currently reparses unchanged plugin-owned files, even when every signed package byte and ABI remains unchanged. This undermines the runtime/plugin separation and makes small operational fixes costly on large projects. Reproduced with shipped binaries in the same isolated project/operator home: core 0.41.8 (4734318a) -> 0.41.9 (44fec715). For the isolation fixture, the 0.41.8 executable directory was composed with the identical signed 0.41.9 package bundle used by the second phase, so Python was already 0.1.4 in BOTH phases. This was a composed test fixture, not a claim about the original 0.41.8 archive. The Python digest remained sha256:bf7ed47e1ce6f70331edd8213c2038f825ee7b4ade7bfec678cf33f8cd18f992, layouts stayed ready/withheld=0, and no project configuration or explicit package-update command was involved. Active generation moved 2 -> 3; all three unchanged files were parsed again. Activation digest moved 3b2dc3b1603333ca1a6135400707da811d67004ef9810ebcad995c5a874b40dd -> 83bd814f067488b160ed1a4f2d6e2ce28d8152326f73f71b117f3116a834e815. Source: activation::identity_of stores CARGO_PKG_VERSION as HostIdentity::host_version; activation_digest hashes it unconditionally. The worker identity_line also includes its release and build identifier. dirty_domain maps host identity changes to total Work::Reparse, and startup cannot recover an older host identity so it falls back to a full build. The host-version rationale is carrying the compiled-in extractor set; shipped plugin-only configurations need a deliberate independent invalidation contract. Investigate a stable extraction-semantics identity covering the actual fact ABI, parser/engine versions and semantic engine flags, embedded dispatch/source-map semantics, and any remaining builtin extractor identity. Keep release/build labels for diagnostics. Do not simply remove identity inputs: changes that alter emitted facts must still invalidate extraction; resolver-only changes must receive the required resolve; incompatible old identities need a conservative transition. Verify both unchanged-package core upgrades and real parser/ABI/extraction-semantic changes. Related #410 handles proving the old package activation for bounded package upgrades. The Windows field case upgraded both core and Python, so a Python-only cause is not established. PR #411 fixes reconnect behavior and intentionally does not change this invalidation policy.
Author
Member

Independent read-only design review identifies why simply dropping release/build labels would be unsafe. I verified the worker's native tree-sitter dependency, concrete fuel/memory/response budgets, and host fact-to-row mapping anchors. The runtime identity must account for native parser semantics, engine/version/flags, fact ABI, validation and extraction limits, embedded/source-map semantics, host fact mapping and writer behavior, and remaining builtin feature/grammar identity. Several of those currently ride implicitly on a release/build change rather than appearing explicitly in the activation identity.

A small candidate design is an explicit host fact-semantics epoch plus a stable worker extraction fingerprint, with release/build labels kept for diagnostics. A meaningful fixed facts-to-stored-rows fixture should guard the epoch: cover tag handling, positions, qualifiers, test/member roles, visibility/grant gating, embedded processing and relevant refusals; fact-changing mutations should force an intentional epoch decision. Parser/engine dependency and budget changes must also change the fingerprint automatically or fail a guard. Unknown legacy identity remains conservative.

For the initial migration, prefer one explicit conservative rebuild over maintaining two digest formulas across multiple releases unless measurements justify that extra machinery. Future equivalent core/worker builds should then retain facts, while real extraction-semantic changes still rebuild. The review also flagged builtin feature-set differences and UNIVERSAL_SKIP withdrawal behavior for verification; those observations are not yet established by a dedicated reproduction. No implementation is included in PR #411.

Independent read-only design review identifies why simply dropping release/build labels would be unsafe. I verified the worker's native tree-sitter dependency, concrete fuel/memory/response budgets, and host fact-to-row mapping anchors. The runtime identity must account for native parser semantics, engine/version/flags, fact ABI, validation and extraction limits, embedded/source-map semantics, host fact mapping and writer behavior, and remaining builtin feature/grammar identity. Several of those currently ride implicitly on a release/build change rather than appearing explicitly in the activation identity. A small candidate design is an explicit host fact-semantics epoch plus a stable worker extraction fingerprint, with release/build labels kept for diagnostics. A meaningful fixed facts-to-stored-rows fixture should guard the epoch: cover tag handling, positions, qualifiers, test/member roles, visibility/grant gating, embedded processing and relevant refusals; fact-changing mutations should force an intentional epoch decision. Parser/engine dependency and budget changes must also change the fingerprint automatically or fail a guard. Unknown legacy identity remains conservative. For the initial migration, prefer one explicit conservative rebuild over maintaining two digest formulas across multiple releases unless measurements justify that extra machinery. Future equivalent core/worker builds should then retain facts, while real extraction-semantic changes still rebuild. The review also flagged builtin feature-set differences and UNIVERSAL_SKIP withdrawal behavior for verification; those observations are not yet established by a dedicated reproduction. No implementation is included in PR #411.
Author
Member

Field confirmation after the operator upgraded all three Windows partners to published 0.41.10 (d2597cd3), 2026-10-22 23:38–23:41 UTC. Each independently confirmed its actual runtime version. Both large projects (20,495 and 22,124 files) kept active generation 4 serving while generation 5 rebuilt; logs explicitly report unrecoverable recorded activation due to host/worker identity movement and full re-extraction. Package bytes were unchanged. The smaller Python project reached steady state after initial reconcile.

Bounded read-only observations found one daemon per root, no redundant daemon, no reconnect timeout/internal_error, and correct representative symbol/text results. During rebuilding, large-project symbol queries were 2–179 ms and text 15–19 ms; health 0–20 ms. Aggregate stats on the 22,124-file project still took 24,587 ms. These are partner observations, not a controlled benchmark; steady-state completion for both large projects is still pending. Existing Python 0.1.3 shadowed grants were left untouched. This confirms the recovery release works in this sample while the core-upgrade rebuild cost remains unresolved.

Field confirmation after the operator upgraded all three Windows partners to published 0.41.10 (d2597cd3), 2026-10-22 23:38–23:41 UTC. Each independently confirmed its actual runtime version. Both large projects (20,495 and 22,124 files) kept active generation 4 serving while generation 5 rebuilt; logs explicitly report unrecoverable recorded activation due to host/worker identity movement and full re-extraction. Package bytes were unchanged. The smaller Python project reached steady state after initial reconcile. Bounded read-only observations found one daemon per root, no redundant daemon, no reconnect timeout/internal_error, and correct representative symbol/text results. During rebuilding, large-project symbol queries were 2–179 ms and text 15–19 ms; health 0–20 ms. Aggregate stats on the 22,124-file project still took 24,587 ms. These are partner observations, not a controlled benchmark; steady-state completion for both large projects is still pending. Existing Python 0.1.3 shadowed grants were left untouched. This confirms the recovery release works in this sample while the core-upgrade rebuild cost remains unresolved.
Author
Member

Post-release field reports for v0.41.13: E3, E4, 16.0 and local-ai-server observed a full re-extraction after upgrading from v0.41.12 with project configuration unchanged. The local-ai-server report measured 1,626 reparsed files and zero carried files. These are partner observations, not a controlled equivalence benchmark.

Source review confirms the current fingerprint conservatively includes every file below the Host role's indexer/core/plugins/abi/package/plugin-supervisor source directories; Worker also includes package sources (crates/extraction_fingerprint.rs). This release changed approval.rs, confirm.rs and package/builtin.rs, so a changed identity is expected under that boundary even though package bytes are unchanged. This does not demonstrate that version-only normalization regressed. It does demonstrate that ordinary runtime fixes can still trigger expensive rebuilds.

Any follow-up narrowing must preserve fact-validation, grants, mapping/writer behavior, parser/engine and budget invalidation. Simply excluding these files or retaining facts across every patch release would be unsafe. A version-only published upgrade remains a separate field witness; no such result is claimed here. E4 independently confirms the v0.41.13 refusal-disclosure correction without database deletion.

Post-release field reports for v0.41.13: E3, E4, 16.0 and local-ai-server observed a full re-extraction after upgrading from v0.41.12 with project configuration unchanged. The local-ai-server report measured 1,626 reparsed files and zero carried files. These are partner observations, not a controlled equivalence benchmark. Source review confirms the current fingerprint conservatively includes every file below the Host role's indexer/core/plugins/abi/package/plugin-supervisor source directories; Worker also includes package sources (crates/extraction_fingerprint.rs). This release changed approval.rs, confirm.rs and package/builtin.rs, so a changed identity is expected under that boundary even though package bytes are unchanged. This does not demonstrate that version-only normalization regressed. It does demonstrate that ordinary runtime fixes can still trigger expensive rebuilds. Any follow-up narrowing must preserve fact-validation, grants, mapping/writer behavior, parser/engine and budget invalidation. Simply excluding these files or retaining facts across every patch release would be unsafe. A version-only published upgrade remains a separate field witness; no such result is claimed here. E4 independently confirms the v0.41.13 refusal-disclosure correction without database deletion.
Author
Member

Follow-up field readiness checks after the v0.41.13 rebuilds: all four reporting roots reached ready without database deletion or new reported errors. E3: 23,991 reparsed / 0 carried, ~25.3 minutes startup-to-ready, zero parse errors. 16.0: 24,920 reparsed / 0 carried, ~25.8 minutes, zero parse errors; reported reference resolution alone took about 12.5 minutes. E4: 12,221 reparsed / 0 carried, ~1.8 minutes, zero parse errors (one pre-existing partial source remains). local-ai-server: 1,626 reparsed / 0 carried, ~59 seconds. These are independent partner log observations rather than controlled performance measurements. Both large startups completed under the 30-minute idle threshold, so neither establishes a field witness for #415. The conservative host-source invalidation cost remains as described above.

Follow-up field readiness checks after the v0.41.13 rebuilds: all four reporting roots reached ready without database deletion or new reported errors. E3: 23,991 reparsed / 0 carried, ~25.3 minutes startup-to-ready, zero parse errors. 16.0: 24,920 reparsed / 0 carried, ~25.8 minutes, zero parse errors; reported reference resolution alone took about 12.5 minutes. E4: 12,221 reparsed / 0 carried, ~1.8 minutes, zero parse errors (one pre-existing partial source remains). local-ai-server: 1,626 reparsed / 0 carried, ~59 seconds. These are independent partner log observations rather than controlled performance measurements. Both large startups completed under the 30-minute idle threshold, so neither establishes a field witness for #415. The conservative host-source invalidation cost remains as described above.
Author
Member

Published v0.41.15→v0.41.16 field witness from W100/E3/E4: four real project roots upgraded without re-extraction, schema reset, or approval/digest changes. All partners verified matching core binaries at 18151656.

  • local-ai-server: 1,626 files seen, 0 parsed, 1,623 stat-skipped; initial reconciliation completed.
  • 16.0: 24,920 seen, 0 parsed, 22,131 stat-skipped, 0 parse errors. Generation 7 / activation 8e73d1ff retained. Bound at 12:44:42Z, opened at 12:44:56Z, index complete at 12:45:02Z. C# 38194755 and XAML 0c65e958 are live.
  • e3: 23,991 seen, 0 parsed, 20,497 stat-skipped, 0 parse errors. Opened at 12:44:33Z, complete at 12:50:10Z. The private bOk_Click bridge still resolves from XAML:77 to C# target_id 888980, unchanged from before the upgrade.
  • E4: 12,221 seen, 0 parsed, 4,141 stat-skipped, 0 parse errors. Opened at 12:45:16Z, complete at 12:45:21Z. Generation 5 is active and owes nothing. All eight approved package IDs were checked sequentially through the named updater; each exited 0 / up to date. MCP overview: 4,141 files, 111,576 symbols, with the one expected malformed dataset partial.

These are field receipts, not a general proof for future source changes. Independent published-binary isolated upgrade also retained the exact active generation/digest and DB inode: #438 (comment)

Published v0.41.15→v0.41.16 field witness from W100/E3/E4: four real project roots upgraded without re-extraction, schema reset, or approval/digest changes. All partners verified matching core binaries at `18151656`. - local-ai-server: 1,626 files seen, 0 parsed, 1,623 stat-skipped; initial reconciliation completed. - 16.0: 24,920 seen, 0 parsed, 22,131 stat-skipped, 0 parse errors. Generation 7 / activation `8e73d1ff` retained. Bound at 12:44:42Z, opened at 12:44:56Z, index complete at 12:45:02Z. C# `38194755` and XAML `0c65e958` are live. - e3: 23,991 seen, 0 parsed, 20,497 stat-skipped, 0 parse errors. Opened at 12:44:33Z, complete at 12:50:10Z. The private `bOk_Click` bridge still resolves from XAML:77 to C# target_id `888980`, unchanged from before the upgrade. - E4: 12,221 seen, 0 parsed, 4,141 stat-skipped, 0 parse errors. Opened at 12:45:16Z, complete at 12:45:21Z. Generation 5 is active and owes nothing. All eight approved package IDs were checked sequentially through the named updater; each exited 0 / up to date. MCP overview: 4,141 files, 111,576 symbols, with the one expected malformed dataset partial. These are field receipts, not a general proof for future source changes. Independent published-binary isolated upgrade also retained the exact active generation/digest and DB inode: https://git.h-dv.de/h-dv/code-index/issues/438#issuecomment-10307
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#412
No description provided.