Reference resolution is one non-resumable transaction, so one-shot queries after an upgrade can never converge #301

Closed
opened 2026-09-26 01:12:43 +02:00 by buildagent · 4 comments
Member

After a schema upgrade that invalidates code caches (m0070, m0071), a code-index query with no daemon running never finishes reconciling. Each call answers warming_up, and then the server is killed.

What lane H measured (I075, on a django copy upgraded from schema 69, killing code-index index every 15 s):

  • File reparse IS durable across kills. Files awaiting reparse went 2,970 → 1,570 → 370 → 0 over three windows.
  • Reference resolution is NOT durable. It runs as a single transaction (about 32 s on django), and each kill discards it. Twelve consecutive 15 s windows never completed it.
  • A one-shot call's window is about 2 s, and the transaction grows with repository size. Per-call progress therefore cannot converge unless the resolver itself becomes resumable.

What v0.32.1 ships: disclosure only. A one-shot warming_up carries reconcile_pending { converges_on_retry: false, … } and names the remedy: run code-index index once, or start the daemon. This is honest, but hooks such as the blast-radius hook still get no answer until a user acts.

Proposed fix: make resolution resumable. Commit per batch or per file group, with a generation-safe cursor, so each short-lived process makes durable progress. This must keep the generation-gating invariants (generation_build, generation_isolation) and the reader-epoch guarantees. A partially resolved generation must never be visible as complete.

🤖 Generated with Claude Code

After a schema upgrade that invalidates code caches (m0070, m0071), a `code-index query` with no daemon running never finishes reconciling. Each call answers `warming_up`, and then the server is killed. **What lane H measured** (I075, on a django copy upgraded from schema 69, killing `code-index index` every 15 s): - File reparse IS durable across kills. Files awaiting reparse went 2,970 → 1,570 → 370 → 0 over three windows. - Reference resolution is NOT durable. It runs as a single transaction (about 32 s on django), and each kill discards it. Twelve consecutive 15 s windows never completed it. - A one-shot call's window is about 2 s, and the transaction grows with repository size. Per-call progress therefore cannot converge unless the resolver itself becomes resumable. **What v0.32.1 ships:** disclosure only. A one-shot `warming_up` carries `reconcile_pending { converges_on_retry: false, … }` and names the remedy: run `code-index index` once, or start the daemon. This is honest, but hooks such as the blast-radius hook still get no answer until a user acts. **Proposed fix:** make resolution resumable. Commit per batch or per file group, with a generation-safe cursor, so each short-lived process makes durable progress. This must keep the generation-gating invariants (`generation_build`, `generation_isolation`) and the reader-epoch guarantees. A partially resolved generation must never be visible as complete. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Author
Member

Still current as of 0bcdac1 (checked by Phase 1 lane R on 2026-09-26): resolution is one transaction in apply_resolution_observed, so a one-shot code-index query answers warming_up with converges_on_retry: false.

Two options:

  • Full fix: batch the roughly 6,000-line Full resolver by file, with a durable cursor and pools rebuilt per batch. This puts baseline equivalence and the cost ratchet at risk.
  • Cheaper: a one-shot query that finds a pending reconcile and no daemon starts a detached code-index index to finish it, and says so in its reply. Retries then converge without a daemon.

This is a prerequisite before any language flips to a package in Phase 3.

Still current as of 0bcdac1 (checked by Phase 1 lane R on 2026-09-26): resolution is one transaction in `apply_resolution_observed`, so a one-shot `code-index query` answers `warming_up` with `converges_on_retry: false`. Two options: - **Full fix:** batch the roughly 6,000-line Full resolver by file, with a durable cursor and pools rebuilt per batch. This puts baseline equivalence and the cost ratchet at risk. - **Cheaper:** a one-shot query that finds a pending reconcile and no daemon starts a detached `code-index index` to finish it, and says so in its reply. Retries then converge without a daemon. This is a prerequisite before any language flips to a package in Phase 3.
Author
Member

The nightly cargo test on master (run 5643, master at 0bcdac1) failed in query_cli::the_cli_log_level_reaches_the_spawned_server. It hit this issue directly: the one-shot code-index query answered warming_up ("the index is still reconciling and retrying this query will not finish it"), so the test never reached the premise it asserts. The same test passed in every PR run of this code and in a local full run, so it is timing-dependent: it fails when the runner is loaded, and the nightly runs heavier jobs alongside it.

Until this issue is fixed, any test that runs a one-shot query right after indexing can flake like this. Fixing this issue removes the flake class; a separate test-side workaround would only hide it.

The nightly `cargo test` on master (run 5643, master at 0bcdac1) failed in `query_cli::the_cli_log_level_reaches_the_spawned_server`. It hit this issue directly: the one-shot `code-index query` answered `warming_up` ("the index is still reconciling and retrying this query will not finish it"), so the test never reached the premise it asserts. The same test passed in every PR run of this code and in a local full run, so it is timing-dependent: it fails when the runner is loaded, and the nightly runs heavier jobs alongside it. Until this issue is fixed, any test that runs a one-shot query right after indexing can flake like this. Fixing this issue removes the flake class; a separate test-side workaround would only hide it.
Author
Member

Additional local reproduction of the existing one-shot log-level test's startup premise failure (the test comments already name this issue and earlier loaded-runner occurrences). During PR#439 validation, the full default-profile CLI nextest run completed446 cases:445 passed, and query_cli::the_cli_log_level_reaches_the_spawned_server failed after10.329s because search_symbols returned warming_up/reconcile_pending for its explicit temporary --db. The latin1 warning had been emitted, so logging had begun; the committed queue was not observed before the existing one-shot deadline. This test and its query path are unchanged by PR439 (the source edits are plugin CLI mutation locking/notice).

An isolated single-thread invocation of the same fixture, with unchanged source and deadlines, passed in17.390s for its two queries. Load sensitivity is consistent with the fixture's documented history; this does not establish the precise startup bottleneck. The original failure remains reported in the PR validation. No retry policy, runtime deadline, assertion or CI gate was weakened. Normal CI qualification of the corrected PR head remains required.

Additional local reproduction of the existing one-shot log-level test's startup premise failure (the test comments already name this issue and earlier loaded-runner occurrences). During PR#439 validation, the full default-profile CLI nextest run completed446 cases:445 passed, and `query_cli::the_cli_log_level_reaches_the_spawned_server` failed after10.329s because search_symbols returned warming_up/reconcile_pending for its explicit temporary --db. The latin1 warning had been emitted, so logging had begun; the committed queue was not observed before the existing one-shot deadline. This test and its query path are unchanged by PR439 (the source edits are plugin CLI mutation locking/notice). An isolated single-thread invocation of the same fixture, with unchanged source and deadlines, passed in17.390s for its two queries. Load sensitivity is consistent with the fixture's documented history; this does not establish the precise startup bottleneck. The original failure remains reported in the PR validation. No retry policy, runtime deadline, assertion or CI gate was weakened. Normal CI qualification of the corrected PR head remains required.
Author
Member

Current-source audit on v0.41.45 confirms this is implemented in crates/indexer/src/resolve_resume.rs: generation/content identity-fenced durable batches commit references and aggregates together; pending work remains visible until completion. All 14 resolve_resume unit tests passed freshly, including cancellation/kill/reopen convergence, snapshot changes and retained/pending generation isolation. The actual no_daemon_retries_resume_resolution_to_the_atomic_projection MCP test also passed: it requires more than one process and an observed durable queue, then checks exact equality with the uninterrupted projection and consistent ref counts after every committed batch. Existing v0.41.45 Linux/native-Windows qualification and release are green. This issue is ready to close as delivered, without another resolver implementation.

Current-source audit on v0.41.45 confirms this is implemented in crates/indexer/src/resolve_resume.rs: generation/content identity-fenced durable batches commit references and aggregates together; pending work remains visible until completion. All 14 resolve_resume unit tests passed freshly, including cancellation/kill/reopen convergence, snapshot changes and retained/pending generation isolation. The actual no_daemon_retries_resume_resolution_to_the_atomic_projection MCP test also passed: it requires more than one process and an observed durable queue, then checks exact equality with the uninterrupted projection and consistent ref counts after every committed batch. Existing v0.41.45 Linux/native-Windows qualification and release are green. This issue is ready to close as delivered, without another resolver implementation.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
h-dv/code-index#301
No description provided.