h-dv/code-index

Watch

Fork 0

After a daemon restart a session can refuse every filter as "daemon predates the filter", and calls during the restart are refused instead of waiting #584

New issue

Closed

opened 2026-11-13 12:54:04 +01:00 by buildagent · 5 comments

buildagent commented

2026-11-13 12:54:04 +01:00

Member

Reported by a customer on v0.41.50 (client and daemon both v0.41.50, five language packages, plugin doctor all OK), 2026-11-13.

What the user saw

search_symbols with path_glob was refused: "the running daemon predates the search_symbols.path_glob filter and would silently ignore it … Stop it (or wait out its idle timeout) and retry". Both binaries were the same release. The refusal stayed for the rest of the session. search_text with path_glob kept working.
While the daemon restarted after its idle timeout, several agents of the same session got daemon_restarting and "existing owner … is still starting". They fell back to grep.

Part A: a failed capability probe is remembered as "this daemon has no filters" (cause known)

RpcIndex::honors (crates/daemon/src/rpc_index.rs) asks the daemon for its filter capabilities once per handle and keeps the answer in a OnceCell. If that one request fails for ANY reason, the error is logged and an EMPTY set is stored:

a daemon that is restarting or still starting when the first filtered call arrives gives exactly such an error;
from then on every guarded filter of that MCP session is refused with the "daemon predates the filter" text, which names a wrong cause and a remedy that does not help (stopping the daemon does not clear the client's memory);
the stored set also survives a reconnect or respawn, so it can describe a daemon process that no longer exists.

Only restarting the MCP server clears it. Present in every release that has the capability probe, including v0.41.52.

Expected:

only a real answer is remembered: the capability list, or "unknown method" from a daemon that really is older;
the remembered answer belongs to one daemon identity and is dropped when the handle connects to another daemon;
a probe that failed for a transient reason gives the caller the transient refusal (retryable), never the "predates" text.

Part B: calls made during a daemon restart are refused instead of answered a moment later (to measure)

After an idle timeout the next call starts a new daemon. Calls from other clients of the same project that arrive in that window are refused (daemon_restarting, "existing owner … is still starting"). An agent treats a refusal as "tool broken" and leaves for grep, even when the refusal says retryable.

To establish before changing anything:

how long the window is on v0.41.50 and on v0.41.52 with five packages installed (v0.41.50 carried the startup regression of #581, so the window was several times longer than it is now);
what each client path does today: which of them wait, for how long, and which refuse at once;
whether a bounded wait inside the client (the caller gets the answer late instead of a refusal, and a refusal only after the bound, saying how long was waited) is the right default for the calls that need the daemon.

Acceptance

Test for A that can fail: the first filtered call meets a starting or restarting daemon; a later filtered call on the same handle is honoured once the daemon is up. Mutation run. Second test: after a respawn the capabilities are asked again.
The refusal text for a transient probe failure does not say "predates".
B: the measured windows and the per-path behaviour are posted here; the change, if any, is covered by a test with two clients and one restart, on both platforms' transport.

Workaround until released: restart the MCP server (the client session) once the daemon is up.

Reported by a customer on v0.41.50 (client and daemon both v0.41.50, five language packages, `plugin doctor` all OK), 2026-11-13. ## What the user saw 1. `search_symbols` with `path_glob` was refused: "the running daemon predates the `search_symbols.path_glob` filter and would silently ignore it … Stop it (or wait out its idle timeout) and retry". Both binaries were the same release. The refusal stayed for the rest of the session. `search_text` with `path_glob` kept working. 2. While the daemon restarted after its idle timeout, several agents of the same session got `daemon_restarting` and "existing owner … is still starting". They fell back to grep. ## Part A: a failed capability probe is remembered as "this daemon has no filters" (cause known) `RpcIndex::honors` (`crates/daemon/src/rpc_index.rs`) asks the daemon for its filter capabilities once per handle and keeps the answer in a `OnceCell`. If that one request fails for ANY reason, the error is logged and an EMPTY set is stored: - a daemon that is restarting or still starting when the first filtered call arrives gives exactly such an error; - from then on every guarded filter of that MCP session is refused with the "daemon predates the filter" text, which names a wrong cause and a remedy that does not help (stopping the daemon does not clear the client's memory); - the stored set also survives a reconnect or respawn, so it can describe a daemon process that no longer exists. Only restarting the MCP server clears it. Present in every release that has the capability probe, including v0.41.52. Expected: - only a real answer is remembered: the capability list, or "unknown method" from a daemon that really is older; - the remembered answer belongs to one daemon identity and is dropped when the handle connects to another daemon; - a probe that failed for a transient reason gives the caller the transient refusal (retryable), never the "predates" text. ## Part B: calls made during a daemon restart are refused instead of answered a moment later (to measure) After an idle timeout the next call starts a new daemon. Calls from other clients of the same project that arrive in that window are refused (`daemon_restarting`, "existing owner … is still starting"). An agent treats a refusal as "tool broken" and leaves for grep, even when the refusal says retryable. To establish before changing anything: - how long the window is on v0.41.50 and on v0.41.52 with five packages installed (v0.41.50 carried the startup regression of #581, so the window was several times longer than it is now); - what each client path does today: which of them wait, for how long, and which refuse at once; - whether a bounded wait inside the client (the caller gets the answer late instead of a refusal, and a refusal only after the bound, saying how long was waited) is the right default for the calls that need the daemon. ## Acceptance - Test for A that can fail: the first filtered call meets a starting or restarting daemon; a later filtered call on the same handle is honoured once the daemon is up. Mutation run. Second test: after a respawn the capabilities are asked again. - The refusal text for a transient probe failure does not say "predates". - B: the measured windows and the per-path behaviour are posted here; the change, if any, is covered by a test with two clients and one restart, on both platforms' transport. Workaround until released: restart the MCP server (the client session) once the daemon is up.

buildagent added the

labels

2026-11-13 12:54:04 +01:00

buildagent commented

2026-11-13 13:07:19 +01:00

Author

Member

First data point for Part B, measured on Linux with the release smoke steps (fresh project, seven languages enabled, no daemon running, a copy of a real store with 45 installed packages):

release	first query after start	second query
v0.41.50	19.6 s, refused `index_incomplete` (state `warming_up`)	17.6 s, answered while `reconciling`
v0.41.52	1.7 s, answered	1.8 s, answered

So on the reporter's release a start with a filled package store took long enough for several calls to be refused; that part is the startup regression of #581, fixed in v0.41.52. This is the one-shot CLI path, not yet the daemon restart with two MCP clients that the report describes; that measurement is part of the lane.

For the reporter: upgrading to v0.41.52 shortens the window; Part A (the remembered "predates the filter" refusal) is not fixed by it, the workaround stays restarting the MCP server.

First data point for Part B, measured on Linux with the release smoke steps (fresh project, seven languages enabled, no daemon running, a copy of a real store with 45 installed packages): | release | first query after start | second query | | :-- | :-- | :-- | | v0.41.50 | 19.6 s, refused `index_incomplete` (state `warming_up`) | 17.6 s, answered while `reconciling` | | v0.41.52 | 1.7 s, answered | 1.8 s, answered | So on the reporter's release a start with a filled package store took long enough for several calls to be refused; that part is the startup regression of #581, fixed in v0.41.52. This is the one-shot CLI path, not yet the daemon restart with two MCP clients that the report describes; that measurement is part of the lane. For the reporter: upgrading to v0.41.52 shortens the window; Part A (the remembered "predates the filter" refusal) is not fixed by it, the workaround stays restarting the MCP server.

buildagent commented

2026-11-13 14:22:51 +01:00

Author

Member

Part A is fixed on the next-train branch (lane r34, merged locally as ab56843c; not released yet).

Reproduced as the report describes it: a guarded call that met a starting owner made every later guarded call on the same handle fail with the "predates the filter" text. Two regression tests failed before the fix.
The capability answer is now remembered only when it is a real answer (the list, or the daemon's own "unknown method"), and it belongs to one daemon identity (pid and token). A reconnect to another daemon, including the first time a token becomes known, drops it. A probe that fails for a transient reason returns that error with its own classification.
The same mistake in other spellings was checked and tightened: every "the daemon is old" verdict now needs the daemon's typed not-found answer for an unknown method, not any error whose text contains those words. This covers invoke_versioned, health, search_symbols_ex, wait_for_startup, the optional readers (resolve_progress, resolve_costs, partial_sources, lang_spellings, lang_census, declared_profiles, read_epoch, path_claims, outline_source) and the exclude_tests refusals of references and callers.
Guards audited (all pass the probe's error on): change_impact.role_filter, resolution_gaps.{exclude_tests,ref_kind,qualifier_captured}, search_symbols.path_glob, file_outline.top_level_only, find_references/find_callers.resolution.
Lead's check: the 13 unit tests of the reconnect module and the 7 reconnect end-to-end tests pass; keeping the remembered answer across a daemon replacement (mutation) fails one unit test and one end-to-end test. The worker reported six further mutations red.

Part B, measured. Restart after a real idle exit, 300 files, five package-owned languages, warm store and index: first answer after 0.21 s and settled after 1.16 s on v0.41.50; 0.24 s and 1.40 s on the branch in the shipped build shape. So a warm restart is short on both, and the long window in the report needs the large package store of #581 (see the figures above: 19.6 s to a refused first query with 45 packages on v0.41.50, 1.7 s on v0.41.52).

What each client path does today while a daemon starts:

path	waits	then
MCP session whose call triggers the start	up to 3 s reconnect, 45 s readiness	`daemon_restarting`, no `retryable` field
another existing MCP session	up to 3 s + 3 s, often no wait at all	`daemon_restarting` ("existing owner … is still starting"), no `retryable` field
newly started MCP server	2 s per tool call, attach continues in the background	`index_incomplete`, `retryable: true`
CLI `query`	10 s shared startup wait	`index_incomplete`; answers from a snapshot when no daemon answers
hook	20 s per query	fails open

The second row is the report's case. A lane is now running for it: a call that finds a positively identified starting owner waits for it, bounded by 10 s and by the call's own budget, and answers; a refusal after that carries retryable: true, the measured wait and the owner's startup phase.

**Part A is fixed on the next-train branch** (lane r34, merged locally as `ab56843c`; not released yet). - Reproduced as the report describes it: a guarded call that met a starting owner made every later guarded call on the same handle fail with the "predates the filter" text. Two regression tests failed before the fix. - The capability answer is now remembered only when it is a real answer (the list, or the daemon's own "unknown method"), and it belongs to one daemon identity (pid and token). A reconnect to another daemon, including the first time a token becomes known, drops it. A probe that fails for a transient reason returns that error with its own classification. - The same mistake in other spellings was checked and tightened: every "the daemon is old" verdict now needs the daemon's typed not-found answer for an unknown method, not any error whose text contains those words. This covers `invoke_versioned`, `health`, `search_symbols_ex`, `wait_for_startup`, the optional readers (`resolve_progress`, `resolve_costs`, `partial_sources`, `lang_spellings`, `lang_census`, `declared_profiles`, `read_epoch`, `path_claims`, `outline_source`) and the `exclude_tests` refusals of references and callers. - Guards audited (all pass the probe's error on): `change_impact.role_filter`, `resolution_gaps.{exclude_tests,ref_kind,qualifier_captured}`, `search_symbols.path_glob`, `file_outline.top_level_only`, `find_references`/`find_callers.resolution`. - Lead's check: the 13 unit tests of the reconnect module and the 7 reconnect end-to-end tests pass; keeping the remembered answer across a daemon replacement (mutation) fails one unit test and one end-to-end test. The worker reported six further mutations red. **Part B, measured.** Restart after a real idle exit, 300 files, five package-owned languages, warm store and index: first answer after 0.21 s and settled after 1.16 s on v0.41.50; 0.24 s and 1.40 s on the branch in the shipped build shape. So a warm restart is short on both, and the long window in the report needs the large package store of #581 (see the figures above: 19.6 s to a refused first query with 45 packages on v0.41.50, 1.7 s on v0.41.52). What each client path does today while a daemon starts: | path | waits | then | | :-- | :-- | :-- | | MCP session whose call triggers the start | up to 3 s reconnect, 45 s readiness | `daemon_restarting`, no `retryable` field | | another existing MCP session | up to 3 s + 3 s, often no wait at all | `daemon_restarting` ("existing owner … is still starting"), no `retryable` field | | newly started MCP server | 2 s per tool call, attach continues in the background | `index_incomplete`, `retryable: true` | | CLI `query` | 10 s shared startup wait | `index_incomplete`; answers from a snapshot when no daemon answers | | hook | 20 s per query | fails open | The second row is the report's case. A lane is now running for it: a call that finds a positively identified starting owner waits for it, bounded by 10 s and by the call's own budget, and answers; a refusal after that carries `retryable: true`, the measured wait and the owner's startup phase.

buildagent commented

2026-11-13 15:16:29 +01:00

Author

Member

Part B is on the next-train branch (lane r38, merged locally as 15eab3e5; not released yet).

Reproduced with two client handles on one project: on the code before the change, the second client's query was refused after 0.01 s while the first client's replacement daemon was starting.
Now the second client waits for that daemon and gets its answer. The wait sits in the respawn guard of an existing client handle and applies only to a positively identified starting owner: the kernel lock is held, the process is alive, the recorded root matches, the capability token is present and the startup heartbeat is fresh. Each poll authenticates and checks the root before a channel is used. It never starts a second daemon. A stale or unidentified owner is handled as before. A cancelled call closes its pending channel.
The bound is 10 s, clamped to the tool call's remaining daemon budget. Basis: warm restarts measured at 1.16 s and 1.40 s to settled.
Every daemon_restarting refusal now carries retryable: true, waited_ms (0 when no wait was made) and startup_phase (the daemon's latest lockfile phase, or unavailable). Example from the renderer: {"error":"daemon_restarting","retryable":true,"waited_ms":80,"startup_phase":"opening index", …}. The lockfile phases describe the time before the listener serves (opening index, resetting index schema, checkpointing write-ahead log); the phases a reader reports in state_detail afterwards are the separate closed list in docs/operations.md.
Covered: Linux TCP at runtime; the Windows code compiles under the windows-gnu clippy and was not exercised at runtime.
Lead's check: 17 unit tests of the reconnect module and 8 reconnect end-to-end tests pass in my run; the refusal registry and the documentation size test pass. The worker reported seven mutations, each red in its intended test.

Unchanged and still as in the table above: a newly started MCP server (2 s, then index_incomplete with retryable: true), the CLI query (10 s shared startup wait) and the hook (fails open).

With Part A and Part B merged, what remains of this ticket is the release.

**Part B is on the next-train branch** (lane r38, merged locally as `15eab3e5`; not released yet). - Reproduced with two client handles on one project: on the code before the change, the second client's query was refused after 0.01 s while the first client's replacement daemon was starting. - Now the second client waits for that daemon and gets its answer. The wait sits in the respawn guard of an existing client handle and applies only to a positively identified starting owner: the kernel lock is held, the process is alive, the recorded root matches, the capability token is present and the startup heartbeat is fresh. Each poll authenticates and checks the root before a channel is used. It never starts a second daemon. A stale or unidentified owner is handled as before. A cancelled call closes its pending channel. - The bound is 10 s, clamped to the tool call's remaining daemon budget. Basis: warm restarts measured at 1.16 s and 1.40 s to settled. - Every `daemon_restarting` refusal now carries `retryable: true`, `waited_ms` (0 when no wait was made) and `startup_phase` (the daemon's latest lockfile phase, or `unavailable`). Example from the renderer: `{"error":"daemon_restarting","retryable":true,"waited_ms":80,"startup_phase":"opening index", …}`. The lockfile phases describe the time before the listener serves (`opening index`, `resetting index schema`, `checkpointing write-ahead log`); the phases a reader reports in `state_detail` afterwards are the separate closed list in `docs/operations.md`. - Covered: Linux TCP at runtime; the Windows code compiles under the windows-gnu clippy and was not exercised at runtime. - Lead's check: 17 unit tests of the reconnect module and 8 reconnect end-to-end tests pass in my run; the refusal registry and the documentation size test pass. The worker reported seven mutations, each red in its intended test. Unchanged and still as in the table above: a newly started MCP server (2 s, then `index_incomplete` with `retryable: true`), the CLI `query` (10 s shared startup wait) and the hook (fails open). With Part A and Part B merged, what remains of this ticket is the release.

buildagent commented

2026-11-13 20:13:20 +01:00

Author

Member

A gap in Part B found and closed before release (lane r44, merged locally as cf2d3ee4).

A full daemon-suite run under load failed once in daemon_death_mcp_e2e::a_daemon_killed_mid_call_still_answers_that_call: the call that was in flight when the daemon was killed came back as internal_error ("existing owner … is still starting. No replacement was started").

Cause, traced. My first guess (a freshly spawned daemon that has not published its identity yet) was not what happened in that run. The killed daemon itself still held the kernel lock for a moment and still looked alive, with phase serving and no heartbeat. That shape fell into the older branch of the recovery guard: one more connect attempt, then a plain error, which the server renders as internal_error when the previous daemon recorded no exit. Under 12 busy loops the test failed 1 time in 20 before the change.

Change. One predicate now decides whether recovery may wait for a lock holder (its truth table is in the doc comment of starting_owner): the lock is held, the process is alive, no exit was announced, the recorded root and identity are compatible, and the heartbeat is not stale. Identity fields that are still unknown may become known during the wait; a known value that changes ends it. A channel is used only after the authenticated root handshake, as before. If the holder releases the lock during the wait (the killed predecessor finishing its teardown), the same call goes on to start the replacement. The older one-attempt branch is gone.

A call that gives up because a holder is still starting is now daemon_restarting in every case, with retryable: true, waited_ms and startup_phase; when the previous daemon was killed, the hint says that its exit was not recorded instead of naming a cause.

Checked. After the change the same loop passes 20 of 20 under the same load (worker's run). In my run: 21 unit tests of the reconnect module, 10 reconnect and 8 daemon-death end-to-end tests pass; removing the "start the replacement once the lock is released" arm (mutation) fails the test written for it. The worker reported six further mutations red and the whole daemon suite and both MCP legs green.

**A gap in Part B found and closed before release** (lane r44, merged locally as `cf2d3ee4`). A full daemon-suite run under load failed once in `daemon_death_mcp_e2e::a_daemon_killed_mid_call_still_answers_that_call`: the call that was in flight when the daemon was killed came back as `internal_error` ("existing owner … is still starting. No replacement was started"). **Cause, traced.** My first guess (a freshly spawned daemon that has not published its identity yet) was not what happened in that run. The **killed** daemon itself still held the kernel lock for a moment and still looked alive, with phase `serving` and no heartbeat. That shape fell into the older branch of the recovery guard: one more connect attempt, then a plain error, which the server renders as `internal_error` when the previous daemon recorded no exit. Under 12 busy loops the test failed 1 time in 20 before the change. **Change.** One predicate now decides whether recovery may wait for a lock holder (its truth table is in the doc comment of `starting_owner`): the lock is held, the process is alive, no exit was announced, the recorded root and identity are compatible, and the heartbeat is not stale. Identity fields that are still unknown may become known during the wait; a known value that changes ends it. A channel is used only after the authenticated root handshake, as before. If the holder releases the lock during the wait (the killed predecessor finishing its teardown), the same call goes on to start the replacement. The older one-attempt branch is gone. A call that gives up because a holder is still starting is now `daemon_restarting` in every case, with `retryable: true`, `waited_ms` and `startup_phase`; when the previous daemon was killed, the hint says that its exit was not recorded instead of naming a cause. **Checked.** After the change the same loop passes 20 of 20 under the same load (worker's run). In my run: 21 unit tests of the reconnect module, 10 reconnect and 8 daemon-death end-to-end tests pass; removing the "start the replacement once the lock is released" arm (mutation) fails the test written for it. The worker reported six further mutations red and the whole daemon suite and both MCP legs green.

buildagent referenced this issue from a commit

2026-11-14 00:43:18 +01:00

Merge lane r44: recovery waits through a starting or dying lock holder and reports restarts as retryable (#584)

buildagent referenced this issue

2026-11-14 00:43:40 +01:00

v0.41.53: restart and update behaviour a session can rely on; link add keeps the language decision; skill command removed #587

buildagent referenced this issue from a commit

2026-11-14 11:54:09 +01:00

Merge lane r34: a failed capability probe is no longer remembered as an old daemon (#584)

buildagent referenced this issue from a commit

2026-11-14 11:54:09 +01:00

Merge lane r38: a call that finds a starting daemon waits for it; restart refusals carry retryable, the wait and the phase (#584)

buildagent referenced this issue from a commit

2026-11-14 11:54:09 +01:00

Merge PR #587: v0.41.53, restart and update behaviour a session can rely on

buildagent closed this issue

2026-11-14 11:54:09 +01:00

buildagent commented

2026-11-14 12:28:46 +01:00

Author

Member

Released in v0.41.53 (59b8bc3d); verified on the installed binaries.
Filters no longer stay refused after a transient capability request failure, and an existing session waits for a replacement daemon within its call budget. A restart refusal is retryable and includes waited_ms and startup_phase.

Verified on the installed release:

584_first_filtered_call: a filtered call receives an answer without a false older-daemon refusal. PASS: answered after 1 call; replies: ['ok']
584_after_kill: the session answers after its daemon is killed, with valid restart refusals if any. PASS: the session answered 0.2s after its daemon was killed

Remaining scope: a newly started MCP server, the CLI query and the hook keep their own waiting rules while a daemon starts.

Released in v0.41.53 (59b8bc3d); verified on the installed binaries. Filters no longer stay refused after a transient capability request failure, and an existing session waits for a replacement daemon within its call budget. A restart refusal is retryable and includes `waited_ms` and `startup_phase`. Verified on the installed release: - `584_first_filtered_call`: a filtered call receives an answer without a false older-daemon refusal. PASS: answered after 1 call; replies: ['ok'] - `584_after_kill`: the session answers after its daemon is killed, with valid restart refusals if any. PASS: the session answered 0.2s after its daemon was killed Remaining scope: a newly started MCP server, the CLI `query` and the hook keep their own waiting rules while a daemon starts.

buildagent referenced this issue

2026-11-14 20:02:50 +01:00

An upgrade leaves the old daemon running and the new MCP server silently attaches to it #591

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

h-dv/code-index#584

No description provided.

Rows
Columns