h-dv/code-index

Fork 0

daemon lifecycle: no progress/status signal, no health-based takeover of a wedged holder, shim never re-discovers daemon #66

New issue

Closed

opened 2026-08-05 12:38:05 +02:00 by buildagent · 1 comment

buildagent commented

2026-08-05 12:38:05 +02:00

Member

Source: a1x dogfood. Four related lifecycle gaps that together turned the resolve-stage wedge (#65) into ~50 min of ambiguity for the caller. Not yet fixed.

(a) `warming_up` is indistinguishable from a wedged daemon

warming_up tells the caller "retry in 1-2s"; the reporter observed >14 min of it, then a silent flip to project_not_available. There is NO status/progress RPC in the wire surface (crates/daemon/src/rpc_index.rs exposes stats + graph methods only), and stats/project_overview is unavailable for the entire indexing window. A caller cannot tell "indexing, wait" from "wedged, kill it". Even a monotonic files_indexed/refs_resolved counter would settle it in seconds. Consider also a degraded/partial-ready mode that serves reads off the partial index while the resolve stage runs (the read pool is already open at that point).

(b) No health-based takeover of a wedged holder

The daemon lock is a kernel file lock (crates/daemon/src/lifecycle.rs, try_lock_exclusive). A DEAD holder frees it and a replacement takes over cleanly. But a holder that is ALIVE yet WEDGED (pegging a core, no longer serving) keeps the lock, so a replacement correctly gets LockBusy and defers forever. pid_is_alive() gates the MCP attach probe, not daemon-side takeover. Suggest: a replacement should evict a holder that is alive but fails a TCP/health probe.

(c) daemon.toml can advertise a port nothing is listening on

A wedged daemon published port=58124 with no listener for its entire life (only 3 threads). The state file is not a usable readiness signal on its own — pair it with a bind-liveness/health check.

(d) MCP shim single_process fallback never retries daemon rediscovery

When daemon attach fails, the MCP server (crates/mcp-server/src/main.rs) falls back to opening the DB directly (LocalIndex) and never retries daemon discovery. A shim started before the daemon therefore stays on the direct DB, surfaces project_not_available, and only a client reconnect (/mcp) recovers. The direct-DB hold is fallback-only, not the intended daemon path. Suggest: on the fallback path, periodically re-attempt daemon discovery and switch over when one appears.

These are the ergonomics half of #65: even with the tier-3 fix, the daemon should never leave a caller unable to distinguish slow from wedged, or unable to recover without a manual reconnect.

**Source:** a1x dogfood. Four related lifecycle gaps that together turned the resolve-stage wedge (#65) into ~50 min of ambiguity for the caller. Not yet fixed. ## (a) `warming_up` is indistinguishable from a wedged daemon `warming_up` tells the caller "retry in 1-2s"; the reporter observed >14 min of it, then a silent flip to `project_not_available`. There is NO status/progress RPC in the wire surface (crates/daemon/src/rpc_index.rs exposes `stats` + graph methods only), and `stats`/`project_overview` is unavailable for the entire indexing window. A caller cannot tell "indexing, wait" from "wedged, kill it". Even a monotonic `files_indexed`/`refs_resolved` counter would settle it in seconds. Consider also a degraded/partial-ready mode that serves reads off the partial index while the resolve stage runs (the read pool is already open at that point). ## (b) No health-based takeover of a wedged holder The daemon lock is a kernel file lock (crates/daemon/src/lifecycle.rs, `try_lock_exclusive`). A DEAD holder frees it and a replacement takes over cleanly. But a holder that is ALIVE yet WEDGED (pegging a core, no longer serving) keeps the lock, so a replacement correctly gets `LockBusy` and defers forever. `pid_is_alive()` gates the MCP attach probe, not daemon-side takeover. Suggest: a replacement should evict a holder that is alive but fails a TCP/health probe. ## (c) daemon.toml can advertise a port nothing is listening on A wedged daemon published `port=58124` with no listener for its entire life (only 3 threads). The state file is not a usable readiness signal on its own — pair it with a bind-liveness/health check. ## (d) MCP shim single_process fallback never retries daemon rediscovery When daemon attach fails, the MCP server (crates/mcp-server/src/main.rs) falls back to opening the DB directly (`LocalIndex`) and never retries daemon discovery. A shim started before the daemon therefore stays on the direct DB, surfaces `project_not_available`, and only a client reconnect (`/mcp`) recovers. The direct-DB hold is fallback-only, not the intended daemon path. Suggest: on the fallback path, periodically re-attempt daemon discovery and switch over when one appears. These are the ergonomics half of #65: even with the tier-3 fix, the daemon should never leave a caller unable to distinguish slow from wedged, or unable to recover without a manual reconnect.

buildagent added the

labels

2026-08-05 12:38:46 +02:00

dhoyer referenced this issue from a commit

2026-08-05 13:49:06 +02:00

fix(mcp-server): self-heal single-process fallback via background daemon rediscovery (part of #66)

dhoyer referenced this issue from a commit

2026-08-05 13:54:22 +02:00

fix(mcp-server): fold live index counts into the warming_up hint (part of #66)

dhoyer referenced this issue from a commit

2026-08-05 15:44:01 +02:00

feat(daemon): health-based takeover of wedged daemon + bind-liveness (parts b, c of #66)

dhoyer referenced this issue from a commit

2026-08-05 15:44:01 +02:00

harden(daemon): verify holder image before eviction to survive PID reuse (part b of #66)

buildagent commented

2026-08-07 21:26:49 +02:00

Author

Member

Closing — all four parts verified in-tree.

(a) progress signal — warming_up is no longer opaque. warming_progress_counts (mcp-server) reads files / symbols / refs WHERE target_id IS NOT NULL on the warming slow path and rides along with the retry hint, so a caller can watch the counters move and distinguish "indexing, wait" from "wedged, kill it". That is the monotonic counter this issue asked for.

(b) health-based takeover — crates/daemon/src/takeover.rs states the distinction explicitly: "Takeover distinguishes the two by health, not mere liveness", backed by probe_reachable. An alive-but-wedged holder no longer keeps the lock forever.

(c) port with no listener — same probe_reachable primitive doubles as bind-liveness, so daemon.toml advertising a dead port is detectable rather than trusted.

(d) shim never re-discovers — crates/mcp-server/src/main.rs has spawn_daemon_rediscovery, kicked proactively and re-triggered rather than latching on the direct-DB fallback.

Note this is the ergonomics half of #65 and it is now genuinely done; #65 itself stays open for its unresolved secondary (index size ~115 KB per source file). The tier-3 wedge that motivated both is fixed, and as of today the degradation is also disclosed — see the #65 comment.

Closing — all four parts verified in-tree. **(a) progress signal** — `warming_up` is no longer opaque. `warming_progress_counts` (mcp-server) reads `files` / `symbols` / `refs WHERE target_id IS NOT NULL` on the warming slow path and rides along with the retry hint, so a caller can watch the counters move and distinguish "indexing, wait" from "wedged, kill it". That is the monotonic counter this issue asked for. **(b) health-based takeover** — `crates/daemon/src/takeover.rs` states the distinction explicitly: *"Takeover distinguishes the two by health, not mere liveness"*, backed by `probe_reachable`. An alive-but-wedged holder no longer keeps the lock forever. **(c) port with no listener** — same `probe_reachable` primitive doubles as bind-liveness, so `daemon.toml` advertising a dead port is detectable rather than trusted. **(d) shim never re-discovers** — `crates/mcp-server/src/main.rs` has `spawn_daemon_rediscovery`, kicked proactively and re-triggered rather than latching on the direct-DB fallback. Note this is the *ergonomics* half of #65 and it is now genuinely done; #65 itself stays open for its unresolved secondary (index size ~115 KB per source file). The tier-3 wedge that motivated both is fixed, and as of today the degradation is also **disclosed** — see the #65 comment.

buildagent closed this issue

2026-08-07 21:26:53 +02:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

h-dv/code-index#66

No description provided.

Rows
Columns

daemon lifecycle: no progress/status signal, no health-based takeover of a wedged holder, shim never re-discovers daemon #66

(a) warming_up is indistinguishable from a wedged daemon

(b) No health-based takeover of a wedged holder

(c) daemon.toml can advertise a port nothing is listening on

(d) MCP shim single_process fallback never retries daemon rediscovery

(a) `warming_up` is indistinguishable from a wedged daemon