Watcher/writer: files on disk drop out of the live index after churn (dir bounce, git stash pop) and never heal until the 30-min reconcile #381

Closed
opened 2026-10-15 15:37:41 +02:00 by buildagent · 1 comment
Member

Found by the 2026-10-15 review and reproduced on 0.39.0 (d79756f1). Pre-existing.

Repro

  • Run a daemon on a copy of rust-ripgrep.
  • Trigger one soft re-walk with an empty commit.
  • Run mv crates/ignore away and back 0.4 s later.
  • Result: all 9 files of crates/ignore/src are still on disk but gone from the index, still gone after 105 s. 17 paths are lost in total.
  • index_freshness reports state: lagging, added: 17, so it measures the loss but nothing acts on it.
  • index_coverage tells the agent "picks new files up in about a second — retry", which is false.
  • git stash && git stash pop loses files the same way (delete-recreate with identical content): a StatTouch lands on a row that was already deleted. A new file renamed away before its insert commits leaves a phantom row.

Root cause

Decisions are made against the committed read connection while the writer still has messages queued.

  • soft_rewalk (watcher.rs ~1949) sends a Withdraw or Delete for paths the racing walk did not see.
  • The event path decides StatSkip against the old committed row.
  • The removal then commits, and nothing re-adds the file.

Direction

Two generic fixes:

  1. Re-validate against disk when the writer applies a message. Delete only if the path is absent; Withdraw only if classification now says skip; a StatTouch that updates 0 rows becomes a re-parse.
  2. Self-heal. When freshness::compare measures added/changed, enqueue those paths.

Test: a deterministic storm harness that compares the settled live index with a cold index of the same tree on files, symbols, refs and binds.

Found by the 2026-10-15 review and reproduced on 0.39.0 (d79756f1). Pre-existing. ## Repro - Run a daemon on a copy of rust-ripgrep. - Trigger one soft re-walk with an empty commit. - Run `mv crates/ignore` away and back 0.4 s later. - Result: all 9 files of `crates/ignore/src` are still on disk but gone from the index, still gone after 105 s. 17 paths are lost in total. - `index_freshness` reports `state: lagging, added: 17`, so it measures the loss but nothing acts on it. - `index_coverage` tells the agent "picks new files up in about a second — retry", which is false. - `git stash && git stash pop` loses files the same way (delete-recreate with identical content): a `StatTouch` lands on a row that was already deleted. A new file renamed away before its insert commits leaves a phantom row. ## Root cause Decisions are made against the committed read connection while the writer still has messages queued. - `soft_rewalk` (`watcher.rs` ~1949) sends a `Withdraw` or `Delete` for paths the racing walk did not see. - The event path decides `StatSkip` against the old committed row. - The removal then commits, and nothing re-adds the file. ## Direction Two generic fixes: 1. **Re-validate against disk when the writer applies a message.** `Delete` only if the path is absent; `Withdraw` only if classification now says skip; a `StatTouch` that updates 0 rows becomes a re-parse. 2. **Self-heal.** When `freshness::compare` measures `added`/`changed`, enqueue those paths. Test: a deterministic storm harness that compares the settled live index with a cold index of the same tree on files, symbols, refs and binds.
Author
Member

Released in v0.40.0 (https://git.h-dv.de/h-dv/code-index/releases/tag/v0.40.0, PR #388, merge a48fb945): every watcher write is checked against the disk as it applies; an edit/rename/delete storm ends equal to a cold index; index_freshness re-queues lagging paths; the pathless-event panic that killed the watcher thread on Linux is fixed; mass deletes 8.4 s → 0.48 s for 10,000 files. zammad-dev has been asked to retest their live-deletion probe on 0.40.

Released in v0.40.0 (https://git.h-dv.de/h-dv/code-index/releases/tag/v0.40.0, PR #388, merge a48fb945): every watcher write is checked against the disk as it applies; an edit/rename/delete storm ends equal to a cold index; `index_freshness` re-queues lagging paths; the pathless-event panic that killed the watcher thread on Linux is fixed; mass deletes 8.4 s → 0.48 s for 10,000 files. zammad-dev has been asked to retest their live-deletion probe on 0.40.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#381
No description provided.