mcp-server: search_text whole_word=true has a fixed 1000-candidate cap and over-reports total #15

Closed
opened 2026-07-07 13:19:31 +02:00 by buildagent · 1 comment
Member

Severity: Medium · Category: Correctness + performance

For whole_word=true, search_text fetches a fixed 1000-candidate cap per project at offset 0, then post-filters (crates/mcp-server/src/server.rs:608,630,662-694 fan-out; :2896,2957 narrow). Whole-word matches beyond the first 1000 trigram candidates are unreachable, and in the fan-out path grand_total reports the raw FTS total (:657) — so total overstates the reachable whole-word count and next_cursor promises pages that never yield the missing hits.

Companion perf issue: contains_as_word (:3026-3055) allocates a full to_ascii_lowercase() copy of each candidate file (up to the ~2 MiB read cap) per candidate → up to ~2 GiB transient allocation across a fan-out whole-word query.

Fix

  • Paginate the underlying fetch (loop until limit whole-word hits or FTS exhausted), or return an approximate: true flag and cap the reported total to what's reachable.
  • Case-fold incrementally / match without a full lowercase copy (e.g. char_indices + eq_ignore_ascii_case over windows).

Filed from a deep multi-agent code review of the workspace (v0.5.7). Severity/category per the review; file:line refs were accurate at review time — verify against current master.

**Severity:** Medium · **Category:** Correctness + performance For `whole_word=true`, `search_text` fetches a **fixed 1000-candidate** cap per project at offset 0, then post-filters (`crates/mcp-server/src/server.rs:608,630,662-694` fan-out; `:2896,2957` narrow). Whole-word matches beyond the first 1000 trigram candidates are **unreachable**, and in the fan-out path `grand_total` reports the *raw* FTS total (`:657`) — so `total` overstates the reachable whole-word count and `next_cursor` promises pages that never yield the missing hits. Companion perf issue: `contains_as_word` (`:3026-3055`) allocates a full `to_ascii_lowercase()` copy of each candidate file (up to the ~2 MiB read cap) per candidate → up to ~2 GiB transient allocation across a fan-out whole-word query. ### Fix - Paginate the underlying fetch (loop until `limit` whole-word hits or FTS exhausted), or return an `approximate: true` flag and cap the reported `total` to what's reachable. - Case-fold incrementally / match without a full lowercase copy (e.g. `char_indices` + `eq_ignore_ascii_case` over windows). --- _Filed from a deep multi-agent code review of the workspace (v0.5.7). Severity/category per the review; file:line refs were accurate at review time — verify against current `master`._
Author
Member

Fixed in 73f5881 (branch fix/ultradeep-review-findings).

Reassessed on current master:

  • Total over-reporting — was present on the fan-out path only (fan_out_search_text reported grand_total, the raw FTS trigram total). Fixed: it now accumulates the per-project whole-word-filtered count and reports that for total, next_cursor, and the broad-query hint. (The narrow single-project path already reported the filtered total_filtered, so it wasn't over-reporting — verified.)
  • contains_as_word allocation — was allocating a full to_ascii_lowercase() copy of each candidate file (≤2 MiB × every fan-out candidate). Rewritten to scan byte windows with eq_ignore_ascii_case in place — behavior-identical (both sides already ASCII-folded; a window can only align to a char boundary since the needle's first byte is never a continuation byte), and all existing contains_as_word unit tests stay green.

Test: mcp_smoke::search_text_whole_word_total_counts_only_word_matches — todo matches TODO (whole word) and autodone (subword) by trigram; asserts whole_word=true returns total == 1, not the raw trigram total.

Left as a documented known bound (separate from the over-report bug): the whole-word post-filter fetches a fixed 1000-candidate window per project, so matches ranked beyond it aren't reachable. Full pagination (loop until limit whole-word hits or FTS exhausted) is a larger change; happy to file it separately if you want it pursued.

Fixed in `73f5881` (branch `fix/ultradeep-review-findings`). Reassessed on current master: - **Total over-reporting** — was present on the **fan-out** path only (`fan_out_search_text` reported `grand_total`, the raw FTS trigram total). Fixed: it now accumulates the per-project whole-word-filtered count and reports that for `total`, `next_cursor`, and the broad-query hint. (The narrow single-project path already reported the filtered `total_filtered`, so it wasn't over-reporting — verified.) - **`contains_as_word` allocation** — was allocating a full `to_ascii_lowercase()` copy of each candidate file (≤2 MiB × every fan-out candidate). Rewritten to scan byte windows with `eq_ignore_ascii_case` in place — behavior-identical (both sides already ASCII-folded; a window can only align to a char boundary since the needle's first byte is never a continuation byte), and all existing `contains_as_word` unit tests stay green. **Test:** `mcp_smoke::search_text_whole_word_total_counts_only_word_matches` — `todo` matches `TODO` (whole word) and `autodone` (subword) by trigram; asserts `whole_word=true` returns `total == 1`, not the raw trigram total. **Left as a documented known bound** (separate from the over-report bug): the whole-word post-filter fetches a fixed 1000-candidate window per project, so matches ranked beyond it aren't reachable. Full pagination (loop until `limit` whole-word hits or FTS exhausted) is a larger change; happy to file it separately if you want it pursued.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#15
No description provided.