search_text: cost of one query grows quadratically with the literal's length (Windows: a 400,000-character literal blocks the server for more than 30 s)

buildagent commented

2026-11-12 09:20:15 +01:00

Member

Found while qualifying v0.41.50 (PR #574, Windows CI round 2). Present in v0.41.49 and earlier; not fixed in v0.41.50.

Observed

Windows CI: search_text with a quoted literal of 400,000 characters did not answer within 30 s. The server process was alive and silent; every other call in the same session answered in about 1 ms.
Linux, debug build, same call driven over stdio (--no-daemon), literal of a repeated n times:

n	`whole_word: false`
400,000	0.51 s
800,000	0.95 s
3,200,000	about 14 s
6,400,000	25.7 s

Cause (established by stack samples on Linux)

Every sample during the call is in SQLite's FTS5 query parser:

fts5ParseTokenize        sqlite3.c:248479   sqlite3_realloc64(pPhrase, SZ_FTS5EXPRPHRASE(nNew+1))
fts5TriTokenize
sqlite3Fts5ParseTerm
sqlite3Fts5ExprNew

fts5ParseTokenize grows the phrase by SZALLOC = 8 terms per reallocation. A literal of n characters is n-2 trigram tokens, so n/8 reallocations of a block that grows to about 40 bytes per token (200 MB observed at 6.4 M characters). glibc resizes a large block with mremap and rarely copies; an allocator that copies on every growth (the Windows heap for large blocks) moves on the order of n²/16 × 40 bytes: about 400 GB for n = 400,000.

Not established: the exact Windows duration (the CI harness gave up at 30 s), and whether other tools hand caller text to FTS5 the same way.

Why it matters

One request can keep a server busy for seconds (Linux) to minutes (Windows). No ordinary agent query is that long; a pasted file is.

Proposal

Bound the length of a search_text literal in the argument policy (#570 registry), so that the schema states it (maxLength), the refusal is invalid_argument_value with the accepted length, and docs/mcp-arguments.md lists it. For scale: at 16 KiB the copying worst case is about 0.7 GB (well under a second); at 64 KiB about 11 GB.

Consequences to handle in the same change:

whole_word_needle_unsupported becomes unreachable below about 31 KiB of literal (the regex engine's smallest compile limit, measured: 31,208 × ι), so it moves to the unreachable kept codes in argument-legacy-acceptance.json if the bound is lower.
Check search_symbols and every other argument that reaches an FTS5 MATCH for the same growth.

What v0.41.50 did instead

Only the test changed: the replayed case uses the shortest literal the regex engine refuses (40,000 × ι) instead of 400,000 × a (commit b28850b0). The product path is unchanged.

Found while qualifying v0.41.50 (PR #574, Windows CI round 2). Present in v0.41.49 and earlier; not fixed in v0.41.50. ## Observed - Windows CI: `search_text` with a quoted literal of 400,000 characters did not answer within 30 s. The server process was alive and silent; every other call in the same session answered in about 1 ms. - Linux, debug build, same call driven over stdio (`--no-daemon`), literal of `a` repeated n times: | n | `whole_word: false` | |---|---| | 400,000 | 0.51 s | | 800,000 | 0.95 s | | 3,200,000 | about 14 s | | 6,400,000 | 25.7 s | ## Cause (established by stack samples on Linux) Every sample during the call is in SQLite's FTS5 query parser: ``` fts5ParseTokenize sqlite3.c:248479 sqlite3_realloc64(pPhrase, SZ_FTS5EXPRPHRASE(nNew+1)) fts5TriTokenize sqlite3Fts5ParseTerm sqlite3Fts5ExprNew ``` `fts5ParseTokenize` grows the phrase by `SZALLOC = 8` terms per reallocation. A literal of n characters is n-2 trigram tokens, so n/8 reallocations of a block that grows to about 40 bytes per token (200 MB observed at 6.4 M characters). glibc resizes a large block with `mremap` and rarely copies; an allocator that copies on every growth (the Windows heap for large blocks) moves on the order of n²/16 × 40 bytes: about 400 GB for n = 400,000. Not established: the exact Windows duration (the CI harness gave up at 30 s), and whether other tools hand caller text to FTS5 the same way. ## Why it matters One request can keep a server busy for seconds (Linux) to minutes (Windows). No ordinary agent query is that long; a pasted file is. ## Proposal Bound the length of a `search_text` literal in the argument policy (#570 registry), so that the schema states it (`maxLength`), the refusal is `invalid_argument_value` with the accepted length, and `docs/mcp-arguments.md` lists it. For scale: at 16 KiB the copying worst case is about 0.7 GB (well under a second); at 64 KiB about 11 GB. Consequences to handle in the same change: - `whole_word_needle_unsupported` becomes unreachable below about 31 KiB of literal (the regex engine's smallest compile limit, measured: 31,208 × `ι`), so it moves to the unreachable kept codes in `argument-legacy-acceptance.json` if the bound is lower. - Check `search_symbols` and every other argument that reaches an FTS5 `MATCH` for the same growth. ## What v0.41.50 did instead Only the test changed: the replayed case uses the shortest literal the regex engine refuses (40,000 × `ι`) instead of 400,000 × `a` (commit b28850b0). The product path is unchanged.

buildagent referenced this issue

2026-11-12 09:44:16 +01:00

v0.41.50: module evidence for qualified references, one argument contract, one extraction census #574

buildagent referenced this issue

2026-11-12 11:41:34 +01:00

Arguments, settings and documented facts disagree or lack one checked source #570

buildagent referenced this issue

2026-11-12 11:41:41 +01:00

Tool discovery and replies cost extra calls, while review and navigation still have gaps #521

buildagent referenced this issue

2026-11-12 13:41:32 +01:00

Tool discovery and replies cost extra calls, while review and navigation still have gaps #521

buildagent referenced this issue from a commit

2026-11-12 18:41:23 +01:00

Merge codex-r24: one length rule for every string argument, missing paths answered as missing (#575, #578)

buildagent referenced this issue

2026-11-12 18:41:44 +01:00

v0.41.51: bounded text arguments, truthful path and startup answers, review grading by effective visibility #580

buildagent referenced this issue from a commit

2026-11-12 20:47:09 +01:00

fix(mcp): bound text queries through argument policy (#575)

buildagent referenced this issue from a commit

2026-11-12 20:47:09 +01:00

test(mcp): replay text length refusals and pin Unicode boundaries (#575)

buildagent referenced this issue from a commit

2026-11-12 20:47:09 +01:00

Merge PR #580: v0.41.51 — bounded text arguments, truthful path and startup answers, review grading by effective visibility

buildagent referenced this issue

2026-11-12 22:02:47 +01:00

Startup with language packages is about six times slower since v0.41.49; the v0.41.51 release run failed on it #581

buildagent referenced this issue

2026-11-12 22:22:54 +01:00

Arguments, settings and documented facts disagree or lack one checked source #570

buildagent added the

labels

2026-11-12 22:23:30 +01:00

buildagent commented

2026-11-13 13:06:46 +01:00

Author

Member

Released in v0.41.52 (7fc7fb20, release run 8656, 15 assets). Verified on the installed Linux x86_64 binaries (code-index 0.41.52 (7fc7fb20)):

search_text with a query of 8,193 characters: refused, invalid_argument_value, received.characters: 8193, accepted.maxLength: 8192, exit status 1.
8,192 characters: accepted.
A list of queries with one over-long entry: refused and names the entry (entry: 1).
An over-long path argument: refused the same way.
The published tool schema carries maxLength for the query.

Released in [v0.41.52](https://git.h-dv.de/h-dv/code-index/releases/tag/v0.41.52) (`7fc7fb20`, release run 8656, 15 assets). Verified on the installed Linux x86_64 binaries (`code-index 0.41.52 (7fc7fb20)`): - `search_text` with a query of 8,193 characters: refused, `invalid_argument_value`, `received.characters: 8193`, `accepted.maxLength: 8192`, exit status 1. - 8,192 characters: accepted. - A list of queries with one over-long entry: refused and names the entry (`entry: 1`). - An over-long path argument: refused the same way. - The published tool schema carries `maxLength` for the query.

buildagent closed this issue

2026-11-13 13:06:46 +01:00

Rows
Columns

search_text: cost of one query grows quadratically with the literal's length (Windows: a 400,000-character literal blocks the server for more than 30 s) #575

Observed

Cause (established by stack samples on Linux)

Why it matters

Proposal

What v0.41.50 did instead