perf/hang: daemon resolve stage wedges on large cross-project C# repos (tier-3 import-boost GLOB cross-product) #65
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
h-dv/code-index#65
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Source: a1x dogfood on a 235-project .NET solution (~9k source files after build-output exclusion). Extreme instance of #53 (tier-3 scaling): here it does not terminate on human timescales. Release-blocker for that class of repo — not yet fixed.
Symptom
Fresh index: ~70s of real indexing (index.db 0 → 713 MB), then the daemon pegs ONE core with ZERO further I/O — index.db AND index.db-wal mtimes frozen — indefinitely (>10 min observed; 4 of 5 runs). Thread dump: 18 threads, exactly one Running (all process CPU), flat ~173 MB WS → spinning, not allocating. Last log before the freeze:
outline_changed=8958atindex stage: resolving references. The RPC listener stays up (TCP connects) but every tool returnswarming_up.Bisect (reporter)
Subtrees resolve fine and fast; only the full 235-project root wedges.
tier3_groups: 0 on a single project, 297 on a mid subtree, and it explodes at full cross-project import fan-out. The wedge scales with tier-3 group count.Root cause (source analysis)
tier-3 import-boost resolution in
resolve_ref_targets_in_tx(crates/indexer/src/index.rs, the ambiguous-remain block) joins each still-ambiguous ref's candidate pool againsttemp.file_keyswith NON-INDEXABLE leading-*GLOBs, e.g.{IMPORT_SEGMENTS} GLOB ('*.' || k.key || '.*')(~lines 1408-1435), plus the reachability predicate. A leading-*GLOB can't use an index → full scan each time → evaluated across {ambiguous refs} × {candidate symbols} × {file_keys}. At 235-project fan-out this is super-linear and effectively non-terminating. Verified there is NO recursive CTE / worklist loop — it terminates in theory, just not in practice. Same class as I020 (Cap'n Proto quadratic); the de-correlation note at index.rs:1478 isn't enough at this scale.Suggested direction
*GLOBs.name_fallbackwith disclosure instead of scanning the cross-product.Secondary (same repo) — index size
~115 KB of index per source file (~9k files → 1.04 GB) with build output EXCLUDED — not generated JSON. Candidate: per-occurrence ref rows carrying qualified-name strings (52,989 refs / 114 files in one subtree ⇒ tens of millions of rows at full scale). Worth checking whether ref storage is the real size driver. May share a cause with the tier-3 working-set blowup.
Related: #53.
Status update — NOT closing. Suggested direction 3 is now shipped; the secondary remains open.
Landed today on
master(a5adb00), completing suggested direction 3 ("a hard budget so a pathological repo degrades-with-disclosure rather than wedging"):The tier-3 budget degradation already existed, but its only outputs were a
tracing::warn!and a#[cfg(test)]atomic — a workspace grep found no consumer outsideindex.rs. So an entire resolution tier could be skipped and no MCP tool could report it: affected refs looked like ordinary misses and every published resolution rate was quietly wrong. The "with disclosure" half of the degrade was missing.Now:
resolver_healthtable (m0032, schema v32), re-derived on every resolve — written on both branches, so a set-only flag cannot latch on after one degraded run;project_overview.resolver_degradationandresolution_gaps.resolver_degradation, carrying the flag, the budget in force (so it is actionable, not merely announced), and a semantics string stating the consequence — forresolution_gapsspecifically, that an unknown share of the counts below are budget artifacts, not resolver blind spots, and the table should not be worked as a recall backlog until the budget is raised;Some(false). Mutation-proven — collapsing that withunwrap_or(false)fails the test, which matters because it looks like harmless defensive coding and would tell every older index "no degradation occurred".Directions 1 (indexable tier-3 reachability) and 2 (budget →
name_fallback) were addressed by the v0.10.0 work; the wedge itself is fixed.Still open: the Secondary section — ~115 KB of index per source file (~9k files → 1.04 GB) with build output excluded, and the hypothesis that per-occurrence ref rows carrying qualified-name strings are the size driver. Nothing in this session touched storage size. That question deserves its own issue if you'd rather close this one; leaving it here for now so the measurement isn't lost.
Secondary (index size) — measured. The hypothesis in the issue body is wrong.
The body proposes "per-occurrence ref rows carrying qualified-name strings" as the size driver. Measured with
dbstaton two indexes:files_fts_data(trigram index)files_fts_content(stored doc copy)symbols+ its indexesRef rows are the second driver, not the first. Full-text search is 57–81% of the index in both.
And the obvious FTS lever was already evaluated and rejected.
files_fts_contentholds a full document copy (27.2 MB on django, avg 6.4 KB/file), which looks like waste until you read_prdoc/records/I001-A001-fts5-contentless.md— "FTS5 contentless mode is operationally untenable". Contentless FTS5 cannot delete or replace a single row (only a fulldelete-all), which breaks per-file incremental updates, andsearch_text'ssnippet(files_fts, 1, …)(local_index.rs:1782) needs the stored content to produce excerpts. So that ~19% is a deliberate, documented trade-off, not a regression.On the reported 115 KB/source-file: django measures ~34.9 KB per indexed file (147.6 MB / 4235). The C# case is therefore ~3.3× heavier per file than django, which suggests something specific to that repo or to the C# plugin rather than a general per-ref cost — worth isolating before optimising anything. Note also that FTS covers text/metadata files too (django: 4235 FTS rows vs 2970 code files), by design, so file count alone understates FTS scope.
Where the remaining levers actually are, in rough order of size: the trigram tokenizer (
files_fts_data, the single largest component — trigrams of every document are inherently bulky; a cheaper tokenizer or a size cap on FTS'd files would move real bytes); then ref-row width, which is the issue's original hypothesis and is worth ~29% at django scale rather than the majority.Recommend re-titling this secondary around FTS storage cost, since that is where the bytes are, and carrying forward the constraint that contentless mode is off the table for the reasons in I001-A001.