h-dv/code-index
Watch
4
0
Fork
You've already forked code-index
 0
Code Issues 46 Pull requests Projects Releases 98 Packages Wiki Activity Actions 1

perf/reliability: tier 1R and C# partial resolution scale quadratically and hide progress #65

New issue
Closed
opened 2026-08-05 12:37:51 +02:00 by buildagent · 5 comments
buildagent commented 2026-08-05 12:37:51 +02:00
Member
Copy link

Current live defect

The original tier-3 leading-wildcard GLOB implementation was replaced, but the production failure remains in other resolver stages.

A synthetic C# monorepo matching the reported 235-project shape demonstrates:

projects files tier 1R tier 3 C# partial total
10 80 0.34s 0.18s 0.03s 0.70s
40 320 6.68s 2.66s 0.60s 10.5s
80 640 27.85s 9.43s 2.71s 40.9s
160 1,280 110.27s 33.49s 12.71s 158s

Tier 1R scales at approximately N^2.06 and C# partial at N^2.12. Extrapolation to the reported ~9,000 files remains human-scale non-termination.

This is a production blocker for large multi-project repositories and a prerequisite for #77: runtime plugin profiles and bridges must not add new candidate populations to an unbounded resolver.

Root cause

Tier 1R expands each receiver binding across every same-named candidate container, then evaluates correlated origin evidence repeatedly for each candidate triple.

The query-plan shape includes:

  • recv_bound scan;
  • same-name parent fan-out through symbols;
  • child member lookup;
  • repeated correlated import/file-key/package evidence checks using non-indexable pattern work.

With common project-local type names such as Program, Settings, Repository, Logger and IService, candidate cardinality grows with project count. Controlled ablation making type names unique reduced the N=80 tier-1R stage from ~27.8 seconds to ~0.34 seconds.

C# partial-member resolution separately joins same-named members across partial scopes and has its own quadratic population.

The tier-3 budget does not guard either stage.

User-visible diagnosability defect

The resolve transaction can run for minutes while:

  • RPC accepts connections but tools return warming_up;
  • committed file/symbol/ref counters remain frozen because resolution is one transaction;
  • the useful heartbeat goes only to daemon stderr;
  • an auto-spawned daemon discards stderr;
  • the warming hint interprets frozen counters as a stuck daemon.

Thus the only operator-visible signal can assert the wrong diagnosis while the resolver is actively consuming CPU.

Required design

1. Materialize origin evidence

Compute receiver/container origin eligibility once per bounded key such as (source file, bound type, candidate file/container), then join by equality.

Do not re-evaluate import segments, file keys and package pattern predicates inside the candidate-member cross product. Reuse the indexable materialization discipline already applied to tier 3.

C# partial scope likewise needs a bounded, indexed scope/member representation rather than an unconstrained same-name join.

2. Resolver-wide work budgets

Budget every stage capable of candidate fan-out, including:

  • tier 1R;
  • C# partial;
  • tier 1Q/type-FQN;
  • tier 3;
  • future dynamic-profile and bridge passes from #77.

Budgets are based on a conservative pre-count or actual work units, not wall time alone. Exceeding a budget leaves affected refs unresolved and records:

  • stage/rule;
  • work estimate/actual work;
  • configured budget;
  • affected ref groups/sites;
  • generation/profile/package influence;
  • stable semantics explaining that reported recall is a floor.

A budget must not silently change no_candidate/internal_missed meaning.

3. Observable progress

Expose in-memory resolver progress through daemon health/RPC without requiring a write inside the long transaction:

  • active stage;
  • elapsed time;
  • completed/total work when measurable;
  • heartbeat timestamp;
  • degradation/budget state;
  • active index/plugin generation.

Auto-spawned daemon logs must remain recoverable, but logs are not the API. warming_up text must not claim frozen committed counters imply a wedge while a fresh resolver heartbeat exists.

4. Cancellation and recovery

  • CLI/daemon shutdown interrupts the current SQLite work.
  • Pending work remains durable.
  • Restart resumes or safely recomputes.
  • A budget/cancellation cannot promote a partial plugin generation under #78.

Regression shape

Keep a generated many-project corpus with:

  • repeated container names across projects;
  • repeated member names;
  • receiver bindings such as var x = new Settings(); x.Get();
  • cross-project imports;
  • C# partial declarations;
  • negative decoys proving no new phantom edges.

Assertions:

  • materialized relation cardinality has a documented upper bound;
  • EQP contains no correlated large-table/pattern scan in tier 1R or partial;
  • doubling project count does not quadruple measured work units;
  • every stage is budgeted;
  • a deliberately tiny budget degrades with exact disclosure;
  • health progress advances while the resolve transaction is uncommitted;
  • tools/health remain responsive;
  • resolved target projection is identical to the pre-optimization implementation on bounded corpora;
  • phantom count remains zero.

Wall-clock remains a generous hang ceiling, not the primary proof, because shared-runner timing is noisy.

Relationship to other issues

  • #53 tracks residual end-to-end superlinearity after individual blockers are fixed.
  • #41 owns large-corpus ceilings and scale artifacts.
  • #73 owns project_overview aggregate latency, not resolve latency.
  • #77 must reuse the stage-budget/materialized-evidence framework for dynamic languages and bridges.
  • #78 must treat resolver degradation/failure as a failed pending generation, never promote partial semantic state.

Acceptance

  1. Tier 1R and C# partial have no correlated unbounded same-name/pattern cross product.
  2. Structural work growth on the regression corpus is sub-quadratic and bounded.
  3. Resolver-wide budgets cover all fan-out stages and disclose skipped populations.
  4. A 235-project-equivalent corpus completes or deliberately degrades within the committed ceiling.
  5. Operator-visible health distinguishes active slow progress from a wedge.
  6. Existing resolved targets and zero-phantom gates remain unchanged on non-degraded runs.
  7. Dynamic resolver capabilities cannot bypass the same budgets.
## Current live defect The original tier-3 leading-wildcard GLOB implementation was replaced, but the production failure remains in other resolver stages. A synthetic C# monorepo matching the reported 235-project shape demonstrates: | projects | files | tier 1R | tier 3 | C# partial | total | |---:|---:|---:|---:|---:|---:| | 10 | 80 | 0.34s | 0.18s | 0.03s | 0.70s | | 40 | 320 | 6.68s | 2.66s | 0.60s | 10.5s | | 80 | 640 | 27.85s | 9.43s | 2.71s | 40.9s | | 160 | 1,280 | 110.27s | 33.49s | 12.71s | 158s | Tier 1R scales at approximately N^2.06 and C# partial at N^2.12. Extrapolation to the reported ~9,000 files remains human-scale non-termination. This is a production blocker for large multi-project repositories and a prerequisite for #77: runtime plugin profiles and bridges must not add new candidate populations to an unbounded resolver. ## Root cause Tier 1R expands each receiver binding across every same-named candidate container, then evaluates correlated origin evidence repeatedly for each candidate triple. The query-plan shape includes: - recv_bound scan; - same-name parent fan-out through symbols; - child member lookup; - repeated correlated import/file-key/package evidence checks using non-indexable pattern work. With common project-local type names such as Program, Settings, Repository, Logger and IService, candidate cardinality grows with project count. Controlled ablation making type names unique reduced the N=80 tier-1R stage from ~27.8 seconds to ~0.34 seconds. C# partial-member resolution separately joins same-named members across partial scopes and has its own quadratic population. The tier-3 budget does not guard either stage. ## User-visible diagnosability defect The resolve transaction can run for minutes while: - RPC accepts connections but tools return warming_up; - committed file/symbol/ref counters remain frozen because resolution is one transaction; - the useful heartbeat goes only to daemon stderr; - an auto-spawned daemon discards stderr; - the warming hint interprets frozen counters as a stuck daemon. Thus the only operator-visible signal can assert the wrong diagnosis while the resolver is actively consuming CPU. ## Required design ### 1. Materialize origin evidence Compute receiver/container origin eligibility once per bounded key such as (source file, bound type, candidate file/container), then join by equality. Do not re-evaluate import segments, file keys and package pattern predicates inside the candidate-member cross product. Reuse the indexable materialization discipline already applied to tier 3. C# partial scope likewise needs a bounded, indexed scope/member representation rather than an unconstrained same-name join. ### 2. Resolver-wide work budgets Budget every stage capable of candidate fan-out, including: - tier 1R; - C# partial; - tier 1Q/type-FQN; - tier 3; - future dynamic-profile and bridge passes from #77. Budgets are based on a conservative pre-count or actual work units, not wall time alone. Exceeding a budget leaves affected refs unresolved and records: - stage/rule; - work estimate/actual work; - configured budget; - affected ref groups/sites; - generation/profile/package influence; - stable semantics explaining that reported recall is a floor. A budget must not silently change no_candidate/internal_missed meaning. ### 3. Observable progress Expose in-memory resolver progress through daemon health/RPC without requiring a write inside the long transaction: - active stage; - elapsed time; - completed/total work when measurable; - heartbeat timestamp; - degradation/budget state; - active index/plugin generation. Auto-spawned daemon logs must remain recoverable, but logs are not the API. warming_up text must not claim frozen committed counters imply a wedge while a fresh resolver heartbeat exists. ### 4. Cancellation and recovery - CLI/daemon shutdown interrupts the current SQLite work. - Pending work remains durable. - Restart resumes or safely recomputes. - A budget/cancellation cannot promote a partial plugin generation under #78. ## Regression shape Keep a generated many-project corpus with: - repeated container names across projects; - repeated member names; - receiver bindings such as var x = new Settings(); x.Get(); - cross-project imports; - C# partial declarations; - negative decoys proving no new phantom edges. Assertions: - materialized relation cardinality has a documented upper bound; - EQP contains no correlated large-table/pattern scan in tier 1R or partial; - doubling project count does not quadruple measured work units; - every stage is budgeted; - a deliberately tiny budget degrades with exact disclosure; - health progress advances while the resolve transaction is uncommitted; - tools/health remain responsive; - resolved target projection is identical to the pre-optimization implementation on bounded corpora; - phantom count remains zero. Wall-clock remains a generous hang ceiling, not the primary proof, because shared-runner timing is noisy. ## Relationship to other issues - #53 tracks residual end-to-end superlinearity after individual blockers are fixed. - #41 owns large-corpus ceilings and scale artifacts. - #73 owns project_overview aggregate latency, not resolve latency. - #77 must reuse the stage-budget/materialized-evidence framework for dynamic languages and bridges. - #78 must treat resolver degradation/failure as a failed pending generation, never promote partial semantic state. ## Acceptance 1. Tier 1R and C# partial have no correlated unbounded same-name/pattern cross product. 2. Structural work growth on the regression corpus is sub-quadratic and bounded. 3. Resolver-wide budgets cover all fan-out stages and disclose skipped populations. 4. A 235-project-equivalent corpus completes or deliberately degrades within the committed ceiling. 5. Operator-visible health distinguishes active slow progress from a wedge. 6. Existing resolved targets and zero-phantom gates remain unchanged on non-degraded runs. 7. Dynamic resolver capabilities cannot bypass the same budgets.
buildagent commented 2026-08-07 21:27:33 +02:00
Author
Member
Copy link

Status update — NOT closing. Suggested direction 3 is now shipped; the secondary remains open.

Landed today on master (a5adb00), completing suggested direction 3 ("a hard budget so a pathological repo degrades-with-disclosure rather than wedging"):

The tier-3 budget degradation already existed, but its only outputs were a tracing::warn! and a #[cfg(test)] atomic — a workspace grep found no consumer outside index.rs. So an entire resolution tier could be skipped and no MCP tool could report it: affected refs looked like ordinary misses and every published resolution rate was quietly wrong. The "with disclosure" half of the degrade was missing.

Now:

  • persisted in a new resolver_health table (m0032, schema v32), re-derived on every resolve — written on both branches, so a set-only flag cannot latch on after one degraded run;
  • surfaced as project_overview.resolver_degradation and resolution_gaps.resolver_degradation, carrying the flag, the budget in force (so it is actionable, not merely announced), and a semantics string stating the consequence — for resolution_gaps specifically, that an unknown share of the counts below are budget artifacts, not resolver blind spots, and the table should not be worked as a recall backlog until the budget is raised;
  • three-state: absent means "this index did not report" (pre-m0032 database) and is deliberately not Some(false). Mutation-proven — collapsing that with unwrap_or(false) fails the test, which matters because it looks like harmless defensive coding and would tell every older index "no degradation occurred".

Directions 1 (indexable tier-3 reachability) and 2 (budget → name_fallback) were addressed by the v0.10.0 work; the wedge itself is fixed.

Still open: the Secondary section — ~115 KB of index per source file (~9k files → 1.04 GB) with build output excluded, and the hypothesis that per-occurrence ref rows carrying qualified-name strings are the size driver. Nothing in this session touched storage size. That question deserves its own issue if you'd rather close this one; leaving it here for now so the measurement isn't lost.

**Status update — NOT closing. Suggested direction 3 is now shipped; the secondary remains open.** Landed today on `master` (`a5adb00`), completing suggested direction **3** ("a hard budget so a pathological repo degrades-with-disclosure rather than wedging"): The tier-3 budget degradation already existed, but its *only* outputs were a `tracing::warn!` and a `#[cfg(test)]` atomic — a workspace grep found no consumer outside `index.rs`. So an entire resolution tier could be skipped and **no MCP tool could report it**: affected refs looked like ordinary misses and every published resolution rate was quietly wrong. The "with disclosure" half of the degrade was missing. Now: - persisted in a new `resolver_health` table (m0032, schema v32), re-derived on **every** resolve — written on both branches, so a set-only flag cannot latch on after one degraded run; - surfaced as `project_overview.resolver_degradation` **and** `resolution_gaps.resolver_degradation`, carrying the flag, the budget in force (so it is actionable, not merely announced), and a semantics string stating the consequence — for `resolution_gaps` specifically, that an unknown share of the counts below are **budget artifacts, not resolver blind spots**, and the table should not be worked as a recall backlog until the budget is raised; - three-state: absent means *"this index did not report"* (pre-m0032 database) and is deliberately **not** `Some(false)`. Mutation-proven — collapsing that with `unwrap_or(false)` fails the test, which matters because it looks like harmless defensive coding and would tell every older index "no degradation occurred". Directions 1 (indexable tier-3 reachability) and 2 (budget → `name_fallback`) were addressed by the v0.10.0 work; the wedge itself is fixed. **Still open:** the *Secondary* section — ~115 KB of index per source file (~9k files → 1.04 GB) with build output excluded, and the hypothesis that per-occurrence ref rows carrying qualified-name strings are the size driver. Nothing in this session touched storage size. That question deserves its own issue if you'd rather close this one; leaving it here for now so the measurement isn't lost.
buildagent commented 2026-08-07 21:46:22 +02:00
Author
Member
Copy link

Secondary (index size) — measured. The hypothesis in the issue body is wrong.

The body proposes "per-occurrence ref rows carrying qualified-name strings" as the size driver. Measured with dbstat on two indexes:

component cosi-mcp (194 code files, 48.3 MB) py-django (2970 code files, 147.6 MB)
FTS5 total 80.9% 57.5%
├ files_fts_data (trigram index) 70.9% 38.4%
└ files_fts_content (stored doc copy) 10.0% 18.9%
refs + its indexes 13.7% 29.4%
symbols + its indexes ~3.6% ~7%

Ref rows are the second driver, not the first. Full-text search is 57–81% of the index in both.

And the obvious FTS lever was already evaluated and rejected. files_fts_content holds a full document copy (27.2 MB on django, avg 6.4 KB/file), which looks like waste until you read _prdoc/records/I001-A001-fts5-contentless.md — "FTS5 contentless mode is operationally untenable". Contentless FTS5 cannot delete or replace a single row (only a full delete-all), which breaks per-file incremental updates, and search_text's snippet(files_fts, 1, …) (local_index.rs:1782) needs the stored content to produce excerpts. So that ~19% is a deliberate, documented trade-off, not a regression.

On the reported 115 KB/source-file: django measures ~34.9 KB per indexed file (147.6 MB / 4235). The C# case is therefore ~3.3× heavier per file than django, which suggests something specific to that repo or to the C# plugin rather than a general per-ref cost — worth isolating before optimising anything. Note also that FTS covers text/metadata files too (django: 4235 FTS rows vs 2970 code files), by design, so file count alone understates FTS scope.

Where the remaining levers actually are, in rough order of size: the trigram tokenizer (files_fts_data, the single largest component — trigrams of every document are inherently bulky; a cheaper tokenizer or a size cap on FTS'd files would move real bytes); then ref-row width, which is the issue's original hypothesis and is worth ~29% at django scale rather than the majority.

Recommend re-titling this secondary around FTS storage cost, since that is where the bytes are, and carrying forward the constraint that contentless mode is off the table for the reasons in I001-A001.

**Secondary (index size) — measured. The hypothesis in the issue body is wrong.** The body proposes *"per-occurrence ref rows carrying qualified-name strings"* as the size driver. Measured with `dbstat` on two indexes: | component | cosi-mcp (194 code files, 48.3 MB) | py-django (2970 code files, 147.6 MB) | |---|---|---| | **FTS5 total** | **80.9%** | **57.5%** | | ├ `files_fts_data` (trigram index) | 70.9% | 38.4% | | └ `files_fts_content` (stored doc copy) | 10.0% | 18.9% | | **refs + its indexes** | **13.7%** | **29.4%** | | `symbols` + its indexes | ~3.6% | ~7% | Ref rows are the *second* driver, not the first. Full-text search is 57–81% of the index in both. **And the obvious FTS lever was already evaluated and rejected.** `files_fts_content` holds a full document copy (27.2 MB on django, avg 6.4 KB/file), which looks like waste until you read `_prdoc/records/I001-A001-fts5-contentless.md` — *"FTS5 contentless mode is operationally untenable"*. Contentless FTS5 cannot delete or replace a single row (only a full `delete-all`), which breaks per-file incremental updates, and `search_text`'s `snippet(files_fts, 1, …)` (`local_index.rs:1782`) needs the stored content to produce excerpts. So that ~19% is a deliberate, documented trade-off, not a regression. **On the reported 115 KB/source-file:** django measures ~34.9 KB per indexed file (147.6 MB / 4235). The C# case is therefore ~3.3× heavier per file than django, which suggests something specific to that repo or to the C# plugin rather than a general per-ref cost — worth isolating before optimising anything. Note also that FTS covers text/metadata files too (django: 4235 FTS rows vs 2970 code files), by design, so file count alone understates FTS scope. **Where the remaining levers actually are**, in rough order of size: the trigram tokenizer (`files_fts_data`, the single largest component — trigrams of every document are inherently bulky; a cheaper tokenizer or a size cap on FTS'd files would move real bytes); then ref-row width, which is the issue's original hypothesis and is worth ~29% at django scale rather than the majority. Recommend re-titling this secondary around **FTS storage cost**, since that is where the bytes are, and carrying forward the constraint that contentless mode is off the table for the reasons in I001-A001.
buildagent commented 2026-08-18 15:18:16 +02:00
Author
Member
Copy link

Correction: the wedge is NOT fixed. It moved to tier 1R.

Comment 4856 says "the wedge itself is fixed." That is true about the site it names and false about the stage that now dominates. Re-verified today against v0.14.0 with a synthetic reproduction, because the claim was asserted by the same work that made the fix.

Tier 3 really was fixed

The leading-* GLOBs are gone from tier 3 — import_key_candidates() (index.rs:748) now enumerates segment runs in Rust and probes a HashMap by equality into temp.import_key_rel. EXPLAIN QUERY PLAN on all three edge joins plus the aggregate shows no SCAN of a large table and no correlated subquery:

T3 EDGE1: |--SCAN g |--SEARCH fdr USING PRIMARY KEY `--SEARCH db USING INDEX ix_dir_bucket (...)
T3 EDGE2: |--SCAN g |--SEARCH b USING INDEX ix_symbol_buckets_name `--SEARCH ikr USING PRIMARY KEY (...)

t3_reach pairs grow linearly: 326k → 827k → 1676k at N=20/40/80. The "O(reachable pairs)" claim at index.rs:985 holds.

But the hang relocated

Synthetic C# monorepo, N projects × 8 files, 25 cross-project using each, 8 shared type names, 6 shared method names — the reporter's shape. Release binary, one-shot code-index index:

N files tier1r ms tier3 ms tier_partial ms total
10 80 344 175 34 0.70 s
20 160 1 740 884 154 3.04 s
40 320 6 678 2 663 602 10.5 s
80 640 27 847 9 425 2 705 40.9 s
160 1 280 110 267 33 489 12 708 158 s

Log-log exponents: tier1r 2.06, tier_partial 2.12, tier3 step 1.86, total 1.94. Extrapolating to 9 000 files: ≈115 minutes — one core, no I/O, never finishing. The reported symptom.

Tier 1R's plan shows why (index.rs:3307-3337):

|--SCAN rb
|--SEARCH par USING INDEX idx_symbols_name (name=?)   <-- fan-out = # same-named types = O(projects)
|--SEARCH m USING INDEX idx_symbols_parent_name
|--CORRELATED SCALAR SUBQUERY 1   (imports x file_keys, leading-* GLOB)
|--CORRELATED SCALAR SUBQUERY 2   (SEARCH qr USING idx_refs_file, GLOB each)
|--CORRELATED SCALAR SUBQUERY 3   (imports x file_pkg, leading-* GLOB)

The same leading-* GLOB pattern the tier-3 rewrite removed, still correlated — now per (recv_bound × same-named container × same-named method) triple, and that triple count is quadratic. Inner-join cardinality: 115 200 / 460 800 / 1 843 200 at N=20/40/80, exactly ×4 per doubling.

Controlled ablation — the decisive proof

Same N=80 corpus, type names made unique per project. Identical 640 files / 15 360 refs / 16 640 imports / 10 240 symbols:

tier1r:  27 847 ms  ->  335 ms      (83x)

The driver is par.name = rb.ty same-name fan-out. Nothing else.

The budget cannot save you

tier3_work_budget() is referenced only around run_tier3_reachability (index.rs:2636-2641) — it is the sole guard in the entire resolver and it gates only tier 3. Ablation at N=80 with CODE_INDEX_TIER3_WORK_BUDGET=0: tier 3 skipped, tier3_degraded=1 persisted, and tier1r still burned 26 326 ms of the 29.8 s wall.

Worse for this repo specifically: the estimate itself grows quadratically (exponent 1.99) and at 235 projects lands at ≈45 M against the 50 M default — just under. So tier 3 most likely ran rather than degrading, which is why tier3_import_boost_skipped was never observed here.

Why the earlier fix missed it

git show across all four #65 commits (5ac6920, 2fbb837, 150d21a, acae471) → zero hits for recv_bound|recv_calls|pc_calls|cs_scope_fqn. The work was scoped to tier 3 / import_key_rel / tier 1Q. And tier_partial (bb1ddac, C#-only, quadratic on JOIN symbols m ON m.name = c.name at index.rs:3654) landed 2026-08-06, one day after the last #65 commit.

The diagnosability finding is the sharpest part

The #65 heartbeat is installed correctly on the resolve transaction (index.rs:1174-1191, verified live — 49 monotonic lines to elapsed_s=128). But:

  • it only reaches tracing::info! → daemon stderr, and an auto-spawned daemon's stderr is Stdio::null() (mcp-server/src/main.rs:606-608). In exactly the #65 scenario — agent calls a tool, daemon auto-spawns, tool returns warming_up — the heartbeat is discarded;
  • the entire resolve runs inside one uncommitted transaction, so the warming_hint counters (files/symbols/resolved_refs) are frozen for its whole duration by construction;
  • and the hint text tells the operator that frozen counters mean the daemon is stuck.

So the only operator-visible signal actively misdiagnoses working-but-slow as wedged. That is very plausibly why 4 of 5 runs were killed.

Trigger shape, for the regression test

Any repo where (a) a type name is declared in many files/projects, (b) those types carry methods, and (c) call sites use local-variable receivers (var x = new T(); x.M();). C# monorepos are canonical — Program, Startup, Settings, Repository, Logger, IService once per project — and pay tier_partial on top. The other five languages hit tier 1R alone.

Fix order

  1. Materialise tier-1R origin evidence per (file, type) once instead of re-evaluating three correlated GLOB EXISTS per triple — exactly the treatment tier 3 already got.
  2. Extend a work budget (or a cheap pre-count of Σ recv_bound × same-name container fan-out) to tier 1R and tier_partial, not just tier 3.
  3. Make the heartbeat reachable: route the auto-spawned daemon's stderr to a log file, and stop the warming_up hint asserting "frozen ⇒ stuck" while the resolve transaction is open.

Reopening scope accordingly. The Secondary (index size) is addressed in the next comment.

## Correction: the wedge is NOT fixed. It moved to tier 1R. [Comment 4856](https://git.h-dv.de/h-dv/code-index/issues/65#issuecomment-4856) says *"the wedge itself is fixed."* That is true about the site it names and false about the stage that now dominates. Re-verified today against v0.14.0 with a synthetic reproduction, because the claim was asserted by the same work that made the fix. ### Tier 3 really was fixed The leading-`*` GLOBs are gone from tier 3 — `import_key_candidates()` (`index.rs:748`) now enumerates segment runs in Rust and probes a HashMap by equality into `temp.import_key_rel`. `EXPLAIN QUERY PLAN` on all three edge joins plus the aggregate shows no SCAN of a large table and no correlated subquery: ``` T3 EDGE1: |--SCAN g |--SEARCH fdr USING PRIMARY KEY `--SEARCH db USING INDEX ix_dir_bucket (...) T3 EDGE2: |--SCAN g |--SEARCH b USING INDEX ix_symbol_buckets_name `--SEARCH ikr USING PRIMARY KEY (...) ``` `t3_reach` pairs grow **linearly**: 326k → 827k → 1676k at N=20/40/80. The "O(reachable pairs)" claim at `index.rs:985` holds. ### But the hang relocated Synthetic C# monorepo, N projects × 8 files, 25 cross-project `using` each, 8 shared type names, 6 shared method names — the reporter's shape. Release binary, one-shot `code-index index`: | N | files | **tier1r ms** | tier3 ms | tier_partial ms | total | |---|---|---|---|---|---| | 10 | 80 | 344 | 175 | 34 | 0.70 s | | 20 | 160 | 1 740 | 884 | 154 | 3.04 s | | 40 | 320 | 6 678 | 2 663 | 602 | 10.5 s | | 80 | 640 | 27 847 | 9 425 | 2 705 | 40.9 s | | 160 | 1 280 | **110 267** | 33 489 | **12 708** | **158 s** | Log-log exponents: **tier1r 2.06**, tier_partial 2.12, tier3 step 1.86, total **1.94**. Extrapolating to 9 000 files: **≈115 minutes** — one core, no I/O, never finishing. The reported symptom. Tier 1R's plan shows why (`index.rs:3307-3337`): ``` |--SCAN rb |--SEARCH par USING INDEX idx_symbols_name (name=?) <-- fan-out = # same-named types = O(projects) |--SEARCH m USING INDEX idx_symbols_parent_name |--CORRELATED SCALAR SUBQUERY 1 (imports x file_keys, leading-* GLOB) |--CORRELATED SCALAR SUBQUERY 2 (SEARCH qr USING idx_refs_file, GLOB each) |--CORRELATED SCALAR SUBQUERY 3 (imports x file_pkg, leading-* GLOB) ``` The same leading-`*` GLOB pattern the tier-3 rewrite removed, still correlated — now per `(recv_bound × same-named container × same-named method)` triple, and that triple count is quadratic. Inner-join cardinality: **115 200 / 460 800 / 1 843 200** at N=20/40/80, exactly ×4 per doubling. ### Controlled ablation — the decisive proof Same N=80 corpus, type names made unique per project. **Identical** 640 files / 15 360 refs / 16 640 imports / 10 240 symbols: ``` tier1r: 27 847 ms -> 335 ms (83x) ``` The driver is `par.name = rb.ty` same-name fan-out. Nothing else. ### The budget cannot save you `tier3_work_budget()` is referenced **only** around `run_tier3_reachability` (`index.rs:2636-2641`) — it is the sole guard in the entire resolver and it gates only tier 3. Ablation at N=80 with `CODE_INDEX_TIER3_WORK_BUDGET=0`: tier 3 skipped, `tier3_degraded=1` persisted, and **tier1r still burned 26 326 ms of the 29.8 s wall**. Worse for this repo specifically: the estimate itself grows quadratically (exponent 1.99) and at 235 projects lands at **≈45 M against the 50 M default — just under**. So tier 3 most likely *ran* rather than degrading, which is why `tier3_import_boost_skipped` was never observed here. ### Why the earlier fix missed it `git show` across all four #65 commits (`5ac6920`, `2fbb837`, `150d21a`, `acae471`) → **zero** hits for `recv_bound|recv_calls|pc_calls|cs_scope_fqn`. The work was scoped to tier 3 / `import_key_rel` / tier 1Q. And `tier_partial` (`bb1ddac`, C#-only, quadratic on `JOIN symbols m ON m.name = c.name` at `index.rs:3654`) landed **2026-08-06, one day after** the last #65 commit. ### The diagnosability finding is the sharpest part The #65 heartbeat is installed correctly on the resolve transaction (`index.rs:1174-1191`, verified live — 49 monotonic lines to `elapsed_s=128`). But: - it only reaches `tracing::info!` → daemon stderr, and an auto-spawned daemon's stderr is **`Stdio::null()`** (`mcp-server/src/main.rs:606-608`). In exactly the #65 scenario — agent calls a tool, daemon auto-spawns, tool returns `warming_up` — the heartbeat is discarded; - the entire resolve runs inside **one uncommitted transaction**, so the `warming_hint` counters (`files`/`symbols`/`resolved_refs`) are frozen for its whole duration **by construction**; - and the hint text tells the operator that frozen counters mean the daemon is stuck. So the only operator-visible signal actively misdiagnoses working-but-slow as wedged. That is very plausibly why 4 of 5 runs were killed. ### Trigger shape, for the regression test Any repo where (a) a type name is declared in many files/projects, (b) those types carry methods, and (c) call sites use local-variable receivers (`var x = new T(); x.M();`). C# monorepos are canonical — `Program`, `Startup`, `Settings`, `Repository`, `Logger`, `IService` once per project — and pay `tier_partial` on top. The other five languages hit tier 1R alone. ### Fix order 1. Materialise tier-1R origin evidence per `(file, type)` once instead of re-evaluating three correlated GLOB `EXISTS` per triple — exactly the treatment tier 3 already got. 2. Extend a work budget (or a cheap pre-count of `Σ recv_bound × same-name container fan-out`) to tier 1R and tier_partial, not just tier 3. 3. Make the heartbeat reachable: route the auto-spawned daemon's stderr to a log file, and stop the `warming_up` hint asserting "frozen ⇒ stuck" while the resolve transaction is open. Reopening scope accordingly. The *Secondary* (index size) is addressed in the next comment.
buildagent commented 2026-08-18 15:18:42 +02:00
Author
Member
Copy link

Secondary (index size) — the remaining hypothesis is refuted. It is average file size, not C#.

Comment 4857 correctly killed the issue body's "ref rows are the size driver" hypothesis and replaced it with a new one:

The C# case is therefore ~3.3× heavier per file than django, which suggests something specific to that repo or to the C# plugin rather than a general per-ref cost — worth isolating before optimising anything.

Isolated. There is nothing C#-specific. KB-per-file is a file-size-distribution artifact, and the 3.3× is entirely explained by average file size.

Measured on three real repos, v0.14.0

repo files avg source file expansion KB/file FTS share
cosi-mcp 294 14.7 KB 6.90× 101.1 71.2%
a1x 1 650 12.8 KB 5.24× 67.1 67.5%
brain-mcp 947 8.3 KB 5.87× 48.8 62.9%

Expansion (index bytes ÷ source bytes) is roughly constant at 5.2–6.9×. KB-per-file varies 2× across the same three repos — purely with average file size, because the two are related by construction:

KB/file  =  expansion  ×  avg_file_size

cosi-mcp: 14.7 KB × 6.90 = 101.1 ✓ · brain-mcp: 8.3 KB × 5.87 = 48.8 ✓

Applying that to the two reported numbers

Divide each by the measured expansion:

expansion implied avg C# file implied avg django file ratio
5.24× 21.9 KB 6.7 KB 3.30×
6.00× 19.2 KB 5.8 KB 3.30×
6.90× 16.7 KB 5.1 KB 3.30×

The ratio is invariant, because it is the same constant on both sides. ~19 KB average C# files against ~5.8 KB average django files is entirely ordinary — C# is verbose and django ships many small modules.

Cross-check against the reported total: 9 000 files × 19.2 KB × 6.0 = 1 012 MB. The reported 1.04 GB is precisely what this design produces at that file-size profile. No anomaly, no C# bug, nothing to isolate.

Where the bytes actually are, and why it is not obviously fixable

files_fts_data — the trigram index — is 56–63% of every index measured; FTS total is 63–71%. Source text expands into trigrams roughly 3.8× on cosi-mcp (4.44 MB source → 17.0 MB trigram data), plus a ~1× stored content copy.

Every lever trades away the capability a real user measured as this tool's win:

  • A cheaper tokenizer ends substring/subword matching. The 2026-08-18 customer assessment's only measured win was search_text finding 11 hits a hand-scoped grep missed, and TODO matching ToDouble is documented behaviour.
  • detail=none / detail=column would shrink files_fts_data substantially but break snippet() (local_index.rs:1782) and phrase queries.
  • Contentless FTS5 (the ~15–19% content copy) was already evaluated and rejected — see _prdoc/records/I001-A001-fts5-contentless.md: it cannot delete or replace a single row, which breaks per-file incremental update.
  • A size cap on FTS'd files loses searchability of exactly the largest files, which are usually the ones worth searching.

So the honest position is that ~6× expansion is structural to a substring-searchable index, not a defect. If index size must come down, it is a product decision to weaken search_text, and should be filed as that rather than as a perf bug.

One separate finding, not the reporter's problem

A long-lived, watcher-updated index accumulates free pages that are never returned to the OS:

cosi-mcp (long-lived):  62.8 MB file / 30.4 MB live / 32.4 MB free  (51.6%), auto_vacuum=0
a1x      (fresh):      113.8 MB file / 113.5 MB live / 0.4 MB free

This is already known and documented — wal_maintenance.rs:50-57 states that optimize returns pages to the freelist where they are reused but the .db file does not shrink, and that only VACUUM would. Recording the measurement because the 2× gap is larger than that note implies.

It does not explain this issue: the reported index was fresh, and fresh indexes carry essentially no freelist.

Recommendation

Split the Secondary out of #65 entirely. It is now measured, explained, and has no defect in it — what remains is a deliberate size/searchability trade-off worth its own issue if anyone wants to reopen the decision. #65 should carry only the wedge, which per the previous comment is still live in tier 1R.

## Secondary (index size) — the remaining hypothesis is refuted. It is average file size, not C#. [Comment 4857](https://git.h-dv.de/h-dv/code-index/issues/65#issuecomment-4857) correctly killed the issue body's "ref rows are the size driver" hypothesis and replaced it with a new one: > The C# case is therefore ~3.3× heavier per file than django, which suggests something specific to that repo or to the C# plugin rather than a general per-ref cost — worth isolating before optimising anything. Isolated. There is nothing C#-specific. **KB-per-file is a file-size-distribution artifact**, and the 3.3× is entirely explained by average file size. ### Measured on three real repos, v0.14.0 | repo | files | avg source file | **expansion** | KB/file | FTS share | |---|---|---|---|---|---| | cosi-mcp | 294 | 14.7 KB | 6.90× | 101.1 | 71.2% | | a1x | 1 650 | 12.8 KB | 5.24× | 67.1 | 67.5% | | brain-mcp | 947 | 8.3 KB | 5.87× | 48.8 | 62.9% | Expansion (index bytes ÷ source bytes) is roughly **constant at 5.2–6.9×**. KB-per-file varies **2×** across the same three repos — purely with average file size, because the two are related by construction: ``` KB/file = expansion × avg_file_size ``` cosi-mcp: 14.7 KB × 6.90 = 101.1 ✓ · brain-mcp: 8.3 KB × 5.87 = 48.8 ✓ ### Applying that to the two reported numbers Divide each by the measured expansion: | expansion | implied avg C# file | implied avg django file | ratio | |---|---|---|---| | 5.24× | 21.9 KB | 6.7 KB | **3.30×** | | 6.00× | 19.2 KB | 5.8 KB | **3.30×** | | 6.90× | 16.7 KB | 5.1 KB | **3.30×** | The ratio is invariant, because it is the same constant on both sides. ~19 KB average C# files against ~5.8 KB average django files is entirely ordinary — C# is verbose and django ships many small modules. Cross-check against the reported total: 9 000 files × 19.2 KB × 6.0 = **1 012 MB**. The reported 1.04 GB is precisely what this design produces at that file-size profile. **No anomaly, no C# bug, nothing to isolate.** ### Where the bytes actually are, and why it is not obviously fixable `files_fts_data` — the trigram index — is 56–63% of every index measured; FTS total is 63–71%. Source text expands into trigrams roughly **3.8×** on cosi-mcp (4.44 MB source → 17.0 MB trigram data), plus a ~1× stored content copy. Every lever trades away the capability a real user measured as this tool's win: - **A cheaper tokenizer** ends substring/subword matching. The 2026-08-18 customer assessment's *only* measured win was `search_text` finding 11 hits a hand-scoped grep missed, and `TODO` matching `ToDouble` is documented behaviour. - **`detail=none` / `detail=column`** would shrink `files_fts_data` substantially but break `snippet()` (`local_index.rs:1782`) and phrase queries. - **Contentless FTS5** (the ~15–19% content copy) was already evaluated and rejected — see `_prdoc/records/I001-A001-fts5-contentless.md`: it cannot delete or replace a single row, which breaks per-file incremental update. - **A size cap on FTS'd files** loses searchability of exactly the largest files, which are usually the ones worth searching. So the honest position is that ~6× expansion is structural to a substring-searchable index, not a defect. If index size must come down, it is a **product decision to weaken `search_text`**, and should be filed as that rather than as a perf bug. ### One separate finding, not the reporter's problem A long-lived, watcher-updated index accumulates free pages that are never returned to the OS: ``` cosi-mcp (long-lived): 62.8 MB file / 30.4 MB live / 32.4 MB free (51.6%), auto_vacuum=0 a1x (fresh): 113.8 MB file / 113.5 MB live / 0.4 MB free ``` This is **already known and documented** — `wal_maintenance.rs:50-57` states that `optimize` returns pages to the freelist where they are reused but the `.db` file does not shrink, and that only `VACUUM` would. Recording the measurement because the 2× gap is larger than that note implies. It does **not** explain this issue: the reported index was fresh, and fresh indexes carry essentially no freelist. ### Recommendation Split the *Secondary* out of #65 entirely. It is now measured, explained, and has no defect in it — what remains is a deliberate size/searchability trade-off worth its own issue if anyone wants to reopen the decision. #65 should carry only the wedge, which per the previous comment is still live in tier 1R.
buildagent changed title from perf/hang: daemon resolve stage wedges on large cross-project C# repos (tier-3 import-boost GLOB cross-product) to perf/reliability: tier 1R and C# partial resolution scale quadratically and hide progress 2026-08-26 13:38:05 +02:00
buildagent commented 2026-09-04 15:20:03 +02:00
Author
Member
Copy link

Closing: this shipped in v0.23.0. The issue text describes pre-fix code.

Audited against the tree at 4555887, and every citation below was opened and read rather than grepped for.

crates/indexer/src/resolve_budget.rs was created by 653324e ("I064: extract the resolver stage-budget mechanism (#65)"); git tag --contains → v0.23.0.

Five stages are budgeted, against the four this issue named (resolve_budget.rs:130-136):

pub(crate) const ALL: &'static [Stage] = &[
    Stage::Tier3, Stage::Tier1rOrigin, Stage::Tier1rIdent,
    Stage::CsharpPartial, Stage::Tier1qScands,
];

with five distinct guard sites at crates/indexer/src/index.rs:5290, 5720, 6407, 6477, 6837.

Progress is agent-visible, not log-only. It is a real RPC (crates/daemon/src/server.rs:357, 650) that rides on project_overview (crates/mcp-server/src/server.rs:7769), so an agent watching a slow index sees it move. warming_hint (server.rs:1686) carries an explicit // #65: comment removing this issue's "frozen counters ⇒ wedged" reading.

Logs remain recoverable, which this issue required: the auto-spawned daemon's stderr is still Stdio::null(), but the daemon tees to a rotating log (crates/daemon/src/main.rs:83-89, 1286-1326), covered by crates/daemon/tests/daemon_log_e2e.rs.

Worth noting what Stage::ALL honestly does not do — the doc says it itself, and says so because an earlier version of that doc overclaimed:

It does NOT force a call site or a corpus fixture — an earlier version of this doc claimed it did, and a Stage::Ghost with neither passed the whole suite.

That is the right kind of comment and it is why this close can be trusted: the registry's own limits are written down where the next reader will hit them.

One residual, split out rather than left buried

Acceptance item 7 — "dynamic resolver capabilities cannot bypass the same budgets" — is a decision, not an oversight, but nothing grades it. The bridge tier (index.rs:~6890) is deliberately unguarded, with a written justification: it does not run unless a package declared and the project granted a bridge, and its populations are one component's refs of one kind × one language's symbols of one kind. Stage has no bridge variant, so a bridge fan-out cannot degrade-with-disclosure.

Filed as its own issue so the argument is graded rather than trusted.

Also stale, flagged for whoever reads it next

_prdoc/missions/I064-resolver-fanout-and-progress.md still says "unreleased — the branch ships when #80's gate is green". Master carries and has released all of it.

Why this close matters beyond bookkeeping

#80 quotes this issue as live grounds — "#65 resolver fan-out can still monopolize a daemon" — and that sentence is false today. Leaving it open is part of what makes #80's gate unreadable. This is the same failure #85 caused: an open issue read as evidence its defect is open, and a lane nearly spent rebuilding something that already existed.

## Closing: this shipped in **v0.23.0**. The issue text describes pre-fix code. Audited against the tree at `4555887`, and every citation below was opened and read rather than grepped for. `crates/indexer/src/resolve_budget.rs` was created by `653324e` ("I064: extract the resolver stage-budget mechanism (#65)"); `git tag --contains` → **v0.23.0**. **Five stages are budgeted, against the four this issue named** (`resolve_budget.rs:130-136`): ```rust pub(crate) const ALL: &'static [Stage] = &[ Stage::Tier3, Stage::Tier1rOrigin, Stage::Tier1rIdent, Stage::CsharpPartial, Stage::Tier1qScands, ]; ``` with five distinct guard sites at `crates/indexer/src/index.rs:5290, 5720, 6407, 6477, 6837`. **Progress is agent-visible, not log-only.** It is a real RPC (`crates/daemon/src/server.rs:357, 650`) that rides on `project_overview` (`crates/mcp-server/src/server.rs:7769`), so an agent watching a slow index sees it move. `warming_hint` (`server.rs:1686`) carries an explicit `// #65:` comment removing this issue's "frozen counters ⇒ wedged" reading. **Logs remain recoverable**, which this issue required: the auto-spawned daemon's stderr is still `Stdio::null()`, but the daemon tees to a rotating log (`crates/daemon/src/main.rs:83-89, 1286-1326`), covered by `crates/daemon/tests/daemon_log_e2e.rs`. Worth noting what `Stage::ALL` honestly does **not** do — the doc says it itself, and says so because an earlier version of that doc overclaimed: > It does NOT force a call site or a corpus fixture — an earlier version of this doc claimed it did, and a `Stage::Ghost` with neither passed the whole suite. That is the right kind of comment and it is why this close can be trusted: the registry's own limits are written down where the next reader will hit them. ### One residual, split out rather than left buried Acceptance item 7 — "dynamic resolver capabilities cannot bypass the same budgets" — is **a decision, not an oversight**, but nothing grades it. The bridge tier (`index.rs:~6890`) is deliberately unguarded, with a written justification: it does not run unless a package declared *and* the project granted a bridge, and its populations are one component's refs of one kind × one language's symbols of one kind. `Stage` has no bridge variant, so a bridge fan-out cannot degrade-with-disclosure. Filed as its own issue so the argument is graded rather than trusted. ### Also stale, flagged for whoever reads it next `_prdoc/missions/I064-resolver-fanout-and-progress.md` still says *"unreleased — the branch ships when #80's gate is green"*. Master carries and has released all of it. ### Why this close matters beyond bookkeeping **#80 quotes this issue as live grounds** — *"#65 resolver fan-out can still monopolize a daemon"* — and that sentence is false today. Leaving it open is part of what makes #80's gate unreadable. This is the same failure #85 caused: an open issue read as evidence its defect is open, and a lane nearly spent rebuilding something that already existed.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
p1-integration
p0-integration
release/v0.32.2
release/v0.32.1
epic-i072-release-032
windows-installer
i269b-registry-truth
probe-win-baseline
fix/246-shadowed-imports
release/0.28.1
fix/review-followthrough-20260911
iwin-msvc-archive
v0.32.2
v0.32.1
v0.32.0
v0.31.2
v0.31.1
v0.31.0
v0.30.1
v0.30.0
v0.29.0
v0.28.3
v0.28.2
v0.28.1
v0.28.0
v0.27.1
v0.27.0
v0.26.1
v0.26.0
v0.25.0
v0.24.1
v0.24.0
v0.23.1
v0.23.0
v0.22.3
v0.22.2
v0.22.1
v0.22.0
v0.21.0
v0.20.3
v0.20.2
v0.20.1
v0.20.0
v0.19.5
v0.19.4
v0.19.3
v0.19.2
v0.19.1
v0.19.0
v0.18.1
v0.18.0
v0.17.0
v0.16.0
v0.15.3
v0.14.0
v0.13.3
v0.13.2
v0.13.1
v0.13.0
v0.12.0
v0.11.7
v0.11.6
v0.11.5
v0.11.4
v0.11.3
v0.11.2
v0.11.1
v0.11.0
v0.10.2
v0.10.1
v0.10.0
v0.9.0
v0.8.7
v0.8.6
v0.8.5
v0.8.4
v0.8.3
v0.8.2
v0.8.1
v0.8.0
v0.7.1
v0.7.0
v0.6.1
v0.6.0
v0.5.21
v0.5.20
v0.5.19
v0.5.18
v0.5.17
v0.5.16
v0.5.15
v0.5.14
v0.5.13
v0.5.12
v0.5.11
v0.5.10
v0.5.9
v0.5.8
v0.5.7
v0.5.6
v0.5.5
v0.5.4
v0.5.3
v0.5.2
v0.5.1
v0.5.0
v0.4.2
v0.4.1
v0.4.0
v0.3.3
v0.3.2
v0.3.1
v0.3.0
v0.2.0
Labels
Clear labels   
code-review

   
correctness

   
dos

   
performance

   
security

   
severity/high

   
severity/low

   
severity/medium

   
tech-debt
  
Kind/Breaking
Breaking change that won't be backward compatible

  
Kind/Bug
Something is not working

  
Kind/Documentation
Documentation changes

  
Kind/Enhancement
Improve existing functionality

  
Kind/Feature
New functionality

  
Kind/Security
This is security issue

  
Kind/Testing
Issue or pull request related to testing

  
Priority
Critical
 The priority is critical

  
Priority
High
 The priority is high

  
Priority
Low
 The priority is low

  
Priority
Medium
 The priority is medium

  
Reviewed
Confirmed
 Issue has been confirmed

  
Reviewed
Duplicate
 This issue or pull request already exists

  
Reviewed
Invalid
 Invalid issue

  
Reviewed
Won't Fix
 This issue won't be fixed

  
Status
Abandoned
 Somebody has started to work on this but abandoned work

  
Status
Blocked
 Something is blocking this issue or pull request

  
Status
Need More Info
 Feedback is required to reproduce issue or to continue work

No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#41 test: scale ceilings for graph caps, resolver fan-out and dynamic plugin generations
h-dv/code-index
#53 perf: bound residual resolver superlinearity across builtin and dynamic profiles
h-dv/code-index
#77 resolver: row-level provenance, capability-gated pools and explicit language bridges
h-dv/code-index
#80 gate: plugin conformance, hostile-runtime containment and production disclosure
h-dv/code-index
Reference
h-dv/code-index#65
No description provided.