h-dv/code-index

Fork 0

test: tier-3 scale ceilings — 20k+ file repos, 1M-edge cap, peak RSS, query p99, FTS growth #41

New issue

Open

opened 2026-07-28 20:38:02 +02:00 by buildagent · 1 comment

buildagent commented

2026-07-28 20:38:02 +02:00

Member

P1 — the second axis the 2026-07-28 spike could not reach.

From _prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md §0.

Why

Everything measured in the spike was 170–580 files. At that size the indexer is clean on every axis tested. The remaining risk is entirely at scale, and it is currently wholly unmeasured:

Unmeasured	Why it matters
1M-edge cap	shipped in I029b; never exercised — what happens at the boundary is untested behaviour, and the cap is user-visible in `graph_semantics`
peak RSS	no ceiling asserted anywhere; a 50k-file repo could OOM a user's machine
query p50/p99	`bench_find_references.rs` / `bench_search_symbols.rs` benchmark 4–14-file fixtures, i.e. noise
FTS growth	trigram index size vs source bytes — unknown scaling curve
DB bytes / source KB	guards against schema bloat across migrations
pagerank at scale	behaviour on a large graph never observed
index wall-clock	spike range 0.4–3.4s at ~200 files; scaling unknown

Spike reference points (tier-1, for the ratchet baseline): ripgrep 231 files → 4127 symbols / 36607 refs / 1.5s; ts-zod 558 files → 8489 symbols / 85027 refs / 3.4s. Note zod already produces 172 refs/file — a 20k-file repo of that density is ~3.4M refs.

What to build

add tier-3 repos to corpus.toml (#40) — candidates: rust-analyzer, django, vscode, roslyn (verify size/licence before pinning)
measure and record per repo: wall-clock, peak RSS, DB bytes, FTS bytes, symbol/ref/edge counts, query p50/p99 for the hot tools
assert ceilings (hard gate): no OOM, no hang, completes within a per-repo budget
ratchet the rest against a committed baseline (#45)
deliberately include a repo that crosses the 1M-edge cap so the cap path is executed and its disclosure verified

Acceptance

≥2 tier-3 repos indexed in CI (weekly cadence)
peak RSS + wall-clock hard ceilings asserted
1M-edge cap path actually exercised and its graph_semantics disclosure verified
p50/p99 recorded for find_callers/find_references/search_symbols/search_text on a large graph
all metrics land in the ratchet baseline

**P1 — the second axis the 2026-07-28 spike could not reach.** From `_prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md` §0. ## Why Everything measured in the spike was **170–580 files**. At that size the indexer is clean on every axis tested. The remaining risk is entirely at scale, and it is currently **wholly unmeasured**: | Unmeasured | Why it matters | |---|---| | 1M-edge cap | shipped in I029b; never exercised — what happens at the boundary is untested behaviour, and the cap is user-visible in `graph_semantics` | | peak RSS | no ceiling asserted anywhere; a 50k-file repo could OOM a user's machine | | query p50/p99 | `bench_find_references.rs` / `bench_search_symbols.rs` benchmark 4–14-file fixtures, i.e. noise | | FTS growth | trigram index size vs source bytes — unknown scaling curve | | DB bytes / source KB | guards against schema bloat across migrations | | pagerank at scale | behaviour on a large graph never observed | | index wall-clock | spike range 0.4–3.4s at ~200 files; scaling unknown | Spike reference points (tier-1, for the ratchet baseline): ripgrep 231 files → 4127 symbols / 36607 refs / 1.5s; ts-zod 558 files → 8489 symbols / **85027 refs** / 3.4s. Note zod already produces 172 refs/file — a 20k-file repo of that density is ~3.4M refs. ## What to build - add tier-3 repos to `corpus.toml` (#40) — candidates: rust-analyzer, django, vscode, roslyn (verify size/licence before pinning) - measure and record per repo: wall-clock, peak RSS, DB bytes, FTS bytes, symbol/ref/edge counts, query p50/p99 for the hot tools - **assert ceilings** (hard gate): no OOM, no hang, completes within a per-repo budget - **ratchet** the rest against a committed baseline (#45) - deliberately include a repo that crosses the 1M-edge cap so the cap path is executed and its disclosure verified ## Acceptance - [ ] ≥2 tier-3 repos indexed in CI (weekly cadence) - [ ] peak RSS + wall-clock hard ceilings asserted - [ ] 1M-edge cap path actually exercised and its `graph_semantics` disclosure verified - [ ] p50/p99 recorded for find_callers/find_references/search_symbols/search_text on a large graph - [ ] all metrics land in the ratchet baseline

buildagent added the

labels

2026-07-28 20:40:43 +02:00

buildagent referenced this issue

2026-07-28 20:52:17 +02:00

bug: resolution is STICKY across incremental runs — renaming an anchor file leaves refs resolved that a cold index leaves unresolved #52

buildagent referenced this issue

2026-07-28 22:06:08 +02:00

perf: cold-index throughput degrades ~5-6x from tier-1 to tier-3 repos (time scales ~refs^1.7) #53

buildagent commented

2026-07-28 22:06:30 +02:00

Author

Member

Landed — `crates/indexer/tests/corpus_scale.rs` + weekly `corpus-scale` CI job

Two tier-3 repos added to the manifest, both permissive and sha-pinned: rust-analyzer (MIT OR Apache-2.0) and django (BSD-3-Clause).

scale[rust-analyzer]: 1778 files / 35461 sym / 340676 refs / 80397 edges / 115997 resolved (34.0%)
  wall=77.1s (4417 refs/s)  db=106.6 MiB  peak_rss=232 MiB
  find_callers_shaped p50=80us p99=321us | search_symbols_prefix p50=118us p99=579us
  search_text_fts p50=1177us p99=4286us

scale[py-django]:     4235 files / 44038 sym / 460978 refs / 52615 edges /  76787 resolved (16.7%)
  wall=98.6s (4673 refs/s)  db=145.4 MiB  peak_rss=300 MiB
  find_callers_shaped p50=80us p99=248us | search_symbols_prefix p50=43us p99=223us
  search_text_fts p50=713us p99=1539us

What was measured, and what it says

Peak RSS 232–300 MiB. An order of magnitude below the 8 GiB I initially set as a ceiling — so I tightened the gate to 2 GiB, since a multi-gigabyte reading would now indicate something genuinely broken rather than a slow machine. No OOM risk on repos of this size.
Query latency is fine at scale. find_callers-shaped p99 ≤ 321µs on a 341k-ref graph; FTS trigram p99 ≤ 4.3ms. The index shape stays queryable.
Cold-index throughput does NOT hold up — 4.4k refs/s here vs ~24k at tier-1, i.e. time ~ refs^1.7. Filed separately as #53; it is a scaling characteristic, not a correctness defect, and it deserves its own investigation rather than being buried in this issue.

Gates vs records

Hard gates: no panic, no hang, non-empty index, wall clock < 900s, peak RSS < 2 GiB. These catch a hang or a pathological blow-up — the actual risk.

Recorded, not gated: wall clock, RSS, DB bytes, counts, query p50/p99 → target/corpus/scale-<repo>.json, seeding the #45 ratchet. Asserting thresholds on these here would just encode this machine's speed.

The 1M-edge cap is still NOT exercised

80 397 and 52 615 edges — far below it. The test reports where each repo sits relative to the cap rather than pretending to test it, and the cap itself lives in the daemon's graph layer, not the indexer, so exercising it needs a daemon-side test with a repo an order of magnitude larger. That acceptance box stays open.

CI

corpus-scale runs Sundays 04:00 and on manual dispatch. fetch.sh now takes COSI_CORPUS_TIERS, so the nightly tier-1 job no longer clones these multi-thousand-file repos into an ephemeral container every night.

Acceptance: boxes 1, 2, 4, 5 met; box 3 (1M-edge cap) explicitly unmet and explained.

## Landed — `crates/indexer/tests/corpus_scale.rs` + weekly `corpus-scale` CI job Two tier-3 repos added to the manifest, both permissive and sha-pinned: **rust-analyzer** (MIT OR Apache-2.0) and **django** (BSD-3-Clause). ``` scale[rust-analyzer]: 1778 files / 35461 sym / 340676 refs / 80397 edges / 115997 resolved (34.0%) wall=77.1s (4417 refs/s) db=106.6 MiB peak_rss=232 MiB find_callers_shaped p50=80us p99=321us | search_symbols_prefix p50=118us p99=579us search_text_fts p50=1177us p99=4286us scale[py-django]: 4235 files / 44038 sym / 460978 refs / 52615 edges / 76787 resolved (16.7%) wall=98.6s (4673 refs/s) db=145.4 MiB peak_rss=300 MiB find_callers_shaped p50=80us p99=248us | search_symbols_prefix p50=43us p99=223us search_text_fts p50=713us p99=1539us ``` ## What was measured, and what it says - **Peak RSS 232–300 MiB.** An order of magnitude below the 8 GiB I initially set as a ceiling — so I tightened the gate to 2 GiB, since a multi-gigabyte reading would now indicate something genuinely broken rather than a slow machine. **No OOM risk on repos of this size.** - **Query latency is fine at scale.** `find_callers`-shaped p99 ≤ 321µs on a 341k-ref graph; FTS trigram p99 ≤ 4.3ms. The index *shape* stays queryable. - **Cold-index throughput does NOT hold up** — 4.4k refs/s here vs ~24k at tier-1, i.e. `time ~ refs^1.7`. Filed separately as **#53**; it is a scaling characteristic, not a correctness defect, and it deserves its own investigation rather than being buried in this issue. ## Gates vs records **Hard gates:** no panic, no hang, non-empty index, wall clock < 900s, peak RSS < 2 GiB. These catch a hang or a pathological blow-up — the actual risk. **Recorded, not gated:** wall clock, RSS, DB bytes, counts, query p50/p99 → `target/corpus/scale-<repo>.json`, seeding the #45 ratchet. Asserting thresholds on these here would just encode this machine's speed. ## The 1M-edge cap is still NOT exercised 80 397 and 52 615 edges — far below it. The test reports where each repo sits relative to the cap rather than pretending to test it, and the cap itself lives in the **daemon's** graph layer, not the indexer, so exercising it needs a daemon-side test with a repo an order of magnitude larger. **That acceptance box stays open.** ## CI `corpus-scale` runs Sundays 04:00 and on manual dispatch. `fetch.sh` now takes `COSI_CORPUS_TIERS`, so the nightly tier-1 job no longer clones these multi-thousand-file repos into an ephemeral container every night. Acceptance: boxes 1, 2, 4, 5 met; box 3 (1M-edge cap) explicitly unmet and explained.

buildagent referenced this issue

2026-07-29 21:26:30 +02:00

I039: OSS corpus test system + the four defects it found (v0.9.0, schema v25) #58

buildagent referenced this issue from a commit

2026-08-03 14:06:57 +02:00

test: OSS corpus test system — metamorphic, mutation, live-watcher replay (#39-#44)

buildagent referenced this issue from a commit

2026-08-03 14:06:57 +02:00

test: tier-3 scale ceilings over rust-analyzer and django (#41)