v0.41.52: startup cost no longer grows with the package store (#581) #583

Merged

buildagent merged 7 commits from integration-1113 into master 2026-11-13 12:22:37 +01:00

Conversation 0 Commits 7 Files changed 12 +328 -39

buildagent commented 2026-11-13 01:03:28 +01:00

Member

Copy link

Fix-forward release for #581. v0.41.51 is tagged on master but was never published: its release run stopped in the Windows archive smoke on this defect. This branch is master plus the fix and the version bump.

The defect (#581)

Since f73682d5 (first shipped in v0.41.49) the answer "which language names are available" read every archive in the user's package store and computed two content hashes for each. One process asked about 22 times, mostly through Config::warnings(), so every start of a process paid for every installed package, used or not.

step, shipped build shape, fresh project, no daemon	v0.41.48	master	this branch
seven packages: languages enable / index / one-shot query	14.2 / 3.7 / 1.7 s	28.2 / 10.3 / 9.1 s	12.5 / 3.4 / 1.5 s
45 packages (141 MiB): languages enable / index / one-shot query	13.6 / 3.6 / 1.4 s	132.7 / 29.4 / 17–19 s, first query refused index_incomplete	12.9 / 3.4 / 1.5 s

Measured by the lead on one machine with the same script; the 45-package store is a copy of this machine's installed packages.

The fix

• The availability answer is advisory text. It now reads only a package's container index and manifest (Store::declared_language_names), never the payload, and computes no hash. It reads fresh on every call, so a package installed, replaced or removed while a daemon runs changes the answer (test advisory_names_follow_installs_replacements_and_removals, mutation run).
• It is not computed when every enabled name is one this build owns, nor when warnings cannot be emitted (log level above warn). PackageSet resolves the known names lazily, as owner_rows_in already did.
• The verified reads are unchanged: Store::get, signature, trust, approval and enablement checks still read and hash every byte of a package that is used.
• A first candidate (a per-process memo of parse results keyed on the archive bytes) was rejected in review: it still read every archive on each round and retained every installed archive in a long-lived daemon. It is not in the final tree.

Gate

unused_packages_add_no_archive_hashes_or_full_reads (crates/cli/tests/plugin_activation_cli.rs) counts archive reads and metadata parses in one single-process startup and one index, with a real loadable package and a non-builtin language enabled, before and after installing eight unused packages. It asserts that the unused packages add nothing and are never read, and that the one used package stays within four reads per startup and three per index. No wall-clock bound.

• Mutation run: the old census restored, 20 → 148 parses, red.
• Mutation run: a second parse in Store::get_unverified_bytes, 8 vs 4, red.

Release smoke (tests/release_language_smoke.ps1)

• Prints the elapsed seconds of each step, so two release runs can be compared.
• A call refused with "retryable": true is retried at most four times within 60 s and the script says so; every other failure stays fatal.
• Stops a daemon the run may have started (PID taken from the scratch project's registration, executable and root verified) on every exit path. A cleanup failure is reported and does not replace the original error.
• Run by the lead in the mcr.microsoft.com/powershell image against the fixed Linux build: passed, query step 1.46 s. Windows is covered by the release run only.

Also

• m2_acceptance gives its fixture an own CODE_INDEX_HOME; before, the benchmark's time depended on the packages installed on the machine that ran it.

Qualification

• Lane: indexer targets, code-index-cli, code-index-mcp on both legs, fmt, clippy, rustdoc: all exit 0 (worker run, logs kept). Lead: counting gate and its mutation, both timing tables, the smoke script.
• The one full local gate ran on 172bcb20 (01:03–03:22, 43 steps): all 43 passed and no record moved (git status clean; tests/corpus/*.json, tests/bench/ratchet.json and the package parity files unchanged). It covers formatting, fast gates, clippy for the host and x86_64-pc-windows-gnu, rustdoc, the workspace and MCP suites on both legs with their timing profiles, precision_gate 7/7 with seven population lines, the CI corpus suites, the eleven package legs, scale and tier-3 corpus, packages-only and shipped-shape acceptance, and the nightly-only cost steps including language_package_overhead (3 passed).
• CI on the head: the package, pilot-plugin and Windows workflows passed. ci.yml run 8637 passed all 16 jobs; it was dispatched by hand, so it also ran the nightly-only jobs (tier-3 scale, grammar rebuild, plugin path cost with language_package_overhead 3 passed). Ten earlier runs on this head tested nothing: the runners could not resolve index.crates.io, static.rust-lang.org or code.forgejo.org for about eight hours (#582). One run in between (8631) failed in a single test, sfc_first_contact_delegates_script_languages_and_refuses_only_unavailable_regions: the index was still reconciling after 60 s. The same commit passed that test in run 8637 and in every local run (36 s alone, 102 s with the machine oversubscribed two-fold; the previous master takes 144 s). The cause of that one failure is not known; it is recorded on #582.

Not in this PR

• The scheduled runs are red for other reasons as well (#582). The package overhead job is dispatched on the fixed tree after the merge to see which of its budgets this fix restores.
• The CLI has no supported command to stop a daemon; the smoke script stops it by verified PID.

🤖 Generated with Claude Code

Fix-forward release for #581. `v0.41.51` is tagged on master but was never published: its release run stopped in the Windows archive smoke on this defect. This branch is master plus the fix and the version bump. ## The defect (#581) Since `f73682d5` (first shipped in v0.41.49) the answer "which language names are available" read every archive in the user's package store and computed two content hashes for each. One process asked about 22 times, mostly through `Config::warnings()`, so every start of a process paid for every installed package, used or not. | step, shipped build shape, fresh project, no daemon | v0.41.48 | master | this branch | |---|---:|---:|---:| | seven packages: `languages enable` / `index` / one-shot query | 14.2 / 3.7 / 1.7 s | 28.2 / 10.3 / 9.1 s | 12.5 / 3.4 / 1.5 s | | 45 packages (141 MiB): `languages enable` / `index` / one-shot query | 13.6 / 3.6 / 1.4 s | 132.7 / 29.4 / 17–19 s, first query refused `index_incomplete` | 12.9 / 3.4 / 1.5 s | Measured by the lead on one machine with the same script; the 45-package store is a copy of this machine's installed packages. ## The fix - The availability answer is advisory text. It now reads only a package's container index and manifest (`Store::declared_language_names`), never the payload, and computes no hash. It reads fresh on every call, so a package installed, replaced or removed while a daemon runs changes the answer (test `advisory_names_follow_installs_replacements_and_removals`, mutation run). - It is not computed when every enabled name is one this build owns, nor when warnings cannot be emitted (log level above `warn`). `PackageSet` resolves the known names lazily, as `owner_rows_in` already did. - The verified reads are unchanged: `Store::get`, signature, trust, approval and enablement checks still read and hash every byte of a package that is used. - A first candidate (a per-process memo of parse results keyed on the archive bytes) was rejected in review: it still read every archive on each round and retained every installed archive in a long-lived daemon. It is not in the final tree. ## Gate `unused_packages_add_no_archive_hashes_or_full_reads` (crates/cli/tests/plugin_activation_cli.rs) counts archive reads and metadata parses in one single-process startup and one `index`, with a real loadable package and a non-builtin language enabled, before and after installing eight unused packages. It asserts that the unused packages add nothing and are never read, and that the one used package stays within four reads per startup and three per `index`. No wall-clock bound. - Mutation run: the old census restored, 20 → 148 parses, red. - Mutation run: a second parse in `Store::get_unverified_bytes`, 8 vs 4, red. ## Release smoke (`tests/release_language_smoke.ps1`) - Prints the elapsed seconds of each step, so two release runs can be compared. - A call refused with `"retryable": true` is retried at most four times within 60 s and the script says so; every other failure stays fatal. - Stops a daemon the run may have started (PID taken from the scratch project's registration, executable and root verified) on every exit path. A cleanup failure is reported and does not replace the original error. - Run by the lead in the `mcr.microsoft.com/powershell` image against the fixed Linux build: passed, query step 1.46 s. Windows is covered by the release run only. ## Also - `m2_acceptance` gives its fixture an own `CODE_INDEX_HOME`; before, the benchmark's time depended on the packages installed on the machine that ran it. ## Qualification - Lane: indexer targets, `code-index-cli`, `code-index-mcp` on both legs, fmt, clippy, rustdoc: all exit 0 (worker run, logs kept). Lead: counting gate and its mutation, both timing tables, the smoke script. - The one full local gate ran on `172bcb20` (01:03–03:22, 43 steps): all 43 passed and no record moved (`git status` clean; `tests/corpus/*.json`, `tests/bench/ratchet.json` and the package parity files unchanged). It covers formatting, fast gates, clippy for the host and `x86_64-pc-windows-gnu`, rustdoc, the workspace and MCP suites on both legs with their timing profiles, `precision_gate` 7/7 with seven population lines, the CI corpus suites, the eleven package legs, scale and tier-3 corpus, packages-only and shipped-shape acceptance, and the nightly-only cost steps including `language_package_overhead` (3 passed). - CI on the head: the package, pilot-plugin and Windows workflows passed. `ci.yml` run 8637 passed all 16 jobs; it was dispatched by hand, so it also ran the nightly-only jobs (tier-3 scale, grammar rebuild, plugin path cost with `language_package_overhead` 3 passed). Ten earlier runs on this head tested nothing: the runners could not resolve `index.crates.io`, `static.rust-lang.org` or `code.forgejo.org` for about eight hours (#582). One run in between (8631) failed in a single test, `sfc_first_contact_delegates_script_languages_and_refuses_only_unavailable_regions`: the index was still `reconciling` after 60 s. The same commit passed that test in run 8637 and in every local run (36 s alone, 102 s with the machine oversubscribed two-fold; the previous master takes 144 s). The cause of that one failure is not known; it is recorded on #582. ## Not in this PR - The scheduled runs are red for other reasons as well (#582). The package overhead job is dispatched on the fixed tree after the merge to see which of its budgets this fix restores. - The CLI has no supported command to stop a daemon; the smoke script stops it by verified PID. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

buildagent added 7 commits 2026-11-13 01:03:28 +01:00

Reuse package metadata parses by exact archive bytes within a process ... 98a5d1886e

Keep all signature, trust and activation checks outside the pure parse memo. Rebind source paths for adoption. Count real parses through the existing loadable startup/index bench.

Mutation checks: duplicating parse on a memo hit fails the startup count (6 vs 1); replacing the byte key with length fails the same-length replacement test. Both restored by verified snapshots.

Time release language smoke steps and bound retryable refusals ... 9c236e64ff

Retry only explicit retryable tool refusals, up to four retries within a 60-second retry budget. Always clean up the scratch daemon with PowerShell Stop-Process after verifying its PID, executable and project root.

Exercised with Docker PowerShell on Linux: actual shipped archive succeeds; controlled warming_up retries then succeeds; a nonretryable refusal remains fatal and cleans up its started scratch daemon.

Isolate the native pipeline benchmark from the operator package store ... a4066f4d53

Give each temporary project a sibling operator home and retain the existing 6-second bound. The full CLI run exposed a 9.698s cold index while inheriting 141MiB of unrelated installed packages.

Both M2 cases pass in 1.36s with the fixture store. Removing isolation reproduces the performance assertion failure at 9.564s; restored with a checksum-verified snapshot.

Keep advisory language availability off the archive verification path ... 9374ef3c3f

Skip builtin-only and unconsumed availability requests, and read current container indexes and manifests for advisory declarations. Remove the archive-byte parse memo so daemons retain no unused or historical archives. Gate startup and index against hashes and full reads added by unused installed packages.

Merge lane r28: startup cost no longer grows with the package store (#581) ... f82c2f913b

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014XLNPtTbPwnHBYfKMb3f1W

Bound verified package reads per used package; report smoke cleanup failures without masking ... 953e888865

The counting gate now also holds the per-package ceiling (four verified
reads in a startup, three in an index). Mutation run: a second
StoredPackage::parse in Store::get_unverified_bytes fails it (8 vs 4).
The release smoke reports a cleanup failure instead of throwing from
finally, where it would replace the error that ended the run.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014XLNPtTbPwnHBYfKMb3f1W

Bump version to 0.41.52 ... 172bcb202a

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014XLNPtTbPwnHBYfKMb3f1W

buildagent referenced this pull request 2026-11-13 06:16:41 +01:00

The nightly runs have been red on 11 of 13 completed runs since 2026-11-06 and nothing reports it: overhead budgets, two intermittent startup failures, a step-guard cascade #582

buildagent merged commit 7fc7fb206e into master 2026-11-13 12:22:37 +01:00

buildagent referenced this pull request from a commit 2026-11-13 12:22:38 +01:00

Merge PR #583: v0.41.52, startup no longer grows with the package store

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

code-review

  

correctness

  

dos

  

performance

  

security

  

severity/high

  

severity/low

  

severity/medium

  

tech-debt

  

Kind/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No labels

code-review

correctness

dos

performance

security

severity/high

severity/low

severity/medium

tech-debt

Kind/Breaking

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

h-dv/code-index!583

No description provided.