h-dv/code-index

Watch

Fork 0

The background update installs and conformance-checks every catalog package again on every run, twice, with nothing to update (about 150 s per daemon start) #586

New issue

Closed

opened 2026-11-13 15:32:50 +01:00 by buildagent · 3 comments

buildagent commented

2026-11-13 15:32:50 +01:00

Member

Found on the development machine while following up #520, 2026-11-13, on the released v0.41.52 (the same pattern is in the daemon log since at least 2026-11-11).

What happens

After a daemon is ready it starts code-index plugin update --root <project> in the background. On a project with eleven catalog packages and nothing to update, one such run

prints update_notifications 0 of 11 checked package(s) have something to act on,
and before that installs each of the eleven packages into the store and runs the full conformance check on each of them, twice: 22 times installed … verdict passed … verdict recorded for 11 distinct digests in one run.

Duration of the last five runs on this machine, from the daemon log ("checking configured plugin updates in the background" to "… check finished"): 162 s, 154 s, 169 s, 66 s, 156 s. Three of them started within 25 minutes of each other today.

The verdict files of these digests already existed before each run; they are rewritten every time.

Why it matters

Every daemon start costs minutes of CPU on the user's machine for a check whose answer is "nothing to do".
Until the next release the run also holds the project's plugin lock for its whole duration (#520: approve refused 19 times in 2.5 minutes). The lock scope is fixed on the next-train branch; the work itself is not.
The same store work is repeated for every project, and by every daemon start.

Expected

A package whose bytes are already in the store with a recorded passing verdict for this host is not installed again and not conformance-checked again. The check for updates compares identities (catalog digest against the store) and does the expensive work only for a digest that is new, or whose verdict is missing or was recorded by a different host or engine configuration.
One run handles each digest at most once.
The run says what it did in numbers: how many packages were compared, how many were already verified, how many were installed and checked.

To establish first

Why each package is handled twice in one run.
Which input makes the run consider a stored, verified digest as needing work (the verdict's host or engine key, the automatic policy, the catalog freshness record, the link set of the project).
Since which release this happens.

Acceptance

Test that can fail: a second update run on an unchanged project and store performs zero installs and zero conformance checks (counted, not timed), and reports that; mutation run.
Test: a new digest in the catalog is installed and checked exactly once.
Test: a verdict recorded by a different host or engine configuration is renewed exactly once.
The measured duration of a no-op background check on a store with eleven verified packages, before and after.

Found on the development machine while following up #520, 2026-11-13, on the released v0.41.52 (the same pattern is in the daemon log since at least 2026-11-11). ## What happens After a daemon is ready it starts `code-index plugin update --root <project>` in the background. On a project with eleven catalog packages and nothing to update, one such run - prints `update_notifications 0 of 11 checked package(s) have something to act on`, - and before that installs each of the eleven packages into the store and runs the full conformance check on each of them, **twice**: 22 times `installed … verdict passed … verdict recorded` for 11 distinct digests in one run. Duration of the last five runs on this machine, from the daemon log ("checking configured plugin updates in the background" to "… check finished"): 162 s, 154 s, 169 s, 66 s, 156 s. Three of them started within 25 minutes of each other today. The verdict files of these digests already existed before each run; they are rewritten every time. ## Why it matters - Every daemon start costs minutes of CPU on the user's machine for a check whose answer is "nothing to do". - Until the next release the run also holds the project's plugin lock for its whole duration (#520: `approve` refused 19 times in 2.5 minutes). The lock scope is fixed on the next-train branch; the work itself is not. - The same store work is repeated for every project, and by every daemon start. ## Expected - A package whose bytes are already in the store with a recorded passing verdict for this host is not installed again and not conformance-checked again. The check for updates compares identities (catalog digest against the store) and does the expensive work only for a digest that is new, or whose verdict is missing or was recorded by a different host or engine configuration. - One run handles each digest at most once. - The run says what it did in numbers: how many packages were compared, how many were already verified, how many were installed and checked. ## To establish first - Why each package is handled twice in one run. - Which input makes the run consider a stored, verified digest as needing work (the verdict's host or engine key, the automatic policy, the catalog freshness record, the link set of the project). - Since which release this happens. ## Acceptance - Test that can fail: a second update run on an unchanged project and store performs zero installs and zero conformance checks (counted, not timed), and reports that; mutation run. - Test: a new digest in the catalog is installed and checked exactly once. - Test: a verdict recorded by a different host or engine configuration is renewed exactly once. - The measured duration of a no-op background check on a store with eleven verified packages, before and after.

buildagent added the

labels

2026-11-13 15:32:50 +01:00

buildagent referenced this issue

2026-11-13 15:33:24 +01:00

Approval writers can race and package admission can miss granted authority #520

buildagent commented

2026-11-13 16:27:39 +01:00

Author

Member

Round 1 of the fix (lane r40, on its branch, not merged): cause found, conformance work removed; a second cause remains.

Cause, both parts present since commit 118b4311, first released in v0.41.24:

An active package whose digest is unchanged was admitted again unconditionally, and admission installs and runs conformance without looking at the recorded verdict.
The run selects the stored and bundled packages, then walks the catalog and selects the same digests a second time; the "already seen" set of the first pass is not shared with the second.

Round 1 reuses the existing verdict gate (digest and schema, passing outcome, engine version and flags, coverage of the granted capabilities; the code-index build is not part of a verdict's key) and handles each digest once per run. The run now prints what it did, for example update_work 7 compared, 7 already verified, 0 installed, 0 conformance-checked. Named updates (plugin update <id>) already returned early when the catalog pin equals the current digest and did not have this defect.

Lead's measurement, quiet machine (load 1.4), a copy of this machine's package home (45 stored archives, 141 MiB), a scratch project with the seven first-party languages, the real catalog, plugin update --root <project> run twice:

build	wall time	installs	verdicts written
v0.41.52	43.1 s, 43.5 s	14	14
round 1	27.9 s, 26.5 s	0	0

So the repeated conformance was about a third of the cost. The same run at debug level shows the rest: 563 complete archive reads and 581 metadata parses in one run; the archives of the active packages are read 75 times each. This is the defect class of #581 again (the store is enumerated and every archive read and hashed in full, many times per process), on the update path, which the #581 fix did not touch.

Round 2 is running: enumerate the store once per run, read and hash an archive only when a decision needs its bytes, a counting test with a ceiling that does not depend on how many unrelated packages are installed, and the same two counts for the other operator commands (languages enable, plugin status, plugin doctor, approve, link add, languages list) as a table.

**Round 1 of the fix (lane r40, on its branch, not merged): cause found, conformance work removed; a second cause remains.** Cause, both parts present since commit `118b4311`, first released in v0.41.24: - An active package whose digest is unchanged was admitted again unconditionally, and admission installs and runs conformance without looking at the recorded verdict. - The run selects the stored and bundled packages, then walks the catalog and selects the same digests a second time; the "already seen" set of the first pass is not shared with the second. Round 1 reuses the existing verdict gate (digest and schema, passing outcome, engine version and flags, coverage of the granted capabilities; the code-index build is not part of a verdict's key) and handles each digest once per run. The run now prints what it did, for example `update_work 7 compared, 7 already verified, 0 installed, 0 conformance-checked`. Named updates (`plugin update <id>`) already returned early when the catalog pin equals the current digest and did not have this defect. **Lead's measurement**, quiet machine (load 1.4), a copy of this machine's package home (45 stored archives, 141 MiB), a scratch project with the seven first-party languages, the real catalog, `plugin update --root <project>` run twice: | build | wall time | installs | verdicts written | | :-- | --: | --: | --: | | v0.41.52 | 43.1 s, 43.5 s | 14 | 14 | | round 1 | 27.9 s, 26.5 s | 0 | 0 | So the repeated conformance was about a third of the cost. The same run at debug level shows the rest: **563 complete archive reads and 581 metadata parses in one run**; the archives of the active packages are read 75 times each. This is the defect class of #581 again (the store is enumerated and every archive read and hashed in full, many times per process), on the update path, which the #581 fix did not touch. Round 2 is running: enumerate the store once per run, read and hash an archive only when a decision needs its bytes, a counting test with a ceiling that does not depend on how many unrelated packages are installed, and the same two counts for the other operator commands (`languages enable`, `plugin status`, `plugin doctor`, `approve`, `link add`, `languages list`) as a table.

buildagent commented

2026-11-13 17:30:52 +01:00

Author

Member

Fixed on the next-train branch (lane r40, two rounds, merged locally as 968a57eb; not released yet).

Second cause, attributed. In the run of 563 archive reads, 38 archives were read once and the seven active ones 75 times each. The reads came from: the store scan (45), consider → resolve / displacers / withdrawn → get for local candidates (203), the catalog prefilter (154), the catalog consider (147), and two more get per package (14). Every get read the archive and hashed it twice.

Change.

One update run owns a snapshot of the store: the directory is listed once and an archive is read and hashed at most once; later lookups in the same run reuse it. The snapshot belongs to the operation's own store value; an ordinary store still reads fresh bytes, and nothing is cached for the life of a daemon.
Candidates are selected from the manifests (container index and manifest only, no payload, no hash). Bytes are read and verified only for a candidate that is selected, once, and the same verified bytes are used for admission and conformance grading. Signature and trust checks still run on every lookup.
plugin status and plugin doctor use the same snapshot.

Lead's measurement, copy of this machine's package home (45 archives, 141 MiB), scratch project with the seven first-party languages, real catalog. The machine was not quiet for the last row (load 8), which makes that figure an upper bound:

build	no-op `plugin update --root`	full archive reads	metadata parses	installs / verdicts
v0.41.52	43.1 s, 43.5 s	584	630	14 / 14
round 1	27.9 s, 26.5 s	563	595	0 / 0
round 2	1.36 s, 1.38 s	7	7	0 / 0

(The read and parse counts of the first row are the worker's; the other figures are mine.)

Other commands on the same store, after the change (my run): plugin status 2.1 s (45 reads), plugin doctor 1.6 s (45 reads; 164 before, per the worker), languages list 0.6 s (7 reads).

Tests. A no-op update on a store with 0 and with 8 unrelated installed packages performs the same number of full reads (1, for the first verification of content in the fixture) and never more; a new catalog digest is installed and checked exactly once; a verdict recorded for another engine configuration is renewed exactly once; one run never handles a digest twice. The worker reported the mutations red (four in each round).

Not changed, counted for a later lane (worker's figures, same store):

languages enable for seven languages on a fresh project: 77 full reads, 98 parses, 12.7 s. The caller chain is bundled adoption → cmd_add → admit → activation, each with its own store value.
plugin add of an installed digest: 29 reads.
languages list: 7 reads, 14 parses.
plugin status and plugin doctor still read every archive once (45); they could work from manifests.

One red in the worker's full indexer run, not attributed to this change: member_lookup::a_thousand_overriding_users_cost_one_decision_not_one_per_site took 94 s against its 60 s ceiling while three workers were building on the machine. Alone it takes 5.2 s on this lane's tree (two runs) and 12.8 s on the branch before the merge under the same load. The full gate run on a quiet machine decides it.

**Fixed on the next-train branch** (lane r40, two rounds, merged locally as `968a57eb`; not released yet). **Second cause, attributed.** In the run of 563 archive reads, 38 archives were read once and the seven active ones 75 times each. The reads came from: the store scan (45), `consider` → resolve / displacers / withdrawn → `get` for local candidates (203), the catalog prefilter (154), the catalog `consider` (147), and two more `get` per package (14). Every `get` read the archive and hashed it twice. **Change.** - One update run owns a snapshot of the store: the directory is listed once and an archive is read and hashed at most once; later lookups in the same run reuse it. The snapshot belongs to the operation's own store value; an ordinary store still reads fresh bytes, and nothing is cached for the life of a daemon. - Candidates are selected from the manifests (container index and manifest only, no payload, no hash). Bytes are read and verified only for a candidate that is selected, once, and the same verified bytes are used for admission and conformance grading. Signature and trust checks still run on every lookup. - `plugin status` and `plugin doctor` use the same snapshot. **Lead's measurement**, copy of this machine's package home (45 archives, 141 MiB), scratch project with the seven first-party languages, real catalog. The machine was not quiet for the last row (load 8), which makes that figure an upper bound: | build | no-op `plugin update --root` | full archive reads | metadata parses | installs / verdicts | | :-- | --: | --: | --: | --: | | v0.41.52 | 43.1 s, 43.5 s | 584 | 630 | 14 / 14 | | round 1 | 27.9 s, 26.5 s | 563 | 595 | 0 / 0 | | round 2 | 1.36 s, 1.38 s | 7 | 7 | 0 / 0 | (The read and parse counts of the first row are the worker's; the other figures are mine.) Other commands on the same store, after the change (my run): `plugin status` 2.1 s (45 reads), `plugin doctor` 1.6 s (45 reads; 164 before, per the worker), `languages list` 0.6 s (7 reads). **Tests.** A no-op update on a store with 0 and with 8 unrelated installed packages performs the same number of full reads (1, for the first verification of content in the fixture) and never more; a new catalog digest is installed and checked exactly once; a verdict recorded for another engine configuration is renewed exactly once; one run never handles a digest twice. The worker reported the mutations red (four in each round). **Not changed, counted for a later lane** (worker's figures, same store): - `languages enable` for seven languages on a fresh project: 77 full reads, 98 parses, 12.7 s. The caller chain is bundled adoption → `cmd_add` → admit → activation, each with its own store value. - `plugin add` of an installed digest: 29 reads. - `languages list`: 7 reads, 14 parses. - `plugin status` and `plugin doctor` still read every archive once (45); they could work from manifests. **One red in the worker's full indexer run, not attributed to this change:** `member_lookup::a_thousand_overriding_users_cost_one_decision_not_one_per_site` took 94 s against its 60 s ceiling while three workers were building on the machine. Alone it takes 5.2 s on this lane's tree (two runs) and 12.8 s on the branch before the merge under the same load. The full gate run on a quiet machine decides it.

buildagent referenced this issue

2026-11-14 00:43:40 +01:00

v0.41.53: restart and update behaviour a session can rely on; link add keeps the language decision; skill command removed #587

buildagent referenced this issue from a commit

2026-11-14 11:54:09 +01:00

Merge lane r40: a background update with nothing to do verifies nothing again and reads each archive once (#586)

buildagent referenced this issue from a commit

2026-11-14 11:54:09 +01:00

Merge PR #587: v0.41.53, restart and update behaviour a session can rely on

buildagent commented

2026-11-14 12:28:46 +01:00

Author

Member

Released in v0.41.53 (59b8bc3d); verified on the installed binaries.
A background update with nothing to do no longer reinstalls or rechecks packages already verified for this engine, and reads each archive at most once per run. On a store with 45 packages, it went from 43 s and 584 complete archive reads to 1.4 s and 7 reads; plugin status and plugin doctor also read each installed archive once.

Verified on the installed release:

586_noop_update: the update installs and conformance-checks nothing and completes in under 10 s. PASS: .260299055s; 0 installed, 0 conformance-checked

Remaining scope: languages enable for several languages, plugin add and languages list still read installed archives more often than needed: 77, 29 and 7 complete reads on a store with 45 archives.

Released in v0.41.53 (59b8bc3d); verified on the installed binaries. A background update with nothing to do no longer reinstalls or rechecks packages already verified for this engine, and reads each archive at most once per run. On a store with 45 packages, it went from 43 s and 584 complete archive reads to 1.4 s and 7 reads; `plugin status` and `plugin doctor` also read each installed archive once. Verified on the installed release: - `586_noop_update`: the update installs and conformance-checks nothing and completes in under 10 s. PASS: .260299055s; 0 installed, 0 conformance-checked Remaining scope: `languages enable` for several languages, `plugin add` and `languages list` still read installed archives more often than needed: 77, 29 and 7 complete reads on a store with 45 archives.

buildagent closed this issue

2026-11-14 12:28:47 +01:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

h-dv/code-index#586

No description provided.

Rows
Columns