Release 0.36.1: a full connection cap delays the startup self-probe, it does not fail it #353

Closed
buildagent wants to merge 0 commits from hotfix-0.36.1 into master
Member

What broke (field report, zammad-dev, 12k-file Rails tree)

After the schema v80+v81 migrations (~3 min), the first daemon aborted 10 ms after its accept loop started:

RPC accept loop starting port=41911
64 × WARN connection cap reached; dropping connection … cap=64
ERROR … self-probe failed: listener on 127.0.0.1:41911 is not serving within 5s; aborting startup

Nothing served for ~25 min until a later daemon took over.

Cause: the daemon publishes its port before it migrates, so clients retrying during the migration queue on it. When the accept loop started, they took all 64 slots and the rest were closed over the cap — the self-probe among them. probe_reachable made ONE attempt, read EOF, and answered "unreachable" without using its 5 s budget. Pre-existing (one cap warning on 2026-10-10 under 0.35); v81 doubled the window.

Fix

probe_reachable retries one-shot health round trips until its deadline. A refused attempt proves the cap is full, not that the loop is wedged. The takeover probe of a live holder gets the same reading. Against a live holder that refuses connections, it now takes its full 2 s budget instead of answering at once.

Evidence

  • New test server::preauth_tests::a_full_connection_cap_delays_the_self_probe_but_does_not_fail_it: 64 live peers fill the real accept loop's cap and leave after 500 ms. The probe must answer true, and only after they left.
  • MUTATION (RUN): the probe returns its first attempt → RED on reachable. 30/30 green in parallel.
  • Local: fmt, clippy, doc (-D warnings --document-private-items), nextest ci 5029/5029, ci-timing 34/34, daemon leg 919/919, fast_gates rc=0, guest_gates rc=0.
  • Version bump to 0.36.1 in this PR (version_bump.sh), no schema change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YDDBdWU4yThi8Zi51E6FCj

## What broke (field report, zammad-dev, 12k-file Rails tree) After the schema v80+v81 migrations (~3 min), the first daemon aborted 10 ms after its accept loop started: ``` RPC accept loop starting port=41911 64 × WARN connection cap reached; dropping connection … cap=64 ERROR … self-probe failed: listener on 127.0.0.1:41911 is not serving within 5s; aborting startup ``` Nothing served for ~25 min until a later daemon took over. **Cause:** the daemon publishes its port before it migrates, so clients retrying during the migration queue on it. When the accept loop started, they took all 64 slots and the rest were closed over the cap — the self-probe among them. `probe_reachable` made ONE attempt, read EOF, and answered "unreachable" without using its 5 s budget. Pre-existing (one cap warning on 2026-10-10 under 0.35); v81 doubled the window. ## Fix `probe_reachable` retries one-shot `health` round trips until its deadline. A refused attempt proves the cap is full, not that the loop is wedged. The takeover probe of a live holder gets the same reading. Against a live holder that refuses connections, it now takes its full 2 s budget instead of answering at once. ## Evidence - New test `server::preauth_tests::a_full_connection_cap_delays_the_self_probe_but_does_not_fail_it`: 64 live peers fill the real accept loop's cap and leave after 500 ms. The probe must answer true, and only after they left. - MUTATION (RUN): the probe returns its first attempt → RED on `reachable`. 30/30 green in parallel. - Local: fmt, clippy, doc (`-D warnings --document-private-items`), nextest ci 5029/5029, ci-timing 34/34, daemon leg 919/919, fast_gates rc=0, guest_gates rc=0. - Version bump to 0.36.1 in this PR (`version_bump.sh`), no schema change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01YDDBdWU4yThi8Zi51E6FCj
Field report (zammad-dev, 12k-file Rails tree): after the schema v80+v81
migrations (~3 min), the first daemon aborted with "self-probe failed:
listener ... is not serving within 5s" 10 ms after its accept loop started,
and nothing served for ~25 min. The port is published before migrations,
so clients retrying meanwhile queue on it; the accept loop took 64 of them
at once, closed the rest over MAX_CONCURRENT_CONNECTIONS, and the self-probe
was among the closed. `probe_reachable` made ONE attempt, read EOF, and
answered "unreachable" without using its budget.

`probe_reachable` now retries one-shot `health` round trips until its
deadline. A refused attempt proves the cap is full, not that the loop is
wedged; the takeover probe of a live holder gets the same reading.

Test: `a_full_connection_cap_delays_the_self_probe_but_does_not_fail_it`
fills the cap with 64 live peers on the real accept loop, releases them
after 500 ms, and requires the probe to answer true, and only after the
release.

MUTATION (RUN): probe returns its first attempt -> RED on `reachable`.
30/30 green in parallel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDDBdWU4yThi8Zi51E6FCj
Release 0.36.1: version_bump.sh
All checks were successful
Package CI / package suites (clippy) (pull_request) Has been skipped
Package CI / package (javascript) (pull_request) Successful in 21s
Package CI / package (php) (pull_request) Successful in 21s
Package CI / package (ruby) (pull_request) Successful in 22s
Package CI / package (csharp) (pull_request) Successful in 30s
Package CI / package (python) (pull_request) Successful in 31s
Package CI / package (svelte) (pull_request) Successful in 21s
Package CI / package (timeline) (pull_request) Successful in 20s
Package CI / package (rust) (pull_request) Successful in 25s
CI / changes (what this run must test) (pull_request) Successful in 1m26s
Package CI / package (typescript) (pull_request) Successful in 24s
Package CI / package (vue) (pull_request) Successful in 23s
CI (Windows) / changes (what this run must test) (pull_request) Successful in 1m33s
CI / cargo fmt (pull_request) Successful in 1m33s
CI / OSS corpus tier-3 scale (nightly) (pull_request) Has been skipped
CI / Grammar rebuild from source (nightly) (pull_request) Has been skipped
Package CI / package (xaml) (pull_request) Successful in 20s
CI / CI lane wall-clock headroom (pull_request) Successful in 1m48s
CI / guest crates (fmt, clippy, doc) (pull_request) Successful in 2m20s
CI / cargo doc (intra-doc links) (pull_request) Successful in 12m47s
CI / cargo deny (pull_request) Successful in 14m40s
CI / cargo test (abi, 32-bit + wasm32) (pull_request) Successful in 15m49s
CI / cargo check (MSRV 1.98) (pull_request) Successful in 17m18s
CI / cargo clippy (pull_request) Successful in 17m29s
CI / cargo check (windows-gnu) (pull_request) Successful in 18m29s
CI (Windows) / fmt + clippy + build + test (windows) (pull_request) Successful in 37m40s
CI / OSS corpus (tier 1) (pull_request) Successful in 1h0m43s
CI / cargo test (pull_request) Successful in 45m40s
CI / cargo test (daemon transport) (pull_request) Successful in 12m14s
CI / Plugin path cost + pool throughput (nightly) (pull_request) Has been skipped
ca32787b4a
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDDBdWU4yThi8Zi51E6FCj
package URLs: a package's own release is spelled pkg%2F<id>%2Fv<ver>, and the channel takes the host's URL
All checks were successful
Package CI / package (vue) (push) Successful in 26m37s
Package CI / package (svelte) (push) Successful in 26m55s
Package CI / package (php) (push) Successful in 46m48s
Package CI / package (csharp) (push) Successful in 47m25s
Package CI / package (rust) (push) Successful in 47m34s
Package CI / package (javascript) (push) Successful in 47m52s
Package CI / package (typescript) (push) Successful in 48m4s
Package CI / package (python) (push) Successful in 48m40s
Package CI / package (ruby) (push) Successful in 54m31s
Release Build / Generate Version (push) Successful in 19s
Release Build / Required CI green (push) Successful in 54s
Release Build / Build linux-aarch64 (push) Successful in 12m50s
Release Build / Build linux-x86_64 (push) Successful in 14m36s
Release Build / Build linux-x86_64-musl (push) Successful in 15m26s
Release Build / Pack the first-party packages-1 (push) Successful in 53s
Release Build / Pack the first-party packages-4 (push) Successful in 54s
Release Build / Pack the first-party packages-3 (push) Successful in 1m1s
Release Build / Pack the first-party packages-2 (push) Successful in 1m9s
Release Build / Pack the first-party packages-6 (push) Successful in 1m4s
Release Build / Pack the first-party packages-8 (push) Successful in 1m17s
Release Build / Pack the first-party packages (push) Successful in 1m19s
Release Build / Pack the first-party packages-5 (push) Successful in 1m19s
Release Build / Pack the first-party packages-7 (push) Successful in 1m31s
Release Build / Pack the first-party packages-10 (push) Successful in 54s
Release Build / Pack the first-party packages-9 (push) Successful in 1m3s
Release Build / Build windows-x86_64 (push) Successful in 19m56s
Release Build / Assemble the release archives (push) Successful in 3m16s
Release Build / Windows archive smoke (msvc) (push) Successful in 25s
Release Build / Create Forgejo Release (push) Successful in 10m3s
Release Build / Seed the package channel (push) Successful in 2m34s
876b78b581
The host serves a `pkg/<id>/v<version>` release's assets only with the
tag's slashes percent-encoded (measured: ruby 0.10.0's literal-slash URL
answers 404, the encoded one 200; the API's browser_download_url is the
encoded form). The tree spelled the literal-slash form as canonical and
registry_schema_gate REQUIRED it, so the catalog shipped a dead link.

(a) One spelling: `code_index_core::registry::own_release_segment` (and
OWN_RELEASE_SEGMENT_PREFIX). The catalog's ruby row, registry_schema_gate,
inputs::move_catalog_row/release_of (package_records --write), the
ruby_source_ownership catalog check, registry_synthesis's retag mirror,
release_registry.sh's retag + audit, version_bump.sh's three non-site
recognisers, and package_publish.sh's DRY_RUN synthesis (`tag_segment`)
all use the encoded form; a literal-slash `pkg/` row is now refused.

(b) build_channel merges the package released NOW with `merge_released`:
a same-digest version already listed takes the incoming url and
signature_url (the host's answer) and is reported `merge_relocated`
instead of `merge_kept`. Base/include rows keep the old rule; a different
digest is still refused.

(c) package_channel_check.sh --downloadable fetches the released
version's url and signature_url from the PUBLISHED channel, anonymously,
and fails on anything but 2xx; package-release.yml's re-read runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDDBdWU4yThi8Zi51E6FCj
Author
Member

Added: package download URLs (876b78b5)

Folded into 0.36.1 by operator decision. ruby 0.10.0's package release (run 1094) published its assets, but the package channel listed them at a URL that returns 404.

Measured on the host: …/releases/download/pkg/de.h-dv.ruby/v0.10.0/… returns 404. …/releases/download/pkg%2Fde.h-dv.ruby%2Fv0.10.0/… returns 200, and is the API's own browser_download_url.

Three causes, three fixes:

  • (a) The literal-slash spelling was canonical. distribution/registry.v1.json used it, and registry_schema_gate required it.
    • New code_index_core::registry::own_release_segment gives one encoded spelling, used by every producer and gate (catalog row, schema gate, move_catalog_row, release_registry.sh, version_bump.sh, publish DRY_RUN).
    • A literal-slash pkg/ row is now refused.
  • (b) The channel merge kept the stale row. build_channel reported a same-digest version as merge_kept, so the old URL from the core catalog beat the host's answer.
    • The package released now goes through merge_released: same digest takes the incoming URLs and is reported merge_relocated.
    • Base and include rows are unchanged, and a different digest is still refused.
  • (c) The read-back never downloaded. package_channel_check.sh --downloadable now fetches the released version's url and signature_url from the PUBLISHED channel, anonymously, and requires 2xx. package-release.yml runs it.

Verification (mine, beyond the lane's 12 mutations):

  • 5-crate nextest on the combined head: 2776/2776. Also clippy, doc, fast_gates and guest_gates.
  • MUTATION (RUN): includes also relocate → RED, a_released_version_takes_the_hosts_urls_over_a_same_digest_row.
  • MUTATION (RUN): --downloadable always 200 → RED, the_channel_check_refuses_a_listed_url_that_does_not_download.
  • Against the real host: the live channel is refused (HTTP 404 named). The same channel with the encoded URL passes (200 for both url and signature_url).
  • The runtime catalog validation (crates/core) does not check URL shape, so v0.36.0 clients accept the encoded URL. A v0.36.0 plugin add from the encoded URL installs, as observed in the field.

After the 0.36.1 release: re-run package-release for pkg/de.h-dv.ruby/v0.10.0. With (b) it rewrites the channel's URL, and with (c) it proves the new one downloads.

🤖 Generated with Claude Code

## Added: package download URLs (876b78b5) Folded into 0.36.1 by operator decision. ruby 0.10.0's package release (run 1094) published its assets, but the package channel listed them at a URL that returns 404. **Measured on the host:** `…/releases/download/pkg/de.h-dv.ruby/v0.10.0/…` returns 404. `…/releases/download/pkg%2Fde.h-dv.ruby%2Fv0.10.0/…` returns 200, and is the API's own `browser_download_url`. **Three causes, three fixes:** - **(a) The literal-slash spelling was canonical.** `distribution/registry.v1.json` used it, and `registry_schema_gate` *required* it. - New `code_index_core::registry::own_release_segment` gives one encoded spelling, used by every producer and gate (catalog row, schema gate, `move_catalog_row`, `release_registry.sh`, `version_bump.sh`, publish DRY_RUN). - A literal-slash `pkg/` row is now refused. - **(b) The channel merge kept the stale row.** `build_channel` reported a same-digest version as `merge_kept`, so the old URL from the core catalog beat the host's answer. - The package released now goes through `merge_released`: same digest takes the incoming URLs and is reported `merge_relocated`. - Base and include rows are unchanged, and a different digest is still refused. - **(c) The read-back never downloaded.** `package_channel_check.sh --downloadable` now fetches the released version's `url` and `signature_url` from the PUBLISHED channel, anonymously, and requires 2xx. `package-release.yml` runs it. **Verification (mine, beyond the lane's 12 mutations):** - 5-crate nextest on the combined head: 2776/2776. Also clippy, doc, fast_gates and guest_gates. - MUTATION (RUN): includes also relocate → RED, `a_released_version_takes_the_hosts_urls_over_a_same_digest_row`. - MUTATION (RUN): `--downloadable` always 200 → RED, `the_channel_check_refuses_a_listed_url_that_does_not_download`. - Against the real host: the live channel is refused (HTTP 404 named). The same channel with the encoded URL passes (200 for both `url` and `signature_url`). - The runtime catalog validation (`crates/core`) does not check URL shape, so v0.36.0 clients accept the encoded URL. A v0.36.0 `plugin add` from the encoded URL installs, as observed in the field. **After the 0.36.1 release:** re-run package-release for `pkg/de.h-dv.ruby/v0.10.0`. With (b) it rewrites the channel's URL, and with (c) it proves the new one downloads. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Author
Member

Shipped. Head 876b78b5 is on master's first-parent line, because master was fast-forwarded to it rather than merged through this PR, and it is tagged v0.36.1. Closing as delivered.

Shipped. Head 876b78b5 is on master's first-parent line, because master was fast-forwarded to it rather than merged through this PR, and it is tagged **v0.36.1**. Closing as delivered.
buildagent closed this pull request 2026-10-13 10:15:23 +02:00
All checks were successful
CI / changes (what this run must test) (pull_request) Successful in 21s
Package CI / changes (which packages this run must test) (pull_request) Successful in 21s
CI (Windows) / changes (what this run must test) (pull_request) Successful in 40s
CI / cargo fmt (pull_request) Successful in 1m2s
CI / OSS corpus tier-3 scale (nightly) (pull_request) Has been skipped
CI / Grammar rebuild from source (nightly) (pull_request) Has been skipped
CI / CI lane wall-clock headroom (pull_request) Successful in 1m50s
CI / guest crates (fmt, clippy, doc) (pull_request) Successful in 2m6s
CI / cargo doc (intra-doc links) (pull_request) Successful in 15m13s
CI / cargo deny (pull_request) Successful in 17m24s
Package CI / package suites (clippy) (pull_request) Successful in 18m10s
CI / cargo test (abi, 32-bit + wasm32) (pull_request) Successful in 17m47s
CI / cargo check (MSRV 1.98) (pull_request) Successful in 20m47s
CI / cargo clippy (pull_request) Successful in 21m39s
CI / cargo check (windows-gnu) (pull_request) Successful in 22m59s
Package CI / package (vue) (pull_request) Successful in 34m39s
Package CI / package (xaml) (pull_request) Successful in 35m4s
Package CI / package (svelte) (pull_request) Successful in 35m19s
Package CI / package (timeline) (pull_request) Successful in 35m36s
CI (Windows) / fmt + clippy + build + test (windows) (pull_request) Successful in 37m26s
Package CI / package (javascript) (pull_request) Successful in 1h1m28s
Package CI / package (php) (pull_request) Successful in 1h2m59s
Package CI / package (csharp) (pull_request) Successful in 1h3m0s
Package CI / package (rust) (pull_request) Successful in 1h3m6s
Package CI / package (typescript) (pull_request) Successful in 1h3m22s
Package CI / package (python) (pull_request) Successful in 1h4m34s
Package CI / package (ruby) (pull_request) Successful in 1h9m41s
CI / OSS corpus (tier 1) (pull_request) Successful in 1h12m28s
CI / cargo test (pull_request) Successful in 57m21s
CI / cargo test (daemon transport) (pull_request) Successful in 12m13s
CI / Plugin path cost + pool throughput (nightly) (pull_request) Has been skipped
Package CI / changes (which packages this run must test) (push) Successful in 23s
CI (Windows) / changes (what this run must test) (push) Successful in 27s
CI (Windows) / fmt + clippy + build + test (windows) (push) Has been skipped
CI / changes (what this run must test) (push) Successful in 41s
CI / cargo fmt (push) Successful in 1m2s
CI / CI lane wall-clock headroom (push) Has been skipped
CI / guest crates (fmt, clippy, doc) (push) Has been skipped
CI / cargo doc (intra-doc links) (push) Has been skipped
CI / OSS corpus (tier 1) (push) Has been skipped
CI / OSS corpus tier-3 scale (nightly) (push) Has been skipped
CI / Grammar rebuild from source (nightly) (push) Has been skipped
CI / cargo clippy (push) Has been skipped
CI / cargo check (MSRV 1.98) (push) Has been skipped
CI / cargo check (windows-gnu) (push) Has been skipped
CI / cargo test (abi, 32-bit + wasm32) (push) Has been skipped
CI / cargo deny (push) Has been skipped
CI / cargo test (push) Has been skipped
CI / cargo test (daemon transport) (push) Has been skipped
CI / Plugin path cost + pool throughput (nightly) (push) Has been skipped
Package CI / package suites (clippy) (push) Successful in 10m36s
Package CI / package (xaml) (push) Successful in 25m50s
Package CI / package (timeline) (push) Successful in 25m57s
Package CI / package (vue) (push) Successful in 26m37s
Package CI / package (svelte) (push) Successful in 26m55s
Package CI / package (php) (push) Successful in 46m48s
Package CI / package (csharp) (push) Successful in 47m25s
Package CI / package (rust) (push) Successful in 47m34s
Package CI / package (javascript) (push) Successful in 47m52s
Package CI / package (typescript) (push) Successful in 48m4s
Package CI / package (python) (push) Successful in 48m40s
Package CI / package (ruby) (push) Successful in 54m31s
Release Build / Generate Version (push) Successful in 19s
Release Build / Required CI green (push) Successful in 54s
Release Build / Build linux-aarch64 (push) Successful in 12m50s
Release Build / Build linux-x86_64 (push) Successful in 14m36s
Release Build / Build linux-x86_64-musl (push) Successful in 15m26s
Release Build / Pack the first-party packages-1 (push) Successful in 53s
Release Build / Pack the first-party packages-4 (push) Successful in 54s
Release Build / Pack the first-party packages-3 (push) Successful in 1m1s
Release Build / Pack the first-party packages-2 (push) Successful in 1m9s
Release Build / Pack the first-party packages-6 (push) Successful in 1m4s
Release Build / Pack the first-party packages-8 (push) Successful in 1m17s
Release Build / Pack the first-party packages (push) Successful in 1m19s
Release Build / Pack the first-party packages-5 (push) Successful in 1m19s
Release Build / Pack the first-party packages-7 (push) Successful in 1m31s
Release Build / Pack the first-party packages-10 (push) Successful in 54s
Release Build / Pack the first-party packages-9 (push) Successful in 1m3s
Release Build / Build windows-x86_64 (push) Successful in 19m56s
Release Build / Assemble the release archives (push) Successful in 3m16s
Release Build / Windows archive smoke (msvc) (push) Successful in 25s
Release Build / Create Forgejo Release (push) Successful in 10m3s
Release Build / Seed the package channel (push) Successful in 2m34s

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index!353
No description provided.