h-dv/code-index

Watch

Fork 0

`mixed_load_ceilings` fails intermittently on the nightly at `ladder x64` with `host.memory_ceiling_exceeded`, on a tree that passes the same job hours earlier #288

New issue

Open

opened 2026-09-21 12:02:59 +02:00 by buildagent · 0 comments

buildagent commented

2026-09-21 12:02:59 +02:00

Member

Found while verifying the #276 merge. Pre-existing, not introduced by it — see the identical-trees proof below.

The failure

Nightly run 856 (bea410f, scheduled, 03:01) — job corpus-scale, test mixed_load_ceilings in crates/plugin-host/tests/mixed_load_bench.rs:

test mixed_load_ceilings ... FAILED
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13 filtered out; finished in 95.68s

The IPC-at-size ladder crosses the ceiling only at its last rung:

  file          scan     src B     nodes   stream B  reply  host us  trip us  us/KiB outcome
  ladder x32   first  15527649   2772675   44362800     71  2001763  2374378   156.6 facts
  ladder x64   first  31055297   5545347   88725552      5  3938727  4718800   155.6 host.memory_ceiling_exceeded

x32 produces facts; x64 — 31 MB of source, 5.5 M nodes, an 88 MB stream — does not. Note reply drops from 71 to 5, i.e. the refusal, not a short answer.

It is NOT a code regression, and this is provable

The same job passed on run 851 at 14:00 and again on run 857.

Run 851 ran 7a5fd70; run 856 ran bea410f. bea410f is the merge of 7a5fd70 into a master whose head was that commit's own parent, so:

$ git diff --stat 7a5fd70 bea410f
(empty)

Byte-identical trees passed and failed. No change to the tree can explain it.

Frequency — one verified, one suspected

Two of the last 14 scheduled ci.yml runs failed, both aborting at a near-identical duration:

856  failure  1559s  bea410f   <- VERIFIED: mixed_load_ceilings, host.memory_ceiling_exceeded
725  failure  1541s  bd1c4e2   <- SUSPECTED ONLY

Run 725 could not be verified. Its logs return 404 on every job index and it is past the /actions/tasks feed window, so the only thing linking it is the 18-second-similar early-abort duration. That is a hint, not evidence, and it is recorded here as such rather than counted as a second occurrence. If someone has a way to reach logs that old, it is worth one look — two confirmed occurrences would change this from "seen once" to a rate.

Why a host-conditions hypothesis is worth testing first

mixed_load_ceilings is #[ignore]d ("reads ~/.cache/cosi-corpus and produces timings"), so it runs only on the nightly and on workflow_dispatch — the two configurations where the three nightly jobs run concurrently with everything else.

The ceiling it trips is a host memory ceiling on the largest rung of a doubling ladder. A rung that needs ~88 MB of stream buffer either fits or does not depending on what else the box is doing at 03:01. This file's own history shows the area is sensitive: eb92cf6 ("an OOM kill no longer pushes a blameless package toward quarantine") and 0fa644b ("a 25% flake was a test asserting inside a window with no evidence").

This is NOT a claim that the ceiling is wrong. It is that nothing in the failure output distinguishes:

the guest genuinely needing more than the ceiling allows, from
the host being unable to give it what the ceiling permits.

Those want different fixes and the current message cannot tell them apart.

What would close this

Make the refusal print its state. host.memory_ceiling_exceeded should carry what was asked for, what the ceiling was, and what the host had available at that moment. A derived value cannot currently tell a contention artifact from a real over-budget request.
Decide whether ladder x64 belongs in a gate that runs under contention. It is the only rung that fails and it is 2× the largest rung that passes. Either it is load-bearing and needs a reserve (cf. #285's shape), or it is a measurement that should not gate a nightly.
A second confirmed occurrence before treating a rate as known. One verified failure is one.

Scope note

Deliberately not "raise the memory ceiling". The ceiling may be exactly right; what is missing is the evidence to tell which of the two cases above produced the refusal.

🤖 Generated with Claude Code

https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

Found while verifying the #276 merge. **Pre-existing, not introduced by it** — see the identical-trees proof below. ## The failure Nightly run [856](https://git.h-dv.de/h-dv/code-index/actions/runs/856) (`bea410f`, scheduled, 03:01) — job `corpus-scale`, test `mixed_load_ceilings` in `crates/plugin-host/tests/mixed_load_bench.rs`: ``` test mixed_load_ceilings ... FAILED test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13 filtered out; finished in 95.68s ``` The IPC-at-size ladder crosses the ceiling only at its last rung: ``` file scan src B nodes stream B reply host us trip us us/KiB outcome ladder x32 first 15527649 2772675 44362800 71 2001763 2374378 156.6 facts ladder x64 first 31055297 5545347 88725552 5 3938727 4718800 155.6 host.memory_ceiling_exceeded ``` `x32` produces facts; `x64` — 31 MB of source, 5.5 M nodes, an 88 MB stream — does not. Note `reply` drops from 71 to 5, i.e. the refusal, not a short answer. ## It is NOT a code regression, and this is provable The same job **passed** on run [851](https://git.h-dv.de/h-dv/code-index/actions/runs/851) at 14:00 and again on run [857](https://git.h-dv.de/h-dv/code-index/actions/runs/857). Run 851 ran `7a5fd70`; run 856 ran `bea410f`. `bea410f` is the merge of `7a5fd70` into a master whose head was that commit's own parent, so: ``` $ git diff --stat 7a5fd70 bea410f (empty) ``` **Byte-identical trees passed and failed.** No change to the tree can explain it. ## Frequency — one verified, one suspected Two of the last 14 scheduled `ci.yml` runs failed, both aborting at a near-identical duration: ``` 856 failure 1559s bea410f <- VERIFIED: mixed_load_ceilings, host.memory_ceiling_exceeded 725 failure 1541s bd1c4e2 <- SUSPECTED ONLY ``` **Run 725 could not be verified.** Its logs return 404 on every job index and it is past the `/actions/tasks` feed window, so the only thing linking it is the 18-second-similar early-abort duration. That is a hint, not evidence, and it is recorded here as such rather than counted as a second occurrence. If someone has a way to reach logs that old, it is worth one look — two confirmed occurrences would change this from "seen once" to a rate. ## Why a host-conditions hypothesis is worth testing first `mixed_load_ceilings` is `#[ignore]`d (`"reads ~/.cache/cosi-corpus and produces timings"`), so it runs only on the nightly and on `workflow_dispatch` — the two configurations where the three nightly jobs run concurrently with everything else. The ceiling it trips is a **host memory** ceiling on the largest rung of a doubling ladder. A rung that needs ~88 MB of stream buffer either fits or does not depending on what else the box is doing at 03:01. This file's own history shows the area is sensitive: `eb92cf6` ("an OOM kill no longer pushes a blameless package toward quarantine") and `0fa644b` ("a 25% flake was a test asserting inside a window with no evidence"). This is NOT a claim that the ceiling is wrong. It is that nothing in the failure output distinguishes: 1. the guest genuinely needing more than the ceiling allows, from 2. the host being unable to give it what the ceiling permits. Those want different fixes and the current message cannot tell them apart. ## What would close this 1. **Make the refusal print its state.** `host.memory_ceiling_exceeded` should carry what was asked for, what the ceiling was, and what the host had available at that moment. A derived value cannot currently tell a contention artifact from a real over-budget request. 2. **Decide whether `ladder x64` belongs in a gate that runs under contention.** It is the only rung that fails and it is 2× the largest rung that passes. Either it is load-bearing and needs a reserve (cf. #285's shape), or it is a measurement that should not gate a nightly. 3. **A second confirmed occurrence** before treating a rate as known. One verified failure is one. ## Scope note Deliberately not "raise the memory ceiling". The ceiling may be exactly right; what is missing is the evidence to tell which of the two cases above produced the refusal. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

h-dv/code-index#288

No description provided.

Rows
Columns

mixed_load_ceilings fails intermittently on the nightly at ladder x64 with host.memory_ceiling_exceeded, on a tree that passes the same job hours earlier #288