h-dv/code-index

Watch

Fork 0

XAML `$max_facts` is the next wall behind #222 — raising the tree budget alone will turn `truncated_tree` into `node_limit_reached` on the same files #223

New issue

Closed

opened 2026-09-08 09:41:09 +02:00 by buildagent · 1 comment

buildagent commented

2026-09-08 09:41:09 +02:00

Member

Filed alongside #222 so that fixing one does not silently produce the other and read as a regression.

The situation

The XAML extractor declares two independent budgets that report two different codes:

(global $max_tree_bytes i32 (i32.const 320000))   ;; WALK   -> extract.truncated_tree
(global $max_facts      i32 (i32.const 4000))     ;; OUTPUT -> extract.node_limit_reached

plus a third, unnamed guard inside the scan loop, checked on every iteration:

(if (i32.or (i32.ge_u (global.get $nf) (global.get $max_facts))
            (i32.gt_u (global.get $bp) (i32.const 196608)))
  (then (local.set $full (i32.const 1)) (br $scan)))

Today the tree budget cuts first on every large file measured in production — six probes, all extract.truncated_tree, never node_limit_reached. That means $max_facts and the $bp guard are currently untested against real markup: nothing reaches them, so nobody knows where they bite.

Why this must be settled with #222 rather than after it

#222 raises $max_tree_bytes so files up to some new size complete. The moment the tree stops cutting, the fact budget becomes the binding constraint on exactly the files #222 is meant to fix — the theme dictionaries. Generic.xaml at 246.6 KB and BaseTheme.Net.xaml at 221.2 KB are dense x:Name/x:Class/Click markup, which is precisely what mints facts.

The observable outcome would be: the same files still incomplete, the same tools still missing symbols, and a different diagnostic code. To anyone watching the code rather than the outcome that looks like a new defect introduced by the fix. To anyone watching the outcome it looks like the fix did nothing.

Both budgets must be sized against the same measured corpus, in one change, or neither number means anything.

What must be measured, not reasoned about

For real markup at 100 KB, 250 KB and 1.3 MB (the production sizes in #222):

facts emitted per KB of XAML — the density that decides whether 4,000 is anywhere near right;
which budget binds first at each size, stated as a table;
bytes consumed in the fact buffer at each size, against the hard $bp > 196608 guard — that guard is a third ceiling and it is not one of the two documented knobs;
total linear memory in use, against (memory 8) = 512 KiB initial with src_offset at 262,144, shared by source + tree + facts. The tree budget is the largest single consumer, so raising it moves this too.

If the answer is that Generic.xaml-scale files cannot fit any budget triple within one wasm instance's memory, that is a legitimate finding and should be stated rather than papered over with a number that merely moves the wall past the current corpus. A streaming or chunked walk would then be the real fix, and this issue becomes its motivation.

What a fix must prove

A fixture that exceeds $max_facts while the tree budget is comfortably clear, so the node_limit_reached path is graded at all — it currently is not, against real markup.
A fixture that exceeds the $bp buffer guard, or a written argument (with the arithmetic) for why $max_facts must always bind first. Right now that guard is a silent third ceiling.
The truncated_tree path still fires and still discloses, for a file beyond whatever new tree budget lands.
Both codes distinguishable end-to-end through index_coverage, not just at the extractor boundary.

Mutations

Set $max_facts above anything reachable → the fact-budget assertion must go RED, or that path is untested.
Set the $bp guard above reachable → its assertion must go RED. If no test fails, the guard is ungraded and the fix has not covered the third ceiling.
Make the extractor report truncated_tree where it means node_limit_reached → the code-discrimination test must go RED. The two codes carry different operator meaning ("too big to read" vs "too rich to report") and must not be interchangeable.

Cost, same as #222

Any change here is a package edit: every byte under tests/packages/xaml/ is packed, so the .cip digest moves, tests/packages/xaml.digest must be re-recorded, the package version bumps, and the release notes must say so or operators meet a bare digest_mismatch. Doing this once, together with #222, is materially cheaper than doing it twice.

Filed alongside #222 so that fixing one does not silently produce the other and read as a regression. ## The situation The XAML extractor declares two independent budgets that report two different codes: ```wat (global $max_tree_bytes i32 (i32.const 320000)) ;; WALK -> extract.truncated_tree (global $max_facts i32 (i32.const 4000)) ;; OUTPUT -> extract.node_limit_reached ``` plus a third, unnamed guard inside the scan loop, checked on **every iteration**: ```wat (if (i32.or (i32.ge_u (global.get $nf) (global.get $max_facts)) (i32.gt_u (global.get $bp) (i32.const 196608))) (then (local.set $full (i32.const 1)) (br $scan))) ``` Today the **tree** budget cuts first on every large file measured in production — six probes, all `extract.truncated_tree`, never `node_limit_reached`. That means `$max_facts` and the `$bp` guard are currently **untested against real markup**: nothing reaches them, so nobody knows where they bite. ## Why this must be settled with #222 rather than after it #222 raises `$max_tree_bytes` so files up to some new size complete. The moment the tree stops cutting, the fact budget becomes the binding constraint on exactly the files #222 is meant to fix — the theme dictionaries. `Generic.xaml` at 246.6 KB and `BaseTheme.Net.xaml` at 221.2 KB are dense `x:Name`/`x:Class`/`Click` markup, which is precisely what mints facts. The observable outcome would be: the same files still incomplete, the same tools still missing symbols, and a *different* diagnostic code. To anyone watching the code rather than the outcome that looks like a new defect introduced by the fix. To anyone watching the outcome it looks like the fix did nothing. **Both budgets must be sized against the same measured corpus, in one change, or neither number means anything.** ## What must be measured, not reasoned about For real markup at 100 KB, 250 KB and 1.3 MB (the production sizes in #222): 1. facts emitted per KB of XAML — the density that decides whether 4,000 is anywhere near right; 2. which budget binds first at each size, stated as a table; 3. bytes consumed in the fact buffer at each size, against the hard `$bp > 196608` guard — that guard is a *third* ceiling and it is not one of the two documented knobs; 4. total linear memory in use, against `(memory 8)` = 512 KiB initial with `src_offset` at 262,144, shared by source + tree + facts. The tree budget is the largest single consumer, so raising it moves this too. If the answer is that `Generic.xaml`-scale files cannot fit any budget triple within one wasm instance's memory, **that is a legitimate finding and should be stated** rather than papered over with a number that merely moves the wall past the current corpus. A streaming or chunked walk would then be the real fix, and this issue becomes its motivation. ## What a fix must prove * A fixture that exceeds `$max_facts` while the tree budget is comfortably clear, so the `node_limit_reached` path is graded at all — it currently is not, against real markup. * A fixture that exceeds the `$bp` buffer guard, or a written argument (with the arithmetic) for why `$max_facts` must always bind first. Right now that guard is a silent third ceiling. * The `truncated_tree` path still fires and still discloses, for a file beyond whatever new tree budget lands. * Both codes distinguishable end-to-end through `index_coverage`, not just at the extractor boundary. ## Mutations * Set `$max_facts` above anything reachable → the fact-budget assertion must go RED, or that path is untested. * Set the `$bp` guard above reachable → its assertion must go RED. If no test fails, the guard is ungraded and the fix has not covered the third ceiling. * Make the extractor report `truncated_tree` where it means `node_limit_reached` → the code-discrimination test must go RED. The two codes carry different operator meaning ("too big to read" vs "too rich to report") and must not be interchangeable. ## Cost, same as #222 Any change here is a package edit: every byte under `tests/packages/xaml/` is packed, so the `.cip` digest moves, `tests/packages/xaml.digest` must be re-recorded, the package version bumps, and the release notes must say so or operators meet a bare `digest_mismatch`. Doing this once, together with #222, is materially cheaper than doing it twice.

dhoyer referenced this issue from a commit

2026-09-08 13:23:49 +02:00

fix: the XAML extractor's three budgets, sized against a measurement (#222, #223)

buildagent referenced this issue

2026-09-08 16:07:57 +02:00

XAML $max_tree_bytes truncates real-world files from ~80 KB — measured on a 14,056-file production repo, ~50 files affected #222

buildagent commented

2026-09-08 16:08:28 +02:00

Author

Member

Fixed in ea74dbc, together with #222 — which is the whole point of this issue existing separately.

The premise held, and both budgets moved together

$max_facts 4,000 → 48,000, and the unnamed $bp > 196608 guard became a named $max_fact_bytes = 2,097,152 — a LENGTH from the buffer base rather than a raw address, and a global parameterised exactly like the other two so tests can drive it.

src_offset moved 262,144 → 2,162,688 (= 33 pages, matching the declared (memory 33)), which is what actually bought the fact buffer its room. That was the knob this issue was looking for.

The arithmetic this issue asked for DOES NOT EXIST, and that is the answer

This issue offered a choice: "a fixture that exceeds the $bp buffer guard, or a written argument with the arithmetic for why $max_facts must always bind first."

There is no such argument. A fact record spans 30 to 4,130 bytes (29 + name / 34 + name, name ceiling 4,096) — a 138× spread. No pair of constants can order a count against a byte total across that range. Which output budget binds first is a property of the document, not of the numbers.

So it is graded with a fixture instead, and both ceilings stay reachable on real documents: the count binds below a mean record of 2,097,152/48,000 = 43.7 bytes, which is — to two decimals — the measured production mean of 43.67. The byte ceiling binds above it. That is a much better place to sit than either budget being unreachable.

The third ceiling is named and graded, not silent

This issue's core worry was that raising the tree budget would move the wall invisibly. Measured, before → after, all under the old triple:

size	old	new
~100 KB dense	`truncated_tree`	complete
~250 KB dense / theme	`truncated_tree`	complete
~1.3 MB dense	`truncated_tree`	complete
2 MiB dense (the cap)	`truncated_tree`	complete

The walk cut first every time under the old budgets, confirming this issue's claim that $max_facts was ungraded against real markup. Both codes now fire on their own fixtures and are asserted end to end through symbols_overlapping().extraction_diagnostics — the exact carrier index_coverage reads — so truncated_tree and node_limit_reached are distinguishable at the surface, not merely at the extractor boundary.

extract.node_limit_reached is reachable at 49,000 × 7-byte x:Name (1,029,008 bytes of source, 48.8 facts/KB). The tree disclosure stays reachable inside the indexer's own 2 MiB limit via 880 KB of <b/> filler at a 20.00 expansion ratio.

The finding that outranks the budgets

Mutating the byte term out of the guard let 38,000 facts emit 2,432,000 buffer bytes — 271,384 of them past src_offset, over the source. It survived only because the walk is forward and the overwritten prefix had already been read.

That guard is a memory boundary wearing a budget's clothes. This issue called it "a silent third ceiling"; it is worse than that — it was the only thing preventing a buffer overrun into the guest's own view of the input. The new layout keeps 59,334 bytes of unreachable margin above the worst-case overshoot (guard plus one maximal 4,130-byte record).

Correcting this issue's memory model

Both errors were mine and both pointed the fix in the wrong direction:

"the tree budget is the largest single consumer [of memory]" — false. The host serialises the whole tree and grows guest memory to fit it before calling extract; the budget bounds fuel, not bytes.
"(memory 8) = 512 KiB initial … shared by source, tree and facts" — the real ceiling is MEMORY_CEILING_BYTES = 64 MiB. 512 KiB bounded only the fact buffer.

Consequence: this issue's speculation that Generic.xaml scale might be "architecturally out of range" and need a streaming walk was wrong. At 2 MiB of dense markup with the new budgets: 29,845 facts, no diagnostic. Memory at the extreme is ~14.8 MB against 64 MiB.

Eight mutations run, all RED, including both directions on each budget and a code-swap proving the two diagnostics are not interchangeable. Closing.

**Fixed in `ea74dbc`, together with #222** — which is the whole point of this issue existing separately. ## The premise held, and both budgets moved together `$max_facts` 4,000 → **48,000**, and the unnamed `$bp > 196608` guard became a named `$max_fact_bytes` = **2,097,152** — a LENGTH from the buffer base rather than a raw address, and a global parameterised exactly like the other two so tests can drive it. `src_offset` moved 262,144 → 2,162,688 (= 33 pages, matching the declared `(memory 33)`), which is what actually bought the fact buffer its room. That was the knob this issue was looking for. ## The arithmetic this issue asked for DOES NOT EXIST, and that is the answer This issue offered a choice: *"a fixture that exceeds the `$bp` buffer guard, **or** a written argument with the arithmetic for why `$max_facts` must always bind first."* There is no such argument. A fact record spans **30 to 4,130 bytes** (`29 + name` / `34 + name`, name ceiling 4,096) — a 138× spread. No pair of constants can order a count against a byte total across that range. Which output budget binds first is a property of the document, not of the numbers. So it is graded with a fixture instead, and both ceilings stay reachable on real documents: the count binds below a mean record of 2,097,152/48,000 = **43.7 bytes**, which is — to two decimals — the measured production mean of 43.67. The byte ceiling binds above it. That is a much better place to sit than either budget being unreachable. ## The third ceiling is named and graded, not silent This issue's core worry was that raising the tree budget would move the wall invisibly. Measured, before → after, all under the old triple: | size | old | new | |---|---|---| | ~100 KB dense | `truncated_tree` | complete | | ~250 KB dense / theme | `truncated_tree` | complete | | ~1.3 MB dense | `truncated_tree` | complete | | 2 MiB dense (the cap) | `truncated_tree` | complete | The walk cut first every time under the old budgets, confirming this issue's claim that `$max_facts` was ungraded against real markup. Both codes now fire on their own fixtures and are asserted end to end through `symbols_overlapping().extraction_diagnostics` — the exact carrier `index_coverage` reads — so `truncated_tree` and `node_limit_reached` are distinguishable at the surface, not merely at the extractor boundary. `extract.node_limit_reached` is reachable at 49,000 × 7-byte `x:Name` (1,029,008 bytes of source, 48.8 facts/KB). The tree disclosure stays reachable inside the indexer's own 2 MiB limit via 880 KB of `<b/>` filler at a 20.00 expansion ratio. ## The finding that outranks the budgets Mutating the byte term out of the guard let 38,000 facts emit **2,432,000 buffer bytes — 271,384 of them past `src_offset`, over the source**. It survived only because the walk is forward and the overwritten prefix had already been read. **That guard is a memory boundary wearing a budget's clothes.** This issue called it "a silent third ceiling"; it is worse than that — it was the only thing preventing a buffer overrun into the guest's own view of the input. The new layout keeps 59,334 bytes of unreachable margin above the worst-case overshoot (guard plus one maximal 4,130-byte record). ## Correcting this issue's memory model Both errors were mine and both pointed the fix in the wrong direction: * "the tree budget is the largest single consumer [of memory]" — **false**. The host serialises the whole tree and grows guest memory to fit it before calling `extract`; the budget bounds fuel, not bytes. * "`(memory 8)` = 512 KiB initial … shared by source, tree and facts" — the real ceiling is `MEMORY_CEILING_BYTES` = **64 MiB**. 512 KiB bounded only the fact buffer. Consequence: this issue's speculation that `Generic.xaml` scale might be "architecturally out of range" and need a streaming walk was wrong. At 2 MiB of dense markup with the new budgets: 29,845 facts, no diagnostic. Memory at the extreme is ~14.8 MB against 64 MiB. Eight mutations run, all RED, including both directions on each budget and a code-swap proving the two diagnostics are not interchangeable. Closing.

buildagent closed this issue

2026-09-08 16:08:32 +02:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

h-dv/code-index#223

No description provided.

Rows
Columns

XAML $max_facts is the next wall behind #222 — raising the tree budget alone will turn truncated_tree into node_limit_reached on the same files #223