h-dv/code-index

Watch

Fork 0

XAML `$max_tree_bytes` truncates real-world files from ~80 KB — measured on a 14,056-file production repo, ~50 files affected #222

New issue

Closed

opened 2026-09-08 09:40:42 +02:00 by buildagent · 1 comment

buildagent commented

2026-09-08 09:40:42 +02:00

Member

Found in production use, not in a fixture. Reported from a 14,056-file WPF repository with 30,461 XAML field symbols.

Measured

Every affected file reports exactly extract.truncated_tree, never extract.node_limit_reached, across six probes. parse_errors: 0 — this is not a parse failure and not host give-up. It is the extractor bounding its own walk, writing what it had, and declaring it, which is the departure from "budget exhaustion is a refusal" that #80 explicitly permits.

file	size	verdict
`ScaleBackgroundLayerModels.xaml`	1361.6 KB	truncated
`wndKommissionierenForBA.xaml`	98.6 KB	truncated
`wndProduktionsKontrollplanElectronics.xaml`	97.1 KB	truncated
`wndAbstempelnBDE.xaml`	90.0 KB	truncated
`xRichTextBoxDevExpr.xaml`	77.5 KB	clean
`wndShopItem.xaml`	64.1 KB	clean

The cut sits between ~78 and ~90 KB of source. Reported count was 42, correctly identified by the reporter as a FLOOR rather than a total — the 43rd is truncated too, the index was mid schema-upgrade rebuild (5,041 of 14,056 files carrying the invalidation sentinel), and 44 candidates are ≥97 KB with 58 ≥78 KB. Expect it to settle in the low-to-mid 50s.

The files affected are the ones that matter most in that repo: Generic.xaml (246.6 KB), BaseTheme.Net.xaml (221.2 KB), wndBeleg.xaml (160.3 KB), BaseTheme.xaml (133.3 KB), AiChatControl.xaml (124.3 KB) — the theme dictionaries and the main controls.

The mechanism, and it is NOT node count

tests/packages/xaml.extractor.wat:

(global $max_tree_bytes i32 (i32.const 320000))   ;; bounds the WALK   -> extract.truncated_tree
(global $max_facts      i32 (i32.const 4000))     ;; bounds the OUTPUT -> extract.node_limit_reached

$max_tree_bytes bounds the serialized tree-sitter tree ($tl, the length the host passes alongside $tp), not source bytes and not node count:

(if (i32.gt_u (local.get $tl) (global.get $max_tree_bytes))
  (then (local.set $end (i32.add (local.get $tp) (global.get $max_tree_bytes)))
        (local.set $cut (i32.const 1))))

A serialized tree is several times its source, which is exactly why a 320,000-byte budget bites at ~80 KB of XAML, and why markup density moves the threshold. The reporter's inference that it was "node-count driven" is the one budget that is NOT firing — $max_facts reports a different code that never appeared in any probe.

Why this is worth fixing rather than documenting

For a truncated file, symbols past the cut are missing from every structural tool: file_outline, search_symbols, find_references, change_impact. The disclosure is honest and it is per-file, so a reader who checks is not misled — but the reader has to check, and the affected set is the largest and most-referenced files in the repository.

It also compounds with the XAML use-channel gap: the package declares resolver = ["bridge_source"] with both bridges pointing outward, so nothing emits a ref targeting a XAML symbol and ref_count on an x:Name field is structurally unmeasured (30,461 symbols in kinds_without_use_channel). On a truncated large file, a "this style is unused" conclusion is unsupported twice — the symbol may not be recorded at all, and even where it is, its zero was never measured against anything.

What the fix must weigh

The budget is deliberately one line — the extractor's own comment says the globals exist "so that one line is the whole knob", and plugin_pkg.rs::xaml_extractor rewrites exactly those two lines to make a budget fire on a small file. So the edit is trivial and the consequences are not:

The fact budget becomes the next wall. Filed separately — a 246 KB Generic.xaml full of x:Name may exceed $max_facts = 4000 once the tree budget stops cutting first, turning truncated_tree into node_limit_reached on the same files. Raising one budget alone would look like a regression to anyone watching the diagnostic rather than the outcome.
$bp > 196608 is a hard guard on the fact output buffer, checked every iteration of the scan.
(memory 8) — 512 KiB initial linear memory, with src_offset at 262,144, shared between source, tree and fact buffer.
The package digest moves. Every byte under tests/packages/xaml/ is packed, so this is a package version bump (0.1.0 → next), a re-record of tests/packages/xaml.digest, and a release note — an operator who does not re-pin gets a bare digest_mismatch and no other explanation. xaml_package_e2e.rs::the_shipped_package_is_the_artifact_that_was_recorded stays RED until the digests are re-recorded.

What a fix must prove

A fixture at each side of the new threshold, so the boundary is graded rather than assumed. The current fixtures are tiny; nothing in the tree exercises an 80 KB XAML file.
That a file which previously truncated now completes — and one that still truncates still says so. Both directions, or the fix cannot be told from removing the diagnostic.
The interaction with $max_facts measured, not reasoned about: state which budget binds first at 100 KB, 250 KB and 1.3 MB of real markup.
Memory headroom stated as a number, since source + tree + facts share 512 KiB initial and the tree budget is the largest single consumer.

Mutations

Restore $max_tree_bytes to 320000 → the "large file completes" assertion must go RED.
Set it to a value that cannot cut anything → the "still truncates and says so" assertion must go RED, or the fix has simply deleted the disclosure.
Leave the digest un-re-recorded → the_shipped_package_is_the_artifact_that_was_recorded must go RED (it already does; confirm it still does after the bump).

Related: the fact-budget follow-up, and the XAML use-channel question (a bridge_source-only package means ref_count on x:Name fields is structurally unmeasured).

Found in **production use**, not in a fixture. Reported from a 14,056-file WPF repository with 30,461 XAML field symbols. ## Measured Every affected file reports exactly `extract.truncated_tree`, never `extract.node_limit_reached`, across six probes. `parse_errors: 0` — this is not a parse failure and not host give-up. It is the extractor bounding its own walk, writing what it had, and declaring it, which is the departure from "budget exhaustion is a refusal" that #80 explicitly permits. | file | size | verdict | |---|---:|---| | `ScaleBackgroundLayerModels.xaml` | 1361.6 KB | truncated | | `wndKommissionierenForBA.xaml` | 98.6 KB | truncated | | `wndProduktionsKontrollplanElectronics.xaml` | 97.1 KB | truncated | | `wndAbstempelnBDE.xaml` | 90.0 KB | truncated | | `xRichTextBoxDevExpr.xaml` | 77.5 KB | **clean** | | `wndShopItem.xaml` | 64.1 KB | **clean** | The cut sits between **~78 and ~90 KB of source**. Reported count was 42, correctly identified by the reporter as a FLOOR rather than a total — the 43rd is truncated too, the index was mid schema-upgrade rebuild (5,041 of 14,056 files carrying the invalidation sentinel), and 44 candidates are ≥97 KB with 58 ≥78 KB. Expect it to settle in the low-to-mid 50s. The files affected are the ones that matter most in that repo: `Generic.xaml` (246.6 KB), `BaseTheme.Net.xaml` (221.2 KB), `wndBeleg.xaml` (160.3 KB), `BaseTheme.xaml` (133.3 KB), `AiChatControl.xaml` (124.3 KB) — the theme dictionaries and the main controls. ## The mechanism, and it is NOT node count `tests/packages/xaml.extractor.wat`: ```wat (global $max_tree_bytes i32 (i32.const 320000)) ;; bounds the WALK -> extract.truncated_tree (global $max_facts i32 (i32.const 4000)) ;; bounds the OUTPUT -> extract.node_limit_reached ``` `$max_tree_bytes` bounds the **serialized tree-sitter tree** (`$tl`, the length the host passes alongside `$tp`), not source bytes and not node count: ```wat (if (i32.gt_u (local.get $tl) (global.get $max_tree_bytes)) (then (local.set $end (i32.add (local.get $tp) (global.get $max_tree_bytes))) (local.set $cut (i32.const 1)))) ``` A serialized tree is several times its source, which is exactly why a 320,000-byte budget bites at ~80 KB of XAML, and why markup density moves the threshold. The reporter's inference that it was "node-count driven" is the one budget that is NOT firing — `$max_facts` reports a different code that never appeared in any probe. ## Why this is worth fixing rather than documenting For a truncated file, symbols past the cut are missing from **every** structural tool: `file_outline`, `search_symbols`, `find_references`, `change_impact`. The disclosure is honest and it is per-file, so a reader who checks is not misled — but the reader has to check, and the affected set is the largest and most-referenced files in the repository. It also **compounds with the XAML use-channel gap**: the package declares `resolver = ["bridge_source"]` with both bridges pointing outward, so nothing emits a ref targeting a XAML symbol and `ref_count` on an `x:Name` field is structurally unmeasured (30,461 symbols in `kinds_without_use_channel`). On a truncated large file, a "this style is unused" conclusion is unsupported **twice** — the symbol may not be recorded at all, and even where it is, its zero was never measured against anything. ## What the fix must weigh The budget is deliberately one line — the extractor's own comment says the globals exist "so that one line is the whole knob", and `plugin_pkg.rs::xaml_extractor` rewrites exactly those two lines to make a budget fire on a small file. So the edit is trivial and the consequences are not: * **The fact budget becomes the next wall.** Filed separately — a 246 KB `Generic.xaml` full of `x:Name` may exceed `$max_facts = 4000` once the tree budget stops cutting first, turning `truncated_tree` into `node_limit_reached` on the same files. Raising one budget alone would look like a regression to anyone watching the diagnostic rather than the outcome. * **`$bp > 196608`** is a hard guard on the fact output buffer, checked every iteration of the scan. * **`(memory 8)`** — 512 KiB initial linear memory, with `src_offset` at 262,144, shared between source, tree and fact buffer. * **The package digest moves.** Every byte under `tests/packages/xaml/` is packed, so this is a package version bump (`0.1.0` → next), a re-record of `tests/packages/xaml.digest`, and a release note — an operator who does not re-pin gets a bare `digest_mismatch` and no other explanation. `xaml_package_e2e.rs::the_shipped_package_is_the_artifact_that_was_recorded` stays RED until the digests are re-recorded. ## What a fix must prove * A fixture at each side of the new threshold, so the boundary is graded rather than assumed. The current fixtures are tiny; nothing in the tree exercises an 80 KB XAML file. * That a file which previously truncated now completes — and one that still truncates still says so. **Both directions**, or the fix cannot be told from removing the diagnostic. * The interaction with `$max_facts` measured, not reasoned about: state which budget binds first at 100 KB, 250 KB and 1.3 MB of real markup. * Memory headroom stated as a number, since source + tree + facts share 512 KiB initial and the tree budget is the largest single consumer. ## Mutations * Restore `$max_tree_bytes` to 320000 → the "large file completes" assertion must go RED. * Set it to a value that cannot cut anything → the "still truncates and says so" assertion must go RED, or the fix has simply deleted the disclosure. * Leave the digest un-re-recorded → `the_shipped_package_is_the_artifact_that_was_recorded` must go RED (it already does; confirm it still does after the bump). Related: the fact-budget follow-up, and the XAML use-channel question (a `bridge_source`-only package means `ref_count` on `x:Name` fields is structurally unmeasured).

buildagent referenced this issue

2026-09-08 09:41:09 +02:00

XAML $max_facts is the next wall behind #222 — raising the tree budget alone will turn truncated_tree into node_limit_reached on the same files #223

dhoyer referenced this issue from a commit

2026-09-08 13:23:49 +02:00

fix: the XAML extractor's three budgets, sized against a measurement (#222, #223)

buildagent commented

2026-09-08 16:07:57 +02:00

Author

Member

Fixed in ea74dbc (package de.h-dv.xaml 0.2.0). Landed together with #223, because fixing either alone would have moved the wall rather than removed it.

The measurement, which is the deliverable

Bisected through the real grammar, real worker and real parent validator — tree length recovered by bisecting the budget itself (the smallest budget at which truncated_tree does not fire IS the tree length).

shape	source	tree bytes	tree/src	facts	facts/KB	bytes/fact
dense WPF window	102,574	516,000	5.03	1,465	14.6	43.67
dense WPF window	256,294	1,288,992	5.03	3,661	14.6	43.67
dense WPF window	1,331,559	6,685,152	5.02	18,991	14.6	43.67
theme dictionary	293,172	1,536,384	5.24	0	0.0	—
`<b/>` filler	160,045	3,200,528	20.00	2	0.0	38.50

Three results worth keeping:

The fact-record formula is exact — 43.67 / 41.00 / 38.50 bytes per fact are 29 + name (ref) and 34 + name (symbol) to the byte on three unrelated shapes. That licenses arithmetic assertions rather than fitted constants.
The tree ratio is a property of the markup, not the size — 5.02–5.03 across a 13× range, 5.24 for theme dictionaries, 20.00 for empty elements. So 320,000 cut at ~64 KB here and ~80 KB on the reporter's sparser files. The mechanism in this issue is confirmed exactly.
Which output budget binds first is a property of the document, not something a pair of numbers can order: facts span 30–4,130 bytes, a 138× spread.

What shipped

All three budgets now derive from FILE_SIZE_LIMIT_BYTES (2 MiB — the largest file the indexer hands any extractor), not from the corpus of the day, which was this issue's actual root cause.

	old	new	derivation
`$max_tree_bytes`	320,000	16,777,216	2 MiB × 8; worst measured ratio 5.24 → 1.53× margin
`$max_facts`	4,000	48,000	2 MiB at 14.6 facts/KB = 29,900 → 1.61×
`$max_fact_bytes`	unnamed `196608`	2,097,152	one buffer byte per source byte; measured 0.624 → 1.60×

Every production size in this report now completes. At 2 MiB of dense markup: 29,845 facts, no diagnostic.

Two things this issue got wrong about the memory model

Both were mine, and both mattered:

$max_tree_bytes costs no memory at all. The host serialises the whole tree into guest memory and grows it to fit before calling extract. The budget bounds work (fuel), not bytes. This issue's claim that it was "the largest single consumer" of memory was simply false.
(memory 8) = 512 KiB was never the ceiling. The real one is MEMORY_CEILING_BYTES = 64 MiB. What 512 KiB actually bounded was the fact buffer, because it sits below src_offset — so the knob for the third guard was src_offset all along. A one-line move, not the streaming rewrite this issue contemplated.

Generic.xaml scale is reachable. Streaming is not needed.

One finding worth more than the budgets

With the byte guard deleted as a mutation, 38,000 facts wrote 271,384 bytes past src_offset, over the source. It survived only because the walk is forward and the overwritten prefix had already been read. That guard is a memory boundary, not a budget — the new layout keeps 59,334 bytes of margin above the worst-case overshoot.

Operator impact

Digest moved to sha256:7b572f5cc32ffcd5af550900a451bbef37e255804776d7e3e3e71d2dd7d0aa79; the release asset is de.h-dv.xaml-0.2.0.cip. Re-pin, or get a bare digest_mismatch. extraction_identity moved too, so every project with this package enabled re-extracts its .xaml files once — which is the point.

Documented in the v0.27.0 release notes as its own upgrade section. Note v0.27.0 is tagged but not yet published — blocked on an unrelated windows-gnu archive fault — so the asset is not downloadable yet.

Eight mutations run, all RED. Closing.

**Fixed in `ea74dbc`** (package `de.h-dv.xaml` 0.2.0). Landed together with #223, because fixing either alone would have moved the wall rather than removed it. ## The measurement, which is the deliverable Bisected through the real grammar, real worker and real parent validator — tree length recovered by bisecting the budget itself (the smallest budget at which `truncated_tree` does not fire IS the tree length). | shape | source | tree bytes | tree/src | facts | facts/KB | bytes/fact | |---|---:|---:|---:|---:|---:|---:| | dense WPF window | 102,574 | 516,000 | 5.03 | 1,465 | 14.6 | 43.67 | | dense WPF window | 256,294 | 1,288,992 | 5.03 | 3,661 | 14.6 | 43.67 | | dense WPF window | 1,331,559 | 6,685,152 | 5.02 | 18,991 | 14.6 | 43.67 | | theme dictionary | 293,172 | 1,536,384 | 5.24 | 0 | 0.0 | — | | `<b/>` filler | 160,045 | 3,200,528 | **20.00** | 2 | 0.0 | 38.50 | Three results worth keeping: * **The fact-record formula is exact** — 43.67 / 41.00 / 38.50 bytes per fact are `29 + name` (ref) and `34 + name` (symbol) to the byte on three unrelated shapes. That licenses arithmetic assertions rather than fitted constants. * **The tree ratio is a property of the markup, not the size** — 5.02–5.03 across a 13× range, 5.24 for theme dictionaries, 20.00 for empty elements. So `320,000` cut at ~64 KB here and ~80 KB on the reporter's sparser files. The mechanism in this issue is confirmed exactly. * **Which output budget binds first is a property of the document**, not something a pair of numbers can order: facts span 30–4,130 bytes, a 138× spread. ## What shipped All three budgets now derive from `FILE_SIZE_LIMIT_BYTES` (2 MiB — the largest file the indexer hands **any** extractor), not from the corpus of the day, which was this issue's actual root cause. | | old | new | derivation | |---|---:|---:|---| | `$max_tree_bytes` | 320,000 | 16,777,216 | 2 MiB × 8; worst measured ratio 5.24 → 1.53× margin | | `$max_facts` | 4,000 | 48,000 | 2 MiB at 14.6 facts/KB = 29,900 → 1.61× | | `$max_fact_bytes` | unnamed `196608` | 2,097,152 | one buffer byte per source byte; measured 0.624 → 1.60× | Every production size in this report now completes. At 2 MiB of dense markup: **29,845 facts, no diagnostic.** ## Two things this issue got wrong about the memory model Both were mine, and both mattered: 1. **`$max_tree_bytes` costs no memory at all.** The host serialises the whole tree into guest memory and grows it to fit *before* calling `extract`. The budget bounds **work** (fuel), not bytes. This issue's claim that it was "the largest single consumer" of memory was simply false. 2. **`(memory 8)` = 512 KiB was never the ceiling.** The real one is `MEMORY_CEILING_BYTES` = 64 MiB. What 512 KiB actually bounded was the *fact buffer*, because it sits below `src_offset` — so the knob for the third guard was `src_offset` all along. A one-line move, not the streaming rewrite this issue contemplated. **`Generic.xaml` scale is reachable.** Streaming is not needed. ## One finding worth more than the budgets With the byte guard deleted as a mutation, 38,000 facts wrote **271,384 bytes past `src_offset`, over the source**. It survived only because the walk is forward and the overwritten prefix had already been read. **That guard is a memory boundary, not a budget** — the new layout keeps 59,334 bytes of margin above the worst-case overshoot. ## Operator impact Digest moved to `sha256:7b572f5cc32ffcd5af550900a451bbef37e255804776d7e3e3e71d2dd7d0aa79`; the release asset is `de.h-dv.xaml-0.2.0.cip`. Re-pin, or get a bare `digest_mismatch`. `extraction_identity` moved too, so every project with this package enabled re-extracts its `.xaml` files once — which is the point. Documented in the v0.27.0 release notes as its own upgrade section. Note v0.27.0 is tagged but **not yet published** — blocked on an unrelated windows-gnu archive fault — so the asset is not downloadable yet. Eight mutations run, all RED. Closing.

buildagent closed this issue

2026-09-08 16:08:03 +02:00

buildagent referenced this issue

2026-09-08 16:08:28 +02:00

XAML $max_facts is the next wall behind #222 — raising the tree budget alone will turn truncated_tree into node_limit_reached on the same files #223

dhoyer referenced this issue from a commit

2026-09-10 00:16:08 +02:00

fix(#221): the shared bless verdict gets a third arm — a CONDITION move

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

h-dv/code-index#222

No description provided.

Rows
Columns

XAML $max_tree_bytes truncates real-world files from ~80 KB — measured on a 14,056-file production repo, ~50 files affected #222