h-dv/code-index

Watch

Fork 0

One oversized/deep/invalid construct drops the whole file, and four disclosure surfaces disagree about it (all languages) #545

New issue

Open

opened 2026-11-09 08:35:49 +01:00 by buildagent · 1 comment

buildagent commented

2026-11-09 08:35:49 +01:00

Member

Consolidated scope (triage 2026-11-09)

Priority: High: one bad construct drops a whole file and four disclosure surfaces disagree.

This issue is now the canonical tracker for:

From #42: fold its CRLF/BOM/refused-content corpus mutations into the hostile matrix here
From #288: a memory_ceiling_exceeded refusal reports observed: None; report the requested bytes

Original report

Summary

Across all languages, one bad construct throws away a whole file's extraction, and the disclosures disagree about it. Reproduced in Python, JavaScript, Rust, C#, PHP and XAML (installed build 2dcfc615):

Trigger	Result	Languages
A single fact string over the ABI maximum (a long signature or initializer: 64 KB string, 100 KB `1+1+…`, many-arg signature, deep type)	`fact.string_too_long`: the whole file is refused, including innocent symbols (`after_type`)	Python, JS, Rust
A 350-segment attribute/member chain (a 705-byte line); 50k-link chain	`host.deadline_exceeded (fuel)`: the whole file is refused. Python's `attribute` arm re-walks nested chains, roughly O(D²)–O(D³) (inferred)	Python, JS
A 1.3–1.4 MB file under the 2 MB cap	`host.memory_ceiling_exceeded`: whole file refused	Rust, C#
One stray non-UTF-8 byte (for example legacy Windows-1252 C#)	`content_refused`: whole file refused	all
Nesting past the depth guard	silent truncation: PHP extracted 257 of 3,000 nested functions, C# drops refs past about 200 levels, with no diagnostic	PHP, C#

The disclosures contradict each other:

project_overview.parse_error_files names the refused files.
resolution_gaps.partial_source_census reports total: 0: "every project … reported no incomplete extraction".
index_coverage says extraction_diagnostics: [], which is documented as "the measurement that there were none", even for the depth-truncated PHP file.
list_files reports a refused file as symbols: 0 without symbols_unmeasured.

The depth guard in each walker returns early without setting a flag. TRUNCATED_TREE fires only on a tree-decode failure (crates/guest/languages/src/wasm.rs:96).

Proposal (generic, in guest-kit and the host)

guest-kit clamps every emitted string to the ABI maximum on a char boundary, with a truncation marker. Ruby's private truncate_at_char already does this, so move it to guest-kit. Signatures stop at the first line or 512 bytes.
guest-kit records a depth_limit_reached diagnostic whenever any walker hits its guard. The file becomes partial, never silently complete.
Python/JS walkers do not re-descend into a member chain they have already emitted.
Decode invalid UTF-8 lossily and record a diagnostic, instead of refusing the file.
One partial-source census: fact refusals, fuel/memory refusals, depth truncation and lossy decode all count in partial_source_census, extraction_diagnostics and list_files.symbols_unmeasured. A registry test asserts that the three surfaces agree on a hostile fixture set.

Acceptance

A shared hostile-fixture matrix across the 6 languages and the framework packages: deep nesting, long lines, oversized strings, invalid bytes, long chains.
Each case either extracts with a diagnostic, or is refused and reported identically by all four surfaces.
Mutation: remove the clamp or the depth flag, and the matrix goes red.

Related: #448 (false-zero disclosures), #41 (scale ceilings).

Filed from the per-language review on 2026-11-09: 8 agents, one per language plus one for Vue/Svelte/XAML/timeline, run against copies of the pinned corpus repos (tests/corpus/corpus.toml) using installed build 2dcfc615, which differs from master only by XAML/CI commits. The top claim of every issue was spot-checked in the main session. Artifacts live under the review session's scratchpad lang-out/<lang>/ and are available on request.

## Consolidated scope (triage 2026-11-09) **Priority: High**: one bad construct drops a whole file and four disclosure surfaces disagree. This issue is now the canonical tracker for: - [ ] From #42: fold its CRLF/BOM/refused-content corpus mutations into the hostile matrix here - [ ] From #288: a `memory_ceiling_exceeded` refusal reports `observed: None`; report the requested bytes --- ## Original report ## Summary Across all languages, **one bad construct throws away a whole file's extraction**, and the disclosures **disagree** about it. Reproduced in Python, JavaScript, Rust, C#, PHP and XAML (installed build `2dcfc615`): | Trigger | Result | Languages | |---|---|---| | A single fact string over the ABI maximum (a long signature or initializer: 64 KB string, 100 KB `1+1+…`, many-arg signature, deep type) | `fact.string_too_long`: **the whole file is refused**, including innocent symbols (`after_type`) | Python, JS, Rust | | A 350-segment attribute/member chain (a 705-byte line); 50k-link chain | `host.deadline_exceeded (fuel)`: the whole file is refused. Python's `attribute` arm re-walks nested chains, roughly O(D²)–O(D³) (inferred) | Python, JS | | A 1.3–1.4 MB file under the 2 MB cap | `host.memory_ceiling_exceeded`: whole file refused | Rust, C# | | One stray non-UTF-8 byte (for example legacy Windows-1252 C#) | `content_refused`: whole file refused | all | | Nesting past the depth guard | **silent truncation**: PHP extracted 257 of 3,000 nested functions, C# drops refs past about 200 levels, with no diagnostic | PHP, C# | **The disclosures contradict each other:** - `project_overview.parse_error_files` names the refused files. - `resolution_gaps.partial_source_census` reports `total: 0`: "every project … reported no incomplete extraction". - `index_coverage` says `extraction_diagnostics: []`, which is documented as "the measurement that there were none", even for the depth-truncated PHP file. - `list_files` reports a refused file as `symbols: 0` **without** `symbols_unmeasured`. The depth guard in each walker returns early without setting a flag. `TRUNCATED_TREE` fires only on a tree-decode failure (`crates/guest/languages/src/wasm.rs:96`). ## Proposal (generic, in guest-kit and the host) 1. **guest-kit clamps every emitted string** to the ABI maximum on a char boundary, with a truncation marker. Ruby's private `truncate_at_char` already does this, so move it to guest-kit. Signatures stop at the first line or 512 bytes. 2. **guest-kit records a `depth_limit_reached` diagnostic** whenever any walker hits its guard. The file becomes *partial*, never silently complete. 3. Python/JS walkers do not re-descend into a member chain they have already emitted. 4. Decode invalid UTF-8 lossily and record a diagnostic, instead of refusing the file. 5. **One partial-source census**: fact refusals, fuel/memory refusals, depth truncation and lossy decode all count in `partial_source_census`, `extraction_diagnostics` and `list_files.symbols_unmeasured`. A registry test asserts that the three surfaces agree on a hostile fixture set. ## Acceptance - A shared hostile-fixture matrix across the 6 languages and the framework packages: deep nesting, long lines, oversized strings, invalid bytes, long chains. - Each case either extracts with a diagnostic, or is refused **and** reported identically by all four surfaces. - Mutation: remove the clamp or the depth flag, and the matrix goes red. Related: #448 (false-zero disclosures), #41 (scale ceilings). --- _Filed from the per-language review on 2026-11-09: 8 agents, one per language plus one for Vue/Svelte/XAML/timeline, run against copies of the pinned corpus repos (`tests/corpus/corpus.toml`) using installed build `2dcfc615`, which differs from master only by XAML/CI commits. The top claim of every issue was spot-checked in the main session. Artifacts live under the review session's scratchpad `lang-out/<lang>/` and are available on request._

buildagent added the

labels

2026-11-09 08:35:49 +01:00

buildagent referenced this issue

2026-11-09 08:35:50 +01:00

First contact: an SFC whose script language is not enabled loses all facts; package-only extensions never activate; answers precede activation #547

buildagent referenced this issue

2026-11-09 08:35:51 +01:00

Symbols carry no documentation and signatures are unreadable (attribute lists, CRLF; 11 Query overloads look identical) #550

buildagent referenced this issue

2026-11-09 08:36:25 +01:00

v0.41.17 field assessment: false-zero disclosures, import precision and daemon approval concurrency need reproduction #448

buildagent referenced this issue

2026-11-12 05:46:24 +01:00

v0.41.50: module evidence for qualified references, one argument contract, one extraction census #574

buildagent commented

2026-11-12 11:41:24 +01:00

Author

Member

Released in v0.41.50 (host side).

One census of incomplete extraction now feeds project_overview (extraction_diagnostics.files, parse_error_files), resolution_gaps.partial_source_census, index_coverage.extraction_diagnostics, list_files and evidence_gaps. A file the host refused (fact.string_too_long, host.deadline_exceeded, host.memory_ceiling_exceeded, content_refused) is counted by all of them.
A partially extracted file keeps its symbol count in list_files, marked symbols_are_floor: true with its diagnostic codes.
A source file that is not valid UTF-8 is recorded as content_refused instead of being absent.

Still open here:

Guest side: clamping oversized strings, the depth-limit diagnostic, the member-chain re-walk, lossy decoding. Each needs a release of every language package.
The requested bytes of a host.memory_ceiling_exceeded refusal are not reported; that needs a worker/supervisor protocol change.

Released in v0.41.50 (host side). - One census of incomplete extraction now feeds `project_overview` (`extraction_diagnostics.files`, `parse_error_files`), `resolution_gaps.partial_source_census`, `index_coverage.extraction_diagnostics`, `list_files` and `evidence_gaps`. A file the host refused (`fact.string_too_long`, `host.deadline_exceeded`, `host.memory_ceiling_exceeded`, `content_refused`) is counted by all of them. - A partially extracted file keeps its symbol count in `list_files`, marked `symbols_are_floor: true` with its diagnostic codes. - A source file that is not valid UTF-8 is recorded as `content_refused` instead of being absent. Still open here: - Guest side: clamping oversized strings, the depth-limit diagnostic, the member-chain re-walk, lossy decoding. Each needs a release of every language package. - The requested bytes of a `host.memory_ceiling_exceeded` refusal are not reported; that needs a worker/supervisor protocol change.

buildagent referenced this issue

2026-11-12 18:41:44 +01:00

v0.41.51: bounded text arguments, truthful path and startup answers, review grading by effective visibility #580

buildagent referenced this issue

2026-11-12 22:22:54 +01:00

Regression gates miss corpus invariants, independent targets and test-build cost #42