Phase 0 of pluggable languages: route language rules through profiles, measure costs, add gates #306

Closed

buildagent wants to merge 0 commits from p0-integration into master

pull from: p0-integration

merge into: h-dv:master

h-dv:master

h-dv:p1-integration

h-dv:release/v0.32.2

h-dv:release/v0.32.1

h-dv:epic-i072-release-032

h-dv:windows-installer

h-dv:i269b-registry-truth

h-dv:probe-win-baseline

h-dv:fix/246-shadowed-imports

h-dv:release/0.28.1

h-dv:fix/review-followthrough-20260911

h-dv:iwin-msvc-archive

Conversation 1 Commits - Files changed -

buildagent commented 2026-09-26 19:07:10 +02:00

Member

Copy link

Phase 0 prepares the move of all languages into plugin packages: it measures the costs, adds the gates and makes rules follow the language profile. It does not move any language yet. Built-in results are unchanged: a bind-by-bind comparison over all nine corpus repositories found 0 differences among 1,140,041 refs.

Runtime and measurement (lane M)

• OOM priority: plugin workers set oom_score_adj=1000, so the Linux OOM killer picks a worker before the daemon.
• Real-grammar costs: worker costs were measured with the grammars we actually ship (_prdoc/records/P0-real-grammar-costs.md). Ruby takes about 175 ms to start cold and uses 35 MB for the first worker.
• Parallel precompile: package discovery compiles every approved extractor in parallel.
• Idle retirement: pool lanes retire a worker after 60 s idle.

Language rules follow the profile (lane N)

• Profile routing: every language id that was written inline outside the exempt files now goes through lang_profile. There were 57 literals at 41 sites; there are now 0, and lang_literal_gate enforces that with no allowlist.
  • The resolver's SQL reads profile traits from a per-pass temp.lang_profile table built in Rust.
• Language filters: a lang filter given as a profile name (lang: "ruby") also matches packaged languages that use that profile.
• W0: a packaged Ruby makes the same local-variable decisions as the builtin, with a fixture that fails when either half of the old check is restored.
• W7: a transient package refusal (worker unavailable, deadline exceeded, restart backoff, quarantine) keeps the file's previous facts and marks the file stale instead of deleting them.

Gates and release infrastructure (lane O)

• Grammar builds: one parameterized build recipe for grammars, plus the wchar.h shim PHP needs.
• Parity gate: a generic package_parity gate driven by a LangSpec, with two new axes.
• Per-language corpus runs: COSI_CORPUS_LANGS runs the corpus ratchets for selected languages only.

Integration fixes

• Packaged C# is isolated from builtin C#, and gated. Routing through the profile let a packaged <pkg>/csharp language enter the builtin C# partial-class and reclassification rules. It had no language key and no capability gate, so a packaged Widget could merge with a builtin Widget.
  • All five relations now key on lang, and the partial-class join requires the same language on both sides.
  • Their symbol reads take member_candidate or type_position_candidate.
  • The capability registry records each relation's new stance.
  • New tests use the real package id, and their mutations were run.
• Fork lock: the parallel_precompile test and the package fixture's host build now take the fork lock.
• Test isolation: plugin_activation_cli's bench no longer inherits COSI_CORPUS_DIR. The inherited variable made it reconcile every corpus checkout and read another root's log line.
• Test module: ruby_package_local_decisions declares package_parity, which the fixture now imports.

Validation

• Full local gate set, run once on the integrated tree:
  • fmt and strict rustdoc clean.
  • Workspace clippy and Windows clippy clean.
  • Daemon leg passed.
  • All 13 CI corpus suites passed: 71 tests, corpus baselines untouched.
  • agent_task_bench passed.
  • Precision gate 7/7 with 0 phantoms.
• Workspace suite on that run: 6 targets failed.
  • 3 were real defects, fixed above.
  • 3 were build-hash mismatches caused by HEAD moving during the run.
  • All 6 pass on this head.
• Tests re-run after the fixes:
  • The full indexer crate: 1,305 passed.
  • Release corpus ratchet, mutation, stage and cost; precision 7/7.
  • I ran the partial-join mutation independently: the test fails when pf.lang = c.lang is removed.
• Known and not fixed: pc_calls picks the enclosing class from ungated symbols. This would only matter with two producers in one file, which no shipped path creates yet.

🤖 Generated with Claude Code

https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

Phase 0 prepares the move of all languages into plugin packages: it measures the costs, adds the gates and makes rules follow the language profile. It does not move any language yet. Built-in results are unchanged: a bind-by-bind comparison over all nine corpus repositories found 0 differences among 1,140,041 refs. ## Runtime and measurement (lane M) - **OOM priority:** plugin workers set `oom_score_adj=1000`, so the Linux OOM killer picks a worker before the daemon. - **Real-grammar costs:** worker costs were measured with the grammars we actually ship (`_prdoc/records/P0-real-grammar-costs.md`). Ruby takes about 175 ms to start cold and uses 35 MB for the first worker. - **Parallel precompile:** package discovery compiles every approved extractor in parallel. - **Idle retirement:** pool lanes retire a worker after 60 s idle. ## Language rules follow the profile (lane N) - **Profile routing:** every language id that was written inline outside the exempt files now goes through `lang_profile`. There were 57 literals at 41 sites; there are now 0, and `lang_literal_gate` enforces that with no allowlist. - The resolver's SQL reads profile traits from a per-pass `temp.lang_profile` table built in Rust. - **Language filters:** a `lang` filter given as a profile name (`lang: "ruby"`) also matches packaged languages that use that profile. - **W0:** a packaged Ruby makes the same local-variable decisions as the builtin, with a fixture that fails when either half of the old check is restored. - **W7:** a transient package refusal (worker unavailable, deadline exceeded, restart backoff, quarantine) keeps the file's previous facts and marks the file stale instead of deleting them. ## Gates and release infrastructure (lane O) - **Grammar builds:** one parameterized build recipe for grammars, plus the `wchar.h` shim PHP needs. - **Parity gate:** a generic `package_parity` gate driven by a `LangSpec`, with two new axes. - **Per-language corpus runs:** `COSI_CORPUS_LANGS` runs the corpus ratchets for selected languages only. ## Integration fixes - **Packaged C# is isolated from builtin C#, and gated.** Routing through the profile let a packaged `<pkg>/csharp` language enter the builtin C# partial-class and reclassification rules. It had no language key and no capability gate, so a packaged `Widget` could merge with a builtin `Widget`. - All five relations now key on `lang`, and the partial-class join requires the same language on both sides. - Their symbol reads take `member_candidate` or `type_position_candidate`. - The capability registry records each relation's new stance. - New tests use the real package id, and their mutations were run. - **Fork lock:** the `parallel_precompile` test and the package fixture's host build now take the fork lock. - **Test isolation:** `plugin_activation_cli`'s bench no longer inherits `COSI_CORPUS_DIR`. The inherited variable made it reconcile every corpus checkout and read another root's log line. - **Test module:** `ruby_package_local_decisions` declares `package_parity`, which the fixture now imports. ## Validation - **Full local gate set, run once on the integrated tree:** - fmt and strict rustdoc clean. - Workspace clippy and Windows clippy clean. - Daemon leg passed. - All 13 CI corpus suites passed: 71 tests, corpus baselines untouched. - `agent_task_bench` passed. - Precision gate 7/7 with 0 phantoms. - **Workspace suite on that run:** 6 targets failed. - 3 were real defects, fixed above. - 3 were build-hash mismatches caused by HEAD moving during the run. - All 6 pass on this head. - **Tests re-run after the fixes:** - The full indexer crate: 1,305 passed. - Release corpus ratchet, mutation, stage and cost; precision 7/7. - I ran the partial-join mutation independently: the test fails when `pf.lang = c.lang` is removed. - **Known and not fixed:** `pc_calls` picks the enclosing class from ungated symbols. This would only matter with two producers in one file, which no shipped path creates yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

buildagent added 22 commits 2026-09-26 19:07:11 +02:00

grammars: one parameterized build recipe, and the wchar.h shim php needs ... d53b64dc34

tests/grammars/build-tree-sitter.sh is the recipe the three per-grammar
scripts each claimed to be a copy of. It takes every value as a required
flag (crate, version, crate sha256, source subdir, export symbol, wasm
sha256, output name); a default would be one grammar's value applied to
another. `--wasm-sha256 -` builds an UNPINNED grammar, prints its digest
and exits 3, never 0.

build-tree-sitter-{xml,ruby,svelte}.sh keep their provenance blocks and
their plain VERSION=/CRATE_SHA256=/WASM_SHA256= assignments, because
grammar_provenance.rs and ruby_kind_table.rs read those lines, and then
exec the recipe.

shim/wchar.h: tree-sitter-php 0.24.2's common/scanner.h includes it and
calls nothing from it, so it declares types only (wint_t, and wchar_t
via stddef.h) - no function a WasmStore could refuse to import.

Verified:
  * ruby, xml, svelte rebuilt through the wrappers: all three
    byte-identical (ruby c54cc209...e3d8, the checked-in sha).
  * php 0.24.2 (php/src, tree_sitter_php) builds unpinned:
    71c92662...ce51, 15 imports, every function among them
    (calloc/free/malloc/realloc, memcpy/memcmp, iswspace/iswxdigit/
    iswdigit/iswalnum) already in the svelte/ruby import sets.
  * grammar_provenance 7/7, ruby_kind_table 6/6.
Mutations (run, restored, md5-verified):
  * remove shim/wchar.h -> php build RED: 'wchar.h' file not found.
  * xml wrapper --src-subdir xml/src -> dtd/src -> RED, digest mismatch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

W8: workers set oom_score_adj=1000 so the OOM killer picks a worker, not the daemon ... cf0b3880be

Written in pre_exec (async-signal-safe open/write/close on static bytes),
inherited across exec, best effort where /proc is absent.
crates/plugin-host/tests/oom_score.rs reads it back off a real worker
beside the parent's own value. Mutation (removing the call) RUN: RED.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

W1/W15: measure worker costs with the REAL shipped grammars; say which grammar each quoted figure is ... 196955677e

crates/plugin-host/tests/real_grammar_costs.rs (#[ignore], linux) packs
tests/packages/{ruby,svelte,timeline,xaml} and the JSON fixture and reports
extractor precompile, grammar JIT wall/CPU, cold start (+worker CPU), warm
trip, RSS/Pss and marginal Pss. Ruby: ~175 ms cold (~135 ms of it the
grammar JIT), ~35 MiB Pss, ~28 MiB marginal - 14x the JSON fixture's 2 MiB.
Recorded in _prdoc/records/P0-real-grammar-costs.md.

packages.rs docs (75 us / 18.4 ms / 2-3 MiB) now state they are the JSON
fixture grammar's and give the Ruby figures, including that the pool's
IDLE_WORKER_MARGINAL_PSS_CEILING_KIB understates a Ruby worker ~3.4x.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

W14: discovery compiles every approved package's extractor in parallel ... cf8953a200

load_one is split into prepare_one (read, validate, own claims, stage),
precompile, and admit_one (package-vs-package claim check against the
packages REALLY accepted so far, then the staging and compile verdicts).
discover_with_store prepares in approval order, runs every precompile at
once (rayon), and admits in order - so every verdict and its refusal
order is the sequential one. Cost difference: a package later refused for
a claim conflict has been staged and compiled (it is operator-approved).

crates/indexer/tests/parallel_precompile.rs discovers a 4-package fleet
through a wrapper host binary that records whether it saw another compile
running. Mutation (.par_iter -> .iter) RUN: RED, 0 of 4 overlapped.
package_fixture: approved_fleet() split out of live_host_source;
shipped_entries() generalises xaml_entries().

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

lang_literal_gate: census every inline builtin language id (ratchet) ... cf767b9809

A packaged language's wire id (`de.h-dv.ruby/ruby`) is disjoint from
every builtin id by construction, so each `lang == "ruby"`, each
`lang = 'python'` in a resolver statement, silently excludes the package
that adopts that profile. This gate scans every non-test line of every
`crates/*/src/**.rs` (comments and `#[cfg(test)]` items lexed out) for a
builtin id spelled as a literal: a Rust string, a SQL string inside one,
or escaped JSON.

It starts as a RATCHET. The census, taken with this lexer and
cross-checked by an independent script: 136 literals in 235 files;
after the permanent exemptions (lang_profile.rs, kinds.rs, the
extractors, the builtin catalogue, frozen migrations, one extension list
and the default config template) 57 literals at 41 sites, every one
listed as a TODO. A new site is RED; a listed site that disappears is
RED too, so the list can only shrink.

Mutations run, all RED: drop a TODO row; stop blanking line comments;
drop `php` from the scanned ids; empty `cfg_test_lines`; ignore `'`
quotes; add a new `lang == "ruby"` to refactor.rs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

package_parity: the #84 parity gate, driven by a LangSpec, with two new axes ... 2023fd6f84

axis-B W2. `ruby_package_parity.rs`'s mechanism moves to
`crates/indexer/tests/package_parity/`, parameterized by `LangSpec`
(builtin lang, package id/lang/dir, shadow extension, qualified-name
separator, grant) and per-repo `RepoParity` (floors, pinned resolved
count, staged/qname floors). `RUBY` carries the numbers the Ruby file
pinned, unchanged: 1260 / 20446 / 231 / 2700 floors, 2966 on both legs.
`ruby_package_parity.rs` keeps its history and its test name and now
calls `package_parity::run(&RUBY)`; `ruby_parity_fixture` is a facade
over the same recipe so its three other callers did not change.

Assertions are the old ones in the old order. Added:
  * imports ROW FOR ROW (path, module, alias, bound_name, lang,
    start_line) - the old gate had only a count(*) floor per leg;
  * `corpus::project_symbol_detail`: signature and doc hashes,
    attr_start_line, is_extension - columns SYM_SQL omits (it must, for
    cross-binary comparisons). Signature is required populated on the
    builtin leg (1170/1260); doc and attr_start_line are COUNTED and
    printed, and for Ruby both are 0 - so those two columns are graded
    only as "both legs leave it NULL", and the control line says so.

`[[displaces]]` was NOT adopted: using it would mean packing a rewritten
plugin.toml (not the shipped digest) and would re-admit Gemfile (moving
FLOOR_REFS and the pinned resolved count). Reasons in the module doc.

Ruby parity: GREEN, executed=1, controls=5, 3.4 s.
Mutations (run; guest rebuilt with build.sh, whose unmodified output was
first confirmed byte-identical; restored by cp, md5-verified):
  * guest import encoder alias Some("zz") on each file's first import:
    RED at the import rows, "231 builtin rows, 231 package rows" - the
    old count floor and the resolution equality both held.
  * signature_until_body one byte short: RED at symbol detail.
  * compare returns 0: RED, "examined 0 row pairs against 20446".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

W3: pool lanes retire resting workers after IDLE_WORKER_RETIREMENT_MS (60 s) ... 2266fb1c67

The lane loop waits with recv_timeout (poll = threshold/2, clamped to
10 ms..5 s) and calls Supervisor::retire_idle on every timeout and after
every answer; retirement kills and reaps and charges no restart history.
PackageHost::live_workers() publishes the pool's worker count (per lane,
last-published); set_idle_retirement() lets a test shorten the threshold.

Threshold from the W1 numbers: the next edit pays one cold start, ~175 ms
for Ruby at worker level (416-494 ms through the 4-lane host at load avg
~16 on 12 cores), 45-65 ms for the small grammars; a resting Ruby worker
holds ~35 MiB Pss.

crates/indexer/tests/pool_idle_retirement.rs: shipped Ruby package behind
a 4-lane host; a burst forks 4 workers, a 300 ms threshold drains them to
0 (host count AND kernel child list), the next edit respawns and answers.
Mutations RUN: both calls removed RED; timeout-arm call removed RED;
after-answer call removed SURVIVES (recorded: needs a lane that never goes
quiet, not arrangeable from outside).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

W1: waive real_grammar_costs in ignored_test_reachability (measurement, no bound) ... 97835b2666

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

Route every inline builtin language test through the profile ... 105dc518b9

Every site the lang_literal_gate census found (57 literals at 41 sites)
now asks `code_index_core::lang_profile` instead of spelling a builtin
id, and the gate is a hard rule with no allowlist.

lang_profile gains `ProfileTraits`, one closed value per question,
compiled in per builtin (`PROFILE_TRAITS`) and read through `traits_of`,
which goes through `profile_name` — so `de.h-dv.ruby/ruby` answers as
`ruby`. `ProfileTraits::NONE` is the fail-closed default and is, field
by field, what each replaced site did for an id it did not spell.

Fields: own_import_proof (RustModuleFiles|PythonDotted|Unproven),
import_scope (ModuleBlock|ClassOpaque|Lexical), local_decision
(ExtractorMarks|ExtractorDecides|RowBased), header_parameters,
function_scoped_locals, import_row_names_entity,
type_position_may_be_local, self_is_a_parameter, conftest_fixtures,
module_static_is_value, own_file_alias_fallback, package_name_unique,
receiver_field_veto, partial_types, reclassify_nontype_refs,
names_case_insensitive, member_attribution
(ImplBlocks|SameKindRedeclarations|None).

SQL no longer spells a language: the resolver materialises
`temp.lang_profile` once per pass (every files.lang plus the builtin
names, one column per `SQL_TRAIT_COLUMNS` entry computed in Rust through
the same gate) and statements ask `sql_lang_has(col, trait)`, which
interpolates a checked COLUMN name only. `filter_glob` is added for the
user-facing filters (next commit).

Sites: index.rs (own-import proof and scope, module-static values,
package-name uniqueness, receiver field veto, alias fallback, C#
partial and type-ref reclassification), index/local_scope.rs (all four
lang consts, the ruby exemption, python type-position/self/conftest),
python_imports.rs (all seven), daemon refactor.rs (php case), and the
member-attribution lines of mcp-server's type disclosure.

Unit grades in lang_profile::tests, mutations run, all RED: delete the
ruby traits row; drop php's names_case_insensitive; give javascript
ClassOpaque; bypass profile_name in traits_of; drop sql_lang_has's
column check; widen filter_glob past profile names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

corpus: COSI_CORPUS_LANGS runs and blesses the ratchets per language ... acb812f452

axis-C W1. `COSI_CORPUS_LANGS=ruby,php` (corpus.toml `lang` spelling)
narrows corpus_ratchet, corpus_cost, corpus_stage, corpus_tier3_ratchet
and package_parity to those languages' repos, through ONE door:
`corpus::Coverage::select`. `corpus::tier` stays unfiltered.

Why it cannot weaken a full run:
  * unset/empty/whitespace = no filter; `select` is then the identity;
  * a value naming nothing (unknown lang, only commas) PANICS at
    `Coverage::new`, naming it and the known languages (#259's shape);
  * deselected repos are recorded in the coverage manifest
    (`lang_filter`, `deselected`), never counted executed or
    unavailable, and every filtered run prints `FILTERED RUN ... This is
    NOT a full corpus run.`;
  * a selected repo that is unavailable still fails under REQUIRE; only
    "the filter selected none of this suite's repos" is excused, and
    that state is unreachable without a filter;
  * corpus_lang_filter.rs fails if ci.yml's `corpus` or `corpus-scale`
    job ever sets the variable.
A filtered bless is PARTIAL by construction: every ratchet writer
already merges (per-repo replace), and the filter note now goes into the
committed control roster, naming the rows kept unmeasured.

Baselines are NOT split. baseline.json must not move (CLAUDE.md), and
per-repo rows already make a per-language bless touch only its rows;
the one shared line two concurrent per-language blesses would conflict
on is the `_blessed` block. Splitting that is a decision about bless
history, not a cheap mechanical change.

ci.yml: a comment block documents the per-language job shape (its own
workflow with `paths:` or an `if:` over changed files - both unmeasured
on this Forgejo), and the three rules. No job, step or env line changed.

Tests: corpus_lang_filter 6/6, corpus_require_floor 11/11, ci_cadence
9/9. End to end: COSI_CORPUS_LANGS=ruby corpus_ratchet GREEN executed=1
with the six others named as deselected; =rubyy RED naming it;
COSI_CORPUS_LANGS=ruby corpus_tier3_ratchet GREEN, "selected none".
Mutations (run, restored, md5-verified):
  * drop the unknown-language check -> RED a_filter_that_names_nothing
  * select_by_lang drops the deselected half -> RED ...loses_nothing
  * empty value parses as Some(empty) -> RED no_value_is_no_filter
  * COSI_CORPUS_LANGS: "ruby" in the corpus job env -> RED, naming it

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

User-facing lang filters accept a profile name ... f4b8d01317

`lang: "ruby"` now matches rows stamped `de.h-dv.ruby/ruby` as well as
`ruby`, at every user-facing language filter: search_symbols (prefix
path, token tier and the imports half), search_text, and
resolution_gaps. A wire id (`de.h-dv.ruby/ruby`) still matches only
itself, and a bare name that is no profile (`md`, `myruby`) matches
exactly, as before. The value is always bound: `sql_lang_filter`
renders `(col = ?a OR col GLOB ?b)` with `?b = '*/ruby'` from
`filter_glob`, and interpolates only the caller's column constant.

Tests, mutations run, all RED: restore `s.lang = ?` on the prefix path;
restore it on the token tier; restore `r.lang = ?` in resolution_gaps;
misnumber the GLOB placeholder. The imports half and search_text share
the helper and are not separately graded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

W0: a packaged Ruby gets the builtin's local decisions (fixture) ... 22dfe97f8e

local_scope.rs kept the row-based local pass away from Ruby with two
spellings of the BUILTIN id (`f.lang <> 'ruby'` and
`EXTRACTOR_DECIDES_LOCALS`), so `de.h-dv.ruby/ruby` ran the
approximation the builtin is exempt from. The routing commit made both
read `local_decision == ExtractorDecides`; this pins it.

The fixture indexes one file on a builtin leg and, through the shipped
`tests/packages/ruby` package and the production index entry point, on
a package leg. A default value that assigns (`def run(x = (helper = 2))`)
is the one Ruby construct that writes a HEADER binding row, which makes
`helper` a "parameter" to the row-based pass and let R1 refuse
`helper(3)` its same-file `def helper`. Result: both legs bind
`helper(3)` to `def helper`, and the whole resolved projection agrees.

MUTATION (run): restore the pre-fix coupling (SQL exemption spelled
`(f.lang = 'ruby')` and `decides_locals` true for packaged ids) — RED,
the package leg leaves `helper(3)` unresolved. Each half alone
survives, by design: either routed half closes the coupling.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

W7: a transient package refusal keeps the file's previous facts ... d5be34bc32

`extract_for` turned every package refusal into `Parse::Err`, and the
writer then deleted the file's previous symbols and refs. A refusal that
is about the WORKER — killed from outside or unavailable
(`HostWorkerUnavailable`, which is where the supervisor's
`killed_from_outside` split already lands), `HostDeadlineExceeded`,
`HostRestartBackoff`, `HostPackageQuarantined` — is not a verdict on the
file, yet it wiped the file's facts until the next edit.

`refusal_is_transient` names those four; `parse_of_refusal` maps a
refusal to `Parse::Transient` or `Parse::Err`. On `Transient`, `run_task`
writes NOTHING when the file has a previous row: its facts and recorded
stat are left as they were, the stat mismatch is the stale mark
(freshness reports the file as changed), and the next pass re-extracts
it. With no previous row — including every generation build, which
passes an empty snapshot — the file is recorded as refused exactly as
before, so `build::extract_one`'s accounting is unchanged. Traps, memory
ceilings and every abi/fact validation refusal still delete.

Tests drive real `index_path` passes through a path-keyed test seam at
the top of `extract_for` that stands in for the worker's answer.
Mutations run, all RED: treat transient as Err (facts deleted); treat
every refusal as transient (stale facts survive a verdict); ignore
`Transient` in `run_task`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

Docs: cite the profile traits instead of the removed local_scope consts ... 2e512e5551

rustdoc -D warnings caught one intra-doc link to HEADER_PARAMETER_LANGS;
local_scope_review_fixes named FUNCTION_SCOPED_LOCALS in a RUN mutation,
which was re-run against the trait (drop typescript's
function_scoped_locals): RED, two tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

Tests that flush through the writer hold RESOLVE_HOOK_LOCK ... a23792ead9

The first full `code-index-indexer --lib` run after W7 went red on three
tests: `the_shutdown_flush_gives_up_after_its_cap` and the m0070/m0071
upgrade tests, all with "injected busy (test seam)". FAIL_NEXT_COMMITS
is process-global, and every test whose `index_path` flushes through
the writer must hold `index::RESOLVE_HOOK_LOCK` (its doc says so). The
new W7 tests did not — six `index_path` passes each, consuming the
shutdown test's armed failures — and the two migration tests never
did, a latent race the extra traffic exposed.

All three now hold the lock. Result: 438 passed, 0 failed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

Merge branch 'p0-runtime-measure' into p0-integration eae051f560

Merge branch 'p0-profile-literals' into p0-integration 13fbf4f884

Merge branch 'p0-gates-release-infra' into p0-integration 709d902f78

ruby_package_local_decisions: declare package_parity, which ruby_parity_fixture now imports ... fcb9690010

Lane N's new test predates lane O's LangSpec move of the fixture onto
crate::package_parity; each test crate that mounts the fixture must
mount its sibling too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

C# partial + reclassification tiers: key on lang and take the gate ... c0512a6f4b

#112 phase 0 replaced these tiers' `lang = 'csharp'` literal with the
`partial_types` / `reclassify_nontype_refs` profile traits over
`temp.lang_profile`. `traits_of` resolves by LOCAL id, so a package
language `<pkg>/csharp` now enters `cs_scope_fqn`, `pc_calls`,
`cs_type_names`, `cs_nontype_names` and `cs_seg_reclass` — and the
registry's NeverDynamic claim for all five ("no dynamic row can reach
it") stopped being true.

Two production defects followed, both fixed here:

* CROSS-LANGUAGE DECISIONS. None of the relations carried `lang`, so a
  packaged `Widget` and a builtin `Widget` were one partial-class scope
  (a builtin call bound into the package's member — a cross-language
  edge outside the bridge tier), and a packaged member reclassified a
  builtin type ref. Every relation now keys on `lang`, and each
  language's tier judges only its own rows.
* UNGATED ADMISSION. An ungranted package's rows could enter all five.
  `cs_scope_fqn` and `pc_calls` now take `member_candidate` (the ref
  side exactly as `recv_calls`/`ext_calls` do); the reclassification
  name sets take `type_position_candidate` on every symbol read
  (member, namespace-segment and type-FQN exclusion fills).

Registry: the four relations are Gated (24 -> 28, count pin updated
with the reason), `cs_seg_reclass` is Bookkeeping naming
`tqt_decisions`, and `temp.lang_profile` is NeverDynamic on
`let t = traits_of(lang);` — a per-pass lookup table with no symbol,
candidate or verdict, the `file_ancestor` shape. Its INSERT is spelled
`VALUES (?1, …)` so the scanner grades it against the loop that
computes each row.

Tests, every mutation RUN:
* index.rs `packaged_profile_isolation_tests` (calls the resolver
  directly — `index_path` re-extracts any file whose `files.lang` is
  stamped with a package id): drop `pf.lang = c.lang` -> RED; drop the
  reclassification SET's `n.lang` key -> RED. Both with a same-language
  control arm that must decide.
* capability_isolation: empty the `pc_calls` ref gate -> RED
  (region-split ungranted call); empty the member / namespace /
  type-name reclassification gates -> RED each, each with a granted
  control.
* LAYERED, recorded as behavioural survivors: emptying the
  `cs_scope_fqn` gate or the type-FQN exclusion gate survives every
  behavioural suite (downstream gates hold); the registry's admission
  count kills both (`cs_scope_fqn: … says 1 … carries 0`,
  `cs_nontype_names: … says 5 … carries 4`). Rewriting the
  `lang_profile` predicate -> a_never_dynamic… RED.

Corpus: corpus_ratchet 3/3, corpus_mutation 6/6, corpus_stage 11/11
(release, COSI_CORPUS_REQUIRE=1); baseline files untouched.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

parallel_precompile: write the host wrapper under the fork lock ... 6e42426432

`discovery_runs_its_precompiles_concurrently_and_admits_every_package`
wrote its `#!/bin/sh` host wrapper with a bare `fs::write` + chmod and
then had discovery exec it from a rayon pool — the ETXTBSY window
`exec_copy_registry::every_written_executable_goes_through_the_fork_lock`
exists for. It now uses `code_index_test_support::exec::write_executable`.

Adopting it brings the file under the paired spawn gate, which then
named `package_fixture/mod.rs:95` — the fixture's `cargo build` of the
plugin host, an unlocked fork. It goes through `locked_status`, since
the lock closes nothing unless the fork takes it too.

MUTATION (RUN): restore the bare `fs::write` + `set_permissions(0o755)`.
RESULT: RED — `crates/indexer/tests/parallel_precompile.rs:37 fn
discovery_runs_its_precompiles_concurrently_and_admits_every_package(…)`.
The fixture's bare `.status()` was seen RED by
`a_file_that_installs_an_executable_locks_its_spawns_too` before the
second half of this change. exec_copy_registry 9/9.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

plugin_activation_cli: the --no-daemon bench serves exactly one root ... c49a523159

`a_no_daemon_mcp_startup_preserves_the_active_generations_package_record`
failed in the full gate run with "the startup reconcile stat-skipped
everything", and the failure's own stderr says why: the
`single-process reconcile complete` line it stopped on read
`files_seen: 206, parse_count: 0`, beside
`index_path{root=/home/master/.cache/cosi-corpus/js-express}`.

The gate run sets COSI_CORPUS_DIR for its corpus suites. The spawned
`code-index-mcp` inherited it and (#191) linked every pinned corpus
checkout, and `--no-daemon` reconciles each linked root, each emitting
its own `complete` line that names no root. The helper stopped at
whichever finished first. NOT Phase 0 and not W7: neither file changed
since v0.32.2, where both the test and `corpus_links` already existed.
Reproduced with COSI_CORPUS_DIR set: 2 of 3 runs RED, including the
sibling `a_rollback_across_an_mcp_startup_…` ("did not run").

The helper now removes COSI_CORPUS_DIR and COSI_WORKTREES_DIR (#238,
the same shape) from the server, and refuses any `index_path` line
naming a root other than the bench, so a future leak fails by name
instead of grading another project's stats. Nothing the tests assert
is weakened. With COSI_CORPUS_DIR set: 5/5 green.

MUTATION (RUN): drop both `env_remove`s, COSI_CORPUS_DIR set, 10 runs.
RESULT: RED in 7 of 10 ("reconciled a root other than this bench"); the
3 survivors are runs whose own reconcile reported first — the race the
leak is, which is why the precondition is removed rather than detected.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

buildagent added 1 commit 2026-09-26 20:07:09 +02:00

packaged_profile_isolation tests hold RESOLVE_HOOK_LOCK ... 0bcdac136f

Both index and resolve through flush, so a parallel test's injected busy
commit (FAIL_NEXT_COMMITS, process-global) could land in them: native
Windows CI failed a_packaged_csharp_member_does_not_reclassify_a_builtin_type_ref
with "injected busy (test seam)" at csharp_project's index_path. Same rule
RESOLVE_HOOK_LOCK's doc states, and the same repair a23792e made for the
W7 and m0070/m0071 tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126PDDLB4wNHxKXvWM1VNmu

buildagent commented 2026-09-26 21:20:08 +02:00

Author

Member

Copy link

Merged: master was fast-forwarded to 0bcdac1 after CI on that exact commit passed on Linux (run 5634) and native Windows (run 5633). The Windows failure from the first run is fixed in 0bcdac1: the two new isolation tests now hold RESOLVE_HOOK_LOCK. I'm closing this PR by hand because Forgejo does not detect a fast-forward merge.

Merged: master was fast-forwarded to 0bcdac1 after CI on that exact commit passed on Linux (run 5634) and native Windows (run 5633). The Windows failure from the first run is fixed in 0bcdac1: the two new isolation tests now hold `RESOLVE_HOOK_LOCK`. I'm closing this PR by hand because Forgejo does not detect a fast-forward merge.

buildagent closed this pull request 2026-09-26 21:20:08 +02:00

buildagent referenced this pull request 2026-09-27 15:03:26 +02:00

epic: every builtin language becomes a first-party plugin package #309

buildagent referenced this pull request 2026-09-27 15:03:30 +02:00

epic: runtime plugin architecture — dynamic grammar, extraction and resolution packages #75

All checks were successful

Hide checks

CI / cargo fmt (pull_request) Successful in 50s

Details

CI / OSS corpus tier-3 scale (nightly) (pull_request) Has been skipped

Details

CI / Grammar rebuild from source (nightly) (pull_request) Has been skipped

Details

CI / guest crates (fmt, clippy, doc) (pull_request) Successful in 1m22s

Details

CI / CI lane wall-clock headroom (pull_request) Successful in 1m25s

Details

CI / cargo doc (intra-doc links) (pull_request) Successful in 4m28s

Details

CI / cargo check (MSRV 1.98) (pull_request) Successful in 6m1s

Details

CI / cargo deny (pull_request) Successful in 6m14s

Details

CI / cargo clippy (pull_request) Successful in 6m33s

Details

CI / cargo test (abi, 32-bit + wasm32) (pull_request) Successful in 6m37s

Details

CI / cargo check (windows-gnu) (pull_request) Successful in 6m40s

Details

CI / OSS corpus (tier 1) (pull_request) Successful in 27m54s

Details

CI / cargo test (pull_request) Successful in 32m37s

Details

CI / cargo test (daemon transport) (pull_request) Successful in 10m10s

Details

CI / Plugin path cost + pool throughput (nightly) (pull_request) Has been skipped

Details

CI (Windows) / fmt + clippy + build + test (windows) (pull_request) Successful in 1h12m33s

Details

CI / cargo fmt (push) Successful in 1m3s

Details

CI / OSS corpus tier-3 scale (nightly) (push) Has been skipped

Details

CI / Grammar rebuild from source (nightly) (push) Has been skipped

Details

CI / guest crates (fmt, clippy, doc) (push) Successful in 58s

Details

CI / CI lane wall-clock headroom (push) Successful in 1m36s

Details

CI / cargo doc (intra-doc links) (push) Successful in 4m47s

Details

CI / cargo check (MSRV 1.98) (push) Successful in 5m37s

Details

CI / cargo test (abi, 32-bit + wasm32) (push) Successful in 6m14s

Details

CI / cargo deny (push) Successful in 6m20s

Details

CI / cargo clippy (push) Successful in 6m28s

Details

CI / cargo check (windows-gnu) (push) Successful in 6m41s

Details

CI / OSS corpus (tier 1) (push) Successful in 32m20s

Details

CI / cargo test (push) Successful in 52m14s

Details

CI (Windows) / fmt + clippy + build + test (windows) (push) Successful in 1h20m56s

Details

CI / cargo test (daemon transport) (push) Successful in 23m56s

Details

CI / Plugin path cost + pool throughput (nightly) (push) Has been skipped

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

code-review

  

correctness

  

dos

  

performance

  

security

  

severity/high

  

severity/low

  

severity/medium

  

tech-debt

  

Kind/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No labels

code-review

correctness

dos

performance

security

severity/high

severity/low

severity/medium

tech-debt

Kind/Breaking

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

h-dv/code-index!306

No description provided.