feat: shrink corpus and plugin-generation findings into reproducible fixtures #48

New issue

Open

opened 2026-07-28 20:39:43 +02:00 by buildagent · 0 comments

buildagent commented

2026-07-28 20:39:43 +02:00

Member

P3 — build when the first real corpus finding lands; premature before then.

From _prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md §5.2.

The problem it solves

A phantom in a 5000-file repo is nearly impossible to reason about, and without this the finding stays only in the corpus — a slow test forever, re-run on every CI pass, debugged from scratch every time.

What

Given (repo, ref site, wrong target): bisect the file set — delete half, re-index, does it still reproduce — down to the minimal reproducing set. Emit a new micro-fixture under tests/fixtures/ plus a generated oracle.toml probe in the existing schema (name/kind/path/container/find/expect_resolved/forbid_resolved).

The division of labour it establishes

The corpus is the discovery engine. The fixtures are the regression suite.
Nothing found in the corpus is allowed to stay only in the corpus.

This is what keeps PR CI fast permanently: the lasting cost of each finding becomes a handful of files, not a repository. It also preserves the existing precision_gate.rs as the fast gate rather than growing a parallel slow one.

Design notes

bisection must preserve indexability — deleting a file that the repro depends on ends that branch; standard delta-debugging applies
the generated probe must satisfy the existing I033 anti-vacuity guard: at least one forbid_resolved probe whose (name, kind) resolves to exactly one in-project definition, or the generated oracle is vacuous by construction
emit the note field explaining provenance (source repo, sha, original site) so the fixture is traceable back to the real code

Acceptance

shrinker reduces a seeded known finding to a minimal file set
emits a fixture + valid oracle.toml probe that the existing precision_gate.rs picks up unmodified
generated oracle passes the I033 single-candidate anti-vacuity requirement
provenance recorded in the fixture
round-trip proven on a deliberately injected defect

Runtime-plugin architecture expansion

A reproducer is now more than source files. The shrinker must preserve the minimal semantic environment:

exact plugin package digest(s), supplied from a local test registry;
capability and bridge grants;
active/pending generation transition when relevant;
wrapper/embedded source maps;
the smallest file set and claim-domain configuration;
host ABI/engine version required to reproduce.

For a package finding, emit either a normal oracle fixture plus package metadata or a conformance fixture consumed by #80. Never copy an untrusted remote package implicitly; record digest and approved test artifact provenance.

Delta-debug both axes: files and plugin capabilities/rules. Positive controls must prove the reduced fixture still executes the relevant component and moves the expected candidate population.

**P3 — build when the first real corpus finding lands; premature before then.** From `_prdoc/records/brainstorm-2026-07-28-oss-corpus-test-system.md` §5.2. ## The problem it solves A phantom in a 5000-file repo is nearly impossible to reason about, and without this the finding stays *only* in the corpus — a slow test forever, re-run on every CI pass, debugged from scratch every time. ## What Given `(repo, ref site, wrong target)`: bisect the file set — delete half, re-index, does it still reproduce — down to the minimal reproducing set. Emit a new micro-fixture under `tests/fixtures/` plus a generated `oracle.toml` probe in the existing schema (`name`/`kind`/`path`/`container`/`find`/`expect_resolved`/`forbid_resolved`). ## The division of labour it establishes > **The corpus is the discovery engine. The fixtures are the regression suite.** > Nothing found in the corpus is allowed to stay only in the corpus. This is what keeps PR CI fast permanently: the lasting cost of each finding becomes a handful of files, not a repository. It also preserves the existing `precision_gate.rs` as the fast gate rather than growing a parallel slow one. ## Design notes - bisection must preserve indexability — deleting a file that the repro depends on ends that branch; standard delta-debugging applies - the generated probe must satisfy the **existing I033 anti-vacuity guard**: at least one `forbid_resolved` probe whose `(name, kind)` resolves to exactly one in-project definition, or the generated oracle is vacuous by construction - emit the `note` field explaining provenance (source repo, sha, original site) so the fixture is traceable back to the real code ## Acceptance - [ ] shrinker reduces a seeded known finding to a minimal file set - [ ] emits a fixture + valid `oracle.toml` probe that the existing `precision_gate.rs` picks up unmodified - [ ] generated oracle passes the I033 single-candidate anti-vacuity requirement - [ ] provenance recorded in the fixture - [ ] round-trip proven on a deliberately injected defect ## Runtime-plugin architecture expansion A reproducer is now more than source files. The shrinker must preserve the minimal semantic environment: - exact plugin package digest(s), supplied from a local test registry; - capability and bridge grants; - active/pending generation transition when relevant; - wrapper/embedded source maps; - the smallest file set and claim-domain configuration; - host ABI/engine version required to reproduce. For a package finding, emit either a normal oracle fixture plus package metadata or a conformance fixture consumed by #80. Never copy an untrusted remote package implicitly; record digest and approved test artifact provenance. Delta-debug both axes: files and plugin capabilities/rules. Positive controls must prove the reduced fixture still executes the relevant component and moves the expected candidate population.

buildagent added the

Kind/Feature

Priority

Low

labels

2026-07-28 20:40:54 +02:00

buildagent changed title from ~~feat: shrinker — bisect a corpus finding into a minimal fixture + generated oracle.toml probe~~ to feat: shrink corpus and plugin-generation findings into reproducible fixtures

2026-08-26 13:39:48 +02:00

buildagent added a new dependency

2026-08-26 13:43:40 +02:00

#76 plugin ABI: immutable package format and validated extraction-fact protocol

buildagent added a new dependency

2026-08-26 13:43:41 +02:00

#77 resolver: row-level provenance, capability-gated pools and explicit language bridges

buildagent added a new dependency

2026-08-26 13:43:43 +02:00

#78 index: generation-safe plugin activation, invalidation, promotion and rollback

buildagent added a new dependency

2026-08-26 13:43:46 +02:00