A workspace cargo test with COSI_CORPUS_DIR set indexes all nine corpus checkouts in place, which turns every corpus gate red #314

Closed
opened 2026-10-05 08:50:02 +02:00 by buildagent · 3 comments
Member

Observed (2026-10-05, v0.33.0 release candidate 8879fd84). The local full gate run exported COSI_CORPUS_DIR=$HOME/.cache/cosi-corpus for every step, including cargo test --workspace. CI never does that: only the corpus jobs set the variable.

At 06:37:19–06:37:23Z, a .code-index/ directory appeared in all nine pinned checkouts (cs-dapper … ts-zod), during the workspace run. As a result:

  • the corpus guard (crates/indexer/tests/corpus/mod.rs:263) refused every corpus suite;
  • query_cli::the_cli_log_level_reaches_the_spawned_server answered warming_up, because indexing Django and rust-analyzer in the background starved it. It passes 20/20, three times, on a quiet machine.

Cause. code-index-mcp honours COSI_CORPUS_DIR (#191 corpus links). It links each pinned repo as a project and indexes it IN PLACE, so its state lands in the checkout. tests/support/mcp.rs scrubs the variable, but at least one other spawner of the server does not. Candidates that spawn code-index-mcp without env_remove("COSI_CORPUS_DIR"):

  • crates/daemon/tests/common/mod.rs
  • mcp_smoke.rs
  • warmup_e2e.rs
  • workspace_e2e.rs
  • harness_store_isolation.rs
  • cli/tests/blast_radius_hook.rs

These were not yet bisected.

Why it matters. CLAUDE.md tells developers to run the corpus gates with COSI_CORPUS_DIR set. Anyone who also has it exported in their shell, or runs the workspace and corpus gates together, gets a corpus that every gate refuses. Without the corpus guard, they would get silently skewed numbers instead.

Fix shape (structural, not per test). Every test spawn of the server should go through one helper that scrubs COSI_CORPUS_DIR, with a registry test that fails when a test spawns CARGO_BIN_EXE_code-index-mcp some other way. Optionally, corpus links could keep their index state outside the checkout.

Found during the v0.33.0 full gate run. It does not block the release, because CI's test job does not set the variable.

**Observed (2026-10-05, v0.33.0 release candidate 8879fd84).** The local full gate run exported `COSI_CORPUS_DIR=$HOME/.cache/cosi-corpus` for every step, including `cargo test --workspace`. CI never does that: only the corpus jobs set the variable. At 06:37:19–06:37:23Z, a `.code-index/` directory appeared in all nine pinned checkouts (cs-dapper … ts-zod), during the workspace run. As a result: - the corpus guard (`crates/indexer/tests/corpus/mod.rs:263`) refused every corpus suite; - `query_cli::the_cli_log_level_reaches_the_spawned_server` answered `warming_up`, because indexing Django and rust-analyzer in the background starved it. It passes 20/20, three times, on a quiet machine. **Cause.** `code-index-mcp` honours `COSI_CORPUS_DIR` (#191 corpus links). It links each pinned repo as a project and indexes it IN PLACE, so its state lands in the checkout. `tests/support/mcp.rs` scrubs the variable, but at least one other spawner of the server does not. Candidates that spawn `code-index-mcp` without `env_remove("COSI_CORPUS_DIR")`: - `crates/daemon/tests/common/mod.rs` - `mcp_smoke.rs` - `warmup_e2e.rs` - `workspace_e2e.rs` - `harness_store_isolation.rs` - `cli/tests/blast_radius_hook.rs` These were not yet bisected. **Why it matters.** CLAUDE.md tells developers to run the corpus gates with `COSI_CORPUS_DIR` set. Anyone who also has it exported in their shell, or runs the workspace and corpus gates together, gets a corpus that every gate refuses. Without the corpus guard, they would get silently skewed numbers instead. **Fix shape (structural, not per test).** Every test spawn of the server should go through one helper that scrubs `COSI_CORPUS_DIR`, with a registry test that fails when a test spawns `CARGO_BIN_EXE_code-index-mcp` some other way. Optionally, corpus links could keep their index state outside the checkout. Found during the v0.33.0 full gate run. It does not block the release, because CI's test job does not set the variable.
Author
Member

Now breaks CI, not just local runs. Nightly run 5754 (master 407c4925), job "Plugin path cost + pool throughput (nightly)": all 4 language_package_overhead tests stopped at the corpus guard (crates/indexer/tests/corpus/mod.rs:263):

corpus checkout /tmp/cosi-corpus/rust-ripgrep carries a .code-index/ directory (languages.toml, languages-enabled, index.db, daemon.log, index.db-shm, index.db-wal, daemon.lock, daemon.toml)

Source: the step right before it, python3 tests/phase2_acceptance.py --bin-dir target/phase2-packages-only/release … (ci.yml ~1599).

  • It never names the corpus, but it inherits the job-wide COSI_CORPUS_DIR: /tmp/cosi-corpus (ci.yml ~1548).
  • The code-index-mcp it spawns links every corpus repo (#191) and indexes them in place.

Fix in progress with #318 phase 2:

  • The acceptance script strips COSI_CORPUS_DIR from its children, and the step runs with it unset.
  • Every test spawn of code-index-mcp goes through one scrubbing helper.
  • A registry test forbids spawning it any other way.
**Now breaks CI, not just local runs.** Nightly run 5754 (master 407c4925), job "Plugin path cost + pool throughput (nightly)": all 4 `language_package_overhead` tests stopped at the corpus guard (`crates/indexer/tests/corpus/mod.rs:263`): > corpus checkout /tmp/cosi-corpus/rust-ripgrep carries a `.code-index/` directory (languages.toml, languages-enabled, index.db, daemon.log, index.db-shm, index.db-wal, daemon.lock, daemon.toml) **Source:** the step right before it, `python3 tests/phase2_acceptance.py --bin-dir target/phase2-packages-only/release …` (ci.yml ~1599). - It never names the corpus, but it inherits the job-wide `COSI_CORPUS_DIR: /tmp/cosi-corpus` (ci.yml ~1548). - The `code-index-mcp` it spawns links every corpus repo (#191) and indexes them in place. **Fix in progress with #318 phase 2:** - The acceptance script strips `COSI_CORPUS_DIR` from its children, and the step runs with it unset. - Every test spawn of `code-index-mcp` goes through one scrubbing helper. - A registry test forbids spawning it any other way.
Author
Member

A local reproduction of CI's corpus job (2026-10-07, on the zammad-fixes integration branch) left .code-index/ directories in three corpus checkouts: cs-dapper, python-flask and rust-ripgrep.

  • Likely source: mcp agent_task_bench. ci.yml's corpus job sets COSI_CORPUS_DIR at job level, and this suite spawns the MCP server with it. The integration lane attributed the directories to it.
  • What phase 2 covered: the #314 fix scrubbed the variable from test-spawned servers, via code_index_test_support::product::command and the mcp_spawn_registry guard. Either agent_task_bench does not go through that spawn path, or it indexes the corpus on purpose.
  • Why it matters: CI probably writes the same directories into /tmp/cosi-corpus. Those directories are what the corpus contamination guard refuses on later runs that share the cache.
  • To do: confirm which suite creates them, then either route it through the scrubbing spawn helper or point it at a copy of the corpus.
A local reproduction of CI's corpus job (2026-10-07, on the zammad-fixes integration branch) left `.code-index/` directories in three corpus checkouts: cs-dapper, python-flask and rust-ripgrep. - **Likely source:** `mcp agent_task_bench`. ci.yml's corpus job sets `COSI_CORPUS_DIR` at job level, and this suite spawns the MCP server with it. The integration lane attributed the directories to it. - **What phase 2 covered:** the #314 fix scrubbed the variable from test-spawned servers, via `code_index_test_support::product::command` and the `mcp_spawn_registry` guard. Either `agent_task_bench` does not go through that spawn path, or it indexes the corpus on purpose. - **Why it matters:** CI probably writes the same directories into `/tmp/cosi-corpus`. Those directories are what the corpus contamination guard refuses on later runs that share the cache. - **To do:** confirm which suite creates them, then either route it through the scrubbing spawn helper or point it at a copy of the corpus.
Author
Member

Verified the structural fix and the remaining benchmark contamination path are already shipped in v0.41.34; the relevant source is byte-identical at current candidate d641f8ec.

  • Test product spawns use code_index_test_support::product::command, which removes the exact corpus variable the server reads. The registry currently scans 593 sources and identifies 23 sources reaching the MCP server; all three permanent registry checks pass.
  • Recompiled those unchanged standard-library-only registry/lexer modules directly, avoiding a workspace rebuild: seven checks pass, including a real child process launched with COSI_CORPUS_DIR inherited by the parent and absent from the child.
  • Negative control: removing only env_remove in a private copy makes both the permanent helper check and the actual-child check fail. No repository source was changed.
  • The follow-up agent_task_bench path now creates a disposable shared Git clone, checks out the pinned oracle revision there, and indexes a database under its temporary directory. Published qualification job 60198 ran all three real repositories (cs-dapper, python-flask and rust-ripgrep); all five benchmark tests passed. The full Linux qualification passed 5,355 tests.
  • The Python signed-package acceptance runner also scrubs the corpus settings from every child; its full candidate E2E suite passed again during PR #490 qualification.

Closing the original inherited-environment and benchmark-in-place defects with verified shipped behavior. No extra CI run or production change was needed. Local receipt: /tmp/cosi-314-verification.json.

Verified the structural fix and the remaining benchmark contamination path are already shipped in v0.41.34; the relevant source is byte-identical at current candidate d641f8ec. - Test product spawns use `code_index_test_support::product::command`, which removes the exact corpus variable the server reads. The registry currently scans 593 sources and identifies 23 sources reaching the MCP server; all three permanent registry checks pass. - Recompiled those unchanged standard-library-only registry/lexer modules directly, avoiding a workspace rebuild: seven checks pass, including a real child process launched with COSI_CORPUS_DIR inherited by the parent and absent from the child. - Negative control: removing only `env_remove` in a private copy makes both the permanent helper check and the actual-child check fail. No repository source was changed. - The follow-up `agent_task_bench` path now creates a disposable shared Git clone, checks out the pinned oracle revision there, and indexes a database under its temporary directory. Published qualification job 60198 ran all three real repositories (cs-dapper, python-flask and rust-ripgrep); all five benchmark tests passed. The full Linux qualification passed 5,355 tests. - The Python signed-package acceptance runner also scrubs the corpus settings from every child; its full candidate E2E suite passed again during PR #490 qualification. Closing the original inherited-environment and benchmark-in-place defects with verified shipped behavior. No extra CI run or production change was needed. Local receipt: `/tmp/cosi-314-verification.json`.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#314
No description provided.