h-dv/code-index

Watch

Fork 0

review_diff grades an edited top-level Rust `#[test]` function as an untested production change #577

New issue

Closed

opened 2026-11-12 13:40:49 +01:00 by buildagent · 2 comments

buildagent commented

2026-11-12 13:40:49 +01:00

Member

Found by a worker in the v0.41.51 train (dogfood batch 18) and reproduced by the lead on the installed release, 0.41.50 (3d0a584c), in a small git fixture.

Reproduction: lib.rs at the base revision holds a few ordinary functions and one unit test at file level, with no enclosing #[cfg(test)] module:

pub fn validate_base() {}
pub fn validate_values() {}
#[test]
fn standalone_test() { validate_base(); }

The only change adds validate_values(); to the test body. Then:

code-index query review_diff '{"path_glob":"lib.rs","envelope":"minimal","response_format":"concise"}'

Actual: one finding, severity: "medium", reason: "untested_change", name: "standalone_test"; summary.changed_test_files: []. Nothing is degraded: classification_degraded: false, resolution 100 %, no truncation.

Control: rustc --test lib.rs runs and passes exactly that test. A changed file under tests/ is recognised as a changed test file in the same fixture.

Expected: a function annotated #[test] has the test role wherever it stands. Editing only it raises no production untested_change, and it counts as changed-test evidence without making the other symbols of the file tests.

Impact: a test-only edit gets a medium production-risk finding and contributes no counter-evidence. The usual low-confidence disclosures do not apply here, so a reader has no reason to doubt the finding.

Scope note: not measured whether the same holds for other languages' file-level test markers (for example a pytest function in a non-test file, or a C# [Fact] method in a production class).

Found by a worker in the v0.41.51 train (dogfood batch 18) and reproduced by the lead on the installed release, 0.41.50 (3d0a584c), in a small git fixture. **Reproduction:** `lib.rs` at the base revision holds a few ordinary functions and one unit test at file level, with no enclosing `#[cfg(test)]` module: ```rust pub fn validate_base() {} pub fn validate_values() {} #[test] fn standalone_test() { validate_base(); } ``` The only change adds `validate_values();` to the test body. Then: ``` code-index query review_diff '{"path_glob":"lib.rs","envelope":"minimal","response_format":"concise"}' ``` **Actual:** one finding, `severity: "medium"`, `reason: "untested_change"`, `name: "standalone_test"`; `summary.changed_test_files: []`. Nothing is degraded: `classification_degraded: false`, resolution 100 %, no truncation. **Control:** `rustc --test lib.rs` runs and passes exactly that test. A changed file under `tests/` is recognised as a changed test file in the same fixture. **Expected:** a function annotated `#[test]` has the test role wherever it stands. Editing only it raises no production `untested_change`, and it counts as changed-test evidence without making the other symbols of the file tests. **Impact:** a test-only edit gets a medium production-risk finding and contributes no counter-evidence. The usual low-confidence disclosures do not apply here, so a reader has no reason to doubt the finding. **Scope note:** not measured whether the same holds for other languages' file-level test markers (for example a pytest function in a non-test file, or a C# `[Fact]` method in a production class).

buildagent referenced this issue

2026-11-12 13:41:32 +01:00

Tool discovery and replies cost extra calls, while review and navigation still have gaps #521

buildagent commented

2026-11-12 14:43:25 +01:00

Author

Member

Investigated in the v0.41.51 train; not fixable in the host alone, so it is not in that release.

Where the test role is decided: in the host, by one rule for all seven languages: the file's path (test directories and file-name patterns per language, crates/indexer/src/test_paths.rs), then the names of enclosing modules, namespaces or classes (test, tests, *_test, *_tests; crates/mcp-server/src/server.rs, symbol_is_test_role). A symbol's own attributes, decorators or name are never read. The container rule is a name heuristic and does not inspect #[cfg(test)], so a #[cfg(test)] mod checks { … } is affected in the same way as the file-level test in the report.

Why the host cannot fix it: the language packages emit, per symbol, the line where its attributes start, but not the attributes themselves. A line number cannot tell #[test] from #[inline].

What a fix needs: the packages emit a per-symbol test marker (or normalized attribute/decorator names) through the fact ABI, the index stores it, and the grader uses it next to the path and container rules. That is a change to the ABI and a release of every language package; it belongs with the extraction work in #540.

The same gap applies to the other languages (pytest functions in a non-test file, [Fact]/[Test] methods in a production class): no language's own markers are read.

Investigated in the v0.41.51 train; not fixable in the host alone, so it is not in that release. **Where the test role is decided:** in the host, by one rule for all seven languages: the file's path (test directories and file-name patterns per language, `crates/indexer/src/test_paths.rs`), then the names of enclosing modules, namespaces or classes (`test`, `tests`, `*_test`, `*_tests`; `crates/mcp-server/src/server.rs`, `symbol_is_test_role`). A symbol's own attributes, decorators or name are never read. The container rule is a name heuristic and does not inspect `#[cfg(test)]`, so a `#[cfg(test)] mod checks { … }` is affected in the same way as the file-level test in the report. **Why the host cannot fix it:** the language packages emit, per symbol, the line where its attributes start, but not the attributes themselves. A line number cannot tell `#[test]` from `#[inline]`. **What a fix needs:** the packages emit a per-symbol test marker (or normalized attribute/decorator names) through the fact ABI, the index stores it, and the grader uses it next to the path and container rules. That is a change to the ABI and a release of every language package; it belongs with the extraction work in #540. The same gap applies to the other languages (pytest functions in a non-test file, `[Fact]`/`[Test]` methods in a production class): no language's own markers are read.

buildagent referenced this issue

2026-11-12 18:41:44 +01:00

v0.41.51: bounded text arguments, truthful path and startup answers, review grading by effective visibility #580

buildagent referenced this issue

2026-11-12 22:22:53 +01:00

Missing package facts hide live uses, public members, framework bindings and test roles #540

buildagent commented

2026-11-12 22:23:08 +01:00

Author

Member

Merged into #540.
Host symbol_is_test_role in crates/mcp-server/src/server.rs still uses path/container evidence. No package-marker fix shipped.
Remaining: Per-symbol test facts; the ABI, storage, grader and package releases belong together.

Merged into #540. Host symbol_is_test_role in crates/mcp-server/src/server.rs still uses path/container evidence. No package-marker fix shipped. Remaining: Per-symbol test facts; the ABI, storage, grader and package releases belong together.

buildagent closed this issue

2026-11-12 22:23:08 +01:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

h-dv/code-index#577

No description provided.

Rows
Columns

review_diff grades an edited top-level Rust #[test] function as an untested production change #577

review_diff grades an edited top-level Rust `#[test]` function as an untested production change #577