h-dv/code-index

Watch

Fork 0

Agents lack a measured, client-compatible path to adopt code-index #571

New issue

Open

opened 2026-11-11 09:20:56 +01:00 by buildagent · 4 comments

buildagent commented

2026-11-11 09:20:56 +01:00

Member

Scope (consolidated 2026-11-12)

MCP instructions, an on-demand skill and an optional shell hook are different ways to bring agents to the tools. Their actual consumption and effect on tool choice have not been measured. Installation must fit the client and preserve user-owned instructions. A scripted tool replay measures answer quality and calls, but cannot establish agent adoption.

Priority: Medium · Risk: cost · Size: M
Merged into this ticket: #560 (Optional shell translation hook)

Done

The rules-file command and instruction-file writes were removed in v0.41.49; write_agent_skill in crates/cli/src/skill.rs installs only its own project skill path with --force for a differing copy.
Batch inputs exist on search_symbols/search_text/read_code and code-index query is a CLI path (crates/cli/src/query.rs); the issue body’s claimed MCP query tool does not exist.

Remaining

Client consumption and installation

Observe which client versions consume MCP-served skills and their on-disk search paths; choose project/user destination, detect stale copies and implement the selected differing-copy policy. (#571)

Real-agent measurement

Choose measurement scope and run a pilot before claiming adoption; use fresh sessions, fixed binary/index and oracle answers, with a rules-file comparison arm only inside fixtures if selected. (#571)

Optional shell hook

Resolve whether a blast-radius hook is wanted and how it can be registered without rules; the shell-interception proposal below is a separate optional channel. (#571)
If selected, translate supported indexed-code shell lookups using coverage policy, return the direct tool call, pass through non-code/excluded/unsupported cases with fallback logs, and allow session disable; test p95 ≤50ms and adoption effect. (#560)

Acceptance

Observe skill consumption per client/version; compare selected channels with fresh real-agent sessions and report first-tool choice, calls, correctness and provider tokens. Publish only supported claims.
No install writes instruction text into user-owned files; opt-in channels and differing-copy behavior are documented in one place.
If the operator selects a hook, indexed rg is translated with the suggested direct call, excluded target paths/non-code/unsupported cases pass through and log fallback, per-session disable works, p95 overhead is ≤50ms and fallback drops measurably.

Decisions needed from the operator

“Which clients read MCP-served skills today, and which need the on-disk copy? Where does each client look for skills?” — #571.
“How do we tell a user that an installed on-disk skill is older than the binary’s skill?” — #571.
“Is a blast-radius hook still wanted, and if so, how is it installed without rules?” — #571.
“Measurement scope: none / one client, arms A and B (recommended first step) / two clients with the hook arm.” — #571.
“Should the on-disk skill be installed for the user (home directory) rather than per project?” — #571.
“A differing installed copy: keep --force / refuse and show the difference only (recommended, to hold “never overwrite text the user owns”).” — #571.
“Hook: leave removed (recommended until an edit-task experiment is wanted) / ship as a separate opt-in package with user-applied registration.” — #571.

Suggested order

Choose experiment scope, install destinations and hook policy; operator decision, no package release or bind account.
Implement selected client/scope and version/copy handling; host only, no bind change intended.
Pilot real-agent A/B, then the approved hook arm with pass-through/latency tests; no package release or bind account.

Original report

Operator decision of 2026-11-11: the rules-file feature is removed in the v0.41.49 train. code-index init no longer writes instruction text into a repository, and the rules command with its hook installer is gone. Reason: the written blocks overwrote or cluttered users' own agent instructions (AGENTS.md, CLAUDE.md, editor rule files) and created a mess.

What remains, and what this issue is to decide:

Channels that do not touch a user's instruction files

MCP server instructions and tool descriptions. Loaded by every MCP client, always in context. This is the only always-on channel, so it has to carry "use this instead of shell search".
The skill. The same bytes are served over MCP (skills/list, skills/get, skill://code-index/SKILL.md) and can be installed on disk with the explicit, opt-in code-index skill install, which writes only the skill's own directory. A skill is loaded on demand, so it carries "how to use the tools so the conclusions hold"; it does not by itself make an agent prefer the tools.
The batched query tool and the code-index query CLI, which reduce round trips.

Open questions

Which clients read MCP-served skills today, and which need the on-disk copy? Where does each client look for skills?
Should the on-disk skill be installed for the user (home directory) rather than per project?
Is a blast-radius hook still wanted, and if so, how is it installed without rules?
How do we tell a user that an installed on-disk skill is older than the binary's skill?

Measurement before any claim. None of these channels has been measured against the removed rules files. Use the agent task bench with and without each channel (server instructions only; plus skill; plus hook) and report tool-choice rate, round trips and answer correctness. An earlier lever ("shrink the envelope") was refuted by measurement; the lever that held was fewer calls.

Constraints

Never write instruction text into a file the user owns.
Everything opt-in and documented in one place.
No claim about precision or adoption that the measurement does not support.

## Scope (consolidated 2026-11-12) MCP instructions, an on-demand skill and an optional shell hook are different ways to bring agents to the tools. Their actual consumption and effect on tool choice have not been measured. Installation must fit the client and preserve user-owned instructions. A scripted tool replay measures answer quality and calls, but cannot establish agent adoption. **Priority:** Medium · **Risk:** cost · **Size:** M **Merged into this ticket:** #560 (Optional shell translation hook) ## Done - The rules-file command and instruction-file writes were removed in v0.41.49; `write_agent_skill` in `crates/cli/src/skill.rs` installs only its own project skill path with --force for a differing copy. - Batch inputs exist on search_symbols/search_text/read_code and `code-index query` is a CLI path (`crates/cli/src/query.rs`); the issue body’s claimed MCP query tool does not exist. ## Remaining ### Client consumption and installation - Observe which client versions consume MCP-served skills and their on-disk search paths; choose project/user destination, detect stale copies and implement the selected differing-copy policy. (#571) ### Real-agent measurement - Choose measurement scope and run a pilot before claiming adoption; use fresh sessions, fixed binary/index and oracle answers, with a rules-file comparison arm only inside fixtures if selected. (#571) ### Optional shell hook - Resolve whether a blast-radius hook is wanted and how it can be registered without rules; the shell-interception proposal below is a separate optional channel. (#571) - If selected, translate supported indexed-code shell lookups using coverage policy, return the direct tool call, pass through non-code/excluded/unsupported cases with fallback logs, and allow session disable; test p95 ≤50ms and adoption effect. (#560) ## Acceptance - Observe skill consumption per client/version; compare selected channels with fresh real-agent sessions and report first-tool choice, calls, correctness and provider tokens. Publish only supported claims. - No install writes instruction text into user-owned files; opt-in channels and differing-copy behavior are documented in one place. - If the operator selects a hook, indexed rg is translated with the suggested direct call, excluded target paths/non-code/unsupported cases pass through and log fallback, per-session disable works, p95 overhead is ≤50ms and fallback drops measurably. ## Decisions needed from the operator - “Which clients read MCP-served skills today, and which need the on-disk copy? Where does each client look for skills?” — #571. - “How do we tell a user that an installed on-disk skill is older than the binary’s skill?” — #571. - “Is a blast-radius hook still wanted, and if so, how is it installed without `rules`?” — #571. - “Measurement scope: none / one client, arms A and B (recommended first step) / two clients with the hook arm.” — #571. - “Should the on-disk skill be installed for the user (home directory) rather than per project?” — #571. - “A differing installed copy: keep `--force` / refuse and show the difference only (recommended, to hold “never overwrite text the user owns”).” — #571. - “Hook: leave removed (recommended until an edit-task experiment is wanted) / ship as a separate opt-in package with user-applied registration.” — #571. ## Suggested order 1. Choose experiment scope, install destinations and hook policy; operator decision, no package release or bind account. 2. Implement selected client/scope and version/copy handling; host only, no bind change intended. 3. Pilot real-agent A/B, then the approved hook arm with pass-through/latency tests; no package release or bind account. ## Original report Operator decision of 2026-11-11: the rules-file feature is removed in the v0.41.49 train. `code-index init` no longer writes instruction text into a repository, and the `rules` command with its hook installer is gone. Reason: the written blocks overwrote or cluttered users' own agent instructions (`AGENTS.md`, `CLAUDE.md`, editor rule files) and created a mess. What remains, and what this issue is to decide: **Channels that do not touch a user's instruction files** 1. **MCP server instructions and tool descriptions.** Loaded by every MCP client, always in context. This is the only always-on channel, so it has to carry "use this instead of shell search". 2. **The skill.** The same bytes are served over MCP (`skills/list`, `skills/get`, `skill://code-index/SKILL.md`) and can be installed on disk with the explicit, opt-in `code-index skill install`, which writes only the skill's own directory. A skill is loaded on demand, so it carries "how to use the tools so the conclusions hold"; it does not by itself make an agent prefer the tools. 3. **The batched `query` tool and the `code-index query` CLI**, which reduce round trips. **Open questions** - Which clients read MCP-served skills today, and which need the on-disk copy? Where does each client look for skills? - Should the on-disk skill be installed for the user (home directory) rather than per project? - Is a blast-radius hook still wanted, and if so, how is it installed without `rules`? - How do we tell a user that an installed on-disk skill is older than the binary's skill? **Measurement before any claim.** None of these channels has been measured against the removed rules files. Use the agent task bench with and without each channel (server instructions only; plus skill; plus hook) and report tool-choice rate, round trips and answer correctness. An earlier lever ("shrink the envelope") was refuted by measurement; the lever that held was fewer calls. **Constraints** - Never write instruction text into a file the user owns. - Everything opt-in and documented in one place. - No claim about precision or adoption that the measurement does not support.

buildagent referenced this issue

2026-11-11 11:22:48 +01:00

v0.41.49: PHP namespace paths, settled answers, validated configuration, rules files removed #567

buildagent commented

2026-11-11 18:46:54 +01:00

Author

Member

Analysis for the measurement (2026-11-11, on the v0.41.49 release commit). Nothing was run against a model.

What exists, measured on the installed binary

Server instructions: 3,986 bytes for a two-root deployment (test ceiling 4,096).
tools/list: 23 tools, 59,311 bytes, of which 33,623 bytes are description text. This is the largest fixed cost of a session and is paid by every client, with or without a skill.
Skill: 15,781 bytes, served by skills/list, skills/get and skill://code-index/SKILL.md; code-index skill install writes only .claude/skills/code-index/SKILL.md in the project.
Correction to the issue text: there is no MCP tool named query. Batching is the list form of query on search_symbols/search_text and of target on read_code; code-index query is the CLI.

What the existing bench can and cannot say
agent_task_bench replays a scripted tool plan per question (60 questions over ripgrep, Flask and Dapper) and grades answers, round trips and response size. No model chooses a tool in it. It therefore cannot measure tool choice, skill loading or instruction adherence, and no adoption claim can come from it.

Proposed experiment (real agent, not run)

Arms: A server instructions and descriptions only; B plus the on-disk skill in the client's own skill directory, with no mention of the skill in the task prompt; C plus a hook, on edit tasks only.
Same binary, schemas and index state in every arm; fresh session per task; fixtures outside any repository that carries agent instructions; shell search allowed.
Primary outcome: share of tasks whose first code-search or impact action is code-index. Secondary: round trips, answer correctness against the oracle, provider-reported tokens.
Size: to detect a difference of about 25 points between two arms, 60 tasks per arm are enough; 10 points would need about 390 per arm.
Cheapest useful version: one client, arms A and B, the 60 existing questions once each = 120 runs. Budget assumption 60k–150k input tokens per run, so roughly 7–19 million tokens in total; a 10-task pilot would replace the assumption with a measurement.
A comparison with the removed rules files needs a fourth arm that restores them inside the fixture only.

Where clients look for on-disk skills (from vendor documentation as read by the analysis task; not checked against installed clients): Claude Code .claude/skills and ~/.claude/skills; Codex .agents/skills and ~/.agents/skills; Cursor, Gemini CLI, Copilot and Cline each read .agents/skills or .claude/skills besides their own directory. Whether any of these clients consumes the MCP-served skill automatically is not established by their documentation; it has to be observed per client version.

Decisions needed

Measurement scope: none / one client, arms A and B (recommended first step) / two clients with the hook arm.
Skill destination: keep project-only / add an explicit --client and --scope choice (recommended; Codex needs .agents/skills, which the installer does not write today) / user-level by default.
A differing installed copy: keep --force / refuse and show the difference only (recommended, to hold "never overwrite text the user owns").
Hook: leave removed (recommended until an edit-task experiment is wanted) / ship as a separate opt-in package with user-applied registration.
Claims: publish only what the experiment measured, per client and version.

**Analysis for the measurement (2026-11-11, on the v0.41.49 release commit). Nothing was run against a model.** **What exists, measured on the installed binary** - Server instructions: 3,986 bytes for a two-root deployment (test ceiling 4,096). - `tools/list`: 23 tools, 59,311 bytes, of which 33,623 bytes are description text. This is the largest fixed cost of a session and is paid by every client, with or without a skill. - Skill: 15,781 bytes, served by `skills/list`, `skills/get` and `skill://code-index/SKILL.md`; `code-index skill install` writes only `.claude/skills/code-index/SKILL.md` in the project. - Correction to the issue text: there is no MCP tool named `query`. Batching is the list form of `query` on `search_symbols`/`search_text` and of `target` on `read_code`; `code-index query` is the CLI. **What the existing bench can and cannot say** `agent_task_bench` replays a scripted tool plan per question (60 questions over ripgrep, Flask and Dapper) and grades answers, round trips and response size. No model chooses a tool in it. It therefore cannot measure tool choice, skill loading or instruction adherence, and no adoption claim can come from it. **Proposed experiment (real agent, not run)** - Arms: A server instructions and descriptions only; B plus the on-disk skill in the client's own skill directory, with no mention of the skill in the task prompt; C plus a hook, on edit tasks only. - Same binary, schemas and index state in every arm; fresh session per task; fixtures outside any repository that carries agent instructions; shell search allowed. - Primary outcome: share of tasks whose first code-search or impact action is code-index. Secondary: round trips, answer correctness against the oracle, provider-reported tokens. - Size: to detect a difference of about 25 points between two arms, 60 tasks per arm are enough; 10 points would need about 390 per arm. - Cheapest useful version: one client, arms A and B, the 60 existing questions once each = 120 runs. Budget assumption 60k–150k input tokens per run, so roughly 7–19 million tokens in total; a 10-task pilot would replace the assumption with a measurement. - A comparison with the removed rules files needs a fourth arm that restores them inside the fixture only. **Where clients look for on-disk skills** (from vendor documentation as read by the analysis task; not checked against installed clients): Claude Code `.claude/skills` and `~/.claude/skills`; Codex `.agents/skills` and `~/.agents/skills`; Cursor, Gemini CLI, Copilot and Cline each read `.agents/skills` or `.claude/skills` besides their own directory. Whether any of these clients consumes the MCP-served skill automatically is not established by their documentation; it has to be observed per client version. **Decisions needed** 1. Measurement scope: none / one client, arms A and B (recommended first step) / two clients with the hook arm. 2. Skill destination: keep project-only / add an explicit `--client` and `--scope` choice (recommended; Codex needs `.agents/skills`, which the installer does not write today) / user-level by default. 3. A differing installed copy: keep `--force` / refuse and show the difference only (recommended, to hold "never overwrite text the user owns"). 4. Hook: leave removed (recommended until an edit-task experiment is wanted) / ship as a separate opt-in package with user-applied registration. 5. Claims: publish only what the experiment measured, per client and version.

buildagent referenced this issue

2026-11-12 05:46:24 +01:00

v0.41.50: module evidence for qualified references, one argument contract, one extraction census #574

buildagent referenced this issue

2026-11-12 22:22:54 +01:00

Agents still need shell fallbacks to finish code investigation and review #555

buildagent changed title from ~~Rethink how agents are brought to use code-index, after removing the rules files~~ to Agents lack a measured, client-compatible path to adopt code-index

2026-11-12 22:22:54 +01:00

buildagent referenced this issue

2026-11-12 22:23:10 +01:00

Shell interception hook: answer grep/rg/cat on indexed code via code-index and record real fallbacks (zero-fallback) #560

buildagent added the

Kind/Feature

Priority

Medium

labels

2026-11-12 22:23:30 +01:00

buildagent referenced this issue

2026-11-13 01:09:02 +01:00

Tool discovery and replies cost extra calls, while review and navigation still have gaps #521

buildagent referenced this issue

2026-11-13 07:24:00 +01:00

Tool discovery and replies cost extra calls, while review and navigation still have gaps #521

buildagent commented

2026-11-13 16:50:48 +01:00

Author

Member

Operator decisions, 2026-11-13 (asked as concrete questions, answered in the session):

Channel for the skill: "use skill as mcp resource and let agent decide". The skill served over MCP is the supported channel; the client or agent decides whether and where to load it. Asked what should happen when an installed on-disk copy differs from the binary's skill, the answer was "mcp": the MCP-served skill is the reference, and the product does not manage on-disk copies further.
Hook: stays removed.
First measurement: one client, two arms (server instructions only; server instructions plus the skill), reporting tool-choice rate, round trips and answer correctness on the agent task bench.

Lead's reading of the first point, to be corrected if wrong: no work goes into code-index skill install (no default location change, no new handling of differing copies); whether that command is kept as it is or removed is not decided by these answers and is asked separately before anything is changed.

That closes five of the seven questions in the body. The two that are research, not decisions ("which clients read MCP-served skills today", "how does a user learn that an on-disk copy is older") are answered as part of the measurement lane; the second one falls away if on-disk copies are not managed.

**Operator decisions, 2026-11-13** (asked as concrete questions, answered in the session): - **Channel for the skill:** "use skill as mcp resource and let agent decide". The skill served over MCP is the supported channel; the client or agent decides whether and where to load it. Asked what should happen when an installed on-disk copy differs from the binary's skill, the answer was "mcp": the MCP-served skill is the reference, and the product does not manage on-disk copies further. - **Hook:** stays removed. - **First measurement:** one client, two arms (server instructions only; server instructions plus the skill), reporting tool-choice rate, round trips and answer correctness on the agent task bench. Lead's reading of the first point, to be corrected if wrong: no work goes into `code-index skill install` (no default location change, no new handling of differing copies); whether that command is kept as it is or removed is not decided by these answers and is asked separately before anything is changed. That closes five of the seven questions in the body. The two that are research, not decisions ("which clients read MCP-served skills today", "how does a user learn that an on-disk copy is older") are answered as part of the measurement lane; the second one falls away if on-disk copies are not managed.

buildagent commented

2026-11-13 16:59:20 +01:00

Author

Member

Operator decision, 2026-11-13, on the question left open above: "remove the skill cli command". The code-index skill command and everything that exists only for an on-disk copy (the doctor check, the --force advice, the install instructions) are removed; the skill stays available as the MCP resource skill://code-index/SKILL.md and through skills/list / skills/get. A lane is doing this now. Release notes will name it as a removed command.

**Operator decision, 2026-11-13, on the question left open above:** "remove the skill cli command". The `code-index skill` command and everything that exists only for an on-disk copy (the doctor check, the `--force` advice, the install instructions) are removed; the skill stays available as the MCP resource `skill://code-index/SKILL.md` and through `skills/list` / `skills/get`. A lane is doing this now. Release notes will name it as a removed command.

buildagent referenced this issue

2026-11-14 00:43:40 +01:00

v0.41.53: restart and update behaviour a session can rely on; link add keeps the language decision; skill command removed #587

buildagent referenced this issue from a commit

2026-11-14 11:54:09 +01:00

Merge lane r43: the skill CLI command is removed; the skill is served over MCP only (#571)

buildagent commented

2026-11-14 12:28:47 +01:00

Author

Member

code-index skill is removed; the skill remains available as the MCP resource skill://code-index/SKILL.md via skills/list and skills/get. Clients that cannot read MCP resources can save the resource text themselves.

`code-index skill` is removed; the skill remains available as the MCP resource `skill://code-index/SKILL.md` via `skills/list` and `skills/get`. Clients that cannot read MCP resources can save the resource text themselves.

buildagent referenced this issue

2026-11-14 20:02:49 +01:00

Tool discovery and replies cost extra calls, while review and navigation still have gaps #521

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

h-dv/code-index#571

No description provided.

Rows
Columns