After a schema upgrade the daemon serves for the whole re-parse from an index whose stat and hash were all invalidated — and nothing is known to disclose it #155
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
h-dv/code-index#155
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found by tracing a question a Windows session asked about #144: does the startup grace cover only the migration chain, or the re-parse m0016 forces?
The answer, and why it inverts the concern
crates/daemon/src/main.rsorder:The reconcile — which is what actually re-parses the files m0016 invalidated — runs after the accept loop is serving.
main.rs:390states it: "Clients are still free to connect during reconcile — the accept loop is already up (started above)."Good news for #144: takeover can only evict a silent holder, and during the re-parse the daemon is answering. So the 60 s grace is compared against the right quantity, and the re-parse is not part of the killable window.
Bad news, and it is the sharper finding. 14 of the ~60 migrations invalidate every code file's mtime and hash. On a large repo that forces a full re-parse. A Windows operator reports a full re-parse of a ~1 GB C# index (TimeLine 16.0) taking ~90 minutes with no log output at all.
So after a schema upgrade the daemon spends potentially ninety minutes serving — never silent, never evicted, never refused — from an index where every code file's stat and hash have been invalidated. Not a brick. A long window of confidently degraded answers.
The question this issue is
Does anything in the payload say so during that window?
I have not checked. The candidates are the
index_stale/ freshness family,resolve_progress, andindex_coverage'spending— all of which exist for exactly this class. What matters is whether a user in that window is told the index is mid-rebuild, or is quietly handed thin answers that look complete.This is the same shape as #145 (
index_coveragetelling you to retry in a second for a population whose real horizon is a reconcile interval): apendingwhose true horizon is ninety minutes must not render like one whose horizon is a debounce.Cheap way to answer it
Does not need a v15 database or a 1 GB index. Upgrade a modest project across a schema version that invalidates hashes, then call
project_overviewandindex_coverageduring the reconcile and read what they say. Minutes, not ninety.Adjacent, filed together
main.rs:190-193still reads:That was true when the accept loop started after reconcile. It moved,
main.rs:390records the move, and this comment was not updated — so the file states both orderings 200 lines apart. Either reader gets the wrong answer depending which one they find first.Related
pendinghorizons)doctoris blind to index staleness)🤖 Generated with Claude Code
https://claude.ai/code/session_01K1zj5VcFJvJt3pQxe9259K
Half answered already, by accident, and the answer is better than expected
A Windows session hit the degraded window twice in one day without trying, on a daemon-restart reconcile. Two
project_overviewcalls, same repo, different daemon pids:Both carried:
The disclosure is real, accurate, and load-bearing for a consumer. Resolution moved 21.5% → 44.8% between two reads of the same repo. A caller taking the first at face value would have reported this codebase as resolving 14% of its Rust references. They did not, and said they only avoided it because that block was there — their write-up called the figures "a partial floor rather than a steady-state figure", which is correct and which they could not have known to say otherwise.
They also tried to catch it out and failed to:
pass_seq: 0beside counters that had clearly doubled looks like a stale progress block. It is not — the daemon pid changed, so "no resolve transaction has run yet in this daemon" is literally true and the counters belong to its predecessor. The three words carrying the honesty are "in this daemon." Worth recording that the wording survives an adversarial reader.What this does NOT settle — the half this issue is about
That was a reconcile after a daemon restart, not after a schema upgrade. Nothing exercised m0016's forced re-parse. Still open:
state: "reconciling", or something weaker?state_detailsays committed counters may be partial. After m0016 the counters describe rows whose files were all invalidated — a different and worse claim, and the same sentence stays technically true while understating it. That is the sharp residue.index_coveragehas not been exercised in either window.So the channel exists and works; whether it says enough after a schema upgrade is untested.
Adjacent, now fixed
main.rs:190-193stated the opposite ordering to line 390 — two contradictory orderings 200 lines apart in the file that decides this behaviour, with the stale one reading as authoritative. Corrected, with the correction recording what it used to say and why the distinction is load-bearing twice: it is why the 60 s grace is compared against the migration and not the re-parse, and why that re-parse is a window of degraded answers rather than of silence.🤖 Generated with Claude Code
https://claude.ai/code/session_01K1zj5VcFJvJt3pQxe9259K
index_coveragemeasured in the window: it says "Indexed and current." and CANNOT say otherwiseThe third open question from the previous comment, answered in one call. Same daemon (pid 27128) that
project_overviewreports asstate: "reconciling":This is not a lie, and that is what makes it the residue
The per-path answer is true: the row exists, the content hash matches,
stale: falseis correct for the question asked. The two tools answer different questions and both answer honestly.But
coverage_reasonsis documented as "#80's closed Coverage family … qualifying THIS answer", with an empty list explicitly a measurement. The field exists precisely to say what qualifies a verdict — and the closed vocabulary has no member for "the index this path sits in is mid-reconcile."That makes it a SCHEMA gap, not a wording one — a different repair from the
state_detailresidue above:index_coverageon any of them returnsstale: false,"Indexed and current."— because the row and hash are still there — right up until the re-parse rewrites it.index_coverage(path)to settle coverage in one call, instead of a defensive grep) gets a confident "current" with nothing qualifying it. That is exactly the consumer the tool was optimised for.index_reconciling, emitted whenever the daemon'sstateis not steady. One code, and it keeps the "empty list is a measurement" invariant — which a prose-only fix would break.The asymmetry is the sharp part
project_overviewdiscloses the window;index_coveragedoes not — and the one that does not is the cheaper call this repository's own guidance tells agents to prefer. The disclosure is strongest exactly where a careful caller is least likely to be looking.Same shape as everything else in this family, one level further out: the population
coverage_reasonscan describe was fixed before anyone had stood in this state with a reader paying attention.Consequence for the repair
This issue now has two distinct repairs, and they are not the same change:
state_detailcannot distinguish "stale but complete" from "invalidated and being rebuilt" (previous comment).coverage_reasonshas no code for a mid-reconcile index, soindex_coveragestructurally cannot qualify its answer.(2) is the one that matters more, because it is the tool agents are told to reach for.
🤖 Generated with Claude Code
https://claude.ai/code/session_01K1zj5VcFJvJt3pQxe9259K
Both repairs from the second comment are DONE. Verdict: FIXED.
Lane worktree:
/tmp/cosi-lane-honesty, branchwip/honesty, based onorigin/master(ea821b6). Not pushed.The comments supersede the body, so I worked from them: repair (1) is the WORDING gap in
state_detail, repair (2) is the SCHEMA gap incoverage_reasons. Both are in. The adjacentmain.rs:190-193comment was already corrected on master before this lane started — verified, not re-done.Repair 2 (the one that matters more):
coverage_reasonsgains a memberindex_reconciling, and why ONE code for three statesNew closed-family member
coverage::INDEX_RECONCILING = "index_reconciling", emitted by the single producercoverage::qualifyfrom a new three-stateCoverageState::lifecycle_steady: Option<bool>.One predicate, deliberately wider than "reconciling": the lifecycle state of the index that answered is not steady. That covers
reconciling(the post-upgrade re-parse and the startup one),resyncing(a full re-walk) anddegraded. They differ in cause and remedy —project_overview.state/state_detailis where a caller reads WHICH — but they make the same claim about a per-path verdict: the row this answer rests on may be replaced by work already in flight. Three codes would put the burden of knowing all three on every consumer, and a consumer filtering for two would be qualified by nothing in the third. That is #124's defect, re-created.Three states, and neither default is safe
None= did not report (a daemon predating the field) → emits nothing.Some(true)= a measurement → emits nothing.Some(false)→ emits the code. Afalsedefault would make every older daemon carry a code about a state nobody measured; atruedefault would make it claim a steadiness nobody measured.The wire
PathClaimsgainslifecycle_state: Option<String>— the per-path reply that already carriesrequested_not_installedandpackage_refusalsfor exactly this reason: the measurement that qualifies a per-path verdict, taken in the same round trip so the two cannot disagree. Stamped byLocalIndex::path_claims, not byProjectExtractors::claims, because the extractor set belongs to the PROJECT and the lifecycle belongs to the PROCESS — the same set answers in a--no-daemonsnapshot and inside a reconciling daemon, and only one of those is steady.Decoding rules, all four graded:
None— did not report"ready"Some(true)"reconciling"/"resyncing"/"degraded"Some(false)Some(false)--no-daemonreports"ready"deliberately: built once at startup, never re-walked, nothing in flight by construction. Reporting absence there would make a snapshot indistinguishable from an old daemon — the exact collapse.The asymmetry the comment called sharp is closed at the producer, not per surface.
project_overview'scoverage_reasons_forandindex_coverage'scoverage_forboth feed the samequalify, so the two tools cannot come to disagree about whether the index was steady — which is exactly what they DID disagree about.FINISHING THE PAIR: the hint
A code beside a sentence that denies it is worse than no code (#140's lesson, in this very tool).
hintin the indexed/current arm is now derived from the same value the code is:The arithmetic guard, kept exact
every_coverage_code_is_flat_lower_snake_and_uniquepins the family size at "#80's four minus the retired ones". The cheapest way to add a code would have been to loosen it to>=, after which a typo'd duplicate or a stray string would be invisible. Instead the count grows only through a new register,coverage::BEYOND_ISSUE_80, which costs one row and a written reason, and which the same test asserts the family actually declares.index_reconcilingis its first and only member, and it is explicitly NOT presented as a #80 code — that catalogue is a contract with a spec.Catalogued in
code-index://docs/reason-codes(the registry demanded it:every_declared_code_is_in_the_cataloguewent RED until it was there) and named incoverage::SEMANTICS.Repair 1:
state_detailsays WHICH kind of reconcileMeasured, not inferred from which migration ran
Every invalidating migration writes the SAME sentinel —
files.mtime_ns = -1, hash zeroed — because that is what makes the next reconcile re-parse the file, and the reconcile clears it per file as it goes. So one count overfilesanswers it for all fourteen, for any future one, and for a half-finished rebuild, where a per-migration registry would need a row per migration and would still say nothing about progress.stats()takes that census only whilestate == "reconciling", and appends one of three sentences:n > 0→ "SCHEMA-UPGRADE REBUILD: {n} of {files} indexed files currently carry the invalidation sentinel … the counters above are not merely partial - they describe rows that are being REPLACED."n == 0→ "…this is an ordinary reconcile and not a schema-upgrade rebuild - the counters above are partial, not rows that are all being replaced."The third arm is the point. My first cut used
.ok(), which would have made "could not look" and "no rebuild" render identically — the defect this change is about, re-created inside the fix.Mutations (ALL RUN)
M1 —
coverage_forgetsNonefor the lifecycle instead ofelig_lifecycle(9 call sites):M2 — the DAEMON stamps
LIFECYCLE_READYunconditionally (proves the fact is carried by the daemon, not guessed client-side):M3 — an absent
lifecycle_statedecodes as steady (is_none_orinstead ofmap):M4 — an unknown phase decodes as steady (
!= "reconciling"instead of== LIFECYCLE_READY):M5 — the
state_detailunmeasured arm falls back to the plain detail: RED onreconciling_says_whether_every_row_is_being_rebuilt, arm 3.Existing gates that went RED on their own and had to be satisfied (recorded because they are evidence the family's guards work):
every_coverage_code_is_flat_lower_snake_and_unique(left: 5, right: 4),the_semantics_names_every_code_the_field_can_carry(`SEMANTICS` can carry `index_reconciling` and does not name it),the_payload_disclosure_does_not_name_a_retired_code, andreason_code_registry::every_declared_code_is_in_the_catalogue.Tests added
warmup_e2e.rs::index_coverage_discloses_a_mid_rebuild_index_and_only_then— real daemon, real reconcile window, both directions on ONE daemon: the code appears during reconcile, and after the same reconcile settles it is gone withverdict: "indexed"still present (so the absence is not an error envelope). A code that rides on every answer qualifies none of them.mcp-server/src/eligibility.rs::the_lifecycle_state_decodes_into_four_distinct_readings— the four wire readings plus the builtin degrade path, then end-to-end throughqualifyso it grades the reason code and not just a getter.daemon/src/local_index.rs::reconciling_says_whether_every_row_is_being_rebuilt— four arms: ordinary reconcile (the control, first), schema-upgrade rebuild (asserting the sentence carries2 of 3, so a whole-tree rebuild is distinguishable from three stragglers), census-failed, and steady (says nothing even with sentinel rows in the table).coverage.rs::each_coverage_code_has_its_own_trigger— all three lifecycle states asserted, because the whole finding is that two of them rendered alike.Gates
cargo fmt --all -- --check0 ·cargo clippy --workspace --all-targets -- -D warnings0 ·RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --document-private-items0.tests/corpus/baseline.jsonunmoved at534084b856c22566c48e386bc41ed67e.What I did NOT do
I did not exercise a real schema upgrade end to end — the e2e drives a startup reconcile with an injected delay, and the
mtime_ns = -1census is graded at the unit level against a planted sentinel. The sentinel itself is pinned by the existing migration tests (m0016,m0025,m0031,m0028,m0018,m0020,m0021all assertmtime == -1), which is what makes the one count sound across all fourteen; a real cross-version upgrade in the e2e harness would be strictly better and is not here.🤖 Generated with Claude Code
https://claude.ai/code/session_01K1zj5VcFJvJt3pQxe9259K
Follow-up after rebasing onto master (
095eba6): three things the rebase and the gates addedThe work above is unchanged in substance. Three things came out of finishing it that belong on the record, two of which are findings rather than fixes.
1. A DERIVATION DEBT, recorded rather than absorbed
Making
Stats::statefeed a renderer —lifecycle_steady_of(stats)derives one boolean socoverage_reasonscan carryindex_reconciling— trippeddisclosure_derivation_registry::a_renderer_owned_field_never_reaches_the_wire_raw:That gate is right in general and its premise does not hold for THIS field, which is why the two rows I added say so instead of the ceiling being quietly raised:
DERIVATION_DEBT_CEILING3 → 5. Mutation RUN (keep the rows, restore the ceiling to 3):Master's newer
every_stats_reading_function_is_a_declared_surface_or_helperalso caught the helper on its first run —lifecycle_steady_ofis now a declaredSTATS_HELPERSrow. Worth noting because it is that gate working exactly as #137 designed it, on a function that did not exist when it was written.2. A FINDING:
#[serde(default)]on anOptionis decoration on this wire tooI wrote a wire-skew assertion (an old daemon's reply without
lifecycle_statemust still decode, and aNonemust be ABSENT rather thannull) and then ran the obvious mutation — dropping#[serde(default)]from the field. It SURVIVED.That is a finding about my test, so I measured the cause rather than re-aiming:
So
serde_jsonalready yieldsNonefor a missingOption<T>and already omits aNoneunderskip_serializing_if. This generalises the silent-loss lane's measurement on thetomlwire to the JSON RPC wire: the attribute is not what makes skew-tolerance true on either. There are ~30 such fields onPathClaims/Statsalone; none of them is protected by that attribute, and a reader who believes otherwise will misjudge which changes are safe.The assertion is kept, its comment now says what actually makes it true, and the mutation that DOES break it was found and run — collapsing the three-state
Option<String>into a two-stateString:3. A FINDING:
code-index://docs/reason-codesis AT its 4,000-token resource capAdding one catalogue entry pushed the topic over, and the overflow cascaded into three more gates (
every_docs_topic_is_listed_as_a_concrete_resourceserved it as an 8,343-char stub;every_json_resource_discloses_a_truncated_source…served a JSON envelope instead of the document). The entry now fits, paid for by trimming provenance prose elsewhere in the file — and the page says so out loud rather than leaving the next author to rediscover it:The next reason code this project mints cannot be documented without a trim or a topic split. That is #120's crowding in the resource layer and is worth its own issue.
Also, one process note against myself: I used
git checkout <file>to undo a mutation and destroyed the uncommitted wire-skew block — the exact move this repo's own notes forbid. Rewritten from scratch; nothing else was lost, and no other undo in this lane used it.Gates, post-rebase onto
095eba6cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningsRUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --document-private-itemscargo test --workspace --no-fail-fastCOSI_E2E_LEG=daemon cargo test -p code-index-mcp --no-fail-fastcorpus_ratchet(require floor)tests/corpus/baseline.jsonis unmoved at534084b856c22566c48e386bc41ed67eand was NOT blessed. The ratchet drift is resolver counts on rust-ripgrep / ts-zod / ruby-sinatra (refs,resolved,edges,resolved_call/method_call/type/read/member_access), which is2f16e22's work and the withdrawn bless the #165 lane owns. Structurally not mine:corpus_ratchetruns-p code-index-indexer, and this lane's only production change in that crate iscoverage.rs, which no module on the index path (index.rs,writer.rs,reconcile.rs,collect.rs,walker.rs,dirty.rs) references at all.One honest caveat on the daemon leg: an earlier run of it failed three tests, two of them
project index never finished warming up in 15sat load average 12-15 with five lanes building. All three pass isolated, and the full leg is green on the re-run. Recorded rather than silently re-run, because "a correctness suite cannot see a slowdown" cuts both ways: a contended run is not evidence in either direction.🤖 Generated with Claude Code
https://claude.ai/code/session_01K1zj5VcFJvJt3pQxe9259K
Triage 2026-09-06: CLOSING. Both repairs the comments asked for are in, and the e2e that grades them is not vacuous.
Verified against master. The
wip/honestywork a lane described as "not pushed" did land — merge833aaa5.Repair 2 — the schema half
coverage::INDEX_RECONCILING = "index_reconciling"—crates/indexer/src/coverage.rs:190, and it is in the closed family at:199, soindex_coveragestructurally can qualify its answer instead of returning an empty list that this project's doctrine reads as "nothing qualifies this".CoverageState::lifecycle_steady: Option<bool>—coverage.rs:843. Three-state, so absent ≠ steady.coverage.rs:924-926, insidequalify.PathClaims::lifecycle_statestamped byLocalIndex::path_claims(crates/daemon/src/local_index.rs:1754,:4800); client decodelifecycle_steady()atcrates/mcp-server/src/eligibility.rs:191.project_overviewderives its side fromStats::statevialifecycle_steady_of(crates/mcp-server/src/server.rs:4200) into the samecoverage_reasons_for— so the two surfaces cannot disagree, which is the failure mode #148/#152 were about.coverage::BEYOND_ISSUE_80register (coverage.rs:482-493).Repair 1 — the wording half
The
mtime_ns = -1sentinel census with three arms — rebuild / ordinary reconcile / census-failed —crates/daemon/src/local_index.rs:2362-2375. The third arm is the one that matters: "could not look" is now distinct from "no rebuild", which was the specific trap.The paired hint was finished too:
server.rs:13538no longer says the bare "Indexed and current." but "Indexed and current AS OF THIS ROW …". Fixing one half of a paired field is what makes the other half the bug, so this is the right shape.Adjacent
The contradictory ordering comment at
main.rs:190-193is corrected — nowcrates/daemon/src/main.rs:202-215, recording what it used to say and stating the true ordering.Runs (all exit 0)
index_coverage_discloses_a_mid_rebuild_index_and_only_thencrates/mcp-server/tests/warmup_e2e.rs:664the_lifecycle_state_decodes_into_four_distinct_readingscrates/mcp-server/src/eligibility.rs:830reconciling_says_whether_every_row_is_being_rebuiltcrates/daemon/src/local_index.rs:7328each_coverage_code_has_its_own_triggercrates/indexer/src/coverage.rs:1103The e2e is not vacuous: it polls for the code during reconcile on a real spawned daemon, asserts the hint is not the bare sentence, then on the same daemon waits for steady and asserts the code is gone with
verdict: "indexed"still present. That last clause is the anti-vacuity control — without it, a test that broke the whole reply would pass.Residuals, named and not blocking
mtime_ns = -1census is graded at unit level against a planted sentinel. "One count answers it for all fourteen migrations" rests on the existing per-migration tests assertingmtime == -1, not on an upgrade run.DERIVATION_DEBT_CEILINGraised 3 → 5 (crates/mcp-server/tests/disclosure_derivation_registry.rs:705). Paying it down means emitting"ready"explicitly forStats::state— a wire change.code-index://docs/reason-codesis at its 4,000-token resource cap (3,979 used). The next reason code this project mints cannot be documented without a trim or a topic split. That is a real finding and deserves its own issue — it is not this one.🤖 Triage lane, 2026-09-06, master
45cf6e4code-index://docs/reason-codesis at 3,979 of its 4,000-token cap, so the next reason code this project mints cannot be documented #184