Generation rollback outcome depends on SQLite rowid reuse: an edited file can lose its live rows #321

Closed
opened 2026-10-06 03:26:12 +02:00 by buildagent · 2 comments
Member

Found while root-causing a CI-only failure of generation_promotion::a_rollback_skips_queued_files_that_no_longer_exist. The test is fixed; this product defect is not.

Mechanism

  • files.id is a plain INTEGER PRIMARY KEY, not AUTOINCREMENT (crates/indexer/src/migrations/m0001_initial.rs:5).
  • write_upsert replaces a file's row with a DELETE followed by an INSERT (crates/indexer/src/writer.rs:306-340).
  • SQLite hands the new row max(rowid)+1. If the edited file held the highest id, it gets the SAME id back.
  • Which file holds the highest id depends on directory-listing order and parse-thread completion order (walker.rs:967, index.rs:574-611).

Effect, measured with debug output in the test

  • When the id is reused, a rollback withdraws the edited file's NEW live rows. src/notes.md ends with no live contribution at all.
  • With a fresh id, it keeps its live row.

So the rollback's result depends on rowid allocation. The cause is that generation_build_files.file_id is assumed never to be reissued (generations.rs:1168-1172, withdraw at :1003-1016). A reissued id can also name a DIFFERENT path: delete the highest-id file, add a new file, and the new file takes the old id.

Separately, write_upsert has no "retained rows" branch like write_delete's, so editing a file also deletes the old generation's retained rows.

Reproduce: check out the test before its fix (PR #320), run it with TMPDIR=/dev/shm/... and RAYON_NUM_THREADS=1. It fails 10 out of 10 times.

Fix options (design decision needed):

  • (a) files.id AUTOINCREMENT. That is a table rebuild, with the known parent-table cascade hazard.
  • (b) Give the build queue a stable identity (path plus content identity) instead of file_id.

Either way, add a test that stages the reuse deliberately rather than by luck.

Found while root-causing a CI-only failure of `generation_promotion::a_rollback_skips_queued_files_that_no_longer_exist`. The test is fixed; this product defect is not. **Mechanism** - `files.id` is a plain `INTEGER PRIMARY KEY`, not AUTOINCREMENT (`crates/indexer/src/migrations/m0001_initial.rs:5`). - `write_upsert` replaces a file's row with a DELETE followed by an INSERT (`crates/indexer/src/writer.rs:306-340`). - SQLite hands the new row `max(rowid)+1`. If the edited file held the highest id, it gets the SAME id back. - Which file holds the highest id depends on directory-listing order and parse-thread completion order (`walker.rs:967`, `index.rs:574-611`). **Effect, measured with debug output in the test** - When the id is reused, a rollback withdraws the edited file's NEW live rows. `src/notes.md` ends with no live contribution at all. - With a fresh id, it keeps its live row. So the rollback's result depends on rowid allocation. The cause is that `generation_build_files.file_id` is assumed never to be reissued (`generations.rs:1168-1172`, withdraw at `:1003-1016`). A reissued id can also name a DIFFERENT path: delete the highest-id file, add a new file, and the new file takes the old id. Separately, `write_upsert` has no "retained rows" branch like `write_delete`'s, so editing a file also deletes the old generation's retained rows. **Reproduce:** check out the test before its fix (PR #320), run it with `TMPDIR=/dev/shm/...` and `RAYON_NUM_THREADS=1`. It fails 10 out of 10 times. **Fix options (design decision needed):** - (a) `files.id` AUTOINCREMENT. That is a table rebuild, with the known parent-table cascade hazard. - (b) Give the build queue a stable identity (path plus content identity) instead of `file_id`. Either way, add a test that stages the reuse deliberately rather than by luck.
Author
Member

Current-source audit on b7c34798 (isolated worktree; no release source changes): both forced reuse cases pass. With the edited path as the only indexed file, SQLite demonstrably reused id 1 → 1; package activation → edit → rollback → builtin pass matched a cold builtin index of current bytes. A second case deleted the queued path, reconciled to zero files, then created a different path which demonstrably reused id 1; rollback plus the builtin pass again matched the cold whole-tree projection.

The current promotion code refreshes generation_build_files.file_id through files.path before withdrawing rows; it also marks missing incoming facts as owed. The original rowid-dependent outcome did not reproduce in either deterministic case. This does not assert retained obsolete rows survive an edit: the checked contract is correct current-byte facts after rollback and convergence.

Keeping this issue open pending a permanent, deliberately forced-reuse regression (the existing owner_switch_rollback edit test exercises the corrected path lookup but does not force SQLite allocation). No AUTOINCREMENT/table rebuild appears necessary for these witnessed cases.

Current-source audit on b7c34798 (isolated worktree; no release source changes): both forced reuse cases pass. With the edited path as the only indexed file, SQLite demonstrably reused id 1 → 1; package activation → edit → rollback → builtin pass matched a cold builtin index of current bytes. A second case deleted the queued path, reconciled to zero files, then created a different path which demonstrably reused id 1; rollback plus the builtin pass again matched the cold whole-tree projection. The current promotion code refreshes generation_build_files.file_id through files.path before withdrawing rows; it also marks missing incoming facts as owed. The original rowid-dependent outcome did not reproduce in either deterministic case. This does not assert retained obsolete rows survive an edit: the checked contract is correct current-byte facts after rollback and convergence. Keeping this issue open pending a permanent, deliberately forced-reuse regression (the existing owner_switch_rollback edit test exercises the corrected path lookup but does not force SQLite allocation). No AUTOINCREMENT/table rebuild appears necessary for these witnessed cases.
Author
Member

PR#462 adds the missing permanent forced-reuse regression. Both edited-path and delete/replacement-path cases start from an empty files table, assert actual id reuse, and compare the post-rollback/pass whole projection to a cold index of current bytes. It additionally asserts durable owed work for invalidated retained facts and that rollback does not withdraw the replacement path's current facts. The current path-based queue refresh already handles these cases: disabling that refresh makes this test RED at the replacement's live-contribution assertion. No AUTOINCREMENT/table rebuild is needed for these witnesses. Normal CI is running before closure.

PR#462 adds the missing permanent forced-reuse regression. Both edited-path and delete/replacement-path cases start from an empty files table, assert actual id reuse, and compare the post-rollback/pass whole projection to a cold index of current bytes. It additionally asserts durable owed work for invalidated retained facts and that rollback does not withdraw the replacement path's current facts. The current path-based queue refresh already handles these cases: disabling that refresh makes this test RED at the replacement's live-contribution assertion. No AUTOINCREMENT/table rebuild is needed for these witnesses. Normal CI is running before closure.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#321
No description provided.