Field (zammad-dev, 0.34.0): first daemon after upgrade ran a full resolve for 40+ min, then exited "sqlite error: interrupted"; refs_resolved read 0 and the watcher starved meanwhile #334

Closed
opened 2026-10-08 16:51:11 +02:00 by buildagent · 4 comments
Member

This is a field report from zammad-dev (A1X code-index-dev, 2026-10-08). Their setup: a Zammad 7.0.2 fork with Rails and Vue/TS, 10,009 files, build 0.34.0 (014ed555), Ruby owned by de.h-dv.ruby 0.8.1 (plugin generation 4), and a 700 MB index DB. Not yet reproduced here; a debugging lane is on it.

Observed

  1. The hang. Daemon pid 803642 ran a FULL resolve (pass_seq 1) for more than 40 min at about 100% CPU and 1.7 GB RSS.
    • last_completed_step stayed at access_ref_to_member_access for about 37 min while the heartbeat was alive.
    • It then exited: fatal error: startup generation 4 committed; resolution remains pending: sqlite error: interrupted (exit code 1).
  2. Empty answers during the pass. project_overview reported refs_resolved 0 (81,059 earlier in the same run), 0% for every language and no top symbols. index_state did say "floors". Even so, find_callers, find_references and change_impact answered from nothing for about 40 min.
  3. Watcher starvation. Edits were not picked up for 40+ min: index_freshness showed lagging with 154 added and 39 changed paths. Line numbers in line_text went stale and new files were missing from list_files.
  4. After a restart (pid 813834) the daemon is steady: metadata_agrees, 0 lagging. Generation 2 is still "awaiting collection".

Likely trigger (unconfirmed)

This was probably the first daemon to open the index after the upgrade from 0.33.1. Migration v79 (2a2e2629) runs one Full Ruby re-resolve on upgrade. For comparison, a cold CLI index of a local Zammad 7.0.2 copy with the 0.34.0 code takes 67 s in total, with lexical_constant at 0.86 s. So the difference is in the upgrade, daemon, package-owner or generations path.

Hypotheses

  • (a) The step after access_ref_to_member_access is pathological on this path.
  • (b) An interrupt in the STARTUP resolve, after the generation commit, is treated as fatal rather than as preempted or cancelled (compare b13303f3 and 7c69a192).
  • (c) Watcher preemption loops, so the pass never completes.
  • (d) A Full pass clears bindings where readers can see them.

Asked of the reporter

daemon.log for pid 803642, and whether it was the first daemon after the upgrade. Their question about whether plugin retry collects generation 2 is still open.

This is a field report from zammad-dev (A1X code-index-dev, 2026-10-08). Their setup: a Zammad 7.0.2 fork with Rails and Vue/TS, 10,009 files, **build 0.34.0 (014ed555)**, Ruby owned by de.h-dv.ruby 0.8.1 (plugin generation 4), and a 700 MB index DB. **Not yet reproduced here; a debugging lane is on it.** ## Observed 1. **The hang.** Daemon pid 803642 ran a FULL resolve (pass_seq 1) for more than 40 min at about 100% CPU and 1.7 GB RSS. - `last_completed_step` stayed at `access_ref_to_member_access` for about 37 min while the heartbeat was alive. - It then exited: `fatal error: startup generation 4 committed; resolution remains pending: sqlite error: interrupted (exit code 1)`. 2. **Empty answers during the pass.** `project_overview` reported `refs_resolved 0` (81,059 earlier in the same run), 0% for every language and no top symbols. `index_state` did say "floors". Even so, `find_callers`, `find_references` and `change_impact` answered from nothing for about 40 min. 3. **Watcher starvation.** Edits were not picked up for 40+ min: `index_freshness` showed `lagging` with 154 added and 39 changed paths. Line numbers in `line_text` went stale and new files were missing from `list_files`. 4. **After a restart** (pid 813834) the daemon is steady: `metadata_agrees`, 0 lagging. Generation 2 is still "awaiting collection". ## Likely trigger (unconfirmed) This was probably the first daemon to open the index after the upgrade from 0.33.1. Migration v79 (2a2e2629) runs one Full Ruby re-resolve on upgrade. For comparison, a cold CLI index of a local Zammad 7.0.2 copy with the 0.34.0 code takes 67 s in total, with `lexical_constant` at 0.86 s. So the difference is in the upgrade, daemon, package-owner or generations path. ## Hypotheses - (a) The step after `access_ref_to_member_access` is pathological on this path. - (b) An interrupt in the STARTUP resolve, after the generation commit, is treated as fatal rather than as preempted or cancelled (compare b13303f3 and 7c69a192). - (c) Watcher preemption loops, so the pass never completes. - (d) A Full pass clears bindings where readers can see them. ## Asked of the reporter `daemon.log` for pid 803642, and whether it was the first daemon after the upgrade. Their question about whether `plugin retry` collects generation 2 is still open.
Author
Member

The reporter's daemon.log evidence, summarised. It narrows the picture.

The trigger was the first open after the upgrade.

  • 13:47:53 applying schema migration version=79 name="lexical_constants".
  • 13:49:11 migration applied … elapsed_ms=78698.
  • 13:49:17 index opened; schema is current elapsed_ms=84694.
  • So the v79 migration finished in 79 s and is not the hang.

The installed package profile was refused, four times during the migration and three times after it: declared profile not readable by this build; the builtin answer stands lang="de.h-dv.ruby/ruby" refusal=profile.(fields) = "17 of 18". de.h-dv.ruby 0.8.1 declares 17 profile fields and 0.34.0 expects 18, presumably the new lexical_constant_lookup trait. This may be a compatibility defect of its own: an older package's profile ought to stay readable, with the new trait defaulted.

The hang is the post-migration startup resolve.

  • One stage logged resolve stage: still advancing 1,079 times, the last at 14:35:14 with elapsed_s=2694.
  • There is no stage-complete line.
  • The last completed step was access_ref_to_member_access.

The fatal exit is probably a cancel or shutdown interrupt treated as fatal.

  • 14:33:16: an older MCP client (pid 550882) exited on SIGTERM.
  • 14:35:16.9: daemon exiting on a fatal error … startup generation 4 committed; resolution remains pending: sqlite error: interrupted.

The restart has a question attached.

  • 14:40 the new daemon opened in 4.5 s, and its reconcile finished with reference resolution … scope=Skipped.
  • Whether the work the dead pass owed was completed or dropped still needs to be checked.

So this issue now covers up to three defects:

  1. a pathological resolve stage after access_ref_to_member_access on this path;
  2. a cancel interrupt in the startup-resolve path treated as fatal;
  3. possibly the package-profile refusal.

#336 data from the reporter (resolved_by):

  • message.name and last.name (assistant.rb): tier3_import_boost.
  • record.name (nextcloud.rb): tier3_import_boost.
  • record.name inside Space#name itself (docs.rb:70): tier2_same_file.
  • All four are builtin_only.

This last item is recorded on #336.

The reporter's `daemon.log` evidence, summarised. It narrows the picture. **The trigger was the first open after the upgrade.** - 13:47:53 `applying schema migration version=79 name="lexical_constants"`. - 13:49:11 `migration applied … elapsed_ms=78698`. - 13:49:17 `index opened; schema is current elapsed_ms=84694`. - So **the v79 migration finished in 79 s and is not the hang.** **The installed package profile was refused**, four times during the migration and three times after it: `declared profile not readable by this build; the builtin answer stands lang="de.h-dv.ruby/ruby" refusal=profile.(fields) = "17 of 18"`. de.h-dv.ruby 0.8.1 declares 17 profile fields and 0.34.0 expects 18, presumably the new `lexical_constant_lookup` trait. This may be a compatibility defect of its own: an older package's profile ought to stay readable, with the new trait defaulted. **The hang is the post-migration startup resolve.** - One stage logged `resolve stage: still advancing` 1,079 times, the last at 14:35:14 with elapsed_s=2694. - There is no stage-complete line. - The last completed step was `access_ref_to_member_access`. **The fatal exit is probably a cancel or shutdown interrupt treated as fatal.** - 14:33:16: an older MCP client (pid 550882) exited on SIGTERM. - 14:35:16.9: `daemon exiting on a fatal error … startup generation 4 committed; resolution remains pending: sqlite error: interrupted`. **The restart has a question attached.** - 14:40 the new daemon opened in 4.5 s, and its reconcile finished with `reference resolution … scope=Skipped`. - Whether the work the dead pass owed was completed or dropped still needs to be checked. So this issue now covers up to three defects: 1. a pathological resolve stage after `access_ref_to_member_access` on this path; 2. a cancel interrupt in the startup-resolve path treated as fatal; 3. possibly the package-profile refusal. **#336 data from the reporter (resolved_by):** - `message.name` and `last.name` (`assistant.rb`): `tier3_import_boost`. - `record.name` (`nextcloud.rb`): `tier3_import_boost`. - `record.name` inside `Space#name` itself (`docs.rb:70`): `tier2_same_file`. - All four are `builtin_only`. This last item is recorded on #336.
Author
Member

Root-caused and fixed locally: commits 68bca7b5 and 7a5f4d5f. They are not pushed yet.

Reproduction

  • Copied the local Zammad 7.0.2 tree and indexed it with 0.33.1, with de.h-dv.ruby owning Ruby.
  • Started the 0.34.0 daemon on that index.
  • The v79 migration finished in 24.6 s.
  • The upgrade changed the host identity, so the startup activation rebuilt the tree as a new generation and promoted it.
  • The resolve after that promotion hung in resolution_influence, the step after access_ref_to_member_access, for 1,370,778 ms: step 3 took 59 s and the anchor arm about 22 min. The same step inside the migration's own resolve took 61–82 ms.
  • During the hang, refs_resolved = 0 and last_completed_step access_ref_to_member_access, as in the field.
  • A SIGTERM mid-pass reproduced the fatal exit exactly (exit 1).

(a) Root cause of the hang

  • build::request wrote the pipeline-built generation through activation::write_generation, which defaults builtin_equivalent = 0. Only activation::record, on the ordinary pass, ever set it to 1.
  • So the promoted generation counted as third party. That changed the grant fingerprint, which sends influence::apply_general over every ref, unscoped (646k refs). Its anchor arm compares string prefixes and cannot use an index.
  • This is NOT new in 0.34.0: any release upgrade of a package-owned project hits it.

Fix (68bca7b5): build::request records builtin-equivalence from the live PackageHost, the same way the ordinary pass does.

  • After the fix: resolution_influence takes 1,070 ms on the same copy.
  • Test: first_party_influence::a_pipeline_built_generation_keeps_the_first_party_exemption.
  • Mutation (removing the call) goes RED with …must be recorded builtin-equivalent: [0]. I re-ran it and confirmed.

(b) The fatal "interrupted" exit

  • A SIGTERM during the startup activation's promotion resolve was classified StartupFailure::Fatal.
  • Before the watcher is armed, only the SIGINT/SIGTERM handler can interrupt that pass. Takeover uses SIGKILL and idle timeout cannot reach it.

Fix (7a5f4d5f): a requested stop routes to the clean-shutdown path, which reports that the pending work is durable.

  • Test: a_requested_stop_never_makes_a_startup_failure_fatal.
  • Both of its mutations go RED (re-run and confirmed).
  • The call-site wiring is covered by a live SIGTERM run on the Zammad copy (exit 1 before, exit 0 after), not by a test.

(c) and (d)

  • (c) rejected: the promotion resolve checks only for cancellation, never for watcher preemption. The watcher was not armed yet.
  • (d) separate design issue, filed as a follow-up: the promotion commits the pointer switch before resolving, so readers see an unresolved generation for as long as that resolve runs.

The watcher starvation and refs_resolved = 0 are symptoms of the same long pass.

Owed work after the crash

In the reproduction the restart ran the owed Full resolve (55 s) and the pending tables ended empty. The field restart logged scope=Skipped, so I have asked the reporter for its resolve lines.

Follow-ups (filed separately)

  • The package profile "17 of 18" compatibility defect.
  • A third-party reachability_anchor package still pays the anchor arm per ref; it should run once per distinct qualifier.
  • Readers see an unresolved generation between the pointer switch and the resolve.
**Root-caused and fixed locally:** commits 68bca7b5 and 7a5f4d5f. They are not pushed yet. ## Reproduction - Copied the local Zammad 7.0.2 tree and indexed it with 0.33.1, with de.h-dv.ruby owning Ruby. - Started the 0.34.0 daemon on that index. - The v79 migration finished in 24.6 s. - The upgrade changed the host identity, so the startup activation rebuilt the tree as a new generation and promoted it. - The resolve after that promotion hung in `resolution_influence`, the step after `access_ref_to_member_access`, for **1,370,778 ms**: step 3 took 59 s and the anchor arm about 22 min. The same step inside the migration's own resolve took 61–82 ms. - During the hang, `refs_resolved = 0` and `last_completed_step access_ref_to_member_access`, as in the field. - A SIGTERM mid-pass reproduced the fatal exit exactly (exit 1). ## (a) Root cause of the hang - `build::request` wrote the pipeline-built generation through `activation::write_generation`, which defaults `builtin_equivalent = 0`. Only `activation::record`, on the ordinary pass, ever set it to 1. - So the promoted generation counted as third party. That changed the grant fingerprint, which sends `influence::apply_general` over every ref, unscoped (646k refs). Its anchor arm compares string prefixes and cannot use an index. - This is NOT new in 0.34.0: any release upgrade of a package-owned project hits it. **Fix (68bca7b5):** `build::request` records builtin-equivalence from the live `PackageHost`, the same way the ordinary pass does. - After the fix: `resolution_influence` takes 1,070 ms on the same copy. - Test: `first_party_influence::a_pipeline_built_generation_keeps_the_first_party_exemption`. - Mutation (removing the call) goes RED with `…must be recorded builtin-equivalent: [0]`. I re-ran it and confirmed. ## (b) The fatal "interrupted" exit - A SIGTERM during the startup activation's promotion resolve was classified `StartupFailure::Fatal`. - Before the watcher is armed, only the SIGINT/SIGTERM handler can interrupt that pass. Takeover uses SIGKILL and idle timeout cannot reach it. **Fix (7a5f4d5f):** a requested stop routes to the clean-shutdown path, which reports that the pending work is durable. - Test: `a_requested_stop_never_makes_a_startup_failure_fatal`. - Both of its mutations go RED (re-run and confirmed). - The call-site wiring is covered by a live SIGTERM run on the Zammad copy (exit 1 before, exit 0 after), not by a test. ## (c) and (d) - **(c) rejected:** the promotion resolve checks only for cancellation, never for watcher preemption. The watcher was not armed yet. - **(d) separate design issue, filed as a follow-up:** the promotion commits the pointer switch before resolving, so readers see an unresolved generation for as long as that resolve runs. **The watcher starvation and `refs_resolved = 0` are symptoms of the same long pass.** ## Owed work after the crash In the reproduction the restart ran the owed Full resolve (55 s) and the pending tables ended empty. The field restart logged `scope=Skipped`, so I have asked the reporter for its resolve lines. ## Follow-ups (filed separately) - The package profile "17 of 18" compatibility defect. - A third-party `reachability_anchor` package still pays the anchor arm per ref; it should run once per distinct qualifier. - Readers see an unresolved generation between the pointer switch and the resolve.
Author
Member

The reporter answered the two open questions.

1. The kill was a SIGTERM. daemon.log line 6681, 14:35:16.793:

INFO … pid=803642 shutting down: termination signal received (SIGINT/SIGTERM, …)

The fatal sqlite error: interrupted line follows 115 ms later. That matches the reproduction and is fixed by 7a5f4d5f.

2. The owed resolve was deferred, not lost. The restarted daemon (813834) logged at 14:40:48:

index stage complete: reference resolution elapsed_ms=0 resolved=0 invalidated=0 scope=Skipped

So it did not run the owed Full resolve at startup. The reproduction did run it (55 s). The work was caught up only by later incremental passes:

  • 14:53:28: a small pass;
  • 15:00:40: tier1_unique=12233 …, the catch-up;
  • 15:02:59: a small pass.

The current daemon reports refs_resolved 82,571 against 81,059 before the upgrade, with Ruby at 11.8% both times, so nothing appears lost.

Still to explain. Why did the startup reconcile choose scope=Skipped while m0007 pending work from the killed promotion existed? Between 14:40 and 15:00, readers saw floors with no stated cause. This is NOT addressed by #342. It needs its own reproduction, starting from the exact killed state; the debug lane's snapshot of a killed DB was taken before the fix.

generations_awaiting_collection is now 0: generation 2 was collected at a later daemon start without operator action.

The reporter answered the two open questions. **1. The kill was a SIGTERM.** `daemon.log` line 6681, 14:35:16.793: `INFO … pid=803642 shutting down: termination signal received (SIGINT/SIGTERM, …)` The fatal `sqlite error: interrupted` line follows 115 ms later. That matches the reproduction and is fixed by 7a5f4d5f. **2. The owed resolve was deferred, not lost.** The restarted daemon (813834) logged at 14:40:48: `index stage complete: reference resolution elapsed_ms=0 resolved=0 invalidated=0 scope=Skipped` So it did not run the owed Full resolve at startup. The reproduction did run it (55 s). The work was caught up only by later incremental passes: - 14:53:28: a small pass; - 15:00:40: `tier1_unique=12233 …`, the catch-up; - 15:02:59: a small pass. The current daemon reports `refs_resolved` 82,571 against 81,059 before the upgrade, with Ruby at 11.8% both times, so nothing appears lost. **Still to explain.** Why did the startup reconcile choose `scope=Skipped` while m0007 pending work from the killed promotion existed? Between 14:40 and 15:00, readers saw floors with no stated cause. This is NOT addressed by #342. It needs its own reproduction, starting from the exact killed state; the debug lane's snapshot of a killed DB was taken before the fix. `generations_awaiting_collection` is now 0: generation 2 was collected at a later daemon start without operator action.
Author
Member

Released in v0.34.1 (https://git.h-dv.de/h-dv/code-index/releases/tag/v0.34.1), via PR #342:

  • 68bca7b5: a generation built by the activation pipeline keeps its first-party exemption, so resolution_influence takes about 1 s instead of tens of minutes;
  • 7a5f4d5f: a requested stop during the startup activation is a clean shutdown, not exit 1.

I'm closing this issue for the hang and the fatal exit. Items still open elsewhere:

  • #339: a package profile stored by an older host can't be read by a newer one;
  • #340: the influence anchor arm and step 3 cost;
  • #341: readers see an unresolved generation during promotion.

One item has no issue yet: the restarted daemon chose scope=Skipped while owed work was pending (see the comment above). I'll file it separately so it gets its own reproduction.

Released in v0.34.1 (https://git.h-dv.de/h-dv/code-index/releases/tag/v0.34.1), via PR #342: - **68bca7b5:** a generation built by the activation pipeline keeps its first-party exemption, so `resolution_influence` takes about 1 s instead of tens of minutes; - **7a5f4d5f:** a requested stop during the startup activation is a clean shutdown, not exit 1. I'm closing this issue for the hang and the fatal exit. Items still open elsewhere: - #339: a package profile stored by an older host can't be read by a newer one; - #340: the influence anchor arm and step 3 cost; - #341: readers see an unresolved generation during promotion. One item has no issue yet: the restarted daemon chose `scope=Skipped` while owed work was pending (see the comment above). I'll file it separately so it gets its own reproduction.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#334
No description provided.