Field (zammad-dev, 0.34.0): first daemon after upgrade ran a full resolve for 40+ min, then exited "sqlite error: interrupted"; refs_resolved read 0 and the watcher starved meanwhile #334
Labels
No labels
code-review
correctness
dos
performance
security
severity/high
severity/low
severity/medium
tech-debt
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
h-dv/code-index#334
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This is a field report from zammad-dev (A1X code-index-dev, 2026-10-08). Their setup: a Zammad 7.0.2 fork with Rails and Vue/TS, 10,009 files, build 0.34.0 (
014ed555), Ruby owned by de.h-dv.ruby 0.8.1 (plugin generation 4), and a 700 MB index DB. Not yet reproduced here; a debugging lane is on it.Observed
last_completed_stepstayed ataccess_ref_to_member_accessfor about 37 min while the heartbeat was alive.fatal error: startup generation 4 committed; resolution remains pending: sqlite error: interrupted (exit code 1).project_overviewreportedrefs_resolved 0(81,059 earlier in the same run), 0% for every language and no top symbols.index_statedid say "floors". Even so,find_callers,find_referencesandchange_impactanswered from nothing for about 40 min.index_freshnessshowedlaggingwith 154 added and 39 changed paths. Line numbers inline_textwent stale and new files were missing fromlist_files.metadata_agrees, 0 lagging. Generation 2 is still "awaiting collection".Likely trigger (unconfirmed)
This was probably the first daemon to open the index after the upgrade from 0.33.1. Migration v79 (
2a2e2629) runs one Full Ruby re-resolve on upgrade. For comparison, a cold CLI index of a local Zammad 7.0.2 copy with the 0.34.0 code takes 67 s in total, withlexical_constantat 0.86 s. So the difference is in the upgrade, daemon, package-owner or generations path.Hypotheses
access_ref_to_member_accessis pathological on this path.b13303f3and7c69a192).Asked of the reporter
daemon.logfor pid 803642, and whether it was the first daemon after the upgrade. Their question about whetherplugin retrycollects generation 2 is still open.The reporter's
daemon.logevidence, summarised. It narrows the picture.The trigger was the first open after the upgrade.
applying schema migration version=79 name="lexical_constants".migration applied … elapsed_ms=78698.index opened; schema is current elapsed_ms=84694.The installed package profile was refused, four times during the migration and three times after it:
declared profile not readable by this build; the builtin answer stands lang="de.h-dv.ruby/ruby" refusal=profile.(fields) = "17 of 18". de.h-dv.ruby 0.8.1 declares 17 profile fields and 0.34.0 expects 18, presumably the newlexical_constant_lookuptrait. This may be a compatibility defect of its own: an older package's profile ought to stay readable, with the new trait defaulted.The hang is the post-migration startup resolve.
resolve stage: still advancing1,079 times, the last at 14:35:14 with elapsed_s=2694.access_ref_to_member_access.The fatal exit is probably a cancel or shutdown interrupt treated as fatal.
daemon exiting on a fatal error … startup generation 4 committed; resolution remains pending: sqlite error: interrupted.The restart has a question attached.
reference resolution … scope=Skipped.So this issue now covers up to three defects:
access_ref_to_member_accesson this path;#336 data from the reporter (resolved_by):
message.nameandlast.name(assistant.rb):tier3_import_boost.record.name(nextcloud.rb):tier3_import_boost.record.nameinsideSpace#nameitself (docs.rb:70):tier2_same_file.builtin_only.This last item is recorded on #336.
Root-caused and fixed locally: commits
68bca7b5and7a5f4d5f. They are not pushed yet.Reproduction
resolution_influence, the step afteraccess_ref_to_member_access, for 1,370,778 ms: step 3 took 59 s and the anchor arm about 22 min. The same step inside the migration's own resolve took 61–82 ms.refs_resolved = 0andlast_completed_step access_ref_to_member_access, as in the field.(a) Root cause of the hang
build::requestwrote the pipeline-built generation throughactivation::write_generation, which defaultsbuiltin_equivalent = 0. Onlyactivation::record, on the ordinary pass, ever set it to 1.influence::apply_generalover every ref, unscoped (646k refs). Its anchor arm compares string prefixes and cannot use an index.Fix (
68bca7b5):build::requestrecords builtin-equivalence from the livePackageHost, the same way the ordinary pass does.resolution_influencetakes 1,070 ms on the same copy.first_party_influence::a_pipeline_built_generation_keeps_the_first_party_exemption.…must be recorded builtin-equivalent: [0]. I re-ran it and confirmed.(b) The fatal "interrupted" exit
StartupFailure::Fatal.Fix (
7a5f4d5f): a requested stop routes to the clean-shutdown path, which reports that the pending work is durable.a_requested_stop_never_makes_a_startup_failure_fatal.(c) and (d)
The watcher starvation and
refs_resolved = 0are symptoms of the same long pass.Owed work after the crash
In the reproduction the restart ran the owed Full resolve (55 s) and the pending tables ended empty. The field restart logged
scope=Skipped, so I have asked the reporter for its resolve lines.Follow-ups (filed separately)
reachability_anchorpackage still pays the anchor arm per ref; it should run once per distinct qualifier.The reporter answered the two open questions.
1. The kill was a SIGTERM.
daemon.logline 6681, 14:35:16.793:INFO … pid=803642 shutting down: termination signal received (SIGINT/SIGTERM, …)The fatal
sqlite error: interruptedline follows 115 ms later. That matches the reproduction and is fixed by7a5f4d5f.2. The owed resolve was deferred, not lost. The restarted daemon (813834) logged at 14:40:48:
index stage complete: reference resolution elapsed_ms=0 resolved=0 invalidated=0 scope=SkippedSo it did not run the owed Full resolve at startup. The reproduction did run it (55 s). The work was caught up only by later incremental passes:
tier1_unique=12233 …, the catch-up;The current daemon reports
refs_resolved82,571 against 81,059 before the upgrade, with Ruby at 11.8% both times, so nothing appears lost.Still to explain. Why did the startup reconcile choose
scope=Skippedwhile m0007 pending work from the killed promotion existed? Between 14:40 and 15:00, readers saw floors with no stated cause. This is NOT addressed by #342. It needs its own reproduction, starting from the exact killed state; the debug lane's snapshot of a killed DB was taken before the fix.generations_awaiting_collectionis now 0: generation 2 was collected at a later daemon start without operator action.Released in v0.34.1 (https://git.h-dv.de/h-dv/code-index/releases/tag/v0.34.1), via PR #342:
68bca7b5: a generation built by the activation pipeline keeps its first-party exemption, soresolution_influencetakes about 1 s instead of tens of minutes;7a5f4d5f: a requested stop during the startup activation is a clean shutdown, not exit 1.I'm closing this issue for the hang and the fatal exit. Items still open elsewhere:
One item has no issue yet: the restarted daemon chose
scope=Skippedwhile owed work was pending (see the comment above). I'll file it separately so it gets its own reproduction.plugin statusreports a live daemon build as "interrupted mid-build — run plugin retry" #346