influence anchor arm runs per reference with an unindexable prefix compare; a third-party reachability_anchor package pays ~20 min on a 10k-file tree whenever the grant moves #340

Closed
opened 2026-10-08 17:37:18 +02:00 by buildagent · 1 comment
Member

This was found while working on #334.

Measurement. On a 10k-file Rails tree (Zammad 7.0.2, 646k refs), the general influence pass (influence::apply_general) takes about 23 min when it runs unscoped:

  • the anchor arm (step 4) takes about 22 min;
  • step 3 ("COMPETED") takes 59 s.

Why the anchor arm is slow. It compares each qualified ref's qualifier against every file key with a prefix test that cannot use an index. That is 150k qualified refs, even though there are only about 9.6k distinct qualifiers.

Scope. #334's fix (68bca7b5) removes the first-party trigger. A genuinely third-party package that declares reachability_anchor still pays this cost on every pass where the grant fingerprint moves.

Proposed fix. Evaluate the anchor arm once per DISTINCT qualifier and join the result back to the refs, the way path_evidence_match_sql already does. The estimate is about 15x faster. Measure it with cost_attribution and the #275 Svelte cost test.

Related: step 3 ("COMPETED"). It computes the same correlated competitor subquery twice per ref, once in SET and once in WHERE EXISTS. #275's cost attribution measured it at 86% of the Svelte package's extra SQLite work. An UPDATE … FROM (SELECT ref_id, MIN(component) … GROUP BY ref_id) would remove the duplicate.

Do not move any cost baseline to absorb either cost; reduce the cost first.

This was found while working on #334. **Measurement.** On a 10k-file Rails tree (Zammad 7.0.2, 646k refs), the general influence pass (`influence::apply_general`) takes about 23 min when it runs unscoped: - the anchor arm (step 4) takes about 22 min; - step 3 ("COMPETED") takes 59 s. **Why the anchor arm is slow.** It compares each qualified ref's qualifier against every file key with a prefix test that cannot use an index. That is 150k qualified refs, even though there are only about 9.6k distinct qualifiers. **Scope.** #334's fix (68bca7b5) removes the first-party trigger. A genuinely third-party package that declares `reachability_anchor` still pays this cost on every pass where the grant fingerprint moves. **Proposed fix.** Evaluate the anchor arm once per DISTINCT qualifier and join the result back to the refs, the way `path_evidence_match_sql` already does. The estimate is about 15x faster. Measure it with `cost_attribution` and the #275 Svelte cost test. **Related: step 3 ("COMPETED").** It computes the same correlated competitor subquery twice per ref, once in `SET` and once in `WHERE EXISTS`. #275's cost attribution measured it at 86% of the Svelte package's extra SQLite work. An `UPDATE … FROM (SELECT ref_id, MIN(component) … GROUP BY ref_id)` would remove the duplicate. Do not move any cost baseline to absorb either cost; reduce the cost first.
Author
Member

Released in v0.35.0 (https://git.h-dv.de/h-dv/code-index/releases/tag/v0.35.0), via PR #351.

What changed:

  • Step 3 now computes the competitors once per pass instead of once per ref, folded by name.
  • The anchor arm now matches each distinct qualifier's prefixes against the anchored roots by key, instead of scanning.

Results: identical. A differential test against an independently written oracle of the old pass checks this.

Cost:

  • Step 3 on Zammad with Vue: 516M → 35M vm_step.
  • The Ruby package leg's full scans dropped below their pre-#340 level.

The bridge_arm finding is tracked separately in #349.

Released in v0.35.0 (https://git.h-dv.de/h-dv/code-index/releases/tag/v0.35.0), via PR #351. **What changed:** - Step 3 now computes the competitors once per pass instead of once per ref, folded by name. - The anchor arm now matches each distinct qualifier's prefixes against the anchored roots by key, instead of scanning. **Results:** identical. A differential test against an independently written oracle of the old pass checks this. **Cost:** - Step 3 on Zammad with Vue: 516M → 35M vm_step. - The Ruby package leg's full scans dropped below their pre-#340 level. The `bridge_arm` finding is tracked separately in #349.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
h-dv/code-index#340
No description provided.