Repository navigation
fix: eliminate redundant incident-candidate DB fetches in new-issue intake #SUPERLOG - #537
Open
superlog-app[bot] wants to merge 1 commit into
Open
superlog-app[bot] wants to merge 1 commit into
superlog-app[bot] wants to merge 1 commit into
Conversation
…ntake #SUPERLOG Delivery-Id: 67d338ea005db7279e254af5fc2197d7009362a5322158c37de0ddda94b9d80e Delivery-Base: main
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New-issue transitions were averaging ~95 s execution time (40% above the ~68 s baseline) for the full observed hour. The elevated duration is caused by two compounding defects in the new-issue intake path, both of which are addressed here.
Root Cause
1. Redundant DB round-trips in
findMatchingIncidentFor every new issue that falls through heuristic matching,
findMatchingIncidentwas callingfindOpenIncidentCandidates(filterService: false)andloadLinkedIncidentIssuestwice — once insidefindSameTraceMatchingIncidentand again insidefindLlmMatchingIncident. The comment infindSameTraceMatchingIncidenteven acknowledged this: "so no-trace intakes don't do them here only for the LLM path to immediately repeat them" — but only the no-trace early-exit was implemented; the duplicate fetch for trace-having issues was never eliminated.Under the
perProjectkeyed mutex, each additional second of per-transition latency serializes for every concurrent new issue in the same project, multiplying the wall-clock delay across any project experiencing an issue storm.2. No per-incident row limit in
loadLinkedIncidentIssuesloadLinkedIncidentIssueshad noORDER BYand noLIMIT. An incident with thousands of linked issues (observed at 14,000+ during a Railway backfill event) caused the query to return the entire result set on every call. Combined with the redundant calls above, that same enormous result set was fetched and transferred 2–3 times per transition, even thoughbuildGroupingCandidateonly uses the 5 most-recent issues per incident.Remediation
apps/worker/src/incidents/intake.ts—findMatchingIncidentnow loads broad candidates + linked issues once and passes them into both the same-trace check (now inlined as a call to the purefindSameTraceIncidentMatch) andfindLlmMatchingIncident. ThefindSameTraceMatchingIncidentasync helper is unchanged because it is still called from the recurrence path insideensureIncidentForIssueAttempt.apps/worker/src/issues/repository.ts—loadLinkedIncidentIssuesnow orders results by(incident_id ASC, last_seen DESC)and caps the returned rows toLINKED_ISSUES_PER_INCIDENT_CAP = 5per incident in TypeScript. This matches theGROUPING_CANDIDATE_ISSUE_LIMIT = 5indomain.tsand prevents a single high-issue-count incident from returning an unbounded result set to callers.Expected Impact
findOpenIncidentCandidates+ oneloadLinkedIncidentIssuesDB round-trip per new-issue transition that goes through the LLM path (the common case for new issues)loadLinkedIncidentIssuesto return 14,000+ rows on every callIncident: c9757c47-db13-4139-9b81-674f321ed80d
Was this PR helpful? Leave feedback — goes straight to the Superlog team.