Skip to content

fix: cap incident issue load in agent-run context to prevent unbounded IN clause #SUPERLOG - #538

Open
superlog-app[bot] wants to merge 1 commit into
mainfrom
superlog/fix-agent-run-context-issue-limit
Open

superlog-app[bot] wants to merge 1 commit into
mainfrom
superlog/fix-agent-run-context-issue-limit

Conversation

@superlog-app

@superlog-app superlog-app Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Agent PR delivery stalled completely after the worker restart at 22:05 UTC (incident knotty-manatee). Three independent metrics confirmed zero deliveries: superlog.prs.created, superlog.agent_pr_delivery.guard, and pr_observability.review_passes all flatlined, while agent investigations kept running at above-baseline rate. The worker emitted no api.github.com calls across 115 minutes of runtime.

Root Cause

In loadAgentRunContext, the two-step issue-loading query is unbounded:

// step 1 — fetches ALL issue IDs for the incident, no limit
const issueIds = await db
  .select({ issueId: schema.incidentIssues.issueId })
  .from(schema.incidentIssues)
  .where(eq(schema.incidentIssues.incidentId, incident.id));

// step 2 — passes every ID as a bind parameter
const issueRows = issueIds.length > 0
  ? await db.query.issues.findMany({
      where: inArray(schema.issues.id, issueIds.map(r => r.issueId)),
      ...
    })
  : [];

When an incident accumulates many linked issues, the inArray call grows proportionally. Production logs showed 13,854+ bind parameters (the log line was itself truncated at 30 kB, so the actual count may have been higher). Postgres rejects queries that exceed its 65,535 bind parameter limit with a hard "Failed query" error. The failure is caught in queue.ts and logged as "agent-run advance failed; the next sweep re-enqueues it" — but the next sweep calls the same unbounded loadAgentRunContext again, creating a permanent retry loop that prevents any advance from succeeding and blocks all PR delivery.

Remediation

Add a JOIN on issues to the first query so rows can be sorted by lastSeen DESC before applying a LIMIT 200. The second inArray call is now bounded at ≤ 200 parameters, well under the Postgres limit. The 200 most-recently-seen issues are always included, matching existing downstream caps (MAX_TRACE_ENRICHED_ISSUES = 50 in prompt-context.ts).

const ISSUE_CONTEXT_LIMIT = 200;
const issueIds = await db
  .select({ issueId: schema.incidentIssues.issueId })
  .from(schema.incidentIssues)
  .innerJoin(schema.issues, eq(schema.issues.id, schema.incidentIssues.issueId))
  .where(eq(schema.incidentIssues.incidentId, incident.id))
  .orderBy(desc(schema.issues.lastSeen))
  .limit(ISSUE_CONTEXT_LIMIT);

Typecheck: ✅ tsc --noEmit passes
Tests: ✅ test:agent-run-queue, test:agent-run-start pass

Related incident: c7813ad1-000c-4ee0-b76d-d1b7b3707d03


Was this PR helpful? Leave feedback — goes straight to the Superlog team.

…d IN clause #SUPERLOG

Delivery-Id: 4edfa37d3357f2b56f3f44a51f5c782bef5ace93a093263f2dd1ab0154c1511a
Delivery-Base: main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant