Skip to content

fix(live-alerts): retry a failed Twitch/Kick poll once and log a run of failures once - #374

Merged
Ferdinand99 merged 3 commits into
mainfrom
fix/live-alerts-transient-errors
Nov 16, 2026
Merged

Ferdinand99 merged 3 commits into
mainfrom
fix/live-alerts-transient-errors

Conversation

@Ferdinand99

@Ferdinand99 Ferdinand99 commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

What

Production 5.2.0 logs Twitch alerts — poll failed: fetch failed (and the same for Kick) every now and then. "fetch failed" is Node's generic network error, a single transient blip, and each one is a log.error.

  • A failed poll request is retried once after 3 s for network errors, 5xx and 429 (a Retry-After is honoured, up to 15 s). 4xx answers other than 429 are not retried.
  • A single failed poll is no longer logged. Consecutive failed polls are counted per platform: one log.warn after 3 in a row and one log.warn when polling works again. The line names the cause code from err.cause, e.g. fetch failed (ECONNRESET). It holds no URL or credential.
  • log.error stays for what needs a fix: programming errors (a TypeError in our code) and 4xx answers such as bad credentials.
  • Nothing changes in which streams are announced; a recovered poll announces as usual.

The shared part is src/modules/lib/pollErrors.js (classification, withRetry, createPollHealth), used by twitchAlerts.js and kickAlerts.js. tick() is exported from both for the tests, like on the 5.3 branch.

Tests

test/liveAlertsPollErrors.test.js, for both Twitch and Kick: a blip is retried silently and the stream is announced; 3 failed polls give one warning with the cause code, none while it stays down, one recovery line; 503 and 429 (Retry-After) are retried; a programming error and a 403 are logged as errors and not retried. Plus unit tests of the helper.

Notes

Review round 2

  • The every() callbacks now return the tick promise, so the scheduler's no-overlap guard applies (a tick that waits on a retry is not started twice).
  • A rejected token (401) is refreshed and the request retried once at once; a second 401 is an error.
  • Errors that are not transient (a TypeError, 401/403, a refused token) are logged with their stack the first time and when the message changes, then at most once an hour as "still failing after N polls"; one line when polling works again. A network outage that lasts also gets an hourly reminder.
  • UND_ERR_* codes: only the connection and timeout ones count as network errors. Ratelimit-Reset is only read for a 429.
  • Token endpoint errors carry no value-fault status (tokenError).

…of failures once

A single network blip ("fetch failed") no longer logs an error. A failed
request is retried once after a short delay for network errors, 5xx and 429
(honouring Retry-After). Failed polls are counted per platform: one warning
after 3 in a row, naming the cause code (e.g. ECONNRESET), and one line when
polling works again. Programming errors and 4xx answers are still logged as
errors.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

No issues found by static checks — no coupling regressions or blocking issues in the code graph. That is not the same as safe to merge: see what was not checked.

Not checked: tests were not run; no formal proof of the changed code.


Graphify review — findings

Adds a shared pollErrors helper so transient Twitch and Kick poll failures stop spamming error logs. withRetry repeats a request once after a short delay (3s by default) on network errors, 5xx and 429, honouring Retry-After up to 15s. Other 4xx answers are not retried and fail immediately. createPollHealth replaces the per-poll log.error in both pollers: transient failures log a single warning after 3 consecutive failed polls and one more when polling recovers. Non-transient errors, such as bad credentials or a bug in our own code, are still logged as errors on every poll. Failure messages now come from httpError, which carries the status and Retry-After, and describeError adds the underlying cause code (e.g. ECONNRESET) to Node's generic "fetch failed" and strips URLs so credentials stay out of the logs. Kick's tick is now exported for the new tests.

No blocking issues surfaced. 2 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 55 functions depend on the 50 functions this change touches.

Health — grade A; 2 existing hotspot(s) in the area this change touches (pre-existing, not introduced here):

  • tick() — 1 callers, 18 callees (high)
  • tick() — 1 callers, 16 callees (high)

Verification — 55 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 55 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

4 of 673 test file(s) selected (1%) via static blast radius.

  • test/configUndo.test.js — impact
  • test/kickAlerts.test.js — impact
  • test/liveAlertsPollErrors.test.js — impact, changed-test
  • test/twitchAlerts.test.js — impact

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Return the tick promise to the scheduler so a slow tick is not started twice.
Read Ratelimit-Reset only for a 429. Count only the connection and timeout
codes of undici as network errors. Refresh a rejected token and retry once at
once. Log an error that is not transient the first time, when it changes and
then once an hour, with its stack, and remind hourly during a long network
outage. Mark token endpoint errors so a refused token is never read as a
rejected value.
… the error throttle by kind

A TypeError('fetch failed') with a cause is now a network error only when the
cause code is a connection or timeout code; UND_ERR_INVALID_ARG and similar
stay errors. The hourly throttle for errors that are not transient keys on
name, status and the message without numbers, so alternating errors or a
changing value in the message are not logged on every poll.
@Ferdinand99
Ferdinand99 merged commit 51538e1 into main Nov 16, 2026
12 checks passed
@Ferdinand99
Ferdinand99 deleted the fix/live-alerts-transient-errors branch November 16, 2026 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant