Skip to content

fix(live-alerts): retry a failed Twitch/Kick poll once and log a run of failures once (5.3) - #375

Open
Ferdinand99 wants to merge 3 commits into
fix/53-audit-leftoversfrom
fix/live-alerts-transient-errors-53
Open

Ferdinand99 wants to merge 3 commits into
fix/53-audit-leftoversfrom
fix/live-alerts-transient-errors-53

Conversation

@Ferdinand99

@Ferdinand99 Ferdinand99 commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

What

The 5.3 version of #374, on top of the batch isolation in #367 (lib/batchIsolation.js). Base is fix/53-audit-leftovers; change the base to v5.3 once #367 is in.

A transient network error ("fetch failed") in the Twitch or Kick poll no longer logs an error.

  • The retry sits inside the request (helix(), getChannels()): a network error, 5xx or 429 is retried once after 3 s (Retry-After honoured, up to 15 s). fetchIsolated only sees an error once the retry failed too, so its semantics are unchanged: 400/404/422 stay value faults, network errors and 5xx/429 are not value faults and fail the poll (unless the canary rule applies), and nothing is quarantined because of them.
  • The Twitch user lookup (decoration only, failure already swallowed) is not retried.
  • Failed polls are counted per platform: one log.warn after 3 in a row, one when polling works again. The line names the cause code from err.cause (fetch failed (ECONNRESET)), no URL or credential. A poll counts as working when fetchLiveStreams / fetchLiveChannels returns, including the inconclusive case.
  • log.error stays for programming errors and for 4xx answers that need a fix (bad credentials, a request refused as a whole).
  • No change to which streams are announced.

Shared helper: src/modules/lib/pollErrors.js, the same file as in #374.

Tests

test/liveAlertsPollErrors.test.js, for Twitch and Kick: a blip retried silently; 3 failed polls give one warning with the cause code, and one recovery line; 503 and 429 retried; a programming error and a 403 still logged as errors, not retried. On top of that, in combination with the isolation: a blip plus a rejected value in the same poll is retried, isolated, and the other server's stream is announced; a network error while values are being isolated fails the poll as a network failure and quarantines nothing; a failed Twitch user lookup is neither retried nor logged.

Notes

Review round 2

Same changes as #374 (tick promise returned to the scheduler, 401 refresh and retry, hourly throttle for repeated errors with a stack, precise UND_ERR_* codes, Ratelimit-Reset only for 429), plus for the isolation:

  • A refused token-endpoint answer (400/403...) no longer carries a status that fetchIsolated reads as a rejected value (tokenError). Before, bad credentials started a search of about 120 token requests a poll and the poll counted as working.
  • A network failure of the canary request fails the poll as that network failure and keeps the canary (with nothing succeeded), or quarantines nothing and keeps the canary (when the platform flaps), instead of rethrowing the earlier 400 and dropping the canary. batchIsolation.js imports isTransient from pollErrors.js for this.

…of failures once

A single network blip ("fetch failed") no longer logs an error. A failed
request is retried once after a short delay for network errors, 5xx and 429
(honouring Retry-After); the retry sits inside the request, so the batch
isolation only sees an error after the retry failed too. Failed polls are
counted per platform: one warning after 3 in a row, naming the cause code
(e.g. ECONNRESET), and one line when polling works again. Programming errors
and 4xx answers are still logged as errors.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Code graph out of date: Health delta baseline: last indexed commit 2431870, 18 commit(s) behind this PR's base. Impact and hotspot numbers below may be wrong.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.

Not checked: tests were not run; no formal proof of the changed code.


Graphify review — findings

Adds a shared withRetry so Twitch and Kick requests retry once after a network error, 5xx or 429 (honouring Retry-After, capped at 15s), while other 4xx answers fail immediately. Replaces the per-poll error log in both tick loops with createPollHealth, which stays quiet on transient failures until three polls in a row fail, logs once when polling recovers, and still logs non-transient errors such as bad credentials every time. Kick's getChannels 401 and token failures now go through httpError and carry a status, the field batch isolation reads when deciding whether Kick rejected a slug. Twitch's helix takes retry: false for lookups that only decorate an announcement.

Worth a look

  • Auth/token failures now carry an HTTP status, which batch isolation may read as a rejected-slug answer — src/modules/kickAlerts.js:104 · Escalate · medium
    • NOT verified (no proof, no reproducing execution) — needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 57 functions depend on the 52 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: tick() — 1 callers, 20 callees
  • worse: tick() — 1 callers, 18 callees

Verification — 57 functions in the blast radius were not formally verified this run (proofs are advisory here).

Health delta baseline: last indexed commit 2431870, 18 commit(s) behind this PR's base.

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 57 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

4 of 681 test file(s) selected (1%) via static blast radius.

  • test/configUndo.test.js — impact
  • test/kickAlerts.test.js — impact
  • test/liveAlertsPollErrors.test.js — impact, changed-test
  • test/twitchAlerts.test.js — impact

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Return the tick promise to the scheduler so a slow tick is not started twice.
Read Ratelimit-Reset only for a 429. Count only the connection and timeout
codes of undici as network errors. Refresh a rejected token and retry once at
once. Log an error that is not transient the first time, when it changes and
then once an hour, with its stack, and remind hourly during a long network
outage. Mark token endpoint errors so batch isolation cannot read a refused
token as a rejected value (no more search of ~120 token requests per poll).
A network failure of the canary request now fails the poll as that failure
and keeps the canary, instead of rethrowing the earlier 400.
… the error throttle by kind

A TypeError('fetch failed') with a cause is now a network error only when the
cause code is a connection or timeout code; UND_ERR_INVALID_ARG and similar
stay errors. The hourly throttle for errors that are not transient keys on
name, status and the message without numbers, so alternating errors or a
changing value in the message are not logged on every poll.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant