Repository navigation
fix(live-alerts): retry a failed Twitch/Kick poll once and log a run of failures once (5.3) - #375
Ferdinand99 wants to merge 3 commits into
Conversation
…of failures once
A single network blip ("fetch failed") no longer logs an error. A failed
request is retried once after a short delay for network errors, 5xx and 429
(honouring Retry-After); the retry sits inside the request, so the batch
isolation only sees an error after the retry failed too. Failed polls are
counted per platform: one warning after 3 in a row, naming the cause code
(e.g. ECONNRESET), and one line when polling works again. Programming errors
and 4xx answers are still logged as errors.
There was a problem hiding this comment.
Graphify reviewed this change.
Code graph out of date: Health delta baseline: last indexed commit 2431870, 18 commit(s) behind this PR's base. Impact and hotspot numbers below may be wrong.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.
Not checked: tests were not run; no formal proof of the changed code.
Graphify review — findings
Adds a shared withRetry so Twitch and Kick requests retry once after a network error, 5xx or 429 (honouring Retry-After, capped at 15s), while other 4xx answers fail immediately. Replaces the per-poll error log in both tick loops with createPollHealth, which stays quiet on transient failures until three polls in a row fail, logs once when polling recovers, and still logs non-transient errors such as bad credentials every time. Kick's getChannels 401 and token failures now go through httpError and carry a status, the field batch isolation reads when deciding whether Kick rejected a slug. Twitch's helix takes retry: false for lookups that only decorate an announcement.
Worth a look
- Auth/token failures now carry an HTTP status, which batch isolation may read as a rejected-slug answer —
src/modules/kickAlerts.js:104· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 57 functions depend on the 52 functions this change touches.
Health — this change adds coupling hotspots:
- worse:
tick()— 1 callers, 20 callees - worse:
tick()— 1 callers, 18 callees
Verification — 57 functions in the blast radius were not formally verified this run (proofs are advisory here).
Health delta baseline: last indexed commit 2431870, 18 commit(s) behind this PR's base.
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 57 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
4 of 681 test file(s) selected (1%) via static blast radius.
test/configUndo.test.js— impacttest/kickAlerts.test.js— impacttest/liveAlertsPollErrors.test.js— impact, changed-testtest/twitchAlerts.test.js— impact
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Return the tick promise to the scheduler so a slow tick is not started twice. Read Ratelimit-Reset only for a 429. Count only the connection and timeout codes of undici as network errors. Refresh a rejected token and retry once at once. Log an error that is not transient the first time, when it changes and then once an hour, with its stack, and remind hourly during a long network outage. Mark token endpoint errors so batch isolation cannot read a refused token as a rejected value (no more search of ~120 token requests per poll). A network failure of the canary request now fails the poll as that failure and keeps the canary, instead of rethrowing the earlier 400.
… the error throttle by kind
A TypeError('fetch failed') with a cause is now a network error only when the
cause code is a connection or timeout code; UND_ERR_INVALID_ARG and similar
stay errors. The hourly throttle for errors that are not transient keys on
name, status and the message without numbers, so alternating errors or a
changing value in the message are not logged on every poll.
What
The 5.3 version of #374, on top of the batch isolation in #367 (
lib/batchIsolation.js). Base isfix/53-audit-leftovers; change the base tov5.3once #367 is in.A transient network error ("fetch failed") in the Twitch or Kick poll no longer logs an error.
helix(),getChannels()): a network error, 5xx or 429 is retried once after 3 s (Retry-Afterhonoured, up to 15 s).fetchIsolatedonly sees an error once the retry failed too, so its semantics are unchanged: 400/404/422 stay value faults, network errors and 5xx/429 are not value faults and fail the poll (unless the canary rule applies), and nothing is quarantined because of them.log.warnafter 3 in a row, one when polling works again. The line names the cause code fromerr.cause(fetch failed (ECONNRESET)), no URL or credential. A poll counts as working whenfetchLiveStreams/fetchLiveChannelsreturns, including the inconclusive case.log.errorstays for programming errors and for 4xx answers that need a fix (bad credentials, a request refused as a whole).Shared helper:
src/modules/lib/pollErrors.js, the same file as in #374.Tests
test/liveAlertsPollErrors.test.js, for Twitch and Kick: a blip retried silently; 3 failed polls give one warning with the cause code, and one recovery line; 503 and 429 retried; a programming error and a 403 still logged as errors, not retried. On top of that, in combination with the isolation: a blip plus a rejected value in the same poll is retried, isolated, and the other server's stream is announced; a network error while values are being isolated fails the poll as a network failure and quarantines nothing; a failed Twitch user lookup is neither retried nor logged.Notes
mainintov5.3later will report an add/add conflict onliveAlertsPollErrors.test.jsif fix(live-alerts): retry a failed Twitch/Kick poll once and log a run of failures once #374 lands first (pollErrors.jsis identical in both); keep this branch's version of the test.Review round 2
Same changes as #374 (tick promise returned to the scheduler, 401 refresh and retry, hourly throttle for repeated errors with a stack, precise
UND_ERR_*codes,Ratelimit-Resetonly for 429), plus for the isolation:fetchIsolatedreads as a rejected value (tokenError). Before, bad credentials started a search of about 120 token requests a poll and the poll counted as working.batchIsolation.jsimportsisTransientfrompollErrors.jsfor this.