[superlog-app]

Summary

Railway's OAuth server is currently returning {"error":"invalid_client"} for every refresh_token grant, indicating Superlog's Railway OAuth app credentials (RAILWAY_CLIENT_ID / RAILWAY_CLIENT_SECRET) have been rejected or revoked. This permanently stalls log/metric ingestion for all 12+ connected Railway installations and produces 12+ railway token refresh failed ERROR logs per minute (one per installation, every pull cycle).

Root cause

invalid_client is an application-level credential rejection — Railway refuses the OAuth app's client credentials, not the per-installation refresh token. Because all installations share the same client_id + client_secret, every installation fails identically on every tick. The puller previously treated invalid_client identically to invalid_grant (user-revoked per-installation access): it logged railway token refresh failed per installation and continued to the next one, hammering Railway's token endpoint 12+ times per minute with requests that will all fail.

Remediation

External action required (unblocks ingestion): Rotate the Railway OAuth application credentials in the Railway developer portal and update RAILWAY_CLIENT_ID / RAILWAY_CLIENT_SECRET in production secrets. This code change alone does not restore ingestion.

This PR (prevents token-endpoint hammering and improves signal clarity): Adds an appLevelOAuthError closure flag in runRailwayPullOnce. When the first invalid_client response is detected:

1. The error is logged once with clear credential-rotation guidance instead of per-installation.
2. The installation loop breaks immediately — no further token requests are issued for the remaining installations in that pull cycle.

This reduces Railway token-endpoint traffic from 12+ requests/minute to 1, and collapses the error signal from 12+ per cycle to 1 per cycle, making the failure immediately actionable.

Test

Added puller.test.ts case: two installations with expiring tokens, Railway token endpoint returns {"error":"invalid_client"}. Asserts token endpoint is hit exactly once, stats.errors === 1, rotation message in logs, and no intake traffic.

---

Incident: drifting-narwhal (e3d893f6-dc5d-40ad-9a78-321d178321ab)

---

Was this PR helpful? Leave feedback — goes straight to the Superlog team.

---

Summary by cubic

Circuit-breaks the Railway pull cycle when Railway rejects the OAuth application credentials (invalid_client) instead of treating it like a per-installation revocation and failing 12+ times per cycle.

Previously, invalid_client was handled like invalid_grant, so every installation hit the token endpoint with requests that all failed, producing 12+ railway token refresh failed errors per minute. Now the first invalid_client stops the loop: one error is logged with credential-rotation guidance, no further token requests are issued in that cycle, and token-endpoint traffic drops from 12+/minute to 1. Added a test asserting the token endpoint is hit exactly once and no intake traffic occurs.

Rollout action

Rotate the Railway OAuth app credentials in the developer portal and update RAILWAY_CLIENT_ID/RAILWAY_CLIENT_SECRET in production secrets — this code change alone does not restore ingestion.

^{Written for commit 537ddd9. Summary will update on new commits.}