feat: Discord alert relay for Locomotive log shipping

[joemurrell]

Summary

Restores production log alerting to Discord. Investigation while following up on the scheduled-posting fix (#68/#69) found the locomotive logging Railway service — meant to forward critical bot logs to Discord — had been crash-looping and dead since 2026-05-20 (the tool it runs, ferretcode/locomotive, was archived upstream on 2025-08-06). It was dead for the entire lifetime of the scheduled-posting bug, which is very likely why no one caught it earlier.

Migrating to the maintained fork (ghcr.io/brody192/locomotive) surfaced a second gap: v2 has no Discord webhook mode at all (confirmed in source — only json/jsonl/datadog/axiom/betterstack/loki/sentry/signoz/victorialogs/otel_http). Pointing it straight at a Discord webhook URL fails outright — confirmed live in the new Locomotive service logs: dispatch failed, retrying → dropping item after exhausting all retries, because Discord's webhook API rejects Locomotive's raw JSON (it expects a specific {content, embeds} schema).

alert_relay.py is the missing piece: a small standalone service that sits between Locomotive and Discord, translating one into the other.

What it does

• POST /webhook receives Locomotive's JSON array or JSONL payload.
• Filters to WARNING/ERROR/CRITICAL only, via the app-reported level field.
• Reformats matches into Discord embeds (color-coded by severity, service/environment/logger as fields, message truncated to Discord limits) and posts them to DISCORD_WEBHOOK_URL.
• Batches of >10 alertable lines (Discord's per-message embed cap) are chunked plus a rolled-up summary line.
• X-Relay-Secret header check against RELAY_SHARED_SECRET so the endpoint isn't open to anyone who finds the URL.
• GET /health for Railway's health check.

Important: requires LOG_FORMAT=json on the bot to actually filter correctly

Confirmed by comparing two data points from this investigation: Locomotive's own Go structured logs came through Railway correctly tagged ("dispatch failed, retrying" → severity: "warn", "dropping item..." → severity: "error"), while every log line pulled from the bot all session — including ones logged via logger.warning()/logger.error() in the code — showed severity: "info" uniformly. Railway only extracts a real severity from apps emitting structured/leveled log output; the bot's current plain-text format (LOG_FORMAT unset) means every line looks identical to Railway regardless of actual Python log level.

extract_level() in this relay deliberately reads the app's own level field (populated by JSONFormatter when LOG_FORMAT=json) rather than Railway's severity — falling back to severity would have reproduced the exact blind spot that let the original bug go unnoticed. This PR alone does not fix alerting — LOG_FORMAT=json still needs to be set on the discord bot Railway service, and this relay needs to be deployed with Locomotive pointed at it, for alerts to actually flow. Those are infra changes to make after this merges, not part of the diff.

Test plan

• pytest tests/ — 120 passed (96 existing + 24 new)
• Manual end-to-end smoke test: real aiohttp server + real HTTP client + fake Discord endpoint — verified health check, unauthenticated-request rejection, mixed-severity batch (INFO dropped, WARNING+ERROR forwarded as 2 embeds), and JSONL body parsing all work against the actual running server, not just the unit-tested pure functions.

---

Generated by Claude Code

[railway-app]

🚅 Deployed to the brevitybot-pr-70 environment in BrevityBot

Service	Status	Web	Updated (UTC)
discord bot	✅ Success (View Logs)		Jul 25, 2026 at 5:33 am