Add optional source-reliability layer, new data sources & SOURCES health panel - #230
Open
stephenlthorn wants to merge 1 commit into
Open
Conversation
…anel Purely additive — no existing route or behavior is changed. Adds an opt-in reliability toolkit, five new data sources built on it, and a panel to see feed health. Reliability layer (src/lib/sources/**, unit-tested): - SourceAdapter framework with TTL caching (fresh-vs-stale), per-source rate limiting, health tracking (ok/degraded/down/unknown), fallback chains (runWithFallback), and params-aware cache keys. - HMR-safe singleton + a registry. Existing routes are left untouched; they can adopt runSource incrementally if/when desired. New data sources (adapter + route + toggleable map layer): - NOAA Tsunami warnings, NOAA NDBC ocean buoys (900+ stations), ransomware.live recent victims (jittered country centroid, since records carry only a country). Opt-in keyed sources — disabled cleanly with no errors until configured: - Shodan "Exposed Devices" layer (SHODAN_API_KEY) - WiGLE wireless-network lookup in the region dossier (WIGLE_API_NAME + WIGLE_API_TOKEN) Both documented in .env.example. SOURCES panel + GET /api/sources/health: - Endpoint reports every registered source's status, freshness and attribution. A new SOURCES tab (desktop tool strip + mobile nav) groups them with status dots and provider links; polls every 45s. The only edits to existing files are additive: new layer toggles in LayerPanel, new map layers/popups in OsirisMap, and the new fetch blocks + SOURCES button in page.tsx. Existing layers behave exactly as before. Verification: clean production build; 88 unit tests pass (adapters, runner, cache, rate-limit, health, registry, health endpoint); dev server confirmed the new routes serve live data and the keyed sources degrade gracefully when unset.
stephenlthorn
force-pushed
the
feat/source-reliability-layer
branch
from
July 6, 2026 13:46
8b6bff8 to
b11d273
Compare
stephenlthorn
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An optional, purely additive enhancement to Osiris — it doesn't change or replace any existing route or behavior. It adds a reliability toolkit, five new data sources built on it, and a panel to see feed health. Existing layers work exactly as they do today; everything here is new files plus a handful of additive edits (new layer toggles, new map layers, a new SOURCES button).
Huge thanks for building Osiris — this is meant to build on it, not restructure it. Happy to adjust anything to fit your conventions, or trim it down.
🛡️ Optional reliability layer (
src/lib/sources/**, unit-tested)A
SourceAdapterframework any route can opt into viarunSource(...):runWithFallback, e.g. FIRMS VIIRS → MODIS).Your existing routes are left untouched — they can adopt this incrementally if/when you want. Nothing is forced onto them.
🌐 New sources (adapter + route + toggleable map layer)
Keyless: NOAA Tsunami, NOAA NDBC ocean buoys (900+ stations), ransomware.live recent victims (jittered country centroid, since records carry only a country).
Opt-in keyed, disabled with no errors until configured: Shodan "Exposed Devices" (
SHODAN_API_KEY), WiGLE wireless lookup in the region dossier (WIGLE_API_NAME+WIGLE_API_TOKEN). Documented in.env.example.📡 SOURCES panel +
GET /api/sources/healthReports each registered source's status, freshness and attribution. A new SOURCES tab (desktop tool strip + mobile nav) groups them with status dots and provider links; polls every 45s.
Scope
LayerPanel(new toggles),OsirisMap(new layers/popups),page.tsx(new fetch blocks + SOURCES button), and.env.example(new opt-in keys).✅ Verification
Clean production build; 88 unit tests pass; dev server confirmed the new routes serve live data and keyed sources degrade gracefully when unset.
Note: I also have a couple of small, unrelated fixes (a privacy fix for the analytics middleware, and a few data-source bugs I noticed) that I'm happy to send as separate focused PRs if useful.