Skip to content

feat: serve fort map data and filter pick lists from Golbat's fort API - #175

Draft
jfberry wants to merge 17 commits into
ccev:mainfrom
jfberry:feat/golbat-fort-api
Draft

feat: serve fort map data and filter pick lists from Golbat's fort API#175
jfberry wants to merge 17 commits into
ccev:mainfrom
jfberry:feat/golbat-fort-api

Conversation

@jfberry

@jfberry jfberry commented Aug 3, 2026

Copy link
Copy Markdown

Implements the design agreed in #174: gyms, pokéstops and stations are served from Golbat's fort HTTP API (UnownHash/Golbat#385) instead of raw SQL, and filter pick lists are sourced from GET api/fort/available — with automatic detection and full SQL fallback. Draft until the live parity checks below are done.

What's here (Phase 1 + Phase 2 of #174)

  • Detection with SQL fallback (golbatFortApi.ts): a 60s poll of api/fort/available doubles as feature detection (the endpoints are 503-gated on fort_in_memory). No config option — golbat.url/golbat.secret are all that's needed. Golbat can be toggled/upgraded without restarting Diadem. Unreachable-Golbat network errors are tolerated at startup and at runtime; the detection probe is bounded by a 10s timeout.
  • Three API-backed query classes (ApiGymQuery/ApiPokestopQuery/ApiStationQuery) subclassing the SQL classes and overriding only query()/querySingle() — all filter()/prepare() logic is inherited, same as the PokemonQuery pattern. If a scan or by-id call fails mid-request (e.g. Golbat dies inside the 60s detection window), the request falls back to the parent SQL implementation instead of 500ing.
  • DNF translation (fortDnf.ts, unit-tested): each SQL OR-branch becomes one DNF clause; everything DNF can't express (boss temp_evolution_id, quest title/target, ranking_standard, exact item amounts, bread_mode) is loosened to a superset and re-checked by the existing local filter logic.
  • Pick lists from availability (Phase 2, merged into MasterStats at /api/stats request time — zero component churn): raid bosses, quest rewards (incl. counts), max battles and showcases now refresh within ~60s instead of hourly. The three live-table SQL queries in queryMasterStats() (incl. the quest UNION ALL — the most expensive recurring query) are skipped while the fort API is on. On an API→SQL fallback transition the stats provider refreshes immediately so pick lists don't sit empty until the hourly tick.
  • Docs: Golbat-side requirements documented in the configuration reference.

Accepted behavioral deltas (from #174 / Appendix B)

  • Raid/battle/showcase pick-list entries lose count (nothing rendered them; the availability index can't provide them by design).
  • Mega bosses appear as their base form and showcase ranking_standard is 0 in pick lists until the Golbat availability-enrichment PR lands.
  • bread_mode on max-battle pick entries is derived from battle_level >= 6 (to be confirmed against live data, see checklist).
  • enabled is now actually populated on gyms/pokestops (was declared but never selected by the SQL).
  • Rate-limit charging uses Golbat's examined counts, which can run higher than SQL's matched-row counts (same as the pokemon scan path).

Pending live verification (why this is a draft)

Needs a Golbat with fort_in_memory = true (preload = true recommended):

  • Parity pan-around vs the SQL path: plain forts, raid filter w/ boss list, quest filter w/ stardust range + item, invasion filter w/ characters, max-battle filter w/ boss + hasGmax — same forts render; popups show defenders, RSVPs, quests, incidents (confirmed lineups), showcases, stationed Pokémon
  • By-id queries via marker clicks (gym/pokéstop/station)
  • Limit notice (hitting map object limits should be handled nicer #151 toast) still fires when zoomed out
  • Stop Golbat → ≤60s flip to SQL (log line + working map); start → flips back
  • Pick lists populate from availability; egg hint in GymPopup
  • Golbat DNF egg semantics: {pokemon_id: 0} must match null-boss raids like SQL's COALESCE(raid_pokemon_id, 0) = 0
  • Wire shapes: quest_rewards/showcase_focus/showcase_rankings/stationed_pokemon arrive as JSON strings (inherited prepare() parses strings)
  • SELECT DISTINCT battle_level, battle_pokemon_bread_mode FROM station WHERE battle_pokemon_bread_mode IS NOT NULL; — confirm gmax ⇔ level 6

Upstream Golbat follow-ups (filed separately, none block this PR)

  1. Status/feature-flags endpoint (would replace probing; also exposes server scan caps)
  2. limit_reached on fort scan responses (mirror of feat: expose v3 pokemon scan limit status UnownHash/Golbat#392 — the interim limit + 1 overflow check covers until then)
  3. Availability enrichment: temp_evolution_id on raids, ranking_standard on showcases
  4. ApiPokestopResult.FirstSeenTimestamp is int16 — unix timestamps truncate

Notes for review

  • The gym field typo availble_slots is intentionally preserved (DB column name); the API's available_slots is mapped onto it.
  • Pre-existing (upstream main) svelte-check errors: 9 errors in 5 unrelated UI files; this branch adds none.
  • Found while implementing, out of scope here: pokestop.d.ts has QuestRewardTempEvoBranch (and three sibling types) misplaced in the ContestFocus union plus a dangling QuestRewardPokemonEgg reference — masked by skipLibCheck. Worth a separate issue; the one place it bites here is documented at the cast site in queryStats.ts.

🤖 Generated with Claude Code

jfberry and others added 16 commits August 3, 2026 13:38
Plan for serving gym/pokestop/station map data from Golbat's fort API
(UnownHash/Golbat#385) with automatic detection and SQL fallback, plus
filter pick lists from fort availability. Agreed design: ccev#174.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SQL "boss" OR-branch (COALESCE(raid_pokemon_id, 0) != 0) is
unconditional and doesn't fold in `levels`. buildGymDnfFilters was
narrowing it to filterset.levels when present, which could exclude a
hatched raid at a level outside that filterset's `levels` list -
silently hiding it from the map. Push an any-active-raid clause
instead, matching the egg branch's unconditional style; the local
filter pass re-checks the hatched/level constraint.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lure

AMOUNT_MAX capped quest reward amount ranges at 10000, silently dropping
quests above that threshold since the SQL fallback omitted the max clause
entirely. Raise it to int32 max so the DNF range can never exclude a real
amount.

Also add per-request SQL fallback to ApiGymQuery/ApiPokestopQuery/
ApiStationQuery: a thrown fetch error (e.g. Golbat dying mid-request) or an
undefined scan/by-id result now falls back to the parent SQL query instead
of always 500ing or silently missing data. Found-but-deleted objects still
short-circuit to [] as before.
Golbat's fort API sends quest_rewards / alternative_quest_rewards as
real JSON arrays, while parseQuestReward (via the inherited prepare)
expects the SQL rows' serialized string form — every quest-bearing
pokestop threw and the route 500'd. Re-serialize both fields in the
mapper, correct the wire type that hid the mismatch, and move the
mapper to its own type-only-import module so it is unit-testable
under bare vitest.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jfberry added a commit to UnownHash/Golbat that referenced this pull request Aug 4, 2026
The fort API's design is native JSON for blob columns, but the pass
was incomplete: gyms were fully converted while stationed_pokemon,
showcase_focus, showcase_rankings and both quest_conditions still
arrived as escaped strings. Convert all five via jsonRaw, matching
the gym rsvps precedent. Consumers are unmerged (ccev/diadem#175,
WatWowMap/ReactMap#1228 — the latter already handles both forms), so
now is the time to break the wire.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Golbat is finishing its fort-API native-JSON conversion
(UnownHash/Golbat#393): stationed_pokemon, showcase_focus and
showcase_rankings switch from serialized strings to real JSON. Accept
both wire generations via blobToString so either Golbat works, and
type the wire fields honestly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jfberry

jfberry commented Aug 4, 2026

Copy link
Copy Markdown
Author

Heads-up on a cross-PR dependency: UnownHash/Golbat#393 now finishes the fort API's native-JSON conversion — stationed_pokemon, showcase_focus and showcase_rankings become real JSON instead of serialized strings. This branch handles both wire generations (blobToString in the mappers), so it works against current Golbat main and against #393 once merged. Also fixed here: quest rewards already arrive as native JSON on the current wire, which 500'd every quest-bearing pokestop scan — mappers now re-serialize to the SQL row shape the shared prepare() expects (with unit tests).

@jfberry

jfberry commented Aug 4, 2026

Copy link
Copy Markdown
Author

Proposal for a follow-up PR (deliberately not part of this one) — flipping the internal fort-data contract to parsed JSON:

Right now the internal contract after a query is "the SQL row's shape", i.e. JSON blobs as serialized strings, and prepare() parses them. That's why the API mappers here re-serialize Golbat's native JSON back to strings just for prepare() to re-parse — and why the quest-rewards bug existed at all: any field whose wire form diverges from the DB form is a trap. (The gym path already half-escapes this: its mapper assigns parsed defenders/rsvps directly and the string branches in prepare() no-op.)

The follow-up would normalize at each source boundary instead:

  • Db*Query gets a row-conversion step doing the JSON.parse + dropping the *_raw aliases right after the query
  • the API mappers emit parsed objects (parse-if-string stays for the old-Golbat transition window, then dies)
  • prepare() shrinks to form normalization + permission stripping
  • the transient defenders_raw / raw_rsvps / raw_stationed_pokemon fields disappear from the data types
  • mapper tests flip to asserting parsed output

Benefits: one canonical internal shape, the wire-vs-DB trap class is gone, no stringify→parse round-trip per scan. Cost: touches queryGym/queryPokestop/queryStation internals and three type files — which is exactly why it shouldn't ride along in this PR, since the SQL path is the fallback safety net here and deserves its own review.

One counterweight to weigh: if the fort SQL path eventually retires once fort_in_memory graduates, part of the refactor has limited shelf life — the argument for doing it anyway is that the fallback will realistically coexist with the API path for a long while, and that's precisely when the trap bites.

Any objections to this direction? If you're happy I'd do it as a small standalone PR once this one lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant