external-api: serve all market data from the unified user_pool ETL (off Dune/app-DB)#15
Merged
Conversation
… Meteora path Cut /api/market-data Meteora rows over to read futarchy.meteora_daily directly from our served DB via externalDatabaseService.getDailyMeteoraVolumes (validated dollar-to-dollar vs the view: 1077/1077 rows, all cells; route returns source='etl-meteora-daily'). Remove the legacy Dune-sourced Meteora indexing: meteoraVolumeFetcherService, the dead dailyMeteoraVolumeService duplicate, dune-meteora-volumes.sql, dailyMeteoraVolumesRepo, the daily_meteora_volumes table schema, the databaseService meteora delegators + DailyMeteoraVolumeRecord, DUNE_METEORA_VOLUME_QUERY_ID, and all meteoraVolumeFetcher wiring. Tests updated; tsc + suite (112) green. Also includes the in-progress runtime split (main.ts API entrypoint / indexer.ts indexer entrypoint + src/runtime service factory) and the v0.6 spot read paths (coingecko getSpotRolling24hMetrics from futarchy.trades) this branch is built on — intermixed in the same files, so not separable into a Meteora-only commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….6 only) FutarchyAMM volume/24h data is already served from our v0.6 indexer (getDailyTradingActivity -> v06_fee_volume_daily_aggregate); the Dune-sourced 10-min -> hourly -> daily pipeline was dormant in prod (USE_DUNE_DATA=false). Validated the v0.6 path dollar-to-dollar vs the on-chain source public.v0_6_spot_swaps (660K swaps): spot_trade_count 656,158 (Δ0, every day, 9 months); spot base + USDC volume $435,786,572.45 (Δ$0.00); spot_usdc_fees $1,091,210.03 + spot_token_fees 2,661,845.56 (Δ0). Removed the Dune ingest/aggregation services (DuneService, DuneCacheService, TenMinuteVolumeFetcherService, HourlyAggregationService, DailyAggregationService) + their wiring (runtime/services, indexer, routes/types, root, health, metrics), the USE_DUNE_DATA flag, and the dune config block. market-data always reads v0.6 (source 'v06-indexer'). All v0.6 read paths, repos, tables, health counters, and V06ReconciliationService kept intact. tsc + suite (111) green; full-router smoke test passes. Phase 2 (follow-up, NOT in this commit): drop the now-frozen Dune tables (ten_minute_volumes/hourly_volumes/daily_volumes/daily_fees_volumes) + their repos + the dead getDailyBuySellVolumes method; repoint getFirstTradeDates to a v0.6 source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…irst-trade-dates to v0.6 (Phase 2) Phase-2 cleanup after the Dune fetcher removal (ae670b6): the daily_volumes / hourly_volumes / ten_minute_volumes / daily_buy_sell_volumes / daily_fees_volumes tables were frozen (never written) once the Dune services were gone. Remove them and all dead code that read them. Deleted repos: intervalVolumesRepo, dailyFeesVolumesRepo, dailyBuySellVolumesRepo. Stripped dailyVolumesRepo to only getV06Rolling24hMetrics (v06_spot_ohlcv_1m, a v0.6 table — kept). Removed all dead databaseService delegators + now-unused record interfaces. schemaManager: dropped the 5 Dune CREATE TABLEs + the aggregate_10min_to_hourly / aggregate_hourly_to_daily / calculate_rolling_24h functions (kept sync_metadata, metrics_history, service_health_snapshots, v06_*). health.ts/metrics.ts: removed the dead Dune record-count snapshots. Repointed coingecko first-trade-dates off the frozen Dune daily_buy_sell_volumes to a new externalDatabaseService.getFirstTradeDates() that reads the v0.6 indexer (v0_6_spot_swaps JOIN v0_6_daos, MIN date per base mint) in the served DB — so new tokens get a startDate again. Verified: tsc clean; bun test 113 pass; schemaManager DDL executes on a fresh DB; getFirstTradeDates returns 35 tokens with correct dates vs real v0.6 data (Avici 2025-10-18, Jurassic 2026-05-15, Ranger 2026-01-10). The actual prod DROP TABLE for the 5 dead tables is a manual DBA step. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…h checks external DB, remove dead Dune scripts, docs H5: removed broken/obsolete Dune backfill scripts that imported deleted services (backfill.ts, backfillExtendedFields.ts, backfillMeteoraVolumes.ts, compareDuneVsV06.ts) + their package.json entries; cleaned repo-guard.ts + CODEOWNERS references. H2: /api/market-data no longer masks served-DB failures as empty meteora — getDailyMeteoraVolumes now re-throws query/schema errors (→ 500) and the route returns 503 if the served DB is unavailable, instead of 200 with zero volume (financial-data correctness). H4: /api/health now reports externalDatabase connectivity and degrades when the served indexer DB (the source for Meteora/tickers/DexScreener/first-trade-dates) is down. M1: README — removed stale USE_DUNE_DATA / Dune env + pipeline docs; documented the served indexer DB as a required dependency and the Dune-free data sources; dropped deleted-file references from the architecture tree. Deferred (noted, pre-existing): H1 per-DAO ticker fallback, H3 indexer startup gating (superseded by the planned full indexer removal), M2 extra regression tests. L1 (route tests not validating) not reproduced — bun test 113 pass / 253 assertions incl. tests/routes/*. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…llback; surface served-DB failures H1: /api/tickers v0.6 fallback is now per-DAO — fills only the DAOs futarchy.trades didn't cover and merges, instead of all-or-nothing (partial primary coverage no longer zeroes the rest). H2: getSpotRolling24hMetrics now throws on query/schema failure (returns empty only for genuine zero/not-connected), so served-DB contract drift surfaces as a 5xx via asyncHandler instead of silently degrading to the v0.6 fallback. M1: removed the stale Dune-cache block (DUNE_CACHE_REFRESH_INTERVAL/DUNE_FETCH_TIMEOUT) from GET / metadata; replaced with a read-only/no-Dune note. Indexer removal (this repo is now a pure read-only API): deleted src/indexer.ts, V06ReconciliationService, backfillV06/backfillV06Fees scripts, the dev/start:indexer + backfill:v06 package scripts, and all wiring (runtime/services RuntimeMode narrowed to 'api', Services field, repo-guard/CODEOWNERS/README). tsc clean; bun test 112 pass. DEPLOY GATE: the v0.6 READ methods (getDailyTradingActivity → v06_fee_volume_daily_aggregate, getV06Rolling24hMetrics → v06_spot_ohlcv_1m) are KEPT, but this repo no longer POPULATES those app-DB aggregates. Before deploy, the v0.6 daily/OHLCV aggregates must be produced elsewhere (reid's indexer) OR these reads repointed to compute on-the-fly from the served DB's v0_6_spot_swaps/v0_6_conditional_swaps — otherwise FutarchyAMM market-data/tickers serve stale data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…output Every market-data read now comes from the unified user_pool ETL tables in the served DB — no app-DB v06 rollups, no futarchy.trades, no raw-swap aggregation: - /api/market-data futarchy → user_pool_daily (getFutarchyAmmDailyActivity); already wired. meteora → user_pool_daily (source='meteora') instead of the meteora_daily view (validated row-exact, 0 mismatches over 1082 rows; token_per_usdc derived = base_volume/target_volume = 1/average_price). - /api/tickers 24h metrics → user_pool_spot_ohlcv (1m candles, source= 'futarchy_amm'), keyed by token → dao. Drops BOTH the futarchy.trades primary and the dead app-DB v06_spot_ohlcv_1m fallback. Values are human units (no /1e6). - getFirstTradeDates → user_pool_daily MIN(date) instead of raw v0_6_spot_swaps. - Remove dead getDailyTradingActivity + getV06Rolling24hMetrics and their repos (v06TradingActivityRepo, dailyVolumesRepo). dexscreener stays on raw v0_6_spot_swaps by design — it's a per-swap event + post-swap-reserves (amm_base/quote_amount) adapter, data the aggregated ETL doesn't carry. Tests: add getFutarchyAmmDailyActivity to the mock; rewrite market.test with a seeded happy-path asserting the pivoted spot/conditional/total + protocol/LP split and meteora values (was a vacuous 200/500/503 check); repoint the tickers test to token-keyed user_pool_spot_ohlcv; drop the obsolete fallback test. 111 pass / 0 fail, typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…source) Repoint all three DEX Screener swap queries off the raw v0_6_* indexer tables onto the unified ETL output, so the entire external-api now reads from one source: - /events: user_pool_swaps (source='futarchy_amm', market_kind='spot'); columns aliased to the legacy v0_6 shape (side→swap_type; input/output reconstructed from base/quote+side) so the event builder is unchanged. reserves now come from our decoded amm_base_reserves/amm_quote_reserves — non-NULL for EVERY spot swap (vs the nullable raw column). Ordered by on-chain order (slot, inner_group, inner_ix) for stable txnIndex/eventIndex. - /latest-block + /pair: user_pool_swaps too (pair identity + creation in one query; drops the v0_6_daos lookup). Validated vs the LIVE API (market-api.metadao.fi/dexscreener/events): the migrated feed matches byte-for-byte on the full economic payload (signature + input + output + reserves, raw u64) — 6,555/6,555 swaps in a 400k-slot window, 0 mismatches. Adds dexscreener route tests (none existed): buy/sell leg mapping, priceNative, reserves, and per-block txnIndex/eventIndex grouping. 116 pass, typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nup, hardening Bundles the remaining external-api work for the unified user_pool migration (branch is local; nothing was pushed). All from this effort + the concurrent hardening/cleanup pass; typecheck clean, 116 tests pass. - coingecko /tickers: single source (user_pool_spot_ohlcv), 503 on served-DB down instead of masking zero volume. - externalDatabaseService: served-data contract health check (checkServedDataContract) surfaced via /health (degraded when the ETL tables are missing). - Security (info disclosure): market/health/metrics no longer return raw error.message — log server-side, return generic error + requestId. - Dead-code removal: drop the legacy Dune schema files, the app-DB v06 OHLCV/fee tables (schemaManager) and dead metricsService helpers — all unread after the cutover to user_pool_*. - Docs/config/tests updated to match (README, AGENTS, example.env, validation, resilience, root, test mocks + suites). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Package Protection
Repository GuardExact dependency versions
Package minimum age
Sensitive wallet / program changes
Overall status: pass Transitive supply-chain threats are covered by the Socket scanner (via |
Contributor
Author
|
@greptileai review |
…table inner_group/inner_ix are WITHIN-transaction coordinates, so two distinct transactions in the same slot can carry the same (inner_group, inner_ix). The /events query ordered only by (slot, inner_group, inner_ix), which (a) is non-deterministic among those tied rows and (b) interleaves one txn's events with another's. The txnIndex builder increments whenever the signature changes from the previous row, so an interleaved (non-contiguous) signature was assigned TWO different txnIndex values within a slot — a DEX Screener contract violation. Add signature to both ORDER BYs (events + the /pair first-swap lookup) so each transaction's events stay contiguous → stable, consistent txnIndex/eventIndex and a deterministic createdAtTxnId. Validated end-to-end against the served DB (real app over HTTP): a 500k-slot, 8,207-event window now reconciles to user_pool_swaps with 0 bad on amounts, priceNative, post-swap reserves (present on every event), execution order, and txnIndex. All futarchy base mints are 6-decimal, so the route's 1e6 scaling is correct for current data. 116 tests pass; typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…llers retry) Audit of every financial endpoint for one rule: if an upstream (Solana RPC / served DB) fails, the API must ERROR (consumers are pollers and retry) — it must never serve a wrong or fabricated-zero number as if real. Found and fixed several paths that silently degraded to wrong/zero: WRONG PRICE - futarchyService.getTokenDecimals: returned a hardcoded `9` on RPC failure. All futarchy tokens are 6-decimal, so a fabricated 9 made the served price 10^3 = 1000x wrong (and it was served, not skipped). Now propagates — decimals are immutable so the long cache shields the steady state. OVERSTATED CIRCULATING SUPPLY / MARKET CAP (the dangerous class) - launchpadService.fetchLaunchFromProgram: caught ALL errors and returned null, so an RPC blip fetching the launch account looked identical to "no launch" → getTokenAllocationBreakdown returned the all-zero emptyBreakdown → circulating supply = total supply (every locked allocation counted as circulating). Now only a genuine "Account does not exist" returns null; RPC errors propagate. - getFutarchyAmmLiquidity / getMeteoraLpLiquidity / performance-package / DAO-treasury fetches: each caught getAccount() failures and assumed 0, conflating "account genuinely absent" (0 correct) with "RPC failed" (0 wrong → undercounts locked → overstates circulating). New util isTokenAccountAbsent() classifies via the spl-token TokenAccountNotFoundError/TokenInvalidAccountOwnerError types: 0 only for genuine absence, otherwise propagate. EMPTY TICKER SET - futarchyService.getAllDaos: per-DAO fetch errors were swallowed. A full RPC outage already throws on the DAO-list fetch, but a degraded RPC that failed every per-DAO fetch would return [] → /api/tickers 200 with no tickers. Now tracks per-DAO errors and throws if it produced 0 tickers from >0 DAOs with errors. Left intentionally: per-token 0 24h-volume in /api/tickers (getSpotRolling24hMetrics already throws on query failure; an empty map = a genuinely untraded pair, where 0 is the correct CoinGecko value and the live price still serves). market-data + dexscreener already 503 on DB-down and propagate query errors. typecheck clean; 116 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…ol served DB Complete the cutover so the API is a pure read layer over the served ETL DB: - remove the embedded app-DB/indexer runtime (databaseService, dbRuntime, schemaManager, metricsRepo) and meteoraService; all market data now reads futarchy.user_pool_* in the served DB (the unified user_pool ETL output) - add a background heartbeat self-check (served-DB connectivity, data freshness, served-data contract drift) with cooldown'd webhook alerts + Prometheus gauges - harden external-DB TLS: verify the server cert by default (system CAs or EXTERNAL_DATABASE_CA_CERT), explicit EXTERNAL_DATABASE_SSL_NO_VERIFY opt-out - per-IP rate limiting via an explicit trust-proxy hop count (never blanket true) - graceful shutdown: drain in-flight requests before closing DB pools - add CI workflow + failure-path and launchpad service tests Validated against the prod read replica (railway): served-data contract OK; swap coverage exact vs Dune query 7729524 (540,178/540,178 across all 13 pools); per-pool realized fees match Dune earned within ~9% (authoritative realized collected >= rate-based accrual). typecheck clean; 114/114 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- contract check: user_pool_daily required-columns now covers EVERY column the market-data queries select (buy/sell_volume, token_fees, token_fees_usdc, average_price, futarchy_protocol_fee_usdc, futarchy_lp_fee_usdc, conditional_reconciled, pending_open_proposals) — previously the heartbeat could report ok:true while a dropped column made every query throw at runtime - getDailyMeteoraVolumes: token_per_usdc typed string | null (it is SQL NULL on zero-volume days via nullif(target_volume,0); null is the correct "undefined ratio" value, so widen the type rather than coalesce to a misleading 0) - /api/market-data: validate params before the served-DB availability check so a malformed/missing date returns 400 (client error) instead of 503 during a DB outage - example.env: document RPCPOOL_RPC_URL / RPCPOOL_WS_URL (take precedence over the SOLANA_RPC_URL/WS_URL public fallback) typecheck clean; 114/114 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
example.env was missing the required DB connection (it was commented out) and carried dead/over-verbose entries. Now: - FRONTEND_READER_PG_URL is uncommented and clearly marked REQUIRED (without it the market-data/tickers/dexscreener routes return 503) - removed env vars the code never reads: DEV_MODE, SOLANA_WS_URL/RPCPOOL_WS_URL (config.solana.wsUrl is unused) — and dropped the matching dead fields from config.ts (devMode, solana.wsUrl) - every remaining documented var maps 1:1 to a process.env read in config.ts (23/23); EXCLUDED_DAOS keeps its 15 unique mints (one duplicate removed) - trimmed verbose prose to concise comments Kept PROTOCOL_FEE_RATE / config.fees — it IS used (dexscreener fee-bps). No hardcoded credentials. typecheck clean; 114/114 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The served-DB connection was reachable under two names (FRONTEND_READER_PG_URL preferred, EXTERNAL_DATABASE_URL fallback) — a leftover from the vault-keyed-env migration that was never finished. Collapse to a single DATABASE_PG_URL and rename the SSL family to match (DATABASE_PG_SSL / _CA_CERT / _SSL_NO_VERIFY). Hard cut, no fallback: the Northflank vault key and any local .env must be renamed to DATABASE_PG_URL before deploy or market routes 503. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
metajinglun
marked this pull request as ready for review
June 16, 2026 22:25
metanallok
approved these changes
Jun 17, 2026
metanallok
left a comment
metanallok
left a comment
Contributor
There was a problem hiding this comment.
@meta-reid would like your approval too, but this looks good to me.
meta-reid
self-requested a review
June 17, 2026 01:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Migrates every external-api market-data read to the unified, on-chain-derived
user_poolETL output in the served (railway) DB — one source of truth. No Dune, no app-DB v06 rollups, nofutarchy.trades, no raw-swap aggregation./api/market-datafutarchyv06_fee_volume_daily_aggregate(flat 0.5%)futarchy.user_pool_daily(source=futarchy_amm) — pivoted spot/conditional/total + protocol/LP split/api/market-datameteorameteora_dailyviewuser_pool_daily(source=meteora)/api/tickers24h hi/lo/volfutarchy.trades+ dead app-DBv06_spot_ohlcv_1mfallbackuser_pool_spot_ohlcv(1m candles), single source/dexscreener/*v0_6_spot_swaps/v0_6_daosuser_pool_swaps(+ decoded post-swap reserves)v0_6_spot_swapsuser_pool_dailyAlso in here
schemaManager),v06TradingActivityRepo+dailyVolumesRepo, and deadmetricsServicehelpers — all unread after the cutover./health(degraded when the ETL tables are missing), and/api/tickersreturns 503 on served-DB down instead of masking zero volume.market/health/metricsno longer return rawerror.message— log server-side, return a generic error +requestId.Validation
user_pool_dailyvalidated row-exact vsmeteora_daily(0 mismatches / 1082 rows;token_per_usdcderived =base/target)./dexscreener/eventsAPI — 6,555/6,555 swaps on a 400k-slot window (full economic payload: amounts + reserves, raw u64). Reserves separately validated exact vs live for 6,628 swaps.dexscreenerroute tests added (none existed);markettest rewritten with a seeded happy-path. 116 pass, typecheck clean.The ETL tables (
futarchy.user_pool_daily/user_pool_spot_ohlcv/user_pool_swaps) must exist + be populated on the prod served DB before this ships, or/api/market-data,/api/tickers, and/dexscreener/*go empty. Order:0005+0006+0007migrations +rebuildAll) on prod railway.🤖 Generated with Claude Code
Greptile Summary
This PR consolidates all external-API market-data reads onto a single unified ETL output (
futarchy.user_pool_daily,user_pool_spot_ohlcv,user_pool_swaps) and removes roughly 950 lines of legacy Dune/app-DB code paths./api/market-data,/api/tickers, and/dexscreener/*all read theuser_poolETL tables in the served (Railway) DB; the oldfutarchy.trades,v06_fee_volume_daily_aggregate, and raw-swap aggregations are gone./api/healthsurfaces all three axes; Prometheus gauges are updated on every scrape.Confidence Score: 5/5
Safe to merge once the ETL tables are live on prod per the documented deploy ordering.
The cutover is a clean hard swap with no partial fallback paths left in code. The contract check, heartbeat, and failure-path tests together ensure outages surface as 5xx rather than silent zeroes. The only non-trivial findings are a dead u.id column in the events SELECT (unused and absent from the contract check) and sequential DB calls in /api/tickers that could be parallelized — neither affects correctness.
No files require special attention beyond the deploy prerequisite (ETL tables must exist before this ships).
Important Files Changed
Reviews (5): Last reviewed commit: "refactor(external-api): collapse DB env ..." | Re-trigger Greptile