From 6216ca11cddcc409d7c17ee1e6ee8fc61faa4351 Mon Sep 17 00:00:00 2001 From: Jinglun Date: Thu, 4 Jun 2026 17:49:39 -0700 Subject: [PATCH 01/14] feat(meteora): serve /api/market-data Meteora from our ETL; drop Dune Meteora path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cut /api/market-data Meteora rows over to read futarchy.meteora_daily directly from our served DB via externalDatabaseService.getDailyMeteoraVolumes (validated dollar-to-dollar vs the view: 1077/1077 rows, all cells; route returns source='etl-meteora-daily'). Remove the legacy Dune-sourced Meteora indexing: meteoraVolumeFetcherService, the dead dailyMeteoraVolumeService duplicate, dune-meteora-volumes.sql, dailyMeteoraVolumesRepo, the daily_meteora_volumes table schema, the databaseService meteora delegators + DailyMeteoraVolumeRecord, DUNE_METEORA_VOLUME_QUERY_ID, and all meteoraVolumeFetcher wiring. Tests updated; tsc + suite (112) green. Also includes the in-progress runtime split (main.ts API entrypoint / indexer.ts indexer entrypoint + src/runtime service factory) and the v0.6 spot read paths (coingecko getSpotRolling24hMetrics from futarchy.trades) this branch is built on — intermixed in the same files, so not separable into a Meteora-only commit. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 30 +- docs/tickers-volume-cutover-validation.md | 147 ++++++ package.json | 3 + src/config.ts | 2 - src/indexer.ts | 137 ++++++ src/main.ts | 218 +-------- src/routes/coingecko.ts | 143 +++--- src/routes/market.ts | 13 +- src/routes/root.ts | 6 +- src/routes/types.ts | 4 - src/runtime/services.ts | 95 ++++ src/schema/dune-meteora-volumes.sql | 355 --------------- src/services/dailyMeteoraVolumeService.ts | 415 ----------------- .../internal/repos/dailyMeteoraVolumesRepo.ts | 244 ---------- .../database/internal/schemaManager.ts | 27 -- src/services/databaseService.ts | 81 +--- src/services/externalDatabaseService.ts | 136 +++++- src/services/meteoraService.ts | 3 - src/services/meteoraVolumeFetcherService.ts | 419 ------------------ tests/api.test.ts | 53 ++- tests/helpers/testApp.ts | 4 +- tests/runtime/services.test.ts | 35 ++ 22 files changed, 724 insertions(+), 1846 deletions(-) create mode 100644 docs/tickers-volume-cutover-validation.md create mode 100644 src/indexer.ts create mode 100644 src/runtime/services.ts delete mode 100644 src/schema/dune-meteora-volumes.sql delete mode 100644 src/services/dailyMeteoraVolumeService.ts delete mode 100644 src/services/database/internal/repos/dailyMeteoraVolumesRepo.ts delete mode 100644 src/services/meteoraVolumeFetcherService.ts create mode 100644 tests/runtime/services.test.ts diff --git a/README.md b/README.md index 0a1b76c..c7fb738 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ Returns all DAO tickers with pricing, volume, and liquidity information. Automat | `high_24h` / `low_24h` | 24h high/low (when available) | | `startDate` | First trade date for the token | -**Volume sources** (priority order): 10-minute DB → hourly DB → Dune cache. +**Volume sources** (priority order): 10-minute DB → hourly DB → optional legacy cache. --- @@ -199,8 +199,14 @@ bun run build # Start the server (runs build first) bun run start +# Start indexing workers separately (runs build first) +bun run start:indexer + # Development with hot reload bun run dev + +# Development indexer with hot reload +bun run dev:indexer ``` ## Configuration @@ -242,7 +248,10 @@ Create a `.env` file in the root directory (see `example.env` for reference): ``` src/ ├── app.ts # Express app setup & middleware -├── main.ts # Entry point, service wiring, scheduled tasks +├── main.ts # API entry point (serves routes, no indexing workers) +├── indexer.ts # Indexer entry point (Dune collection, rollups, v0.6 reconciliation) +├── runtime/ +│ └── services.ts # API/indexer service composition ├── config.ts # Environment variables & configuration ├── routes/ │ ├── index.ts # Route registration @@ -281,9 +290,24 @@ src/ ## Data Pipeline ### Volume Sources (priority order) +The API process serves data from the app DB and read-only external indexer DB. It does not start indexing, Dune fetchers, rollups, reconciliation workers, or app-DB schema setup. + +Run the indexer process separately: + +```bash +bun run dev:indexer + +# production +bun run start:indexer +``` + +Indexer-owned jobs: + 1. **10-minute volumes** — from Dune, stored in `ten_minute_volumes`, rolled up to hourly/daily 2. **v0.6 Reconciliation** — hourly job reads `v0_6_spot_swaps` + `v0_6_conditional_swaps` from the external indexer DB, writes OHLCV and fee breakdowns to app DB -3. **Dune cache** — fallback for 24h metrics when DB sources unavailable +3. **Dune cache health** — maintained only by the indexer runtime for compatibility with existing health/metrics views + +The API's `/api/tickers` reads 24h metrics directly from the app DB in priority order: `ten_minute_volumes` → `hourly_volumes` → optional legacy cache if a caller intentionally wires it. ### DexScreener Pipeline The DexScreener adapter reads **directly from the external indexer DB** (`v0_6_spot_swaps` + `v0_6_daos`) and serves real-time swap events indexed by Solana slot. No intermediate aggregation — raw swap data mapped to the DexScreener schema. diff --git a/docs/tickers-volume-cutover-validation.md b/docs/tickers-volume-cutover-validation.md new file mode 100644 index 0000000..2530962 --- /dev/null +++ b/docs/tickers-volume-cutover-validation.md @@ -0,0 +1,147 @@ +# `/api/tickers` volume cutover — validation plan + +Status: **design / pre-cutover**. This is the checkable contract we validate against +before deleting the Dune tickers pipeline. + +## 0. Goal & scope + +`/api/tickers` now serves 24h volume metrics from the indexer DB's `futarchy.trades` +(read-through via `ExternalDatabaseService.getSpotRolling24hMetrics`), with the v0.6 +reconciliation OHLCV (`v06_spot_ohlcv_1m`) as a Dune-free fallback. The Dune pipeline +(10-minute / hourly / daily fetchers + Dune cache) is **superseded but not yet +deleted**. This plan defines how we prove the new source is correct — equal to, or +correctly-better-than, what we serve today — before tearing the Dune pipeline out. + +**In scope** — the per-DAO 24h spot fields tickers exposes: +`base_volume` (base-token), `target_volume` (USDC), `high_24h`, `low_24h`, `trade_count`. + +**Out of scope** — `last_price`, `bid`/`ask`/spread, `liquidity_in_usd`: computed live +from on-chain reserves, unchanged by this cutover. `startDate` (`getFirstTradeDates`, +still Dune-sourced `daily_buy_sell_volumes`) — tracked as a separate follow-up. + +## 1. What we're validating against (baselines, increasing authority) + +| baseline | source | role | +|---|---|---| +| **Live API** | prod `GET /api/tickers` (today: Dune-sourced volume) | the incumbent served truth — what consumers see now | +| **New** | `futarchy.trades` rolling-24h (local/staging ETL DB) | the candidate | +| **v0.6 OHLCV** | `v06_spot_ohlcv_1m` (reconciliation, already non-Dune, live in prod) | independent same-chain cross-check | +| **On-chain** | raw swap txs / `futarchy.trades` rows | adjudicator — final tie-breaker | + +Principle (mirrors the Meteora/v0.6 parity work): the live API / Dune is the +**incumbent**, not infallible. A divergence is **adjudicated against chain**, not +assumed to be an ETL bug. A Dune-side error is a pass-with-note. + +## 2. The core challenge: rolling window vs closed window + +The tickers metric is a **rolling last-24h** number. Comparing it live across two +systems is inherently noisy: + +- **Clock skew** — both sides must evaluate `now()` at the same instant. +- **Freshness lag** — Dune refreshes on a delay (10-min query batched hourly); the + real-time ETL includes the most recent ~hour Dune hasn't ingested yet. So the new + source reads **higher** on recently-active DAOs. *Expected, not a bug.* +- **Coverage** — the two pipelines may not cover the identical swap set at the edge. + +Therefore the **rigorous gate is a closed-UTC-day backtest** (stable, complete +windows — no moving edge, no freshness noise). The live rolling comparison is a +valuable **end-to-end sanity check**, run with a looser tolerance that accounts for +the open-window lag. + +## 3. Validation methods + +### A. Closed-day backtest — DB↔DB (primary gate) +For each **completed** UTC day over a fixed window (target **≥30 days**), per DAO, +aggregate from each source and compare every in-scope field: +- New: `Σ` over `futarchy.trades` (spot) for that day, decimal-aware. +- Dune: the day's `daily_volumes` / hourly rollup. +- v0.6: `v06_spot_ohlcv_1d` (or `1m` summed). + +Full set, **no sampling**. Tolerances per §4. Days present in one source but not +another are reported as **coverage gaps** (investigate), not silently dropped. +Tool: `scripts/compareTickersVolume.ts` (§5). + +### B. v0.6 ↔ `futarchy.trades` cross-check (strongest, lowest-noise) +Both derive from the **same** indexer chain data (legacy `v0_6_spot_swaps` vs new +`futarchy.trades`) — no third-party, no Dune. If these two agree, the new source is +validated against the path already trusted in prod under `USE_DUNE_DATA=false`. +This is the highest-signal check because the only differences can come from +derivation logic, not data provenance. Expect **tight** agreement; any gap is a +real logic difference to root-cause. + +### C. Live API ↔ local ETL — end-to-end rolling spot-check (the E2E proof) +The user-requested axis. Capture prod `GET /api/tickers` and, at the same instant, +compute the new rolling-24h from the local/staging ETL DB; diff per DAO. +- Captures the **actual served path**, not just DB aggregates. +- Looser tolerance (§4) absorbing the open-window freshness lag. +- A large, *unexplained-by-lag* gap (esp. on a closed-history DAO with no recent + trades) is a real finding → adjudicate via D. +Tool: `scripts/compareTickersLiveVsLocal.ts` (§5). + +### D. On-chain adjudication (tie-breaker) +For any divergence beyond tolerance in A/B/C, pull the underlying swaps for that +`(dao, window)` on-chain, recompute by hand, and record which side is right. This is +the guard against *both* the ETL and the incumbent being wrong. + +## 4. Tolerances & expected (non-failing) divergences + +| field class | tolerance | notes | +|---|---|---| +| `trade_count` | **exact** (closed-day) | integer; rolling allows freshness slack | +| `base_volume` / `target_volume` (closed-day, 6-dec token) | **≤ 0.5%** rel. | mechanical aggregation | +| `high_24h` / `low_24h` (closed-day) | **≤ 0.5%** rel. | price extremes; watch unit (§ decimals) | +| any field (live rolling, method C) | **≤ ~5%** rel. | absorbs open-window freshness lag | +| v0.6 ↔ trades (method B) | **≤ 0.5%** rel. | same chain data; tighter is better | + +**Expected divergences — treat as pass-with-note, NOT failures:** +1. **Decimals correction.** The new source scales by each token's real + `futarchy.tokens.decimals`; Dune/v0.6 assume `÷1e6`. For a **non-6-decimal base + token**, `base_volume` (and `high/low`, scaled by `10^(baseDec−quoteDec)`) will + legitimately differ — the new value is *more correct*. The harness must flag the + token's decimals so a decimals-driven delta is recognised, not chased as a bug. +2. **Freshness lag** on the open rolling window (method C only) — new ≥ incumbent on + recently-active DAOs. +3. **Coverage:** DAOs the new source has but Dune doesn't → **new coverage** (good). + DAOs Dune has but the new source is **missing** → **failure** (investigate). + +Note: tickers uses only base/target **totals**, not a buy/sell split, so the +direction-label inversion that affected Meteora does **not** affect these fields. + +## 5. Tooling to build + +1. **`scripts/compareTickersVolume.ts`** — DB↔DB harness (methods A & B). Reads the + `dao_addr → base_mint` map from `futarchy.daos` (no RPC), then lines up the three + sources through the *actual serving functions* (`getSpotRolling24hMetrics`, + `getV06Rolling24hMetrics`, `getRolling24hFromTenMinute`/`getRolling24hMetrics`) + and/or closed-day aggregates. Per-DAO field deltas + Δ%, tolerance flags, coverage + gaps, per-token decimals column. Modeled on `scripts/compareDuneVsV06.ts`. +2. **`scripts/compareTickersLiveVsLocal.ts`** — E2E (method C). `fetch(/api/tickers)` + (prod, incumbent) vs the local new-source rolling-24h for the same DAOs at one + instant; per-DAO diff of base/target/high/low with the looser rolling tolerance, + annotating likely freshness-lag rows. `API_BASE` via env/arg. +3. **Flag-gate** — `ENABLE_LEGACY_DUNE_TICKERS` (default off). The serving path is + already cut over; this flag keeps the Dune **fetchers** runnable in the comparison + env so `daily_volumes`/hourly stay populated as the method-A incumbent. Off in + clean/serve-only deploys; **on** wherever we run the backtest. + +## 6. Cutover gate (delete the Dune tickers pipeline when ALL hold) + +- **B** (v0.6 ↔ trades): within 0.5% across the full DAO set on the shared window. +- **A** (closed-day vs Dune): within tolerance over ≥30 complete days, every + out-of-tolerance day adjudicated by **D** (and decimals-deltas accounted). +- **C** (live vs local): agreement within the rolling tolerance, residual gaps fully + explained by freshness/decimals — no unexplained divergence on a closed-history DAO. +- **No missing-DAO failures** (new source covers every DAO Dune serves today). + +On pass: delete `TenMinuteVolumeFetcherService`, `HourlyAggregationService`, +`DailyAggregationService`, `DuneCacheService`, the Dune SQL, the tickers-only tables +(`ten_minute_volumes`/`hourly_volumes`/`daily_volumes`) + repos, and the +`ENABLE_LEGACY_DUNE_TICKERS` flag. Keep v0.6 reconciliation (feeds `/api/market-data`). + +## 7. Rollback + +The new source and the v0.6 fallback are both Dune-free and the fallback is live in +prod today, so the serving path degrades safely (new → v0.6) even pre-cutover. If a +blocking issue is found post-cutover, re-enabling Dune requires reverting the +teardown PR — which is why teardown is gated on this plan passing, and is a separate +PR from the read-through change. diff --git a/package.json b/package.json index 579c37e..f1ad985 100644 --- a/package.json +++ b/package.json @@ -6,8 +6,11 @@ "version": "2.0.0", "scripts": { "dev": "bun --watch src/main.ts", + "dev:indexer": "bun --watch src/indexer.ts", "build": "tsc", "start": "bun dist/main.js", + "prestart:indexer": "bun run build", + "start:indexer": "bun dist/indexer.js", "prestart": "bun run build", "test": "bun test", "backfill": "bun run scripts/backfill.ts", diff --git a/src/config.ts b/src/config.ts index 9823913..1d47d50 100644 --- a/src/config.ts +++ b/src/config.ts @@ -54,8 +54,6 @@ export const config = { apiKey: process.env.DUNE_API_KEY || '', // ACTIVE: 10-minute query - single source of truth, all other data aggregated from this tenMinuteVolumeQueryId: process.env.DUNE_TEN_MINUTE_VOLUME_QUERY_ID ? parseInt(process.env.DUNE_TEN_MINUTE_VOLUME_QUERY_ID) : undefined, - // ACTIVE: Meteora daily volumes query - tracks Meteora pool fees per owner (service currently disabled) - meteoraVolumeQueryId: process.env.DUNE_METEORA_VOLUME_QUERY_ID ? parseInt(process.env.DUNE_METEORA_VOLUME_QUERY_ID) : undefined, }, alerts: { webhookUrl: process.env.ALERT_WEBHOOK_URL || 'https://telegram-webhook-relay.themetadao-org.workers.dev', diff --git a/src/indexer.ts b/src/indexer.ts new file mode 100644 index 0000000..02af9e1 --- /dev/null +++ b/src/indexer.ts @@ -0,0 +1,137 @@ +import { saveHealthSnapshots } from './routes/health.js'; +import { createServiceGetters } from './routes/types.js'; +import { closeDataStores, createServices, initializeRuntimeDataStores } from './runtime/services.js'; +import { logger } from './utils/logger.js'; +import { scheduleDailyAtUTC, scheduleWithoutPileup, type ScheduledTask } from './utils/scheduling.js'; +import type { Services } from './app.js'; + +async function startIndexingServices(services: Services): Promise { + if (services.hourlyAggregationService) { + logger.info('Starting Hourly Aggregation service'); + try { + await services.hourlyAggregationService.start(); + logger.info('Hourly Aggregation service started'); + } catch (error) { + logger.error('Failed to start Hourly Aggregation service', error); + } + } + + if (services.tenMinuteVolumeFetcherService) { + logger.info('Starting 10-Minute Volume Fetcher service'); + try { + await services.tenMinuteVolumeFetcherService.start(); + logger.info('10-Minute Volume Fetcher service started'); + } catch (error) { + logger.error('Failed to start 10-Minute Volume Fetcher service', error); + } + } + + if (services.dailyAggregationService) { + logger.info('Starting Daily Aggregation service'); + try { + await services.dailyAggregationService.initialize(); + services.dailyAggregationService.start(); + logger.info('Daily Aggregation service started'); + } catch (error) { + logger.error('Failed to start Daily Aggregation service', error); + } + } + + if (services.duneCacheService) { + logger.info('Starting Dune cache service'); + try { + await services.duneCacheService.start(); + logger.info('Dune cache service started'); + } catch (error) { + logger.error('Failed to start Dune cache service', error); + } + } + + if (services.externalDatabaseService?.isAvailable() && services.v06ReconciliationService) { + logger.info('Starting v0.6 Reconciliation service'); + services.v06ReconciliationService.start(); + logger.info('v0.6 Reconciliation service started'); + } +} + +function startIndexerScheduledTasks(services: Services): ScheduledTask[] { + const tasks: ScheduledTask[] = []; + const serviceGetters = createServiceGetters(services); + + const healthSnapshotTask = scheduleWithoutPileup( + async () => { + await saveHealthSnapshots(serviceGetters); + }, + { + name: 'IndexerHealthSnapshot', + intervalMs: 5 * 60 * 1000, + onError: (error) => logger.error('Error saving indexer health snapshot', error), + } + ); + tasks.push(healthSnapshotTask); + logger.info('Indexer health snapshots scheduled every 5 minutes'); + + const metricsPruneTask = scheduleDailyAtUTC( + async () => { + if (services.databaseService.isAvailable()) { + await services.databaseService.pruneOldMetrics(30); + logger.info('Old metrics pruned (keeping last 30 days)'); + } + }, + { + name: 'IndexerMetricsPrune', + hourUTC: 3, + onError: (error) => logger.error('Error pruning indexer metrics', error), + } + ); + tasks.push(metricsPruneTask); + logger.info('Indexer metrics pruning scheduled daily at 03:00 UTC'); + + return tasks; +} + +async function stopIndexingServices(services: Services, scheduledTasks: ScheduledTask[]): Promise { + scheduledTasks.forEach(task => task.stop()); + services.duneCacheService?.stop(); + services.hourlyAggregationService?.stop(); + services.tenMinuteVolumeFetcherService?.stop(); + services.dailyAggregationService?.stop(); + services.v06ReconciliationService?.stop(); + await closeDataStores(services); +} + +async function main(): Promise { + const services = createServices('indexer'); + let scheduledTasks: ScheduledTask[] = []; + + await initializeRuntimeDataStores(services, 'Indexer', { ensureAppSchema: true }); + await startIndexingServices(services); + scheduledTasks = startIndexerScheduledTasks(services); + + const serviceGetters = createServiceGetters(services); + setTimeout(async () => { + try { + await saveHealthSnapshots(serviceGetters); + logger.info('Initial indexer health snapshot saved'); + } catch (error) { + logger.error('Error saving initial indexer health snapshot', error); + } + }, 10000); + + logger.info('Indexer runtime started'); + + const shutdown = async (signal: string): Promise => { + logger.info(`${signal} received, shutting down indexer gracefully`); + await stopIndexingServices(services, scheduledTasks); + logger.info('Indexer stopped'); + process.exit(0); + }; + + process.on('SIGTERM', () => shutdown('SIGTERM')); + process.on('SIGINT', () => shutdown('SIGINT')); +} + +main().catch((error) => { + logger.error('Failed to start indexer runtime', error); + process.exit(1); +}); diff --git a/src/main.ts b/src/main.ts index 2066213..b33768a 100644 --- a/src/main.ts +++ b/src/main.ts @@ -1,199 +1,21 @@ -import { createApp, type Services } from './app.js'; -import { FutarchyService } from './services/futarchyService.js'; -import { PriceService } from './services/priceService.js'; -import { DuneService } from './services/duneService.js'; -import { DuneCacheService } from './services/duneCacheService.js'; -import { SolanaService } from './services/solanaService.js'; -import { LaunchpadService } from './services/launchpadService.js'; -import { DatabaseService } from './services/databaseService.js'; -import { ExternalDatabaseService } from './services/externalDatabaseService.js'; -import { HourlyAggregationService } from './services/hourlyAggregationService.js'; -import { TenMinuteVolumeFetcherService } from './services/tenMinuteVolumeFetcherService.js'; -import { DailyAggregationService } from './services/dailyAggregationService.js'; -import { MeteoraVolumeFetcherService } from './services/meteoraVolumeFetcherService.js'; -import { V06ReconciliationService } from './services/v06ReconciliationService.js'; +import { createApp } from './app.js'; import { config } from './config.js'; +import { closeDataStores, createServices, initializeRuntimeDataStores } from './runtime/services.js'; import { logger } from './utils/logger.js'; -import { scheduleWithoutPileup, scheduleDailyAtUTC, type ScheduledTask } from './utils/scheduling.js'; -import { saveHealthSnapshots } from './routes/health.js'; -import { createServiceGetters, type ServiceGetters } from './routes/types.js'; import type { Server } from 'http'; -function initializeServices(): Services { - const futarchyService = new FutarchyService(); - const priceService = new PriceService(); - const databaseService = new DatabaseService(); - const externalDatabaseService = new ExternalDatabaseService(); - const solanaService = new SolanaService(); - const launchpadService = new LaunchpadService(); - - let duneService: DuneService | null = null; - if (config.dune.apiKey) { - duneService = new DuneService(); - } - - let duneCacheService: DuneCacheService | null = null; - let hourlyAggregationService: HourlyAggregationService | null = null; - let tenMinuteVolumeFetcherService: TenMinuteVolumeFetcherService | null = null; - let dailyAggregationService: DailyAggregationService | null = null; - let meteoraVolumeFetcherService: MeteoraVolumeFetcherService | null = null; - - if (duneService) { - duneCacheService = new DuneCacheService(duneService, databaseService, futarchyService); - hourlyAggregationService = new HourlyAggregationService(databaseService, duneService, futarchyService); - tenMinuteVolumeFetcherService = new TenMinuteVolumeFetcherService(duneService, databaseService, futarchyService); - dailyAggregationService = new DailyAggregationService(duneService, databaseService, futarchyService); - meteoraVolumeFetcherService = new MeteoraVolumeFetcherService(duneService, databaseService); - } - - const v06ReconciliationService = new V06ReconciliationService(databaseService, externalDatabaseService); - - return { - futarchyService, - priceService, - databaseService, - externalDatabaseService, - duneService, - duneCacheService, - solanaService, - launchpadService, - hourlyAggregationService, - tenMinuteVolumeFetcherService, - dailyAggregationService, - meteoraVolumeFetcherService, - v06ReconciliationService, - }; -} - -async function startServices(services: Services): Promise { - // Initialize database connection first - const dbConnected = await services.databaseService.initialize(); - if (dbConnected) { - logger.info('Database service initialized and connected'); - } else { - logger.warn('Database service not available - volume history will use in-memory cache only'); - } - - if (services.hourlyAggregationService) { - logger.info('Starting Hourly Aggregation service'); - try { - await services.hourlyAggregationService.start(); - logger.info('Hourly Aggregation service started'); - } catch (error) { - logger.error('Failed to start Hourly Aggregation service', error); - } - } - - if (services.tenMinuteVolumeFetcherService) { - logger.info('Starting 10-Minute Volume Fetcher service'); - try { - await services.tenMinuteVolumeFetcherService.start(); - logger.info('10-Minute Volume Fetcher service started'); - } catch (error) { - logger.error('Failed to start 10-Minute Volume Fetcher service', error); - } - } - - if (services.dailyAggregationService) { - logger.info('Starting Daily Aggregation service'); - try { - await services.dailyAggregationService.initialize(); - services.dailyAggregationService.start(); - logger.info('Daily Aggregation service started'); - } catch (error) { - logger.error('Failed to start Daily Aggregation service', error); - } - } - - if (services.duneCacheService) { - logger.info('Starting Dune cache service'); - try { - await services.duneCacheService.start(); - logger.info('Dune cache service started'); - } catch (error) { - logger.error('Failed to start Dune cache service', error); - } - } - - if (services.meteoraVolumeFetcherService) { - logger.info('Starting Meteora Volume Fetcher service'); - try { - await services.meteoraVolumeFetcherService.initialize(); - services.meteoraVolumeFetcherService.start(); - logger.info('Meteora Volume Fetcher service started'); - } catch (error) { - logger.error('Failed to start Meteora Volume Fetcher service', error); - } - } - - // Initialize external indexer DB and start v0.6 reconciliation - if (services.externalDatabaseService) { - const extConnected = await services.externalDatabaseService.initialize(); - if (extConnected && services.v06ReconciliationService) { - logger.info('Starting v0.6 Reconciliation service'); - services.v06ReconciliationService.start(); - logger.info('v0.6 Reconciliation service started'); - } - } -} - -async function stopServices(services: Services, scheduledTasks: ScheduledTask[]): Promise { - scheduledTasks.forEach(task => task.stop()); - services.duneCacheService?.stop(); - services.hourlyAggregationService?.stop(); - services.tenMinuteVolumeFetcherService?.stop(); - services.dailyAggregationService?.stop(); - services.meteoraVolumeFetcherService?.stop(); - services.v06ReconciliationService?.stop(); - await services.externalDatabaseService?.close(); - await services.databaseService.close(); -} - -function startScheduledTasks(services: Services): ScheduledTask[] { - const tasks: ScheduledTask[] = []; - const serviceGetters = createServiceGetters(services); - - // Health snapshots every 5 minutes - const healthSnapshotTask = scheduleWithoutPileup( - async () => { - await saveHealthSnapshots(serviceGetters); - }, - { - name: 'HealthSnapshot', - intervalMs: 5 * 60 * 1000, - onError: (error) => logger.error('Error saving health snapshot', error), - } - ); - tasks.push(healthSnapshotTask); - logger.info('Health snapshots scheduled every 5 minutes'); - - // Prune old metrics daily at 03:00 UTC - const metricsPruneTask = scheduleDailyAtUTC( - async () => { - if (services.databaseService.isAvailable()) { - await services.databaseService.pruneOldMetrics(30); - logger.info('Old metrics pruned (keeping last 30 days)'); - } - }, - { - name: 'MetricsPrune', - hourUTC: 3, - onError: (error) => logger.error('Error pruning metrics', error), - } - ); - tasks.push(metricsPruneTask); - logger.info('Metrics pruning scheduled daily at 03:00 UTC'); - - return tasks; +async function stopApi(services: ReturnType): Promise { + await closeDataStores(services); } async function main(): Promise { - const services = initializeServices(); + const services = createServices('api'); const app = createApp({ services }); - let scheduledTasks: ScheduledTask[] = []; - const server: Server = app.listen(config.server.port, async () => { - logger.info('Server started', { + await initializeRuntimeDataStores(services, 'API', { ensureAppSchema: false }); + + const server: Server = app.listen(config.server.port, () => { + logger.info('API server started', { port: config.server.port, tickersUrl: `http://localhost:${config.server.port}/api/tickers`, healthUrl: `http://localhost:${config.server.port}/health`, @@ -202,20 +24,6 @@ async function main(): Promise { logger.info('Trusted API keys loaded', { count: config.server.trustedApiKeys.size, }); - - await startServices(services); - scheduledTasks = startScheduledTasks(services); - - // Save initial health snapshot after startup - const serviceGetters = createServiceGetters(services); - setTimeout(async () => { - try { - await saveHealthSnapshots(serviceGetters); - logger.info('Initial health snapshot saved'); - } catch (error) { - logger.error('Error saving initial health snapshot', error); - } - }, 10000); }); server.timeout = config.server.requestTimeout; @@ -223,10 +31,10 @@ async function main(): Promise { server.headersTimeout = config.server.keepAliveTimeout + 1000; const shutdown = async (signal: string): Promise => { - logger.info(`${signal} received, shutting down gracefully`); - await stopServices(services, scheduledTasks); + logger.info(`${signal} received, shutting down API gracefully`); + await stopApi(services); server.close(() => { - logger.info('Server closed'); + logger.info('API server closed'); process.exit(0); }); }; @@ -236,6 +44,6 @@ async function main(): Promise { } main().catch((error) => { - logger.error('Failed to start server', error); + logger.error('Failed to start API server', error); process.exit(1); }); diff --git a/src/routes/coingecko.ts b/src/routes/coingecko.ts index d6c8d4c..5921be2 100644 --- a/src/routes/coingecko.ts +++ b/src/routes/coingecko.ts @@ -4,114 +4,75 @@ import type { ServiceGetters } from './types.js'; import { asyncHandler } from '../middleware/errorHandler.js'; import { logger } from '../utils/logger.js'; import { sendAlert } from '../utils/alerts.js'; -import { config } from '../config.js'; export function createCoinGeckoRouter(services: ServiceGetters): Router { const router = Router(); - const { getFutarchyService, getPriceService, getDuneCacheService, getTenMinuteVolumeFetcherService, getHourlyAggregationService, getDatabaseService } = services; + const { getFutarchyService, getPriceService, getDatabaseService, getExternalDatabaseService } = services; // CoinGecko Endpoint: /tickers router.get('/api/tickers', asyncHandler(async (req: Request, res: Response) => { const futarchyService = getFutarchyService(); const priceService = getPriceService(); - const duneCacheService = getDuneCacheService(); - const tenMinuteVolumeFetcherService = getTenMinuteVolumeFetcherService(); - const hourlyAggregationService = getHourlyAggregationService(); const databaseService = getDatabaseService(); - + const externalDatabaseService = getExternalDatabaseService(); + const allDaos = await futarchyService.getAllDaos(); - - const firstTradeDates = databaseService?.isAvailable() - ? await databaseService.getFirstTradeDates() + + const firstTradeDates = databaseService?.isAvailable() + ? await databaseService.getFirstTradeDates() : new Map(); - + const tokenToDaoMap = new Map(); for (const dao of allDaos) { tokenToDaoMap.set(dao.baseMint.toString(), dao.daoAddress.toString()); } - - let volumeMetricsMap = new Map(); + + const volumeMetricsMap = new Map(); let volumeSource = 'none'; - if (!config.useDuneData) { - // USE_DUNE_DATA=false — source FutarchyAMM 24h volume from v0.6 indexer data - if (databaseService?.isAvailable()) { - const baseMints = allDaos.map(dao => dao.baseMint.toString()); - const v06Metrics = await databaseService.getV06Rolling24hMetrics(baseMints); - - if (v06Metrics.size > 0) { - for (const [tokenAddress, metrics] of v06Metrics.entries()) { - const daoAddress = tokenToDaoMap.get(tokenAddress); - if (daoAddress) { - volumeMetricsMap.set(daoAddress, { - base_volume_24h: metrics.base_volume_24h, - target_volume_24h: metrics.target_volume_24h, - high_24h: metrics.high_24h, - low_24h: metrics.low_24h, - }); - } - } - volumeSource = 'v06-indexer'; - logger.debug('Using v0.6 indexer rolling 24h metrics', { daoCount: volumeMetricsMap.size, requestId: req.requestId }); - } + // Primary: read rolling-24h spot metrics straight from the indexer DB + // (futarchy.trades, keyed by dao_addr). No Dune, no app-DB rollup. + if (externalDatabaseService?.isAvailable()) { + const daoAddresses = allDaos.map(dao => dao.daoAddress.toString()); + const spotMetrics = await externalDatabaseService.getSpotRolling24hMetrics(daoAddresses); + + for (const [daoAddress, metrics] of spotMetrics.entries()) { + volumeMetricsMap.set(daoAddress, { + base_volume_24h: metrics.base_volume_24h, + target_volume_24h: metrics.target_volume_24h, + high_24h: metrics.high_24h, + low_24h: metrics.low_24h, + }); } - } else { - // USE_DUNE_DATA=true (default) — Dune pipeline: 10-min → hourly → cache - // Try TenMinuteVolumeFetcherService first - if (tenMinuteVolumeFetcherService?.isInitialized && tenMinuteVolumeFetcherService.isDatabaseConnected()) { - const baseMints = allDaos.map(dao => dao.baseMint.toString()); - const tenMinMetrics = await tenMinuteVolumeFetcherService.getRolling24hMetrics(baseMints); - - if (tenMinMetrics.size > 0) { - for (const [tokenAddress, metrics] of tenMinMetrics.entries()) { - const daoAddress = tokenToDaoMap.get(tokenAddress); - if (daoAddress) { - volumeMetricsMap.set(daoAddress, { - base_volume_24h: String(metrics.base_volume_24h), - target_volume_24h: String(metrics.target_volume_24h), - high_24h: String(metrics.high_24h), - low_24h: String(metrics.low_24h), - }); - } - } - volumeSource = '10-minute'; - logger.debug('Using 10-minute rolling 24h metrics', { daoCount: volumeMetricsMap.size, requestId: req.requestId }); - } + + if (volumeMetricsMap.size > 0) { + volumeSource = 'futarchy-trades-db'; + logger.debug('Using indexer futarchy.trades rolling 24h metrics', { daoCount: volumeMetricsMap.size, requestId: req.requestId }); } + } - // Fall back to HourlyAggregationService - if (volumeMetricsMap.size === 0 && hourlyAggregationService?.isInitialized && hourlyAggregationService.isDatabaseConnected()) { - const baseMints = allDaos.map(dao => dao.baseMint.toString()); - const hourlyMetrics = await hourlyAggregationService.getRolling24hMetrics(baseMints); - - if (hourlyMetrics.size > 0) { - for (const [tokenAddress, metrics] of hourlyMetrics.entries()) { - const daoAddress = tokenToDaoMap.get(tokenAddress); - if (daoAddress) { - volumeMetricsMap.set(daoAddress, { - base_volume_24h: metrics.base_volume_24h, - target_volume_24h: metrics.target_volume_24h, - high_24h: metrics.high_24h, - low_24h: metrics.low_24h, - }); - } - } - volumeSource = 'hourly'; - logger.debug('Using hourly rolling 24h metrics', { daoCount: volumeMetricsMap.size, requestId: req.requestId }); + // Fallback: v0.6 indexer OHLCV (app DB, populated by reconciliation from + // the indexer DB). Live in prod and Dune-free — covers the window before + // futarchy.trades is populated for a DAO. + if (volumeMetricsMap.size === 0 && databaseService?.isAvailable()) { + const baseMints = allDaos.map(dao => dao.baseMint.toString()); + const v06Metrics = await databaseService.getV06Rolling24hMetrics(baseMints); + + for (const [tokenAddress, metrics] of v06Metrics.entries()) { + const daoAddress = tokenToDaoMap.get(tokenAddress); + if (daoAddress) { + volumeMetricsMap.set(daoAddress, { + base_volume_24h: metrics.base_volume_24h, + target_volume_24h: metrics.target_volume_24h, + high_24h: metrics.high_24h, + low_24h: metrics.low_24h, + }); } } - // Fall back to DuneCacheService - if (volumeMetricsMap.size === 0 && duneCacheService) { - const cachedMetrics = duneCacheService.getPoolMetrics(); - if (cachedMetrics && cachedMetrics.size > 0) { - const cacheStatus = duneCacheService.getCacheStatus(); - logger.debug('Using Dune cache metrics', { cacheAgeSeconds: Math.round(cacheStatus.cacheAgeMs / 1000), entryCount: cachedMetrics.size, requestId: req.requestId }); - volumeMetricsMap = cachedMetrics; - volumeSource = 'dune-cache'; - } else { - logger.warn('No cached metrics available yet', { requestId: req.requestId }); - } + if (volumeMetricsMap.size > 0) { + volumeSource = 'v06-indexer-fallback'; + logger.debug('Using v0.6 indexer rolling 24h metrics (fallback)', { daoCount: volumeMetricsMap.size, requestId: req.requestId }); } } @@ -165,17 +126,17 @@ export function createCoinGeckoRouter(services: ServiceGetters): Router { if (!liquidityUsd) continue; - const duneMetrics = volumeMetricsMap.get(poolId); + const volumeMetrics = volumeMetricsMap.get(poolId); let baseVolume: string; let targetVolume: string; let high24h: string | undefined; let low24h: string | undefined; - if (duneMetrics) { - baseVolume = duneMetrics.base_volume_24h; - targetVolume = duneMetrics.target_volume_24h; - high24h = duneMetrics.high_24h !== '0' ? duneMetrics.high_24h : undefined; - low24h = duneMetrics.low_24h !== '0' ? duneMetrics.low_24h : undefined; + if (volumeMetrics) { + baseVolume = volumeMetrics.base_volume_24h; + targetVolume = volumeMetrics.target_volume_24h; + high24h = volumeMetrics.high_24h !== '0' ? volumeMetrics.high_24h : undefined; + low24h = volumeMetrics.low_24h !== '0' ? volumeMetrics.low_24h : undefined; } else { baseVolume = '0'; targetVolume = '0'; diff --git a/src/routes/market.ts b/src/routes/market.ts index eaed9ec..fe8e5f7 100644 --- a/src/routes/market.ts +++ b/src/routes/market.ts @@ -5,7 +5,7 @@ import type { ServiceGetters } from './types.js'; export function createMarketRouter(services: ServiceGetters): Router { const router = Router(); - const { getDatabaseService } = services; + const { getDatabaseService, getExternalDatabaseService } = services; // Get daily market data with date range and optional token filtering // When USE_DUNE_DATA=false, sources from v0.6 indexer aggregate table; @@ -46,13 +46,19 @@ export function createMarketRouter(services: ServiceGetters): Router { const useV06 = !config.useDuneData; + // Meteora rows are served directly from our meteora accounting ETL + // (futarchy.meteora_daily in the served DB, via externalDatabase). + const externalDatabaseService = getExternalDatabaseService(); + const [futarchyData, meteoraData] = await Promise.all([ useV06 ? databaseService.getDailyTradingActivity(queryOptions) : databaseService.getDailyBuySellVolumes(queryOptions), - databaseService.getDailyMeteoraVolumes(queryOptions), + externalDatabaseService?.isAvailable() + ? externalDatabaseService.getDailyMeteoraVolumes(queryOptions) + : Promise.resolve([]), ]); - + res.json({ filters: { tokens: tokensResult.value || 'all', @@ -65,6 +71,7 @@ export function createMarketRouter(services: ServiceGetters): Router { data: futarchyData, }, meteora: { + source: 'etl-meteora-daily', count: meteoraData.length, data: meteoraData, }, diff --git a/src/routes/root.ts b/src/routes/root.ts index dec8528..4c06dbf 100644 --- a/src/routes/root.ts +++ b/src/routes/root.ts @@ -44,16 +44,16 @@ export function createRootRouter(services: ServiceGetters): Router { meteoraLpLiquidity: 'Tokens in the external Meteora DAMM pool (POL) - IS circulating', }, caching: { - description: 'Dune data is cached and refreshed hourly to improve response times', + description: 'Ticker volume is served from app DB aggregates populated by the separate indexer runtime', refreshInterval: `${parseInt(process.env.DUNE_CACHE_REFRESH_INTERVAL || '3600')} seconds`, fetchTimeout: `${parseInt(process.env.DUNE_FETCH_TIMEOUT || '240')} seconds`, status: cacheStatus ? { isInitialized: cacheStatus.isInitialized, poolMetricsCount: cacheStatus.poolMetricsCount, lastUpdated: cacheStatus.lastUpdated.toISOString(), - } : 'Not available (DUNE_API_KEY not set)', + } : 'No live cache in API runtime', }, - note: 'This API automatically discovers and aggregates all DAOs from the Futarchy protocol.', + note: 'This API discovers DAOs for serving responses; background indexing runs separately.', }); }); diff --git a/src/routes/types.ts b/src/routes/types.ts index 5b3c0ed..5f8911f 100644 --- a/src/routes/types.ts +++ b/src/routes/types.ts @@ -8,7 +8,6 @@ import type { DatabaseService } from '../services/databaseService.js'; import type { HourlyAggregationService } from '../services/hourlyAggregationService.js'; import type { TenMinuteVolumeFetcherService } from '../services/tenMinuteVolumeFetcherService.js'; import type { DailyAggregationService } from '../services/dailyAggregationService.js'; -import type { MeteoraVolumeFetcherService } from '../services/meteoraVolumeFetcherService.js'; import type { ExternalDatabaseService } from '../services/externalDatabaseService.js'; import type { V06ReconciliationService } from '../services/v06ReconciliationService.js'; import { AppError } from '../middleware/errorHandler.js'; @@ -32,7 +31,6 @@ export interface Services { hourlyAggregationService: HourlyAggregationService | null; tenMinuteVolumeFetcherService: TenMinuteVolumeFetcherService | null; dailyAggregationService: DailyAggregationService | null; - meteoraVolumeFetcherService: MeteoraVolumeFetcherService | null; v06ReconciliationService: V06ReconciliationService | null; solanaService?: SolanaService; @@ -55,7 +53,6 @@ export interface ServiceGetters { getHourlyAggregationService: () => HourlyAggregationService | null; getTenMinuteVolumeFetcherService: () => TenMinuteVolumeFetcherService | null; getDailyAggregationService: () => DailyAggregationService | null; - getMeteoraVolumeFetcherService: () => MeteoraVolumeFetcherService | null; getExternalDatabaseService: () => ExternalDatabaseService | null; } @@ -79,7 +76,6 @@ export function createServiceGetters(services: Services): ServiceGetters { getHourlyAggregationService: () => optionalService(services.hourlyAggregationService), getTenMinuteVolumeFetcherService: () => optionalService(services.tenMinuteVolumeFetcherService), getDailyAggregationService: () => optionalService(services.dailyAggregationService), - getMeteoraVolumeFetcherService: () => optionalService(services.meteoraVolumeFetcherService), getExternalDatabaseService: () => optionalService(services.externalDatabaseService), getSolanaService: () => requireService(services.solanaService, 'Solana'), diff --git a/src/runtime/services.ts b/src/runtime/services.ts new file mode 100644 index 0000000..ea54903 --- /dev/null +++ b/src/runtime/services.ts @@ -0,0 +1,95 @@ +import type { Services } from '../app.js'; +import { config } from '../config.js'; +import { DailyAggregationService } from '../services/dailyAggregationService.js'; +import { DatabaseService } from '../services/databaseService.js'; +import { DuneCacheService } from '../services/duneCacheService.js'; +import { DuneService } from '../services/duneService.js'; +import { ExternalDatabaseService } from '../services/externalDatabaseService.js'; +import { FutarchyService } from '../services/futarchyService.js'; +import { HourlyAggregationService } from '../services/hourlyAggregationService.js'; +import { LaunchpadService } from '../services/launchpadService.js'; +import { PriceService } from '../services/priceService.js'; +import { SolanaService } from '../services/solanaService.js'; +import { TenMinuteVolumeFetcherService } from '../services/tenMinuteVolumeFetcherService.js'; +import { V06ReconciliationService } from '../services/v06ReconciliationService.js'; +import { logger } from '../utils/logger.js'; + +export type RuntimeMode = 'api' | 'indexer'; + +export function createServices(mode: RuntimeMode): Services { + const futarchyService = new FutarchyService(); + const priceService = new PriceService(); + const databaseService = new DatabaseService(); + const externalDatabaseService = new ExternalDatabaseService(); + const solanaService = new SolanaService(); + const launchpadService = new LaunchpadService(); + + let duneService: DuneService | null = null; + let duneCacheService: DuneCacheService | null = null; + let hourlyAggregationService: HourlyAggregationService | null = null; + let tenMinuteVolumeFetcherService: TenMinuteVolumeFetcherService | null = null; + let dailyAggregationService: DailyAggregationService | null = null; + let v06ReconciliationService: V06ReconciliationService | null = null; + + if (mode === 'indexer') { + if (config.dune.apiKey) { + duneService = new DuneService(); + duneCacheService = new DuneCacheService(duneService, databaseService, futarchyService); + hourlyAggregationService = new HourlyAggregationService(databaseService, duneService, futarchyService); + tenMinuteVolumeFetcherService = new TenMinuteVolumeFetcherService(duneService, databaseService, futarchyService); + dailyAggregationService = new DailyAggregationService(duneService, databaseService, futarchyService); + } else { + logger.warn('DUNE_API_KEY not configured - Dune indexing services will not be created'); + } + + v06ReconciliationService = new V06ReconciliationService(databaseService, externalDatabaseService); + } + + return { + futarchyService, + priceService, + databaseService, + externalDatabaseService, + duneService, + duneCacheService, + solanaService, + launchpadService, + hourlyAggregationService, + tenMinuteVolumeFetcherService, + dailyAggregationService, + v06ReconciliationService, + }; +} + +export interface RuntimeDataStoreOptions { + ensureAppSchema: boolean; +} + +export async function initializeRuntimeDataStores( + services: Services, + runtimeName: string, + options: RuntimeDataStoreOptions +): Promise { + const dbConnected = await services.databaseService.initialize({ + ensureSchema: options.ensureAppSchema, + }); + if (dbConnected) { + logger.info(`${runtimeName} app database initialized and connected`); + } else { + logger.warn(`${runtimeName} app database not available - historical volume routes may be degraded`); + } + + if (services.externalDatabaseService) { + const extConnected = await services.externalDatabaseService.initialize(); + if (extConnected) { + logger.info(`${runtimeName} external indexer database initialized and connected`); + } else { + logger.warn(`${runtimeName} external indexer database not available`); + } + } +} + +export async function closeDataStores(services: Services): Promise { + await services.externalDatabaseService?.close(); + await services.databaseService.close(); +} diff --git a/src/schema/dune-meteora-volumes.sql b/src/schema/dune-meteora-volumes.sql deleted file mode 100644 index 8bb6379..0000000 --- a/src/schema/dune-meteora-volumes.sql +++ /dev/null @@ -1,355 +0,0 @@ --- Dune Query: Meteora Daily Volumes per Owner --- Deploy: paste this file into the query behind DUNE_METEORA_VOLUME_QUERY_ID (see .env). --- --- Parameters: --- start_date: DATE - fetch data from this date onwards (default: '2025-10-09') --- end_date: DATE (optional) - fetch data up to this date (inclusive). If not provided, fetches all data from start_date onwards. --- --- Returns daily aggregated data per owner with all required fields --- Cumulative fields are calculated in PostgreSQL, not here - -WITH -params AS ( - SELECT - TIMESTAMP '{{start_date}}' AS start_ts, - CASE - WHEN '{{end_date}}' = '9999-12-31' -- Sentinel value means no constraint - THEN TIMESTAMP '9999-12-31' -- No end date constraint if not provided - ELSE (CAST('{{end_date}}' AS TIMESTAMP) + INTERVAL '1' DAY) -- Make end_date inclusive by adding 1 day - END AS end_ts -), - -target_pools AS ( - SELECT '7dVri3qjYD3uobSZL3Zth8vSCgU6r6R2nvFsh7uVfDte' AS pool -- Umbra / USDC - UNION ALL SELECT '59WuweKV7DAg8aUgRhNytScQxioaFYNJdWnox5FxAXFq' -- Ranger / USDC - UNION ALL SELECT '6F88Y6iukU9GuL8CMWnx6YT832vBymNPicJBikQWeYe4' -- Paystream / USDC - UNION ALL SELECT 'BGg7WsK98rhqtTp2uSKMa2yETqgwShFAjyf1RmYqCF7n' -- Loyal / USDC - UNION ALL SELECT '5gB4NPgFB3MHFHSeKN4sbaY6t9MB8ikCe9HyiKYid4Td' -- Avici / USDC - UNION ALL SELECT '57SnL1dxJPgc6TH6DcbRn7Nn5jnYCdcrkpVTy9d5vRuP' -- ZKFG / USDC - UNION ALL SELECT '2zsbECzM7roqnDcuv2TNGpfv5PAnuqGmMo5YPtqmUz5p' -- Solomon / USDC - UNION ALL SELECT 'G63kb4W4nsHFsWExtz5vBipqiESCVYua8PzDmswDNZ8S' -- Omnipair / USDC (DAMM v2, v0.6 config) - UNION ALL SELECT 'AMAYXpujqBETmLz3GRppj5nKdpJLEbLXWbAJq8Zgiqav' -- Superclaw / USDC - UNION ALL SELECT '4cY7i4Pnt8zAgZi5k986AnNdpdHm9hLjnH9xYjRbMybr' -- Futardio cult / USDC -), - -/* First on-chain add_liquidity per pool/owner — supplies position account for newer pools (see meteoraService.ts owners). */ -new_pool_first_position AS ( - SELECT account_pool AS pool, account_owner AS owner, account_position AS position - FROM ( - SELECT - account_pool, - account_owner, - account_position, - ROW_NUMBER() OVER ( - PARTITION BY account_pool, account_owner - ORDER BY call_block_time ASC - ) AS rn - FROM meteora_solana.cp_amm_call_add_liquidity - WHERE account_pool IN ( - 'G63kb4W4nsHFsWExtz5vBipqiESCVYua8PzDmswDNZ8S', - 'AMAYXpujqBETmLz3GRppj5nKdpJLEbLXWbAJq8Zgiqav', - '4cY7i4Pnt8zAgZi5k986AnNdpdHm9hLjnH9xYjRbMybr' - ) - AND account_owner IN ( - '8s6Jdoh7tgUqmU3D2EmpNJHSvuN5U4NybpLAdsiMitwB', - '5ZPnwQDU7dEKdMGqaY5oCQkiuQpwjtYSJNMNpiStTNvU', - 'FeMyhpB3LJuuuA1oLzXFDuZ48EJz46gyyk3w2xuQA8uw' - ) - ) x - WHERE rn = 1 -), - -pool_map AS ( - -- pool, position, owner (one row per pool you want to track) - SELECT '7dVri3qjYD3uobSZL3Zth8vSCgU6r6R2nvFsh7uVfDte' AS pool, - '2b3fM2n9iTPG1xJrPevtdQ7Ju5QHuRbBmmA84k3UF4TA' AS position, - '6VsC8PuKkXm5xo54c2vbrAaSfQipkpGHqNuKTxXFySx6' AS owner - UNION ALL SELECT '59WuweKV7DAg8aUgRhNytScQxioaFYNJdWnox5FxAXFq', - 'GyPSZcXCEGxHrcX5Trs131G13HbwDYZfr2pPAijzAEcg', - '55H1Q1YrHJQ93uhG4jqrBBHx3a8H7TCM8kvf2UM2g5q3' - UNION ALL SELECT '6F88Y6iukU9GuL8CMWnx6YT832vBymNPicJBikQWeYe4', - 'oawFz9eK6eqiTKDuoShc14Tt7sjzgjBY9VGGwpjdNGb', - 'BpXtB2ASf2Tft97ewTd8PayXCqFQ6Wqod33qrwwfK9Vz' - UNION ALL SELECT 'BGg7WsK98rhqtTp2uSKMa2yETqgwShFAjyf1RmYqCF7n', - '5xhd93HfYtsjvDki7ZWs2NSukfKdXzWPVvD7tnQ4Xkb5', - 'AQyyTwCKemeeMu8ZPZFxrXMbVwAYTSbBhi1w4PBrhvYE' - UNION ALL SELECT '5gB4NPgFB3MHFHSeKN4sbaY6t9MB8ikCe9HyiKYid4Td', - '3n3bY2XBcuqXDZ5kXZLKUzFSoSKPJjjZtyDa11CwfDqC', - 'DGgYoUcu1aDZt4GEL5NQiducwHRGbkMWsUzsXh2j622G' - UNION ALL SELECT '57SnL1dxJPgc6TH6DcbRn7Nn5jnYCdcrkpVTy9d5vRuP', - '6PW5FipH8374LuocEAfjLKUJ991hsyBR8UQ1CEYkJgAa', - 'BNvDfXYG2FAyBDYD71Xr9GhKE18MbmhtjsLKsCuXho6z' - UNION ALL SELECT '2zsbECzM7roqnDcuv2TNGpfv5PAnuqGmMo5YPtqmUz5p', - 'w1BDxR4FvN4KryBuJwcEuohYKHkyDzD1beNH3AhF6Wn', - '98SPcyUZ2rqM2dgjCqqSXS4gJrNTLSNUAAVCF38xYj9u' - UNION ALL SELECT pool, position, owner FROM new_pool_first_position -), - -/* ----------------------------- - INIT LIQUIDITY (KEEP THIS) - evtinitializepool lacks owner/position, so we attribute init liquidity - to the first add_liquidity call AFTER init for that pool. - ----------------------------- */ -initial_liquidity AS ( - SELECT - e.evt_block_time AS time, - m.pool, - m.position, - m.owner, - -- if DECIMAL(38,0) overflows for some pools, switch to TRY_CAST(... AS DOUBLE) - CAST(e.liquidity AS DECIMAL(38,0)) AS liquidity_delta - FROM meteora_solana.cp_amm_evt_evtinitializepool e - JOIN pool_map m - ON m.pool = e.pool - WHERE e.evt_block_time >= (SELECT start_ts FROM params) - AND e.evt_block_time < (SELECT end_ts FROM params) -), - -/* ----------------------------- - ADD/REMOVE LIQUIDITY (ROBUST JSON PATHS + NULL-SAFE) - ----------------------------- */ -add_liquidity AS ( - SELECT - call_block_time AS time, - account_pool AS pool, - account_position AS position, - account_owner AS owner, - COALESCE( - TRY_CAST(JSON_EXTRACT_SCALAR(params, '$.AddLiquidityParameters.liquidity_delta') AS DOUBLE), - TRY_CAST(JSON_EXTRACT_SCALAR(params, '$.AddLiquidityParameters2.liquidity_delta') AS DOUBLE), - TRY_CAST(JSON_EXTRACT_SCALAR(params, '$.liquidity_delta') AS DOUBLE), - 0 - ) AS liquidity_delta - FROM meteora_solana.cp_amm_call_add_liquidity - WHERE call_block_time >= (SELECT start_ts FROM params) - AND call_block_time < (SELECT end_ts FROM params) - AND account_pool IN (SELECT pool FROM target_pools) -), - -remove_liquidity AS ( - SELECT - call_block_time AS time, - account_pool AS pool, - account_position AS position, - account_owner AS owner, - -COALESCE( - TRY_CAST(JSON_EXTRACT_SCALAR(params, '$.RemoveLiquidityParameters.liquidity_delta') AS DOUBLE), - TRY_CAST(JSON_EXTRACT_SCALAR(params, '$.RemoveLiquidityParameters2.liquidity_delta') AS DOUBLE), - TRY_CAST(JSON_EXTRACT_SCALAR(params, '$.liquidity_delta') AS DOUBLE), - 0 - ) AS liquidity_delta - FROM meteora_solana.cp_amm_call_remove_liquidity - WHERE call_block_time >= (SELECT start_ts FROM params) - AND call_block_time < (SELECT end_ts FROM params) - AND account_pool IN (SELECT pool FROM target_pools) -), - -all_liquidity_changes AS ( - SELECT * FROM initial_liquidity - UNION ALL SELECT * FROM add_liquidity - UNION ALL SELECT * FROM remove_liquidity -), - -liquidity_deltas_daily AS ( - SELECT - DATE_TRUNC('day', time) AS day, - pool, - position, - owner, - SUM(COALESCE(liquidity_delta, 0)) AS liquidity_delta_day - FROM all_liquidity_changes - GROUP BY 1,2,3,4 -), - -position_liquidity_daily AS ( - SELECT - day, - pool, - position, - owner, - SUM(COALESCE(liquidity_delta_day, 0)) OVER ( - PARTITION BY pool, position, owner - ORDER BY day - ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW - ) AS cumulative_liquidity - FROM liquidity_deltas_daily -), - -pool_liquidity_daily AS ( - SELECT - day, - pool, - SUM(cumulative_liquidity) AS total_pool_liquidity - FROM position_liquidity_daily - WHERE cumulative_liquidity > 0 - GROUP BY 1,2 -), - -ownership_shares_daily AS ( - SELECT - p.day, - p.pool, - p.position, - p.owner, - p.cumulative_liquidity, - t.total_pool_liquidity, - CAST(p.cumulative_liquidity AS DOUBLE) / NULLIF(CAST(t.total_pool_liquidity AS DOUBLE), 0) AS ownership_share - FROM position_liquidity_daily p - JOIN pool_liquidity_daily t - ON p.day = t.day AND p.pool = t.pool - WHERE p.cumulative_liquidity > 0 -), - -/* ----------------------------- - SWAPS (UNCHANGED LOGIC, JUST MULTI-POOL) - ----------------------------- */ -swaps_union AS ( - SELECT - DATE_TRUNC('day', evt_block_time) AS day, - pool, - trade_direction, - CASE WHEN trade_direction = 0 THEN CAST(JSON_EXTRACT_SCALAR(params, '$.SwapParameters.amount_in') AS DOUBLE) END AS usdc_in_raw, - CASE WHEN trade_direction = 1 THEN CAST(JSON_EXTRACT_SCALAR(swap_result, '$.SwapResult.output_amount') AS DOUBLE) END AS usdc_out_raw, - CASE WHEN trade_direction = 1 THEN CAST(JSON_EXTRACT_SCALAR(params, '$.SwapParameters.amount_in') AS DOUBLE) END AS token_in_raw, - CASE WHEN trade_direction = 0 THEN CAST(JSON_EXTRACT_SCALAR(swap_result, '$.SwapResult.output_amount') AS DOUBLE) END AS token_out_raw, - CASE WHEN trade_direction = 0 THEN CAST(JSON_EXTRACT_SCALAR(swap_result, '$.SwapResult.lp_fee') AS DOUBLE) END AS lp_fee_usdc_raw, - CASE WHEN trade_direction = 1 THEN CAST(JSON_EXTRACT_SCALAR(swap_result, '$.SwapResult.lp_fee') AS DOUBLE) END AS lp_fee_token_raw - FROM meteora_solana.cp_amm_evt_evtswap - WHERE evt_block_time >= (SELECT start_ts FROM params) - AND evt_block_time < (SELECT end_ts FROM params) - AND pool IN (SELECT pool FROM target_pools) - - UNION ALL - - SELECT - DATE_TRUNC('day', evt_block_time) AS day, - pool, - trade_direction, - CASE WHEN trade_direction = 0 THEN CAST(JSON_EXTRACT_SCALAR(params, '$.SwapParameters2.amount_0') AS DOUBLE) END AS usdc_in_raw, - CASE WHEN trade_direction = 1 THEN CAST(JSON_EXTRACT_SCALAR(swap_result, '$.SwapResult2.output_amount') AS DOUBLE) END AS usdc_out_raw, - CASE WHEN trade_direction = 1 THEN CAST(JSON_EXTRACT_SCALAR(params, '$.SwapParameters2.amount_0') AS DOUBLE) END AS token_in_raw, - CASE WHEN trade_direction = 0 THEN CAST(JSON_EXTRACT_SCALAR(swap_result, '$.SwapResult2.output_amount') AS DOUBLE) END AS token_out_raw, - CASE WHEN trade_direction = 0 THEN CAST(JSON_EXTRACT_SCALAR(swap_result, '$.SwapResult2.trading_fee') AS DOUBLE) END AS lp_fee_usdc_raw, - CASE WHEN trade_direction = 1 THEN CAST(JSON_EXTRACT_SCALAR(swap_result, '$.SwapResult2.trading_fee') AS DOUBLE) END AS lp_fee_token_raw - FROM meteora_solana.cp_amm_evt_evtswap2 - WHERE evt_block_time >= (SELECT start_ts FROM params) - AND evt_block_time < (SELECT end_ts FROM params) - AND pool IN (SELECT pool FROM target_pools) -), - -daily_fees AS ( - SELECT - day, - pool, - COUNT(*) AS num_swaps, - (SUM(COALESCE(usdc_in_raw, 0)) + SUM(COALESCE(usdc_out_raw, 0))) / 1e6 AS volume_usd_approx, - SUM(COALESCE(lp_fee_usdc_raw, 0)) / 1e6 AS lp_fee_usdc, - SUM(COALESCE(lp_fee_token_raw, 0)) / 1e6 AS lp_fee_token, - - 1 / NULLIF( - COALESCE( - (SUM(COALESCE(token_in_raw, 0)) / 1e6) / NULLIF(SUM(COALESCE(usdc_out_raw, 0)) / 1e6, 0), - (SUM(COALESCE(token_out_raw, 0)) / 1e6) / NULLIF(SUM(COALESCE(usdc_in_raw, 0)) / 1e6, 0) - ), - 0 - ) AS token_per_usdc_raw, - - COALESCE( - (SUM(COALESCE(token_in_raw, 0)) / 1e6) / NULLIF(SUM(COALESCE(usdc_out_raw, 0)) / 1e6, 0), - (SUM(COALESCE(token_out_raw, 0)) / 1e6) / NULLIF(SUM(COALESCE(usdc_in_raw, 0)) / 1e6, 0) - ) AS token_price_usdc, - - (SUM(COALESCE(lp_fee_token_raw, 0)) / 1e6) * - COALESCE( - (SUM(COALESCE(token_in_raw, 0)) / 1e6) / NULLIF(SUM(COALESCE(usdc_out_raw, 0)) / 1e6, 0), - (SUM(COALESCE(token_out_raw, 0)) / 1e6) / NULLIF(SUM(COALESCE(usdc_in_raw, 0)) / 1e6, 0) - ) AS lp_fee_token_usdc, - - (SUM(COALESCE(lp_fee_usdc_raw, 0)) / 1e6) - + - ( - (SUM(COALESCE(lp_fee_token_raw, 0)) / 1e6) * - COALESCE( - (SUM(COALESCE(token_in_raw, 0)) / 1e6) / NULLIF(SUM(COALESCE(usdc_out_raw, 0)) / 1e6, 0), - (SUM(COALESCE(token_out_raw, 0)) / 1e6) / NULLIF(SUM(COALESCE(usdc_in_raw, 0)) / 1e6, 0) - ) - ) AS lp_fee_total_usdc, - - -- Buy volume: sum of USDC in for trade_direction=0 (buy) - SUM(CASE WHEN trade_direction = 0 THEN COALESCE(usdc_in_raw, 0) ELSE 0 END) / 1e6 AS buy_volume, - -- Sell volume: sum of USDC out for trade_direction=1 (sell) - USDC received when selling tokens - SUM(CASE WHEN trade_direction = 1 THEN COALESCE(usdc_out_raw, 0) ELSE 0 END) / 1e6 AS sell_volume - FROM swaps_union - GROUP BY 1,2 -), - -/* ----------------------------- - AS-OF OWNERSHIP: last known share <= fee day - This fixes "no results" when swaps happen on days without LP events. - Gets the most recent ownership share for each owner/position on or before each fee day. - Ensures all pools from pool_map are included with their owners. - ----------------------------- */ -ownership_shares_asof AS ( - SELECT day, pool, owner, position, ownership_share - FROM ( - SELECT - f.day, - f.pool, - m.owner, - m.position, - COALESCE(o.ownership_share, 0) AS ownership_share, - ROW_NUMBER() OVER ( - PARTITION BY f.day, f.pool, m.owner - ORDER BY o.day DESC NULLS LAST - ) AS rn - FROM daily_fees f - CROSS JOIN pool_map m - LEFT JOIN ownership_shares_daily o - ON o.pool = f.pool - AND o.owner = m.owner - AND o.day <= f.day - AND o.ownership_share > 0 - WHERE f.pool = m.pool - ) x - WHERE rn = 1 -), - -fees_earned AS ( - SELECT - f.day, - f.pool, - o.owner, - o.position, - o.ownership_share, - f.num_swaps, - f.volume_usd_approx, - f.lp_fee_usdc, - f.lp_fee_token, - f.token_per_usdc_raw, - f.token_price_usdc AS avg_price_usdc, - f.lp_fee_token_usdc, - f.lp_fee_total_usdc, - f.buy_volume, - f.sell_volume, - f.lp_fee_total_usdc * COALESCE(o.ownership_share, 0) AS earned_fee_usdc - FROM daily_fees f - JOIN ownership_shares_asof o - ON f.day = o.day AND f.pool = o.pool -) - -SELECT - CAST(day AS DATE) AS day, - owner, - pool, - num_swaps, - buy_volume, - sell_volume, - volume_usd_approx, - lp_fee_usdc, - lp_fee_token, - lp_fee_token_usdc, - token_per_usdc_raw, - avg_price_usdc AS token_price_usdc, - earned_fee_usdc -FROM fees_earned -ORDER BY day, pool, owner; diff --git a/src/services/dailyMeteoraVolumeService.ts b/src/services/dailyMeteoraVolumeService.ts deleted file mode 100644 index 90b94fd..0000000 --- a/src/services/dailyMeteoraVolumeService.ts +++ /dev/null @@ -1,415 +0,0 @@ -/** - * MeteoraVolumeFetcherService - * - * Fetches daily Meteora pool fees and volumes from Dune API per owner (mapped to token). - * Stores data in PostgreSQL. - * Cumulative values are calculated in the database, not from Dune. - * - * Schedule: Daily at 00:00 UTC (midnight) - */ - -import { config } from '../config.js'; -import { DuneService } from './duneService.js'; -import { DatabaseService, DailyMeteoraVolumeRecord } from './databaseService.js'; -import { getTokenForOwner } from './meteoraService.js'; -import { logger } from '../utils/logger.js'; - -export class MeteoraVolumeFetcherService { - private duneService: DuneService; - private databaseService: DatabaseService; - private initialized: boolean = false; - private refreshTimer: NodeJS.Timeout | null = null; - private isRefreshing: boolean = false; - - constructor( - duneService: DuneService, - databaseService: DatabaseService - ) { - this.duneService = duneService; - this.databaseService = databaseService; - } - - /** - * Initialize the service - check database for existing data - */ - async initialize(): Promise { - if (!this.databaseService.isAvailable()) { - logger.info('[MeteoraVolume] Database not connected - service disabled'); - return; - } - - logger.info('[MeteoraVolume] Initializing daily Meteora volume service...'); - - try { - const recordCount = await this.databaseService.getMeteoraRecordCount(); - logger.info(`[MeteoraVolume] Current record count in database: ${recordCount}`); - - if (recordCount > 0) { - this.initialized = true; - logger.info('[MeteoraVolume] Service initialized with existing database records.'); - } else { - logger.info('[MeteoraVolume] No existing data - performing initial backfill...'); - await this.backfillFromStart(); - this.initialized = true; - logger.info('[MeteoraVolume] Initialization complete with backfill.'); - } - } catch (error: any) { - logger.error('[MeteoraVolume] Error during initialization:', error); - // Still mark as initialized if we have existing data - const recordCount = await this.databaseService.getMeteoraRecordCount(); - if (recordCount > 0) { - this.initialized = true; - logger.info('[MeteoraVolume] Serving existing data despite initialization error.'); - } - } - } - - /** - * Start the scheduled refresh process - */ - start(): void { - if (!this.databaseService.isAvailable()) { - logger.info('[MeteoraVolume] Service not starting - database not available'); - return; - } - - logger.info('[MeteoraVolume] Starting daily Meteora volume service...'); - this.refresh().catch(err => { - logger.error('[MeteoraVolume] Background refresh error', err); - }); - - this.scheduleDailyRefresh(); - logger.info('[MeteoraVolume] Service started - will refresh daily at 00:00 UTC'); - } - - /** - * Stop the service - */ - stop(): void { - if (this.refreshTimer) { - clearTimeout(this.refreshTimer); - this.refreshTimer = null; - } - logger.info('[MeteoraVolume] Service stopped'); - } - - /** - * Schedule the next refresh at 00:00 UTC (midnight) - */ - private scheduleDailyRefresh(): void { - const now = new Date(); - const nextRefresh = new Date(now); - - // Set to 00:00 UTC (midnight) - nextRefresh.setUTCHours(0, 0, 0, 0); - - // If we're past midnight today, schedule for tomorrow - if (now >= nextRefresh) { - nextRefresh.setUTCDate(nextRefresh.getUTCDate() + 1); - } - - const msUntilRefresh = nextRefresh.getTime() - now.getTime(); - logger.info(`[MeteoraVolume] Next refresh scheduled for ${nextRefresh.toISOString()} (in ${Math.round(msUntilRefresh / 60000)} minutes)`); - - this.refreshTimer = setTimeout(async () => { - await this.refresh(); - // Re-schedule for the next day - this.scheduleDailyRefresh(); - }, msUntilRefresh); - } - - /** - * Perform incremental refresh - fetch data from Dune - */ - async refresh(): Promise { - if (this.isRefreshing) { - logger.info('[MeteoraVolume] Refresh already in progress, skipping...'); - return; - } - - this.isRefreshing = true; - - try { - if (!this.databaseService.isAvailable()) { - logger.info('[MeteoraVolume] Database not available, skipping refresh'); - return; - } - - const lastCompleteDate = await this.databaseService.getLatestMeteoraDate(); - - const startDate = lastCompleteDate - ? this.addDays(lastCompleteDate, 1) - : '2025-10-09'; - - // Constrain to today for daily fetches - const today = new Date().toISOString().split('T')[0]!; - const recordsUpserted = await this.fetchAndStore(startDate, today); - - await this.databaseService.markMeteoraDaysComplete(today); - - logger.info(`[MeteoraVolume] Refresh completed successfully - ${recordsUpserted} records`); - } catch (error: any) { - logger.error('[MeteoraVolume] Refresh error', error); - } finally { - this.isRefreshing = false; - } - } - - /** - * Force a full refresh (useful for manual triggering) - */ - async forceRefresh(): Promise<{ success: boolean; message: string; recordsUpserted?: number }> { - if (this.isRefreshing) { - return { success: false, message: 'Refresh already in progress' }; - } - - this.isRefreshing = true; - - try { - const lastCompleteDate = await this.databaseService.getLatestMeteoraDate(); - const startDate = lastCompleteDate - ? this.addDays(lastCompleteDate, 1) - : '2025-10-09'; - - // Constrain to today for force refresh - const today = new Date().toISOString().split('T')[0]!; - const recordsUpserted = await this.fetchAndStore(startDate, today); - - await this.databaseService.markMeteoraDaysComplete(today); - - this.initialized = true; - return { - success: true, - message: `Refresh completed, fetched from ${startDate}`, - recordsUpserted - }; - } catch (error: any) { - return { success: false, message: error.message }; - } finally { - this.isRefreshing = false; - } - } - - /** - * Backfill from the very beginning (2025-10-09) - */ - private async backfillFromStart(): Promise { - logger.info('[MeteoraVolume] Starting full backfill from Dune...'); - try { - // For full backfill, don't constrain end date - fetch all available data - const recordsUpserted = await this.fetchAndStore('2025-10-09'); - logger.info(`[MeteoraVolume] Backfill complete, upserted ${recordsUpserted} records`); - - const today = new Date().toISOString().split('T')[0]!; - await this.databaseService.markMeteoraDaysComplete(today); - } catch (error: any) { - logger.error('[MeteoraVolume] Backfill failed', error); - throw error; - } - } - - /** - * Public method to fetch data for a specific date range (for chunked backfills) - */ - async fetchDateRange(startDate: string, endDate: string): Promise { - return this.fetchAndStore(startDate, endDate); - } - - /** - * Fetch data from Dune and store in database - */ - private async fetchAndStore(startDate: string, endDate?: string): Promise { - if (!config.dune.meteoraVolumeQueryId) { - throw new Error('No DUNE_METEORA_VOLUME_QUERY_ID configured'); - } - - const params: Record = { - start_date: startDate, - }; - - // Only include end_date if it's explicitly provided and not empty - if (endDate && endDate.trim() !== '') { - params.end_date = endDate; - } - - logger.info(`[MeteoraVolume] Fetching from Dune query ${config.dune.meteoraVolumeQueryId}`); - logger.info(`[MeteoraVolume] Parameters: start_date=${startDate}, end_date=${endDate || '(unconstrained)'}`); - - let result; - try { - result = await (this.duneService as any).executeQueryManually( - config.dune.meteoraVolumeQueryId, - params - ); - } catch (duneError: any) { - logger.error('[MeteoraVolume] Dune query execution failed', duneError); - throw duneError; - } - - if (!result) { - logger.info('[MeteoraVolume] Dune returned null result'); - return 0; - } - - if (!result.rows) { - logger.info('[MeteoraVolume] Dune result has no rows property:', { preview: JSON.stringify(result).slice(0, 500) }); - return 0; - } - - if (result.rows.length === 0) { - logger.info('[MeteoraVolume] No data returned from Dune (empty rows array)'); - return 0; - } - - logger.info(`[MeteoraVolume] Received ${result.rows.length} rows from Dune`); - - try { - if (result.rows.length > 0 && result.rows[0]) { - const sampleRow = result.rows[0] as unknown as Record; - logger.debug('[MeteoraVolume] Sample row fields:', { fields: Object.keys(sampleRow) }); - logger.debug('[MeteoraVolume] Sample row:', { row: JSON.stringify(sampleRow) }); - } - } catch (logError: any) { - logger.error('[MeteoraVolume] Error logging sample row', logError); - } - - logger.info('[MeteoraVolume] Transforming rows to database records...'); - let records: DailyMeteoraVolumeRecord[]; - try { - records = result.rows.map((row: any) => { - let rawDate = row.day || row.date || ''; - let dateStr = ''; - - if (rawDate instanceof Date) { - dateStr = rawDate.toISOString().substring(0, 10); - } else if (typeof rawDate === 'string') { - dateStr = rawDate.substring(0, 10); - } - - const owner = row.owner || ''; - const token = getTokenForOwner(owner); - - if (!token) { - logger.warn(`[MeteoraVolume] No token mapping found for owner: ${owner}`); - } - - // Convert scientific notation for all numeric fields - const convertValue = (value: unknown): string => { - if (value === null || value === undefined || value === '' || value === 0 || value === '0') { - return '0'; - } - if (typeof value === 'number') { - if (isNaN(value) || !isFinite(value)) { - return '0'; - } - return value.toFixed(20).replace(/\.?0+$/, ''); - } - if (typeof value === 'string') { - if (value.includes('E') || value.includes('e')) { - try { - const num = parseFloat(value); - if (isNaN(num)) { - return '0'; - } - return num.toFixed(20).replace(/\.?0+$/, ''); - } catch { - return value; - } - } - return value; - } - return String(value); - }; - - // Calculate target_volume from buy_volume + sell_volume if not provided - const buyVolume = parseFloat(convertValue(row.buy_volume || 0)) || 0; - const sellVolume = parseFloat(convertValue(row.sell_volume || 0)) || 0; - const targetVolume = buyVolume + sellVolume; - - return { - token: token || owner, // Use owner as fallback if mapping not found - date: dateStr, - base_volume: convertValue(row.volume_usd_approx || 0), - target_volume: String(targetVolume), - trade_count: parseInt(String(row.num_swaps || 0)) || 0, - buy_volume: convertValue(row.buy_volume || 0), - sell_volume: convertValue(row.sell_volume || 0), - usdc_fees: convertValue(row.lp_fee_usdc || 0), - token_fees: convertValue(row.lp_fee_token || 0), - token_fees_usdc: convertValue(row.lp_fee_token_usdc || 0), - token_per_usdc: convertValue(row.token_per_usdc_raw || 0), - average_price: convertValue(row.token_price_usdc || 0), - ownership_share: convertValue(row.ownership_share || 0), // Default to 0 if not provided by query - earned_fee_usdc: convertValue(row.earned_fee_usdc || 0), - is_complete: false, - }; - }); - } catch (transformError: any) { - logger.error('[MeteoraVolume] Error transforming rows', transformError); - throw transformError; - } - - const validRecords = records.filter(r => r.token && r.date); - const invalidCount = records.length - validRecords.length; - - logger.info(`[MeteoraVolume] Transformed ${records.length} rows, ${validRecords.length} valid, ${invalidCount} invalid`); - - if (invalidCount > 0 && records.length > 0) { - const invalidSample = records.find(r => !r.token || !r.date); - if (invalidSample) { - logger.info('[MeteoraVolume] Sample invalid record:', { record: JSON.stringify(invalidSample) }); - } - } - - if (validRecords.length === 0) { - logger.info('[MeteoraVolume] No valid records to insert'); - return 0; - } - - const today = new Date().toISOString().split('T')[0]!; - const markComplete = true; - - logger.info(`[MeteoraVolume] Upserting ${validRecords.length} records to database...`); - try { - const upserted = await this.databaseService.upsertDailyMeteoraVolumes(validRecords, markComplete); - logger.info(`[MeteoraVolume] Upsert complete: ${upserted} records`); - return upserted; - } catch (upsertError: any) { - logger.error('[MeteoraVolume] Error upserting to database', upsertError); - throw upsertError; - } - } - - /** - * Helper to add days to a date string - */ - private addDays(dateStr: string, days: number): string { - const date = new Date(dateStr); - date.setDate(date.getDate() + days); - return date.toISOString().split('T')[0]!; - } - - /** - * Check if service is ready - */ - isReady(): boolean { - return this.initialized && this.databaseService.isAvailable(); - } - - /** - * Get service status - */ - getStatus(): { - initialized: boolean; - databaseConnected: boolean; - queryIdConfigured: boolean; - isRefreshing: boolean; - } { - return { - initialized: this.initialized, - databaseConnected: this.databaseService.isAvailable(), - queryIdConfigured: !!config.dune.meteoraVolumeQueryId, - isRefreshing: this.isRefreshing, - }; - } -} diff --git a/src/services/database/internal/repos/dailyMeteoraVolumesRepo.ts b/src/services/database/internal/repos/dailyMeteoraVolumesRepo.ts deleted file mode 100644 index d3a5d18..0000000 --- a/src/services/database/internal/repos/dailyMeteoraVolumesRepo.ts +++ /dev/null @@ -1,244 +0,0 @@ -import { logger } from '../../../../utils/logger.js'; -import type { DbRuntime } from '../dbRuntime.js'; -import type { DailyMeteoraVolumeRecord } from '../../../databaseService.js'; - -export function createDailyMeteoraVolumesRepo(db: DbRuntime) { - return { - /** - * Get daily Meteora volumes with date range filtering - * @param options.token Filter by specific token - * @param options.tokens Filter by array of tokens - * @param options.startDate Start date (inclusive) in YYYY-MM-DD format - * @param options.endDate End date (inclusive) in YYYY-MM-DD format - */ - async getDailyMeteoraVolumes(options?: { - token?: string; - tokens?: string[]; - startDate?: string; - endDate?: string; - }): Promise<{ - token: string; - date: string; - base_volume: string; - target_volume: string; - buy_volume: string; - sell_volume: string; - trade_count: number; - average_price: string; - usdc_fees: string; - token_fees: string; - token_fees_usdc: string; - token_per_usdc: string; - }[]> { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return []; - - try { - const conditions: string[] = []; - const params: any[] = []; - let paramIndex = 1; - - // Support both single token and array of tokens - if (options?.tokens && options.tokens.length > 0) { - const placeholders = options.tokens.map((_, i) => `$${paramIndex + i}`).join(', '); - conditions.push(`token IN (${placeholders})`); - params.push(...options.tokens); - paramIndex += options.tokens.length; - } else if (options?.token) { - conditions.push(`token = $${paramIndex}`); - params.push(options.token); - paramIndex++; - } - - if (options?.startDate) { - conditions.push(`date >= $${paramIndex}`); - params.push(options.startDate); - paramIndex++; - } - - if (options?.endDate) { - conditions.push(`date <= $${paramIndex}`); - params.push(options.endDate); - paramIndex++; - } - - const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(' AND ')}` : ''; - - const result = await pool.query( - `SELECT - token, - date::text, - base_volume::text, - target_volume::text, - buy_volume::text, - sell_volume::text, - trade_count, - average_price::text, - usdc_fees::text, - token_fees::text, - token_fees_usdc::text, - token_per_usdc::text - FROM daily_meteora_volumes - ${whereClause} - ORDER BY token, date ASC`, - params - ); - - return result.rows; - } catch (error: any) { - logger.error('[Database] Error getting daily Meteora volumes:', error); - return []; - } - }, - - /** - * Get the latest complete date from daily_meteora_volumes table - */ - async getLatestMeteoraDate(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return null; - - try { - const result = await pool.query( - 'SELECT MAX(date) as latest_date FROM daily_meteora_volumes WHERE is_complete = true' - ); - return result.rows[0]?.latest_date?.toISOString().split('T')[0] || null; - } catch (error: any) { - logger.error('[Database] Error getting latest Meteora date:', error); - return null; - } - }, - - /** - * Upsert daily Meteora volume records using batched inserts - * @param records Array of daily Meteora volume records - * @param markComplete Whether to mark records as complete - * @returns Number of records upserted - */ - async upsertDailyMeteoraVolumes(records: DailyMeteoraVolumeRecord[], markComplete: boolean = false): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected() || records.length === 0) { - return 0; - } - - const BATCH_SIZE = 500; - let totalUpserted = 0; - - try { - const client = await pool.connect(); - - try { - await client.query('BEGIN'); - - for (let i = 0; i < records.length; i += BATCH_SIZE) { - const batch = records.slice(i, i + BATCH_SIZE); - - const values: any[] = []; - const valuePlaceholders: string[] = []; - - batch.forEach((record, idx) => { - const offset = idx * 15; // 15 parameters per record (14 data fields + is_complete) - valuePlaceholders.push( - `($${offset + 1}, $${offset + 2}, $${offset + 3}, $${offset + 4}, $${offset + 5}, $${offset + 6}, $${offset + 7}, $${offset + 8}, $${offset + 9}, $${offset + 10}, $${offset + 11}, $${offset + 12}, $${offset + 13}, $${offset + 14}, $${offset + 15}, CURRENT_TIMESTAMP)` - ); - values.push( - record.token, - record.date, - record.base_volume, - record.target_volume, - record.trade_count, - record.buy_volume, - record.sell_volume, - record.usdc_fees, - record.token_fees, - record.token_fees_usdc, - record.token_per_usdc, - record.average_price, - record.ownership_share, - record.earned_fee_usdc, - markComplete ? true : (record.is_complete ?? false) - ); - }); - - const batchSQL = ` - INSERT INTO daily_meteora_volumes (token, date, base_volume, target_volume, trade_count, buy_volume, sell_volume, usdc_fees, token_fees, token_fees_usdc, token_per_usdc, average_price, ownership_share, earned_fee_usdc, is_complete, updated_at) - VALUES ${valuePlaceholders.join(', ')} - ON CONFLICT (token, date) - DO UPDATE SET - base_volume = EXCLUDED.base_volume, - target_volume = EXCLUDED.target_volume, - trade_count = EXCLUDED.trade_count, - buy_volume = EXCLUDED.buy_volume, - sell_volume = EXCLUDED.sell_volume, - usdc_fees = EXCLUDED.usdc_fees, - token_fees = EXCLUDED.token_fees, - token_fees_usdc = EXCLUDED.token_fees_usdc, - token_per_usdc = EXCLUDED.token_per_usdc, - average_price = EXCLUDED.average_price, - ownership_share = EXCLUDED.ownership_share, - earned_fee_usdc = EXCLUDED.earned_fee_usdc, - is_complete = CASE WHEN EXCLUDED.is_complete THEN true ELSE daily_meteora_volumes.is_complete END, - updated_at = CURRENT_TIMESTAMP - `; - - await client.query(batchSQL, values); - totalUpserted += batch.length; - - if (records.length > BATCH_SIZE) { - logger.info(`[Database] Meteora volume batch progress: ${totalUpserted}/${records.length}`); - } - } - - await client.query('COMMIT'); - logger.info(`[Database] Upserted ${totalUpserted} daily Meteora volume records`); - return totalUpserted; - } catch (error) { - await client.query('ROLLBACK'); - throw error; - } finally { - client.release(); - } - } catch (error: any) { - logger.error('[Database] Error upserting daily Meteora volumes:', error); - return 0; - } - }, - - /** - * Mark Meteora volume days as complete before a given date - */ - async markMeteoraDaysComplete(beforeDate: string): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return; - - try { - await pool.query( - `UPDATE daily_meteora_volumes SET is_complete = true, updated_at = CURRENT_TIMESTAMP - WHERE date < $1 AND is_complete = false`, - [beforeDate] - ); - logger.info(`[Database] Marked Meteora days before ${beforeDate} as complete`); - } catch (error: any) { - logger.error('[Database] Error marking Meteora days complete:', error); - } - }, - - /** - * Get count of Meteora volume records - */ - async getMeteoraRecordCount(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return 0; - - try { - const result = await pool.query('SELECT COUNT(*) as count FROM daily_meteora_volumes'); - return parseInt(result.rows[0]?.count || '0'); - } catch (error: any) { - logger.error('[Database] Error getting Meteora record count:', error); - return 0; - } - }, - }; -} - -export type DailyMeteoraVolumesRepo = ReturnType; diff --git a/src/services/database/internal/schemaManager.ts b/src/services/database/internal/schemaManager.ts index 5aaf18e..5ccac3c 100644 --- a/src/services/database/internal/schemaManager.ts +++ b/src/services/database/internal/schemaManager.ts @@ -155,33 +155,6 @@ export function createSchemaManager(db: DbRuntime) { CREATE INDEX IF NOT EXISTS idx_daily_fees_volumes_date ON daily_fees_volumes(trading_date); CREATE INDEX IF NOT EXISTS idx_daily_fees_volumes_token_date ON daily_fees_volumes(token, trading_date); - -- Daily Meteora volumes table for tracking Meteora pool fees and volumes per owner - CREATE TABLE IF NOT EXISTS daily_meteora_volumes ( - id SERIAL PRIMARY KEY, - token VARCHAR(64) NOT NULL, - date DATE NOT NULL, - base_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - target_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - trade_count INT NOT NULL DEFAULT 0, - buy_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - sell_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - token_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - token_fees_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - token_per_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - average_price NUMERIC(40, 12) NOT NULL DEFAULT 0, - ownership_share NUMERIC(40, 12) NOT NULL DEFAULT 0, - earned_fee_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - is_complete BOOLEAN NOT NULL DEFAULT false, - created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - UNIQUE(token, date) - ); - - CREATE INDEX IF NOT EXISTS idx_daily_meteora_volumes_token ON daily_meteora_volumes(token); - CREATE INDEX IF NOT EXISTS idx_daily_meteora_volumes_date ON daily_meteora_volumes(date); - CREATE INDEX IF NOT EXISTS idx_daily_meteora_volumes_token_date ON daily_meteora_volumes(token, date); - -- Metrics history table for storing periodic snapshots of system metrics CREATE TABLE IF NOT EXISTS metrics_history ( id SERIAL PRIMARY KEY, diff --git a/src/services/databaseService.ts b/src/services/databaseService.ts index 69661dd..be92409 100644 --- a/src/services/databaseService.ts +++ b/src/services/databaseService.ts @@ -8,7 +8,6 @@ import { createDailyVolumesRepo } from './database/internal/repos/dailyVolumesRe import { createIntervalVolumesRepo } from './database/internal/repos/intervalVolumesRepo.js'; import { createDailyBuySellVolumesRepo } from './database/internal/repos/dailyBuySellVolumesRepo.js'; import { createDailyFeesVolumesRepo } from './database/internal/repos/dailyFeesVolumesRepo.js'; -import { createDailyMeteoraVolumesRepo } from './database/internal/repos/dailyMeteoraVolumesRepo.js'; import { createMetricsRepo } from './database/internal/repos/metricsRepo.js'; import { createV06TradingActivityRepo } from './database/internal/repos/v06TradingActivityRepo.js'; @@ -112,24 +111,6 @@ export interface DailyFeesVolumeRecord { low: string; } -export interface DailyMeteoraVolumeRecord { - token: string; // mapped from owner - date: string; // YYYY-MM-DD - base_volume: string; // volume_usd_approx - target_volume: string; // calculated from buy_volume + sell_volume - trade_count: number; // num_swaps - buy_volume: string; - sell_volume: string; - usdc_fees: string; // lp_fee_usdc - token_fees: string; // lp_fee_token - token_fees_usdc: string; // lp_fee_token_usdc - token_per_usdc: string; // token_per_usdc_raw - average_price: string; // token_price_usdc - ownership_share: string; // ownership_share - earned_fee_usdc: string; // earned_fee_usdc - is_complete: boolean; -} - export interface CumulativeVolumeData { token: string; date: string; @@ -166,6 +147,10 @@ export interface TokenVolumeAggregate { daily_data: DailyVolumeRecord[]; } +export interface DatabaseInitializeOptions { + ensureSchema?: boolean; +} + export class DatabaseService { public pool: pg.Pool | null = null; private isConnected: boolean = false; @@ -184,7 +169,6 @@ export class DatabaseService { private _intervalVolumes = createIntervalVolumesRepo(this.dbRuntime); private _dailyBuySellVolumes = createDailyBuySellVolumesRepo(this.dbRuntime); private _dailyFeesVolumes = createDailyFeesVolumesRepo(this.dbRuntime); - private _dailyMeteoraVolumes = createDailyMeteoraVolumesRepo(this.dbRuntime); private _metrics = createMetricsRepo(this.dbRuntime); private _v06TradingActivity = createV06TradingActivityRepo(this.dbRuntime); @@ -224,9 +208,11 @@ export class DatabaseService { } /** - * Initialize the database connection and create tables if needed + * Initialize the database connection. */ - async initialize(): Promise { + async initialize(options: DatabaseInitializeOptions = {}): Promise { + const { ensureSchema = true } = options; + if (!this.pool) { logger.info('[Database] No database configuration provided, volume history will use in-memory cache only'); return false; @@ -238,11 +224,10 @@ export class DatabaseService { logger.info('[Database] Connected to PostgreSQL'); client.release(); - // Create tables - await this._schema.createTables(); - - // Create aggregation functions - await this._schema.createAggregationFunctions(); + if (ensureSchema) { + await this._schema.createTables(); + await this._schema.createAggregationFunctions(); + } this.isConnected = true; this.consecutiveFailures = 0; @@ -578,48 +563,6 @@ export class DatabaseService { return this._dailyFeesVolumes.getFeesRecordCount(); } - // ============================================ - // Daily Meteora volumes (delegated to DailyMeteoraVolumesRepo) - // ============================================ - - async getDailyMeteoraVolumes(options?: { - token?: string; - tokens?: string[]; - startDate?: string; - endDate?: string; - }): Promise<{ - token: string; - date: string; - base_volume: string; - target_volume: string; - buy_volume: string; - sell_volume: string; - trade_count: number; - average_price: string; - usdc_fees: string; - token_fees: string; - token_fees_usdc: string; - token_per_usdc: string; - }[]> { - return this._dailyMeteoraVolumes.getDailyMeteoraVolumes(options); - } - - async getLatestMeteoraDate(): Promise { - return this._dailyMeteoraVolumes.getLatestMeteoraDate(); - } - - async upsertDailyMeteoraVolumes(records: DailyMeteoraVolumeRecord[], markComplete: boolean = false): Promise { - return this._dailyMeteoraVolumes.upsertDailyMeteoraVolumes(records, markComplete); - } - - async markMeteoraDaysComplete(beforeDate: string): Promise { - return this._dailyMeteoraVolumes.markMeteoraDaysComplete(beforeDate); - } - - async getMeteoraRecordCount(): Promise { - return this._dailyMeteoraVolumes.getMeteoraRecordCount(); - } - // ============================================ // Metrics (delegated to MetricsRepo) // ============================================ diff --git a/src/services/externalDatabaseService.ts b/src/services/externalDatabaseService.ts index 5f8cc83..42584da 100644 --- a/src/services/externalDatabaseService.ts +++ b/src/services/externalDatabaseService.ts @@ -1,6 +1,7 @@ import pg from 'pg'; import { config } from '../config.js'; import { logger } from '../utils/logger.js'; +import type { Rolling24hMetrics } from './databaseService.js'; // Force pg to serialize Date parameters as ISO-8601 UTC strings so PostgreSQL // doesn't receive un-parseable local-timezone names like "GMT-0700". @@ -11,7 +12,8 @@ const { Pool } = pg; /** * Read-only connection pool to the external indexer database. * Used by v0.6 reconciliation to query v0_6_spot_swaps, - * v0_6_conditional_swaps, v0_6_daos, and v0_6_proposals. + * v0_6_conditional_swaps, v0_6_daos, and v0_6_proposals, and by /api/tickers to + * read rolling-24h spot metrics directly from futarchy.trades. */ export class ExternalDatabaseService { private pool: pg.Pool | null = null; @@ -55,6 +57,138 @@ export class ExternalDatabaseService { return this.pool.query(text, params); } + /** + * Rolling 24h spot-market volume metrics per DAO, read directly from the + * indexer's per-swap futarchy.trades table (no Dune, no app-DB rollup). + * + * Keyed by dao_addr (= the ticker's pool_id), so the caller needs no + * token→dao remap. Amounts are raw on-chain integers, so base/quote volume + * are scaled by each side's real decimals (futarchy.tokens.decimals, default + * 6) and price (raw quote/base) is rescaled to human units by + * 10^(baseDecimals − quoteDecimals) — matching the decimal-aware last_price. + * + * Returns an empty Map if the connection is down or no spot trades exist in + * the window (caller treats that as "fall back to another source"). + */ + async getSpotRolling24hMetrics(daoAddrs: string[]): Promise> { + if (!this.pool || !this.isConnected || daoAddrs.length === 0) { + return new Map(); + } + + const cutoff = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); + + try { + const result = await this.pool.query( + `SELECT + t.dao_addr, + (SUM(t.base_amount) / power(10::numeric, COALESCE(bt.decimals, 6)))::text AS base_volume_24h, + (SUM(t.quote_amount) / power(10::numeric, COALESCE(qt.decimals, 6)))::text AS target_volume_24h, + (MAX(t.price) * power(10::numeric, COALESCE(bt.decimals, 6) - COALESCE(qt.decimals, 6)))::text AS high_24h, + (MIN(t.price) FILTER (WHERE t.price > 0) * power(10::numeric, COALESCE(bt.decimals, 6) - COALESCE(qt.decimals, 6)))::text AS low_24h, + COUNT(*)::int AS trade_count_24h + FROM futarchy.trades t + JOIN futarchy.daos d ON d.dao_addr = t.dao_addr + LEFT JOIN futarchy.tokens bt ON bt.mint = d.base_mint + LEFT JOIN futarchy.tokens qt ON qt.mint = d.quote_mint + WHERE t.market_kind = 'spot' + AND t.block_time >= $1 + AND t.dao_addr = ANY($2::text[]) + GROUP BY t.dao_addr, bt.decimals, qt.decimals`, + [cutoff, daoAddrs] + ); + + const metricsMap = new Map(); + for (const row of result.rows) { + metricsMap.set(row.dao_addr, { + token: row.dao_addr, + base_volume_24h: row.base_volume_24h ?? '0', + target_volume_24h: row.target_volume_24h ?? '0', + high_24h: row.high_24h ?? '0', + low_24h: row.low_24h ?? '0', + trade_count_24h: row.trade_count_24h ?? 0, + }); + } + return metricsMap; + } catch (error: any) { + logger.error('[ExternalDB] Error getting spot rolling 24h metrics from futarchy.trades:', error); + return new Map(); + } + } + + /** + * Daily Meteora volumes read DIRECTLY from the meteora accounting ETL's + * `futarchy.meteora_daily` view in our served DB — the source of truth that + * replaces the Dune-sourced `daily_meteora_volumes` app-DB table. + * + * Returns the SAME column contract as the old (Dune) path so /api/market-data + * consumers are unchanged: token (base mint), date, base_volume, target_volume, + * buy_volume, sell_volume, trade_count, average_price, usdc_fees, token_fees, + * token_fees_usdc, token_per_usdc. Returns an empty array if the connection is + * down (the route serves an empty meteora list in that case). + */ + async getDailyMeteoraVolumes(options?: { + token?: string; + tokens?: string[]; + startDate?: string; + endDate?: string; + }): Promise> { + if (!this.pool || !this.isConnected) { + return []; + } + const conditions: string[] = []; + const params: any[] = []; + let i = 1; + if (options?.tokens && options.tokens.length > 0) { + conditions.push(`token = ANY($${i}::text[])`); + params.push(options.tokens); + i++; + } else if (options?.token) { + conditions.push(`token = $${i}`); + params.push(options.token); + i++; + } + if (options?.startDate) { + conditions.push(`date >= $${i}`); + params.push(options.startDate); + i++; + } + if (options?.endDate) { + conditions.push(`date <= $${i}`); + params.push(options.endDate); + i++; + } + const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(' AND ')}` : ''; + try { + const result = await this.pool.query( + `SELECT + token, + date::text, + base_volume::text, + target_volume::text, + buy_volume::text, + sell_volume::text, + trade_count, + average_price::text, + usdc_fees::text, + token_fees::text, + token_fees_usdc::text, + token_per_usdc::text + FROM futarchy.meteora_daily + ${whereClause} + ORDER BY token, date ASC`, + params + ); + return result.rows; + } catch (error: any) { + logger.error('[ExternalDB] Error getting daily Meteora volumes from futarchy.meteora_daily:', error); + return []; + } + } + async close(): Promise { if (this.healthCheckInterval) { clearInterval(this.healthCheckInterval); diff --git a/src/services/meteoraService.ts b/src/services/meteoraService.ts index 7ce707d..78c880b 100644 --- a/src/services/meteoraService.ts +++ b/src/services/meteoraService.ts @@ -4,9 +4,6 @@ * Provides mapping from Meteora pool owner addresses to token (baseMint) addresses. * This mapping is used to normalize Meteora pool data to match existing table structures. * - * Keep `src/schema/dune-meteora-volumes.sql` in sync: every tracked pool must appear in - * `target_pools` + `pool_map` there, or Dune will never return rows for that LP. - * * DAMM v2 pool addresses for base/USDC use the v0.6 Meteora config (same as existing * tracked launches in the Dune query); v0.7 config yields a different pool PDA. */ diff --git a/src/services/meteoraVolumeFetcherService.ts b/src/services/meteoraVolumeFetcherService.ts deleted file mode 100644 index b65443f..0000000 --- a/src/services/meteoraVolumeFetcherService.ts +++ /dev/null @@ -1,419 +0,0 @@ -/** - * MeteoraVolumeFetcherService - * - * Fetches daily Meteora pool fees and volumes from Dune API per owner (mapped to token). - * Stores data in PostgreSQL. - * Cumulative values are calculated in the database, not from Dune. - * - * Schedule: Daily at 00:00 UTC (midnight) - */ - -import { config } from '../config.js'; -import { DuneService } from './duneService.js'; -import { DatabaseService, DailyMeteoraVolumeRecord } from './databaseService.js'; -import { getTokenForOwner } from './meteoraService.js'; -import { logger } from '../utils/logger.js'; - -export class MeteoraVolumeFetcherService { - private duneService: DuneService; - private databaseService: DatabaseService; - private initialized: boolean = false; - private refreshTimer: NodeJS.Timeout | null = null; - private isRefreshing: boolean = false; - - constructor( - duneService: DuneService, - databaseService: DatabaseService - ) { - this.duneService = duneService; - this.databaseService = databaseService; - } - - /** - * Initialize the service - check database for existing data - */ - async initialize(): Promise { - if (!this.databaseService.isAvailable()) { - logger.info('[MeteoraVolume] Database not connected - service disabled'); - return; - } - - logger.info('[MeteoraVolume] Initializing daily Meteora volume service...'); - - try { - const recordCount = await this.databaseService.getMeteoraRecordCount(); - logger.info(`[MeteoraVolume] Current record count in database: ${recordCount}`); - - if (recordCount > 0) { - this.initialized = true; - logger.info('[MeteoraVolume] Service initialized with existing database records.'); - } else { - logger.info('[MeteoraVolume] No existing data - performing initial backfill...'); - await this.backfillFromStart(); - this.initialized = true; - logger.info('[MeteoraVolume] Initialization complete with backfill.'); - } - } catch (error: any) { - logger.error('[MeteoraVolume] Error during initialization:', error); - const recordCount = await this.databaseService.getMeteoraRecordCount(); - if (recordCount > 0) { - this.initialized = true; - logger.info('[MeteoraVolume] Serving existing data despite initialization error.'); - } - } - } - - /** - * Start the scheduled refresh process - */ - start(): void { - if (!this.databaseService.isAvailable()) { - logger.info('[MeteoraVolume] Service not starting - database not available'); - return; - } - - logger.info('[MeteoraVolume] Starting daily Meteora volume service...'); - this.refresh().catch(err => { - logger.error('[MeteoraVolume] Background refresh error', err); - }); - - this.scheduleDailyRefresh(); - logger.info('[MeteoraVolume] Service started - will refresh daily at 00:00 UTC'); - } - - /** - * Stop the service - */ - stop(): void { - if (this.refreshTimer) { - clearTimeout(this.refreshTimer); - this.refreshTimer = null; - } - logger.info('[MeteoraVolume] Service stopped'); - } - - /** - * Schedule the next refresh at 00:00 UTC (midnight) - */ - private scheduleDailyRefresh(): void { - const now = new Date(); - const nextRefresh = new Date(now); - - // Set to 00:00 UTC (midnight) - nextRefresh.setUTCHours(0, 0, 0, 0); - - // If we're past midnight today, schedule for tomorrow - if (now >= nextRefresh) { - nextRefresh.setUTCDate(nextRefresh.getUTCDate() + 1); - } - - const msUntilRefresh = nextRefresh.getTime() - now.getTime(); - logger.info(`[MeteoraVolume] Next refresh scheduled for ${nextRefresh.toISOString()} (in ${Math.round(msUntilRefresh / 60000)} minutes)`); - - this.refreshTimer = setTimeout(async () => { - await this.refresh(); - // Re-schedule for the next day - this.scheduleDailyRefresh(); - }, msUntilRefresh); - } - - /** - * Perform incremental refresh - fetch data from Dune - */ - async refresh(): Promise { - if (this.isRefreshing) { - logger.info('[MeteoraVolume] Refresh already in progress, skipping...'); - return; - } - - this.isRefreshing = true; - - try { - if (!this.databaseService.isAvailable()) { - logger.info('[MeteoraVolume] Database not available, skipping refresh'); - return; - } - - const lastCompleteDate = await this.databaseService.getLatestMeteoraDate(); - - const startDate = lastCompleteDate - ? this.addDays(lastCompleteDate, 1) - : '2025-10-09'; - - // Constrain to today for daily fetches - const today = new Date().toISOString().split('T')[0]!; - const recordsUpserted = await this.fetchAndStore(startDate, today); - - await this.databaseService.markMeteoraDaysComplete(today); - - logger.info(`[MeteoraVolume] Refresh completed successfully - ${recordsUpserted} records`); - } catch (error: any) { - logger.error('[MeteoraVolume] Refresh error', error); - } finally { - this.isRefreshing = false; - } - } - - /** - * Force a full refresh (useful for manual triggering) - */ - async forceRefresh(): Promise<{ success: boolean; message: string; recordsUpserted?: number }> { - if (this.isRefreshing) { - return { success: false, message: 'Refresh already in progress' }; - } - - this.isRefreshing = true; - - try { - const lastCompleteDate = await this.databaseService.getLatestMeteoraDate(); - const startDate = lastCompleteDate - ? this.addDays(lastCompleteDate, 1) - : '2025-10-09'; - - // Constrain to today for force refresh - const today = new Date().toISOString().split('T')[0]!; - const recordsUpserted = await this.fetchAndStore(startDate, today); - - await this.databaseService.markMeteoraDaysComplete(today); - - this.initialized = true; - return { - success: true, - message: `Refresh completed, fetched from ${startDate}`, - recordsUpserted - }; - } catch (error: any) { - return { success: false, message: error.message }; - } finally { - this.isRefreshing = false; - } - } - - /** - * Backfill from the very beginning (2025-10-09) - */ - private async backfillFromStart(): Promise { - logger.info('[MeteoraVolume] Starting full backfill from Dune...'); - try { - // For full backfill, don't constrain end date - fetch all available data - const recordsUpserted = await this.fetchAndStore('2025-10-09'); - logger.info(`[MeteoraVolume] Backfill complete, upserted ${recordsUpserted} records`); - - const today = new Date().toISOString().split('T')[0]!; - await this.databaseService.markMeteoraDaysComplete(today); - } catch (error: any) { - logger.error('[MeteoraVolume] Backfill failed', error); - throw error; - } - } - - /** - * Public method to fetch data for a specific date range (for chunked backfills) - */ - async fetchDateRange(startDate: string, endDate: string): Promise { - return this.fetchAndStore(startDate, endDate); - } - - /** - * Fetch data from Dune and store in database - */ - private async fetchAndStore(startDate: string, endDate?: string): Promise { - if (!config.dune.meteoraVolumeQueryId) { - throw new Error('No DUNE_METEORA_VOLUME_QUERY_ID configured'); - } - - const params: Record = { - start_date: startDate, - }; - - // Only include end_date if it's explicitly provided - // If not provided, use sentinel value '9999-12-31' to indicate unconstrained query - if (endDate !== undefined) { - params.end_date = endDate; - } else { - params.end_date = '9999-12-31'; // Sentinel value means no constraint - } - - logger.info(`[MeteoraVolume] Fetching from Dune query ${config.dune.meteoraVolumeQueryId}`); - logger.info(`[MeteoraVolume] Parameters: start_date=${startDate}, end_date=${endDate || '(unconstrained)'}`); - - let result; - try { - result = await (this.duneService as any).executeQueryManually( - config.dune.meteoraVolumeQueryId, - params - ); - } catch (duneError: any) { - logger.error('[MeteoraVolume] Dune query execution failed', duneError); - throw duneError; - } - - if (!result) { - logger.info('[MeteoraVolume] Dune returned null result'); - return 0; - } - - if (!result.rows) { - logger.info('[MeteoraVolume] Dune result has no rows property:', { preview: JSON.stringify(result).slice(0, 500) }); - return 0; - } - - if (result.rows.length === 0) { - logger.info('[MeteoraVolume] No data returned from Dune (empty rows array)'); - return 0; - } - - logger.info(`[MeteoraVolume] Received ${result.rows.length} rows from Dune`); - - try { - if (result.rows.length > 0 && result.rows[0]) { - const sampleRow = result.rows[0] as unknown as Record; - logger.debug('[MeteoraVolume] Sample row fields:', { fields: Object.keys(sampleRow) }); - logger.debug('[MeteoraVolume] Sample row:', { row: JSON.stringify(sampleRow) }); - } - } catch (logError: any) { - logger.error('[MeteoraVolume] Error logging sample row', logError); - } - - logger.info('[MeteoraVolume] Transforming rows to database records...'); - let records: DailyMeteoraVolumeRecord[]; - try { - records = result.rows.map((row: any) => { - // Parse date - extract YYYY-MM-DD from various formats - let rawDate = row.day || row.date || ''; - let dateStr = ''; - - if (rawDate instanceof Date) { - dateStr = rawDate.toISOString().substring(0, 10); - } else if (typeof rawDate === 'string') { - dateStr = rawDate.substring(0, 10); - } - - // Map owner to token - const owner = row.owner || ''; - const token = getTokenForOwner(owner); - - if (!token) { - logger.warn(`[MeteoraVolume] No token mapping found for owner: ${owner}`); - } - - // Convert scientific notation for all numeric fields - const convertValue = (value: unknown): string => { - if (value === null || value === undefined || value === '' || value === 0 || value === '0') { - return '0'; - } - if (typeof value === 'number') { - if (isNaN(value) || !isFinite(value)) { - return '0'; - } - return value.toFixed(20).replace(/\.?0+$/, ''); - } - if (typeof value === 'string') { - if (value.includes('E') || value.includes('e')) { - try { - const num = parseFloat(value); - if (isNaN(num)) { - return '0'; - } - return num.toFixed(20).replace(/\.?0+$/, ''); - } catch { - return value; - } - } - return value; - } - return String(value); - }; - - // Calculate target_volume from buy_volume + sell_volume if not provided - const buyVolume = parseFloat(convertValue(row.buy_volume || 0)) || 0; - const sellVolume = parseFloat(convertValue(row.sell_volume || 0)) || 0; - const targetVolume = buyVolume + sellVolume; - - return { - token: token || owner, // Use owner as fallback if mapping not found - date: dateStr, - base_volume: convertValue(row.volume_usd_approx || 0), - target_volume: String(targetVolume), - trade_count: parseInt(String(row.num_swaps || 0)) || 0, - buy_volume: convertValue(row.buy_volume || 0), - sell_volume: convertValue(row.sell_volume || 0), - usdc_fees: convertValue(row.lp_fee_usdc || 0), - token_fees: convertValue(row.lp_fee_token || 0), - token_fees_usdc: convertValue(row.lp_fee_token_usdc || 0), - token_per_usdc: convertValue(row.token_per_usdc_raw || 0), - average_price: convertValue(row.token_price_usdc || 0), - ownership_share: convertValue(row.ownership_share || 0), - earned_fee_usdc: convertValue(row.earned_fee_usdc || 0), - is_complete: false, - }; - }); - } catch (transformError: any) { - logger.error('[MeteoraVolume] Error transforming rows', transformError); - throw transformError; - } - - const validRecords = records.filter(r => r.token && r.date); - const invalidCount = records.length - validRecords.length; - - logger.info(`[MeteoraVolume] Transformed ${records.length} rows, ${validRecords.length} valid, ${invalidCount} invalid`); - - if (invalidCount > 0 && records.length > 0) { - const invalidSample = records.find(r => !r.token || !r.date); - if (invalidSample) { - logger.info('[MeteoraVolume] Sample invalid record:', { record: JSON.stringify(invalidSample) }); - } - } - - if (validRecords.length === 0) { - logger.info('[MeteoraVolume] No valid records to insert'); - return 0; - } - - const today = new Date().toISOString().split('T')[0]!; - const markComplete = true; - - logger.info(`[MeteoraVolume] Upserting ${validRecords.length} records to database...`); - try { - const upserted = await this.databaseService.upsertDailyMeteoraVolumes(validRecords, markComplete); - logger.info(`[MeteoraVolume] Upsert complete: ${upserted} records`); - return upserted; - } catch (upsertError: any) { - logger.error('[MeteoraVolume] Error upserting to database', upsertError); - throw upsertError; - } - } - - /** - * Helper to add days to a date string - */ - private addDays(dateStr: string, days: number): string { - const date = new Date(dateStr); - date.setDate(date.getDate() + days); - return date.toISOString().split('T')[0]!; - } - - /** - * Check if service is ready - */ - isReady(): boolean { - return this.initialized && this.databaseService.isAvailable(); - } - - /** - * Get service status - */ - getStatus(): { - initialized: boolean; - databaseConnected: boolean; - queryIdConfigured: boolean; - isRefreshing: boolean; - } { - return { - initialized: this.initialized, - databaseConnected: this.databaseService.isAvailable(), - queryIdConfigured: !!config.dune.meteoraVolumeQueryId, - isRefreshing: this.isRefreshing, - }; - } -} diff --git a/tests/api.test.ts b/tests/api.test.ts index 083b444..f48c2bc 100644 --- a/tests/api.test.ts +++ b/tests/api.test.ts @@ -5,6 +5,7 @@ import { createApp, type Services } from '../src/app.js'; import type { FutarchyService, DaoTickerData } from '../src/services/futarchyService.js'; import type { PriceService } from '../src/services/priceService.js'; import type { DatabaseService } from '../src/services/databaseService.js'; +import type { ExternalDatabaseService } from '../src/services/externalDatabaseService.js'; // Mock DAO data const mockBaseMint = new PublicKey('SoLo9oxzLDpcq1dpqAgMwgce5WqkRDtNXK7EPnbmeta'); @@ -57,13 +58,23 @@ const mockPriceService = { const mockDatabaseService = { isAvailable: jest.fn().mockReturnValue(true), getFirstTradeDates: jest.fn().mockResolvedValue(new Map()), + // v0.6 indexer fallback for /api/tickers 24h volume + getV06Rolling24hMetrics: jest.fn().mockResolvedValue(new Map()), } as unknown as DatabaseService; +// Primary /api/tickers source: rolling-24h spot metrics read straight from the +// indexer DB (futarchy.trades), keyed by dao_addr. +const mockExternalDatabaseService = { + isAvailable: jest.fn().mockReturnValue(true), + getSpotRolling24hMetrics: jest.fn().mockResolvedValue(new Map()), +} as unknown as ExternalDatabaseService; + function createMockServices(): Services { return { futarchyService: mockFutarchyService, priceService: mockPriceService, databaseService: mockDatabaseService, + externalDatabaseService: mockExternalDatabaseService, duneService: null, duneCacheService: null, solanaService: undefined, @@ -71,7 +82,7 @@ function createMockServices(): Services { hourlyAggregationService: null, tenMinuteVolumeFetcherService: null, dailyAggregationService: null, - meteoraVolumeFetcherService: null, + v06ReconciliationService: null, }; } @@ -123,6 +134,46 @@ describe('CoinGecko API', () => { expect(response.body[0].target_volume).toBe('0'); } }); + + it('should read 24h volume from the indexer DB (futarchy.trades) keyed by dao_addr', async () => { + (mockExternalDatabaseService as any).getSpotRolling24hMetrics.mockResolvedValueOnce(new Map([ + [mockDaoAddress.toString(), { + token: mockDaoAddress.toString(), + base_volume_24h: '12.5', + target_volume_24h: '125', + high_24h: '0.06', + low_24h: '0.04', + trade_count_24h: 4, + }], + ])); + + const response = await request(app).get('/api/tickers'); + + expect(response.status).toBe(200); + expect(response.body[0].base_volume).toBe('12.5'); + expect(response.body[0].target_volume).toBe('125'); + expect(response.body[0].high_24h).toBe('0.06'); + expect(response.body[0].low_24h).toBe('0.04'); + }); + + it('should fall back to v0.6 indexer metrics when futarchy.trades is empty', async () => { + (mockDatabaseService as any).getV06Rolling24hMetrics.mockResolvedValueOnce(new Map([ + [mockBaseMint.toString(), { + token: mockBaseMint.toString(), + base_volume_24h: '7', + target_volume_24h: '70', + high_24h: '0.06', + low_24h: '0.04', + trade_count_24h: 3, + }], + ])); + + const response = await request(app).get('/api/tickers'); + + expect(response.status).toBe(200); + expect(response.body[0].base_volume).toBe('7'); + expect(response.body[0].target_volume).toBe('70'); + }); }); describe('GET /health', () => { diff --git a/tests/helpers/testApp.ts b/tests/helpers/testApp.ts index 4c97eda..83d33d2 100644 --- a/tests/helpers/testApp.ts +++ b/tests/helpers/testApp.ts @@ -15,6 +15,8 @@ export function createMockDatabaseService(): DatabaseService { getTenMinuteRecordCount: async () => 0, getDailyRecordCount: async () => 0, getBuySellRecordCount: async () => 0, + getRolling24hFromTenMinute: async () => new Map(), + getRolling24hMetrics: async () => new Map(), insertServiceHealthSnapshot: async () => {}, insertMetricsBatch: async () => {}, pruneOldMetrics: async () => {}, @@ -71,7 +73,7 @@ export function createTestServices(overrides?: Partial): Services { hourlyAggregationService: null, tenMinuteVolumeFetcherService: null, dailyAggregationService: null, - meteoraVolumeFetcherService: null, + v06ReconciliationService: null, ...overrides, }; } diff --git a/tests/runtime/services.test.ts b/tests/runtime/services.test.ts new file mode 100644 index 0000000..35b74ba --- /dev/null +++ b/tests/runtime/services.test.ts @@ -0,0 +1,35 @@ +import { describe, expect, it } from 'bun:test'; +import { config } from '../../src/config.js'; +import { createServices } from '../../src/runtime/services.js'; + +describe('runtime service composition', () => { + it('creates API services without indexing workers', () => { + const services = createServices('api'); + + expect(services.databaseService).toBeTruthy(); + expect(services.externalDatabaseService).toBeTruthy(); + expect(services.duneService).toBeNull(); + expect(services.duneCacheService).toBeNull(); + expect(services.hourlyAggregationService).toBeNull(); + expect(services.tenMinuteVolumeFetcherService).toBeNull(); + expect(services.dailyAggregationService).toBeNull(); + expect(services.v06ReconciliationService).toBeNull(); + }); + + it('creates indexer services for collection and reconciliation', () => { + const services = createServices('indexer'); + + expect(services.v06ReconciliationService).toBeTruthy(); + + if (config.dune.apiKey) { + expect(services.duneService).toBeTruthy(); + expect(services.tenMinuteVolumeFetcherService).toBeTruthy(); + expect(services.hourlyAggregationService).toBeTruthy(); + expect(services.dailyAggregationService).toBeTruthy(); + } else { + expect(services.duneService).toBeNull(); + expect(services.tenMinuteVolumeFetcherService).toBeNull(); + } + }); +}); + From ae670b6de358ce58d5673bcf64917f0d2ee387d4 Mon Sep 17 00:00:00 2001 From: Jinglun Date: Thu, 4 Jun 2026 18:03:15 -0700 Subject: [PATCH 02/14] feat(external-api): remove Dune FutarchyAMM volume pipeline (serve v0.6 only) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FutarchyAMM volume/24h data is already served from our v0.6 indexer (getDailyTradingActivity -> v06_fee_volume_daily_aggregate); the Dune-sourced 10-min -> hourly -> daily pipeline was dormant in prod (USE_DUNE_DATA=false). Validated the v0.6 path dollar-to-dollar vs the on-chain source public.v0_6_spot_swaps (660K swaps): spot_trade_count 656,158 (Δ0, every day, 9 months); spot base + USDC volume $435,786,572.45 (Δ$0.00); spot_usdc_fees $1,091,210.03 + spot_token_fees 2,661,845.56 (Δ0). Removed the Dune ingest/aggregation services (DuneService, DuneCacheService, TenMinuteVolumeFetcherService, HourlyAggregationService, DailyAggregationService) + their wiring (runtime/services, indexer, routes/types, root, health, metrics), the USE_DUNE_DATA flag, and the dune config block. market-data always reads v0.6 (source 'v06-indexer'). All v0.6 read paths, repos, tables, health counters, and V06ReconciliationService kept intact. tsc + suite (111) green; full-router smoke test passes. Phase 2 (follow-up, NOT in this commit): drop the now-frozen Dune tables (ten_minute_volumes/hourly_volumes/daily_volumes/daily_fees_volumes) + their repos + the dead getDailyBuySellVolumes method; repoint getFirstTradeDates to a v0.6 source. Co-Authored-By: Claude Opus 4.8 (1M context) --- example.env | 30 +- src/config.ts | 8 - src/indexer.ts | 45 -- src/routes/health.ts | 81 +-- src/routes/market.ts | 14 +- src/routes/metrics.ts | 30 +- src/routes/root.ts | 14 +- src/routes/types.ts | 20 - src/runtime/services.ts | 26 - src/services/dailyAggregationService.ts | 320 ---------- .../internal/repos/dailyVolumesRepo.ts | 2 +- src/services/duneCacheService.ts | 177 ------ src/services/duneService.ts | 573 ------------------ src/services/hourlyAggregationService.ts | 464 -------------- src/services/tenMinuteVolumeFetcherService.ts | 495 --------------- tests/api.test.ts | 5 - tests/helpers/testApp.ts | 5 - tests/routes/health.test.ts | 7 - tests/runtime/services.test.ts | 19 +- 19 files changed, 13 insertions(+), 2322 deletions(-) delete mode 100644 src/services/dailyAggregationService.ts delete mode 100644 src/services/duneCacheService.ts delete mode 100644 src/services/duneService.ts delete mode 100644 src/services/hourlyAggregationService.ts delete mode 100644 src/services/tenMinuteVolumeFetcherService.ts diff --git a/example.env b/example.env index 6f5708b..3a34463 100644 --- a/example.env +++ b/example.env @@ -42,34 +42,8 @@ PROTOCOL_FEE_RATE=0.005 EXCLUDED_DAOS=DMB74TZgN7Rqfwtqqm3VQBgKBb2WYPdBqVtHbvB4LLeV,AE7jPb9jYzbUE5GYJToKvXaRkJL2Q7Mm3Ek6KqyBGuxe,E3BjsvLSFqUqVtDP76qMw4QbETkxvqvg8RTSbRZxWCK4,CnUUCGbSrAoaJniPifRU8zHRZ6e5uGRVSpCEj2WMeeSv,CLoqV77NtkbrsvtCRDP1vdYxgPZua3nnh7gCNPLzDQQ8,CJCgDqiDtkQvwXT2iiyY7QVajKLH3VRVbcsNQgtttrHn,651uV1hcd7SprwwkumFfkWtx5WrnD53awpjduGtGsHzS,4rW6iVKUq1RWYQ1VBTrjvP9FL4G3Sn7mBj7Yg12kuckv,Eo1BLMVRLJspjP5dDnwzK1m6FxMUcQDG6kDA8CjWPzRW,CTYxPujxrXiiqwG3gSBVNKuBk8u7mPG9qVMUc4aT1L8u,EbcsPbXZa81xUunDSmzYrcAWGURxcZB6BTkgzqvNJBZH,BgNq2V6vea2C7Z3cZhDUJTbmN4Y9bKG6dfEPhH19J7Fb,DHjQLd6LCM4yzZ9e8eabyGofDJLjbouqpuX8wh1rQuBs,BQjNtXjZB7b9WrqgJZQWfR52T1MqZoqMELAoombywDi8,j6Hx7bdAzcj1NsoRBqdafFuRkgEU48QeZ1i5NVXz9fF,AE7jPb9jYzbUE5GYJToKvXaRkJL2Q7Mm3Ek6KqyBGuxe -# Dune Analytics Configuration -# Get your API key from https://dune.com/settings/api -DUNE_API_KEY=your_dune_api_key_here - -# 10-Minute volume query ID - PRIMARY query for rolling 24h metrics on /api/tickers -# This query should accept 'start_time', 'end_time', and 'token_list' parameters -# See src/schema/dune-ten-minute-volume.sql for the query template -# Runs every 10 minutes = 144 queries/day = ~4,320 queries/month -# Hourly and daily data are automatically aggregated from 10-minute data using DB functions -# DUNE_TEN_MINUTE_VOLUME_QUERY_ID= - -# Meteora daily volumes query ID - tracks Meteora pool fees per owner -# This query should accept 'start_date' parameter -# See src/schema/dune-meteora-volumes.sql for the query template -# Runs once daily at 00:00 UTC (midnight) = 1 query/day = ~30 queries/month -# DUNE_METEORA_VOLUME_QUERY_ID= - -# Dune Cache Configuration -# Cache refresh interval in seconds (default: 3600 = 1 hour) -# DUNE_CACHE_REFRESH_INTERVAL=3600 -# Fetch timeout in seconds (default: 240 = 4 minutes) -# DUNE_FETCH_TIMEOUT=240 - -# Optional: Internal cache TTLs in seconds (used by DuneService internally) -# These are less important now that we have the main cache layer -# DUNE_CACHE_TTL=60 -# DUNE_BATCH_CACHE_TTL=300 -# DUNE_AGGREGATE_VOLUME_CACHE_TTL=600 +# FutarchyAMM volume + Meteora data are served from our own indexer / ETL +# (v0.6 indexer aggregates and futarchy.meteora_daily). No Dune dependency. # =========================================== # PostgreSQL Database Configuration diff --git a/src/config.ts b/src/config.ts index 1d47d50..0f84670 100644 --- a/src/config.ts +++ b/src/config.ts @@ -47,14 +47,6 @@ export const config = { // Protocol fee rate (0.005 = 0.5%) protocolFeeRate: parseFloat(process.env.PROTOCOL_FEE_RATE || '0.005'), }, - // When true (default), use Dune-sourced volume data (10-min/hourly/cache) for FutarchyAMM 24h metrics. - // Set USE_DUNE_DATA=false to use v0.6 indexer data (v06_spot_ohlcv_1m) instead. - useDuneData: process.env.USE_DUNE_DATA !== 'false', - dune: { - apiKey: process.env.DUNE_API_KEY || '', - // ACTIVE: 10-minute query - single source of truth, all other data aggregated from this - tenMinuteVolumeQueryId: process.env.DUNE_TEN_MINUTE_VOLUME_QUERY_ID ? parseInt(process.env.DUNE_TEN_MINUTE_VOLUME_QUERY_ID) : undefined, - }, alerts: { webhookUrl: process.env.ALERT_WEBHOOK_URL || 'https://telegram-webhook-relay.themetadao-org.workers.dev', webhookSecret: process.env.ALERT_WEBHOOK_SECRET || '', diff --git a/src/indexer.ts b/src/indexer.ts index 02af9e1..83314ed 100644 --- a/src/indexer.ts +++ b/src/indexer.ts @@ -6,47 +6,6 @@ import { scheduleDailyAtUTC, scheduleWithoutPileup, type ScheduledTask } from '. import type { Services } from './app.js'; async function startIndexingServices(services: Services): Promise { - if (services.hourlyAggregationService) { - logger.info('Starting Hourly Aggregation service'); - try { - await services.hourlyAggregationService.start(); - logger.info('Hourly Aggregation service started'); - } catch (error) { - logger.error('Failed to start Hourly Aggregation service', error); - } - } - - if (services.tenMinuteVolumeFetcherService) { - logger.info('Starting 10-Minute Volume Fetcher service'); - try { - await services.tenMinuteVolumeFetcherService.start(); - logger.info('10-Minute Volume Fetcher service started'); - } catch (error) { - logger.error('Failed to start 10-Minute Volume Fetcher service', error); - } - } - - if (services.dailyAggregationService) { - logger.info('Starting Daily Aggregation service'); - try { - await services.dailyAggregationService.initialize(); - services.dailyAggregationService.start(); - logger.info('Daily Aggregation service started'); - } catch (error) { - logger.error('Failed to start Daily Aggregation service', error); - } - } - - if (services.duneCacheService) { - logger.info('Starting Dune cache service'); - try { - await services.duneCacheService.start(); - logger.info('Dune cache service started'); - } catch (error) { - logger.error('Failed to start Dune cache service', error); - } - } - if (services.externalDatabaseService?.isAvailable() && services.v06ReconciliationService) { logger.info('Starting v0.6 Reconciliation service'); services.v06ReconciliationService.start(); @@ -92,10 +51,6 @@ function startIndexerScheduledTasks(services: Services): ScheduledTask[] { async function stopIndexingServices(services: Services, scheduledTasks: ScheduledTask[]): Promise { scheduledTasks.forEach(task => task.stop()); - services.duneCacheService?.stop(); - services.hourlyAggregationService?.stop(); - services.tenMinuteVolumeFetcherService?.stop(); - services.dailyAggregationService?.stop(); services.v06ReconciliationService?.stop(); await closeDataStores(services); } diff --git a/src/routes/health.ts b/src/routes/health.ts index 772e09e..7da0688 100644 --- a/src/routes/health.ts +++ b/src/routes/health.ts @@ -3,33 +3,20 @@ import type { ServiceGetters } from './types.js'; export function createHealthRouter(services: ServiceGetters): Router { const router = Router(); - const { getDatabaseService, getDuneCacheService, getHourlyAggregationService, getTenMinuteVolumeFetcherService } = services; + const { getDatabaseService } = services; // Basic health check router.get('/health', (req: Request, res: Response) => { - const duneCacheService = getDuneCacheService(); - const cacheStatus = duneCacheService?.getCacheStatus(); - res.json({ status: 'healthy', timestamp: new Date().toISOString(), uptime: process.uptime(), - duneCache: cacheStatus ? { - lastUpdated: cacheStatus.lastUpdated.toISOString(), - isRefreshing: cacheStatus.isRefreshing, - poolMetricsCount: cacheStatus.poolMetricsCount, - cacheAgeSeconds: Math.round(cacheStatus.cacheAgeMs / 1000), - isInitialized: cacheStatus.isInitialized, - } : null, }); }); // Comprehensive health check router.get('/api/health', async (req: Request, res: Response) => { const databaseService = getDatabaseService(); - const duneCacheService = getDuneCacheService(); - const hourlyAggregationService = getHourlyAggregationService(); - const tenMinuteVolumeFetcherService = getTenMinuteVolumeFetcherService(); const health: Record = { status: 'healthy', @@ -40,32 +27,6 @@ export function createHealthRouter(services: ServiceGetters): Router { }, }; - if (duneCacheService) { - const status = duneCacheService.getCacheStatus(); - health.services.dune_cache = { - initialized: status.isInitialized, - refreshing: status.isRefreshing, - lastRefreshTime: status.lastUpdated ? status.lastUpdated.toISOString() : null, - }; - } - - if (hourlyAggregationService) { - health.services.hourly_volume = { - initialized: hourlyAggregationService.isInitialized, - databaseConnected: hourlyAggregationService.isDatabaseConnected(), - }; - } - - if (tenMinuteVolumeFetcherService) { - const status = tenMinuteVolumeFetcherService.getStatus(); - health.services.ten_minute_volume = { - initialized: status.initialized, - running: status.isRunning, - refreshing: status.refreshInProgress, - lastRefreshTime: status.lastRefreshTime, - }; - } - const hasUnhealthyService = Object.values(health.services).some( (s: any) => s.initialized === false ); @@ -139,49 +100,9 @@ export function createHealthRouter(services: ServiceGetters): Router { // Helper function to save health snapshots export async function saveHealthSnapshots(services: ServiceGetters): Promise { const databaseService = services.getDatabaseService(); - const duneCacheService = services.getDuneCacheService(); - const hourlyAggregationService = services.getHourlyAggregationService(); - const tenMinuteVolumeFetcherService = services.getTenMinuteVolumeFetcherService(); if (!databaseService.isAvailable()) return; - if (duneCacheService) { - const status = duneCacheService.getCacheStatus(); - await databaseService.insertServiceHealthSnapshot( - 'dune_cache', - status.isInitialized, - status.lastUpdated, - undefined, - undefined, - { isRefreshing: status.isRefreshing } - ); - } - - if (hourlyAggregationService) { - const recordCount = await databaseService.getHourlyRecordCount(); - await databaseService.insertServiceHealthSnapshot( - 'hourly_volume', - hourlyAggregationService.isInitialized, - undefined, - recordCount, - undefined, - { databaseConnected: hourlyAggregationService.isDatabaseConnected() } - ); - } - - if (tenMinuteVolumeFetcherService) { - const status = tenMinuteVolumeFetcherService.getStatus(); - const recordCount = await databaseService.getTenMinuteRecordCount(); - await databaseService.insertServiceHealthSnapshot( - 'ten_minute_volume', - status.initialized, - status.lastRefreshTime ? new Date(status.lastRefreshTime) : undefined, - recordCount, - undefined, - { isRunning: status.isRunning, refreshInProgress: status.refreshInProgress } - ); - } - const dailyCount = await databaseService.getDailyRecordCount(); const hourlyCount = await databaseService.getHourlyRecordCount(); const tenMinCount = await databaseService.getTenMinuteRecordCount(); diff --git a/src/routes/market.ts b/src/routes/market.ts index fe8e5f7..bad4a59 100644 --- a/src/routes/market.ts +++ b/src/routes/market.ts @@ -1,15 +1,13 @@ import { Router, type Request, type Response } from 'express'; import { parseDateParam, parseCommaSeparatedList } from '../utils/validation.js'; -import { config } from '../config.js'; import type { ServiceGetters } from './types.js'; export function createMarketRouter(services: ServiceGetters): Router { const router = Router(); const { getDatabaseService, getExternalDatabaseService } = services; - // Get daily market data with date range and optional token filtering - // When USE_DUNE_DATA=false, sources from v0.6 indexer aggregate table; - // otherwise uses the Dune-sourced daily_volumes table. + // Get daily market data with date range and optional token filtering. + // FutarchyAMM rows are sourced from the v0.6 indexer aggregate table. router.get('/api/market-data', async (req: Request, res: Response) => { const databaseService = getDatabaseService(); @@ -44,16 +42,12 @@ export function createMarketRouter(services: ServiceGetters): Router { endDate: endDateResult.value!, }; - const useV06 = !config.useDuneData; - // Meteora rows are served directly from our meteora accounting ETL // (futarchy.meteora_daily in the served DB, via externalDatabase). const externalDatabaseService = getExternalDatabaseService(); const [futarchyData, meteoraData] = await Promise.all([ - useV06 - ? databaseService.getDailyTradingActivity(queryOptions) - : databaseService.getDailyBuySellVolumes(queryOptions), + databaseService.getDailyTradingActivity(queryOptions), externalDatabaseService?.isAvailable() ? externalDatabaseService.getDailyMeteoraVolumes(queryOptions) : Promise.resolve([]), @@ -65,7 +59,7 @@ export function createMarketRouter(services: ServiceGetters): Router { startDate: startDateResult.value, endDate: endDateResult.value, }, - source: useV06 ? 'v06-indexer' : 'dune', + source: 'v06-indexer', futarchyAMM: { count: futarchyData.length, data: futarchyData, diff --git a/src/routes/metrics.ts b/src/routes/metrics.ts index 878768e..c71d95f 100644 --- a/src/routes/metrics.ts +++ b/src/routes/metrics.ts @@ -6,7 +6,7 @@ import { logger } from '../utils/logger.js'; export function createMetricsRouter(services: ServiceGetters): Router { const router = Router(); - const { getDatabaseService, getDuneCacheService, getHourlyAggregationService, getTenMinuteVolumeFetcherService, getFutarchyService } = services; + const { getDatabaseService, getFutarchyService } = services; // Prometheus metrics endpoint router.get('/metrics', async (req: Request, res: Response) => { @@ -84,9 +84,6 @@ export function createMetricsRouter(services: ServiceGetters): Router { // Helper function to update all metrics async function updateMetricsSnapshot(services: ServiceGetters): Promise { const databaseService = services.getDatabaseService(); - const duneCacheService = services.getDuneCacheService(); - const hourlyAggregationService = services.getHourlyAggregationService(); - const tenMinuteVolumeFetcherService = services.getTenMinuteVolumeFetcherService(); const futarchyService = services.getFutarchyService(); metricsService.setDatabaseConnected(databaseService.isAvailable()); @@ -122,31 +119,6 @@ async function updateMetricsSnapshot(services: ServiceGetters): Promise { } } - if (duneCacheService) { - const status = duneCacheService.getCacheStatus(); - metricsService.setServiceStatus('dune_cache', status.isInitialized); - metricsService.setRefreshInProgress('dune_cache', status.isRefreshing); - if (status.lastUpdated) { - metricsService.setLastRefreshTime('dune_cache', status.lastUpdated); - metricsService.updateTimeSinceLastRefresh('dune_cache', status.lastUpdated.getTime()); - } - } - - if (hourlyAggregationService) { - metricsService.setServiceStatus('hourly_volume', hourlyAggregationService.isInitialized); - metricsService.setRefreshInProgress('hourly_volume', false); - } - - if (tenMinuteVolumeFetcherService) { - const status = tenMinuteVolumeFetcherService.getStatus(); - metricsService.setServiceStatus('ten_minute_volume', status.initialized); - metricsService.setRefreshInProgress('ten_minute_volume', status.refreshInProgress); - if (status.lastRefreshTime) { - metricsService.setLastRefreshTime('ten_minute_volume', new Date(status.lastRefreshTime)); - metricsService.updateTimeSinceLastRefresh('ten_minute_volume', new Date(status.lastRefreshTime).getTime()); - } - } - try { const daos = await futarchyService.getAllDaos(); metricsService.setActiveDaosCount(daos.length); diff --git a/src/routes/root.ts b/src/routes/root.ts index 4c06dbf..b6c9e6f 100644 --- a/src/routes/root.ts +++ b/src/routes/root.ts @@ -2,22 +2,18 @@ import { Router, type Request, type Response } from 'express'; import { config } from '../config.js'; import type { ServiceGetters } from './types.js'; -export function createRootRouter(services: ServiceGetters): Router { +export function createRootRouter(_services: ServiceGetters): Router { const router = Router(); - const { getDuneCacheService } = services; // Root endpoint with API documentation router.get('/', (req: Request, res: Response) => { - const duneCacheService = getDuneCacheService(); - const cacheStatus = duneCacheService?.getCacheStatus(); - res.json({ name: 'Futarchy AMM - CoinGecko API', version: '1.0.0', documentation: 'https://docs.coingecko.com/reference/exchanges-list', endpoints: { tickers: '/api/tickers - Returns all DAO tickers with pricing and volume', - market_data: '/api/market-data - Daily market data (futarchy AMM + Meteora); uses v0.6 indexer when USE_DUNE_DATA=false', + market_data: '/api/market-data - Daily market data (futarchy AMM + Meteora); FutarchyAMM served from the v0.6 indexer', supply: '/api/supply/:mintAddress - Returns complete supply breakdown with allocation details', supply_total: '/api/supply/:mintAddress/total - Returns total supply only', supply_circulating: '/api/supply/:mintAddress/circulating - Returns circulating supply (excludes team performance package)', @@ -47,11 +43,7 @@ export function createRootRouter(services: ServiceGetters): Router { description: 'Ticker volume is served from app DB aggregates populated by the separate indexer runtime', refreshInterval: `${parseInt(process.env.DUNE_CACHE_REFRESH_INTERVAL || '3600')} seconds`, fetchTimeout: `${parseInt(process.env.DUNE_FETCH_TIMEOUT || '240')} seconds`, - status: cacheStatus ? { - isInitialized: cacheStatus.isInitialized, - poolMetricsCount: cacheStatus.poolMetricsCount, - lastUpdated: cacheStatus.lastUpdated.toISOString(), - } : 'No live cache in API runtime', + status: 'No live cache in API runtime', }, note: 'This API discovers DAOs for serving responses; background indexing runs separately.', }); diff --git a/src/routes/types.ts b/src/routes/types.ts index 5f8911f..3593cab 100644 --- a/src/routes/types.ts +++ b/src/routes/types.ts @@ -1,13 +1,8 @@ import type { FutarchyService } from '../services/futarchyService.js'; import type { PriceService } from '../services/priceService.js'; -import type { DuneService } from '../services/duneService.js'; -import type { DuneCacheService } from '../services/duneCacheService.js'; import type { SolanaService } from '../services/solanaService.js'; import type { LaunchpadService } from '../services/launchpadService.js'; import type { DatabaseService } from '../services/databaseService.js'; -import type { HourlyAggregationService } from '../services/hourlyAggregationService.js'; -import type { TenMinuteVolumeFetcherService } from '../services/tenMinuteVolumeFetcherService.js'; -import type { DailyAggregationService } from '../services/dailyAggregationService.js'; import type { ExternalDatabaseService } from '../services/externalDatabaseService.js'; import type { V06ReconciliationService } from '../services/v06ReconciliationService.js'; import { AppError } from '../middleware/errorHandler.js'; @@ -26,11 +21,6 @@ export interface Services { databaseService: DatabaseService; externalDatabaseService: ExternalDatabaseService | null; - duneService: DuneService | null; - duneCacheService: DuneCacheService | null; - hourlyAggregationService: HourlyAggregationService | null; - tenMinuteVolumeFetcherService: TenMinuteVolumeFetcherService | null; - dailyAggregationService: DailyAggregationService | null; v06ReconciliationService: V06ReconciliationService | null; solanaService?: SolanaService; @@ -45,14 +35,9 @@ export interface Services { export interface ServiceGetters { getFutarchyService: () => FutarchyService; getPriceService: () => PriceService; - getDuneService: () => DuneService | null; - getDuneCacheService: () => DuneCacheService | null; getSolanaService: () => SolanaService; getLaunchpadService: () => LaunchpadService; getDatabaseService: () => DatabaseService; - getHourlyAggregationService: () => HourlyAggregationService | null; - getTenMinuteVolumeFetcherService: () => TenMinuteVolumeFetcherService | null; - getDailyAggregationService: () => DailyAggregationService | null; getExternalDatabaseService: () => ExternalDatabaseService | null; } @@ -71,11 +56,6 @@ export function createServiceGetters(services: Services): ServiceGetters { getPriceService: () => services.priceService, getDatabaseService: () => services.databaseService, - getDuneService: () => optionalService(services.duneService), - getDuneCacheService: () => optionalService(services.duneCacheService), - getHourlyAggregationService: () => optionalService(services.hourlyAggregationService), - getTenMinuteVolumeFetcherService: () => optionalService(services.tenMinuteVolumeFetcherService), - getDailyAggregationService: () => optionalService(services.dailyAggregationService), getExternalDatabaseService: () => optionalService(services.externalDatabaseService), getSolanaService: () => requireService(services.solanaService, 'Solana'), diff --git a/src/runtime/services.ts b/src/runtime/services.ts index ea54903..4926dba 100644 --- a/src/runtime/services.ts +++ b/src/runtime/services.ts @@ -1,16 +1,10 @@ import type { Services } from '../app.js'; -import { config } from '../config.js'; -import { DailyAggregationService } from '../services/dailyAggregationService.js'; import { DatabaseService } from '../services/databaseService.js'; -import { DuneCacheService } from '../services/duneCacheService.js'; -import { DuneService } from '../services/duneService.js'; import { ExternalDatabaseService } from '../services/externalDatabaseService.js'; import { FutarchyService } from '../services/futarchyService.js'; -import { HourlyAggregationService } from '../services/hourlyAggregationService.js'; import { LaunchpadService } from '../services/launchpadService.js'; import { PriceService } from '../services/priceService.js'; import { SolanaService } from '../services/solanaService.js'; -import { TenMinuteVolumeFetcherService } from '../services/tenMinuteVolumeFetcherService.js'; import { V06ReconciliationService } from '../services/v06ReconciliationService.js'; import { logger } from '../utils/logger.js'; @@ -24,24 +18,9 @@ export function createServices(mode: RuntimeMode): Services { const solanaService = new SolanaService(); const launchpadService = new LaunchpadService(); - let duneService: DuneService | null = null; - let duneCacheService: DuneCacheService | null = null; - let hourlyAggregationService: HourlyAggregationService | null = null; - let tenMinuteVolumeFetcherService: TenMinuteVolumeFetcherService | null = null; - let dailyAggregationService: DailyAggregationService | null = null; let v06ReconciliationService: V06ReconciliationService | null = null; if (mode === 'indexer') { - if (config.dune.apiKey) { - duneService = new DuneService(); - duneCacheService = new DuneCacheService(duneService, databaseService, futarchyService); - hourlyAggregationService = new HourlyAggregationService(databaseService, duneService, futarchyService); - tenMinuteVolumeFetcherService = new TenMinuteVolumeFetcherService(duneService, databaseService, futarchyService); - dailyAggregationService = new DailyAggregationService(duneService, databaseService, futarchyService); - } else { - logger.warn('DUNE_API_KEY not configured - Dune indexing services will not be created'); - } - v06ReconciliationService = new V06ReconciliationService(databaseService, externalDatabaseService); } @@ -50,13 +29,8 @@ export function createServices(mode: RuntimeMode): Services { priceService, databaseService, externalDatabaseService, - duneService, - duneCacheService, solanaService, launchpadService, - hourlyAggregationService, - tenMinuteVolumeFetcherService, - dailyAggregationService, v06ReconciliationService, }; } diff --git a/src/services/dailyAggregationService.ts b/src/services/dailyAggregationService.ts deleted file mode 100644 index e39bcca..0000000 --- a/src/services/dailyAggregationService.ts +++ /dev/null @@ -1,320 +0,0 @@ -/** - * Daily Aggregation Service - * - * Extracts/rolls up daily metrics from daily_volumes table (which is populated by aggregate_hourly_to_daily DB function). - * Provides comprehensive daily metrics: volume, price, fees, trade count, cumulative values. - * Part of the aggregation pipeline: 10-min → hourly → daily - * - * Schedule: Daily at 00:05 UTC (after day boundary) - */ - -import { config } from '../config.js'; -import { DuneService } from './duneService.js'; -import { DatabaseService, DailyFeesVolumeRecord } from './databaseService.js'; -import { FutarchyService } from './futarchyService.js'; -import { logger } from '../utils/logger.js'; - -export class DailyAggregationService { - private duneService: DuneService; - private databaseService: DatabaseService; - private futarchyService: FutarchyService; - private initialized: boolean = false; - private refreshTimer: NodeJS.Timeout | null = null; - private isRefreshing: boolean = false; - - constructor( - duneService: DuneService, - databaseService: DatabaseService, - futarchyService: FutarchyService - ) { - this.duneService = duneService; - this.databaseService = databaseService; - this.futarchyService = futarchyService; - } - - /** - * Initialize the service - check database for existing data - */ - async initialize(): Promise { - if (!this.databaseService.isAvailable()) { - logger.info('[DailyAggregation] Database not connected - service disabled'); - return; - } - - logger.info('[DailyAggregation] Initializing daily aggregation service...'); - - try { - const recordCount = await this.databaseService.getFeesRecordCount(); - logger.info(`[DailyAggregation] Current record count in database: ${recordCount}`); - - if (recordCount > 0) { - this.initialized = true; - logger.info('[DailyAggregation] Service initialized with existing database records.'); - } else { - logger.info('[DailyAggregation] No existing data - performing initial backfill...'); - await this.backfillFromStart(); - this.initialized = true; - logger.info('[DailyAggregation] Initialization complete with backfill.'); - } - } catch (error: any) { - logger.error('[DailyAggregation] Error during initialization:', error); - const recordCount = await this.databaseService.getFeesRecordCount(); - if (recordCount > 0) { - this.initialized = true; - logger.info('[DailyAggregation] Serving existing data despite initialization error.'); - } - } - } - - /** - * Start the scheduled refresh process - */ - start(): void { - if (!this.databaseService.isAvailable()) { - logger.info('[DailyAggregation] Service not starting - database not available'); - return; - } - - logger.info('[DailyAggregation] Starting daily aggregation service...'); - this.refresh().catch(err => { - logger.error('[DailyAggregation] Background refresh error', err); - }); - - this.scheduleDailyRefresh(); - logger.info('[DailyAggregation] Service started - will refresh daily at 00:05 UTC'); - } - - /** - * Stop the service - */ - stop(): void { - if (this.refreshTimer) { - clearTimeout(this.refreshTimer); - this.refreshTimer = null; - } - logger.info('[DailyAggregation] Service stopped'); - } - - /** - * Schedule the next refresh at 00:05 UTC - */ - private scheduleDailyRefresh(): void { - const now = new Date(); - const nextRefresh = new Date(now); - - // Set to 00:05 UTC - nextRefresh.setUTCHours(0, 5, 0, 0); - - // If we're past 00:05 today, schedule for tomorrow - if (now >= nextRefresh) { - nextRefresh.setUTCDate(nextRefresh.getUTCDate() + 1); - } - - const msUntilRefresh = nextRefresh.getTime() - now.getTime(); - logger.info(`[DailyAggregation] Next refresh scheduled for ${nextRefresh.toISOString()} (in ${Math.round(msUntilRefresh / 60000)} minutes)`); - - this.refreshTimer = setTimeout(async () => { - await this.refresh(); - // Re-schedule for the next day - this.scheduleDailyRefresh(); - }, msUntilRefresh); - } - - /** - * Perform incremental refresh - extract data from daily_volumes table - */ - async refresh(): Promise { - if (this.isRefreshing) { - logger.info('[DailyAggregation] Refresh already in progress, skipping...'); - return; - } - - this.isRefreshing = true; - - try { - if (!this.databaseService.isAvailable()) { - logger.info('[DailyAggregation] Database not available, skipping refresh'); - return; - } - - const lastCompleteDate = await this.databaseService.getLatestFeesDate(); - - const recordsUpserted = await this.extractFromDailyVolumes(lastCompleteDate || undefined); - - const today = new Date().toISOString().split('T')[0]!; - await this.databaseService.markFeesDaysComplete(today); - - logger.info(`[DailyAggregation] Refresh completed successfully - ${recordsUpserted} records`); - } catch (error: any) { - logger.error('[DailyAggregation] Refresh error', error); - } finally { - this.isRefreshing = false; - } - } - - /** - * Force a full refresh (useful for manual triggering) - */ - async forceRefresh(): Promise<{ success: boolean; message: string; recordsUpserted?: number }> { - if (this.isRefreshing) { - return { success: false, message: 'Refresh already in progress' }; - } - - // Service now uses daily_volumes table instead of Dune - - this.isRefreshing = true; - - try { - const lastCompleteDate = await this.databaseService.getLatestFeesDate(); - - const recordsUpserted = await this.extractFromDailyVolumes(lastCompleteDate || undefined); - - const today = new Date().toISOString().split('T')[0]!; - await this.databaseService.markFeesDaysComplete(today); - - this.initialized = true; - return { - success: true, - message: `Refresh completed, extracted from daily_volumes starting from ${lastCompleteDate || 'beginning'}`, - recordsUpserted - }; - } catch (error: any) { - return { success: false, message: error.message }; - } finally { - this.isRefreshing = false; - } - } - - /** - * Backfill from the very beginning (2025-10-09) - * Now extracts from daily_volumes instead of Dune - */ - private async backfillFromStart(): Promise { - logger.info('[DailyAggregation] Starting full backfill from daily_volumes...'); - try { - const recordsUpserted = await this.extractFromDailyVolumes('2025-10-09'); - logger.info(`[DailyAggregation] Backfill complete, upserted ${recordsUpserted} records`); - - const today = new Date().toISOString().split('T')[0]!; - await this.databaseService.markFeesDaysComplete(today); - } catch (error: any) { - logger.error('[DailyAggregation] Backfill failed', error); - throw error; - } - } - - /** - * Extract fees data from daily_volumes table and store in daily_fees_volumes - */ - private async extractFromDailyVolumes(startDate?: string): Promise { - if (!this.databaseService.isAvailable() || !this.databaseService.pool) { - return 0; - } - - try { - // Query daily_volumes for records with fees and cumulative values - let query = ` - SELECT - token, - date, - base_volume, - target_volume, - buy_volume, - sell_volume, - usdc_fees, - token_fees, - token_fees_usdc, - sell_volume_usdc, - cumulative_usdc_fees, - cumulative_token_in_usdc_fees, - cumulative_target_volume, - cumulative_token_volume, - high, - average_price, - low - FROM daily_volumes - WHERE usdc_fees IS NOT NULL - `; - - const params: any[] = []; - if (startDate) { - query += ` AND date >= $1`; - params.push(startDate); - } - - query += ` ORDER BY token, date`; - - const result = await this.databaseService.pool.query(query, params); - - if (result.rows.length === 0) { - logger.info('[DailyAggregation] No daily_volumes data available to extract'); - return 0; - } - - // Transform to DailyFeesVolumeRecord format - const records: DailyFeesVolumeRecord[] = result.rows.map((row: any) => ({ - token: row.token, - trading_date: row.date, - base_volume: String(row.base_volume || 0), - target_volume: String(row.target_volume || 0), - usdc_fees: String(row.usdc_fees || 0), - token_fees_usdc: String(row.token_fees_usdc || 0), - token_fees: String(row.token_fees || 0), - buy_volume: String(row.buy_volume || 0), - sell_volume: String(row.sell_volume || 0), - sell_volume_usdc: String(row.sell_volume_usdc || 0), - cumulative_usdc_fees: String(row.cumulative_usdc_fees || 0), - cumulative_token_in_usdc_fees: String(row.cumulative_token_in_usdc_fees || 0), - cumulative_target_volume: String(row.cumulative_target_volume || 0), - cumulative_token_volume: String(row.cumulative_token_volume || 0), - high: String(row.high || 0), - average_price: String(row.average_price || 0), - low: String(row.low || 0), - })); - - const upserted = await this.databaseService.upsertDailyFeesVolumes(records, false); - logger.info(`[DailyAggregation] Extracted and upserted ${upserted} records from daily_volumes`); - - return upserted; - } catch (error: any) { - logger.error('[DailyAggregation] Error extracting from daily_volumes', error); - return 0; - } - } - - - /** - * Get daily fees volumes with optional filtering - */ - async getDailyFeesVolumes(options?: { - token?: string; - startDate?: string; - endDate?: string; - }): Promise { - return this.databaseService.getDailyFeesVolumes(options); - } - - /** - * Check if service is ready - */ - isReady(): boolean { - return this.initialized && this.databaseService.isAvailable(); - } - - /** - * Get service status - */ - getStatus(): { - initialized: boolean; - databaseConnected: boolean; - queryIdConfigured: boolean; - isRefreshing: boolean; - } { - return { - initialized: this.initialized, - databaseConnected: this.databaseService.isAvailable(), - queryIdConfigured: false, // Deprecated - now uses DB aggregation from daily_volumes - isRefreshing: this.isRefreshing, - }; - } -} diff --git a/src/services/database/internal/repos/dailyVolumesRepo.ts b/src/services/database/internal/repos/dailyVolumesRepo.ts index ef557a4..a990741 100644 --- a/src/services/database/internal/repos/dailyVolumesRepo.ts +++ b/src/services/database/internal/repos/dailyVolumesRepo.ts @@ -328,7 +328,7 @@ export function createDailyVolumesRepo(db: DbRuntime) { /** * Get rolling 24h metrics from v06_spot_ohlcv_1m table. - * Used when USE_DUNE_DATA=false to source FutarchyAMM volume from the v0.6 indexer. + * Sources FutarchyAMM rolling-24h volume from the v0.6 indexer. * Amounts in this table are raw integers (6 decimals) — caller divides by 1e6. */ async getV06Rolling24hMetrics(tokens?: string[]): Promise> { diff --git a/src/services/duneCacheService.ts b/src/services/duneCacheService.ts deleted file mode 100644 index 67372bf..0000000 --- a/src/services/duneCacheService.ts +++ /dev/null @@ -1,177 +0,0 @@ -import { DuneService, DunePoolMetrics } from './duneService.js'; -import { FutarchyService } from './futarchyService.js'; -import { DatabaseService } from './databaseService.js'; -import { scheduleWithoutPileup, type ScheduledTask } from '../utils/scheduling.js'; -import { logger } from '../utils/logger.js'; - -export interface CachedDuneData { - poolMetrics: Map; - lastUpdated: Date; - isRefreshing: boolean; -} - -export class DuneCacheService { - private duneService: DuneService; - private databaseService: DatabaseService; - private futarchyService: FutarchyService; - private cache: CachedDuneData; - private refreshTask: ScheduledTask | null = null; - private refreshIntervalMs: number; - private isInitialized: boolean = false; - - constructor(duneService: DuneService, databaseService: DatabaseService, futarchyService: FutarchyService) { - this.duneService = duneService; - this.databaseService = databaseService; - this.futarchyService = futarchyService; - - // Default to 1 hour refresh interval, configurable via environment - this.refreshIntervalMs = parseInt(process.env.DUNE_CACHE_REFRESH_INTERVAL || '3600') * 1000; - - // Initialize empty cache - this.cache = { - poolMetrics: new Map(), - lastUpdated: new Date(0), // Epoch - indicates never updated - isRefreshing: false, - }; - } - - /** - * Start the cache refresh cron job - * Should be called when the server starts - */ - async start(): Promise { - logger.info(`[DuneCache] Starting cache service with ${this.refreshIntervalMs / 1000}s refresh interval`); - - this.isInitialized = true; - - this.refreshTask = scheduleWithoutPileup( - () => this.refreshCache(), - { - name: 'DuneCache', - intervalMs: this.refreshIntervalMs, - immediate: true, - onError: (error) => logger.error('[DuneCache] Refresh failed', error), - } - ); - - logger.info(`[DuneCache] Cache service started with ${this.refreshIntervalMs / 1000}s interval`); - } - - /** - * Stop the cache refresh cron job - */ - stop(): void { - if (this.refreshTask) { - this.refreshTask.stop(); - this.refreshTask = null; - } - } - - /** - * Refresh the cache by fetching fresh data from Dune - */ - async refreshCache(): Promise { - if (this.cache.isRefreshing) { - logger.info('[DuneCache] Refresh already in progress, skipping'); - return; - } - - this.cache.isRefreshing = true; - logger.info('[DuneCache] Starting cache refresh...'); - const startTime = Date.now(); - - try { - const allDaos = await this.futarchyService.getAllDaos(); - const baseMintAddresses = allDaos.map(dao => dao.baseMint.toString()); - - logger.info(`[DuneCache] Refreshing data for ${baseMintAddresses.length} tokens`); - - try { - if (!this.databaseService.isAvailable()) { - logger.info('[DuneCache] Database not available, skipping 24h metrics refresh'); - } else { - const rolling24hMetrics = await this.databaseService.getRolling24hFromTenMinute(baseMintAddresses); - - const tokenToDaoMap = new Map(); - for (const dao of allDaos) { - tokenToDaoMap.set(dao.baseMint.toString(), dao.daoAddress.toString()); - } - - const daoMetricsMap = new Map(); - for (const [tokenAddress, metrics] of rolling24hMetrics.entries()) { - const daoAddress = tokenToDaoMap.get(tokenAddress); - if (daoAddress) { - daoMetricsMap.set(daoAddress, { - pool_id: daoAddress, - base_volume_24h: metrics.base_volume_24h, - target_volume_24h: metrics.target_volume_24h, - high_24h: metrics.high_24h, - low_24h: metrics.low_24h, - }); - } - } - - this.cache.poolMetrics = daoMetricsMap; - logger.info(`[DuneCache] Refreshed pool metrics from DB aggregation for ${daoMetricsMap.size} DAOs`); - } - } catch (error: any) { - logger.error('[DuneCache] Error refreshing pool metrics', error); - } - - - this.cache.lastUpdated = new Date(); - const duration = Date.now() - startTime; - logger.info(`[DuneCache] Cache refresh completed in ${duration}ms`); - } catch (error: any) { - logger.error('[DuneCache] Error during cache refresh', error); - } finally { - this.cache.isRefreshing = false; - } - } - - /** - * Force an immediate cache refresh - */ - async forceRefresh(): Promise { - logger.info('[DuneCache] Force refresh requested'); - await this.refreshCache(); - } - - /** - * Get cached pool metrics - * Returns null if cache is empty and not yet initialized - */ - getPoolMetrics(): Map | null { - if (!this.isInitialized && this.cache.poolMetrics.size === 0) { - return null; - } - return this.cache.poolMetrics; - } - - /** - * Get cache status information - */ - getCacheStatus(): { - lastUpdated: Date; - isRefreshing: boolean; - poolMetricsCount: number; - cacheAgeMs: number; - isInitialized: boolean; - } { - return { - lastUpdated: this.cache.lastUpdated, - isRefreshing: this.cache.isRefreshing, - poolMetricsCount: this.cache.poolMetrics.size, - cacheAgeMs: Date.now() - this.cache.lastUpdated.getTime(), - isInitialized: this.isInitialized, - }; - } - - /** - * Check if cache has data - */ - hasData(): boolean { - return this.cache.poolMetrics.size > 0; - } -} - diff --git a/src/services/duneService.ts b/src/services/duneService.ts deleted file mode 100644 index ab990fd..0000000 --- a/src/services/duneService.ts +++ /dev/null @@ -1,573 +0,0 @@ -import { config } from '../config.js'; -import { PublicKey } from '@solana/web3.js'; -import { withRetryAndTimeout, isTransientError, createRetryLogger } from '../utils/resilience.js'; -import { logger } from '../utils/logger.js'; - -export interface DunePoolMetrics { - pool_id: string; - base_volume_24h: string; - target_volume_24h: string; - high_24h: string; - low_24h: string; -} - -export interface DuneQueryResult { - rows: DunePoolMetrics[]; - metadata: { - column_names: string[]; - result_set_bytes: number; - total_row_count: number; - datapoint_count: number; - pending_time_millis: number; - execution_time_millis: number; - }; -} - -export interface DuneQuery { - query_id?: number; - name: string; - query_sql: string; - parameters?: Array<{ - key: string; - type: string; - value?: any; - }>; -} - -// Daily volume data for a single token on a specific date -export interface DuneDailyVolume { - token: string; - date: string; - base_volume: string; - target_volume: string; - high: string; - low: string; -} - -// Aggregate volume data for a single token (totals across all dates) -export interface DuneAggregateTokenVolume { - token: string; - first_trade_date: string; - last_trade_date: string; - total_base_volume: string; - total_target_volume: string; - all_time_high: string; - all_time_low: string; - trading_days: number; - daily_data: DuneDailyVolume[]; -} - -// Complete aggregate volume response -export interface DuneAggregateVolumeResponse { - tokens: DuneAggregateTokenVolume[]; - query_metadata: { - since_start: boolean; - token_count: number; - total_trading_days: number; - execution_time_millis: number; - }; -} - -export class DuneService { - private apiKey: string; - // Deprecated query IDs removed - now using DB aggregation from 10-minute data - private baseUrl: string = 'https://api.dune.com/api/v1'; - private cache: Map; - private batchCache: Map; timestamp: number }>; - private aggregateVolumeCache: Map; - private cacheTTL: number = 60000; // 1 minute cache for individual pools - private batchCacheTTL: number = 300000; // 5 minutes cache for batch queries (Dune queries are heavy) - private aggregateVolumeCacheTTL: number = 600000; // 10 minutes cache for aggregate volume (heavy historical query) - private fetchTimeout: number = 600000; // 10 minutes timeout for Dune API calls - private devMode: boolean; - - constructor() { - this.apiKey = config.dune.apiKey; - // Deprecated query IDs removed - services now use DB aggregation - this.cache = new Map(); - this.batchCache = new Map(); - this.aggregateVolumeCache = new Map(); - this.devMode = config.devMode; - - if (this.devMode) { - logger.info('[Dune] ⚠️ DEV_MODE enabled - external Dune API calls are disabled'); - } - // Allow configurable cache TTLs from environment - if (process.env.DUNE_CACHE_TTL) { - this.cacheTTL = parseInt(process.env.DUNE_CACHE_TTL) * 1000; // Convert seconds to milliseconds - } - if (process.env.DUNE_BATCH_CACHE_TTL) { - this.batchCacheTTL = parseInt(process.env.DUNE_BATCH_CACHE_TTL) * 1000; // Convert seconds to milliseconds - } - if (process.env.DUNE_AGGREGATE_VOLUME_CACHE_TTL) { - this.aggregateVolumeCacheTTL = parseInt(process.env.DUNE_AGGREGATE_VOLUME_CACHE_TTL) * 1000; - } - if (process.env.DUNE_FETCH_TIMEOUT) { - this.fetchTimeout = parseInt(process.env.DUNE_FETCH_TIMEOUT) * 1000; // Convert seconds to milliseconds - } - } - - /** - * Check if dev mode is enabled - */ - isDevMode(): boolean { - return this.devMode; - } - - /** - * Fetch with timeout and retry wrapper. - * Retries on transient errors (network issues, rate limits, 5xx). - * @param url The URL to fetch - * @param options Fetch options - * @returns Response promise - */ - private async fetchWithTimeout(url: string, options: RequestInit = {}): Promise { - return withRetryAndTimeout( - async () => { - const response = await fetch(url, options); - // Throw on transient HTTP errors so they get retried - if (response.status === 429 || response.status >= 500) { - const errorText = await response.text(); - const error = new Error(`Dune API error: ${response.status} - ${errorText}`); - (error as any).status = response.status; - throw error; - } - return response; - }, - { - timeoutMs: this.fetchTimeout, - timeoutMessage: `Dune API request timed out after ${this.fetchTimeout}ms`, - maxRetries: 2, - initialDelayMs: 1000, - maxDelayMs: 5000, - isRetryable: isTransientError, - onRetry: createRetryLogger('[Dune]'), - } - ); - } - - /** - * Convert scientific notation to regular decimal notation - * Handles string, number, and other types from Dune API - * @param value Value that may be in scientific notation (e.g., "3.2259955052399996E5" or 322599.55) - * @returns String in regular decimal notation (e.g., "322599.55052399996") - */ - private convertScientificNotation(value: unknown): string { - // Handle null, undefined, empty values - if (value === null || value === undefined || value === '' || value === 0 || value === '0') { - return '0'; - } - - // Handle number type directly - if (typeof value === 'number') { - if (isNaN(value) || !isFinite(value)) { - return '0'; - } - // Use toFixed with enough precision, then remove trailing zeros - return value.toFixed(20).replace(/\.?0+$/, ''); - } - - // Handle string type - if (typeof value === 'string') { - // Check if it contains scientific notation (E or e) - if (value.includes('E') || value.includes('e')) { - try { - const num = parseFloat(value); - if (isNaN(num)) { - return '0'; - } - return num.toFixed(20).replace(/\.?0+$/, ''); - } catch (error) { - logger.warn(`[Dune] Failed to convert scientific notation: ${value}`, { error: String(error) }); - return value; - } - } - return value; - } - - // Handle any other type (object, boolean, etc.) - convert to string first - try { - const strValue = String(value); - if (strValue === '[object Object]' || strValue === 'undefined' || strValue === 'null') { - return '0'; - } - const num = parseFloat(strValue); - if (isNaN(num) || !isFinite(num)) { - return '0'; - } - return num.toFixed(20).replace(/\.?0+$/, ''); - } catch { - return '0'; - } - } - - - /** - * Create or update a Dune query - * @param query The query definition - * @returns The created/updated query ID - */ - async createOrUpdateQuery(query: DuneQuery): Promise { - const url = query.query_id - ? `${this.baseUrl}/query/${query.query_id}` - : `${this.baseUrl}/query`; - - const method = query.query_id ? 'PATCH' : 'POST'; - - const response = await this.fetchWithTimeout(url, { - method, - headers: { - 'X-Dune-API-Key': this.apiKey, - 'Content-Type': 'application/json', - }, - body: JSON.stringify({ - name: query.name, - query_sql: query.query_sql, - parameters: query.parameters || [], - }), - }); - - if (!response.ok) { - const errorText = await response.text(); - throw new Error(`Dune API error: ${response.status} ${response.statusText} - ${errorText}`); - } - - const result = await response.json() as any; - return result.query_id || result.query?.query_id; - } - - /** - * Execute a Dune query and get execution ID - * Based on DefiLlama's implementation: https://github.com/DefiLlama/dimension-adapters/blob/master/helpers/dune.ts - * @param queryId The Dune query ID - * @param parameters Optional query parameters (e.g., token list, fullQuery) - */ - private async executeQuery( - queryId: number, - parameters?: Record - ): Promise<{ execution_id: string }> { - const executeUrl = `${this.baseUrl}/query/${queryId}/execute`; - - const requestBody: any = {}; - if (parameters && Object.keys(parameters).length > 0) { - // Ensure all parameters are strings (Dune expects string parameters) - const stringParams: Record = {}; - for (const [key, value] of Object.entries(parameters)) { - stringParams[key] = String(value); - } - requestBody.query_parameters = stringParams; - logger.info('[Dune] Sending parameters:', { params: stringParams }); - } - - const response = await this.fetchWithTimeout(executeUrl, { - method: 'POST', - headers: { - 'X-Dune-API-Key': this.apiKey, - 'Content-Type': 'application/json', - }, - body: JSON.stringify(requestBody), - }); - - if (!response.ok) { - const errorText = await response.text(); - if (response.status === 401) { - throw new Error('Dune API Key is invalid'); - } - throw new Error(`Dune API error: ${response.status} ${response.statusText} - ${errorText}`); - } - - const result = await response.json() as any; - - if (result?.execution_id) { - return { execution_id: result.execution_id }; - } - - // DefiLlama pattern: log the query if it fails - if (parameters?.fullQuery) { - logger.info(`Dune query: ${parameters.fullQuery}`); - } else { - logger.info('Dune parameters', { parameters }); - } - - throw new Error(`Error query data: ${JSON.stringify(result)}`); - } - - /** - * Get the execution status - */ - private async getExecutionStatus(executionId: string): Promise { - const statusUrl = `${this.baseUrl}/execution/${executionId}/status`; - - const response = await this.fetchWithTimeout(statusUrl, { - method: 'GET', - headers: { - 'X-Dune-API-Key': this.apiKey, - }, - }); - - if (!response.ok) { - const errorText = await response.text(); - throw new Error(`Dune API error: ${response.status} ${response.statusText} - ${errorText}`); - } - - const result = await response.json(); - return result; - } - - /** - * Fetch all paginated results from a URL - * Dune API returns next_uri when there are more results - */ - private async fetchAllPaginatedResults(initialUrl: string): Promise<{ rows: any[], metadata: any }> { - const allRows: any[] = []; - let currentUrl: string | null = initialUrl; - let metadata: any = null; - let pageCount = 0; - - while (currentUrl) { - pageCount++; - logger.info(`[Dune] Fetching page ${pageCount}...`); - - const response = await this.fetchWithTimeout(currentUrl, { - method: 'GET', - headers: { - 'X-Dune-API-Key': this.apiKey, - }, - }); - - if (!response.ok) { - const errorText = await response.text(); - throw new Error(`Dune API error: ${response.status} ${response.statusText} - ${errorText}`); - } - - const result = await response.json() as any; - - // Extract rows from this page - const pageRows = result.result?.rows || result.rows || []; - allRows.push(...pageRows); - - // Store metadata from first page - if (!metadata && result.result?.metadata) { - metadata = result.result.metadata; - } - - logger.info(`[Dune] Page ${pageCount}: ${pageRows.length} rows (total: ${allRows.length})`); - - // Check for next page - if (result.next_uri) { - // next_uri can be relative or absolute - handle both cases - if (result.next_uri.startsWith('http')) { - currentUrl = result.next_uri; - } else { - currentUrl = `https://api.dune.com${result.next_uri}`; - } - } else { - currentUrl = null; - } - } - - if (pageCount > 1) { - logger.info(`[Dune] Fetched ${pageCount} pages, total rows: ${allRows.length}`); - } - - return { rows: allRows, metadata }; - } - - /** - * Get the execution results - * Based on DefiLlama's implementation: https://github.com/DefiLlama/dimension-adapters/blob/master/helpers/dune.ts - */ - private async getQueryResults(executionId: string, limit: number = 100000): Promise { - const resultsUrl = `${this.baseUrl}/execution/${executionId}/results?limit=${limit}`; - - let attempts = 0; - const maxAttempts = 180; // Wait up to ~10 minutes (parameterized queries can take longer) - - while (attempts < maxAttempts) { - // Check execution status first - const status = await this.getExecutionStatus(executionId); - - if (status.state === 'QUERY_STATE_COMPLETED') { - // Query completed, fetch all paginated results - const { rows, metadata } = await this.fetchAllPaginatedResults(resultsUrl); - - return { - rows, - metadata: { - column_names: metadata?.column_names || [], - result_set_bytes: metadata?.result_set_bytes || 0, - total_row_count: metadata?.total_row_count || rows.length, - datapoint_count: metadata?.datapoint_count || 0, - pending_time_millis: metadata?.pending_time_millis || 0, - execution_time_millis: metadata?.execution_time_millis || 0, - }, - }; - } else if (status.state === 'QUERY_STATE_FAILED') { - // Status already contains error info, no need to fetch again - const errorObj = (status as any).error || {}; - const errorMessage = typeof errorObj === 'string' - ? errorObj - : errorObj.message || errorObj.type || status.state; - const errorDetails = errorObj.metadata - ? `Line ${errorObj.metadata.line}, Column ${errorObj.metadata.column}: ${errorMessage}` - : ''; - - logger.error('[Dune] Query failed with state:', undefined, { state: status.state }); - logger.error('[Dune] Error type:', undefined, { type: errorObj.type }); - logger.error('[Dune] Error message:', undefined, { errorMessage }); - if (errorDetails) { - logger.error('[Dune] Error details:', undefined, { errorDetails }); - } - logger.error('[Dune] Full status response:', undefined, { status }); - - const fullErrorMessage = errorDetails || errorMessage; - throw new Error(`Dune query failed: ${fullErrorMessage}`); - } - - // Query is still running, wait and retry with random delay (like DefiLlama) - const delay = Math.floor(Math.random() * 3) + 2; // 2-4 seconds - await new Promise(resolve => setTimeout(resolve, delay * 1000)); - attempts++; - } - - throw new Error('Dune query execution timeout'); - } - - /** - * Get latest results for a query without executing (if results are recent) - * Based on DefiLlama's implementation - * Now with pagination support - */ - private async getLatestResults(queryId: number, maxAgeHours: number = 3): Promise { - try { - const initialUrl = `${this.baseUrl}/query/${queryId}/results`; - const response = await this.fetchWithTimeout(initialUrl, { - method: 'GET', - headers: { - 'X-Dune-API-Key': this.apiKey, - }, - }); - - if (!response.ok) { - return null; - } - - const result = await response.json() as any; - const submittedAt = result.submitted_at; - - if (!submittedAt) { - return null; - } - - const submittedAtTimestamp = Math.trunc(new Date(submittedAt).getTime() / 1000); - const nowTimestamp = Math.trunc(Date.now() / 1000); - const diff = nowTimestamp - submittedAtTimestamp; - - // If results are older than maxAgeHours, return null to trigger new execution - if (diff >= maxAgeHours * 60 * 60) { - return null; - } - - // Parse result format and handle pagination - if (result.result) { - const allRows = [...(result.result.rows || [])]; - const metadata = result.result.metadata; - - // Check for more pages - let nextUri = result.next_uri; - let pageCount = 1; - - while (nextUri) { - pageCount++; - logger.info(`[Dune] Fetching latest results page ${pageCount}...`); - - // next_uri can be relative or absolute - handle both cases - const nextUrl = nextUri.startsWith('http') ? nextUri : `https://api.dune.com${nextUri}`; - const nextResponse = await this.fetchWithTimeout(nextUrl, { - method: 'GET', - headers: { - 'X-Dune-API-Key': this.apiKey, - }, - }); - - if (!nextResponse.ok) { - break; - } - - const nextResult = await nextResponse.json() as any; - const pageRows = nextResult.result?.rows || nextResult.rows || []; - allRows.push(...pageRows); - - logger.info(`[Dune] Page ${pageCount}: ${pageRows.length} rows (total: ${allRows.length})`); - - nextUri = nextResult.next_uri; - } - - if (pageCount > 1) { - logger.info(`[Dune] Fetched ${pageCount} pages of latest results, total rows: ${allRows.length}`); - } - - return { - rows: allRows, - metadata: { - column_names: metadata?.column_names || [], - result_set_bytes: metadata?.result_set_bytes || 0, - total_row_count: metadata?.total_row_count || allRows.length, - datapoint_count: metadata?.datapoint_count || 0, - pending_time_millis: metadata?.pending_time_millis || 0, - execution_time_millis: metadata?.execution_time_millis || 0, - }, - }; - } - - return null; - } catch (error) { - // If we can't get latest results, return null to trigger new execution - return null; - } - } - - /** - * Manually execute a Dune query with custom parameters - * Useful for testing or one-off queries - * @param queryId The Dune query ID to execute - * @param parameters Optional query parameters - * @returns The raw query results - */ - async executeQueryManually( - queryId: number, - parameters?: Record - ): Promise { - // Skip external calls in dev mode - if (this.devMode) { - logger.info(`[Dune] DEV_MODE: Skipping query ${queryId} execution`); - return { - rows: [], - metadata: { - column_names: [], - result_set_bytes: 0, - total_row_count: 0, - datapoint_count: 0, - pending_time_millis: 0, - execution_time_millis: 0, - } - }; - } - - try { - // Execute the query with parameters - const executeResult = await this.executeQuery(queryId, parameters); - - // Wait for results using the execution_id - const queryResult = await this.getQueryResults(executeResult.execution_id); - - return queryResult; - } catch (error) { - logger.error(`Error executing Dune query ${queryId}:`, error); - throw error; - } - } - -} - - diff --git a/src/services/hourlyAggregationService.ts b/src/services/hourlyAggregationService.ts deleted file mode 100644 index 92b97ba..0000000 --- a/src/services/hourlyAggregationService.ts +++ /dev/null @@ -1,464 +0,0 @@ -import { DatabaseService, HourlyVolumeRecord, Rolling24hMetrics } from './databaseService.js'; -import { DuneService } from './duneService.js'; -import { FutarchyService } from './futarchyService.js'; -import { config } from '../config.js'; -import { logger } from '../utils/logger.js'; - -// The timestamp when Futarchy trading data begins -const FUTARCHY_START_TIME = '2025-10-09 00:00:00'; - -export interface HourlyAggregationStatus { - isInitialized: boolean; - databaseConnected: boolean; - latestHour: string | null; - latestCompleteHour: string | null; - tokenCount: number; - recordCount: number; - lastRefreshTime: Date | null; - isRefreshing: boolean; - hourlyQueryId: number | null; - schedule: { - hourlyRefresh: string; // ":01 past each hour" - tenMinRefresh: string; // "every 10 minutes" - queriesPerDay: number; // 24 + 144 = 168 - }; -} - -export class HourlyAggregationService { - private databaseService: DatabaseService; - private duneService: DuneService; - private futarchyService: FutarchyService; - private _isInitialized: boolean = false; - private isRefreshing: boolean = false; - private lastRefreshTime: Date | null = null; - - // Scheduled timers - private hourlyTimer: ReturnType | null = null; - private tenMinTimer: ReturnType | null = null; - - // In-memory cache for when DB is unavailable - private inMemoryHourlyCache: Map = new Map(); - - constructor(databaseService: DatabaseService, duneService: DuneService, futarchyService: FutarchyService) { - this.databaseService = databaseService; - this.duneService = duneService; - this.futarchyService = futarchyService; - } - - get isInitialized(): boolean { - return this._isInitialized; - } - - isDatabaseConnected(): boolean { - return this.databaseService.isAvailable(); - } - - /** - * Initialize the service - */ - async initialize(): Promise { - logger.info('[HourlyAggregation] Initializing service...'); - - if (this.databaseService.isAvailable()) { - const recordCount = await this.databaseService.getHourlyRecordCount(); - const latestHour = await this.databaseService.getLatestHour(); - - if (recordCount > 0) { - logger.info(`[HourlyAggregation] Found ${recordCount} existing hourly records, latest: ${latestHour}`); - } else { - logger.info('[HourlyAggregation] No existing hourly data, will backfill on first refresh'); - } - } else { - logger.info('[HourlyAggregation] Database not available, will use in-memory cache'); - } - - this._isInitialized = true; - logger.info('[HourlyAggregation] Service initialized'); - } - - /** - * Start the scheduled refresh jobs - * - Hourly: runs at :01 past each hour (fetches complete hours) - * - 10-min: runs at :00, :10, :20, :30, :40, :50 (fetches current incomplete hour) - */ - async start(): Promise { - if (!this._isInitialized) { - await this.initialize(); - } - - if (this.databaseService.isAvailable()) { - const recordCount = await this.databaseService.getHourlyRecordCount(); - if (recordCount > 0) { - logger.info(`[HourlyAggregation] Ready to serve ${recordCount} cached hourly records`); - } - } - - logger.info('[HourlyAggregation] Starting scheduled refresh jobs...'); - logger.info('[HourlyAggregation] - Hourly refresh: :01 past each hour (24 queries/day)'); - logger.info('[HourlyAggregation] - 10-min refresh: every 10 minutes (144 queries/day)'); - - this.scheduleHourlyRefresh(); - this.scheduleTenMinRefresh(); - - logger.info('[HourlyAggregation] Scheduled refresh jobs started'); - - logger.info('[HourlyAggregation] Starting background refresh...'); - this.refresh().catch(err => { - logger.error('[HourlyAggregation] Background refresh failed', err); - }); - } - - /** - * Schedule the next hourly refresh at :01 past the hour - */ - private scheduleHourlyRefresh(): void { - const now = new Date(); - const nextHour = new Date(now); - nextHour.setHours(nextHour.getHours() + 1); - nextHour.setMinutes(1, 0, 0); // :01:00 - - // If we're past :01, schedule for next hour - if (now.getMinutes() >= 1) { - // Already past :01, next run is in the calculated time - } else { - // Before :01, run at :01 this hour - nextHour.setHours(now.getHours()); - } - - const msUntilNext = nextHour.getTime() - now.getTime(); - logger.info(`[HourlyAggregation] Next hourly refresh at ${nextHour.toISOString()} (in ${Math.round(msUntilNext / 1000)}s)`); - - this.hourlyTimer = setTimeout(async () => { - await this.refreshCompleteHours(); - this.scheduleHourlyRefresh(); // Schedule next - }, msUntilNext); - } - - /** - * Schedule the next 10-minute refresh at the next 10-minute mark - */ - private scheduleTenMinRefresh(): void { - const now = new Date(); - const currentMinute = now.getMinutes(); - const nextTenMin = Math.ceil((currentMinute + 1) / 10) * 10; - - const nextRun = new Date(now); - if (nextTenMin >= 60) { - nextRun.setHours(nextRun.getHours() + 1); - nextRun.setMinutes(0, 30, 0); // :00:30 of next hour (30 sec buffer) - } else { - nextRun.setMinutes(nextTenMin, 30, 0); // 30 sec buffer after the mark - } - - const msUntilNext = nextRun.getTime() - now.getTime(); - logger.info(`[HourlyAggregation] Next 10-min refresh at ${nextRun.toISOString()} (in ${Math.round(msUntilNext / 1000)}s)`); - - this.tenMinTimer = setTimeout(async () => { - await this.refreshCurrentHour(); - this.scheduleTenMinRefresh(); // Schedule next - }, msUntilNext); - } - - /** - * Stop the scheduled refresh jobs - */ - stop(): void { - if (this.hourlyTimer) { - clearTimeout(this.hourlyTimer); - this.hourlyTimer = null; - } - if (this.tenMinTimer) { - clearTimeout(this.tenMinTimer); - this.tenMinTimer = null; - } - logger.info('[HourlyAggregation] Scheduled refresh jobs stopped'); - } - - /** - * Refresh only complete hours (called at :01 past each hour) - * Aggregates from 10-minute data using database function - */ - private async refreshCompleteHours(): Promise { - if (this.isRefreshing) { - logger.info('[HourlyAggregation] Refresh already in progress, skipping hourly refresh'); - return; - } - - this.isRefreshing = true; - logger.info('[HourlyAggregation] Hourly refresh - aggregating last complete hour from 10-min data...'); - - try { - if (!this.databaseService.isAvailable()) { - logger.info('[HourlyAggregation] Database not available, skipping aggregation'); - return; - } - - const lastCompleteHour = new Date(); - lastCompleteHour.setMinutes(0, 0, 0); - lastCompleteHour.setHours(lastCompleteHour.getHours() - 1); - const hourISO = lastCompleteHour.toISOString(); - - const recordsAggregated = await this.databaseService.aggregate10MinToHourly(undefined, hourISO); - - if (recordsAggregated > 0) { - logger.info(`[HourlyAggregation] Hourly refresh complete - aggregated ${recordsAggregated} hourly records from 10-min data`); - } else { - logger.info('[HourlyAggregation] No 10-minute data available for aggregation'); - } - - this.lastRefreshTime = new Date(); - } catch (error: any) { - logger.error('[HourlyAggregation] Error in hourly refresh', error); - } finally { - this.isRefreshing = false; - } - } - - /** - * Refresh only the current incomplete hour (called every 10 minutes) - * Aggregates from 10-minute data using database function - */ - private async refreshCurrentHour(): Promise { - if (this.isRefreshing) { - logger.info('[HourlyAggregation] Refresh already in progress, skipping 10-min refresh'); - return; - } - - this.isRefreshing = true; - logger.info('[HourlyAggregation] 10-min refresh - aggregating current hour from 10-min data...'); - - try { - if (!this.databaseService.isAvailable()) { - logger.info('[HourlyAggregation] Database not available, skipping aggregation'); - return; - } - - const currentHourStart = new Date(); - currentHourStart.setMinutes(0, 0, 0); - const hourISO = currentHourStart.toISOString(); - - const recordsAggregated = await this.databaseService.aggregate10MinToHourly(undefined, hourISO); - - if (recordsAggregated > 0) { - await this.databaseService.pool?.query( - `UPDATE hourly_volumes SET is_complete = false WHERE hour = $1`, - [hourISO] - ); - logger.info(`[HourlyAggregation] 10-min refresh complete - aggregated ${recordsAggregated} hourly records (incomplete)`); - } else { - logger.info('[HourlyAggregation] No 10-minute data available for current hour aggregation'); - } - - this.lastRefreshTime = new Date(); - } catch (error: any) { - logger.error('[HourlyAggregation] Error in 10-min refresh', error); - } finally { - this.isRefreshing = false; - } - } - - /** - * Full refresh / backfill hourly data by aggregating from 10-minute data - * Used for: - * - Initial backfill when no data exists - * - Manual force refresh - * Strategy: - * - Aggregates all incomplete hours from 10-minute data - */ - async refresh(): Promise { - if (this.isRefreshing) { - logger.info('[HourlyAggregation] Refresh already in progress, skipping'); - return; - } - - this.isRefreshing = true; - const startTime = Date.now(); - logger.info('[HourlyAggregation] Starting full refresh/backfill from 10-min data...'); - - try { - if (!this.databaseService.isAvailable()) { - logger.info('[HourlyAggregation] Database not available, skipping aggregation'); - return; - } - - const currentHourStart = this.getCurrentHourStart(); - await this.databaseService.markHoursComplete(currentHourStart); - - const recordsAggregated = await this.databaseService.aggregate10MinToHourly(); - - if (recordsAggregated > 0) { - await this.databaseService.pool?.query( - `UPDATE hourly_volumes SET is_complete = true - WHERE hour < $1 AND is_complete = false`, - [currentHourStart] - ); - logger.info(`[HourlyAggregation] Aggregated ${recordsAggregated} hourly records from 10-min data`); - } else { - logger.info('[HourlyAggregation] No 10-minute data available for aggregation'); - } - - this.lastRefreshTime = new Date(); - const duration = Date.now() - startTime; - logger.info(`[HourlyAggregation] Refresh completed in ${duration}ms`); - } catch (error: any) { - logger.error('[HourlyAggregation] Error during refresh', error); - } finally { - this.isRefreshing = false; - } - } - - - /** - * Normalize hour timestamp to consistent ISO format - */ - private normalizeHourTimestamp(hour: string): string { - // Dune may return different formats, normalize to ISO - const date = new Date(hour); - return date.toISOString(); - } - - /** - * Get the start of the current hour as ISO string - */ - private getCurrentHourStart(): string { - const now = new Date(); - now.setMinutes(0, 0, 0); - return now.toISOString(); - } - - /** - * Update in-memory cache with hourly data - */ - private updateInMemoryCache(records: HourlyVolumeRecord[]): void { - for (const record of records) { - let tokenRecords = this.inMemoryHourlyCache.get(record.token); - - if (!tokenRecords) { - tokenRecords = []; - this.inMemoryHourlyCache.set(record.token, tokenRecords); - } - - // Find and update or add - const existingIndex = tokenRecords.findIndex(r => r.hour === record.hour); - if (existingIndex >= 0) { - tokenRecords[existingIndex] = record; - } else { - tokenRecords.push(record); - } - } - - // Prune old data from in-memory cache (keep 48 hours) - const cutoffTime = new Date(Date.now() - 48 * 60 * 60 * 1000).toISOString(); - for (const [token, records] of this.inMemoryHourlyCache.entries()) { - const filtered = records.filter(r => r.hour >= cutoffTime); - this.inMemoryHourlyCache.set(token, filtered); - } - } - - /** - * Get rolling 24h metrics for tokens - * Primary method for serving /api/tickers - */ - async getRolling24hMetrics(tokens?: string[]): Promise> { - // Try database first - if (this.databaseService.isAvailable()) { - const dbMetrics = await this.databaseService.getRolling24hMetrics(tokens); - if (dbMetrics.size > 0) { - return dbMetrics; - } - } - - // Fall back to in-memory cache - return this.calculateRolling24hFromCache(tokens); - } - - /** - * Calculate rolling 24h from in-memory cache - */ - private calculateRolling24hFromCache(tokens?: string[]): Map { - const metricsMap = new Map(); - const cutoffTime = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); - - const tokensToCheck = tokens - ? tokens - : Array.from(this.inMemoryHourlyCache.keys()); - - for (const tokenLower of tokensToCheck) { - const records = this.inMemoryHourlyCache.get(tokenLower); - if (!records) continue; - - const recent = records.filter(r => r.hour >= cutoffTime); - if (recent.length === 0) continue; - - let baseVolume = 0; - let targetVolume = 0; - let high = 0; - let low = Infinity; - let tradeCount = 0; - - for (const record of recent) { - baseVolume += parseFloat(record.base_volume) || 0; - targetVolume += parseFloat(record.target_volume) || 0; - const recordHigh = parseFloat(record.high) || 0; - const recordLow = parseFloat(record.low) || Infinity; - if (recordHigh > high) high = recordHigh; - if (recordLow > 0 && recordLow < low) low = recordLow; - tradeCount += record.trade_count || 0; - } - - metricsMap.set(tokenLower, { - token: tokenLower, - base_volume_24h: baseVolume.toFixed(8), - target_volume_24h: targetVolume.toFixed(8), - high_24h: high > 0 ? high.toFixed(12) : '0', - low_24h: low < Infinity ? low.toFixed(12) : '0', - trade_count_24h: tradeCount, - }); - } - - return metricsMap; - } - - /** - * Force an immediate refresh - */ - async forceRefresh(): Promise { - logger.info('[HourlyAggregation] Force refresh requested'); - await this.refresh(); - } - - /** - * Get service status - */ - async getStatus(): Promise { - const latestHour = this.databaseService.isAvailable() - ? await this.databaseService.getLatestHour() - : null; - const latestCompleteHour = this.databaseService.isAvailable() - ? await this.databaseService.getLatestCompleteHour() - : null; - const tokenCount = this.databaseService.isAvailable() - ? await this.databaseService.getHourlyTokenCount() - : this.inMemoryHourlyCache.size; - const recordCount = this.databaseService.isAvailable() - ? await this.databaseService.getHourlyRecordCount() - : Array.from(this.inMemoryHourlyCache.values()).reduce((sum, records) => sum + records.length, 0); - - return { - isInitialized: this._isInitialized, - databaseConnected: this.databaseService.isAvailable(), - latestHour, - latestCompleteHour, - tokenCount, - recordCount, - lastRefreshTime: this.lastRefreshTime, - isRefreshing: this.isRefreshing, - hourlyQueryId: null, // Deprecated - now uses DB aggregation from 10-minute data - schedule: { - hourlyRefresh: ':01 past each hour', - tenMinRefresh: 'every 10 minutes (:00, :10, :20, :30, :40, :50)', - queriesPerDay: 24 + 144, // 168 queries/day - }, - }; - } -} diff --git a/src/services/tenMinuteVolumeFetcherService.ts b/src/services/tenMinuteVolumeFetcherService.ts deleted file mode 100644 index b795ac9..0000000 --- a/src/services/tenMinuteVolumeFetcherService.ts +++ /dev/null @@ -1,495 +0,0 @@ -/** - * TenMinuteVolumeFetcherService - * - * Fetches 10-minute granularity volume data from Dune API for accurate rolling 24h calculations. - * This is the PRIMARY service for /api/tickers 24h volume data. - * - * Schedule: - * - Every 10 minutes: Fetch current incomplete bucket + update complete buckets - * - Stores last 25 hours of data (150 buckets per token) - * - Prunes data older than 25 hours - * - * 10-minute buckets: :00, :10, :20, :30, :40, :50 - */ - -import { config } from '../config'; -import { DuneService } from './duneService'; -import { DatabaseService, TenMinuteVolumeRecord, Rolling24hMetrics } from './databaseService'; -import { FutarchyService } from './futarchyService'; -import { scheduleAtBoundary, type ScheduledTask } from '../utils/scheduling'; -import { logger } from '../utils/logger.js'; - -export class TenMinuteVolumeFetcherService { - private duneService: DuneService; - private databaseService: DatabaseService; - private futarchyService: FutarchyService; - private refreshTask: ScheduledTask | null = null; - private isRunning: boolean = false; - private initialized: boolean = false; - private lastRefreshTime: Date | null = null; - private refreshInProgress: boolean = false; - - // 10-minute refresh interval (default) - private readonly REFRESH_INTERVAL_MS = 10 * 60 * 1000; // 10 minutes - - constructor( - duneService: DuneService, - databaseService: DatabaseService, - futarchyService: FutarchyService - ) { - this.duneService = duneService; - this.databaseService = databaseService; - this.futarchyService = futarchyService; - } - - get isInitialized(): boolean { - return this.initialized; - } - - isDatabaseConnected(): boolean { - return this.databaseService.isAvailable(); - } - - /** - * Initialize the service - check database and optionally backfill - */ - async initialize(): Promise { - if (!this.databaseService.isAvailable()) { - logger.info('[TenMinVolume] Database not connected - service disabled'); - return; - } - - logger.info('[TenMinVolume] Initializing 10-minute volume service...'); - - try { - const recordCount = await this.databaseService.getTenMinuteRecordCount(); - logger.info(`[TenMinVolume] Current record count in database: ${recordCount}`); - - const latestBucket = await this.databaseService.getLatestTenMinuteBucket(); - if (latestBucket) { - logger.info(`[TenMinVolume] Latest bucket in DB: ${latestBucket}`); - } - - if (recordCount > 0) { - this.initialized = true; - logger.info(`[TenMinVolume] Service initialized with ${recordCount} existing records`); - } - - if (config.dune.tenMinuteVolumeQueryId) { - const now = new Date(); - const twentyFourHoursAgo = new Date(now.getTime() - 24 * 60 * 60 * 1000); - - let startTime: Date; - if (latestBucket) { - const latestDate = new Date(latestBucket); - startTime = new Date(Math.min(latestDate.getTime(), twentyFourHoursAgo.getTime())); - } else { - startTime = twentyFourHoursAgo; - logger.info('[TenMinVolume] No existing data - will backfill last 24 hours'); - } - - logger.info(`[TenMinVolume] Backfilling from ${startTime.toISOString()}`); - await this.fetchAndStore(startTime, true); - - this.initialized = true; - logger.info('[TenMinVolume] Initialization complete'); - } else if (!this.initialized) { - logger.info('[TenMinVolume] No DUNE_TEN_MINUTE_VOLUME_QUERY_ID configured - cannot fetch new data'); - logger.info('[TenMinVolume] Service will not be available until query ID is set or data exists in DB'); - } - } catch (error: any) { - logger.error('[TenMinVolume] Error during initialization', error); - const recordCount = await this.databaseService.getTenMinuteRecordCount(); - if (recordCount > 0) { - this.initialized = true; - logger.info('[TenMinVolume] Serving existing data despite initialization error'); - } - } - } - - /** - * Start the background refresh loop - */ - async start(): Promise { - if (this.isRunning) { - logger.info('[TenMinVolume] Service already running'); - return; - } - - await this.initialize(); - - if (!config.dune.tenMinuteVolumeQueryId) { - if (this.initialized) { - logger.info('[TenMinVolume] Serving existing DB data (no refresh loop - set DUNE_TEN_MINUTE_VOLUME_QUERY_ID to enable)'); - } - return; - } - - this.isRunning = true; - - this.refreshTask = scheduleAtBoundary( - () => this.refresh(), - { - name: 'TenMinVolume', - boundaryMinutes: 10, - bufferSeconds: 5, - onError: (error) => logger.error('[TenMinVolume] Refresh error', error), - } - ); - - logger.info('[TenMinVolume] Started with non-pileup scheduling at 10-minute boundaries'); - } - - /** - * Stop the background refresh loop - */ - stop(): void { - if (this.refreshTask) { - this.refreshTask.stop(); - this.refreshTask = null; - } - this.isRunning = false; - } - - /** - * Perform a refresh - fetch recent 10-minute data - */ - async refresh(): Promise { - if (this.refreshInProgress) { - logger.info('[TenMinVolume] Refresh already in progress, skipping'); - return; - } - - this.refreshInProgress = true; - logger.info('[TenMinVolume] Starting refresh...'); - - try { - const startTime = new Date(Date.now() - 20 * 60 * 1000); - await this.fetchAndStore(startTime, false); - - const currentBucket = this.getCurrentBucketStart(); - await this.databaseService.markTenMinuteBucketsComplete(currentBucket.toISOString()); - - this.lastRefreshTime = new Date(); - logger.info('[TenMinVolume] Refresh complete'); - } catch (error: any) { - logger.error('[TenMinVolume] Error during refresh', error); - } finally { - this.refreshInProgress = false; - } - } - - /** - * Force a full refresh (backfill last 24h) - */ - async forceRefresh(): Promise { - logger.info('[TenMinVolume] Force refresh requested - backfilling last 24h'); - const startTime = new Date(Date.now() - 24 * 60 * 60 * 1000); - await this.fetchAndStore(startTime, true); - this.lastRefreshTime = new Date(); - } - - /** - * Get the start of the current 10-minute bucket - */ - private getCurrentBucketStart(): Date { - const now = new Date(); - const bucketMinute = Math.floor(now.getMinutes() / 10) * 10; - return new Date( - now.getFullYear(), - now.getMonth(), - now.getDate(), - now.getHours(), - bucketMinute, - 0, - 0 - ); - } - - /** - * Public method for backfilling missing extended fields - * Fetches data from Dune for a specific time range and updates only missing fields - * Only fetches buckets that actually need updating (checks database first) - * @param startTime Start time for the query (format: YYYY-MM-DD HH:MM:SS) - * @param endTime Optional end time for the query (format: YYYY-MM-DD HH:MM:SS). If not provided, queries until now. - */ - async backfillExtendedFields(startTime: string, endTime?: string): Promise { - const queryId = config.dune.tenMinuteVolumeQueryId; - if (!queryId) { - logger.info('[TenMinVolume] No query ID configured for backfill'); - return 0; - } - - try { - if (!this.databaseService.pool) { - logger.info('[TenMinVolume] Database not available for backfill check'); - return 0; - } - - const startDate = new Date(startTime); - // Use provided endTime or calculate end of day - const endDate = endTime ? new Date(endTime) : new Date(startDate); - if (!endTime) { - endDate.setDate(endDate.getDate() + 1); // Full day window (24 hours) if no endTime provided - } - - const missingCheck = await this.databaseService.pool.query(` - SELECT COUNT(*) as missing_count - FROM ten_minute_volumes - WHERE bucket >= $1 AND bucket < $2 - AND (buy_volume IS NULL OR buy_volume = 0 OR - sell_volume IS NULL OR sell_volume = 0 OR - average_price IS NULL OR average_price = 0) - `, [startDate.toISOString(), endDate.toISOString()]); - - const missingCount = parseInt(missingCheck.rows[0]?.missing_count || '0'); - if (missingCount === 0) { - // No missing fields in this range, skip - return 0; - } - - // Get the filtered list of DAOs from FutarchyService - const allDaos = await this.futarchyService.getAllDaos(); - const tokenAddresses = allDaos.map(dao => dao.baseMint.toString()); - - logger.info(`[TenMinVolume] Backfilling ${missingCount} records from ${startTime} for ${tokenAddresses.length} tokens`); - - const parameters: Record = { - start_time: startTime, - }; - - // Always provide end_time - use far future date if not provided (effectively no limit) - if (endTime) { - parameters.end_time = endTime; - } else { - // Use far future date to effectively have no limit (avoids SQL parsing issues) - const farFuture = new Date('2099-12-31 23:59:59'); - parameters.end_time = farFuture.toISOString().replace('T', ' ').replace('Z', '').slice(0, 19); - } - - // Format token list - if (tokenAddresses.length > 0) { - parameters.token_list = tokenAddresses.map(token => `'${token}'`).join(', '); - } else { - parameters.token_list = "'__ALL__'"; - } - - const result = await (this.duneService as any).executeQueryManually(queryId, parameters); - - if (!result || !result.rows || result.rows.length === 0) { - logger.info('[TenMinVolume] No results from Dune for backfill'); - return 0; - } - - const rows = result.rows; - logger.info(`[TenMinVolume] Received ${rows.length} rows from Dune for backfill`); - - // Transform to records with extended fields - const records: TenMinuteVolumeRecord[] = rows.map((row: any) => ({ - token: row.token, - bucket: this.parseDuneBucket(row.bucket), - base_volume: row.base_volume || '0', - target_volume: row.target_volume || '0', - buy_volume: row.buy_volume || '0', - sell_volume: row.sell_volume || '0', - high: row.high || '0', - low: row.low || '0', - average_price: row.average_price || '0', - trade_count: parseInt(row.trade_count || '0'), - usdc_fees: row.usdc_fees || '0', - token_fees: row.token_fees || '0', - token_fees_usdc: row.token_fees_usdc || '0', - sell_volume_usdc: row.sell_volume_usdc || '0', - })); - - // Filter to only records that need updating (have missing fields) - const recordsToUpdate = records.filter(record => { - // Check if this specific bucket needs updating - // We'll let the upsert logic handle this, but we can optimize by checking first - return true; // Let upsert handle the filtering - }); - - // Batch upsert records (will only update missing fields due to our safe upsert logic) - const currentBucket = this.getCurrentBucketStart(); - - // Separate complete and incomplete records for batch processing - const completeRecords = recordsToUpdate.filter(r => new Date(r.bucket) < currentBucket); - const incompleteRecords = recordsToUpdate.filter(r => new Date(r.bucket) >= currentBucket); - - let totalUpserted = 0; - - // Batch upsert complete records - if (completeRecords.length > 0) { - const count = await this.databaseService.upsertTenMinuteVolumes(completeRecords, true); - totalUpserted += count; - } - - // Batch upsert incomplete records - if (incompleteRecords.length > 0) { - const count = await this.databaseService.upsertTenMinuteVolumes(incompleteRecords, false); - totalUpserted += count; - } - - logger.info(`[TenMinVolume] Backfilled ${totalUpserted} records (batched)`); - return totalUpserted; - } catch (error: any) { - if (error.message.includes('402') || error.message.includes('Payment Required')) { - throw error; - } - logger.error('[TenMinVolume] Error during backfill', error); - return 0; - } - } - - /** - * Fetch 10-minute data from Dune and store in database - */ - private async fetchAndStore(startTime: Date, markHistoricalComplete: boolean): Promise { - const queryId = config.dune.tenMinuteVolumeQueryId; - if (!queryId) return; - - try { - // Get the filtered list of DAOs from FutarchyService - const allDaos = await this.futarchyService.getAllDaos(); - const tokenAddresses = allDaos.map(dao => dao.baseMint.toString()); - - logger.info(`[TenMinVolume] Fetching from Dune query ${queryId} with start_time: ${startTime.toISOString()} for ${tokenAddresses.length} tokens`); - - const parameters: Record = { - start_time: startTime.toISOString().replace('T', ' ').replace('Z', ''), - }; - - // Format token list - if (tokenAddresses.length > 0) { - parameters.token_list = tokenAddresses.map(token => `'${token}'`).join(', '); - } else { - parameters.token_list = "'__ALL__'"; - } - - const result = await (this.duneService as any).executeQueryManually(queryId, parameters); - - if (!result || !result.rows) { - logger.info('[TenMinVolume] No results from Dune'); - return; - } - - const rows = result.rows; - logger.info(`[TenMinVolume] Received ${rows.length} rows from Dune`); - - if (rows.length === 0) return; - - // Transform to records with extended fields - const records: TenMinuteVolumeRecord[] = rows.map((row: any) => ({ - token: row.token, - bucket: this.parseDuneBucket(row.bucket), - base_volume: row.base_volume || '0', - target_volume: row.target_volume || '0', - buy_volume: row.buy_volume || '0', - sell_volume: row.sell_volume || '0', - high: row.high || '0', - low: row.low || '0', - average_price: row.average_price || '0', - trade_count: parseInt(row.trade_count || '0'), - usdc_fees: row.usdc_fees || '0', - token_fees: row.token_fees || '0', - token_fees_usdc: row.token_fees_usdc || '0', - sell_volume_usdc: row.sell_volume_usdc || '0', - })); - - // For historical backfill, mark all as complete except current bucket - if (markHistoricalComplete) { - const currentBucket = this.getCurrentBucketStart(); - const completeRecords = records.filter(r => new Date(r.bucket) < currentBucket); - const incompleteRecords = records.filter(r => new Date(r.bucket) >= currentBucket); - - if (completeRecords.length > 0) { - await this.databaseService.upsertTenMinuteVolumes(completeRecords, true); - } - if (incompleteRecords.length > 0) { - await this.databaseService.upsertTenMinuteVolumes(incompleteRecords, false); - } - } else { - // Regular refresh - batch upserts like historical mode - const currentBucket = this.getCurrentBucketStart(); - const completeRecords = records.filter(r => new Date(r.bucket) < currentBucket); - const incompleteRecords = records.filter(r => new Date(r.bucket) >= currentBucket); - - if (completeRecords.length > 0) { - await this.databaseService.upsertTenMinuteVolumes(completeRecords, true); - } - if (incompleteRecords.length > 0) { - await this.databaseService.upsertTenMinuteVolumes(incompleteRecords, false); - } - } - - } catch (error: any) { - logger.error('[TenMinVolume] Error fetching from Dune', error); - throw error; - } - } - - /** - * Parse Dune's bucket timestamp format - */ - private parseDuneBucket(bucket: string): string { - // Dune returns bucket as "2026-01-07 12:30:00" or similar - if (bucket.includes('T')) { - return bucket; // Already ISO format - } - // Convert to ISO format - return bucket.replace(' ', 'T') + 'Z'; - } - - /** - * Get rolling 24h metrics from the 10-minute data - * This is the PRIMARY method for /api/tickers 24h volume - */ - async getRolling24hMetrics(baseMintAddresses?: string[]): Promise> { - if (!this.initialized || !this.databaseService.isAvailable()) { - logger.info('[TenMinVolume] Service not ready, returning empty metrics'); - return new Map(); - } - - try { - const dbMetrics = await this.databaseService.getRolling24hFromTenMinute(baseMintAddresses); - - const result = new Map(); - for (const [token, metrics] of dbMetrics) { - result.set(token, { - base_volume_24h: parseFloat(metrics.base_volume_24h) || 0, - target_volume_24h: parseFloat(metrics.target_volume_24h) || 0, - high_24h: parseFloat(metrics.high_24h) || 0, - low_24h: parseFloat(metrics.low_24h) || 0, - }); - } - - logger.info(`[TenMinVolume] Returning rolling 24h metrics for ${result.size} tokens`); - return result; - } catch (error: any) { - logger.error('[TenMinVolume] Error getting rolling 24h metrics', error); - return new Map(); - } - } - - /** - * Get service status for monitoring - */ - getStatus(): { - initialized: boolean; - isRunning: boolean; - databaseConnected: boolean; - lastRefreshTime: string | null; - queryId: number | undefined; - refreshInProgress: boolean; - } { - return { - initialized: this.initialized, - isRunning: this.isRunning, - databaseConnected: this.databaseService.isAvailable(), - lastRefreshTime: this.lastRefreshTime?.toISOString() || null, - queryId: config.dune.tenMinuteVolumeQueryId, - refreshInProgress: this.refreshInProgress, - }; - } -} - diff --git a/tests/api.test.ts b/tests/api.test.ts index f48c2bc..25a06de 100644 --- a/tests/api.test.ts +++ b/tests/api.test.ts @@ -75,13 +75,8 @@ function createMockServices(): Services { priceService: mockPriceService, databaseService: mockDatabaseService, externalDatabaseService: mockExternalDatabaseService, - duneService: null, - duneCacheService: null, solanaService: undefined, launchpadService: undefined, - hourlyAggregationService: null, - tenMinuteVolumeFetcherService: null, - dailyAggregationService: null, v06ReconciliationService: null, }; } diff --git a/tests/helpers/testApp.ts b/tests/helpers/testApp.ts index 83d33d2..5b796dd 100644 --- a/tests/helpers/testApp.ts +++ b/tests/helpers/testApp.ts @@ -66,13 +66,8 @@ export function createTestServices(overrides?: Partial): Services { futarchyService: createMockFutarchyService(), priceService: createMockPriceService(), databaseService: createMockDatabaseService(), - duneService: null, - duneCacheService: null, solanaService: createMockSolanaService(), launchpadService: createMockLaunchpadService(), - hourlyAggregationService: null, - tenMinuteVolumeFetcherService: null, - dailyAggregationService: null, v06ReconciliationService: null, ...overrides, }; diff --git a/tests/routes/health.test.ts b/tests/routes/health.test.ts index 4049246..f28b577 100644 --- a/tests/routes/health.test.ts +++ b/tests/routes/health.test.ts @@ -27,13 +27,6 @@ describe('Health Routes', () => { expect(typeof response.body.uptime).toBe('number'); expect(response.body.uptime).toBeGreaterThanOrEqual(0); }); - - it('should include duneCache info when available', async () => { - const response = await request(app).get('/health'); - - // duneCache may be null if not configured - expect(response.body).toHaveProperty('duneCache'); - }); }); describe('GET /api/health', () => { diff --git a/tests/runtime/services.test.ts b/tests/runtime/services.test.ts index 35b74ba..aea0194 100644 --- a/tests/runtime/services.test.ts +++ b/tests/runtime/services.test.ts @@ -1,5 +1,4 @@ import { describe, expect, it } from 'bun:test'; -import { config } from '../../src/config.js'; import { createServices } from '../../src/runtime/services.js'; describe('runtime service composition', () => { @@ -8,28 +7,12 @@ describe('runtime service composition', () => { expect(services.databaseService).toBeTruthy(); expect(services.externalDatabaseService).toBeTruthy(); - expect(services.duneService).toBeNull(); - expect(services.duneCacheService).toBeNull(); - expect(services.hourlyAggregationService).toBeNull(); - expect(services.tenMinuteVolumeFetcherService).toBeNull(); - expect(services.dailyAggregationService).toBeNull(); expect(services.v06ReconciliationService).toBeNull(); }); - it('creates indexer services for collection and reconciliation', () => { + it('creates indexer services for reconciliation', () => { const services = createServices('indexer'); expect(services.v06ReconciliationService).toBeTruthy(); - - if (config.dune.apiKey) { - expect(services.duneService).toBeTruthy(); - expect(services.tenMinuteVolumeFetcherService).toBeTruthy(); - expect(services.hourlyAggregationService).toBeTruthy(); - expect(services.dailyAggregationService).toBeTruthy(); - } else { - expect(services.duneService).toBeNull(); - expect(services.tenMinuteVolumeFetcherService).toBeNull(); - } }); }); - From c30e1582db20dcf09773f65e04ae9b9f0efade78 Mon Sep 17 00:00:00 2001 From: Jinglun Date: Fri, 5 Jun 2026 12:51:44 -0700 Subject: [PATCH 03/14] refactor(external-api): drop dead Dune volume tables/repos; repoint first-trade-dates to v0.6 (Phase 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase-2 cleanup after the Dune fetcher removal (ae670b6): the daily_volumes / hourly_volumes / ten_minute_volumes / daily_buy_sell_volumes / daily_fees_volumes tables were frozen (never written) once the Dune services were gone. Remove them and all dead code that read them. Deleted repos: intervalVolumesRepo, dailyFeesVolumesRepo, dailyBuySellVolumesRepo. Stripped dailyVolumesRepo to only getV06Rolling24hMetrics (v06_spot_ohlcv_1m, a v0.6 table — kept). Removed all dead databaseService delegators + now-unused record interfaces. schemaManager: dropped the 5 Dune CREATE TABLEs + the aggregate_10min_to_hourly / aggregate_hourly_to_daily / calculate_rolling_24h functions (kept sync_metadata, metrics_history, service_health_snapshots, v06_*). health.ts/metrics.ts: removed the dead Dune record-count snapshots. Repointed coingecko first-trade-dates off the frozen Dune daily_buy_sell_volumes to a new externalDatabaseService.getFirstTradeDates() that reads the v0.6 indexer (v0_6_spot_swaps JOIN v0_6_daos, MIN date per base mint) in the served DB — so new tokens get a startDate again. Verified: tsc clean; bun test 113 pass; schemaManager DDL executes on a fresh DB; getFirstTradeDates returns 35 tokens with correct dates vs real v0.6 data (Avici 2025-10-18, Jurassic 2026-05-15, Ranger 2026-01-10). The actual prod DROP TABLE for the 5 dead tables is a manual DBA step. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/routes/coingecko.ts | 4 +- src/routes/health.ts | 13 +- src/routes/metrics.ts | 31 - .../internal/repos/dailyBuySellVolumesRepo.ts | 379 -------- .../internal/repos/dailyFeesVolumesRepo.ts | 230 ----- .../internal/repos/dailyVolumesRepo.ts | 350 +------ .../internal/repos/intervalVolumesRepo.ts | 913 ------------------ .../database/internal/schemaManager.ts | 425 -------- src/services/databaseService.ts | 358 ------- src/services/externalDatabaseService.ts | 26 + tests/api.test.ts | 25 +- tests/helpers/testApp.ts | 21 +- tests/routes/metrics.test.ts | 2 +- 13 files changed, 69 insertions(+), 2708 deletions(-) delete mode 100644 src/services/database/internal/repos/dailyBuySellVolumesRepo.ts delete mode 100644 src/services/database/internal/repos/dailyFeesVolumesRepo.ts delete mode 100644 src/services/database/internal/repos/intervalVolumesRepo.ts diff --git a/src/routes/coingecko.ts b/src/routes/coingecko.ts index 5921be2..7de3eb7 100644 --- a/src/routes/coingecko.ts +++ b/src/routes/coingecko.ts @@ -18,8 +18,8 @@ export function createCoinGeckoRouter(services: ServiceGetters): Router { const allDaos = await futarchyService.getAllDaos(); - const firstTradeDates = databaseService?.isAvailable() - ? await databaseService.getFirstTradeDates() + const firstTradeDates = externalDatabaseService?.isAvailable() + ? await externalDatabaseService.getFirstTradeDates() : new Map(); const tokenToDaoMap = new Map(); diff --git a/src/routes/health.ts b/src/routes/health.ts index 7da0688..5454e55 100644 --- a/src/routes/health.ts +++ b/src/routes/health.ts @@ -103,15 +103,6 @@ export async function saveHealthSnapshots(services: ServiceGetters): Promise { const futarchyService = services.getFutarchyService(); metricsService.setDatabaseConnected(databaseService.isAvailable()); - - if (databaseService.isAvailable()) { - try { - const dailyCount = await databaseService.getDailyRecordCount(); - const hourlyCount = await databaseService.getHourlyRecordCount(); - const tenMinCount = await databaseService.getTenMinuteRecordCount(); - const buySellCount = await databaseService.getBuySellRecordCount(); - - metricsService.setDatabaseRecordCount('daily_volumes', dailyCount); - metricsService.setDatabaseRecordCount('hourly_volumes', hourlyCount); - metricsService.setDatabaseRecordCount('ten_minute_volumes', tenMinCount); - metricsService.setDatabaseRecordCount('daily_buy_sell_volumes', buySellCount); - - const dailyTokens = await databaseService.getTokenCount(); - const hourlyTokens = await databaseService.getHourlyTokenCount(); - metricsService.setDatabaseTokenCount('daily_volumes', dailyTokens); - metricsService.setDatabaseTokenCount('hourly_volumes', hourlyTokens); - - const latestDaily = await databaseService.getLatestDate(); - const latestHourly = await databaseService.getLatestHour(); - const latestTenMin = await databaseService.getLatestTenMinuteBucket(); - const latestBuySell = await databaseService.getLatestBuySellDate(); - - metricsService.setDatabaseLatestDate('daily_volumes', latestDaily); - metricsService.setDatabaseLatestDate('hourly_volumes', latestHourly); - metricsService.setDatabaseLatestDate('ten_minute_volumes', latestTenMin); - metricsService.setDatabaseLatestDate('daily_buy_sell_volumes', latestBuySell); - } catch (error) { - logger.error('[Metrics] Error fetching database metrics:', error); - } - } try { const daos = await futarchyService.getAllDaos(); diff --git a/src/services/database/internal/repos/dailyBuySellVolumesRepo.ts b/src/services/database/internal/repos/dailyBuySellVolumesRepo.ts deleted file mode 100644 index 091f0e2..0000000 --- a/src/services/database/internal/repos/dailyBuySellVolumesRepo.ts +++ /dev/null @@ -1,379 +0,0 @@ -import { logger } from '../../../../utils/logger.js'; -import type { DbRuntime } from '../dbRuntime.js'; -import type { DailyBuySellVolumeRecord, CumulativeVolumeData } from '../../../databaseService.js'; - -export function createDailyBuySellVolumesRepo(db: DbRuntime) { - return { - /** - * Get the latest complete date from daily_buy_sell_volumes table - */ - async getLatestBuySellDate(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return null; - - try { - const result = await pool.query( - 'SELECT MAX(date) as latest_date FROM daily_buy_sell_volumes WHERE is_complete = true' - ); - return result.rows[0]?.latest_date?.toISOString().split('T')[0] || null; - } catch (error: any) { - logger.error('[Database] Error getting latest buy/sell date:', error); - return null; - } - }, - - async upsertDailyBuySellVolumes(records: DailyBuySellVolumeRecord[], markComplete: boolean = false): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected() || records.length === 0) { - return 0; - } - - const BATCH_SIZE = 500; - let totalUpserted = 0; - - try { - const client = await pool.connect(); - - try { - await client.query('BEGIN'); - - for (let i = 0; i < records.length; i += BATCH_SIZE) { - const batch = records.slice(i, i + BATCH_SIZE); - - const values: any[] = []; - const valuePlaceholders: string[] = []; - - batch.forEach((record, idx) => { - const offset = idx * 9; // 9 parameters per record - valuePlaceholders.push( - `($${offset + 1}, $${offset + 2}, $${offset + 3}, $${offset + 4}, $${offset + 5}, $${offset + 6}, $${offset + 7}, $${offset + 8}, $${offset + 9}, ${markComplete}, CURRENT_TIMESTAMP)` - ); - values.push( - record.token, - record.date, - record.base_volume, - record.target_volume, - record.buy_usdc_volume, - record.sell_token_volume, - record.high, - record.low, - record.trade_count || 0 - ); - }); - - const batchSQL = ` - INSERT INTO daily_buy_sell_volumes (token, date, base_volume, target_volume, buy_usdc_volume, sell_token_volume, high, low, trade_count, is_complete, updated_at) - VALUES ${valuePlaceholders.join(', ')} - ON CONFLICT (token, date) - DO UPDATE SET - base_volume = EXCLUDED.base_volume, - target_volume = EXCLUDED.target_volume, - buy_usdc_volume = EXCLUDED.buy_usdc_volume, - sell_token_volume = EXCLUDED.sell_token_volume, - high = EXCLUDED.high, - low = EXCLUDED.low, - trade_count = EXCLUDED.trade_count, - is_complete = CASE WHEN EXCLUDED.is_complete THEN true ELSE daily_buy_sell_volumes.is_complete END, - updated_at = CURRENT_TIMESTAMP - `; - - await client.query(batchSQL, values); - totalUpserted += batch.length; - - if (records.length > BATCH_SIZE) { - logger.info(`[Database] Buy/sell volume batch progress: ${totalUpserted}/${records.length}`); - } - } - - await client.query('COMMIT'); - logger.info(`[Database] Upserted ${totalUpserted} daily buy/sell volume records`); - return totalUpserted; - } catch (error) { - await client.query('ROLLBACK'); - throw error; - } finally { - client.release(); - } - } catch (error: any) { - logger.error('[Database] Error upserting daily buy/sell volumes:', error); - return 0; - } - }, - - /** - * Get daily buy/sell volumes with cumulative totals calculated from DB - * Cumulative values are computed on-the-fly using window functions - */ - async getDailyBuySellVolumesWithCumulative(token?: string): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return []; - - try { - let whereClause = ''; - let params: any[] = []; - - if (token) { - whereClause = 'WHERE token = $1'; - params = [token]; - } - - const result = await pool.query( - `SELECT - token, - date::text, - base_volume::text, - target_volume::text, - buy_usdc_volume::text, - sell_token_volume::text, - SUM(target_volume) OVER ( - PARTITION BY token - ORDER BY date - ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW - )::text AS cumulative_target_volume, - SUM(base_volume) OVER ( - PARTITION BY token - ORDER BY date - ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW - )::text AS cumulative_base_volume, - SUM(buy_usdc_volume) OVER ( - PARTITION BY token - ORDER BY date - ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW - )::text AS cumulative_buy_usdc_volume, - SUM(sell_token_volume) OVER ( - PARTITION BY token - ORDER BY date - ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW - )::text AS cumulative_sell_token_volume, - high::text, - low::text - FROM daily_buy_sell_volumes - ${whereClause} - ORDER BY token, date ASC`, - params - ); - - return result.rows; - } catch (error: any) { - logger.error('[Database] Error getting cumulative volumes:', error); - return []; - } - }, - - /** - * Get daily buy/sell volumes with date range filtering - * @param options.token Filter by specific token - * @param options.startDate Start date (inclusive) in YYYY-MM-DD format - * @param options.endDate End date (inclusive) in YYYY-MM-DD format - */ - async getDailyBuySellVolumes(options?: { - token?: string; - tokens?: string[]; - startDate?: string; - endDate?: string; - }): Promise<{ - token: string; - date: string; - base_volume: string; - target_volume: string; - buy_usdc_volume: string; - sell_token_volume: string; - high: string; - low: string; - trade_count: number; - average_price: string; - usdc_fees: string; - token_fees: string; - token_fees_usdc: string; - sell_volume_usdc: string; - sell_volume: string; - buy_volume: string; - }[]> { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return []; - - try { - const conditions: string[] = []; - const params: any[] = []; - let paramIndex = 1; - - // Support both single token and array of tokens - if (options?.tokens && options.tokens.length > 0) { - const placeholders = options.tokens.map((_, i) => `$${paramIndex + i}`).join(', '); - conditions.push(`token IN (${placeholders})`); - params.push(...options.tokens); - paramIndex += options.tokens.length; - } else if (options?.token) { - conditions.push(`token = $${paramIndex}`); - params.push(options.token); - paramIndex++; - } - - if (options?.startDate) { - conditions.push(`date >= $${paramIndex}`); - params.push(options.startDate); - paramIndex++; - } - - if (options?.endDate) { - conditions.push(`date <= $${paramIndex}`); - params.push(options.endDate); - paramIndex++; - } - - const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(' AND ')}` : ''; - - const result = await pool.query( - `SELECT - token, - date::text, - base_volume::text, - target_volume::text, - buy_volume::text, - buy_volume::text as buy_usdc_volume, - sell_volume::text, - sell_volume::text as sell_token_volume, - high::text, - low::text, - trade_count, - average_price::text, - usdc_fees::text, - token_fees::text, - token_fees_usdc::text, - sell_volume_usdc::text - FROM daily_volumes - ${whereClause} - ORDER BY token, date ASC`, - params - ); - - return result.rows; - } catch (error: any) { - logger.error('[Database] Error getting daily buy/sell volumes:', error); - return []; - } - }, - - /** - * Get aggregated buy/sell stats for all tokens - */ - async getBuySellAggregates(tokens?: string[]): Promise> { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return new Map(); - - try { - let whereClause = ''; - let params: any[] = []; - - if (tokens && tokens.length > 0) { - const placeholders = tokens.map((_, i) => `$${i + 1}`).join(', '); - whereClause = `WHERE token IN (${placeholders})`; - params = [...tokens]; - } - - const result = await pool.query( - `SELECT - token, - SUM(buy_usdc_volume)::text AS total_buy_usdc, - SUM(sell_token_volume)::text AS total_sell_token, - SUM(base_volume)::text AS total_base_volume, - SUM(target_volume)::text AS total_target_volume, - MIN(date)::text AS first_date, - MAX(date)::text AS last_date, - COUNT(*)::int AS trading_days - FROM daily_buy_sell_volumes - ${whereClause} - GROUP BY token - ORDER BY SUM(target_volume) DESC`, - params - ); - - const aggregates = new Map(); - for (const row of result.rows) { - aggregates.set(row.token, { - total_buy_usdc: row.total_buy_usdc || '0', - total_sell_token: row.total_sell_token || '0', - total_base_volume: row.total_base_volume || '0', - total_target_volume: row.total_target_volume || '0', - first_date: row.first_date, - last_date: row.last_date, - trading_days: row.trading_days || 0, - }); - } - return aggregates; - } catch (error: any) { - logger.error('[Database] Error getting buy/sell aggregates:', error); - return new Map(); - } - }, - - /** - * Get the first trade date for each token (when trading started) - * Returns a Map of token address -> first trade date (YYYY-MM-DD) - */ - async getFirstTradeDates(): Promise> { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return new Map(); - - try { - const result = await pool.query( - `SELECT token, MIN(date)::text AS first_date - FROM daily_buy_sell_volumes - GROUP BY token` - ); - - const map = new Map(); - for (const row of result.rows) { - map.set(row.token, row.first_date); - } - return map; - } catch (error: any) { - logger.error('[Database] Error getting first trade dates:', error); - return new Map(); - } - }, - - /** - * Get buy/sell volume record count - */ - async getBuySellRecordCount(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return 0; - - try { - const result = await pool.query('SELECT COUNT(*) as count FROM daily_buy_sell_volumes'); - return parseInt(result.rows[0]?.count || '0'); - } catch (error: any) { - logger.error('[Database] Error getting buy/sell record count:', error); - return 0; - } - }, - - /** - * Mark days as complete (called when day boundary passes) - */ - async markBuySellDaysComplete(beforeDate: string): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return; - - try { - await pool.query( - `UPDATE daily_buy_sell_volumes SET is_complete = true, updated_at = CURRENT_TIMESTAMP - WHERE date < $1 AND is_complete = false`, - [beforeDate] - ); - logger.info(`[Database] Marked buy/sell days before ${beforeDate} as complete`); - } catch (error: any) { - logger.error('[Database] Error marking buy/sell days complete:', error); - } - }, - }; -} diff --git a/src/services/database/internal/repos/dailyFeesVolumesRepo.ts b/src/services/database/internal/repos/dailyFeesVolumesRepo.ts deleted file mode 100644 index 27f2cb0..0000000 --- a/src/services/database/internal/repos/dailyFeesVolumesRepo.ts +++ /dev/null @@ -1,230 +0,0 @@ -import { logger } from '../../../../utils/logger.js'; -import type { DbRuntime } from '../dbRuntime.js'; -import type { DailyFeesVolumeRecord } from '../../../databaseService.js'; - -export function createDailyFeesVolumesRepo(db: DbRuntime) { - return { - /** - * Get the latest date we have fees data for - */ - async getLatestFeesDate(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return null; - - try { - const result = await pool.query( - 'SELECT MAX(trading_date) as latest_date FROM daily_fees_volumes WHERE is_complete = true' - ); - return result.rows[0]?.latest_date?.toISOString().split('T')[0] || null; - } catch (error: any) { - logger.error('[Database] Error getting latest fees date:', error); - return null; - } - }, - - /** - * Upsert daily fees volume records using batched inserts - * @param records Array of daily fees volume records - * @param markComplete If true, marks these days as complete (for historical data) - */ - async upsertDailyFeesVolumes(records: DailyFeesVolumeRecord[], markComplete: boolean = false): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected() || records.length === 0) { - return 0; - } - - const BATCH_SIZE = 500; - let totalUpserted = 0; - - try { - const client = await pool.connect(); - - try { - await client.query('BEGIN'); - - for (let i = 0; i < records.length; i += BATCH_SIZE) { - const batch = records.slice(i, i + BATCH_SIZE); - - const values: any[] = []; - const valuePlaceholders: string[] = []; - - batch.forEach((record, idx) => { - const offset = idx * 17; // 17 parameters per record - valuePlaceholders.push( - `($${offset + 1}, $${offset + 2}, $${offset + 3}, $${offset + 4}, $${offset + 5}, $${offset + 6}, $${offset + 7}, $${offset + 8}, $${offset + 9}, $${offset + 10}, $${offset + 11}, $${offset + 12}, $${offset + 13}, $${offset + 14}, $${offset + 15}, $${offset + 16}, $${offset + 17}, ${markComplete}, CURRENT_TIMESTAMP)` - ); - values.push( - record.token, - record.trading_date, - record.base_volume, - record.target_volume, - record.usdc_fees, - record.token_fees_usdc, - record.token_fees, - record.buy_volume, - record.sell_volume, - record.sell_volume_usdc, - record.cumulative_usdc_fees, - record.cumulative_token_in_usdc_fees, - record.cumulative_target_volume, - record.cumulative_token_volume, - record.high, - record.average_price, - record.low - ); - }); - - const batchSQL = ` - INSERT INTO daily_fees_volumes (token, trading_date, base_volume, target_volume, usdc_fees, token_fees_usdc, token_fees, buy_volume, sell_volume, sell_volume_usdc, cumulative_usdc_fees, cumulative_token_in_usdc_fees, cumulative_target_volume, cumulative_token_volume, high, average_price, low, is_complete, updated_at) - VALUES ${valuePlaceholders.join(', ')} - ON CONFLICT (token, trading_date) - DO UPDATE SET - base_volume = EXCLUDED.base_volume, - target_volume = EXCLUDED.target_volume, - usdc_fees = EXCLUDED.usdc_fees, - token_fees_usdc = EXCLUDED.token_fees_usdc, - token_fees = EXCLUDED.token_fees, - buy_volume = EXCLUDED.buy_volume, - sell_volume = EXCLUDED.sell_volume, - sell_volume_usdc = EXCLUDED.sell_volume_usdc, - cumulative_usdc_fees = EXCLUDED.cumulative_usdc_fees, - cumulative_token_in_usdc_fees = EXCLUDED.cumulative_token_in_usdc_fees, - cumulative_target_volume = EXCLUDED.cumulative_target_volume, - cumulative_token_volume = EXCLUDED.cumulative_token_volume, - high = EXCLUDED.high, - average_price = EXCLUDED.average_price, - low = EXCLUDED.low, - is_complete = CASE WHEN EXCLUDED.is_complete THEN true ELSE daily_fees_volumes.is_complete END, - updated_at = CURRENT_TIMESTAMP - `; - - await client.query(batchSQL, values); - totalUpserted += batch.length; - - if (records.length > BATCH_SIZE) { - logger.info(`[Database] Fees volume batch progress: ${totalUpserted}/${records.length}`); - } - } - - await client.query('COMMIT'); - logger.info(`[Database] Upserted ${totalUpserted} daily fees volume records`); - return totalUpserted; - } catch (error) { - await client.query('ROLLBACK'); - throw error; - } finally { - client.release(); - } - } catch (error: any) { - logger.error('[Database] Error upserting daily fees volumes:', error); - return 0; - } - }, - - /** - * Get daily fees volumes with date range filtering - * @param options.token Filter by specific token - * @param options.startDate Start date (inclusive) in YYYY-MM-DD format - * @param options.endDate End date (inclusive) in YYYY-MM-DD format - */ - async getDailyFeesVolumes(options?: { - token?: string; - startDate?: string; - endDate?: string; - }): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return []; - - try { - const conditions: string[] = []; - const params: any[] = []; - let paramIndex = 1; - - if (options?.token) { - conditions.push(`token = $${paramIndex}`); - params.push(options.token); - paramIndex++; - } - - if (options?.startDate) { - conditions.push(`trading_date >= $${paramIndex}`); - params.push(options.startDate); - paramIndex++; - } - - if (options?.endDate) { - conditions.push(`trading_date <= $${paramIndex}`); - params.push(options.endDate); - paramIndex++; - } - - const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(' AND ')}` : ''; - - const result = await pool.query( - `SELECT - token, - trading_date::text, - base_volume::text, - target_volume::text, - usdc_fees::text, - token_fees_usdc::text, - token_fees::text, - buy_volume::text, - sell_volume::text, - sell_volume_usdc::text, - cumulative_usdc_fees::text, - cumulative_token_in_usdc_fees::text, - cumulative_target_volume::text, - cumulative_token_volume::text, - high::text, - average_price::text, - low::text - FROM daily_fees_volumes - ${whereClause} - ORDER BY token, trading_date ASC`, - params - ); - - return result.rows; - } catch (error: any) { - logger.error('[Database] Error getting daily fees volumes:', error); - return []; - } - }, - - /** - * Mark days as complete (called when day boundary passes) - */ - async markFeesDaysComplete(beforeDate: string): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return; - - try { - await pool.query( - `UPDATE daily_fees_volumes SET is_complete = true, updated_at = CURRENT_TIMESTAMP - WHERE trading_date < $1 AND is_complete = false`, - [beforeDate] - ); - logger.info(`[Database] Marked fees days before ${beforeDate} as complete`); - } catch (error: any) { - logger.error('[Database] Error marking fees days complete:', error); - } - }, - - /** - * Get fees volume record count - */ - async getFeesRecordCount(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return 0; - - try { - const result = await pool.query('SELECT COUNT(*) as count FROM daily_fees_volumes'); - return parseInt(result.rows[0]?.count || '0'); - } catch (error: any) { - logger.error('[Database] Error getting fees record count:', error); - return 0; - } - }, - }; -} diff --git a/src/services/database/internal/repos/dailyVolumesRepo.ts b/src/services/database/internal/repos/dailyVolumesRepo.ts index a990741..74a69fc 100644 --- a/src/services/database/internal/repos/dailyVolumesRepo.ts +++ b/src/services/database/internal/repos/dailyVolumesRepo.ts @@ -1,331 +1,9 @@ import { logger } from '../../../../utils/logger.js'; import type { DbRuntime } from '../dbRuntime.js'; -import type { DailyVolumeRecord, TokenVolumeAggregate, Rolling24hMetrics } from '../../../databaseService.js'; +import type { Rolling24hMetrics } from '../../../databaseService.js'; export function createDailyVolumesRepo(db: DbRuntime) { return { - /** - * Get the latest date we have data for (across all tokens) - */ - async getLatestDate(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return null; - - try { - const result = await pool.query( - 'SELECT MAX(date) as latest_date FROM daily_volumes' - ); - return result.rows[0]?.latest_date?.toISOString().split('T')[0] || null; - } catch (error: any) { - logger.error('[Database] Error getting latest date:', error); - return null; - } - }, - - /** - * Get the latest date for a specific token - */ - async getLatestDateForToken(token: string): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return null; - - try { - const result = await pool.query( - 'SELECT MAX(date) as latest_date FROM daily_volumes WHERE token = $1', - [token] - ); - return result.rows[0]?.latest_date?.toISOString().split('T')[0] || null; - } catch (error: any) { - logger.error('[Database] Error getting latest date for token:', error); - return null; - } - }, - - /** - * Upsert daily volume records using batched inserts for performance - */ - async upsertDailyVolumes(records: DailyVolumeRecord[]): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected() || records.length === 0) return 0; - - const BATCH_SIZE = 500; // Insert 500 records per batch - let totalUpserted = 0; - - try { - const client = await pool.connect(); - - try { - await client.query('BEGIN'); - - // Process in batches - for (let i = 0; i < records.length; i += BATCH_SIZE) { - const batch = records.slice(i, i + BATCH_SIZE); - - // Build multi-value INSERT statement - const values: any[] = []; - const valuePlaceholders: string[] = []; - - batch.forEach((record, idx) => { - const offset = idx * 6; // 6 parameters per record - valuePlaceholders.push( - `($${offset + 1}, $${offset + 2}, $${offset + 3}, $${offset + 4}, $${offset + 5}, $${offset + 6}, CURRENT_TIMESTAMP)` - ); - values.push( - record.token, - record.date, - record.base_volume, - record.target_volume, - record.high, - record.low - ); - }); - - const batchSQL = ` - INSERT INTO daily_volumes (token, date, base_volume, target_volume, high, low, updated_at) - VALUES ${valuePlaceholders.join(', ')} - ON CONFLICT (token, date) - DO UPDATE SET - -- Only update if existing values are NULL or 0 (preserve existing data) - base_volume = COALESCE(NULLIF(daily_volumes.base_volume, 0), EXCLUDED.base_volume), - target_volume = COALESCE(NULLIF(daily_volumes.target_volume, 0), EXCLUDED.target_volume), - high = GREATEST(COALESCE(daily_volumes.high, 0), COALESCE(EXCLUDED.high, 0)), - low = LEAST( - CASE WHEN daily_volumes.low > 0 THEN daily_volumes.low ELSE EXCLUDED.low END, - CASE WHEN EXCLUDED.low > 0 THEN EXCLUDED.low ELSE daily_volumes.low END - ), - updated_at = CURRENT_TIMESTAMP - `; - - await client.query(batchSQL, values); - totalUpserted += batch.length; - - // Log progress for large batches - if (records.length > BATCH_SIZE) { - logger.info(`[Database] Daily volume batch progress: ${totalUpserted}/${records.length}`); - } - } - - await client.query('COMMIT'); - logger.info(`[Database] Upserted ${totalUpserted} daily volume records`); - return totalUpserted; - } catch (error) { - await client.query('ROLLBACK'); - throw error; - } finally { - client.release(); - } - } catch (error: any) { - logger.error('[Database] Error upserting daily volumes:', error); - return 0; - } - }, - - /** - * Get all daily volumes for a token - */ - async getDailyVolumesForToken(token: string): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return []; - - try { - const result = await pool.query( - `SELECT token, date::text, - base_volume::text, target_volume::text, - high::text, low::text - FROM daily_volumes - WHERE token = $1 - ORDER BY date ASC`, - [token] - ); - return result.rows; - } catch (error: any) { - logger.error('[Database] Error getting daily volumes for token:', error); - return []; - } - }, - - /** - * Get daily volumes for multiple tokens - */ - async getDailyVolumesForTokens(tokens: string[]): Promise> { - const pool = db.getPool(); - if (!pool || !db.isConnected() || tokens.length === 0) { - return new Map(); - } - - try { - const placeholders = tokens.map((_, i) => `$${i + 1}`).join(', '); - const result = await pool.query( - `SELECT token, date::text, - base_volume::text, target_volume::text, - high::text, low::text - FROM daily_volumes - WHERE token IN (${placeholders}) - ORDER BY token, date ASC`, - tokens - ); - - const tokenMap = new Map(); - for (const row of result.rows) { - if (!tokenMap.has(row.token)) { - tokenMap.set(row.token, []); - } - tokenMap.get(row.token)!.push(row); - } - return tokenMap; - } catch (error: any) { - logger.error('[Database] Error getting daily volumes for tokens:', error); - return new Map(); - } - }, - - /** - * Get aggregated volume data for all tokens (for API responses) - */ - async getAggregatedVolumes(tokens?: string[]): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return []; - - try { - let whereClause = ''; - let params: string[] = []; - - if (tokens && tokens.length > 0) { - const placeholders = tokens.map((_, i) => `$${i + 1}`).join(', '); - whereClause = `WHERE token IN (${placeholders})`; - params = [...tokens]; - } - - // Get aggregates - const aggregateResult = await pool.query( - `SELECT - token, - MIN(date)::text as first_trade_date, - MAX(date)::text as last_trade_date, - SUM(base_volume)::text as total_base_volume, - SUM(target_volume)::text as total_target_volume, - MAX(high)::text as all_time_high, - MIN(CASE WHEN low > 0 THEN low END)::text as all_time_low, - COUNT(*)::int as trading_days - FROM daily_volumes - ${whereClause} - GROUP BY token - ORDER BY SUM(base_volume) DESC`, - params - ); - - // Get daily data for each token - const dailyDataMap = await this.getDailyVolumesForTokens( - aggregateResult.rows.map((r: any) => r.token) - ); - - return aggregateResult.rows.map((row: any) => ({ - token: row.token, - first_trade_date: row.first_trade_date, - last_trade_date: row.last_trade_date, - total_base_volume: row.total_base_volume || '0', - total_target_volume: row.total_target_volume || '0', - all_time_high: row.all_time_high || '0', - all_time_low: row.all_time_low || '0', - trading_days: row.trading_days, - daily_data: dailyDataMap.get(row.token) || [], - })); - } catch (error: any) { - logger.error('[Database] Error getting aggregated volumes:', error); - return []; - } - }, - - /** - * Get 24h rolling volume data (last 24 hours from current time) - * This queries data from today and yesterday to cover the 24h window - */ - async get24hVolumes(tokens?: string[]): Promise> { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return new Map(); - - try { - // Get today and yesterday's date - const today = new Date().toISOString().split('T')[0]; - const yesterday = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString().split('T')[0]; - - let whereClause = 'WHERE date >= $1'; - let params: any[] = [yesterday]; - - if (tokens && tokens.length > 0) { - const placeholders = tokens.map((_, i) => `$${i + 2}`).join(', '); - whereClause += ` AND token IN (${placeholders})`; - params = [yesterday, ...tokens]; - } - - const result = await pool.query( - `SELECT - token, - SUM(base_volume)::text as base_volume, - SUM(target_volume)::text as target_volume, - MAX(high)::text as high, - MIN(CASE WHEN low > 0 THEN low END)::text as low - FROM daily_volumes - ${whereClause} - GROUP BY token`, - params - ); - - const volumeMap = new Map(); - for (const row of result.rows) { - volumeMap.set(row.token, { - base_volume: row.base_volume || '0', - target_volume: row.target_volume || '0', - high: row.high || '0', - low: row.low || '0', - }); - } - return volumeMap; - } catch (error: any) { - logger.error('[Database] Error getting 24h volumes:', error); - return new Map(); - } - }, - - /** - * Get total daily volume record count - */ - async getRecordCount(): Promise { - return this.getDailyRecordCount(); - }, - - /** - * Get total daily volume record count - */ - async getDailyRecordCount(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return 0; - - try { - const result = await pool.query('SELECT COUNT(*) as count FROM daily_volumes'); - return parseInt(result.rows[0]?.count || '0'); - } catch (error: any) { - logger.error('[Database] Error getting daily record count:', error); - return 0; - } - }, - - /** - * Get unique token count - */ - async getTokenCount(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return 0; - - try { - const result = await pool.query('SELECT COUNT(DISTINCT token) as count FROM daily_volumes'); - return parseInt(result.rows[0]?.count || '0'); - } catch (error: any) { - logger.error('[Database] Error getting token count:', error); - return 0; - } - }, - /** * Get rolling 24h metrics from v06_spot_ohlcv_1m table. * Sources FutarchyAMM rolling-24h volume from the v0.6 indexer. @@ -378,32 +56,6 @@ export function createDailyVolumesRepo(db: DbRuntime) { return new Map(); } }, - - /** - * Get the first trade date for each token (when trading started) - * Returns a Map of token address -> first trade date (YYYY-MM-DD) - */ - async getFirstTradeDates(): Promise> { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return new Map(); - - try { - const result = await pool.query( - `SELECT token, MIN(date)::text AS first_date - FROM daily_buy_sell_volumes - GROUP BY token` - ); - - const map = new Map(); - for (const row of result.rows) { - map.set(row.token, row.first_date); - } - return map; - } catch (error: any) { - logger.error('[Database] Error getting first trade dates:', error); - return new Map(); - } - }, }; } diff --git a/src/services/database/internal/repos/intervalVolumesRepo.ts b/src/services/database/internal/repos/intervalVolumesRepo.ts deleted file mode 100644 index 339aa6b..0000000 --- a/src/services/database/internal/repos/intervalVolumesRepo.ts +++ /dev/null @@ -1,913 +0,0 @@ -import { logger } from '../../../../utils/logger.js'; -import type { DbRuntime } from '../dbRuntime.js'; -import type { HourlyVolumeRecord, TenMinuteVolumeRecord, Rolling24hMetrics } from '../../../databaseService.js'; - -export function createIntervalVolumesRepo(db: DbRuntime) { - return { - // ============================================ - // SYNC METADATA - // ============================================ - - async setSyncMetadata(key: string, value: string): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return; - - try { - await pool.query( - `INSERT INTO sync_metadata (key, value, updated_at) - VALUES ($1, $2, CURRENT_TIMESTAMP) - ON CONFLICT (key) DO UPDATE SET value = $2, updated_at = CURRENT_TIMESTAMP`, - [key, value] - ); - } catch (error: any) { - logger.error('[Database] Error setting sync metadata:', error); - } - }, - - async getSyncMetadata(key: string): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return null; - - try { - const result = await pool.query( - 'SELECT value FROM sync_metadata WHERE key = $1', - [key] - ); - return result.rows[0]?.value || null; - } catch (error: any) { - logger.error('[Database] Error getting sync metadata:', error); - return null; - } - }, - - // ============================================ - // HOURLY VOLUME METHODS - // ============================================ - - async getLatestHour(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return null; - - try { - const result = await pool.query( - 'SELECT MAX(hour) as latest_hour FROM hourly_volumes' - ); - return result.rows[0]?.latest_hour?.toISOString() || null; - } catch (error: any) { - logger.error('[Database] Error getting latest hour:', error); - return null; - } - }, - - async getLatestCompleteHour(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return null; - - try { - const result = await pool.query( - 'SELECT MAX(hour) as latest_hour FROM hourly_volumes WHERE is_complete = true' - ); - return result.rows[0]?.latest_hour?.toISOString() || null; - } catch (error: any) { - logger.error('[Database] Error getting latest complete hour:', error); - return null; - } - }, - - async upsertHourlyVolumes(records: HourlyVolumeRecord[], markComplete: boolean = false): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected() || records.length === 0) return 0; - - const BATCH_SIZE = 500; - let totalUpserted = 0; - - try { - const client = await pool.connect(); - - try { - await client.query('BEGIN'); - - for (let i = 0; i < records.length; i += BATCH_SIZE) { - const batch = records.slice(i, i + BATCH_SIZE); - - const values: any[] = []; - const valuePlaceholders: string[] = []; - - batch.forEach((record, idx) => { - const offset = idx * 14; - valuePlaceholders.push( - `($${offset + 1}, $${offset + 2}, $${offset + 3}, $${offset + 4}, $${offset + 5}, $${offset + 6}, $${offset + 7}, $${offset + 8}, $${offset + 9}, $${offset + 10}, $${offset + 11}, $${offset + 12}, $${offset + 13}, $${offset + 14}, ${markComplete}, CURRENT_TIMESTAMP)` - ); - values.push( - record.token, - record.hour, - record.base_volume, - record.target_volume, - record.buy_volume || '0', - record.sell_volume || '0', - record.high, - record.low, - record.average_price || '0', - record.trade_count || 0, - record.usdc_fees || '0', - record.token_fees || '0', - record.token_fees_usdc || '0', - record.sell_volume_usdc || '0' - ); - }); - - const batchSQL = ` - INSERT INTO hourly_volumes (token, hour, base_volume, target_volume, buy_volume, sell_volume, high, low, average_price, trade_count, usdc_fees, token_fees, token_fees_usdc, sell_volume_usdc, is_complete, updated_at) - VALUES ${valuePlaceholders.join(', ')} - ON CONFLICT (token, hour) - DO UPDATE SET - base_volume = COALESCE(NULLIF(hourly_volumes.base_volume, 0), EXCLUDED.base_volume), - target_volume = COALESCE(NULLIF(hourly_volumes.target_volume, 0), EXCLUDED.target_volume), - high = GREATEST(COALESCE(hourly_volumes.high, 0), COALESCE(EXCLUDED.high, 0)), - low = LEAST( - CASE WHEN hourly_volumes.low > 0 THEN hourly_volumes.low ELSE EXCLUDED.low END, - CASE WHEN EXCLUDED.low > 0 THEN EXCLUDED.low ELSE hourly_volumes.low END - ), - trade_count = GREATEST(COALESCE(hourly_volumes.trade_count, 0), COALESCE(EXCLUDED.trade_count, 0)), - buy_volume = COALESCE(NULLIF(hourly_volumes.buy_volume, 0), EXCLUDED.buy_volume), - sell_volume = COALESCE(NULLIF(hourly_volumes.sell_volume, 0), EXCLUDED.sell_volume), - average_price = COALESCE(NULLIF(hourly_volumes.average_price, 0), EXCLUDED.average_price), - usdc_fees = COALESCE(NULLIF(hourly_volumes.usdc_fees, 0), EXCLUDED.usdc_fees), - token_fees = COALESCE(NULLIF(hourly_volumes.token_fees, 0), EXCLUDED.token_fees), - token_fees_usdc = COALESCE(NULLIF(hourly_volumes.token_fees_usdc, 0), EXCLUDED.token_fees_usdc), - sell_volume_usdc = COALESCE(NULLIF(hourly_volumes.sell_volume_usdc, 0), EXCLUDED.sell_volume_usdc), - is_complete = CASE WHEN EXCLUDED.is_complete THEN true ELSE hourly_volumes.is_complete END, - updated_at = CURRENT_TIMESTAMP - `; - - await client.query(batchSQL, values); - totalUpserted += batch.length; - - if (records.length > BATCH_SIZE) { - logger.info(`[Database] Hourly volume batch progress: ${totalUpserted}/${records.length}`); - } - } - - await client.query('COMMIT'); - logger.info(`[Database] Upserted ${totalUpserted} hourly volume records (complete: ${markComplete})`); - return totalUpserted; - } catch (error) { - await client.query('ROLLBACK'); - throw error; - } finally { - client.release(); - } - } catch (error: any) { - logger.error('[Database] Error upserting hourly volumes:', error); - return 0; - } - }, - - async markHoursComplete(beforeHour: string): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return; - - try { - await pool.query( - `UPDATE hourly_volumes SET is_complete = true, updated_at = CURRENT_TIMESTAMP - WHERE hour < $1 AND is_complete = false`, - [beforeHour] - ); - logger.info(`[Database] Marked hours before ${beforeHour} as complete`); - } catch (error: any) { - logger.error('[Database] Error marking hours complete:', error); - } - }, - - async getRolling24hMetrics(tokens?: string[]): Promise> { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return new Map(); - - try { - const cutoffTime = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); - - let whereClause = 'WHERE hour >= $1'; - let params: any[] = [cutoffTime]; - - if (tokens && tokens.length > 0) { - const placeholders = tokens.map((_, i) => `$${i + 2}`).join(', '); - whereClause += ` AND token IN (${placeholders})`; - params = [cutoffTime, ...tokens]; - } - - const result = await pool.query( - `SELECT - token, - SUM(base_volume)::text as base_volume_24h, - SUM(target_volume)::text as target_volume_24h, - MAX(high)::text as high_24h, - MIN(CASE WHEN low > 0 THEN low END)::text as low_24h, - SUM(trade_count)::int as trade_count_24h - FROM hourly_volumes - ${whereClause} - GROUP BY token`, - params - ); - - const metricsMap = new Map(); - for (const row of result.rows) { - metricsMap.set(row.token, { - token: row.token, - base_volume_24h: row.base_volume_24h || '0', - target_volume_24h: row.target_volume_24h || '0', - high_24h: row.high_24h || '0', - low_24h: row.low_24h || '0', - trade_count_24h: row.trade_count_24h || 0, - }); - } - return metricsMap; - } catch (error: any) { - logger.error('[Database] Error getting rolling 24h metrics:', error); - return new Map(); - } - }, - - async getHourlyVolumes(startHour: string, endHour?: string, tokens?: string[]): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return []; - - try { - let whereClause = 'WHERE hour >= $1'; - let params: any[] = [startHour]; - let paramIndex = 2; - - if (endHour) { - whereClause += ` AND hour <= $${paramIndex}`; - params.push(endHour); - paramIndex++; - } - - if (tokens && tokens.length > 0) { - const placeholders = tokens.map((_, i) => `$${paramIndex + i}`).join(', '); - whereClause += ` AND token IN (${placeholders})`; - params = [...params, ...tokens]; - } - - const result = await pool.query( - `SELECT - token, - hour::text, - base_volume::text, - target_volume::text, - high::text, - low::text, - trade_count - FROM hourly_volumes - ${whereClause} - ORDER BY token, hour ASC`, - params - ); - - return result.rows; - } catch (error: any) { - logger.error('[Database] Error getting hourly volumes:', error); - return []; - } - }, - - async getHourlyRecordCount(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return 0; - - try { - const result = await pool.query('SELECT COUNT(*) as count FROM hourly_volumes'); - return parseInt(result.rows[0]?.count || '0'); - } catch (error: any) { - logger.error('[Database] Error getting hourly record count:', error); - return 0; - } - }, - - async getHourlyTokenCount(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return 0; - - try { - const result = await pool.query('SELECT COUNT(DISTINCT token) as count FROM hourly_volumes'); - return parseInt(result.rows[0]?.count || '0'); - } catch (error: any) { - logger.error('[Database] Error getting hourly token count:', error); - return 0; - } - }, - - async pruneOldHourlyData(keepHours: number = 48): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return 0; - - try { - const cutoffTime = new Date(Date.now() - keepHours * 60 * 60 * 1000).toISOString(); - const result = await pool.query( - 'DELETE FROM hourly_volumes WHERE hour < $1 RETURNING id', - [cutoffTime] - ); - const deletedCount = result.rowCount || 0; - if (deletedCount > 0) { - logger.info(`[Database] Pruned ${deletedCount} hourly records older than ${keepHours} hours`); - } - return deletedCount; - } catch (error: any) { - logger.error('[Database] Error pruning old hourly data:', error); - return 0; - } - }, - - // ============================================ - // 10-MINUTE VOLUME METHODS - // ============================================ - - async upsertTenMinuteVolumes(records: TenMinuteVolumeRecord[], markComplete: boolean = false): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected() || records.length === 0) return 0; - - const BATCH_SIZE = 500; - let totalUpserted = 0; - - try { - const client = await pool.connect(); - - try { - await client.query('BEGIN'); - - for (let i = 0; i < records.length; i += BATCH_SIZE) { - const batch = records.slice(i, i + BATCH_SIZE); - - const values: any[] = []; - const valuePlaceholders: string[] = []; - - batch.forEach((record, idx) => { - const offset = idx * 14; - valuePlaceholders.push( - `($${offset + 1}, $${offset + 2}, $${offset + 3}, $${offset + 4}, $${offset + 5}, $${offset + 6}, $${offset + 7}, $${offset + 8}, $${offset + 9}, $${offset + 10}, $${offset + 11}, $${offset + 12}, $${offset + 13}, $${offset + 14}, ${markComplete}, CURRENT_TIMESTAMP)` - ); - values.push( - record.token, - record.bucket, - record.base_volume, - record.target_volume, - record.buy_volume || '0', - record.sell_volume || '0', - record.high, - record.low, - record.average_price || '0', - record.trade_count || 0, - record.usdc_fees || '0', - record.token_fees || '0', - record.token_fees_usdc || '0', - record.sell_volume_usdc || '0' - ); - }); - - const batchSQL = ` - INSERT INTO ten_minute_volumes (token, bucket, base_volume, target_volume, buy_volume, sell_volume, high, low, average_price, trade_count, usdc_fees, token_fees, token_fees_usdc, sell_volume_usdc, is_complete, updated_at) - VALUES ${valuePlaceholders.join(', ')} - ON CONFLICT (token, bucket) - DO UPDATE SET - base_volume = COALESCE(NULLIF(ten_minute_volumes.base_volume, 0), EXCLUDED.base_volume), - target_volume = COALESCE(NULLIF(ten_minute_volumes.target_volume, 0), EXCLUDED.target_volume), - high = GREATEST(COALESCE(ten_minute_volumes.high, 0), COALESCE(EXCLUDED.high, 0)), - low = LEAST( - CASE WHEN ten_minute_volumes.low > 0 THEN ten_minute_volumes.low ELSE EXCLUDED.low END, - CASE WHEN EXCLUDED.low > 0 THEN EXCLUDED.low ELSE ten_minute_volumes.low END - ), - buy_volume = COALESCE(NULLIF(ten_minute_volumes.buy_volume, 0), EXCLUDED.buy_volume), - sell_volume = COALESCE(NULLIF(ten_minute_volumes.sell_volume, 0), EXCLUDED.sell_volume), - average_price = COALESCE(NULLIF(ten_minute_volumes.average_price, 0), EXCLUDED.average_price), - trade_count = GREATEST(COALESCE(ten_minute_volumes.trade_count, 0), COALESCE(EXCLUDED.trade_count, 0)), - usdc_fees = COALESCE(NULLIF(ten_minute_volumes.usdc_fees, 0), EXCLUDED.usdc_fees), - token_fees = COALESCE(NULLIF(ten_minute_volumes.token_fees, 0), EXCLUDED.token_fees), - token_fees_usdc = COALESCE(NULLIF(ten_minute_volumes.token_fees_usdc, 0), EXCLUDED.token_fees_usdc), - sell_volume_usdc = COALESCE(NULLIF(ten_minute_volumes.sell_volume_usdc, 0), EXCLUDED.sell_volume_usdc), - is_complete = CASE WHEN EXCLUDED.is_complete THEN true ELSE ten_minute_volumes.is_complete END, - updated_at = CURRENT_TIMESTAMP - `; - - await client.query(batchSQL, values); - totalUpserted += batch.length; - - if (records.length > BATCH_SIZE) { - logger.info(`[Database] 10-min volume batch progress: ${totalUpserted}/${records.length}`); - } - } - - await client.query('COMMIT'); - logger.info(`[Database] Upserted ${totalUpserted} 10-minute volume records (complete: ${markComplete})`); - return totalUpserted; - } catch (error) { - await client.query('ROLLBACK'); - throw error; - } finally { - client.release(); - } - } catch (error: any) { - logger.error('[Database] Error upserting 10-minute volumes:', error); - return 0; - } - }, - - async markTenMinuteBucketsComplete(beforeBucket: string): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return; - - try { - await pool.query( - `UPDATE ten_minute_volumes SET is_complete = true, updated_at = CURRENT_TIMESTAMP - WHERE bucket < $1 AND is_complete = false`, - [beforeBucket] - ); - } catch (error: any) { - logger.error('[Database] Error marking 10-min buckets complete:', error); - } - }, - - async backfillMissingFields(): Promise<{ - tenMinuteUpdated: number; - hourlyUpdated: number; - dailyUpdated: number; - }> { - const pool = db.getPool(); - if (!pool || !db.isConnected()) { - return { tenMinuteUpdated: 0, hourlyUpdated: 0, dailyUpdated: 0 }; - } - - logger.info('[Database] Starting backfill of missing extended fields using DB aggregation functions...'); - const results = { - tenMinuteUpdated: 0, - hourlyUpdated: 0, - dailyUpdated: 0, - }; - - try { - // 1. Find all hours that have 10-minute data but missing/zero extended fields - logger.info('[Database] Finding hours with missing extended fields that have 10-minute data...'); - const hoursToUpdate = await pool.query(` - SELECT DISTINCT - hv.token, - hv.hour - FROM hourly_volumes hv - WHERE EXISTS ( - SELECT 1 - FROM ten_minute_volumes tmv - WHERE tmv.token = hv.token - AND date_trunc('hour', tmv.bucket) = hv.hour - ) - AND ( - hv.average_price IS NULL OR hv.average_price = 0 OR - hv.usdc_fees IS NULL OR hv.usdc_fees = 0 OR - hv.sell_volume_usdc IS NULL OR hv.sell_volume_usdc = 0 OR - hv.buy_volume IS NULL OR hv.buy_volume = 0 OR - hv.sell_volume IS NULL OR hv.sell_volume = 0 OR - hv.token_fees IS NULL OR hv.token_fees = 0 OR - hv.token_fees_usdc IS NULL OR hv.token_fees_usdc = 0 - ) - ORDER BY hv.token, hv.hour - `); - - logger.info(`[Database] Found ${hoursToUpdate.rows.length} hours to update from 10-minute data`); - - if (hoursToUpdate.rows.length > 0) { - logger.info('[Database] Backfilling hourly records from 10-minute data using aggregate_10min_to_hourly function...'); - - for (const row of hoursToUpdate.rows) { - const tokenValue = String(row.token); - const hourValue = row.hour instanceof Date ? row.hour.toISOString() : String(row.hour); - - try { - const client = await pool.connect(); - try { - const escapedToken = client.escapeLiteral(tokenValue); - const escapedHour = client.escapeLiteral(hourValue); - - const aggregated = await client.query( - `SELECT * FROM aggregate_10min_to_hourly(${escapedToken}::VARCHAR, ${escapedHour}::TIMESTAMPTZ)` - ); - - if (aggregated.rows.length > 0) { - const agg = aggregated.rows[0]; - - if (agg.average_price != null || agg.usdc_fees != null || agg.sell_volume_usdc != null) { - const buyVolume = agg.buy_volume != null ? Number(agg.buy_volume) : null; - const sellVolume = agg.sell_volume != null ? Number(agg.sell_volume) : null; - const averagePrice = agg.average_price != null ? Number(agg.average_price) : null; - const usdcFees = agg.usdc_fees != null ? Number(agg.usdc_fees) : null; - const tokenFees = agg.token_fees != null ? Number(agg.token_fees) : null; - const tokenFeesUsdc = agg.token_fees_usdc != null ? Number(agg.token_fees_usdc) : null; - const sellVolumeUsdc = agg.sell_volume_usdc != null ? Number(agg.sell_volume_usdc) : null; - - logger.debug(`[Database] Updating hour ${hourValue} for token ${tokenValue}`, { - buyVolume, sellVolume, averagePrice, usdcFees, tokenFees, tokenFeesUsdc, sellVolumeUsdc, - tokenValue, hourValue, rawAgg: agg - }); - - await client.query( - `UPDATE hourly_volumes - SET - buy_volume = CASE WHEN $1::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(buy_volume, 0), $1::NUMERIC) ELSE buy_volume END, - sell_volume = CASE WHEN $2::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(sell_volume, 0), $2::NUMERIC) ELSE sell_volume END, - average_price = CASE WHEN $3::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(average_price, 0), $3::NUMERIC) ELSE average_price END, - usdc_fees = CASE WHEN $4::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(usdc_fees, 0), $4::NUMERIC) ELSE usdc_fees END, - token_fees = CASE WHEN $5::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(token_fees, 0), $5::NUMERIC) ELSE token_fees END, - token_fees_usdc = CASE WHEN $6::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(token_fees_usdc, 0), $6::NUMERIC) ELSE token_fees_usdc END, - sell_volume_usdc = CASE WHEN $7::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(sell_volume_usdc, 0), $7::NUMERIC) ELSE sell_volume_usdc END, - updated_at = CURRENT_TIMESTAMP - WHERE token = $8::VARCHAR AND hour = $9::TIMESTAMPTZ - AND ( - average_price IS NULL OR average_price = 0 OR - usdc_fees IS NULL OR usdc_fees = 0 OR - sell_volume_usdc IS NULL OR sell_volume_usdc = 0 OR - buy_volume IS NULL OR buy_volume = 0 OR - sell_volume IS NULL OR sell_volume = 0 OR - token_fees IS NULL OR token_fees = 0 OR - token_fees_usdc IS NULL OR token_fees_usdc = 0 - )`, - [buyVolume, sellVolume, averagePrice, usdcFees, tokenFees, tokenFeesUsdc, sellVolumeUsdc, tokenValue, hourValue] - ); - results.hourlyUpdated++; - } - } - } finally { - client.release(); - } - } catch (error: any) { - logger.error(`[Database] Error updating hour ${row.hour} for token ${row.token}:`, { - error: error.message, stack: error.stack, - token: row.token, hour: row.hour, tokenValue, hourValue, - tokenType: typeof row.token, hourType: typeof row.hour, - hourIsDate: row.hour instanceof Date, hourRawValue: row.hour, - }); - } - } - logger.info(`[Database] Updated ${results.hourlyUpdated} hourly records with missing fields`); - } - - // 2. Find all days that have hourly data but missing/zero extended fields - logger.info('[Database] Finding days with missing extended fields that have hourly data...'); - const daysToUpdate = await pool.query(` - SELECT DISTINCT - dv.token, - dv.date - FROM daily_volumes dv - WHERE EXISTS ( - SELECT 1 - FROM hourly_volumes hv - WHERE hv.token = dv.token - AND date_trunc('day', hv.hour)::DATE = dv.date - ) - AND ( - dv.average_price IS NULL OR dv.average_price = 0 OR - dv.usdc_fees IS NULL OR dv.usdc_fees = 0 OR - dv.sell_volume_usdc IS NULL OR dv.sell_volume_usdc = 0 OR - dv.buy_volume IS NULL OR dv.buy_volume = 0 OR - dv.sell_volume IS NULL OR dv.sell_volume = 0 OR - dv.token_fees IS NULL OR dv.token_fees = 0 OR - dv.token_fees_usdc IS NULL OR dv.token_fees_usdc = 0 - ) - ORDER BY dv.token, dv.date - `); - - logger.info(`[Database] Found ${daysToUpdate.rows.length} days to update from hourly data`); - - if (daysToUpdate.rows.length > 0) { - logger.info('[Database] Backfilling daily records from hourly data using aggregate_hourly_to_daily function...'); - - for (const row of daysToUpdate.rows) { - const tokenValue = String(row.token); - const dateValue = row.date instanceof Date ? row.date.toISOString().split('T')[0] : String(row.date); - - try { - const client = await pool.connect(); - try { - const escapedToken = client.escapeLiteral(tokenValue); - const escapedDate = client.escapeLiteral(dateValue); - - const aggregated = await client.query( - `SELECT * FROM aggregate_hourly_to_daily(${escapedToken}::VARCHAR, ${escapedDate}::DATE)` - ); - - if (aggregated.rows.length > 0) { - const agg = aggregated.rows[0]; - - if (agg.average_price != null || agg.usdc_fees != null || agg.sell_volume_usdc != null) { - const buyVolume = agg.buy_volume != null ? Number(agg.buy_volume) : null; - const sellVolume = agg.sell_volume != null ? Number(agg.sell_volume) : null; - const averagePrice = agg.average_price != null ? Number(agg.average_price) : null; - const tradeCount = agg.trade_count != null ? Number(agg.trade_count) : null; - const usdcFees = agg.usdc_fees != null ? Number(agg.usdc_fees) : null; - const tokenFees = agg.token_fees != null ? Number(agg.token_fees) : null; - const tokenFeesUsdc = agg.token_fees_usdc != null ? Number(agg.token_fees_usdc) : null; - const sellVolumeUsdc = agg.sell_volume_usdc != null ? Number(agg.sell_volume_usdc) : null; - const cumulativeUsdcFees = agg.cumulative_usdc_fees != null ? Number(agg.cumulative_usdc_fees) : null; - const cumulativeTokenInUsdcFees = agg.cumulative_token_in_usdc_fees != null ? Number(agg.cumulative_token_in_usdc_fees) : null; - const cumulativeTargetVolume = agg.cumulative_target_volume != null ? Number(agg.cumulative_target_volume) : null; - const cumulativeTokenVolume = agg.cumulative_token_volume != null ? Number(agg.cumulative_token_volume) : null; - - logger.debug(`[Database] Updating day ${dateValue} for token ${tokenValue}`, { - buyVolume, sellVolume, averagePrice, tradeCount, usdcFees, tokenFees, tokenFeesUsdc, - sellVolumeUsdc, cumulativeUsdcFees, cumulativeTokenInUsdcFees, cumulativeTargetVolume, - cumulativeTokenVolume, tokenValue, dateValue, rawAgg: agg - }); - - await client.query( - `UPDATE daily_volumes - SET - buy_volume = CASE WHEN $1::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(buy_volume, 0), $1::NUMERIC) ELSE buy_volume END, - sell_volume = CASE WHEN $2::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(sell_volume, 0), $2::NUMERIC) ELSE sell_volume END, - average_price = CASE WHEN $3::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(average_price, 0), $3::NUMERIC) ELSE average_price END, - trade_count = CASE WHEN $4::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(trade_count, 0), $4::NUMERIC) ELSE trade_count END, - usdc_fees = CASE WHEN $5::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(usdc_fees, 0), $5::NUMERIC) ELSE usdc_fees END, - token_fees = CASE WHEN $6::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(token_fees, 0), $6::NUMERIC) ELSE token_fees END, - token_fees_usdc = CASE WHEN $7::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(token_fees_usdc, 0), $7::NUMERIC) ELSE token_fees_usdc END, - sell_volume_usdc = CASE WHEN $8::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(sell_volume_usdc, 0), $8::NUMERIC) ELSE sell_volume_usdc END, - cumulative_usdc_fees = CASE WHEN $9::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(cumulative_usdc_fees, 0), $9::NUMERIC) ELSE cumulative_usdc_fees END, - cumulative_token_in_usdc_fees = CASE WHEN $10::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(cumulative_token_in_usdc_fees, 0), $10::NUMERIC) ELSE cumulative_token_in_usdc_fees END, - cumulative_target_volume = CASE WHEN $11::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(cumulative_target_volume, 0), $11::NUMERIC) ELSE cumulative_target_volume END, - cumulative_token_volume = CASE WHEN $12::NUMERIC IS NOT NULL THEN COALESCE(NULLIF(cumulative_token_volume, 0), $12::NUMERIC) ELSE cumulative_token_volume END, - updated_at = CURRENT_TIMESTAMP - WHERE token = $13::VARCHAR AND date = $14::DATE - AND ( - average_price IS NULL OR average_price = 0 OR - usdc_fees IS NULL OR usdc_fees = 0 OR - sell_volume_usdc IS NULL OR sell_volume_usdc = 0 OR - buy_volume IS NULL OR buy_volume = 0 OR - sell_volume IS NULL OR sell_volume = 0 OR - token_fees IS NULL OR token_fees = 0 OR - token_fees_usdc IS NULL OR token_fees_usdc = 0 - )`, - [buyVolume, sellVolume, averagePrice, tradeCount, usdcFees, tokenFees, tokenFeesUsdc, - sellVolumeUsdc, cumulativeUsdcFees, cumulativeTokenInUsdcFees, cumulativeTargetVolume, - cumulativeTokenVolume, tokenValue, dateValue] - ); - results.dailyUpdated++; - } - } - } finally { - client.release(); - } - } catch (error: any) { - logger.error(`[Database] Error updating day ${row.date} for token ${row.token}:`, { - error: error.message, stack: error.stack, - token: row.token, date: row.date, tokenValue, dateValue, - tokenType: typeof row.token, dateType: typeof row.date, - dateIsDate: row.date instanceof Date, dateRawValue: row.date, - }); - } - } - logger.info(`[Database] Updated ${results.dailyUpdated} daily records with missing fields`); - } - - logger.info('[Database] Backfill completed:', { results }); - return results; - } catch (error: any) { - logger.error('[Database] Error during backfill:', error); - return results; - } - }, - - async getRolling24hFromTenMinute(tokens?: string[]): Promise> { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return new Map(); - - try { - const metricsMap = new Map(); - - if (tokens && tokens.length > 0) { - for (const token of tokens) { - const result = await pool.query( - `SELECT * FROM calculate_rolling_24h($1)`, - [token] - ); - - for (const row of result.rows) { - metricsMap.set(row.token, { - token: row.token, - base_volume_24h: row.base_volume_24h?.toString() || '0', - target_volume_24h: row.target_volume_24h?.toString() || '0', - high_24h: row.high_24h?.toString() || '0', - low_24h: row.low_24h?.toString() || '0', - trade_count_24h: row.trade_count_24h || 0, - }); - } - } - } else { - const result = await pool.query( - `SELECT * FROM calculate_rolling_24h(NULL)` - ); - - for (const row of result.rows) { - metricsMap.set(row.token, { - token: row.token, - base_volume_24h: row.base_volume_24h?.toString() || '0', - target_volume_24h: row.target_volume_24h?.toString() || '0', - high_24h: row.high_24h?.toString() || '0', - low_24h: row.low_24h?.toString() || '0', - trade_count_24h: row.trade_count_24h || 0, - }); - } - } - - return metricsMap; - } catch (error: any) { - logger.error('[Database] Error getting rolling 24h from 10-min data:', error); - return new Map(); - } - }, - - async aggregate10MinToHourly(token?: string, hour?: string): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return 0; - - try { - const result = await pool.query( - `SELECT * FROM aggregate_10min_to_hourly(CAST($1 AS VARCHAR), CAST($2 AS TIMESTAMPTZ))`, - [token || null, hour || null] - ); - - if (result.rows.length === 0) { - return 0; - } - - const client = await pool.connect(); - try { - await client.query('BEGIN'); - - for (const row of result.rows) { - await client.query( - `INSERT INTO hourly_volumes ( - token, hour, base_volume, target_volume, buy_volume, sell_volume, - high, low, average_price, trade_count, usdc_fees, token_fees, - token_fees_usdc, sell_volume_usdc, is_complete, updated_at - ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, true, CURRENT_TIMESTAMP) - ON CONFLICT (token, hour) DO UPDATE SET - base_volume = COALESCE(NULLIF(hourly_volumes.base_volume, 0), EXCLUDED.base_volume), - target_volume = COALESCE(NULLIF(hourly_volumes.target_volume, 0), EXCLUDED.target_volume), - high = GREATEST(COALESCE(hourly_volumes.high, 0), COALESCE(EXCLUDED.high, 0)), - low = LEAST( - CASE WHEN hourly_volumes.low > 0 THEN hourly_volumes.low ELSE EXCLUDED.low END, - CASE WHEN EXCLUDED.low > 0 THEN EXCLUDED.low ELSE hourly_volumes.low END - ), - trade_count = GREATEST(COALESCE(hourly_volumes.trade_count, 0), COALESCE(EXCLUDED.trade_count, 0)), - buy_volume = COALESCE(NULLIF(hourly_volumes.buy_volume, 0), EXCLUDED.buy_volume), - sell_volume = COALESCE(NULLIF(hourly_volumes.sell_volume, 0), EXCLUDED.sell_volume), - average_price = COALESCE(NULLIF(hourly_volumes.average_price, 0), EXCLUDED.average_price), - usdc_fees = COALESCE(NULLIF(hourly_volumes.usdc_fees, 0), EXCLUDED.usdc_fees), - token_fees = COALESCE(NULLIF(hourly_volumes.token_fees, 0), EXCLUDED.token_fees), - token_fees_usdc = COALESCE(NULLIF(hourly_volumes.token_fees_usdc, 0), EXCLUDED.token_fees_usdc), - sell_volume_usdc = COALESCE(NULLIF(hourly_volumes.sell_volume_usdc, 0), EXCLUDED.sell_volume_usdc), - is_complete = true, - updated_at = CURRENT_TIMESTAMP`, - [ - row.token, row.hour, row.base_volume, row.target_volume, - row.buy_volume, row.sell_volume, row.high, row.low, - row.average_price, row.trade_count, row.usdc_fees, row.token_fees, - row.token_fees_usdc, row.sell_volume_usdc, - ] - ); - } - - await client.query('COMMIT'); - logger.info(`[Database] Aggregated ${result.rows.length} hourly records from 10-minute data`); - return result.rows.length; - } catch (error) { - await client.query('ROLLBACK'); - throw error; - } finally { - client.release(); - } - } catch (error: any) { - logger.error('[Database] Error aggregating 10-min to hourly:', error); - return 0; - } - }, - - async aggregateHourlyToDaily(token?: string, date?: string): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return 0; - - try { - const result = await pool.query( - `SELECT * FROM aggregate_hourly_to_daily(CAST($1 AS VARCHAR), CAST($2 AS DATE))`, - [token || null, date || null] - ); - - if (result.rows.length === 0) { - return 0; - } - - const client = await pool.connect(); - try { - await client.query('BEGIN'); - - for (const row of result.rows) { - await client.query( - `INSERT INTO daily_volumes ( - token, date, base_volume, target_volume, buy_volume, sell_volume, - high, low, average_price, trade_count, usdc_fees, token_fees, - token_fees_usdc, sell_volume_usdc, cumulative_usdc_fees, - cumulative_token_in_usdc_fees, cumulative_target_volume, - cumulative_token_volume, is_complete, updated_at - ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, true, CURRENT_TIMESTAMP) - ON CONFLICT (token, date) DO UPDATE SET - base_volume = COALESCE(NULLIF(daily_volumes.base_volume, 0), EXCLUDED.base_volume), - target_volume = COALESCE(NULLIF(daily_volumes.target_volume, 0), EXCLUDED.target_volume), - high = GREATEST(COALESCE(daily_volumes.high, 0), COALESCE(EXCLUDED.high, 0)), - low = LEAST( - CASE WHEN daily_volumes.low > 0 THEN daily_volumes.low ELSE EXCLUDED.low END, - CASE WHEN EXCLUDED.low > 0 THEN EXCLUDED.low ELSE daily_volumes.low END - ), - trade_count = GREATEST(COALESCE(daily_volumes.trade_count, 0), COALESCE(EXCLUDED.trade_count, 0)), - buy_volume = COALESCE(NULLIF(daily_volumes.buy_volume, 0), EXCLUDED.buy_volume), - sell_volume = COALESCE(NULLIF(daily_volumes.sell_volume, 0), EXCLUDED.sell_volume), - average_price = COALESCE(NULLIF(daily_volumes.average_price, 0), EXCLUDED.average_price), - usdc_fees = COALESCE(NULLIF(daily_volumes.usdc_fees, 0), EXCLUDED.usdc_fees), - token_fees = COALESCE(NULLIF(daily_volumes.token_fees, 0), EXCLUDED.token_fees), - token_fees_usdc = COALESCE(NULLIF(daily_volumes.token_fees_usdc, 0), EXCLUDED.token_fees_usdc), - sell_volume_usdc = COALESCE(NULLIF(daily_volumes.sell_volume_usdc, 0), EXCLUDED.sell_volume_usdc), - cumulative_usdc_fees = COALESCE(NULLIF(daily_volumes.cumulative_usdc_fees, 0), EXCLUDED.cumulative_usdc_fees), - cumulative_token_in_usdc_fees = COALESCE(NULLIF(daily_volumes.cumulative_token_in_usdc_fees, 0), EXCLUDED.cumulative_token_in_usdc_fees), - cumulative_target_volume = COALESCE(NULLIF(daily_volumes.cumulative_target_volume, 0), EXCLUDED.cumulative_target_volume), - cumulative_token_volume = COALESCE(NULLIF(daily_volumes.cumulative_token_volume, 0), EXCLUDED.cumulative_token_volume), - is_complete = true, - updated_at = CURRENT_TIMESTAMP`, - [ - row.token, row.date, row.base_volume, row.target_volume, - row.buy_volume, row.sell_volume, row.high, row.low, - row.average_price, row.trade_count, row.usdc_fees, row.token_fees, - row.token_fees_usdc, row.sell_volume_usdc, row.cumulative_usdc_fees, - row.cumulative_token_in_usdc_fees, row.cumulative_target_volume, - row.cumulative_token_volume, - ] - ); - } - - await client.query('COMMIT'); - logger.info(`[Database] Aggregated ${result.rows.length} daily records from hourly data`); - return result.rows.length; - } catch (error) { - await client.query('ROLLBACK'); - throw error; - } finally { - client.release(); - } - } catch (error: any) { - logger.error('[Database] Error aggregating hourly to daily:', error); - return 0; - } - }, - - async getLatestTenMinuteBucket(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return null; - - try { - const result = await pool.query( - 'SELECT MAX(bucket) as latest_bucket FROM ten_minute_volumes' - ); - return result.rows[0]?.latest_bucket?.toISOString() || null; - } catch (error: any) { - logger.error('[Database] Error getting latest 10-min bucket:', error); - return null; - } - }, - - async getTenMinuteRecordCount(): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return 0; - - try { - const result = await pool.query('SELECT COUNT(*) as count FROM ten_minute_volumes'); - return parseInt(result.rows[0]?.count || '0'); - } catch (error: any) { - logger.error('[Database] Error getting 10-min record count:', error); - return 0; - } - }, - - async pruneOldTenMinuteData(keepHours: number = 25): Promise { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return 0; - - try { - const cutoffTime = new Date(Date.now() - keepHours * 60 * 60 * 1000).toISOString(); - const result = await pool.query( - 'DELETE FROM ten_minute_volumes WHERE bucket < $1 RETURNING id', - [cutoffTime] - ); - const deletedCount = result.rowCount || 0; - if (deletedCount > 0) { - logger.info(`[Database] Pruned ${deletedCount} 10-minute records older than ${keepHours} hours`); - } - return deletedCount; - } catch (error: any) { - logger.error('[Database] Error pruning old 10-min data:', error); - return 0; - } - }, - }; -} - -export type IntervalVolumesRepo = ReturnType; diff --git a/src/services/database/internal/schemaManager.ts b/src/services/database/internal/schemaManager.ts index 5ccac3c..d0f0ea6 100644 --- a/src/services/database/internal/schemaManager.ts +++ b/src/services/database/internal/schemaManager.ts @@ -8,94 +8,6 @@ export function createSchemaManager(db: DbRuntime) { if (!pool) return; const createTableSQL = ` - -- Daily volumes table (aggregated from hourly/10-min data, includes cumulative values) - CREATE TABLE IF NOT EXISTS daily_volumes ( - id SERIAL PRIMARY KEY, - token VARCHAR(64) NOT NULL, - date DATE NOT NULL, - base_volume NUMERIC(40, 12) NOT NULL, - target_volume NUMERIC(40, 12) NOT NULL, - buy_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - sell_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - high NUMERIC(40, 12) NOT NULL, - low NUMERIC(40, 12) NOT NULL, - average_price NUMERIC(40, 12) NOT NULL DEFAULT 0, - trade_count INT NOT NULL DEFAULT 0, - usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - token_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - token_fees_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - sell_volume_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - cumulative_usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - cumulative_token_in_usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - cumulative_target_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - cumulative_token_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - is_complete BOOLEAN NOT NULL DEFAULT false, - created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - UNIQUE(token, date) - ); - - CREATE INDEX IF NOT EXISTS idx_daily_volumes_token ON daily_volumes(token); - CREATE INDEX IF NOT EXISTS idx_daily_volumes_date ON daily_volumes(date); - CREATE INDEX IF NOT EXISTS idx_daily_volumes_token_date ON daily_volumes(token, date); - - -- Hourly volumes table for hourly aggregates (aggregated from 10-min data) - CREATE TABLE IF NOT EXISTS hourly_volumes ( - id SERIAL PRIMARY KEY, - token VARCHAR(64) NOT NULL, - hour TIMESTAMPTZ NOT NULL, - base_volume NUMERIC(40, 12) NOT NULL, - target_volume NUMERIC(40, 12) NOT NULL, - buy_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - sell_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - high NUMERIC(40, 12) NOT NULL, - low NUMERIC(40, 12) NOT NULL, - average_price NUMERIC(40, 12) NOT NULL DEFAULT 0, - trade_count INT NOT NULL DEFAULT 0, - usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - token_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - token_fees_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - sell_volume_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - is_complete BOOLEAN NOT NULL DEFAULT false, - created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - UNIQUE(token, hour) - ); - - CREATE INDEX IF NOT EXISTS idx_hourly_volumes_token ON hourly_volumes(token); - CREATE INDEX IF NOT EXISTS idx_hourly_volumes_hour ON hourly_volumes(hour); - CREATE INDEX IF NOT EXISTS idx_hourly_volumes_token_hour ON hourly_volumes(token, hour); - CREATE INDEX IF NOT EXISTS idx_hourly_volumes_recent ON hourly_volumes(hour DESC); - - -- 10-minute volumes table for accurate rolling 24h calculations - -- Extended with buy/sell volumes and fees (single source of truth) - CREATE TABLE IF NOT EXISTS ten_minute_volumes ( - id SERIAL PRIMARY KEY, - token VARCHAR(64) NOT NULL, - bucket TIMESTAMPTZ NOT NULL, - base_volume NUMERIC(40, 12) NOT NULL, - target_volume NUMERIC(40, 12) NOT NULL, - buy_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - sell_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - high NUMERIC(40, 12) NOT NULL, - low NUMERIC(40, 12) NOT NULL, - average_price NUMERIC(40, 12) NOT NULL DEFAULT 0, - trade_count INT NOT NULL DEFAULT 0, - usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - token_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - token_fees_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - sell_volume_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - is_complete BOOLEAN NOT NULL DEFAULT false, - created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - UNIQUE(token, bucket) - ); - - CREATE INDEX IF NOT EXISTS idx_ten_minute_volumes_token ON ten_minute_volumes(token); - CREATE INDEX IF NOT EXISTS idx_ten_minute_volumes_bucket ON ten_minute_volumes(bucket); - CREATE INDEX IF NOT EXISTS idx_ten_minute_volumes_token_bucket ON ten_minute_volumes(token, bucket); - CREATE INDEX IF NOT EXISTS idx_ten_minute_volumes_recent ON ten_minute_volumes(bucket DESC); - -- Metadata table to track sync status CREATE TABLE IF NOT EXISTS sync_metadata ( key VARCHAR(64) PRIMARY KEY, @@ -103,58 +15,6 @@ export function createSchemaManager(db: DbRuntime) { updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP ); - -- Daily buy/sell volumes table for tracking directional volume - CREATE TABLE IF NOT EXISTS daily_buy_sell_volumes ( - id SERIAL PRIMARY KEY, - token VARCHAR(64) NOT NULL, - date DATE NOT NULL, - base_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - target_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - buy_usdc_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - sell_token_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - high NUMERIC(40, 12) NOT NULL DEFAULT 0, - low NUMERIC(40, 12) NOT NULL DEFAULT 0, - trade_count INT NOT NULL DEFAULT 0, - is_complete BOOLEAN NOT NULL DEFAULT false, - created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - UNIQUE(token, date) - ); - - CREATE INDEX IF NOT EXISTS idx_daily_buy_sell_volumes_token ON daily_buy_sell_volumes(token); - CREATE INDEX IF NOT EXISTS idx_daily_buy_sell_volumes_date ON daily_buy_sell_volumes(date); - CREATE INDEX IF NOT EXISTS idx_daily_buy_sell_volumes_token_date ON daily_buy_sell_volumes(token, date); - - -- Daily fees volumes table for tracking fees and comprehensive volume metrics - CREATE TABLE IF NOT EXISTS daily_fees_volumes ( - id SERIAL PRIMARY KEY, - token VARCHAR(64) NOT NULL, - trading_date DATE NOT NULL, - base_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - target_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - token_fees_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - token_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - buy_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - sell_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - sell_volume_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - cumulative_usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - cumulative_token_in_usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - cumulative_target_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - cumulative_token_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - high NUMERIC(40, 12) NOT NULL DEFAULT 0, - average_price NUMERIC(40, 12) NOT NULL DEFAULT 0, - low NUMERIC(40, 12) NOT NULL DEFAULT 0, - is_complete BOOLEAN NOT NULL DEFAULT false, - created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - UNIQUE(token, trading_date) - ); - - CREATE INDEX IF NOT EXISTS idx_daily_fees_volumes_token ON daily_fees_volumes(token); - CREATE INDEX IF NOT EXISTS idx_daily_fees_volumes_date ON daily_fees_volumes(trading_date); - CREATE INDEX IF NOT EXISTS idx_daily_fees_volumes_token_date ON daily_fees_volumes(token, trading_date); - -- Metrics history table for storing periodic snapshots of system metrics CREATE TABLE IF NOT EXISTS metrics_history ( id SERIAL PRIMARY KEY, @@ -191,9 +51,6 @@ export function createSchemaManager(db: DbRuntime) { // Create v0.6 OHLCV + fee volume tables await manager.createV06Tables(); - - // Run migration to add new columns to existing tables - await manager.migrateTables(); }, async createV06Tables(): Promise { @@ -333,288 +190,6 @@ export function createSchemaManager(db: DbRuntime) { await pool.query(sql); logger.info('[Database] v0.6 tables created/verified'); }, - - async migrateTables(): Promise { - const pool = db.getPool(); - if (!pool) return; - - try { - const migrationSQL = ` - -- Add extended columns to ten_minute_volumes - DO $$ - BEGIN - IF NOT EXISTS (SELECT 1 FROM information_schema.columns - WHERE table_name = 'ten_minute_volumes' AND column_name = 'buy_volume') THEN - ALTER TABLE ten_minute_volumes - ADD COLUMN buy_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN sell_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN average_price NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN token_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN token_fees_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN sell_volume_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0; - RAISE NOTICE 'Added extended columns to ten_minute_volumes'; - END IF; - END $$; - - -- Add extended columns to hourly_volumes - DO $$ - BEGIN - IF NOT EXISTS (SELECT 1 FROM information_schema.columns - WHERE table_name = 'hourly_volumes' AND column_name = 'buy_volume') THEN - ALTER TABLE hourly_volumes - ADD COLUMN buy_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN sell_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN average_price NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN token_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN token_fees_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN sell_volume_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0; - RAISE NOTICE 'Added extended columns to hourly_volumes'; - END IF; - END $$; - - -- Add extended columns to daily_volumes - DO $$ - BEGIN - IF NOT EXISTS (SELECT 1 FROM information_schema.columns - WHERE table_name = 'daily_volumes' AND column_name = 'buy_volume') THEN - ALTER TABLE daily_volumes - ADD COLUMN buy_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN sell_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN average_price NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN trade_count INT NOT NULL DEFAULT 0, - ADD COLUMN usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN token_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN token_fees_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN sell_volume_usdc NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN cumulative_usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN cumulative_token_in_usdc_fees NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN cumulative_target_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN cumulative_token_volume NUMERIC(40, 12) NOT NULL DEFAULT 0, - ADD COLUMN is_complete BOOLEAN NOT NULL DEFAULT false; - RAISE NOTICE 'Added extended columns to daily_volumes'; - END IF; - END $$; - `; - - await pool.query(migrationSQL); - logger.info('[Database] Migration completed - extended columns added if needed'); - } catch (error: any) { - // Migration errors are non-fatal - tables might already have columns - logger.info('[Database] Migration check completed (columns may already exist)'); - } - }, - - async createAggregationFunctions(): Promise { - const pool = db.getPool(); - if (!pool) return; - - try { - const functionsSQL = ` - -- Function to aggregate 10-minute buckets into hourly records - CREATE OR REPLACE FUNCTION aggregate_10min_to_hourly( - p_token VARCHAR DEFAULT NULL, - p_hour TIMESTAMPTZ DEFAULT NULL - ) - RETURNS TABLE ( - token VARCHAR, - hour TIMESTAMPTZ, - base_volume NUMERIC, - target_volume NUMERIC, - buy_volume NUMERIC, - sell_volume NUMERIC, - high NUMERIC, - low NUMERIC, - average_price NUMERIC, - trade_count INT, - usdc_fees NUMERIC, - token_fees NUMERIC, - token_fees_usdc NUMERIC, - sell_volume_usdc NUMERIC - ) AS $$ - BEGIN - RETURN QUERY - SELECT - tmv.token, - date_trunc('hour', tmv.bucket) AS hour, - SUM(tmv.base_volume)::NUMERIC AS base_volume, - SUM(tmv.target_volume)::NUMERIC AS target_volume, - SUM(tmv.buy_volume)::NUMERIC AS buy_volume, - SUM(tmv.sell_volume)::NUMERIC AS sell_volume, - MAX(tmv.high)::NUMERIC AS high, - MIN(CASE WHEN tmv.low > 0 THEN tmv.low END)::NUMERIC AS low, - -- Weighted average price by volume - CASE - WHEN SUM(tmv.base_volume) > 0 - THEN SUM(tmv.average_price * tmv.base_volume) / SUM(tmv.base_volume) - ELSE AVG(tmv.average_price) - END::NUMERIC AS average_price, - SUM(tmv.trade_count)::INT AS trade_count, - SUM(tmv.usdc_fees)::NUMERIC AS usdc_fees, - SUM(tmv.token_fees)::NUMERIC AS token_fees, - SUM(tmv.token_fees_usdc)::NUMERIC AS token_fees_usdc, - SUM(tmv.sell_volume_usdc)::NUMERIC AS sell_volume_usdc - FROM ten_minute_volumes tmv - WHERE - (p_token IS NULL OR tmv.token = p_token) - AND (p_hour IS NULL OR date_trunc('hour', tmv.bucket) = p_hour) - GROUP BY tmv.token, date_trunc('hour', tmv.bucket) - ORDER BY tmv.token, hour; - END; - $$ LANGUAGE plpgsql; - - -- Function to aggregate hourly records into daily records with cumulative values - CREATE OR REPLACE FUNCTION aggregate_hourly_to_daily( - p_token VARCHAR DEFAULT NULL, - p_date DATE DEFAULT NULL - ) - RETURNS TABLE ( - token VARCHAR, - date DATE, - base_volume NUMERIC, - target_volume NUMERIC, - buy_volume NUMERIC, - sell_volume NUMERIC, - high NUMERIC, - low NUMERIC, - average_price NUMERIC, - trade_count INT, - usdc_fees NUMERIC, - token_fees NUMERIC, - token_fees_usdc NUMERIC, - sell_volume_usdc NUMERIC, - cumulative_usdc_fees NUMERIC, - cumulative_token_in_usdc_fees NUMERIC, - cumulative_target_volume NUMERIC, - cumulative_token_volume NUMERIC - ) AS $$ - BEGIN - RETURN QUERY - WITH daily_agg AS ( - SELECT - hv.token, - date_trunc('day', hv.hour)::DATE AS date, - SUM(hv.base_volume)::NUMERIC AS base_volume, - SUM(hv.target_volume)::NUMERIC AS target_volume, - SUM(hv.buy_volume)::NUMERIC AS buy_volume, - SUM(hv.sell_volume)::NUMERIC AS sell_volume, - MAX(hv.high)::NUMERIC AS high, - MIN(CASE WHEN hv.low > 0 THEN hv.low END)::NUMERIC AS low, - -- Weighted average price by volume - CASE - WHEN SUM(hv.base_volume) > 0 - THEN SUM(hv.average_price * hv.base_volume) / SUM(hv.base_volume) - ELSE AVG(hv.average_price) - END::NUMERIC AS average_price, - SUM(hv.trade_count)::INT AS trade_count, - SUM(hv.usdc_fees)::NUMERIC AS usdc_fees, - SUM(hv.token_fees)::NUMERIC AS token_fees, - SUM(hv.token_fees_usdc)::NUMERIC AS token_fees_usdc, - SUM(hv.sell_volume_usdc)::NUMERIC AS sell_volume_usdc - FROM hourly_volumes hv - WHERE - (p_token IS NULL OR hv.token = p_token) - AND (p_date IS NULL OR date_trunc('day', hv.hour)::DATE = p_date) - GROUP BY hv.token, date_trunc('day', hv.hour)::DATE - ) - SELECT - da.token, - da.date, - da.base_volume, - da.target_volume, - da.buy_volume, - da.sell_volume, - da.high, - da.low, - da.average_price, - da.trade_count, - da.usdc_fees, - da.token_fees, - da.token_fees_usdc, - da.sell_volume_usdc, - SUM(da.usdc_fees) OVER ( - PARTITION BY da.token - ORDER BY da.date - ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW - )::NUMERIC AS cumulative_usdc_fees, - SUM(da.token_fees_usdc) OVER ( - PARTITION BY da.token - ORDER BY da.date - ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW - )::NUMERIC AS cumulative_token_in_usdc_fees, - SUM(da.target_volume) OVER ( - PARTITION BY da.token - ORDER BY da.date - ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW - )::NUMERIC AS cumulative_target_volume, - SUM(da.base_volume) OVER ( - PARTITION BY da.token - ORDER BY da.date - ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW - )::NUMERIC AS cumulative_token_volume - FROM daily_agg da - ORDER BY da.token, da.date; - END; - $$ LANGUAGE plpgsql; - - -- Function to calculate rolling 24h metrics from 10-minute data - CREATE OR REPLACE FUNCTION calculate_rolling_24h( - p_token VARCHAR DEFAULT NULL - ) - RETURNS TABLE ( - token VARCHAR, - base_volume_24h NUMERIC, - target_volume_24h NUMERIC, - buy_volume_24h NUMERIC, - sell_volume_24h NUMERIC, - high_24h NUMERIC, - low_24h NUMERIC, - average_price_24h NUMERIC, - trade_count_24h INT, - usdc_fees_24h NUMERIC, - token_fees_24h NUMERIC, - token_fees_usdc_24h NUMERIC, - sell_volume_usdc_24h NUMERIC - ) AS $$ - BEGIN - RETURN QUERY - SELECT - tmv.token, - SUM(tmv.base_volume)::NUMERIC AS base_volume_24h, - SUM(tmv.target_volume)::NUMERIC AS target_volume_24h, - SUM(tmv.buy_volume)::NUMERIC AS buy_volume_24h, - SUM(tmv.sell_volume)::NUMERIC AS sell_volume_24h, - MAX(tmv.high)::NUMERIC AS high_24h, - MIN(CASE WHEN tmv.low > 0 THEN tmv.low END)::NUMERIC AS low_24h, - -- Weighted average price by volume - CASE - WHEN SUM(tmv.base_volume) > 0 - THEN SUM(tmv.average_price * tmv.base_volume) / SUM(tmv.base_volume) - ELSE AVG(tmv.average_price) - END::NUMERIC AS average_price_24h, - SUM(tmv.trade_count)::INT AS trade_count_24h, - SUM(tmv.usdc_fees)::NUMERIC AS usdc_fees_24h, - SUM(tmv.token_fees)::NUMERIC AS token_fees_24h, - SUM(tmv.token_fees_usdc)::NUMERIC AS token_fees_usdc_24h, - SUM(tmv.sell_volume_usdc)::NUMERIC AS sell_volume_usdc_24h - FROM ten_minute_volumes tmv - WHERE - tmv.bucket >= (CURRENT_TIMESTAMP - INTERVAL '24 hours') - AND (p_token IS NULL OR tmv.token = p_token) - GROUP BY tmv.token - ORDER BY tmv.token; - END; - $$ LANGUAGE plpgsql; - `; - - await pool.query(functionsSQL); - logger.info('[Database] Aggregation functions created/updated'); - } catch (error: any) { - logger.error('[Database] Error creating aggregation functions:', error); - throw error; - } - }, }; return manager; diff --git a/src/services/databaseService.ts b/src/services/databaseService.ts index be92409..0aaa3a6 100644 --- a/src/services/databaseService.ts +++ b/src/services/databaseService.ts @@ -5,9 +5,6 @@ import { sendAlert } from '../utils/alerts.js'; import type { DbRuntime } from './database/internal/dbRuntime.js'; import { createSchemaManager } from './database/internal/schemaManager.js'; import { createDailyVolumesRepo } from './database/internal/repos/dailyVolumesRepo.js'; -import { createIntervalVolumesRepo } from './database/internal/repos/intervalVolumesRepo.js'; -import { createDailyBuySellVolumesRepo } from './database/internal/repos/dailyBuySellVolumesRepo.js'; -import { createDailyFeesVolumesRepo } from './database/internal/repos/dailyFeesVolumesRepo.js'; import { createMetricsRepo } from './database/internal/repos/metricsRepo.js'; import { createV06TradingActivityRepo } from './database/internal/repos/v06TradingActivityRepo.js'; @@ -17,115 +14,6 @@ pg.defaults.parseInputDatesAsUTC = true; const { Pool } = pg; -export interface DailyVolumeRecord { - token: string; - date: string; // YYYY-MM-DD - base_volume: string; - target_volume: string; - buy_volume?: string; - sell_volume?: string; - high: string; - low: string; - average_price?: string; - trade_count?: number; - usdc_fees?: string; - token_fees?: string; - token_fees_usdc?: string; - sell_volume_usdc?: string; - cumulative_usdc_fees?: string; - cumulative_token_in_usdc_fees?: string; - cumulative_target_volume?: string; - cumulative_token_volume?: string; -} - -export interface HourlyVolumeRecord { - token: string; - hour: string; // ISO timestamp (YYYY-MM-DD HH:00:00) - base_volume: string; - target_volume: string; - buy_volume?: string; - sell_volume?: string; - high: string; - low: string; - average_price?: string; - trade_count: number; - usdc_fees?: string; - token_fees?: string; - token_fees_usdc?: string; - sell_volume_usdc?: string; -} - -export interface TenMinuteVolumeRecord { - token: string; - bucket: string; // ISO timestamp (YYYY-MM-DD HH:M0:00 where M is 0,1,2,3,4,5) - base_volume: string; - target_volume: string; - buy_volume?: string; - sell_volume?: string; - high: string; - low: string; - average_price?: string; - trade_count: number; - usdc_fees?: string; - token_fees?: string; - token_fees_usdc?: string; - sell_volume_usdc?: string; -} - -export interface DailyBuySellVolumeRecord { - token: string; - date: string; // YYYY-MM-DD - base_volume: string; - target_volume: string; - buy_usdc_volume: string; - sell_token_volume: string; - high: string; - low: string; - trade_count: number; - average_price: string; - usdc_fees: string; - token_fees: string; - token_fees_usdc: string; - sell_volume_usdc: string; - sell_volume: string; - buy_volume: string; -} - -export interface DailyFeesVolumeRecord { - token: string; - trading_date: string; // YYYY-MM-DD - base_volume: string; - target_volume: string; - usdc_fees: string; - token_fees_usdc: string; - token_fees: string; - buy_volume: string; - sell_volume: string; - sell_volume_usdc: string; - cumulative_usdc_fees: string; - cumulative_token_in_usdc_fees: string; - cumulative_target_volume: string; - cumulative_token_volume: string; - high: string; - average_price: string; - low: string; -} - -export interface CumulativeVolumeData { - token: string; - date: string; - base_volume: string; - target_volume: string; - buy_usdc_volume: string; - sell_token_volume: string; - cumulative_target_volume: string; - cumulative_base_volume: string; - cumulative_buy_usdc_volume: string; - cumulative_sell_token_volume: string; - high: string; - low: string; -} - export interface Rolling24hMetrics { token: string; base_volume_24h: string; @@ -135,18 +23,6 @@ export interface Rolling24hMetrics { trade_count_24h: number; } -export interface TokenVolumeAggregate { - token: string; - first_trade_date: string; - last_trade_date: string; - total_base_volume: string; - total_target_volume: string; - all_time_high: string; - all_time_low: string; - trading_days: number; - daily_data: DailyVolumeRecord[]; -} - export interface DatabaseInitializeOptions { ensureSchema?: boolean; } @@ -166,9 +42,6 @@ export class DatabaseService { private _schema = createSchemaManager(this.dbRuntime); private _dailyVolumes = createDailyVolumesRepo(this.dbRuntime); - private _intervalVolumes = createIntervalVolumesRepo(this.dbRuntime); - private _dailyBuySellVolumes = createDailyBuySellVolumesRepo(this.dbRuntime); - private _dailyFeesVolumes = createDailyFeesVolumesRepo(this.dbRuntime); private _metrics = createMetricsRepo(this.dbRuntime); private _v06TradingActivity = createV06TradingActivityRepo(this.dbRuntime); @@ -226,7 +99,6 @@ export class DatabaseService { if (ensureSchema) { await this._schema.createTables(); - await this._schema.createAggregationFunctions(); } this.isConnected = true; @@ -325,244 +197,14 @@ export class DatabaseService { } } - // ============================================ - // Schema management (delegated to SchemaManager) - // ============================================ - - async createAggregationFunctions(): Promise { - return this._schema.createAggregationFunctions(); - } - // ============================================ // Daily volumes (delegated to DailyVolumesRepo) // ============================================ - async getLatestDate(): Promise { - return this._dailyVolumes.getLatestDate(); - } - - async getLatestDateForToken(token: string): Promise { - return this._dailyVolumes.getLatestDateForToken(token); - } - - async upsertDailyVolumes(records: DailyVolumeRecord[]): Promise { - return this._dailyVolumes.upsertDailyVolumes(records); - } - - async getDailyVolumesForToken(token: string): Promise { - return this._dailyVolumes.getDailyVolumesForToken(token); - } - - async getDailyVolumesForTokens(tokens: string[]): Promise> { - return this._dailyVolumes.getDailyVolumesForTokens(tokens); - } - - async getAggregatedVolumes(tokens?: string[]): Promise { - return this._dailyVolumes.getAggregatedVolumes(tokens); - } - - async get24hVolumes(tokens?: string[]): Promise> { - return this._dailyVolumes.get24hVolumes(tokens); - } - - async getRecordCount(): Promise { - return this._dailyVolumes.getDailyRecordCount(); - } - - async getDailyRecordCount(): Promise { - return this._dailyVolumes.getDailyRecordCount(); - } - - async getTokenCount(): Promise { - return this._dailyVolumes.getTokenCount(); - } - async getV06Rolling24hMetrics(tokens?: string[]): Promise> { return this._dailyVolumes.getV06Rolling24hMetrics(tokens); } - // ============================================ - // Interval volumes (delegated to IntervalVolumesRepo) - // ============================================ - - async setSyncMetadata(key: string, value: string): Promise { - return this._intervalVolumes.setSyncMetadata(key, value); - } - - async getSyncMetadata(key: string): Promise { - return this._intervalVolumes.getSyncMetadata(key); - } - - async getLatestHour(): Promise { - return this._intervalVolumes.getLatestHour(); - } - - async getLatestCompleteHour(): Promise { - return this._intervalVolumes.getLatestCompleteHour(); - } - - async upsertHourlyVolumes(records: HourlyVolumeRecord[], markComplete: boolean = false): Promise { - return this._intervalVolumes.upsertHourlyVolumes(records, markComplete); - } - - async markHoursComplete(beforeHour: string): Promise { - return this._intervalVolumes.markHoursComplete(beforeHour); - } - - async getRolling24hMetrics(tokens?: string[]): Promise> { - return this._intervalVolumes.getRolling24hMetrics(tokens); - } - - async getHourlyVolumes(startHour: string, endHour?: string, tokens?: string[]): Promise { - return this._intervalVolumes.getHourlyVolumes(startHour, endHour, tokens); - } - - async getHourlyRecordCount(): Promise { - return this._intervalVolumes.getHourlyRecordCount(); - } - - async getHourlyTokenCount(): Promise { - return this._intervalVolumes.getHourlyTokenCount(); - } - - async pruneOldHourlyData(keepHours: number = 48): Promise { - return this._intervalVolumes.pruneOldHourlyData(keepHours); - } - - async upsertTenMinuteVolumes(records: TenMinuteVolumeRecord[], markComplete: boolean = false): Promise { - return this._intervalVolumes.upsertTenMinuteVolumes(records, markComplete); - } - - async markTenMinuteBucketsComplete(beforeBucket: string): Promise { - return this._intervalVolumes.markTenMinuteBucketsComplete(beforeBucket); - } - - async backfillMissingFields(): Promise<{ - tenMinuteUpdated: number; - hourlyUpdated: number; - dailyUpdated: number; - }> { - return this._intervalVolumes.backfillMissingFields(); - } - - async getRolling24hFromTenMinute(tokens?: string[]): Promise> { - return this._intervalVolumes.getRolling24hFromTenMinute(tokens); - } - - async aggregate10MinToHourly(token?: string, hour?: string): Promise { - return this._intervalVolumes.aggregate10MinToHourly(token, hour); - } - - async aggregateHourlyToDaily(token?: string, date?: string): Promise { - return this._intervalVolumes.aggregateHourlyToDaily(token, date); - } - - async getLatestTenMinuteBucket(): Promise { - return this._intervalVolumes.getLatestTenMinuteBucket(); - } - - async getTenMinuteRecordCount(): Promise { - return this._intervalVolumes.getTenMinuteRecordCount(); - } - - async pruneOldTenMinuteData(keepHours: number = 25): Promise { - return this._intervalVolumes.pruneOldTenMinuteData(keepHours); - } - - // ============================================ - // Daily buy/sell volumes (delegated to DailyBuySellVolumesRepo) - // ============================================ - - async getLatestBuySellDate(): Promise { - return this._dailyBuySellVolumes.getLatestBuySellDate(); - } - - async upsertDailyBuySellVolumes(records: DailyBuySellVolumeRecord[], markComplete: boolean = false): Promise { - return this._dailyBuySellVolumes.upsertDailyBuySellVolumes(records, markComplete); - } - - async getDailyBuySellVolumesWithCumulative(token?: string): Promise { - return this._dailyBuySellVolumes.getDailyBuySellVolumesWithCumulative(token); - } - - async getDailyBuySellVolumes(options?: { - token?: string; - tokens?: string[]; - startDate?: string; - endDate?: string; - }): Promise<{ - token: string; - date: string; - base_volume: string; - target_volume: string; - buy_usdc_volume: string; - sell_token_volume: string; - high: string; - low: string; - trade_count: number; - average_price: string; - usdc_fees: string; - token_fees: string; - token_fees_usdc: string; - sell_volume_usdc: string; - sell_volume: string; - buy_volume: string; - }[]> { - return this._dailyBuySellVolumes.getDailyBuySellVolumes(options); - } - - async getBuySellAggregates(tokens?: string[]): Promise> { - return this._dailyBuySellVolumes.getBuySellAggregates(tokens); - } - - async getFirstTradeDates(): Promise> { - return this._dailyBuySellVolumes.getFirstTradeDates(); - } - - async getBuySellRecordCount(): Promise { - return this._dailyBuySellVolumes.getBuySellRecordCount(); - } - - async markBuySellDaysComplete(beforeDate: string): Promise { - return this._dailyBuySellVolumes.markBuySellDaysComplete(beforeDate); - } - - // ============================================ - // Daily fees volumes (delegated to DailyFeesVolumesRepo) - // ============================================ - - async getLatestFeesDate(): Promise { - return this._dailyFeesVolumes.getLatestFeesDate(); - } - - async upsertDailyFeesVolumes(records: DailyFeesVolumeRecord[], markComplete: boolean = false): Promise { - return this._dailyFeesVolumes.upsertDailyFeesVolumes(records, markComplete); - } - - async getDailyFeesVolumes(options?: { - token?: string; - startDate?: string; - endDate?: string; - }): Promise { - return this._dailyFeesVolumes.getDailyFeesVolumes(options); - } - - async markFeesDaysComplete(beforeDate: string): Promise { - return this._dailyFeesVolumes.markFeesDaysComplete(beforeDate); - } - - async getFeesRecordCount(): Promise { - return this._dailyFeesVolumes.getFeesRecordCount(); - } - // ============================================ // Metrics (delegated to MetricsRepo) // ============================================ diff --git a/src/services/externalDatabaseService.ts b/src/services/externalDatabaseService.ts index 42584da..1613570 100644 --- a/src/services/externalDatabaseService.ts +++ b/src/services/externalDatabaseService.ts @@ -189,6 +189,32 @@ export class ExternalDatabaseService { } } + /** + * First spot-trade date per token (base mint), from the v0.6 indexer's per-swap + * v0_6_spot_swaps in the served DB (replaces the frozen Dune buy/sell volumes table). + * Returns Map. Empty map if the connection is down. + */ + async getFirstTradeDates(): Promise> { + if (!this.pool || !this.isConnected) { + return new Map(); + } + try { + const result = await this.pool.query( + `SELECT d.base_mint_acct AS token, + MIN((to_timestamp(s.unix_timestamp) AT TIME ZONE 'UTC')::date)::text AS first_date + FROM v0_6_spot_swaps s + JOIN v0_6_daos d ON d.dao_addr = s.dao_addr + GROUP BY d.base_mint_acct` + ); + const m = new Map(); + for (const row of result.rows) m.set(row.token, row.first_date); + return m; + } catch (error: any) { + logger.error('[ExternalDB] Error getting first trade dates from v0_6_spot_swaps:', error); + return new Map(); + } + } + async close(): Promise { if (this.healthCheckInterval) { clearInterval(this.healthCheckInterval); diff --git a/tests/api.test.ts b/tests/api.test.ts index 25a06de..d368b71 100644 --- a/tests/api.test.ts +++ b/tests/api.test.ts @@ -57,16 +57,17 @@ const mockPriceService = { const mockDatabaseService = { isAvailable: jest.fn().mockReturnValue(true), - getFirstTradeDates: jest.fn().mockResolvedValue(new Map()), // v0.6 indexer fallback for /api/tickers 24h volume getV06Rolling24hMetrics: jest.fn().mockResolvedValue(new Map()), } as unknown as DatabaseService; // Primary /api/tickers source: rolling-24h spot metrics read straight from the -// indexer DB (futarchy.trades), keyed by dao_addr. +// indexer DB (futarchy.trades), keyed by dao_addr. First-trade dates (startDate) +// also come from the served (external) DB now. const mockExternalDatabaseService = { isAvailable: jest.fn().mockReturnValue(true), getSpotRolling24hMetrics: jest.fn().mockResolvedValue(new Map()), + getFirstTradeDates: jest.fn().mockResolvedValue(new Map()), } as unknown as ExternalDatabaseService; function createMockServices(): Services { @@ -151,6 +152,26 @@ describe('CoinGecko API', () => { expect(response.body[0].low_24h).toBe('0.04'); }); + it('should set startDate from the served DB first-trade dates (keyed by base mint)', async () => { + (mockExternalDatabaseService as any).getFirstTradeDates.mockResolvedValueOnce( + new Map([[mockBaseMint.toString(), '2024-03-07']]) + ); + + const response = await request(app).get('/api/tickers'); + + expect(response.status).toBe(200); + expect(response.body[0].startDate).toBe('2024-03-07'); + expect((mockExternalDatabaseService as any).getFirstTradeDates).toHaveBeenCalled(); + }); + + it('should omit startDate when no first-trade date exists for the base mint', async () => { + // Default mock returns an empty Map. + const response = await request(app).get('/api/tickers'); + + expect(response.status).toBe(200); + expect(response.body[0]).not.toHaveProperty('startDate'); + }); + it('should fall back to v0.6 indexer metrics when futarchy.trades is empty', async () => { (mockDatabaseService as any).getV06Rolling24hMetrics.mockResolvedValueOnce(new Map([ [mockBaseMint.toString(), { diff --git a/tests/helpers/testApp.ts b/tests/helpers/testApp.ts index 5b796dd..1cc6529 100644 --- a/tests/helpers/testApp.ts +++ b/tests/helpers/testApp.ts @@ -3,20 +3,16 @@ import { createApp, type Services } from '../../src/app.js'; import type { FutarchyService } from '../../src/services/futarchyService.js'; import type { PriceService } from '../../src/services/priceService.js'; import type { DatabaseService } from '../../src/services/databaseService.js'; +import type { ExternalDatabaseService } from '../../src/services/externalDatabaseService.js'; import type { SolanaService } from '../../src/services/solanaService.js'; import type { LaunchpadService } from '../../src/services/launchpadService.js'; export function createMockDatabaseService(): DatabaseService { return { isAvailable: () => true, - getFirstTradeDates: async () => new Map(), + getV06Rolling24hMetrics: async () => new Map(), getServiceHealthHistory: async () => [], - getHourlyRecordCount: async () => 0, - getTenMinuteRecordCount: async () => 0, - getDailyRecordCount: async () => 0, - getBuySellRecordCount: async () => 0, - getRolling24hFromTenMinute: async () => new Map(), - getRolling24hMetrics: async () => new Map(), + getRecentMetrics: async () => [], insertServiceHealthSnapshot: async () => {}, insertMetricsBatch: async () => {}, pruneOldMetrics: async () => {}, @@ -24,6 +20,16 @@ export function createMockDatabaseService(): DatabaseService { } as unknown as DatabaseService; } +export function createMockExternalDatabaseService(): ExternalDatabaseService { + return { + isAvailable: () => true, + getSpotRolling24hMetrics: async () => new Map(), + getDailyMeteoraVolumes: async () => [], + getFirstTradeDates: async () => new Map(), + close: async () => {}, + } as unknown as ExternalDatabaseService; +} + export function createMockFutarchyService(): FutarchyService { return { getAllDaos: async () => [], @@ -66,6 +72,7 @@ export function createTestServices(overrides?: Partial): Services { futarchyService: createMockFutarchyService(), priceService: createMockPriceService(), databaseService: createMockDatabaseService(), + externalDatabaseService: createMockExternalDatabaseService(), solanaService: createMockSolanaService(), launchpadService: createMockLaunchpadService(), v06ReconciliationService: null, diff --git a/tests/routes/metrics.test.ts b/tests/routes/metrics.test.ts index 86ba612..af5aad8 100644 --- a/tests/routes/metrics.test.ts +++ b/tests/routes/metrics.test.ts @@ -79,7 +79,7 @@ describe('Metrics Routes', () => { it('should accept valid labels JSON', async () => { const response = await request(app) .get('/api/metrics/history/test_metric') - .query({ labels: '{"table":"daily_volumes"}' }); + .query({ labels: '{"table":"v06_spot_ohlcv_1m"}' }); // 200 if database connected, 503 if not, 500 if mock incomplete expect([200, 500, 503]).toContain(response.status); From 1add0695b6d966e545f687a5c8392428055db399 Mon Sep 17 00:00:00 2001 From: Jinglun Date: Fri, 5 Jun 2026 13:39:04 -0700 Subject: [PATCH 04/14] =?UTF-8?q?fix(external-api):=20address=20review=20?= =?UTF-8?q?=E2=80=94=20surface=20served-DB=20failures,=20health=20checks?= =?UTF-8?q?=20external=20DB,=20remove=20dead=20Dune=20scripts,=20docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit H5: removed broken/obsolete Dune backfill scripts that imported deleted services (backfill.ts, backfillExtendedFields.ts, backfillMeteoraVolumes.ts, compareDuneVsV06.ts) + their package.json entries; cleaned repo-guard.ts + CODEOWNERS references. H2: /api/market-data no longer masks served-DB failures as empty meteora — getDailyMeteoraVolumes now re-throws query/schema errors (→ 500) and the route returns 503 if the served DB is unavailable, instead of 200 with zero volume (financial-data correctness). H4: /api/health now reports externalDatabase connectivity and degrades when the served indexer DB (the source for Meteora/tickers/DexScreener/first-trade-dates) is down. M1: README — removed stale USE_DUNE_DATA / Dune env + pipeline docs; documented the served indexer DB as a required dependency and the Dune-free data sources; dropped deleted-file references from the architecture tree. Deferred (noted, pre-existing): H1 per-DAO ticker fallback, H3 indexer startup gating (superseded by the planned full indexer removal), M2 extra regression tests. L1 (route tests not validating) not reproduced — bun test 113 pass / 253 assertions incl. tests/routes/*. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/CODEOWNERS | 1 - README.md | 55 +--- package.json | 5 - scripts/backfill.ts | 160 ---------- scripts/backfillExtendedFields.ts | 284 ------------------ scripts/backfillMeteoraVolumes.ts | 306 ------------------- scripts/compareDuneVsV06.ts | 382 ------------------------ scripts/repo-guard.ts | 1 - src/routes/health.ts | 16 +- src/routes/market.ts | 14 +- src/services/externalDatabaseService.ts | 6 +- 11 files changed, 40 insertions(+), 1190 deletions(-) delete mode 100644 scripts/backfill.ts delete mode 100644 scripts/backfillExtendedFields.ts delete mode 100644 scripts/backfillMeteoraVolumes.ts delete mode 100644 scripts/compareDuneVsV06.ts diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 8640359..4231b5e 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -13,5 +13,4 @@ # Operational scripts /scripts/safe-update.ts @metanallok -/scripts/backfill.ts @metanallok /scripts/backfillV06.ts @metanallok diff --git a/README.md b/README.md index c7fb738..c7a7a7c 100644 --- a/README.md +++ b/README.md @@ -153,11 +153,9 @@ Returns swap events in the given Solana slot range (both inclusive). Events are Returns daily market data for the given date range, split by Futarchy AMM and Meteora sources. -**Data source** is controlled by the `USE_DUNE_DATA` environment variable: -- `USE_DUNE_DATA=false` — reads from `v06_fee_volume_daily_aggregate` (v0.6 indexer), providing spot + conditional volume breakdown with fee calculations and reconciliation status -- `USE_DUNE_DATA=true` (default) — reads from `daily_volumes` (Dune pipeline) - -The response includes a `source` field (`"v06-indexer"` or `"dune"`) indicating which pipeline served the data. +**Data sources** (Dune fully removed — all reads come from our own indexed/ETL data): +- **FutarchyAMM** — `v06_fee_volume_daily_aggregate` (v0.6 indexer): spot + conditional volume breakdown with fee calculations and reconciliation status. Response `source` is always `"v06-indexer"`. +- **Meteora** — read directly from the meteora accounting ETL's `futarchy.meteora_daily` view in the served indexer DB (via `externalDatabase`). The served DB is a hard dependency: if it's unreachable the endpoint returns `503` rather than reporting a DB outage as zero volume. Response `meteora.source` is `"etl-meteora-daily"`. --- @@ -228,15 +226,10 @@ Create a `.env` file in the root directory (see `example.env` for reference): | **Database (App DB)** | | | | `COINGECKO_PG_URL` / `DATABASE_URL` | PostgreSQL connection string | — | | `DATABASE_SSL` | Enable SSL | `false` | -| **External DB (v0.6 Indexer)** | | | -| `EXTERNAL_DATABASE_URL` | Read-only connection to indexer DB | — | +| **Served indexer DB (required)** | | | +| `FRONTEND_READER_PG_URL` / `EXTERNAL_DATABASE_URL` | Read-only connection to the served indexer DB (Meteora, tickers, DexScreener, first-trade-dates). **Required** — `/api/market-data` returns 503 without it. | — | | `EXTERNAL_DATABASE_SSL` | Enable SSL | `false` | -| **Dune** | | | -| `DUNE_API_KEY` | Dune Analytics API key | — | -| `DUNE_TEN_MINUTE_VOLUME_QUERY_ID` | 10-minute volume query ID | — | -| `DUNE_METEORA_VOLUME_QUERY_ID` | Meteora volume query ID | — | | **Protocol** | | | -| `USE_DUNE_DATA` | Use Dune pipeline for volume data (`true`/`false`) | `true` | | `PROTOCOL_FEE_RATE` | Protocol fee rate | `0.005` (0.5%) | | `EXCLUDED_DAOS` | Comma-separated DAO addresses to exclude | — | | **Alerts** | | | @@ -257,7 +250,7 @@ src/ │ ├── index.ts # Route registration │ ├── coingecko.ts # GET /api/tickers │ ├── dexscreener.ts # DexScreener adapter (4 endpoints) -│ ├── market.ts # GET /api/market-data (v0.6 or Dune via USE_DUNE_DATA) +│ ├── market.ts # GET /api/market-data (FutarchyAMM v0.6 + Meteora ETL) │ ├── supply.ts # GET /api/supply/* │ ├── health.ts # Health checks │ ├── metrics.ts # Prometheus metrics @@ -267,13 +260,7 @@ src/ │ ├── priceService.ts # Price, spread, liquidity calculations │ ├── databaseService.ts # App DB (volumes, OHLCV, fees, metrics) │ ├── externalDatabaseService.ts # Read-only indexer DB connection -│ ├── v06ReconciliationService.ts # Hourly v0.6 data reconciliation -│ ├── tenMinuteVolumeFetcherService.ts # 10-min volume from Dune -│ ├── hourlyAggregationService.ts # Hourly rollups -│ ├── dailyAggregationService.ts # Daily rollups -│ ├── meteoraVolumeFetcherService.ts # Meteora pool volumes -│ ├── duneCacheService.ts # Dune API cache layer -│ ├── duneService.ts # Dune API client +│ ├── v06ReconciliationService.ts # v0.6 data reconciliation (served DB → app-DB v0.6 aggregates) │ ├── solanaService.ts # SPL token supply queries │ ├── launchpadService.ts # Token allocation breakdown │ └── metricsService.ts # Prometheus counters/histograms @@ -284,30 +271,17 @@ src/ │ ├── errorHandler.ts # Error handling & asyncHandler │ └── requestId.ts # Request ID injection ├── utils/ # Logger, alerts, validation, scheduling -└── schema/ # Dune SQL queries, v0.6 DDL reference +└── schema/ # v0.6 DDL reference ``` ## Data Pipeline -### Volume Sources (priority order) -The API process serves data from the app DB and read-only external indexer DB. It does not start indexing, Dune fetchers, rollups, reconciliation workers, or app-DB schema setup. - -Run the indexer process separately: - -```bash -bun run dev:indexer - -# production -bun run start:indexer -``` +### Sources (Dune fully removed) +The API process serves data from the app DB (v0.6 aggregates: `v06_fee_volume_daily_aggregate`, `v06_spot_ohlcv_1m`) and the read-only served indexer DB (`futarchy.trades`, `futarchy.meteora_daily`, `v0_6_spot_swaps`). It does not start indexing, fetchers, rollups, or app-DB schema setup. -Indexer-owned jobs: +The only remaining background job is **v0.6 reconciliation** (`bun run start:indexer`): reads `v0_6_spot_swaps` + `v0_6_conditional_swaps` from the served indexer DB and writes the v0.6 OHLCV + fee-breakdown aggregates to the app DB. (This is being phased out — the goal is for the API to read all FutarchyAMM aggregates directly from the served DB, with no indexer in this repo.) -1. **10-minute volumes** — from Dune, stored in `ten_minute_volumes`, rolled up to hourly/daily -2. **v0.6 Reconciliation** — hourly job reads `v0_6_spot_swaps` + `v0_6_conditional_swaps` from the external indexer DB, writes OHLCV and fee breakdowns to app DB -3. **Dune cache health** — maintained only by the indexer runtime for compatibility with existing health/metrics views - -The API's `/api/tickers` reads 24h metrics directly from the app DB in priority order: `ten_minute_volumes` → `hourly_volumes` → optional legacy cache if a caller intentionally wires it. +`/api/tickers` reads 24h metrics primarily from the served DB's `futarchy.trades` (direct), falling back to the app-DB v0.6 OHLCV. ### DexScreener Pipeline The DexScreener adapter reads **directly from the external indexer DB** (`v0_6_spot_swaps` + `v0_6_daos`) and serves real-time swap events indexed by Solana slot. No intermediate aggregation — raw swap data mapped to the DexScreener schema. @@ -323,11 +297,6 @@ bun run backfill:v06 -- --since 2025-06-01 # Chunked for large ranges bun run backfill:v06 -- --since 2025-01-01 --chunk-days 30 - -# Legacy Dune-based backfills -bun run backfill:daily -bun run backfill:hourly -bun run backfill:ten-minute ``` ## Rate Limiting diff --git a/package.json b/package.json index f1ad985..c2f15fc 100644 --- a/package.json +++ b/package.json @@ -13,14 +13,9 @@ "start:indexer": "bun dist/indexer.js", "prestart": "bun run build", "test": "bun test", - "backfill": "bun run scripts/backfill.ts", - "backfill:daily": "bun run scripts/backfill.ts daily", - "backfill:hourly": "bun run scripts/backfill.ts hourly", - "backfill:ten-minute": "bun run scripts/backfill.ts ten-minute", "backfill:v06": "bun run scripts/backfillV06.ts", "safe-update": "bun run scripts/safe-update.ts", "safe-update:dry": "bun run scripts/safe-update.ts --dry-run", - "compare:dune-vs-v06": "bun run scripts/compareDuneVsV06.ts", "typecheck": "tsc --noEmit", "repo:guard": "bun scripts/repo-guard.ts", "deps:check": "knip --include unlisted --reporter compact" diff --git a/scripts/backfill.ts b/scripts/backfill.ts deleted file mode 100644 index 6633d37..0000000 --- a/scripts/backfill.ts +++ /dev/null @@ -1,160 +0,0 @@ -#!/usr/bin/env bun -/** - * Backfill script - Run a full data insert from Dune to PostgreSQL - * - * Usage: - * bun run scripts/backfill.ts [ten-minute] - * - * Note: Hourly and daily data are now aggregated from 10-minute data using DB functions. - * This script only supports 10-minute backfill. Hourly and daily will be automatically - * aggregated from the 10-minute data. - * - * Required environment variables: - * - DATABASE_URL: PostgreSQL connection string - * - DUNE_API_KEY: Your Dune API key - * - DUNE_TEN_MINUTE_VOLUME_QUERY_ID: For 10-minute volumes - */ - -import { config } from '../src/config'; - -// Dynamically import services to avoid circular deps -async function main() { - const args = process.argv.slice(2); - const mode = args[0] || 'all'; - - console.log('🚀 Starting backfill script...'); - console.log(`📊 Mode: ${mode}`); - console.log(''); - - // Check required env vars - if (!process.env.DATABASE_URL) { - console.error('❌ DATABASE_URL is required'); - process.exit(1); - } - - if (!process.env.DUNE_API_KEY) { - console.error('❌ DUNE_API_KEY is required'); - process.exit(1); - } - - // Import services - const { DatabaseService } = await import('../src/services/databaseService'); - const { DuneService } = await import('../src/services/duneService'); - const { FutarchyService } = await import('../src/services/futarchyService'); - - // Initialize database - const databaseService = new DatabaseService(); - await databaseService.initialize(); - - if (!databaseService.isAvailable()) { - console.error('❌ Failed to connect to database'); - process.exit(1); - } - - console.log('✅ Database connected'); - - // Initialize Dune service (uses config internally) - const duneService = new DuneService(); - - // Initialize FutarchyService for token list (uses config internally) - const futarchyService = new FutarchyService(); - - const allDaos = await futarchyService.getAllDaos(); - const tokenAddresses = allDaos.map(dao => dao.baseMint.toString()); - console.log(`📋 Found ${tokenAddresses.length} DAOs to backfill`); - console.log(''); - - try { - // Only support ten-minute mode - hourly and daily are aggregated from 10-minute data - if (mode === 'ten-minute' || mode === 'all' || !mode) { - await backfillTenMinute(databaseService, duneService, tokenAddresses); - } else if (mode === 'daily' || mode === 'hourly') { - console.log(`⚠️ ${mode} mode is deprecated. Hourly and daily data are now aggregated from 10-minute data.`); - console.log(' Please use "ten-minute" mode instead. Hourly and daily will be automatically aggregated.'); - process.exit(1); - } - - console.log(''); - console.log('✅ Backfill complete!'); - console.log(' Note: Hourly and daily records will be automatically aggregated from 10-minute data.'); - - // Print summary - console.log(''); - console.log('📊 Database summary:'); - const tenMinCount = await databaseService.getTenMinuteRecordCount(); - console.log(` 10-minute records: ${tenMinCount}`); - console.log(' (Hourly and daily records are aggregated from 10-minute data)'); - - } catch (error: any) { - console.error('❌ Backfill failed:', error.message); - process.exit(1); - } finally { - await databaseService.close(); - } -} - -async function backfillTenMinute( - databaseService: any, - duneService: any, - tokenAddresses: string[] -) { - const queryId = config.dune.tenMinuteVolumeQueryId; - if (!queryId) { - console.log('⏭️ Skipping 10-minute backfill - DUNE_TEN_MINUTE_VOLUME_QUERY_ID not set'); - return; - } - - console.log('⏱️ Starting 10-MINUTE volume backfill...'); - console.log(` Query ID: ${queryId}`); - - // Always backfill from the beginning - const startTime = new Date('2026-01-24T00:00:00Z'); // Futarchy launch date - const startTimeStr = startTime.toISOString().slice(0, 19).replace('T', ' '); - console.log(` Start time: ${startTimeStr} (always from beginning)`); - - const fetchStart = Date.now(); - - // Build token list parameter - const tokenListParam = tokenAddresses.length > 0 - ? tokenAddresses.map(t => `'${t}'`).join(',') - : "'__ALL__'"; - - // Execute query - const result = await duneService.executeQueryManually(queryId, { - start_time: startTimeStr, - token_list: tokenListParam, - }); - - if (!result || !result.rows || result.rows.length === 0) { - console.log(' No data returned from Dune'); - return; - } - - console.log(` Fetched ${result.rows.length} rows from Dune`); - - // Debug: show first row structure - if (result.rows.length > 0) { - console.log(` Sample row keys: ${Object.keys(result.rows[0]).join(', ')}`); - console.log(` Sample row:`, JSON.stringify(result.rows[0], null, 2)); - } - - // Transform and insert - Dune field names match DB field names - const records = result.rows.map((row: any) => ({ - token: row.token, - bucket: row.bucket, - base_volume: row.base_volume || '0', - target_volume: row.target_volume || '0', - trade_count: parseInt(row.trade_count || row.num_swaps) || 0, - high: row.high || '0', - low: row.low || '0', - })); - - const upserted = await databaseService.upsertTenMinuteVolumes(records); - const duration = Date.now() - fetchStart; - - console.log(` ✅ Upserted ${upserted} 10-minute records in ${duration}ms`); - // Note: Not pruning data - keeping all historical records -} - -main().catch(console.error); - diff --git a/scripts/backfillExtendedFields.ts b/scripts/backfillExtendedFields.ts deleted file mode 100644 index 9f800e5..0000000 --- a/scripts/backfillExtendedFields.ts +++ /dev/null @@ -1,284 +0,0 @@ -/** - * Backfill Script for Extended Fields - * - * This script safely backfills missing extended fields in existing database records. - * It: - * 1. Identifies 10-minute records missing extended fields - * 2. Optionally re-fetches those records from Dune using the extended query (if SKIP_DUNE not set) - * 3. Updates only the missing fields (preserves existing data) - * 4. Aggregates hourly and daily records from the updated 10-minute data - * - * Run this BEFORE starting services to ensure all existing data has extended fields. - * - * Usage: - * bun run scripts/backfillExtendedFields.ts - * SKIP_DUNE=true bun run scripts/backfillExtendedFields.ts # Skip Dune, use DB aggregation only - * RECENT_DAYS=7 bun run scripts/backfillExtendedFields.ts # Only backfill last 7 days from Dune - */ - -import { DatabaseService } from '../src/services/databaseService.js'; -import { DuneService } from '../src/services/duneService.js'; -import { FutarchyService } from '../src/services/futarchyService.js'; -import { TenMinuteVolumeFetcherService } from '../src/services/tenMinuteVolumeFetcherService.js'; -import { config } from '../src/config.js'; - -async function backfillExtendedFields() { - console.log('=== Starting Extended Fields Backfill ===\n'); - - // Check environment variables for options - const skipDune = process.env.SKIP_DUNE === 'true'; - const recentDays = process.env.RECENT_DAYS ? parseInt(process.env.RECENT_DAYS) : null; - - if (skipDune) { - console.log('⚠ SKIP_DUNE=true: Will skip Dune backfill and use database aggregation only\n'); - } - if (recentDays) { - console.log(`⚠ RECENT_DAYS=${recentDays}: Will only backfill last ${recentDays} days from Dune\n`); - } - - // Initialize services - const databaseService = new DatabaseService(); - const duneService = new DuneService(); - const futarchyService = new FutarchyService(); - const tenMinuteService = new TenMinuteVolumeFetcherService(duneService, databaseService, futarchyService); - - try { - // 1. Connect to database - console.log('1. Connecting to database...'); - const dbConnected = await databaseService.initialize(); - if (!dbConnected) { - console.error('Failed to connect to database'); - process.exit(1); - } - console.log('✓ Database connected\n'); - - // 2. Check for existing 10-minute records with missing extended fields - console.log('2. Checking for records with missing extended fields...'); - if (!databaseService.pool) { - console.error('Database pool not available'); - process.exit(1); - } - - const missingFieldsCheck = await databaseService.pool.query(` - SELECT - COUNT(*) as total_records, - COUNT(CASE WHEN average_price IS NULL OR average_price = 0 THEN 1 END) as missing_average_price, - MIN(bucket) as earliest_bucket, - MAX(bucket) as latest_bucket - FROM ten_minute_volumes - `); - - const stats = missingFieldsCheck.rows[0]; - console.log(` Total 10-minute records: ${stats.total_records}`); - console.log(` Missing average_price (needs backfill): ${stats.missing_average_price}`); - console.log(` Date range: ${stats.earliest_bucket} to ${stats.latest_bucket}\n`); - - if (parseInt(stats.missing_average_price) === 0) { - console.log('✓ All 10-minute records already have extended fields\n'); - } else if (skipDune) { - console.log('3. Skipping Dune backfill (SKIP_DUNE=true)\n'); - console.log(' Will use database aggregation to fill missing fields in hourly/daily tables\n'); - } else { - // 3. Find the earliest missing average_price date - console.log('3. Finding earliest missing average_price date...'); - - const earliestMissingResult = await databaseService.pool.query(` - SELECT MIN(bucket) as earliest_missing - FROM ten_minute_volumes - WHERE (average_price IS NULL OR average_price = 0) - `); - - const earliestMissing = earliestMissingResult.rows[0].earliest_missing; - - if (!earliestMissing) { - console.log(' ✓ No missing average_price records found\n'); - } else { - // Use the earliest missing date as start, but truncate to beginning of day - let startDate = new Date(earliestMissing); - startDate.setHours(0, 0, 0, 0); - - // Apply RECENT_DAYS filter if set - if (recentDays) { - const cutoffDate = new Date(); - cutoffDate.setDate(cutoffDate.getDate() - recentDays); - cutoffDate.setHours(0, 0, 0, 0); - if (cutoffDate > startDate) { - startDate = cutoffDate; - console.log(` Using RECENT_DAYS=${recentDays} filter: starting from ${startDate.toISOString().split('T')[0]}\n`); - } else { - console.log(` Starting from earliest missing date: ${startDate.toISOString().split('T')[0]}\n`); - } - } else { - console.log(` Starting from earliest missing date: ${startDate.toISOString().split('T')[0]}\n`); - } - - // 4. Get specific buckets that need backfilling - console.log('4. Identifying specific buckets that need backfilling...'); - - const dateFilter = `AND bucket >= '${startDate.toISOString()}'`; - - const missingBucketsResult = await databaseService.pool.query(` - SELECT DISTINCT - date_trunc('day', bucket)::DATE as date, - COUNT(*) as missing_count - FROM ten_minute_volumes - WHERE (average_price IS NULL OR average_price = 0) - ${dateFilter} - GROUP BY date_trunc('day', bucket)::DATE - ORDER BY date_trunc('day', bucket)::DATE ASC - `); - - const missingDays = missingBucketsResult.rows; - console.log(` Found ${missingDays.length} days with missing extended fields\n`); - - if (missingDays.length === 0) { - console.log('✓ No days need backfilling\n'); - } else { - // Show summary - const totalMissing = missingDays.reduce((sum, day) => sum + parseInt(day.missing_count), 0); - console.log(` Total buckets needing backfill: ${totalMissing}`); - if (missingDays.length > 0) { - console.log(` Date range: ${missingDays[0].date} to ${missingDays[missingDays.length - 1].date}\n`); - } - - // Warn if too many days - if (missingDays.length > 30 && !recentDays) { - console.log(`⚠ WARNING: ${missingDays.length} days need backfilling. This may use significant Dune datapoints.`); - console.log(` Options:`); - console.log(` - Set RECENT_DAYS=30 to only backfill last 30 days`); - console.log(` - Set SKIP_DUNE=true to skip Dune and use DB aggregation only`); - console.log(` - Check your Dune subscription limits\n`); - } - - // Backfill in 7-day increments (1008 buckets per chunk = 7 days * 24 hours * 6 buckets/hour) - console.log('5. Backfilling 10-minute records from Dune (7-day chunks)...'); - console.log(` Processing ${missingDays.length} days in 7-day increments (1008 buckets per chunk)...\n`); - - let totalBackfilled = 0; - let totalUpdated = 0; - let errors = 0; - let apiLimitHit = false; - - // Group days into 7-day chunks - const chunkSize = 7; - const chunks: Array<{ startDate: Date; endDate: Date; days: any[] }> = []; - - for (let i = 0; i < missingDays.length; i += chunkSize) { - const chunkDays = missingDays.slice(i, i + chunkSize); - const startDate = new Date(chunkDays[0].date); - startDate.setHours(0, 0, 0, 0); - - const endDate = new Date(chunkDays[chunkDays.length - 1].date); - endDate.setHours(23, 59, 59, 999); - endDate.setDate(endDate.getDate() + 1); // Exclusive end (start of next day) - - chunks.push({ startDate, endDate, days: chunkDays }); - } - - // Process chunks in chronological order (oldest first) - for (let chunkIdx = 0; chunkIdx < chunks.length; chunkIdx++) { - const chunk = chunks[chunkIdx]; - const startTime = chunk.startDate.toISOString().replace('T', ' ').replace('Z', '').slice(0, 19); - const endTime = chunk.endDate.toISOString().replace('T', ' ').replace('Z', '').slice(0, 19); - - const totalMissingInChunk = chunk.days.reduce((sum, day) => sum + parseInt(day.missing_count), 0); - const dayRange = `${chunk.days[0].date} to ${chunk.days[chunk.days.length - 1].date}`; - - console.log(` [${chunkIdx + 1}/${chunks.length}] ${dayRange} (${totalMissingInChunk} missing buckets across ${chunk.days.length} days)...`); - - try { - // Fetch 7-day chunk from Dune with explicit start_time and end_time (will only update missing fields) - const count = await tenMinuteService.backfillExtendedFields(startTime, endTime); - if (count > 0) { - totalUpdated += count; - console.log(` ✓ Updated ${count} records`); - } else { - console.log(` - Skipped (no missing fields)`); - } - - totalBackfilled++; - - // Add delay between chunks to avoid rate limits (3 seconds between 7-day chunks) - if (chunkIdx < chunks.length - 1) { - await new Promise(resolve => setTimeout(resolve, 3000)); - } - } catch (error: any) { - errors++; - if (error.message.includes('402') || error.message.includes('Payment Required')) { - apiLimitHit = true; - console.error(`\n ✗ Dune API limit reached!`); - console.error(`\n Processed ${totalBackfilled} chunks (${totalBackfilled * 7} days), updated ${totalUpdated} records before limit.`); - console.error(`\n Options to continue:`); - console.error(` 1. Wait for your Dune billing cycle to reset`); - console.error(` 2. Upgrade your Dune subscription`); - console.error(` 3. Run with SKIP_DUNE=true to use database aggregation only:`); - console.error(` SKIP_DUNE=true bun run scripts/backfillExtendedFields.ts`); - console.error(` 4. Resume later - script will skip already-filled records`); - console.error(`\n Note: New data will automatically include extended fields going forward.\n`); - break; - } else { - console.error(` ✗ Error: ${error.message}`); - // Continue with next chunk on non-limit errors - } - } - - if (apiLimitHit) break; - } - - if (!apiLimitHit) { - console.log(`\n✓ Backfilled ${totalBackfilled} chunks from Dune`); - console.log(` Total records updated: ${totalUpdated}`); - if (errors > 0) { - console.log(` Errors encountered: ${errors}`); - } - console.log(''); - } else { - console.log('\n⚠ Backfill incomplete due to API limits. See options above.\n'); - } - } - } - } - - // 6. Run database aggregation backfill (always run this - it's free, no Dune calls) - console.log('6. Running database aggregation backfill...'); - console.log(' (This aggregates from existing 10-minute data - no Dune API calls)\n'); - const backfillResults = await databaseService.backfillMissingFields(); - console.log(` Updated hourly records: ${backfillResults.hourlyUpdated}`); - console.log(` Updated daily records: ${backfillResults.dailyUpdated}\n`); - - if (backfillResults.hourlyUpdated > 0 || backfillResults.dailyUpdated > 0) { - console.log(' ✓ Database aggregation completed - hourly/daily records now have extended fields\n'); - } else { - console.log(' ✓ No hourly/daily records needed updating\n'); - } - - // 7. Verify final state - console.log('7. Verifying final state...'); - const finalCheck = await databaseService.pool.query(` - SELECT - COUNT(*) as total_records, - COUNT(CASE WHEN average_price IS NULL OR average_price = 0 THEN 1 END) as missing_average_price - FROM ten_minute_volumes - `); - - const finalStats = finalCheck.rows[0]; - console.log(` Total 10-minute records: ${finalStats.total_records}`); - console.log(` Still missing average_price: ${finalStats.missing_average_price}\n`); - - if (parseInt(finalStats.missing_average_price) === 0) { - console.log('✓ All records now have extended fields!\n'); - } else { - console.log('⚠ Some records still missing fields (may need additional backfill)\n'); - } - - console.log('=== Backfill Completed ==='); - process.exit(0); - } catch (error: any) { - console.error('Error during backfill:', error); - console.error(error.stack); - process.exit(1); - } -} - -// Run the backfill -backfillExtendedFields(); diff --git a/scripts/backfillMeteoraVolumes.ts b/scripts/backfillMeteoraVolumes.ts deleted file mode 100644 index a7de7a1..0000000 --- a/scripts/backfillMeteoraVolumes.ts +++ /dev/null @@ -1,306 +0,0 @@ -/** - * Backfill Script for Meteora Daily Volumes - * - * This script backfills historical Meteora pool fee data from Dune. - * It fetches data from 2025-10-09 to present and stores it in the database. - * Processes data in chunks to avoid API limits. - * - * Usage: - * bun run scripts/backfillMeteoraVolumes.ts - * RECENT_DAYS=7 bun run scripts/backfillMeteoraVolumes.ts # Only backfill last 7 days - * CHUNK_DAYS=30 bun run scripts/backfillMeteoraVolumes.ts # Use 30-day chunks (default: 30) - * - * Chunk skipping (METEORA_BACKFILL_SKIP_CHUNKS): - * coverage (default) — skip a chunk only if DB already has rows for every mapped token in that date range (adds missing tokens after you extend the map + Dune query). - * any — skip whenever any row exists in the chunk (legacy; misses new tokens in already-filled ranges). - * never — always call Dune (idempotent upserts; highest Dune usage). - */ - -import { DatabaseService } from '../src/services/databaseService.js'; -import { DuneService } from '../src/services/duneService.js'; -import { MeteoraVolumeFetcherService } from '../src/services/meteoraVolumeFetcherService.js'; -import { getAllMappedTokens } from '../src/services/meteoraService.js'; -import { config } from '../src/config.js'; - -/** Chunk skip behavior: avoid re-querying Dune when data already exists. */ -type ChunkSkipMode = 'coverage' | 'any' | 'never'; - -function parseChunkSkipMode(raw: string | undefined): ChunkSkipMode { - if (raw === 'any' || raw === 'never' || raw === 'coverage') { - return raw; - } - return 'coverage'; -} - -/** - * Helper to add days to a date - */ -function addDays(date: Date, days: number): Date { - const result = new Date(date); - result.setDate(result.getDate() + days); - return result; -} - -/** - * Helper to format date as YYYY-MM-DD - */ -function formatDate(date: Date): string { - return date.toISOString().split('T')[0]!; -} - -/** - * Generate date chunks for backfilling - */ -function generateDateChunks(startDate: Date, endDate: Date, chunkDays: number): Array<{ start: Date; end: Date }> { - const chunks: Array<{ start: Date; end: Date }> = []; - let currentStart = new Date(startDate); - - while (currentStart < endDate) { - const currentEnd = new Date(currentStart); - currentEnd.setDate(currentEnd.getDate() + chunkDays - 1); - - // Don't go past the end date - if (currentEnd > endDate) { - currentEnd.setTime(endDate.getTime()); - } - - chunks.push({ - start: new Date(currentStart), - end: new Date(currentEnd), - }); - - // Move to next chunk - currentStart = addDays(currentEnd, 1); - } - - return chunks; -} - -async function backfillMeteoraVolumes() { - console.log('=== Starting Meteora Volumes Backfill ===\n'); - - // Check environment variables for options - const recentDays = process.env.RECENT_DAYS ? parseInt(process.env.RECENT_DAYS) : null; - const chunkDays = process.env.CHUNK_DAYS ? parseInt(process.env.CHUNK_DAYS) : 30; // Default: 30-day chunks - const chunkSkipMode = parseChunkSkipMode(process.env.METEORA_BACKFILL_SKIP_CHUNKS); - const expectedTokens = getAllMappedTokens(); - - if (recentDays) { - console.log(`⚠ RECENT_DAYS=${recentDays}: Will only backfill last ${recentDays} days\n`); - } - console.log(`📦 Using ${chunkDays}-day chunks to avoid API limits`); - console.log(` Mapped Meteora tokens: ${expectedTokens.length} (${chunkSkipMode} chunk skip)\n`); - - // Initialize services - const databaseService = new DatabaseService(); - const duneService = new DuneService(); - const meteoraService = new MeteoraVolumeFetcherService(duneService, databaseService); - - try { - // 1. Connect to database - console.log('1. Connecting to database...'); - const dbConnected = await databaseService.initialize(); - if (!dbConnected) { - console.error('Failed to connect to database'); - process.exit(1); - } - console.log('✓ Database connected\n'); - - // 2. Check for existing records - console.log('2. Checking for existing records...'); - if (!databaseService.pool) { - console.error('Database pool not available'); - process.exit(1); - } - - const existingCheck = await databaseService.pool.query(` - SELECT - COUNT(*) as total_records, - MIN(date) as earliest_date, - MAX(date) as latest_date - FROM daily_meteora_volumes - `); - - const stats = existingCheck.rows[0]; - console.log(` Total records: ${stats.total_records}`); - console.log(` Date range: ${stats.earliest_date || 'N/A'} to ${stats.latest_date || 'N/A'}\n`); - - // 3. Determine date range - const launchDate = new Date('2025-10-09'); - const today = new Date(); - today.setHours(0, 0, 0, 0); - - let actualStartDate = new Date(launchDate); - let actualEndDate = new Date(today); - - if (recentDays) { - const cutoffDate = new Date(); - cutoffDate.setDate(cutoffDate.getDate() - recentDays); - cutoffDate.setHours(0, 0, 0, 0); - if (cutoffDate > launchDate) { - actualStartDate = cutoffDate; - console.log(` Limiting to last ${recentDays} days (since ${formatDate(actualStartDate)})\n`); - } else { - console.log(` Starting from ${formatDate(launchDate)} (launch date)\n`); - } - } else { - console.log(` Date range: ${formatDate(actualStartDate)} to ${formatDate(actualEndDate)}\n`); - } - - // 4. Check if query ID is configured - if (!config.dune.meteoraVolumeQueryId) { - console.error('❌ DUNE_METEORA_VOLUME_QUERY_ID not configured'); - console.error(' Please set DUNE_METEORA_VOLUME_QUERY_ID in your .env file'); - process.exit(1); - } - - console.log(`3. Using Dune query ID: ${config.dune.meteoraVolumeQueryId}\n`); - - // 5. Skip service initialization to avoid automatic backfill - // The backfill script handles chunked fetching manually - console.log('4. Ready to process chunks (skipping service auto-initialization)\n'); - - // 6. Generate date chunks - console.log('5. Generating date chunks...'); - const chunks = generateDateChunks(actualStartDate, actualEndDate, chunkDays); - console.log(` Generated ${chunks.length} chunks of ${chunkDays} days each\n`); - - // 7. Process chunks - console.log('6. Processing chunks from Dune...\n'); - let totalRecords = 0; - let processedChunks = 0; - let errors = 0; - let apiLimitHit = false; - - for (let chunkIdx = 0; chunkIdx < chunks.length; chunkIdx++) { - const chunk = chunks[chunkIdx]!; - const chunkStart = formatDate(chunk.start); - const chunkEnd = formatDate(chunk.end); - - console.log(` [${chunkIdx + 1}/${chunks.length}] Processing ${chunkStart} to ${chunkEnd}...`); - - try { - if (chunkSkipMode !== 'never') { - if (chunkSkipMode === 'any') { - const existingCheck = await databaseService.pool!.query( - ` - SELECT COUNT(*)::int AS count - FROM daily_meteora_volumes - WHERE date >= $1 AND date <= $2 - `, - [chunkStart, chunkEnd], - ); - - const existingCount = Number(existingCheck.rows[0]!.count); - if (existingCount > 0) { - console.log(` ⏭️ Skipping (${existingCount} records already exist, skip mode=any)`); - processedChunks++; - continue; - } - } else { - const coverageCheck = await databaseService.pool!.query( - ` - SELECT COUNT(*)::int AS distinct_tokens - FROM ( - SELECT DISTINCT token - FROM daily_meteora_volumes - WHERE date >= $1 AND date <= $2 - AND token = ANY($3::varchar[]) - ) t - `, - [chunkStart, chunkEnd, expectedTokens], - ); - - const distinctInChunk = Number(coverageCheck.rows[0]!.distinct_tokens); - if (distinctInChunk >= expectedTokens.length) { - console.log( - ` ⏭️ Skipping (${distinctInChunk}/${expectedTokens.length} mapped tokens present in range, skip mode=coverage)`, - ); - processedChunks++; - continue; - } - if (distinctInChunk > 0) { - console.log(` Refetch (${distinctInChunk}/${expectedTokens.length} mapped tokens in range — filling gaps)`); - } - } - } - - // Fetch chunk from Dune - const recordsUpserted = await meteoraService.fetchDateRange(chunkStart, chunkEnd); - - if (recordsUpserted > 0) { - totalRecords += recordsUpserted; - console.log(` ✓ Fetched and stored ${recordsUpserted} records`); - } else { - console.log(` - No new records for this chunk`); - } - - processedChunks++; - - // Add delay between chunks to avoid rate limits (3 seconds between chunks) - if (chunkIdx < chunks.length - 1) { - await new Promise(resolve => setTimeout(resolve, 3000)); - } - } catch (error: any) { - errors++; - if (error.message?.includes('402') || error.message?.includes('Payment Required')) { - apiLimitHit = true; - console.error(`\n ✗ Dune API limit reached!`); - console.error(`\n Processed ${processedChunks} chunks, fetched ${totalRecords} records before limit.`); - console.error(`\n Options to continue:`); - console.error(` 1. Wait for your Dune billing cycle to reset`); - console.error(` 2. Upgrade your Dune subscription`); - console.error(` 3. Resume later - with METEORA_BACKFILL_SKIP_CHUNKS=coverage (default), chunks skip only when every mapped token exists in that date range`); - console.error(`\n Run the script again to resume; use METEORA_BACKFILL_SKIP_CHUNKS=never to force re-querying all chunks.\n`); - break; - } else { - console.error(` ✗ Error: ${error.message}`); - // Continue with next chunk on non-limit errors - } - } - - if (apiLimitHit) break; - } - - if (!apiLimitHit) { - console.log(`\n✓ Backfill completed successfully`); - console.log(` Processed ${processedChunks}/${chunks.length} chunks`); - console.log(` Total records fetched: ${totalRecords}`); - } else { - console.log(`\n⚠ Backfill partially completed`); - console.log(` Processed ${processedChunks}/${chunks.length} chunks`); - console.log(` Total records fetched: ${totalRecords}`); - console.log(` Run the script again to continue from where it left off.`); - } - - if (errors > 0 && !apiLimitHit) { - console.log(`\n⚠ Completed with ${errors} error(s) (non-fatal)`); - } - - // 8. Verify final state - console.log('\n7. Verifying final state...'); - const finalCheck = await databaseService.pool.query(` - SELECT - COUNT(*) as total_records, - MIN(date) as earliest_date, - MAX(date) as latest_date, - COUNT(DISTINCT token) as unique_tokens - FROM daily_meteora_volumes - `); - - const finalStats = finalCheck.rows[0]; - console.log(` Total records: ${finalStats.total_records}`); - console.log(` Date range: ${finalStats.earliest_date || 'N/A'} to ${finalStats.latest_date || 'N/A'}`); - console.log(` Unique tokens: ${finalStats.unique_tokens}\n`); - - console.log('=== Backfill Completed ==='); - process.exit(0); - } catch (error: any) { - console.error('Error during backfill:', error); - console.error(error.stack); - process.exit(1); - } -} - -// Run the backfill -backfillMeteoraVolumes(); diff --git a/scripts/compareDuneVsV06.ts b/scripts/compareDuneVsV06.ts deleted file mode 100644 index b9a5fea..0000000 --- a/scripts/compareDuneVsV06.ts +++ /dev/null @@ -1,382 +0,0 @@ -#!/usr/bin/env bun -/** - * Compare Dune-sourced daily_volumes against v06_fee_volume_daily_aggregate - * to identify divergences per token/day. - * - * Usage: - * bun run scripts/compareDuneVsV06.ts # default last 90 days - * bun run scripts/compareDuneVsV06.ts --since 2025-01-01 - * bun run scripts/compareDuneVsV06.ts --days 30 - * bun run scripts/compareDuneVsV06.ts --days 30 --token META - * bun run scripts/compareDuneVsV06.ts --threshold 5 # flag rows with >5% delta (default: 1%) - * bun run scripts/compareDuneVsV06.ts --basis total # compare Dune to v06 total_* (spot+conditional; expect gaps) - * - * Basis: - * spot (default) — Dune daily_volumes is spot-only; compared to v06 spot_* columns (apples to apples). - * total — compared to v06 total_*; divergences are normal when conditional volume exists. - * - * Units: - * daily_volumes (Dune path) stores human-scale amounts (/ 1e6 in Dune SQL). - * v06_fee_volume_* stores raw on-chain amounts (6 decimals). This script divides - * v06 volumes and fee columns by 1e6 before comparing so the delta is meaningful. - * - * Required env: - * DATABASE_URL (or COINGECKO_PG_URL) - */ - -import 'dotenv/config'; -import { parseArgs } from 'util'; - -// ---- types ---- - -interface ComparisonRow { - token: string; - date: string; - // Dune (daily_volumes) - dune_buy_volume: number; - dune_sell_volume: number; - dune_base_volume: number; - dune_target_volume: number; - dune_trade_count: number; - dune_usdc_fees: number; - dune_token_fees_usdc: number; - // v06 aggregate (spot_* or total_* depending on --basis) - v06_buy_volume: number; - v06_sell_volume: number; - v06_base_volume: number; - v06_target_volume: number; - v06_trade_count: number; - v06_usdc_fees: number; - v06_token_fees_usdc: number; - // flags - has_conditional_volume: boolean; - conditional_reconciled: boolean; - // source presence - in_dune: boolean; - in_v06: boolean; -} - -interface DivergenceReport { - token: string; - date: string; - field: string; - dune_value: number; - v06_value: number; - delta: number; - delta_pct: number; - has_conditional_volume: boolean; - conditional_reconciled: boolean; -} - -// ---- main ---- - -type CompareBasis = 'spot' | 'total'; - -/** v0.6 indexer + reconciliation use raw token amounts (6 dp); Dune daily_volumes uses /1e6. */ -const V06_TO_HUMAN_DIVISOR = 1_000_000; - -async function main() { - const { values } = parseArgs({ - options: { - since: { type: 'string', short: 's' }, - days: { type: 'string', short: 'd' }, - token: { type: 'string', short: 't' }, - threshold: { type: 'string', short: 'p' }, - basis: { type: 'string', short: 'b' }, - }, - strict: false, - }); - - let since: Date; - if (values.since) { - since = new Date(values.since as string); - } else if (values.days) { - since = new Date(Date.now() - Number(values.days) * 86_400_000); - } else { - since = new Date(Date.now() - 90 * 86_400_000); - } - - const thresholdPct = values.threshold ? Number(values.threshold) : 1; - const tokenFilter = values.token as string | undefined; - const basisRaw = (values.basis as string | undefined)?.toLowerCase(); - const basis: CompareBasis = basisRaw === 'total' ? 'total' : 'spot'; - - console.log('🔍 Dune vs v06 Comparison'); - console.log(` Since: ${since.toISOString().slice(0, 10)}`); - console.log(` Token: ${tokenFilter || 'all'}`); - console.log(` Threshold: ${thresholdPct}%`); - console.log(` Basis: ${basis} (v06 ${basis === 'spot' ? 'spot_* ↔ Dune spot-only' : 'total_* includes conditional — gaps vs Dune expected'})`); - console.log(` v06 scale: volumes/fees ÷ ${V06_TO_HUMAN_DIVISOR} (raw 6dp → match Dune)`); - console.log(''); - - // ---- connect ---- - const { DatabaseService } = await import('../src/services/databaseService'); - const db = new DatabaseService(); - const ok = await db.initialize(); - if (!ok) { - console.error('❌ Failed to connect to database'); - process.exit(1); - } - console.log('✅ DB connected\n'); - - try { - // ---- run comparison query ---- - const rows = await runComparison(db, since, tokenFilter, basis); - const divergences = findDivergences(rows, thresholdPct); - - printSummary(rows, divergences, thresholdPct); - printMissingDays(rows); - printDivergences(divergences); - } finally { - await db.close(); - } -} - -// ---- SQL comparison: FULL OUTER JOIN daily_volumes ↔ v06_fee_volume_daily_aggregate ---- - -async function runComparison( - db: any, - since: Date, - tokenFilter: string | undefined, - basis: CompareBasis, -): Promise { - const params: any[] = [since.toISOString().slice(0, 10)]; - if (tokenFilter) { - params.push(tokenFilter); - } - - const v06Cols = - basis === 'total' - ? { - buy: 'v.total_buy_volume', - sell: 'v.total_sell_volume', - base: 'v.total_base_volume', - target: 'v.total_target_volume', - trades: 'v.total_trade_count', - usdc: 'v.total_usdc_fees', - tokUsdc: 'v.total_token_fees_usdc', - } - : { - buy: 'v.spot_buy_volume', - sell: 'v.spot_sell_volume', - base: 'v.spot_base_volume', - target: 'v.spot_target_volume', - trades: 'v.spot_trade_count', - usdc: 'v.spot_usdc_fees', - tokUsdc: 'v.spot_token_fees_usdc', - }; - - // Use a FULL OUTER JOIN so we see rows present in one table but not the other - const sql = ` - SELECT - COALESCE(d.token, v.token) AS token, - COALESCE(d.date, v.date)::text AS date, - -- dune - COALESCE(d.buy_volume, 0)::float8 AS dune_buy_volume, - COALESCE(d.sell_volume, 0)::float8 AS dune_sell_volume, - COALESCE(d.base_volume, 0)::float8 AS dune_base_volume, - COALESCE(d.target_volume, 0)::float8 AS dune_target_volume, - COALESCE(d.trade_count, 0)::int AS dune_trade_count, - COALESCE(d.usdc_fees, 0)::float8 AS dune_usdc_fees, - COALESCE(d.token_fees_usdc, 0)::float8 AS dune_token_fees_usdc, - -- v06 (${basis}) — divide by ${V06_TO_HUMAN_DIVISOR} so amounts match Dune human scale - (COALESCE(${v06Cols.buy}, 0)::numeric / ${V06_TO_HUMAN_DIVISOR})::float8 AS v06_buy_volume, - (COALESCE(${v06Cols.sell}, 0)::numeric / ${V06_TO_HUMAN_DIVISOR})::float8 AS v06_sell_volume, - (COALESCE(${v06Cols.base}, 0)::numeric / ${V06_TO_HUMAN_DIVISOR})::float8 AS v06_base_volume, - (COALESCE(${v06Cols.target}, 0)::numeric / ${V06_TO_HUMAN_DIVISOR})::float8 AS v06_target_volume, - COALESCE(${v06Cols.trades}, 0)::int AS v06_trade_count, - (COALESCE(${v06Cols.usdc}, 0)::numeric / ${V06_TO_HUMAN_DIVISOR})::float8 AS v06_usdc_fees, - (COALESCE(${v06Cols.tokUsdc}, 0)::numeric / ${V06_TO_HUMAN_DIVISOR})::float8 AS v06_token_fees_usdc, - -- flags (non-zero conditional activity on this token-day) - (COALESCE(v.conditional_trade_count, 0) > 0) AS has_conditional_volume, - COALESCE(v.conditional_reconciled, false) AS conditional_reconciled, - (d.token IS NOT NULL) AS in_dune, - (v.token IS NOT NULL) AS in_v06 - FROM daily_volumes d - FULL OUTER JOIN v06_fee_volume_daily_aggregate v - ON LOWER(d.token) = LOWER(v.token) AND d.date = v.date - WHERE COALESCE(d.date, v.date) >= $1 - ${tokenFilter ? `AND (LOWER(d.token) = LOWER($2) OR LOWER(v.token) = LOWER($2))` : ''} - ORDER BY COALESCE(d.token, v.token), COALESCE(d.date, v.date) - `; - - const pool = (db as any).pool; - const result = await pool.query(sql, params); - return result.rows; -} - -// ---- find divergences above threshold ---- - -const COMPARED_FIELDS = [ - 'buy_volume', - 'sell_volume', - 'base_volume', - 'target_volume', - 'trade_count', - 'usdc_fees', - 'token_fees_usdc', -] as const; - -function findDivergences(rows: ComparisonRow[], thresholdPct: number): DivergenceReport[] { - const divergences: DivergenceReport[] = []; - - for (const row of rows) { - if (!row.in_dune || !row.in_v06) continue; // handled separately as "missing" - - for (const field of COMPARED_FIELDS) { - const duneVal = (row as any)[`dune_${field}`] as number; - const v06Val = (row as any)[`v06_${field}`] as number; - const delta = v06Val - duneVal; - const maxAbs = Math.max(Math.abs(duneVal), Math.abs(v06Val)); - const deltaPct = maxAbs > 0 ? (Math.abs(delta) / maxAbs) * 100 : 0; - - if (deltaPct > thresholdPct) { - divergences.push({ - token: row.token, - date: row.date, - field, - dune_value: duneVal, - v06_value: v06Val, - delta, - delta_pct: deltaPct, - has_conditional_volume: row.has_conditional_volume, - conditional_reconciled: row.conditional_reconciled, - }); - } - } - } - - return divergences; -} - -// ---- print helpers ---- - -function printSummary(rows: ComparisonRow[], divergences: DivergenceReport[], thresholdPct: number) { - const tokens = new Set(rows.map((r) => r.token)); - const bothPresent = rows.filter((r) => r.in_dune && r.in_v06); - const duneOnly = rows.filter((r) => r.in_dune && !r.in_v06); - const v06Only = rows.filter((r) => !r.in_dune && r.in_v06); - const divergentDays = new Set(divergences.map((d) => `${d.token}|${d.date}`)); - - console.log('📊 SUMMARY'); - console.log(` Tokens compared: ${tokens.size}`); - console.log(` Total token-day pairs: ${rows.length}`); - console.log(` Both sources present: ${bothPresent.length}`); - console.log(` Dune only (missing v06): ${duneOnly.length}`); - console.log(` v06 only (missing Dune): ${v06Only.length}`); - console.log(` Divergent days (>${thresholdPct}%): ${divergentDays.size}`); - console.log(` Total field divergences: ${divergences.length}`); - console.log(''); -} - -function printMissingDays(rows: ComparisonRow[]) { - const duneOnly = rows.filter((r) => r.in_dune && !r.in_v06); - const v06Only = rows.filter((r) => !r.in_dune && r.in_v06); - - if (duneOnly.length > 0) { - console.log('⚠️ DAYS IN DUNE BUT MISSING FROM V06:'); - const byToken = groupBy(duneOnly, (r) => r.token); - for (const [token, days] of Object.entries(byToken)) { - const dates = days.map((d) => d.date).sort(); - const rangeStr = dates.length <= 5 - ? dates.join(', ') - : `${dates[0]} … ${dates[dates.length - 1]} (${dates.length} days)`; - console.log(` ${token}: ${rangeStr}`); - } - console.log(''); - } - - if (v06Only.length > 0) { - console.log('⚠️ DAYS IN V06 BUT MISSING FROM DUNE:'); - const byToken = groupBy(v06Only, (r) => r.token); - for (const [token, days] of Object.entries(byToken)) { - const dates = days.map((d) => d.date).sort(); - const rangeStr = dates.length <= 5 - ? dates.join(', ') - : `${dates[0]} … ${dates[dates.length - 1]} (${dates.length} days)`; - console.log(` ${token}: ${rangeStr}`); - } - console.log(''); - } - - if (duneOnly.length === 0 && v06Only.length === 0) { - console.log('✅ No missing days — both sources cover the same date ranges.\n'); - } -} - -function printDivergences(divergences: DivergenceReport[]) { - if (divergences.length === 0) { - console.log('✅ No field divergences above threshold.\n'); - return; - } - - console.log('🔴 DIVERGENCES:'); - console.log( - ' ' + - pad('TOKEN', 12) + - pad('DATE', 12) + - pad('FIELD', 18) + - padR('DUNE', 16) + - padR('V06', 16) + - padR('DELTA', 16) + - padR('Δ%', 8) + - pad('COND?', 6) + - pad('RECON?', 6), - ); - console.log(' ' + '-'.repeat(110)); - - for (const d of divergences) { - console.log( - ' ' + - pad(d.token, 12) + - pad(d.date, 12) + - pad(d.field, 18) + - padR(fmt(d.dune_value), 16) + - padR(fmt(d.v06_value), 16) + - padR(fmt(d.delta), 16) + - padR(d.delta_pct.toFixed(1) + '%', 8) + - pad(d.has_conditional_volume ? 'Y' : 'N', 6) + - pad(d.conditional_reconciled ? 'Y' : 'N', 6), - ); - } - - // Per-token summary - console.log(''); - const byToken = groupBy(divergences, (d) => d.token); - console.log(' Per-token divergence counts:'); - for (const [token, items] of Object.entries(byToken)) { - const fields = groupBy(items, (i) => i.field); - const fieldSummary = Object.entries(fields) - .map(([f, arr]) => `${f}:${arr.length}`) - .join(', '); - console.log(` ${token}: ${items.length} divergences (${fieldSummary})`); - } - console.log(''); -} - -// ---- util ---- - -function groupBy(arr: T[], key: (item: T) => string): Record { - const map: Record = {}; - for (const item of arr) { - const k = key(item); - (map[k] ??= []).push(item); - } - return map; -} - -function pad(s: string, n: number) { - return s.padEnd(n); -} -function padR(s: string, n: number) { - return s.padStart(n - 1) + ' '; -} -function fmt(n: number): string { - if (Math.abs(n) < 0.01) return '0'; - return n.toLocaleString('en-US', { maximumFractionDigits: 2 }); -} - -main().catch((err) => { - console.error('❌ Fatal:', err); - process.exit(1); -}); diff --git a/scripts/repo-guard.ts b/scripts/repo-guard.ts index fc271fa..fb887a2 100644 --- a/scripts/repo-guard.ts +++ b/scripts/repo-guard.ts @@ -72,7 +72,6 @@ const sensitiveFiles = [ "src/services/databaseService.ts", "src/services/externalDatabaseService.ts", "scripts/safe-update.ts", - "scripts/backfill.ts", "scripts/backfillV06.ts", ]; diff --git a/src/routes/health.ts b/src/routes/health.ts index 5454e55..9804fc3 100644 --- a/src/routes/health.ts +++ b/src/routes/health.ts @@ -3,7 +3,7 @@ import type { ServiceGetters } from './types.js'; export function createHealthRouter(services: ServiceGetters): Router { const router = Router(); - const { getDatabaseService } = services; + const { getDatabaseService, getExternalDatabaseService } = services; // Basic health check router.get('/health', (req: Request, res: Response) => { @@ -17,6 +17,10 @@ export function createHealthRouter(services: ServiceGetters): Router { // Comprehensive health check router.get('/api/health', async (req: Request, res: Response) => { const databaseService = getDatabaseService(); + const externalDatabaseService = getExternalDatabaseService(); + // The served (external) indexer DB now backs Meteora, tickers, DexScreener and + // first-trade-dates — it's a core dependency, so health must reflect it. + const externalConnected = !!externalDatabaseService?.isAvailable(); const health: Record = { status: 'healthy', @@ -25,15 +29,21 @@ export function createHealthRouter(services: ServiceGetters): Router { database: { connected: databaseService.isAvailable(), }, + externalDatabase: { + connected: externalConnected, + }, }; const hasUnhealthyService = Object.values(health.services).some( (s: any) => s.initialized === false ); - + if (!databaseService.isAvailable()) { health.status = 'degraded'; - health.message = 'Database not connected'; + health.message = 'App database not connected'; + } else if (!externalConnected) { + health.status = 'degraded'; + health.message = 'Served (external) indexer database not connected'; } else if (hasUnhealthyService) { health.status = 'degraded'; health.message = 'One or more services not initialized'; diff --git a/src/routes/market.ts b/src/routes/market.ts index bad4a59..b336bbf 100644 --- a/src/routes/market.ts +++ b/src/routes/market.ts @@ -43,14 +43,20 @@ export function createMarketRouter(services: ServiceGetters): Router { }; // Meteora rows are served directly from our meteora accounting ETL - // (futarchy.meteora_daily in the served DB, via externalDatabase). + // (futarchy.meteora_daily in the served DB, via externalDatabase). The served DB is a + // hard dependency for Meteora — surface its absence/failure rather than masking it as + // empty (a financial feed must never read a DB outage as "zero volume"). const externalDatabaseService = getExternalDatabaseService(); + if (!externalDatabaseService || !externalDatabaseService.isAvailable()) { + return res.status(503).json({ + error: 'Served database not available', + message: 'Meteora data source (served indexer DB) is not connected', + }); + } const [futarchyData, meteoraData] = await Promise.all([ databaseService.getDailyTradingActivity(queryOptions), - externalDatabaseService?.isAvailable() - ? externalDatabaseService.getDailyMeteoraVolumes(queryOptions) - : Promise.resolve([]), + externalDatabaseService.getDailyMeteoraVolumes(queryOptions), ]); res.json({ diff --git a/src/services/externalDatabaseService.ts b/src/services/externalDatabaseService.ts index 1613570..41210e7 100644 --- a/src/services/externalDatabaseService.ts +++ b/src/services/externalDatabaseService.ts @@ -184,8 +184,12 @@ export class ExternalDatabaseService { ); return result.rows; } catch (error: any) { + // Surface query/schema failures — do NOT mask as empty. For a financial endpoint, + // an empty array must mean "genuinely no rows", never "the query failed". The caller + // (market route) guards `isAvailable()` for the connection-down case and lets a real + // failure propagate to a 5xx instead of returning 200 with zero volume. logger.error('[ExternalDB] Error getting daily Meteora volumes from futarchy.meteora_daily:', error); - return []; + throw error; } } From 99e25e6685809ab70baa75409d3d126b2ad9c7fb Mon Sep 17 00:00:00 2001 From: Jinglun Date: Fri, 5 Jun 2026 15:33:27 -0700 Subject: [PATCH 05/14] refactor(external-api): remove the indexer runtime; per-DAO ticker fallback; surface served-DB failures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit H1: /api/tickers v0.6 fallback is now per-DAO — fills only the DAOs futarchy.trades didn't cover and merges, instead of all-or-nothing (partial primary coverage no longer zeroes the rest). H2: getSpotRolling24hMetrics now throws on query/schema failure (returns empty only for genuine zero/not-connected), so served-DB contract drift surfaces as a 5xx via asyncHandler instead of silently degrading to the v0.6 fallback. M1: removed the stale Dune-cache block (DUNE_CACHE_REFRESH_INTERVAL/DUNE_FETCH_TIMEOUT) from GET / metadata; replaced with a read-only/no-Dune note. Indexer removal (this repo is now a pure read-only API): deleted src/indexer.ts, V06ReconciliationService, backfillV06/backfillV06Fees scripts, the dev/start:indexer + backfill:v06 package scripts, and all wiring (runtime/services RuntimeMode narrowed to 'api', Services field, repo-guard/CODEOWNERS/README). tsc clean; bun test 112 pass. DEPLOY GATE: the v0.6 READ methods (getDailyTradingActivity → v06_fee_volume_daily_aggregate, getV06Rolling24hMetrics → v06_spot_ohlcv_1m) are KEPT, but this repo no longer POPULATES those app-DB aggregates. Before deploy, the v0.6 daily/OHLCV aggregates must be produced elsewhere (reid's indexer) OR these reads repointed to compute on-the-fly from the served DB's v0_6_spot_swaps/v0_6_conditional_swaps — otherwise FutarchyAMM market-data/tickers serve stale data. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 25 +- package.json | 4 - scripts/backfillV06.ts | 130 ----- scripts/backfillV06Fees.ts | 110 ---- scripts/repo-guard.ts | 1 - src/indexer.ts | 92 --- src/routes/coingecko.ts | 26 +- src/routes/root.ts | 8 +- src/routes/types.ts | 2 - src/runtime/services.ts | 11 +- src/services/externalDatabaseService.ts | 7 +- src/services/v06ReconciliationService.ts | 698 ----------------------- tests/api.test.ts | 1 - tests/helpers/testApp.ts | 1 - tests/routes/root.test.ts | 11 +- tests/runtime/services.test.ts | 7 - 16 files changed, 32 insertions(+), 1102 deletions(-) delete mode 100644 scripts/backfillV06.ts delete mode 100644 scripts/backfillV06Fees.ts delete mode 100644 src/indexer.ts delete mode 100644 src/services/v06ReconciliationService.ts diff --git a/README.md b/README.md index c7a7a7c..a14246a 100644 --- a/README.md +++ b/README.md @@ -197,14 +197,8 @@ bun run build # Start the server (runs build first) bun run start -# Start indexing workers separately (runs build first) -bun run start:indexer - # Development with hot reload bun run dev - -# Development indexer with hot reload -bun run dev:indexer ``` ## Configuration @@ -242,9 +236,8 @@ Create a `.env` file in the root directory (see `example.env` for reference): src/ ├── app.ts # Express app setup & middleware ├── main.ts # API entry point (serves routes, no indexing workers) -├── indexer.ts # Indexer entry point (Dune collection, rollups, v0.6 reconciliation) ├── runtime/ -│ └── services.ts # API/indexer service composition +│ └── services.ts # API service composition ├── config.ts # Environment variables & configuration ├── routes/ │ ├── index.ts # Route registration @@ -260,7 +253,6 @@ src/ │ ├── priceService.ts # Price, spread, liquidity calculations │ ├── databaseService.ts # App DB (volumes, OHLCV, fees, metrics) │ ├── externalDatabaseService.ts # Read-only indexer DB connection -│ ├── v06ReconciliationService.ts # v0.6 data reconciliation (served DB → app-DB v0.6 aggregates) │ ├── solanaService.ts # SPL token supply queries │ ├── launchpadService.ts # Token allocation breakdown │ └── metricsService.ts # Prometheus counters/histograms @@ -279,26 +271,13 @@ src/ ### Sources (Dune fully removed) The API process serves data from the app DB (v0.6 aggregates: `v06_fee_volume_daily_aggregate`, `v06_spot_ohlcv_1m`) and the read-only served indexer DB (`futarchy.trades`, `futarchy.meteora_daily`, `v0_6_spot_swaps`). It does not start indexing, fetchers, rollups, or app-DB schema setup. -The only remaining background job is **v0.6 reconciliation** (`bun run start:indexer`): reads `v0_6_spot_swaps` + `v0_6_conditional_swaps` from the served indexer DB and writes the v0.6 OHLCV + fee-breakdown aggregates to the app DB. (This is being phased out — the goal is for the API to read all FutarchyAMM aggregates directly from the served DB, with no indexer in this repo.) +This is a pure read-only API with no in-process indexing: FutarchyAMM v0.6 aggregates are read from the app DB, while Meteora data and tickers are read from the served indexer DB. `/api/tickers` reads 24h metrics primarily from the served DB's `futarchy.trades` (direct), falling back to the app-DB v0.6 OHLCV. ### DexScreener Pipeline The DexScreener adapter reads **directly from the external indexer DB** (`v0_6_spot_swaps` + `v0_6_daos`) and serves real-time swap events indexed by Solana slot. No intermediate aggregation — raw swap data mapped to the DexScreener schema. -## Backfill Scripts - -```bash -# Full v0.6 backfill (since 2025-01-01) -bun run backfill:v06 - -# With custom range -bun run backfill:v06 -- --since 2025-06-01 - -# Chunked for large ranges -bun run backfill:v06 -- --since 2025-01-01 --chunk-days 30 -``` - ## Rate Limiting - **Anonymous (default):** 60 requests per minute per IP. Returns `429 Too Many Requests` when exceeded. diff --git a/package.json b/package.json index c2f15fc..c89ffb9 100644 --- a/package.json +++ b/package.json @@ -6,14 +6,10 @@ "version": "2.0.0", "scripts": { "dev": "bun --watch src/main.ts", - "dev:indexer": "bun --watch src/indexer.ts", "build": "tsc", "start": "bun dist/main.js", - "prestart:indexer": "bun run build", - "start:indexer": "bun dist/indexer.js", "prestart": "bun run build", "test": "bun test", - "backfill:v06": "bun run scripts/backfillV06.ts", "safe-update": "bun run scripts/safe-update.ts", "safe-update:dry": "bun run scripts/safe-update.ts --dry-run", "typecheck": "tsc --noEmit", diff --git a/scripts/backfillV06.ts b/scripts/backfillV06.ts deleted file mode 100644 index 65a8e5b..0000000 --- a/scripts/backfillV06.ts +++ /dev/null @@ -1,130 +0,0 @@ -#!/usr/bin/env bun -/** - * v0.6 Backfill — pull historical data from the external indexer DB - * into the app DB's v06_* tables. - * - * Usage: - * bun run scripts/backfillV06.ts # backfill everything (default since 2025-01-01) - * bun run scripts/backfillV06.ts --since 2025-06-01 - * bun run scripts/backfillV06.ts --days 90 # last 90 days - * - * Speed tips: - * - Only need fee / aggregate tables? Use `bun run scripts/backfillV06Fees.ts` - * (skips OHLCV 1m/1d — usually much faster). - * - After 1m, reconciliation runs daily rollup + spot fees + conditional fees - * in parallel on the app DB / indexer pools. - * - For huge date ranges, `--chunk-days` avoids one enormous indexer scan - * (can help with memory/timeouts; total work is similar). - * - * Required env: - * DATABASE_URL (or COINGECKO_PG_URL) — app DB (read-write) - * EXTERNAL_DATABASE_URL — indexer DB (read-only) - */ - -import { parseArgs } from 'util'; - -async function main() { - // ---- CLI args ---- - const { values } = parseArgs({ - options: { - since: { type: 'string', short: 's' }, - days: { type: 'string', short: 'd' }, - 'chunk-days': { type: 'string', short: 'c' }, - }, - strict: false, - }); - - let since: Date; - if (values.since) { - since = new Date(values.since as string); - } else if (values.days) { - since = new Date(Date.now() - Number(values.days) * 86_400_000); - } else { - since = new Date('2025-01-01T00:00:00Z'); - } - - const chunkDays = values['chunk-days'] ? Number(values['chunk-days']) : 0; - - console.log('🚀 v0.6 Backfill'); - console.log(` Since: ${since.toISOString()}`); - console.log(` Chunk days: ${chunkDays || 'none (single pass)'}`); - console.log(''); - - // ---- env checks ---- - if (!process.env.DATABASE_URL && !process.env.COINGECKO_PG_URL) { - console.error('❌ DATABASE_URL (or COINGECKO_PG_URL) is required'); - process.exit(1); - } - if (!process.env.EXTERNAL_DATABASE_URL) { - console.error('❌ EXTERNAL_DATABASE_URL is required'); - process.exit(1); - } - - // ---- init services ---- - const { DatabaseService } = await import('../src/services/databaseService'); - const { ExternalDatabaseService } = await import('../src/services/externalDatabaseService'); - const { V06ReconciliationService } = await import('../src/services/v06ReconciliationService'); - - const appDb = new DatabaseService(); - const extDb = new ExternalDatabaseService(); - - const dbOk = await appDb.initialize(); - if (!dbOk) { - console.error('❌ Failed to connect to app database'); - process.exit(1); - } - console.log('✅ App DB connected'); - - const extOk = await extDb.initialize(); - if (!extOk) { - console.error('❌ Failed to connect to external indexer database'); - await appDb.close(); - process.exit(1); - } - console.log('✅ External DB connected'); - - const reconciler = new V06ReconciliationService(appDb, extDb); - - try { - const totalStart = Date.now(); - - if (chunkDays > 0) { - // ---- chunked backfill ---- - const chunkMs = chunkDays * 86_400_000; - let cursor = new Date(since); - const now = new Date(); - const totalChunks = Math.ceil((now.getTime() - cursor.getTime()) / chunkMs); - let chunkNum = 0; - - while (cursor < now) { - chunkNum++; - const chunkStart = Date.now(); - const chunkEnd = new Date(Math.min(cursor.getTime() + chunkMs, now.getTime())); - console.log(`\n📦 Chunk ${chunkNum}/${totalChunks}: ${cursor.toISOString()} → ${chunkEnd.toISOString()}`); - await reconciler.reconcile(cursor, chunkEnd); - cursor = chunkEnd; - const chunkElapsed = ((Date.now() - chunkStart) / 1000).toFixed(1); - const totalElapsed = ((Date.now() - totalStart) / 1000).toFixed(1); - const remaining = totalChunks - chunkNum; - const avgPerChunk = (Date.now() - totalStart) / chunkNum / 1000; - const eta = remaining > 0 ? `~${(remaining * avgPerChunk / 60).toFixed(1)} min` : 'done'; - console.log(` ✓ Chunk ${chunkNum} done in ${chunkElapsed}s (total: ${totalElapsed}s, remaining: ${remaining} chunks, ETA: ${eta})`); - } - } else { - // ---- single pass ---- - console.log('\n⏳ Running single-pass reconciliation (this may take a while)...'); - await reconciler.reconcile(since); - } - - const totalElapsed = ((Date.now() - totalStart) / 1000).toFixed(1); - console.log(`\n✅ Backfill complete in ${totalElapsed}s!`); - } catch (error: any) { - console.error('❌ Backfill failed:', error.message || error); - process.exit(1); - } finally { - await extDb.close(); - await appDb.close(); - } -} - -main().catch(console.error); diff --git a/scripts/backfillV06Fees.ts b/scripts/backfillV06Fees.ts deleted file mode 100644 index fd37806..0000000 --- a/scripts/backfillV06Fees.ts +++ /dev/null @@ -1,110 +0,0 @@ -#!/usr/bin/env bun -/** - * v0.6 Fee-only Backfill — re-calculates fee volume daily tables - * (spot, conditional, aggregate) without touching OHLCV data. - * - * Usage: - * bun run scripts/backfillV06Fees.ts # default since 2025-01-01 - * bun run scripts/backfillV06Fees.ts --since 2025-06-01 - * bun run scripts/backfillV06Fees.ts --days 90 - * bun run scripts/backfillV06Fees.ts --chunk-days 30 - * - * Spot and conditional fee queries run in parallel against the indexer, - * then the aggregate step runs once. - */ - -import { parseArgs } from 'util'; - -async function main() { - const { values } = parseArgs({ - options: { - since: { type: 'string', short: 's' }, - days: { type: 'string', short: 'd' }, - 'chunk-days': { type: 'string', short: 'c' }, - }, - strict: false, - }); - - let since: Date; - if (values.since) { - since = new Date(values.since as string); - } else if (values.days) { - since = new Date(Date.now() - Number(values.days) * 86_400_000); - } else { - since = new Date('2025-01-01T00:00:00Z'); - } - - const chunkDays = values['chunk-days'] ? Number(values['chunk-days']) : 0; - - console.log('🚀 v0.6 Fee-only Backfill'); - console.log(` Since: ${since.toISOString()}`); - console.log(` Chunk days: ${chunkDays || 'none (single pass)'}`); - console.log(''); - - if (!process.env.DATABASE_URL && !process.env.COINGECKO_PG_URL) { - console.error('❌ DATABASE_URL (or COINGECKO_PG_URL) is required'); - process.exit(1); - } - if (!process.env.EXTERNAL_DATABASE_URL) { - console.error('❌ EXTERNAL_DATABASE_URL is required'); - process.exit(1); - } - - const { DatabaseService } = await import('../src/services/databaseService'); - const { ExternalDatabaseService } = await import('../src/services/externalDatabaseService'); - const { V06ReconciliationService } = await import('../src/services/v06ReconciliationService'); - - const appDb = new DatabaseService(); - const extDb = new ExternalDatabaseService(); - - const dbOk = await appDb.initialize(); - if (!dbOk) { console.error('❌ Failed to connect to app DB'); process.exit(1); } - console.log('✅ App DB connected'); - - const extOk = await extDb.initialize(); - if (!extOk) { console.error('❌ Failed to connect to external DB'); await appDb.close(); process.exit(1); } - console.log('✅ External DB connected'); - - const reconciler = new V06ReconciliationService(appDb, extDb); - - try { - const totalStart = Date.now(); - - if (chunkDays > 0) { - const chunkMs = chunkDays * 86_400_000; - let cursor = new Date(since); - const now = new Date(); - const totalChunks = Math.ceil((now.getTime() - cursor.getTime()) / chunkMs); - let chunkNum = 0; - - while (cursor < now) { - chunkNum++; - const chunkStart = Date.now(); - const chunkEnd = new Date(Math.min(cursor.getTime() + chunkMs, now.getTime())); - console.log(`\n📦 Chunk ${chunkNum}/${totalChunks}: ${cursor.toISOString()} → ${chunkEnd.toISOString()}`); - await reconciler.reconcileFeesOnly(cursor, chunkEnd); - cursor = chunkEnd; - const chunkElapsed = ((Date.now() - chunkStart) / 1000).toFixed(1); - const totalElapsed = ((Date.now() - totalStart) / 1000).toFixed(1); - const remaining = totalChunks - chunkNum; - const avgPerChunk = (Date.now() - totalStart) / chunkNum / 1000; - const eta = remaining > 0 ? `~${(remaining * avgPerChunk / 60).toFixed(1)} min` : 'done'; - console.log(` ✓ Chunk ${chunkNum} done in ${chunkElapsed}s (total: ${totalElapsed}s, remaining: ${remaining} chunks, ETA: ${eta})`); - } - } else { - console.log('\n⏳ Running single-pass fee reconciliation...'); - await reconciler.reconcileFeesOnly(since); - } - - const totalElapsed = ((Date.now() - totalStart) / 1000).toFixed(1); - console.log(`\n✅ Fee backfill complete in ${totalElapsed}s!`); - } catch (error: any) { - console.error('❌ Fee backfill failed:', error.message || error); - process.exit(1); - } finally { - await extDb.close(); - await appDb.close(); - } -} - -main().catch(console.error); diff --git a/scripts/repo-guard.ts b/scripts/repo-guard.ts index fb887a2..edda76e 100644 --- a/scripts/repo-guard.ts +++ b/scripts/repo-guard.ts @@ -72,7 +72,6 @@ const sensitiveFiles = [ "src/services/databaseService.ts", "src/services/externalDatabaseService.ts", "scripts/safe-update.ts", - "scripts/backfillV06.ts", ]; // Files whose purpose is to define the guard rules themselves, or to document diff --git a/src/indexer.ts b/src/indexer.ts deleted file mode 100644 index 83314ed..0000000 --- a/src/indexer.ts +++ /dev/null @@ -1,92 +0,0 @@ -import { saveHealthSnapshots } from './routes/health.js'; -import { createServiceGetters } from './routes/types.js'; -import { closeDataStores, createServices, initializeRuntimeDataStores } from './runtime/services.js'; -import { logger } from './utils/logger.js'; -import { scheduleDailyAtUTC, scheduleWithoutPileup, type ScheduledTask } from './utils/scheduling.js'; -import type { Services } from './app.js'; - -async function startIndexingServices(services: Services): Promise { - if (services.externalDatabaseService?.isAvailable() && services.v06ReconciliationService) { - logger.info('Starting v0.6 Reconciliation service'); - services.v06ReconciliationService.start(); - logger.info('v0.6 Reconciliation service started'); - } -} - -function startIndexerScheduledTasks(services: Services): ScheduledTask[] { - const tasks: ScheduledTask[] = []; - const serviceGetters = createServiceGetters(services); - - const healthSnapshotTask = scheduleWithoutPileup( - async () => { - await saveHealthSnapshots(serviceGetters); - }, - { - name: 'IndexerHealthSnapshot', - intervalMs: 5 * 60 * 1000, - onError: (error) => logger.error('Error saving indexer health snapshot', error), - } - ); - tasks.push(healthSnapshotTask); - logger.info('Indexer health snapshots scheduled every 5 minutes'); - - const metricsPruneTask = scheduleDailyAtUTC( - async () => { - if (services.databaseService.isAvailable()) { - await services.databaseService.pruneOldMetrics(30); - logger.info('Old metrics pruned (keeping last 30 days)'); - } - }, - { - name: 'IndexerMetricsPrune', - hourUTC: 3, - onError: (error) => logger.error('Error pruning indexer metrics', error), - } - ); - tasks.push(metricsPruneTask); - logger.info('Indexer metrics pruning scheduled daily at 03:00 UTC'); - - return tasks; -} - -async function stopIndexingServices(services: Services, scheduledTasks: ScheduledTask[]): Promise { - scheduledTasks.forEach(task => task.stop()); - services.v06ReconciliationService?.stop(); - await closeDataStores(services); -} - -async function main(): Promise { - const services = createServices('indexer'); - let scheduledTasks: ScheduledTask[] = []; - - await initializeRuntimeDataStores(services, 'Indexer', { ensureAppSchema: true }); - await startIndexingServices(services); - scheduledTasks = startIndexerScheduledTasks(services); - - const serviceGetters = createServiceGetters(services); - setTimeout(async () => { - try { - await saveHealthSnapshots(serviceGetters); - logger.info('Initial indexer health snapshot saved'); - } catch (error) { - logger.error('Error saving initial indexer health snapshot', error); - } - }, 10000); - - logger.info('Indexer runtime started'); - - const shutdown = async (signal: string): Promise => { - logger.info(`${signal} received, shutting down indexer gracefully`); - await stopIndexingServices(services, scheduledTasks); - logger.info('Indexer stopped'); - process.exit(0); - }; - - process.on('SIGTERM', () => shutdown('SIGTERM')); - process.on('SIGINT', () => shutdown('SIGINT')); -} - -main().catch((error) => { - logger.error('Failed to start indexer runtime', error); - process.exit(1); -}); diff --git a/src/routes/coingecko.ts b/src/routes/coingecko.ts index 7de3eb7..f4ed7a9 100644 --- a/src/routes/coingecko.ts +++ b/src/routes/coingecko.ts @@ -51,28 +51,32 @@ export function createCoinGeckoRouter(services: ServiceGetters): Router { } } - // Fallback: v0.6 indexer OHLCV (app DB, populated by reconciliation from - // the indexer DB). Live in prod and Dune-free — covers the window before - // futarchy.trades is populated for a DAO. - if (volumeMetricsMap.size === 0 && databaseService?.isAvailable()) { - const baseMints = allDaos.map(dao => dao.baseMint.toString()); - const v06Metrics = await databaseService.getV06Rolling24hMetrics(baseMints); - + // Fallback: v0.6 indexer OHLCV (app DB) for ONLY the DAOs the primary + // (futarchy.trades) didn't cover — per-DAO merge, not all-or-nothing. This + // covers the window before futarchy.trades is populated for a given DAO + // without zeroing out the DAOs that the primary did return. + const missingDaos = allDaos.filter(dao => !volumeMetricsMap.has(dao.daoAddress.toString())); + if (missingDaos.length > 0 && databaseService?.isAvailable()) { + const missingBaseMints = missingDaos.map(dao => dao.baseMint.toString()); + const v06Metrics = await databaseService.getV06Rolling24hMetrics(missingBaseMints); + + let filled = 0; for (const [tokenAddress, metrics] of v06Metrics.entries()) { const daoAddress = tokenToDaoMap.get(tokenAddress); - if (daoAddress) { + if (daoAddress && !volumeMetricsMap.has(daoAddress)) { volumeMetricsMap.set(daoAddress, { base_volume_24h: metrics.base_volume_24h, target_volume_24h: metrics.target_volume_24h, high_24h: metrics.high_24h, low_24h: metrics.low_24h, }); + filled++; } } - if (volumeMetricsMap.size > 0) { - volumeSource = 'v06-indexer-fallback'; - logger.debug('Using v0.6 indexer rolling 24h metrics (fallback)', { daoCount: volumeMetricsMap.size, requestId: req.requestId }); + if (filled > 0) { + volumeSource = volumeSource === 'futarchy-trades-db' ? 'futarchy-trades-db+v06-fallback' : 'v06-indexer-fallback'; + logger.debug('Filled missing DAOs from v0.6 indexer rolling 24h metrics', { filled, requestId: req.requestId }); } } diff --git a/src/routes/root.ts b/src/routes/root.ts index b6c9e6f..5db7f2a 100644 --- a/src/routes/root.ts +++ b/src/routes/root.ts @@ -39,13 +39,7 @@ export function createRootRouter(_services: ServiceGetters): Router { futarchyAmmLiquidity: 'Tokens in the internal FutarchyAMM for spot trading - IS circulating', meteoraLpLiquidity: 'Tokens in the external Meteora DAMM pool (POL) - IS circulating', }, - caching: { - description: 'Ticker volume is served from app DB aggregates populated by the separate indexer runtime', - refreshInterval: `${parseInt(process.env.DUNE_CACHE_REFRESH_INTERVAL || '3600')} seconds`, - fetchTimeout: `${parseInt(process.env.DUNE_FETCH_TIMEOUT || '240')} seconds`, - status: 'No live cache in API runtime', - }, - note: 'This API discovers DAOs for serving responses; background indexing runs separately.', + note: 'Read-only API. Data is served from our own indexed/ETL data (served indexer DB + app-DB v0.6 aggregates); no Dune, no in-process indexing.', }); }); diff --git a/src/routes/types.ts b/src/routes/types.ts index 3593cab..6c0df80 100644 --- a/src/routes/types.ts +++ b/src/routes/types.ts @@ -4,7 +4,6 @@ import type { SolanaService } from '../services/solanaService.js'; import type { LaunchpadService } from '../services/launchpadService.js'; import type { DatabaseService } from '../services/databaseService.js'; import type { ExternalDatabaseService } from '../services/externalDatabaseService.js'; -import type { V06ReconciliationService } from '../services/v06ReconciliationService.js'; import { AppError } from '../middleware/errorHandler.js'; /** @@ -21,7 +20,6 @@ export interface Services { databaseService: DatabaseService; externalDatabaseService: ExternalDatabaseService | null; - v06ReconciliationService: V06ReconciliationService | null; solanaService?: SolanaService; launchpadService?: LaunchpadService; diff --git a/src/runtime/services.ts b/src/runtime/services.ts index 4926dba..572ea01 100644 --- a/src/runtime/services.ts +++ b/src/runtime/services.ts @@ -5,12 +5,12 @@ import { FutarchyService } from '../services/futarchyService.js'; import { LaunchpadService } from '../services/launchpadService.js'; import { PriceService } from '../services/priceService.js'; import { SolanaService } from '../services/solanaService.js'; -import { V06ReconciliationService } from '../services/v06ReconciliationService.js'; import { logger } from '../utils/logger.js'; -export type RuntimeMode = 'api' | 'indexer'; +export type RuntimeMode = 'api'; export function createServices(mode: RuntimeMode): Services { + void mode; const futarchyService = new FutarchyService(); const priceService = new PriceService(); const databaseService = new DatabaseService(); @@ -18,12 +18,6 @@ export function createServices(mode: RuntimeMode): Services { const solanaService = new SolanaService(); const launchpadService = new LaunchpadService(); - let v06ReconciliationService: V06ReconciliationService | null = null; - - if (mode === 'indexer') { - v06ReconciliationService = new V06ReconciliationService(databaseService, externalDatabaseService); - } - return { futarchyService, priceService, @@ -31,7 +25,6 @@ export function createServices(mode: RuntimeMode): Services { externalDatabaseService, solanaService, launchpadService, - v06ReconciliationService, }; } diff --git a/src/services/externalDatabaseService.ts b/src/services/externalDatabaseService.ts index 41210e7..7fd9fe7 100644 --- a/src/services/externalDatabaseService.ts +++ b/src/services/externalDatabaseService.ts @@ -110,8 +110,13 @@ export class ExternalDatabaseService { } return metricsMap; } catch (error: any) { + // Surface query/schema failures (e.g. served-DB contract drift) instead of + // masking them as an empty map — an empty map must mean "genuinely no spot + // trades in the window", not "the query failed". The /api/tickers handler is + // wrapped in asyncHandler, so this propagates to a clean 5xx rather than + // silently degrading to the v0.6 fallback and hiding the problem. logger.error('[ExternalDB] Error getting spot rolling 24h metrics from futarchy.trades:', error); - return new Map(); + throw error; } } diff --git a/src/services/v06ReconciliationService.ts b/src/services/v06ReconciliationService.ts deleted file mode 100644 index 38d8550..0000000 --- a/src/services/v06ReconciliationService.ts +++ /dev/null @@ -1,698 +0,0 @@ -import { config } from '../config.js'; -import { logger } from '../utils/logger.js'; -import { scheduleWithoutPileup, type ScheduledTask } from '../utils/scheduling.js'; -import type { DatabaseService } from './databaseService.js'; -import type { ExternalDatabaseService } from './externalDatabaseService.js'; - -const FEE_RATE = config.fees.protocolFeeRate; // 0.005 = 0.5% - -/** - * Hourly reconciliation service for v0.6 data. - * - * Reads from the external indexer DB (v0_6_spot_swaps, v0_6_conditional_swaps, - * v0_6_daos, v0_6_proposals) and upserts into the app DB's v06_* tables. - * - * Pipeline: - * 1. Spot OHLCV 1m — scan v0_6_spot_swaps JOIN v0_6_daos - * 2. Spot OHLCV 1d — rollup from 1m data - * 3. Fee volume daily spot — aggregate spot swaps by calendar day - * 4. Fee volume daily conditional — winning-market-only swaps - * 5. Fee volume daily aggregate — spot + conditional totals - */ -export class V06ReconciliationService { - private scheduledTask: ScheduledTask | null = null; - private static readonly RECONCILIATION_INTERVAL_MS = 60 * 60 * 1000; // 1 hour - // Window must start at a UTC day boundary — fee queries GROUP BY UTC day and a mid-day start - // would upsert partial-day sums over complete rows. Must also be > proposal voting period (3d) - // so that conditional swaps get re-reconciled after their proposal resolves. - private static readonly LOOKBACK_DAYS = 4; - - constructor( - private readonly appDb: DatabaseService, - private readonly extDb: ExternalDatabaseService, - ) {} - - start(): void { - if (!this.extDb.isAvailable()) { - logger.info('[V06Reconciliation] External DB not available — service will not start'); - return; - } - - this.scheduledTask = scheduleWithoutPileup( - () => this.reconcile(), - { - name: 'V06Reconciliation', - intervalMs: V06ReconciliationService.RECONCILIATION_INTERVAL_MS, - immediate: true, - onError: (err) => logger.error('[V06Reconciliation] Error during reconciliation', err), - }, - ); - logger.info('[V06Reconciliation] Service started — hourly reconciliation enabled'); - } - - stop(): void { - this.scheduledTask?.stop(); - this.scheduledTask = null; - logger.info('[V06Reconciliation] Service stopped'); - } - - isRunning(): boolean { - return this.scheduledTask?.isRunning() ?? false; - } - - getLastRunTime(): Date | null { - return this.scheduledTask?.getLastRunTime() ?? null; - } - - // ------------------------------------------------------------------ - // Top-level reconciliation - // ------------------------------------------------------------------ - - private static startOfUtcDay(d: Date): Date { - const t = new Date(d.getTime()); - t.setUTCHours(0, 0, 0, 0); - return t; - } - - private static defaultWindowStart(): Date { - const floor = V06ReconciliationService.startOfUtcDay(new Date()); - floor.setUTCDate(floor.getUTCDate() - V06ReconciliationService.LOOKBACK_DAYS); - return floor; - } - - // Always floors to UTC midnight — a caller-supplied mid-day `since` would reintroduce the - // partial-day overwrite bug. - private static resolveWindowStart(since?: Date): Date { - return since - ? V06ReconciliationService.startOfUtcDay(since) - : V06ReconciliationService.defaultWindowStart(); - } - - /** - * Run the full reconciliation pipeline. - * @param since Lookback window start (floored to UTC midnight). Defaults to LOOKBACK_DAYS ago. - * @param until Window upper bound (exclusive). Defaults to now. - */ - async reconcile(since?: Date, until?: Date): Promise { - const start = Date.now(); - const windowStart = V06ReconciliationService.resolveWindowStart(since); - const windowEnd = until ?? new Date(); - // Pass ISO strings to sub-methods so the pg driver never calls Date.toString() - // (which can produce un-parseable timezone names like "GMT-0700" under Bun). - const sinceISO = windowStart.toISOString(); - const untilISO = windowEnd.toISOString(); - logger.info(`[V06Reconciliation] Starting reconciliation cycle (since ${sinceISO} until ${untilISO})`); - - await this.reconcileSpotOhlcv1m(sinceISO, untilISO); - // 1d rollup (app DB) and fee aggregates (indexer → app DB) are independent — run in parallel. - await Promise.all([ - this.reconcileSpotOhlcv1d(windowStart), - this.reconcileFeeDailySpot(sinceISO, untilISO), - this.reconcileFeeDailyConditional(sinceISO, untilISO), - ]); - await this.reconcileFeeDailyAggregate(windowStart); - - const elapsed = ((Date.now() - start) / 1000).toFixed(1); - logger.info(`[V06Reconciliation] Cycle complete in ${elapsed}s`); - } - - async reconcileFeesOnly(since?: Date, until?: Date): Promise { - const start = Date.now(); - const windowStart = V06ReconciliationService.resolveWindowStart(since); - const windowEnd = until ?? new Date(); - const sinceISO = windowStart.toISOString(); - const untilISO = windowEnd.toISOString(); - logger.info(`[V06Reconciliation] Starting fee-only reconciliation (since ${sinceISO} until ${untilISO})`); - - await Promise.all([ - this.reconcileFeeDailySpot(sinceISO, untilISO), - this.reconcileFeeDailyConditional(sinceISO, untilISO), - ]); - await this.reconcileFeeDailyAggregate(windowStart); - - const elapsed = ((Date.now() - start) / 1000).toFixed(1); - logger.info(`[V06Reconciliation] Fee-only cycle complete in ${elapsed}s`); - } - - // ------------------------------------------------------------------ - // 1) Spot OHLCV 1-minute - // ------------------------------------------------------------------ - - private async reconcileSpotOhlcv1m(since: string, until: string): Promise { - // #12: Single CTE query replaces 3 separate scans (aggregate + open + close). - // #10: Compare unix_timestamp directly against epoch seconds to allow index use. - const sinceEpoch = Math.floor(new Date(since).getTime() / 1000); - const untilEpoch = Math.floor(new Date(until).getTime() / 1000); - - logger.info('[V06Reconciliation] Spot OHLCV 1m: querying (single CTE)...'); - const rows = await this.extDb.query(` - WITH priced AS ( - SELECT - d.base_mint_acct AS token, - date_trunc('minute', to_timestamp(s.unix_timestamp)) AS bucket, - s.unix_timestamp, - s.id, - CASE WHEN LOWER(TRIM(s.swap_type)) = 'buy' - THEN s.input_amount::numeric / NULLIF(s.output_amount::numeric, 0) - ELSE s.output_amount::numeric / NULLIF(s.input_amount::numeric, 0) - END AS price, - CASE WHEN LOWER(TRIM(s.swap_type)) = 'buy' THEN s.output_amount::numeric ELSE s.input_amount::numeric END AS base_amt, - CASE WHEN LOWER(TRIM(s.swap_type)) = 'buy' THEN s.input_amount::numeric ELSE s.output_amount::numeric END AS target_amt, - CASE WHEN LOWER(TRIM(s.swap_type)) = 'buy' THEN s.input_amount::numeric ELSE 0 END AS buy_amt, - CASE WHEN LOWER(TRIM(s.swap_type)) = 'sell' THEN s.input_amount::numeric ELSE 0 END AS sell_amt - FROM v0_6_spot_swaps s - JOIN v0_6_daos d ON d.dao_addr = s.dao_addr - WHERE s.unix_timestamp >= $1 - AND s.unix_timestamp < $2 - AND s.input_amount > 0 AND s.output_amount > 0 - AND LOWER(TRIM(s.swap_type)) IN ('buy', 'sell') - ), - ranked AS ( - SELECT *, - ROW_NUMBER() OVER (PARTITION BY token, bucket ORDER BY unix_timestamp ASC, id ASC) AS rn_first, - ROW_NUMBER() OVER (PARTITION BY token, bucket ORDER BY unix_timestamp DESC, id DESC) AS rn_last - FROM priced - ) - SELECT - token, - bucket, - MAX(CASE WHEN rn_first = 1 THEN price END) AS open, - MAX(price) AS high, - MIN(price) AS low, - MAX(CASE WHEN rn_last = 1 THEN price END) AS close, - AVG(price) AS average_price, - SUM(base_amt) AS base_volume, - SUM(target_amt) AS target_volume, - SUM(buy_amt) AS buy_volume, - SUM(sell_amt) AS sell_volume, - COUNT(*)::int AS trade_count - FROM ranked - GROUP BY token, bucket - `, [sinceEpoch, untilEpoch]); - logger.info(`[V06Reconciliation] Spot OHLCV 1m: got ${rows.rows.length} rows`); - - if (rows.rows.length === 0) { - logger.info('[V06Reconciliation] Spot OHLCV 1m: no rows'); - return; - } - - // Batch upsert into app DB using multi-value INSERT - const pool = this.appDb.pool; - if (!pool) return; - - const BATCH_SIZE = 500; - const client = await pool.connect(); - try { - await client.query('BEGIN'); - - for (let i = 0; i < rows.rows.length; i += BATCH_SIZE) { - const batch = rows.rows.slice(i, i + BATCH_SIZE); - const values: any[] = []; - const placeholders: string[] = []; - - batch.forEach((row: any, idx: number) => { - const offset = idx * 12; - placeholders.push( - `($${offset + 1}, $${offset + 2}, $${offset + 3}, $${offset + 4}, $${offset + 5}, $${offset + 6}, $${offset + 7}, $${offset + 8}, $${offset + 9}, $${offset + 10}, $${offset + 11}, $${offset + 12}, true, CURRENT_TIMESTAMP)` - ); - values.push( - row.token, row.bucket, - Number(row.open), Number(row.high), Number(row.low), Number(row.close), - Number(row.average_price), - Number(row.base_volume), Number(row.target_volume), - Number(row.buy_volume), Number(row.sell_volume), - row.trade_count, - ); - }); - - await client.query(` - INSERT INTO v06_spot_ohlcv_1m - (token, bucket, open, high, low, close, average_price, - base_volume, target_volume, buy_volume, sell_volume, trade_count, is_complete, updated_at) - VALUES ${placeholders.join(', ')} - ON CONFLICT (token, bucket) DO UPDATE SET - open = EXCLUDED.open, - high = EXCLUDED.high, - low = EXCLUDED.low, - close = EXCLUDED.close, - average_price = EXCLUDED.average_price, - base_volume = EXCLUDED.base_volume, - target_volume = EXCLUDED.target_volume, - buy_volume = EXCLUDED.buy_volume, - sell_volume = EXCLUDED.sell_volume, - trade_count = EXCLUDED.trade_count, - is_complete = true, - updated_at = CURRENT_TIMESTAMP - `, values); - } - - await client.query('COMMIT'); - logger.info(`[V06Reconciliation] Spot OHLCV 1m: upserted ${rows.rows.length} buckets`); - } catch (err) { - await client.query('ROLLBACK'); - throw err; - } finally { - client.release(); - } - } - - // ------------------------------------------------------------------ - // 2) Spot OHLCV daily — rollup from 1m in app DB - // ------------------------------------------------------------------ - - private async reconcileSpotOhlcv1d(since: Date): Promise { - const pool = this.appDb.pool; - if (!pool) return; - - const dayStart = new Date(since); - dayStart.setUTCHours(0, 0, 0, 0); - - await pool.query(` - INSERT INTO v06_spot_ohlcv_1d - (token, bucket, open, high, low, close, average_price, - base_volume, target_volume, buy_volume, sell_volume, trade_count, is_complete, updated_at) - SELECT - token, - date_trunc('day', bucket) AS bucket, - (array_agg(open ORDER BY bucket ASC))[1] AS open, - MAX(high) AS high, - MIN(CASE WHEN low > 0 THEN low END) AS low, - (array_agg(close ORDER BY bucket DESC))[1] AS close, - CASE WHEN SUM(base_volume) > 0 - THEN SUM(average_price * base_volume) / SUM(base_volume) - ELSE 0 END AS average_price, - SUM(base_volume) AS base_volume, - SUM(target_volume) AS target_volume, - SUM(buy_volume) AS buy_volume, - SUM(sell_volume) AS sell_volume, - SUM(trade_count) AS trade_count, - -- A day is complete only if it's in the past - CASE WHEN date_trunc('day', bucket) < date_trunc('day', NOW()) THEN true ELSE false END, - CURRENT_TIMESTAMP - FROM v06_spot_ohlcv_1m - WHERE bucket >= $1 - GROUP BY token, date_trunc('day', bucket) - ON CONFLICT (token, bucket) DO UPDATE SET - open = EXCLUDED.open, - high = EXCLUDED.high, - low = EXCLUDED.low, - close = EXCLUDED.close, - average_price = EXCLUDED.average_price, - base_volume = EXCLUDED.base_volume, - target_volume = EXCLUDED.target_volume, - buy_volume = EXCLUDED.buy_volume, - sell_volume = EXCLUDED.sell_volume, - trade_count = EXCLUDED.trade_count, - is_complete = EXCLUDED.is_complete, - updated_at = CURRENT_TIMESTAMP - `, [dayStart]); - - logger.info('[V06Reconciliation] Spot OHLCV 1d: rollup complete'); - } - - // ------------------------------------------------------------------ - // 3) Fee volume daily — spot - // ------------------------------------------------------------------ - - private async reconcileFeeDailySpot(since: string, until: string): Promise { - // #10: Use epoch-second comparison for index-friendly WHERE. - // #11: Compute fee columns in SQL to avoid JS numeric precision loss. - const sinceEpoch = Math.floor(new Date(since).getTime() / 1000); - const untilEpoch = Math.floor(new Date(until).getTime() / 1000); - - logger.info('[V06Reconciliation] Fee daily spot: querying...'); - // `AT TIME ZONE 'UTC'` keeps bucketing independent of pg session timezone. - const rows = await this.extDb.query(` - SELECT - d.base_mint_acct AS token, - (to_timestamp(s.unix_timestamp) AT TIME ZONE 'UTC')::date AS swap_date, - SUM(CASE WHEN LOWER(TRIM(s.swap_type)) = 'buy' THEN s.input_amount::numeric ELSE 0 END) AS buy_volume, - SUM(CASE WHEN LOWER(TRIM(s.swap_type)) = 'sell' THEN s.input_amount::numeric ELSE 0 END) AS sell_volume, - SUM(CASE WHEN LOWER(TRIM(s.swap_type)) = 'buy' THEN s.output_amount::numeric ELSE s.input_amount::numeric END) AS base_volume, - SUM(CASE WHEN LOWER(TRIM(s.swap_type)) = 'buy' THEN s.input_amount::numeric ELSE s.output_amount::numeric END) AS target_volume, - COUNT(*)::int AS trade_count, - SUM(CASE WHEN LOWER(TRIM(s.swap_type)) = 'sell' THEN s.output_amount::numeric ELSE 0 END) AS sell_output_usdc - FROM v0_6_spot_swaps s - JOIN v0_6_daos d ON d.dao_addr = s.dao_addr - WHERE s.unix_timestamp >= $1 - AND s.unix_timestamp < $2 - AND s.input_amount > 0 AND s.output_amount > 0 - AND LOWER(TRIM(s.swap_type)) IN ('buy', 'sell') - GROUP BY d.base_mint_acct, (to_timestamp(s.unix_timestamp) AT TIME ZONE 'UTC')::date - `, [sinceEpoch, untilEpoch]); - - if (rows.rows.length === 0) { - logger.info('[V06Reconciliation] Fee daily spot: no rows'); - return; - } - - const pool = this.appDb.pool; - if (!pool) return; - - const BATCH_SIZE = 500; - const client = await pool.connect(); - try { - await client.query('BEGIN'); - - for (let i = 0; i < rows.rows.length; i += BATCH_SIZE) { - const batch = rows.rows.slice(i, i + BATCH_SIZE); - const values: any[] = []; - const placeholders: string[] = []; - - batch.forEach((r: any, idx: number) => { - const dateStr = r.swap_date instanceof Date - ? r.swap_date.toISOString().slice(0, 10) - : String(r.swap_date); - const offset = idx * 8; - // #11: Pass raw volumes + fee rate; compute fees in SQL as NUMERIC arithmetic - placeholders.push( - `($${offset + 1}, $${offset + 2}::date, $${offset + 3}::numeric, $${offset + 4}::numeric, $${offset + 5}::numeric, $${offset + 6}::numeric, $${offset + 7}::int, $${offset + 8}::numeric)` - ); - values.push( - r.token, dateStr, - r.buy_volume, r.sell_volume, - r.base_volume, r.target_volume, - r.trade_count, - r.sell_output_usdc, - ); - }); - - await client.query(` - INSERT INTO v06_fee_volume_daily_spot - (token, date, buy_volume, sell_volume, base_volume, target_volume, trade_count, - usdc_fees, token_fees, token_fees_usdc, sell_volume_usdc, updated_at) - SELECT - v.token, v.date, v.buy_volume, v.sell_volume, v.base_volume, v.target_volume, v.trade_count, - v.buy_volume * ${FEE_RATE}::numeric AS usdc_fees, - v.sell_volume * ${FEE_RATE}::numeric AS token_fees, - v.sell_output_usdc * ${FEE_RATE}::numeric AS token_fees_usdc, - v.sell_output_usdc, - CURRENT_TIMESTAMP - FROM (VALUES ${placeholders.join(', ')}) - AS v(token, date, buy_volume, sell_volume, base_volume, target_volume, trade_count, sell_output_usdc) - ON CONFLICT (token, date) DO UPDATE SET - buy_volume = EXCLUDED.buy_volume, - sell_volume = EXCLUDED.sell_volume, - base_volume = EXCLUDED.base_volume, - target_volume = EXCLUDED.target_volume, - trade_count = EXCLUDED.trade_count, - usdc_fees = EXCLUDED.usdc_fees, - token_fees = EXCLUDED.token_fees, - token_fees_usdc = EXCLUDED.token_fees_usdc, - sell_volume_usdc = EXCLUDED.sell_volume_usdc, - updated_at = CURRENT_TIMESTAMP - `, values); - } - - await client.query('COMMIT'); - logger.info(`[V06Reconciliation] Fee daily spot: upserted ${rows.rows.length} rows`); - } catch (err) { - await client.query('ROLLBACK'); - throw err; - } finally { - client.release(); - } - } - - // ------------------------------------------------------------------ - // 4) Fee volume daily — conditional (winning market only) - // ------------------------------------------------------------------ - - private async reconcileFeeDailyConditional(since: string, until: string): Promise { - // #9: Batch upserts (multi-value INSERT) instead of row-by-row. - // #10: Use epoch-second comparison for index-friendly WHERE. - // #11: Compute fee columns in SQL to avoid JS numeric precision loss. - const sinceEpoch = Math.floor(new Date(since).getTime() / 1000); - const untilEpoch = Math.floor(new Date(until).getTime() / 1000); - - // Only include swaps where: - // - proposal is Passed → keep market = 'pass' - // - proposal is Failed → keep market = 'fail' - logger.info('[V06Reconciliation] Fee daily conditional: querying winning-market swaps...'); - const rows = await this.extDb.query(` - SELECT - d.base_mint_acct AS token, - (to_timestamp(c.unix_timestamp) AT TIME ZONE 'UTC')::date AS swap_date, - SUM(CASE WHEN LOWER(TRIM(c.swap_type)) = 'buy' THEN c.input_amount::numeric ELSE 0 END) AS buy_volume, - SUM(CASE WHEN LOWER(TRIM(c.swap_type)) = 'sell' THEN c.input_amount::numeric ELSE 0 END) AS sell_volume, - SUM(CASE WHEN LOWER(TRIM(c.swap_type)) = 'buy' THEN c.output_amount::numeric ELSE c.input_amount::numeric END) AS base_volume, - SUM(CASE WHEN LOWER(TRIM(c.swap_type)) = 'buy' THEN c.input_amount::numeric ELSE c.output_amount::numeric END) AS target_volume, - COUNT(*)::int AS trade_count, - SUM(CASE WHEN LOWER(TRIM(c.swap_type)) = 'sell' THEN c.output_amount::numeric ELSE 0 END) AS sell_output_usdc - FROM v0_6_conditional_swaps c - JOIN v0_6_proposals p ON p.proposal_addr = c.proposal_addr - JOIN v0_6_daos d ON d.dao_addr = p.dao_addr - WHERE c.unix_timestamp >= $1 - AND c.unix_timestamp < $2 - AND c.input_amount > 0 AND c.output_amount > 0 - AND LOWER(TRIM(c.swap_type)) IN ('buy', 'sell') - AND ( - (p.state = 'Passed' AND LOWER(TRIM(c.market)) = 'pass') - OR - (p.state = 'Failed' AND LOWER(TRIM(c.market)) = 'fail') - ) - GROUP BY d.base_mint_acct, (to_timestamp(c.unix_timestamp) AT TIME ZONE 'UTC')::date - `, [sinceEpoch, untilEpoch]); - - // Also count how many open (non-terminal) proposals have swaps in this window - logger.info('[V06Reconciliation] Fee daily conditional: querying pending proposals...'); - const pendingRows = await this.extDb.query(` - SELECT - d.base_mint_acct AS token, - (to_timestamp(c.unix_timestamp) AT TIME ZONE 'UTC')::date AS swap_date, - COUNT(DISTINCT p.proposal_addr)::int AS pending_count - FROM v0_6_conditional_swaps c - JOIN v0_6_proposals p ON p.proposal_addr = c.proposal_addr - JOIN v0_6_daos d ON d.dao_addr = p.dao_addr - WHERE c.unix_timestamp >= $1 - AND c.unix_timestamp < $2 - AND p.state NOT IN ('Passed', 'Failed') - GROUP BY d.base_mint_acct, (to_timestamp(c.unix_timestamp) AT TIME ZONE 'UTC')::date - `, [sinceEpoch, untilEpoch]); - - // Normalise swap_date coming from pg (Date objects) into YYYY-MM-DD strings - // so that Bun's Date.toString() (which emits un-parseable "GMT-0700") never - // leaks into map keys or query parameters. - const normDate = (d: unknown): string => - d instanceof Date ? d.toISOString().slice(0, 10) : String(d); - - const pendingMap = new Map(); - for (const r of pendingRows.rows) { - pendingMap.set(`${r.token}|${normDate(r.swap_date)}`, r.pending_count); - } - - // Build a lookup map for winning-market rows (avoids O(n×m) find inside loop) - const winRowMap = new Map(); - for (const r of rows.rows) { - winRowMap.set(`${r.token}|${normDate(r.swap_date)}`, r); - } - - const pool = this.appDb.pool; - if (!pool) return; - const client = await pool.connect(); - - try { - await client.query('BEGIN'); - - // Collect all token-date keys that need a row - const allKeys = new Set(); - for (const r of rows.rows) allKeys.add(`${r.token}|${normDate(r.swap_date)}`); - for (const key of pendingMap.keys()) allKeys.add(key); - - // Split into rows that have winning-market data vs pending-only - const winEntries: { token: string; date: string; row: any; pending: number; reconciled: boolean }[] = []; - const pendingOnlyEntries: { token: string; date: string; pending: number }[] = []; - - for (const key of allKeys) { - const separatorIdx = key.indexOf('|'); - const token = key.slice(0, separatorIdx); - const swapDateStr = key.slice(separatorIdx + 1); - const winRow = winRowMap.get(key); - const pendingCount = pendingMap.get(key) ?? 0; - if (winRow) { - winEntries.push({ token, date: swapDateStr, row: winRow, pending: pendingCount, reconciled: pendingCount === 0 }); - } else { - pendingOnlyEntries.push({ token, date: swapDateStr, pending: pendingCount }); - } - } - - // Batch upsert winning-market rows with fee math in SQL - const BATCH_SIZE = 500; - for (let i = 0; i < winEntries.length; i += BATCH_SIZE) { - const batch = winEntries.slice(i, i + BATCH_SIZE); - const values: any[] = []; - const placeholders: string[] = []; - - batch.forEach((entry, idx) => { - const offset = idx * 10; - placeholders.push( - `($${offset + 1}, $${offset + 2}::date, $${offset + 3}::numeric, $${offset + 4}::numeric, $${offset + 5}::numeric, $${offset + 6}::numeric, $${offset + 7}::int, $${offset + 8}::numeric, $${offset + 9}::boolean, $${offset + 10}::int)` - ); - values.push( - entry.token, entry.date, - entry.row.buy_volume, entry.row.sell_volume, - entry.row.base_volume, entry.row.target_volume, - entry.row.trade_count, - entry.row.sell_output_usdc, - entry.reconciled, entry.pending, - ); - }); - - await client.query(` - INSERT INTO v06_fee_volume_daily_conditional - (token, date, buy_volume, sell_volume, base_volume, target_volume, trade_count, - usdc_fees, token_fees, token_fees_usdc, sell_volume_usdc, - conditional_reconciled, pending_open_proposals, updated_at) - SELECT - v.token, v.date, v.buy_volume, v.sell_volume, v.base_volume, v.target_volume, v.trade_count, - v.buy_volume * ${FEE_RATE}::numeric AS usdc_fees, - v.sell_volume * ${FEE_RATE}::numeric AS token_fees, - v.sell_output_usdc * ${FEE_RATE}::numeric AS token_fees_usdc, - v.sell_output_usdc, - v.reconciled, v.pending, - CURRENT_TIMESTAMP - FROM (VALUES ${placeholders.join(', ')}) - AS v(token, date, buy_volume, sell_volume, base_volume, target_volume, trade_count, sell_output_usdc, reconciled, pending) - ON CONFLICT (token, date) DO UPDATE SET - buy_volume = EXCLUDED.buy_volume, - sell_volume = EXCLUDED.sell_volume, - base_volume = EXCLUDED.base_volume, - target_volume = EXCLUDED.target_volume, - trade_count = EXCLUDED.trade_count, - usdc_fees = EXCLUDED.usdc_fees, - token_fees = EXCLUDED.token_fees, - token_fees_usdc = EXCLUDED.token_fees_usdc, - sell_volume_usdc = EXCLUDED.sell_volume_usdc, - conditional_reconciled = EXCLUDED.conditional_reconciled, - pending_open_proposals = EXCLUDED.pending_open_proposals, - updated_at = CURRENT_TIMESTAMP - `, values); - } - - // Batch upsert pending-only rows (no winning-market volume yet) - for (let i = 0; i < pendingOnlyEntries.length; i += BATCH_SIZE) { - const batch = pendingOnlyEntries.slice(i, i + BATCH_SIZE); - const values: any[] = []; - const placeholders: string[] = []; - - batch.forEach((entry, idx) => { - const offset = idx * 3; - placeholders.push(`($${offset + 1}, $${offset + 2}::date, $${offset + 3}::int)`); - values.push(entry.token, entry.date, entry.pending); - }); - - await client.query(` - INSERT INTO v06_fee_volume_daily_conditional - (token, date, conditional_reconciled, pending_open_proposals, updated_at) - SELECT v.token, v.date, false, v.pending, CURRENT_TIMESTAMP - FROM (VALUES ${placeholders.join(', ')}) - AS v(token, date, pending) - ON CONFLICT (token, date) DO UPDATE SET - conditional_reconciled = false, - pending_open_proposals = EXCLUDED.pending_open_proposals, - updated_at = CURRENT_TIMESTAMP - `, values); - } - - await client.query('COMMIT'); - logger.info(`[V06Reconciliation] Fee daily conditional: upserted ${allKeys.size} rows`); - } catch (err) { - await client.query('ROLLBACK'); - throw err; - } finally { - client.release(); - } - } - - // ------------------------------------------------------------------ - // 5) Fee volume daily aggregate — join spot + conditional from app DB - // ------------------------------------------------------------------ - - private async reconcileFeeDailyAggregate(since: Date): Promise { - const pool = this.appDb.pool; - if (!pool) return; - - const dayStart = new Date(since); - dayStart.setUTCHours(0, 0, 0, 0); - - await pool.query(` - INSERT INTO v06_fee_volume_daily_aggregate - (token, date, - spot_buy_volume, spot_sell_volume, spot_base_volume, spot_target_volume, - spot_trade_count, spot_usdc_fees, spot_token_fees, spot_token_fees_usdc, - conditional_buy_volume, conditional_sell_volume, conditional_base_volume, conditional_target_volume, - conditional_trade_count, conditional_usdc_fees, conditional_token_fees, conditional_token_fees_usdc, - total_buy_volume, total_sell_volume, total_base_volume, total_target_volume, - total_trade_count, total_usdc_fees, total_token_fees, total_token_fees_usdc, - conditional_reconciled, pending_open_proposals, updated_at) - SELECT - COALESCE(s.token, c.token) AS token, - COALESCE(s.date, c.date) AS date, - -- spot - COALESCE(s.buy_volume, 0), - COALESCE(s.sell_volume, 0), - COALESCE(s.base_volume, 0), - COALESCE(s.target_volume, 0), - COALESCE(s.trade_count, 0), - COALESCE(s.usdc_fees, 0), - COALESCE(s.token_fees, 0), - COALESCE(s.token_fees_usdc, 0), - -- conditional (nullable) - c.buy_volume, - c.sell_volume, - c.base_volume, - c.target_volume, - c.trade_count, - c.usdc_fees, - c.token_fees, - c.token_fees_usdc, - -- totals - COALESCE(s.buy_volume, 0) + COALESCE(c.buy_volume, 0), - COALESCE(s.sell_volume, 0) + COALESCE(c.sell_volume, 0), - COALESCE(s.base_volume, 0) + COALESCE(c.base_volume, 0), - COALESCE(s.target_volume, 0) + COALESCE(c.target_volume, 0), - COALESCE(s.trade_count, 0) + COALESCE(c.trade_count, 0), - COALESCE(s.usdc_fees, 0) + COALESCE(c.usdc_fees, 0), - COALESCE(s.token_fees, 0) + COALESCE(c.token_fees, 0), - COALESCE(s.token_fees_usdc, 0) + COALESCE(c.token_fees_usdc, 0), - -- reconciliation - COALESCE(c.conditional_reconciled, true), - COALESCE(c.pending_open_proposals, 0), - CURRENT_TIMESTAMP - FROM v06_fee_volume_daily_spot s - FULL OUTER JOIN v06_fee_volume_daily_conditional c - ON s.token = c.token AND s.date = c.date - WHERE COALESCE(s.date, c.date) >= $1::date - ON CONFLICT (token, date) DO UPDATE SET - spot_buy_volume = EXCLUDED.spot_buy_volume, - spot_sell_volume = EXCLUDED.spot_sell_volume, - spot_base_volume = EXCLUDED.spot_base_volume, - spot_target_volume = EXCLUDED.spot_target_volume, - spot_trade_count = EXCLUDED.spot_trade_count, - spot_usdc_fees = EXCLUDED.spot_usdc_fees, - spot_token_fees = EXCLUDED.spot_token_fees, - spot_token_fees_usdc = EXCLUDED.spot_token_fees_usdc, - conditional_buy_volume = EXCLUDED.conditional_buy_volume, - conditional_sell_volume = EXCLUDED.conditional_sell_volume, - conditional_base_volume = EXCLUDED.conditional_base_volume, - conditional_target_volume = EXCLUDED.conditional_target_volume, - conditional_trade_count = EXCLUDED.conditional_trade_count, - conditional_usdc_fees = EXCLUDED.conditional_usdc_fees, - conditional_token_fees = EXCLUDED.conditional_token_fees, - conditional_token_fees_usdc = EXCLUDED.conditional_token_fees_usdc, - total_buy_volume = EXCLUDED.total_buy_volume, - total_sell_volume = EXCLUDED.total_sell_volume, - total_base_volume = EXCLUDED.total_base_volume, - total_target_volume = EXCLUDED.total_target_volume, - total_trade_count = EXCLUDED.total_trade_count, - total_usdc_fees = EXCLUDED.total_usdc_fees, - total_token_fees = EXCLUDED.total_token_fees, - total_token_fees_usdc = EXCLUDED.total_token_fees_usdc, - conditional_reconciled = EXCLUDED.conditional_reconciled, - pending_open_proposals = EXCLUDED.pending_open_proposals, - updated_at = CURRENT_TIMESTAMP - `, [dayStart]); - - logger.info('[V06Reconciliation] Fee daily aggregate: upsert complete'); - } -} diff --git a/tests/api.test.ts b/tests/api.test.ts index d368b71..2102d65 100644 --- a/tests/api.test.ts +++ b/tests/api.test.ts @@ -78,7 +78,6 @@ function createMockServices(): Services { externalDatabaseService: mockExternalDatabaseService, solanaService: undefined, launchpadService: undefined, - v06ReconciliationService: null, }; } diff --git a/tests/helpers/testApp.ts b/tests/helpers/testApp.ts index 1cc6529..f93695c 100644 --- a/tests/helpers/testApp.ts +++ b/tests/helpers/testApp.ts @@ -75,7 +75,6 @@ export function createTestServices(overrides?: Partial): Services { externalDatabaseService: createMockExternalDatabaseService(), solanaService: createMockSolanaService(), launchpadService: createMockLaunchpadService(), - v06ReconciliationService: null, ...overrides, }; } diff --git a/tests/routes/root.test.ts b/tests/routes/root.test.ts index 3eb7fb4..a759ba7 100644 --- a/tests/routes/root.test.ts +++ b/tests/routes/root.test.ts @@ -46,12 +46,13 @@ describe('Root Routes', () => { expect(response.body.supplyBreakdown).toHaveProperty('circulatingSupply'); }); - it('should include caching information', async () => { + it('describes itself as a read-only, Dune-free API', async () => { const response = await request(app).get('/'); - - expect(response.body).toHaveProperty('caching'); - expect(response.body.caching).toHaveProperty('description'); - expect(response.body.caching).toHaveProperty('refreshInterval'); + + // The stale Dune-cache block was removed; root now carries a `note` and no caching/Dune cruft. + expect(response.body).toHaveProperty('note'); + expect(response.body.note).toContain('no Dune'); + expect(response.body).not.toHaveProperty('caching'); }); }); }); diff --git a/tests/runtime/services.test.ts b/tests/runtime/services.test.ts index aea0194..2dc36fe 100644 --- a/tests/runtime/services.test.ts +++ b/tests/runtime/services.test.ts @@ -7,12 +7,5 @@ describe('runtime service composition', () => { expect(services.databaseService).toBeTruthy(); expect(services.externalDatabaseService).toBeTruthy(); - expect(services.v06ReconciliationService).toBeNull(); - }); - - it('creates indexer services for reconciliation', () => { - const services = createServices('indexer'); - - expect(services.v06ReconciliationService).toBeTruthy(); }); }); From 15fe27271f90e7b8c508b9f55a490789f7f9d2c5 Mon Sep 17 00:00:00 2001 From: Jinglun Date: Mon, 8 Jun 2026 15:03:29 -0700 Subject: [PATCH 06/14] refactor(external-api): serve all market data from the user_pool ETL output MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every market-data read now comes from the unified user_pool ETL tables in the served DB — no app-DB v06 rollups, no futarchy.trades, no raw-swap aggregation: - /api/market-data futarchy → user_pool_daily (getFutarchyAmmDailyActivity); already wired. meteora → user_pool_daily (source='meteora') instead of the meteora_daily view (validated row-exact, 0 mismatches over 1082 rows; token_per_usdc derived = base_volume/target_volume = 1/average_price). - /api/tickers 24h metrics → user_pool_spot_ohlcv (1m candles, source= 'futarchy_amm'), keyed by token → dao. Drops BOTH the futarchy.trades primary and the dead app-DB v06_spot_ohlcv_1m fallback. Values are human units (no /1e6). - getFirstTradeDates → user_pool_daily MIN(date) instead of raw v0_6_spot_swaps. - Remove dead getDailyTradingActivity + getV06Rolling24hMetrics and their repos (v06TradingActivityRepo, dailyVolumesRepo). dexscreener stays on raw v0_6_spot_swaps by design — it's a per-swap event + post-swap-reserves (amm_base/quote_amount) adapter, data the aggregated ETL doesn't carry. Tests: add getFutarchyAmmDailyActivity to the mock; rewrite market.test with a seeded happy-path asserting the pivoted spot/conditional/total + protocol/LP split and meteora values (was a vacuous 200/500/503 check); repoint the tickers test to token-keyed user_pool_spot_ohlcv; drop the obsolete fallback test. 111 pass / 0 fail, typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/routes/coingecko.ts | 55 +---- src/routes/market.ts | 41 ++-- .../internal/repos/dailyVolumesRepo.ts | 62 ------ .../internal/repos/v06TradingActivityRepo.ts | 125 ----------- src/services/databaseService.ts | 55 ----- src/services/externalDatabaseService.ts | 206 +++++++++++++----- tests/api.test.ts | 32 +-- tests/helpers/testApp.ts | 2 +- tests/routes/market.test.ts | 148 +++++++------ 9 files changed, 271 insertions(+), 455 deletions(-) delete mode 100644 src/services/database/internal/repos/dailyVolumesRepo.ts delete mode 100644 src/services/database/internal/repos/v06TradingActivityRepo.ts diff --git a/src/routes/coingecko.ts b/src/routes/coingecko.ts index f4ed7a9..684eb76 100644 --- a/src/routes/coingecko.ts +++ b/src/routes/coingecko.ts @@ -7,13 +7,12 @@ import { sendAlert } from '../utils/alerts.js'; export function createCoinGeckoRouter(services: ServiceGetters): Router { const router = Router(); - const { getFutarchyService, getPriceService, getDatabaseService, getExternalDatabaseService } = services; + const { getFutarchyService, getPriceService, getExternalDatabaseService } = services; // CoinGecko Endpoint: /tickers router.get('/api/tickers', asyncHandler(async (req: Request, res: Response) => { const futarchyService = getFutarchyService(); const priceService = getPriceService(); - const databaseService = getDatabaseService(); const externalDatabaseService = getExternalDatabaseService(); const allDaos = await futarchyService.getAllDaos(); @@ -28,15 +27,17 @@ export function createCoinGeckoRouter(services: ServiceGetters): Router { } const volumeMetricsMap = new Map(); - let volumeSource = 'none'; - // Primary: read rolling-24h spot metrics straight from the indexer DB - // (futarchy.trades, keyed by dao_addr). No Dune, no app-DB rollup. + // Single source: rolling-24h spot metrics from the unified user_pool ETL + // candles (user_pool_spot_ohlcv via the served DB), keyed by token (base + // mint) → mapped to dao (= pool_id). No futarchy.trades, no app-DB fallback. if (externalDatabaseService?.isAvailable()) { - const daoAddresses = allDaos.map(dao => dao.daoAddress.toString()); - const spotMetrics = await externalDatabaseService.getSpotRolling24hMetrics(daoAddresses); + const baseMints = allDaos.map(dao => dao.baseMint.toString()); + const spotMetrics = await externalDatabaseService.getSpotRolling24hMetrics(baseMints); - for (const [daoAddress, metrics] of spotMetrics.entries()) { + for (const [token, metrics] of spotMetrics.entries()) { + const daoAddress = tokenToDaoMap.get(token); + if (!daoAddress) continue; volumeMetricsMap.set(daoAddress, { base_volume_24h: metrics.base_volume_24h, target_volume_24h: metrics.target_volume_24h, @@ -44,50 +45,16 @@ export function createCoinGeckoRouter(services: ServiceGetters): Router { low_24h: metrics.low_24h, }); } - - if (volumeMetricsMap.size > 0) { - volumeSource = 'futarchy-trades-db'; - logger.debug('Using indexer futarchy.trades rolling 24h metrics', { daoCount: volumeMetricsMap.size, requestId: req.requestId }); - } - } - - // Fallback: v0.6 indexer OHLCV (app DB) for ONLY the DAOs the primary - // (futarchy.trades) didn't cover — per-DAO merge, not all-or-nothing. This - // covers the window before futarchy.trades is populated for a given DAO - // without zeroing out the DAOs that the primary did return. - const missingDaos = allDaos.filter(dao => !volumeMetricsMap.has(dao.daoAddress.toString())); - if (missingDaos.length > 0 && databaseService?.isAvailable()) { - const missingBaseMints = missingDaos.map(dao => dao.baseMint.toString()); - const v06Metrics = await databaseService.getV06Rolling24hMetrics(missingBaseMints); - - let filled = 0; - for (const [tokenAddress, metrics] of v06Metrics.entries()) { - const daoAddress = tokenToDaoMap.get(tokenAddress); - if (daoAddress && !volumeMetricsMap.has(daoAddress)) { - volumeMetricsMap.set(daoAddress, { - base_volume_24h: metrics.base_volume_24h, - target_volume_24h: metrics.target_volume_24h, - high_24h: metrics.high_24h, - low_24h: metrics.low_24h, - }); - filled++; - } - } - - if (filled > 0) { - volumeSource = volumeSource === 'futarchy-trades-db' ? 'futarchy-trades-db+v06-fallback' : 'v06-indexer-fallback'; - logger.debug('Filled missing DAOs from v0.6 indexer rolling 24h metrics', { filled, requestId: req.requestId }); - } } if (volumeMetricsMap.size === 0) { logger.warn('No volume metrics available', { requestId: req.requestId }); sendAlert( - 'No volume metrics available — all sources returned empty', + 'No volume metrics available — user_pool_spot_ohlcv returned empty', { cooldownKey: 'no-volume-data', cooldownMs: 10 * 60 * 1000 } ); } else { - logger.debug('Volume source selected', { volumeSource, daoCount: volumeMetricsMap.size, requestId: req.requestId }); + logger.debug('Volume source: user_pool ETL (user_pool_spot_ohlcv)', { daoCount: volumeMetricsMap.size, requestId: req.requestId }); } const tickers: CoinGeckoTicker[] = []; diff --git a/src/routes/market.ts b/src/routes/market.ts index b336bbf..de1a2f3 100644 --- a/src/routes/market.ts +++ b/src/routes/market.ts @@ -4,17 +4,21 @@ import type { ServiceGetters } from './types.js'; export function createMarketRouter(services: ServiceGetters): Router { const router = Router(); - const { getDatabaseService, getExternalDatabaseService } = services; + const { getExternalDatabaseService } = services; - // Get daily market data with date range and optional token filtering. - // FutarchyAMM rows are sourced from the v0.6 indexer aggregate table. + // Daily market data with date range + optional token filtering. + // BOTH FutarchyAMM and Meteora are served from the unified, on-chain-derived ETL in + // the served DB (futarchy.user_pool_daily / futarchy.meteora_daily), via externalDatabase. + // FutarchyAMM no longer reads the flat-0.5% app-DB v06_fee_volume_daily_aggregate. router.get('/api/market-data', async (req: Request, res: Response) => { - const databaseService = getDatabaseService(); - - if (!databaseService || !databaseService.isAvailable()) { + // The served DB is the source of truth for market data; surface its absence/failure + // rather than masking it as empty (a financial feed must never read a DB outage as + // "zero volume"). + const externalDatabaseService = getExternalDatabaseService(); + if (!externalDatabaseService || !externalDatabaseService.isAvailable()) { return res.status(503).json({ - error: 'Database not available', - message: 'Service is initializing or database is not connected', + error: 'Served database not available', + message: 'Market data source (served indexer DB) is not connected', }); } @@ -23,12 +27,12 @@ export function createMarketRouter(services: ServiceGetters): Router { if (!startDateResult.success) { return res.status(400).json(startDateResult.error); } - + const endDateResult = parseDateParam(req.query.endDate as string, 'endDate', { required: true }); if (!endDateResult.success) { return res.status(400).json(endDateResult.error); } - + // Validate tokens list const tokensResult = parseCommaSeparatedList(req.query.tokens as string, 'tokens'); if (!tokensResult.success) { @@ -42,20 +46,8 @@ export function createMarketRouter(services: ServiceGetters): Router { endDate: endDateResult.value!, }; - // Meteora rows are served directly from our meteora accounting ETL - // (futarchy.meteora_daily in the served DB, via externalDatabase). The served DB is a - // hard dependency for Meteora — surface its absence/failure rather than masking it as - // empty (a financial feed must never read a DB outage as "zero volume"). - const externalDatabaseService = getExternalDatabaseService(); - if (!externalDatabaseService || !externalDatabaseService.isAvailable()) { - return res.status(503).json({ - error: 'Served database not available', - message: 'Meteora data source (served indexer DB) is not connected', - }); - } - const [futarchyData, meteoraData] = await Promise.all([ - databaseService.getDailyTradingActivity(queryOptions), + externalDatabaseService.getFutarchyAmmDailyActivity(queryOptions), externalDatabaseService.getDailyMeteoraVolumes(queryOptions), ]); @@ -65,8 +57,9 @@ export function createMarketRouter(services: ServiceGetters): Router { startDate: startDateResult.value, endDate: endDateResult.value, }, - source: 'v06-indexer', + source: 'user-pool-etl', futarchyAMM: { + source: 'etl-user-pool-daily', count: futarchyData.length, data: futarchyData, }, diff --git a/src/services/database/internal/repos/dailyVolumesRepo.ts b/src/services/database/internal/repos/dailyVolumesRepo.ts deleted file mode 100644 index 74a69fc..0000000 --- a/src/services/database/internal/repos/dailyVolumesRepo.ts +++ /dev/null @@ -1,62 +0,0 @@ -import { logger } from '../../../../utils/logger.js'; -import type { DbRuntime } from '../dbRuntime.js'; -import type { Rolling24hMetrics } from '../../../databaseService.js'; - -export function createDailyVolumesRepo(db: DbRuntime) { - return { - /** - * Get rolling 24h metrics from v06_spot_ohlcv_1m table. - * Sources FutarchyAMM rolling-24h volume from the v0.6 indexer. - * Amounts in this table are raw integers (6 decimals) — caller divides by 1e6. - */ - async getV06Rolling24hMetrics(tokens?: string[]): Promise> { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return new Map(); - - try { - const cutoffTime = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); - - let whereClause = 'WHERE bucket >= $1'; - let params: any[] = [cutoffTime]; - - if (tokens && tokens.length > 0) { - const placeholders = tokens.map((_, i) => `$${i + 2}`).join(', '); - whereClause += ` AND token IN (${placeholders})`; - params = [cutoffTime, ...tokens]; - } - - const result = await pool.query( - `SELECT - token, - (SUM(base_volume) / 1e6)::text AS base_volume_24h, - (SUM(target_volume) / 1e6)::text AS target_volume_24h, - MAX(high)::text AS high_24h, - MIN(CASE WHEN low > 0 THEN low END)::text AS low_24h, - SUM(trade_count)::int AS trade_count_24h - FROM v06_spot_ohlcv_1m - ${whereClause} - GROUP BY token`, - params - ); - - const metricsMap = new Map(); - for (const row of result.rows) { - metricsMap.set(row.token, { - token: row.token, - base_volume_24h: row.base_volume_24h || '0', - target_volume_24h: row.target_volume_24h || '0', - high_24h: row.high_24h || '0', - low_24h: row.low_24h || '0', - trade_count_24h: row.trade_count_24h || 0, - }); - } - return metricsMap; - } catch (error: any) { - logger.error('[Database] Error getting v0.6 rolling 24h metrics:', error); - return new Map(); - } - }, - }; -} - -export type DailyVolumesRepo = ReturnType; diff --git a/src/services/database/internal/repos/v06TradingActivityRepo.ts b/src/services/database/internal/repos/v06TradingActivityRepo.ts deleted file mode 100644 index a11945d..0000000 --- a/src/services/database/internal/repos/v06TradingActivityRepo.ts +++ /dev/null @@ -1,125 +0,0 @@ -import { logger } from '../../../../utils/logger.js'; -import type { DbRuntime } from '../dbRuntime.js'; - -export function createV06TradingActivityRepo(db: DbRuntime) { - return { - /** - * Get daily trading activity from the v06_fee_volume_daily_aggregate table. - * Returns spot + conditional breakdown with a has_conditional_volume flag per row. - */ - async getDailyTradingActivity(options?: { - token?: string; - tokens?: string[]; - startDate?: string; - endDate?: string; - }): Promise<{ - token: string; - date: string; - has_conditional_volume: boolean; - spot_buy_volume: string; - spot_sell_volume: string; - spot_base_volume: string; - spot_target_volume: string; - spot_trade_count: number; - spot_usdc_fees: string; - spot_token_fees: string; - spot_token_fees_usdc: string; - conditional_buy_volume: string | null; - conditional_sell_volume: string | null; - conditional_base_volume: string | null; - conditional_target_volume: string | null; - conditional_trade_count: number | null; - conditional_usdc_fees: string | null; - conditional_token_fees: string | null; - conditional_token_fees_usdc: string | null; - total_buy_volume: string; - total_sell_volume: string; - total_base_volume: string; - total_target_volume: string; - total_trade_count: number; - total_usdc_fees: string; - total_token_fees: string; - total_token_fees_usdc: string; - conditional_reconciled: boolean; - pending_open_proposals: number; - }[]> { - const pool = db.getPool(); - if (!pool || !db.isConnected()) return []; - - try { - const conditions: string[] = []; - const params: any[] = []; - let paramIndex = 1; - - if (options?.tokens && options.tokens.length > 0) { - const placeholders = options.tokens.map((_, i) => `$${paramIndex + i}`).join(', '); - conditions.push(`token IN (${placeholders})`); - params.push(...options.tokens); - paramIndex += options.tokens.length; - } else if (options?.token) { - conditions.push(`token = $${paramIndex}`); - params.push(options.token); - paramIndex++; - } - - if (options?.startDate) { - conditions.push(`date >= $${paramIndex}`); - params.push(options.startDate); - paramIndex++; - } - - if (options?.endDate) { - conditions.push(`date <= $${paramIndex}`); - params.push(options.endDate); - paramIndex++; - } - - const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(' AND ')}` : ''; - - const result = await pool.query( - `SELECT - token, - date::text, - (conditional_buy_volume IS NOT NULL) AS has_conditional_volume, - (spot_buy_volume / 1e6)::text AS spot_buy_volume, - (spot_sell_volume / 1e6)::text AS spot_sell_volume, - (spot_base_volume / 1e6)::text AS spot_base_volume, - (spot_target_volume / 1e6)::text AS spot_target_volume, - spot_trade_count, - (spot_usdc_fees / 1e6)::text AS spot_usdc_fees, - (spot_token_fees / 1e6)::text AS spot_token_fees, - (spot_token_fees_usdc / 1e6)::text AS spot_token_fees_usdc, - (conditional_buy_volume / 1e6)::text AS conditional_buy_volume, - (conditional_sell_volume / 1e6)::text AS conditional_sell_volume, - (conditional_base_volume / 1e6)::text AS conditional_base_volume, - (conditional_target_volume / 1e6)::text AS conditional_target_volume, - conditional_trade_count, - (conditional_usdc_fees / 1e6)::text AS conditional_usdc_fees, - (conditional_token_fees / 1e6)::text AS conditional_token_fees, - (conditional_token_fees_usdc / 1e6)::text AS conditional_token_fees_usdc, - (total_buy_volume / 1e6)::text AS total_buy_volume, - (total_sell_volume / 1e6)::text AS total_sell_volume, - (total_base_volume / 1e6)::text AS total_base_volume, - (total_target_volume / 1e6)::text AS total_target_volume, - total_trade_count, - (total_usdc_fees / 1e6)::text AS total_usdc_fees, - (total_token_fees / 1e6)::text AS total_token_fees, - (total_token_fees_usdc / 1e6)::text AS total_token_fees_usdc, - conditional_reconciled, - pending_open_proposals - FROM v06_fee_volume_daily_aggregate - ${whereClause} - ORDER BY token, date ASC`, - params - ); - - return result.rows; - } catch (error: any) { - logger.error('[Database] Error getting daily trading activity:', error); - return []; - } - }, - }; -} - -export type V06TradingActivityRepo = ReturnType; diff --git a/src/services/databaseService.ts b/src/services/databaseService.ts index 0aaa3a6..f2b9d81 100644 --- a/src/services/databaseService.ts +++ b/src/services/databaseService.ts @@ -4,9 +4,7 @@ import { logger } from '../utils/logger.js'; import { sendAlert } from '../utils/alerts.js'; import type { DbRuntime } from './database/internal/dbRuntime.js'; import { createSchemaManager } from './database/internal/schemaManager.js'; -import { createDailyVolumesRepo } from './database/internal/repos/dailyVolumesRepo.js'; import { createMetricsRepo } from './database/internal/repos/metricsRepo.js'; -import { createV06TradingActivityRepo } from './database/internal/repos/v06TradingActivityRepo.js'; // Force pg to serialize Date parameters as ISO-8601 UTC strings so PostgreSQL // doesn't receive un-parseable local-timezone names like "GMT-0700". @@ -41,9 +39,7 @@ export class DatabaseService { }; private _schema = createSchemaManager(this.dbRuntime); - private _dailyVolumes = createDailyVolumesRepo(this.dbRuntime); private _metrics = createMetricsRepo(this.dbRuntime); - private _v06TradingActivity = createV06TradingActivityRepo(this.dbRuntime); constructor() { // Only initialize if database config is provided @@ -197,14 +193,6 @@ export class DatabaseService { } } - // ============================================ - // Daily volumes (delegated to DailyVolumesRepo) - // ============================================ - - async getV06Rolling24hMetrics(tokens?: string[]): Promise> { - return this._dailyVolumes.getV06Rolling24hMetrics(tokens); - } - // ============================================ // Metrics (delegated to MetricsRepo) // ============================================ @@ -254,47 +242,4 @@ export class DatabaseService { async pruneOldMetrics(keepDays: number = 30): Promise<{ metricsDeleted: number; healthDeleted: number }> { return this._metrics.pruneOldMetrics(keepDays); } - - // ============================================ - // V06 Trading Activity (delegated to V06TradingActivityRepo) - // ============================================ - - async getDailyTradingActivity(options?: { - token?: string; - tokens?: string[]; - startDate?: string; - endDate?: string; - }): Promise<{ - token: string; - date: string; - has_conditional_volume: boolean; - spot_buy_volume: string; - spot_sell_volume: string; - spot_base_volume: string; - spot_target_volume: string; - spot_trade_count: number; - spot_usdc_fees: string; - spot_token_fees: string; - spot_token_fees_usdc: string; - conditional_buy_volume: string | null; - conditional_sell_volume: string | null; - conditional_base_volume: string | null; - conditional_target_volume: string | null; - conditional_trade_count: number | null; - conditional_usdc_fees: string | null; - conditional_token_fees: string | null; - conditional_token_fees_usdc: string | null; - total_buy_volume: string; - total_sell_volume: string; - total_base_volume: string; - total_target_volume: string; - total_trade_count: number; - total_usdc_fees: string; - total_token_fees: string; - total_token_fees_usdc: string; - conditional_reconciled: boolean; - pending_open_proposals: number; - }[]> { - return this._v06TradingActivity.getDailyTradingActivity(options); - } } diff --git a/src/services/externalDatabaseService.ts b/src/services/externalDatabaseService.ts index 7fd9fe7..3306a20 100644 --- a/src/services/externalDatabaseService.ts +++ b/src/services/externalDatabaseService.ts @@ -10,10 +10,10 @@ pg.defaults.parseInputDatesAsUTC = true; const { Pool } = pg; /** - * Read-only connection pool to the external indexer database. - * Used by v0.6 reconciliation to query v0_6_spot_swaps, - * v0_6_conditional_swaps, v0_6_daos, and v0_6_proposals, and by /api/tickers to - * read rolling-24h spot metrics directly from futarchy.trades. + * Read-only connection pool to the served indexer database. All market-data reads + * come from the unified user_pool ETL output here: user_pool_daily (futarchy + + * meteora daily), user_pool_spot_ohlcv (24h metrics), and user_pool_swaps. Also + * reads the raw v0_6_* decoded tables for dexscreener per-swap event feeds. */ export class ExternalDatabaseService { private pool: pg.Pool | null = null; @@ -58,20 +58,21 @@ export class ExternalDatabaseService { } /** - * Rolling 24h spot-market volume metrics per DAO, read directly from the - * indexer's per-swap futarchy.trades table (no Dune, no app-DB rollup). + * Rolling 24h spot-market metrics per token, aggregated from the unified ETL's + * 1-minute candles (`futarchy.user_pool_spot_ohlcv`, source='futarchy_amm') in + * the served DB. These candles are already in HUMAN units (price = USD/token, + * base_volume = token, target_volume = USD), so no decimal scaling is needed. * - * Keyed by dao_addr (= the ticker's pool_id), so the caller needs no - * token→dao remap. Amounts are raw on-chain integers, so base/quote volume - * are scaled by each side's real decimals (futarchy.tokens.decimals, default - * 6) and price (raw quote/base) is rescaled to human units by - * 10^(baseDecimals − quoteDecimals) — matching the decimal-aware last_price. + * Keyed by token (base mint); the /api/tickers caller maps token→dao. This is + * the SINGLE source for 24h metrics — it replaces both the old futarchy.trades + * read and the app-DB v06_spot_ohlcv_1m fallback. Everything now comes from the + * user_pool ETL output. * - * Returns an empty Map if the connection is down or no spot trades exist in - * the window (caller treats that as "fall back to another source"). + * Returns an empty Map if the connection is down or no spot candles exist in the + * window. Throws on query failure (never masks a failure as empty for a feed). */ - async getSpotRolling24hMetrics(daoAddrs: string[]): Promise> { - if (!this.pool || !this.isConnected || daoAddrs.length === 0) { + async getSpotRolling24hMetrics(tokens: string[]): Promise> { + if (!this.pool || !this.isConnected || tokens.length === 0) { return new Map(); } @@ -80,27 +81,25 @@ export class ExternalDatabaseService { try { const result = await this.pool.query( `SELECT - t.dao_addr, - (SUM(t.base_amount) / power(10::numeric, COALESCE(bt.decimals, 6)))::text AS base_volume_24h, - (SUM(t.quote_amount) / power(10::numeric, COALESCE(qt.decimals, 6)))::text AS target_volume_24h, - (MAX(t.price) * power(10::numeric, COALESCE(bt.decimals, 6) - COALESCE(qt.decimals, 6)))::text AS high_24h, - (MIN(t.price) FILTER (WHERE t.price > 0) * power(10::numeric, COALESCE(bt.decimals, 6) - COALESCE(qt.decimals, 6)))::text AS low_24h, - COUNT(*)::int AS trade_count_24h - FROM futarchy.trades t - JOIN futarchy.daos d ON d.dao_addr = t.dao_addr - LEFT JOIN futarchy.tokens bt ON bt.mint = d.base_mint - LEFT JOIN futarchy.tokens qt ON qt.mint = d.quote_mint - WHERE t.market_kind = 'spot' - AND t.block_time >= $1 - AND t.dao_addr = ANY($2::text[]) - GROUP BY t.dao_addr, bt.decimals, qt.decimals`, - [cutoff, daoAddrs] + token, + SUM(base_volume)::text AS base_volume_24h, + SUM(target_volume)::text AS target_volume_24h, + MAX(high)::text AS high_24h, + (MIN(low) FILTER (WHERE low > 0))::text AS low_24h, + SUM(trade_count)::int AS trade_count_24h + FROM futarchy.user_pool_spot_ohlcv + WHERE source = 'futarchy_amm' + AND "interval" = '1m' + AND bucket_start >= $1 + AND token = ANY($2::text[]) + GROUP BY token`, + [cutoff, tokens] ); const metricsMap = new Map(); for (const row of result.rows) { - metricsMap.set(row.dao_addr, { - token: row.dao_addr, + metricsMap.set(row.token, { + token: row.token, base_volume_24h: row.base_volume_24h ?? '0', target_volume_24h: row.target_volume_24h ?? '0', high_24h: row.high_24h ?? '0', @@ -112,24 +111,25 @@ export class ExternalDatabaseService { } catch (error: any) { // Surface query/schema failures (e.g. served-DB contract drift) instead of // masking them as an empty map — an empty map must mean "genuinely no spot - // trades in the window", not "the query failed". The /api/tickers handler is - // wrapped in asyncHandler, so this propagates to a clean 5xx rather than - // silently degrading to the v0.6 fallback and hiding the problem. - logger.error('[ExternalDB] Error getting spot rolling 24h metrics from futarchy.trades:', error); + // candles in the window", not "the query failed". The /api/tickers handler is + // wrapped in asyncHandler, so this propagates to a clean 5xx. + logger.error('[ExternalDB] Error getting spot rolling 24h metrics from user_pool_spot_ohlcv:', error); throw error; } } /** - * Daily Meteora volumes read DIRECTLY from the meteora accounting ETL's - * `futarchy.meteora_daily` view in our served DB — the source of truth that - * replaces the Dune-sourced `daily_meteora_volumes` app-DB table. + * Daily Meteora volumes from the unified `futarchy.user_pool_daily` (source= + * 'meteora') in the served DB — the user_pool ETL output (a faithful, v0_6_daos- + * filtered map of the meteora accounting ETL's meteora_daily view). Reading the + * unified table (not meteora_daily directly) keeps every market-data read on the + * one ETL output; validated row-exact vs meteora_daily. * - * Returns the SAME column contract as the old (Dune) path so /api/market-data - * consumers are unchanged: token (base mint), date, base_volume, target_volume, - * buy_volume, sell_volume, trade_count, average_price, usdc_fees, token_fees, - * token_fees_usdc, token_per_usdc. Returns an empty array if the connection is - * down (the route serves an empty meteora list in that case). + * Returns the SAME column contract so /api/market-data consumers are unchanged: + * token (base mint), date, base_volume, target_volume, buy_volume, sell_volume, + * trade_count, average_price, usdc_fees, token_fees, token_fees_usdc, + * token_per_usdc (derived = base_volume/target_volume = 1/average_price, since + * user_pool_daily doesn't store it). Empty array if the connection is down. */ async getDailyMeteoraVolumes(options?: { token?: string; @@ -166,7 +166,8 @@ export class ExternalDatabaseService { params.push(options.endDate); i++; } - const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(' AND ')}` : ''; + // source='meteora' is always applied; user-supplied filters are AND-ed after it. + const whereClause = `WHERE source = 'meteora'${conditions.length > 0 ? ' AND ' + conditions.join(' AND ') : ''}`; try { const result = await this.pool.query( `SELECT @@ -181,8 +182,8 @@ export class ExternalDatabaseService { usdc_fees::text, token_fees::text, token_fees_usdc::text, - token_per_usdc::text - FROM futarchy.meteora_daily + (base_volume / nullif(target_volume, 0))::text AS token_per_usdc + FROM futarchy.user_pool_daily ${whereClause} ORDER BY token, date ASC`, params @@ -193,15 +194,107 @@ export class ExternalDatabaseService { // an empty array must mean "genuinely no rows", never "the query failed". The caller // (market route) guards `isAvailable()` for the connection-down case and lets a real // failure propagate to a 5xx instead of returning 200 with zero volume. - logger.error('[ExternalDB] Error getting daily Meteora volumes from futarchy.meteora_daily:', error); + logger.error('[ExternalDB] Error getting daily Meteora volumes from user_pool_daily:', error); + throw error; + } + } + + /** + * Daily FutarchyAMM trading activity from `futarchy.user_pool_daily` (served DB) — + * the unified, ON-CHAIN-derived table that replaces the flat-0.5% + * `v06_fee_volume_daily_aggregate` (and the Dune-style `v06ReconciliationService`). + * + * user_pool_daily stores spot and conditional as SEPARATE rows (market_kind); we + * pivot them into the same spot / conditional / total column shape the old app-DB + * aggregate path returned, so /api/market-data consumers are unchanged. + * Values are already USD / token-UI (no /1e6). ADDS the protocol/LP fee split + * (collected-to-treasury vs retained), which the old flat-rate aggregate could not + * provide. Throws on query failure (never masks as empty for a financial feed). + */ + async getFutarchyAmmDailyActivity(options?: { + token?: string; + tokens?: string[]; + startDate?: string; + endDate?: string; + }): Promise { + if (!this.pool || !this.isConnected) { + return []; + } + const conditions: string[] = [`source = 'futarchy_amm'`]; + const params: any[] = []; + let i = 1; + if (options?.tokens && options.tokens.length > 0) { + conditions.push(`token = ANY($${i}::text[])`); + params.push(options.tokens); + i++; + } else if (options?.token) { + conditions.push(`token = $${i}`); + params.push(options.token); + i++; + } + if (options?.startDate) { conditions.push(`date >= $${i}`); params.push(options.startDate); i++; } + if (options?.endDate) { conditions.push(`date <= $${i}`); params.push(options.endDate); i++; } + const whereClause = `WHERE ${conditions.join(' AND ')}`; + try { + const result = await this.pool.query( + `WITH d AS (SELECT * FROM futarchy.user_pool_daily ${whereClause}), + spot AS (SELECT * FROM d WHERE market_kind = 'spot'), + cond AS (SELECT * FROM d WHERE market_kind = 'conditional') + SELECT + COALESCE(s.token, c.token) AS token, + COALESCE(s.date, c.date)::text AS date, + (c.token IS NOT NULL) AS has_conditional_volume, + -- spot + COALESCE(s.buy_volume,0)::text AS spot_buy_volume, + COALESCE(s.sell_volume,0)::text AS spot_sell_volume, + COALESCE(s.base_volume,0)::text AS spot_base_volume, + COALESCE(s.target_volume,0)::text AS spot_target_volume, + COALESCE(s.trade_count,0) AS spot_trade_count, + COALESCE(s.usdc_fees,0)::text AS spot_usdc_fees, + COALESCE(s.token_fees,0)::text AS spot_token_fees, + COALESCE(s.token_fees_usdc,0)::text AS spot_token_fees_usdc, + COALESCE(s.futarchy_protocol_fee_usdc,0)::text AS spot_protocol_fee_usd, + COALESCE(s.futarchy_lp_fee_usdc,0)::text AS spot_lp_fee_usd, + -- conditional (NULL when no conditional row for the day) + c.buy_volume::text AS conditional_buy_volume, + c.sell_volume::text AS conditional_sell_volume, + c.base_volume::text AS conditional_base_volume, + c.target_volume::text AS conditional_target_volume, + c.trade_count AS conditional_trade_count, + c.usdc_fees::text AS conditional_usdc_fees, + c.token_fees::text AS conditional_token_fees, + c.token_fees_usdc::text AS conditional_token_fees_usdc, + c.futarchy_protocol_fee_usdc::text AS conditional_protocol_fee_usd, + c.futarchy_lp_fee_usdc::text AS conditional_lp_fee_usd, + -- total = spot + conditional + (COALESCE(s.buy_volume,0)+COALESCE(c.buy_volume,0))::text AS total_buy_volume, + (COALESCE(s.sell_volume,0)+COALESCE(c.sell_volume,0))::text AS total_sell_volume, + (COALESCE(s.base_volume,0)+COALESCE(c.base_volume,0))::text AS total_base_volume, + (COALESCE(s.target_volume,0)+COALESCE(c.target_volume,0))::text AS total_target_volume, + (COALESCE(s.trade_count,0)+COALESCE(c.trade_count,0)) AS total_trade_count, + (COALESCE(s.usdc_fees,0)+COALESCE(c.usdc_fees,0))::text AS total_usdc_fees, + (COALESCE(s.token_fees,0)+COALESCE(c.token_fees,0))::text AS total_token_fees, + (COALESCE(s.token_fees_usdc,0)+COALESCE(c.token_fees_usdc,0))::text AS total_token_fees_usdc, + (COALESCE(s.futarchy_protocol_fee_usdc,0)+COALESCE(c.futarchy_protocol_fee_usdc,0))::text AS total_protocol_fee_usd, + (COALESCE(s.futarchy_lp_fee_usdc,0)+COALESCE(c.futarchy_lp_fee_usdc,0))::text AS total_lp_fee_usd, + COALESCE(c.conditional_reconciled, false) AS conditional_reconciled, + COALESCE(c.pending_open_proposals, 0) AS pending_open_proposals + FROM spot s FULL OUTER JOIN cond c ON s.token = c.token AND s.date = c.date + ORDER BY token, date ASC`, + params + ); + return result.rows; + } catch (error: any) { + logger.error('[ExternalDB] Error getting FutarchyAMM daily activity from user_pool_daily:', error); throw error; } } /** - * First spot-trade date per token (base mint), from the v0.6 indexer's per-swap - * v0_6_spot_swaps in the served DB (replaces the frozen Dune buy/sell volumes table). - * Returns Map. Empty map if the connection is down. + * First spot-trade date per token (base mint), from the unified ETL output + * `futarchy.user_pool_daily` (source='futarchy_amm', spot) — MIN(date) per token. + * Keeps every market-data read on the one ETL output. Returns + * Map. Empty map if the connection is down. */ async getFirstTradeDates(): Promise> { if (!this.pool || !this.isConnected) { @@ -209,17 +302,16 @@ export class ExternalDatabaseService { } try { const result = await this.pool.query( - `SELECT d.base_mint_acct AS token, - MIN((to_timestamp(s.unix_timestamp) AT TIME ZONE 'UTC')::date)::text AS first_date - FROM v0_6_spot_swaps s - JOIN v0_6_daos d ON d.dao_addr = s.dao_addr - GROUP BY d.base_mint_acct` + `SELECT token, MIN(date)::text AS first_date + FROM futarchy.user_pool_daily + WHERE source = 'futarchy_amm' AND market_kind = 'spot' + GROUP BY token` ); const m = new Map(); for (const row of result.rows) m.set(row.token, row.first_date); return m; } catch (error: any) { - logger.error('[ExternalDB] Error getting first trade dates from v0_6_spot_swaps:', error); + logger.error('[ExternalDB] Error getting first trade dates from user_pool_daily:', error); return new Map(); } } diff --git a/tests/api.test.ts b/tests/api.test.ts index 2102d65..32f5b37 100644 --- a/tests/api.test.ts +++ b/tests/api.test.ts @@ -57,13 +57,11 @@ const mockPriceService = { const mockDatabaseService = { isAvailable: jest.fn().mockReturnValue(true), - // v0.6 indexer fallback for /api/tickers 24h volume - getV06Rolling24hMetrics: jest.fn().mockResolvedValue(new Map()), } as unknown as DatabaseService; -// Primary /api/tickers source: rolling-24h spot metrics read straight from the -// indexer DB (futarchy.trades), keyed by dao_addr. First-trade dates (startDate) -// also come from the served (external) DB now. +// /api/tickers 24h metrics come from the unified user_pool ETL candles +// (user_pool_spot_ohlcv), keyed by token (base mint). First-trade dates (startDate) +// also come from the served (external) DB. Single source, no app-DB fallback. const mockExternalDatabaseService = { isAvailable: jest.fn().mockReturnValue(true), getSpotRolling24hMetrics: jest.fn().mockResolvedValue(new Map()), @@ -130,10 +128,10 @@ describe('CoinGecko API', () => { } }); - it('should read 24h volume from the indexer DB (futarchy.trades) keyed by dao_addr', async () => { + it('should read 24h volume from user_pool_spot_ohlcv keyed by token (base mint)', async () => { (mockExternalDatabaseService as any).getSpotRolling24hMetrics.mockResolvedValueOnce(new Map([ - [mockDaoAddress.toString(), { - token: mockDaoAddress.toString(), + [mockBaseMint.toString(), { + token: mockBaseMint.toString(), base_volume_24h: '12.5', target_volume_24h: '125', high_24h: '0.06', @@ -171,24 +169,6 @@ describe('CoinGecko API', () => { expect(response.body[0]).not.toHaveProperty('startDate'); }); - it('should fall back to v0.6 indexer metrics when futarchy.trades is empty', async () => { - (mockDatabaseService as any).getV06Rolling24hMetrics.mockResolvedValueOnce(new Map([ - [mockBaseMint.toString(), { - token: mockBaseMint.toString(), - base_volume_24h: '7', - target_volume_24h: '70', - high_24h: '0.06', - low_24h: '0.04', - trade_count_24h: 3, - }], - ])); - - const response = await request(app).get('/api/tickers'); - - expect(response.status).toBe(200); - expect(response.body[0].base_volume).toBe('7'); - expect(response.body[0].target_volume).toBe('70'); - }); }); describe('GET /health', () => { diff --git a/tests/helpers/testApp.ts b/tests/helpers/testApp.ts index f93695c..970b3f8 100644 --- a/tests/helpers/testApp.ts +++ b/tests/helpers/testApp.ts @@ -10,7 +10,6 @@ import type { LaunchpadService } from '../../src/services/launchpadService.js'; export function createMockDatabaseService(): DatabaseService { return { isAvailable: () => true, - getV06Rolling24hMetrics: async () => new Map(), getServiceHealthHistory: async () => [], getRecentMetrics: async () => [], insertServiceHealthSnapshot: async () => {}, @@ -25,6 +24,7 @@ export function createMockExternalDatabaseService(): ExternalDatabaseService { isAvailable: () => true, getSpotRolling24hMetrics: async () => new Map(), getDailyMeteoraVolumes: async () => [], + getFutarchyAmmDailyActivity: async () => [], getFirstTradeDates: async () => new Map(), close: async () => {}, } as unknown as ExternalDatabaseService; diff --git a/tests/routes/market.test.ts b/tests/routes/market.test.ts index 41c3219..dd5911e 100644 --- a/tests/routes/market.test.ts +++ b/tests/routes/market.test.ts @@ -1,89 +1,115 @@ import { describe, it, expect } from 'bun:test'; import request from 'supertest'; import { createTestApp } from '../helpers/testApp.js'; +import type { ExternalDatabaseService } from '../../src/services/externalDatabaseService.js'; const app = createTestApp(); +// A served-DB stub seeded with one futarchy row (spot + conditional pivoted by +// getFutarchyAmmDailyActivity) and one meteora row, so the happy path asserts the +// actual user_pool ETL → /api/market-data contract, not just the HTTP envelope. +function seededExternalDb(): ExternalDatabaseService { + return { + isAvailable: () => true, + getFutarchyAmmDailyActivity: async () => [ + { + token: 'TOK', date: '2024-01-02', has_conditional_volume: true, + spot_target_volume: '1000', spot_usdc_fees: '5', + spot_protocol_fee_usd: '4', spot_lp_fee_usd: '1', + conditional_target_volume: '200', conditional_usdc_fees: '1', + total_target_volume: '1200', total_usdc_fees: '6', + total_protocol_fee_usd: '4.8', total_lp_fee_usd: '1.2', + conditional_reconciled: true, pending_open_proposals: 0, + }, + ], + getDailyMeteoraVolumes: async () => [ + { + token: 'TOK', date: '2024-01-02', base_volume: '50', target_volume: '500', + buy_volume: '300', sell_volume: '200', trade_count: 7, average_price: '10', + usdc_fees: '2', token_fees: '0.1', token_fees_usdc: '1', token_per_usdc: '0.1', + }, + ], + } as unknown as ExternalDatabaseService; +} + describe('Market Routes', () => { describe('GET /api/market-data', () => { - it('should require startDate parameter', async () => { - const response = await request(app) - .get('/api/market-data') - .query({ endDate: '2024-01-15' }); - - // 400 for validation error, or 503 if database check happens first - expect([400, 503]).toContain(response.status); - if (response.status === 400) { - expect(response.body.error).toBe('Missing required parameter'); - expect(response.body.field).toBe('startDate'); - } + it('returns 400 with field=startDate when startDate is missing', async () => { + const response = await request(app).get('/api/market-data').query({ endDate: '2024-01-15' }); + expect(response.status).toBe(400); + expect(response.body.error).toBe('Missing required parameter'); + expect(response.body.field).toBe('startDate'); }); - it('should require endDate parameter', async () => { - const response = await request(app) - .get('/api/market-data') - .query({ startDate: '2024-01-01' }); - - expect([400, 503]).toContain(response.status); - if (response.status === 400) { - expect(response.body.error).toBe('Missing required parameter'); - expect(response.body.field).toBe('endDate'); - } + it('returns 400 with field=endDate when endDate is missing', async () => { + const response = await request(app).get('/api/market-data').query({ startDate: '2024-01-01' }); + expect(response.status).toBe(400); + expect(response.body.error).toBe('Missing required parameter'); + expect(response.body.field).toBe('endDate'); }); - it('should validate date format for startDate', async () => { + it('returns 400 Invalid date format for a malformed startDate', async () => { const response = await request(app) .get('/api/market-data') .query({ startDate: '01-01-2024', endDate: '2024-01-15' }); - - expect([400, 503]).toContain(response.status); - if (response.status === 400) { - expect(response.body.error).toBe('Invalid date format'); - } + expect(response.status).toBe(400); + expect(response.body.error).toBe('Invalid date format'); }); - it('should validate date format for endDate', async () => { - const response = await request(app) + it('returns 503 when the served DB is unavailable', async () => { + const downApp = createTestApp({ + externalDatabaseService: { isAvailable: () => false } as unknown as ExternalDatabaseService, + }); + const response = await request(downApp) .get('/api/market-data') - .query({ startDate: '2024-01-01', endDate: '15/01/2024' }); - - expect([400, 503]).toContain(response.status); - if (response.status === 400) { - expect(response.body.error).toBe('Invalid date format'); - } + .query({ startDate: '2024-01-01', endDate: '2024-01-15' }); + expect(response.status).toBe(503); + expect(response.body.error).toBe('Served database not available'); }); - it('should accept valid date parameters', async () => { - const response = await request(app) + it('serves futarchy + meteora rows from the user_pool ETL with exact values', async () => { + const seededApp = createTestApp({ externalDatabaseService: seededExternalDb() }); + const response = await request(seededApp) .get('/api/market-data') .query({ startDate: '2024-01-01', endDate: '2024-01-15' }); - - // 200 with data, 503 if database not connected, or 500 if mock incomplete - expect([200, 500, 503]).toContain(response.status); - - if (response.status === 200) { - expect(response.body).toHaveProperty('filters'); - expect(response.body).toHaveProperty('count'); - expect(response.body).toHaveProperty('data'); - expect(response.body.filters.startDate).toBe('2024-01-01'); - expect(response.body.filters.endDate).toBe('2024-01-15'); - } + + expect(response.status).toBe(200); + // envelope + expect(response.body.source).toBe('user-pool-etl'); + expect(response.body.filters.startDate).toBe('2024-01-01'); + expect(response.body.filters.endDate).toBe('2024-01-15'); + + // futarchy block — source-tagged, counted, and the pivoted spot/conditional/total + // + protocol/LP split passed through verbatim from getFutarchyAmmDailyActivity. + expect(response.body.futarchyAMM.source).toBe('etl-user-pool-daily'); + expect(response.body.futarchyAMM.count).toBe(1); + const fut = response.body.futarchyAMM.data[0]; + expect(fut.token).toBe('TOK'); + expect(fut.spot_target_volume).toBe('1000'); + expect(fut.spot_protocol_fee_usd).toBe('4'); + expect(fut.spot_lp_fee_usd).toBe('1'); + expect(fut.total_target_volume).toBe('1200'); + expect(fut.total_protocol_fee_usd).toBe('4.8'); + expect(fut.total_lp_fee_usd).toBe('1.2'); + expect(fut.conditional_reconciled).toBe(true); + + // meteora block + expect(response.body.meteora.source).toBe('etl-meteora-daily'); + expect(response.body.meteora.count).toBe(1); + const met = response.body.meteora.data[0]; + expect(met.token).toBe('TOK'); + expect(met.target_volume).toBe('500'); + expect(met.usdc_fees).toBe('2'); + expect(met.token_per_usdc).toBe('0.1'); }); - it('should accept tokens parameter', async () => { - const response = await request(app) + it('passes the tokens filter through to the response envelope', async () => { + const seededApp = createTestApp({ externalDatabaseService: seededExternalDb() }); + const response = await request(seededApp) .get('/api/market-data') - .query({ - startDate: '2024-01-01', - endDate: '2024-01-15', - tokens: 'token1,token2' - }); - - expect([200, 500, 503]).toContain(response.status); - - if (response.status === 200) { - expect(response.body.filters.tokens).toEqual(['token1', 'token2']); - } + .query({ startDate: '2024-01-01', endDate: '2024-01-15', tokens: 'token1,token2' }); + expect(response.status).toBe(200); + expect(response.body.filters.tokens).toEqual(['token1', 'token2']); }); }); }); From 8b40a2834722518d48cc0c6b76dad8e8dad6b488 Mon Sep 17 00:00:00 2001 From: Jinglun Date: Mon, 8 Jun 2026 15:32:18 -0700 Subject: [PATCH 07/14] refactor(external-api): serve DEX Screener from user_pool_swaps (one source) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Repoint all three DEX Screener swap queries off the raw v0_6_* indexer tables onto the unified ETL output, so the entire external-api now reads from one source: - /events: user_pool_swaps (source='futarchy_amm', market_kind='spot'); columns aliased to the legacy v0_6 shape (side→swap_type; input/output reconstructed from base/quote+side) so the event builder is unchanged. reserves now come from our decoded amm_base_reserves/amm_quote_reserves — non-NULL for EVERY spot swap (vs the nullable raw column). Ordered by on-chain order (slot, inner_group, inner_ix) for stable txnIndex/eventIndex. - /latest-block + /pair: user_pool_swaps too (pair identity + creation in one query; drops the v0_6_daos lookup). Validated vs the LIVE API (market-api.metadao.fi/dexscreener/events): the migrated feed matches byte-for-byte on the full economic payload (signature + input + output + reserves, raw u64) — 6,555/6,555 swaps in a 400k-slot window, 0 mismatches. Adds dexscreener route tests (none existed): buy/sell leg mapping, priceNative, reserves, and per-block txnIndex/eventIndex grouping. 116 pass, typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/routes/dexscreener.ts | 91 +++++++++++++++--------------- tests/routes/dexscreener.test.ts | 97 ++++++++++++++++++++++++++++++++ 2 files changed, 142 insertions(+), 46 deletions(-) create mode 100644 tests/routes/dexscreener.test.ts diff --git a/src/routes/dexscreener.ts b/src/routes/dexscreener.ts index 08d4928..3e916d6 100644 --- a/src/routes/dexscreener.ts +++ b/src/routes/dexscreener.ts @@ -38,8 +38,9 @@ export function createDexScreenerRouter(services: ServiceGetters): Router { } const result = await extDb.query(` - SELECT slot, unix_timestamp - FROM v0_6_spot_swaps + SELECT slot, extract(epoch FROM block_time)::bigint AS unix_timestamp + FROM futarchy.user_pool_swaps + WHERE source = 'futarchy_amm' AND market_kind = 'spot' ORDER BY slot DESC LIMIT 1 `); @@ -147,47 +148,39 @@ export function createDexScreenerRouter(services: ServiceGetters): Router { return res.status(503).json({ error: 'External database not available' }); } - // Look up the pair (DAO) in the external DB - const daoResult = await extDb.query( - `SELECT dao_addr, base_mint_acct, quote_mint_acct - FROM v0_6_daos - WHERE dao_addr = $1`, + // Pair identity + creation (first spot swap) from the unified ETL output. One + // query: the earliest futarchy spot swap for the DAO carries its base/quote + // mints AND the creation block/txn. A DAO with no spot swaps has no tradeable + // pair → 404 (same status the old v0_6_daos miss returned). + const pairResult = await extDb.query( + `SELECT dao_addr, base_mint, quote_mint, + slot, extract(epoch FROM block_time)::bigint AS unix_timestamp, signature + FROM futarchy.user_pool_swaps + WHERE source = 'futarchy_amm' AND market_kind = 'spot' AND dao_addr = $1 + ORDER BY slot ASC, inner_group ASC, inner_ix ASC + LIMIT 1`, [id], ); - if (daoResult.rows.length === 0) { + if (pairResult.rows.length === 0) { return res.status(404).json({ error: 'Pair not found' }); } - const dao = daoResult.rows[0]; - - // Get creation info (first swap for this pair) - const creationResult = await extDb.query( - `SELECT slot, unix_timestamp, signature - FROM v0_6_spot_swaps - WHERE dao_addr = $1 - ORDER BY slot ASC, id ASC - LIMIT 1`, - [id], - ); + const dao = pairResult.rows[0]; const response: DexScreenerPairResponse = { pair: { id: dao.dao_addr, dexKey: DEX_KEY, - asset0Id: dao.base_mint_acct, - asset1Id: dao.quote_mint_acct, + asset0Id: dao.base_mint, + asset1Id: dao.quote_mint, feeBps: FEE_BPS, + createdAtBlockNumber: Number(dao.slot), + createdAtBlockTimestamp: Number(dao.unix_timestamp), + createdAtTxnId: dao.signature, }, }; - if (creationResult.rows.length > 0) { - const creation = creationResult.rows[0]; - response.pair.createdAtBlockNumber = Number(creation.slot); - response.pair.createdAtBlockTimestamp = Number(creation.unix_timestamp); - response.pair.createdAtTxnId = creation.signature; - } - pairCache.set(id, { data: response, expiresAt: Date.now() + CACHE_TTL_MS }); res.json(response); })); @@ -217,26 +210,32 @@ export function createDexScreenerRouter(services: ServiceGetters): Router { return res.status(503).json({ error: 'External database not available' }); } - // Query swap events in the slot range (both inclusive) - // amm_base_amount / amm_quote_amount = post-swap pool reserves (nullable for older rows) + // Query swap events in the slot range (both inclusive) from the unified ETL + // output. Columns are aliased to the legacy v0_6 shape so the builder below is + // unchanged: side→swap_type, and input/output reconstructed from base/quote + + // side (Buy: USDC in / token out; Sell: token in / USDC out). amm_base/quote + // reserves are our decoded post-swap reserves — non-NULL for EVERY spot swap + // (validated dollar-exact vs the live feed), unlike the nullable raw column. + // Ordered by on-chain execution order (slot, inner_group, inner_ix) for stable + // txnIndex/eventIndex. const result = await extDb.query( `SELECT - s.id, - s.signature, - s.slot, - s.unix_timestamp, - s.dao_addr, - s.user_addr, - s.swap_type, - s.input_amount, - s.output_amount, - s.amm_base_amount, - s.amm_quote_amount - FROM v0_6_spot_swaps s - WHERE s.slot >= $1 AND s.slot <= $2 - AND s.input_amount > 0 AND s.output_amount > 0 - AND LOWER(TRIM(s.swap_type)) IN ('buy', 'sell') - ORDER BY s.slot ASC, s.id ASC`, + u.id, + u.signature, + u.slot, + extract(epoch FROM u.block_time)::bigint AS unix_timestamp, + u.dao_addr, + u.user_addr, + u.side AS swap_type, + CASE WHEN u.side = 'buy' THEN u.quote_amount ELSE u.base_amount END AS input_amount, + CASE WHEN u.side = 'buy' THEN u.base_amount ELSE u.quote_amount END AS output_amount, + u.amm_base_reserves AS amm_base_amount, + u.amm_quote_reserves AS amm_quote_amount + FROM futarchy.user_pool_swaps u + WHERE u.source = 'futarchy_amm' AND u.market_kind = 'spot' + AND u.slot >= $1 AND u.slot <= $2 + AND u.base_amount > 0 AND u.quote_amount > 0 + ORDER BY u.slot ASC, u.inner_group ASC, u.inner_ix ASC`, [fromBlock, toBlock], ); diff --git a/tests/routes/dexscreener.test.ts b/tests/routes/dexscreener.test.ts new file mode 100644 index 0000000..d0f7e57 --- /dev/null +++ b/tests/routes/dexscreener.test.ts @@ -0,0 +1,97 @@ +import { describe, it, expect } from 'bun:test'; +import request from 'supertest'; +import { createTestApp } from '../helpers/testApp.js'; +import type { ExternalDatabaseService } from '../../src/services/externalDatabaseService.js'; + +// Stub the served DB with canned user_pool_swaps-shaped rows (the aliased column +// shape the migrated /events SQL returns) so we test the event-building transform: +// buy/sell leg mapping, priceNative, reserves, and txnIndex/eventIndex grouping. +function extDbReturning(rows: any[]): ExternalDatabaseService { + return { + isAvailable: () => true, + query: async (text: string) => { + if (/ORDER BY slot DESC/i.test(text)) { + return { rows: [{ slot: 999, unix_timestamp: '1700000999' }] } as any; + } + return { rows } as any; // /events + }, + } as unknown as ExternalDatabaseService; +} + +describe('DexScreener Routes', () => { + describe('GET /dexscreener/events', () => { + it('maps buy/sell legs, price, reserves, and txn/event indices from user_pool_swaps', async () => { + // Two swaps in one signature (same txn → eventIndex 0,1), one in a second txn. + const rows = [ + { id: 1, signature: 'SIGA', slot: 10, unix_timestamp: '1700', dao_addr: 'DAO1', user_addr: 'U1', + swap_type: 'buy', input_amount: '2000000', output_amount: '40000000', + amm_base_amount: '1000000000', amm_quote_amount: '50000000' }, + { id: 2, signature: 'SIGA', slot: 10, unix_timestamp: '1700', dao_addr: 'DAO1', user_addr: 'U1', + swap_type: 'sell', input_amount: '10000000', output_amount: '500000', + amm_base_amount: '1010000000', amm_quote_amount: '49500000' }, + // Same slot as SIGA, new signature → txnIndex increments, eventIndex resets. + { id: 3, signature: 'SIGB', slot: 10, unix_timestamp: '1700', dao_addr: 'DAO1', user_addr: 'U2', + swap_type: 'buy', input_amount: '1000000', output_amount: '20000000', + amm_base_amount: '990000000', amm_quote_amount: '50500000' }, + ]; + const app = createTestApp({ externalDatabaseService: extDbReturning(rows) }); + + const res = await request(app).get('/dexscreener/events').query({ fromBlock: 0, toBlock: 100 }); + expect(res.status).toBe(200); + expect(res.body.events).toHaveLength(3); + + const [buy, sell, buy2] = res.body.events; + + // Buy: USDC in (asset1In), token out (asset0Out); price = USDC/token = 2/40 = 0.05 + expect(buy.eventType).toBe('swap'); + expect(buy.txnId).toBe('SIGA'); + expect(buy.txnIndex).toBe(0); + expect(buy.eventIndex).toBe(0); + expect(buy.maker).toBe('U1'); + expect(buy.pairId).toBe('DAO1'); + expect(buy.asset1In).toBe(2); + expect(buy.asset0Out).toBe(40); + expect(buy.priceNative).toBeCloseTo(0.05, 9); + expect(buy.reserves).toEqual({ asset0: 1000, asset1: 50 }); + + // Second swap in the SAME signature → same txnIndex, eventIndex increments. + expect(sell.txnId).toBe('SIGA'); + expect(sell.txnIndex).toBe(0); + expect(sell.eventIndex).toBe(1); + // Sell: token in (asset0In=10), USDC out (asset1Out=0.5); price = USDC/token = 0.5/10 = 0.05 + expect(sell.asset0In).toBe(10); + expect(sell.asset1Out).toBe(0.5); + expect(sell.priceNative).toBeCloseTo(0.05, 9); + expect(sell.reserves).toEqual({ asset0: 1010, asset1: 49.5 }); + + // New signature in the SAME slot → txnIndex increments, eventIndex resets. + expect(buy2.txnId).toBe('SIGB'); + expect(buy2.txnIndex).toBe(1); + expect(buy2.eventIndex).toBe(0); + }); + + it('returns 503 when the served DB is unavailable', async () => { + const app = createTestApp({ + externalDatabaseService: { isAvailable: () => false } as unknown as ExternalDatabaseService, + }); + const res = await request(app).get('/dexscreener/events').query({ fromBlock: 0, toBlock: 100 }); + expect(res.status).toBe(503); + }); + + it('rejects an out-of-order block range', async () => { + const app = createTestApp({ externalDatabaseService: extDbReturning([]) }); + const res = await request(app).get('/dexscreener/events').query({ fromBlock: 100, toBlock: 10 }); + expect(res.status).toBe(400); + }); + }); + + describe('GET /dexscreener/latest-block', () => { + it('returns the latest block from user_pool_swaps', async () => { + const app = createTestApp({ externalDatabaseService: extDbReturning([]) }); + const res = await request(app).get('/dexscreener/latest-block'); + expect(res.status).toBe(200); + expect(res.body.block.blockNumber).toBe(999); + expect(res.body.block.blockTimestamp).toBe(1700000999); + }); + }); +}); From 36cce17ad8619eb802b1569bc4afeb558c1fc5de Mon Sep 17 00:00:00 2001 From: Jinglun Date: Mon, 8 Jun 2026 16:02:20 -0700 Subject: [PATCH 08/14] =?UTF-8?q?external-api:=20finalize=20user=5Fpool=20?= =?UTF-8?q?cutover=20=E2=80=94=20served-data=20contract,=20cleanup,=20hard?= =?UTF-8?q?ening?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bundles the remaining external-api work for the unified user_pool migration (branch is local; nothing was pushed). All from this effort + the concurrent hardening/cleanup pass; typecheck clean, 116 tests pass. - coingecko /tickers: single source (user_pool_spot_ohlcv), 503 on served-DB down instead of masking zero volume. - externalDatabaseService: served-data contract health check (checkServedDataContract) surfaced via /health (degraded when the ETL tables are missing). - Security (info disclosure): market/health/metrics no longer return raw error.message — log server-side, return generic error + requestId. - Dead-code removal: drop the legacy Dune schema files, the app-DB v06 OHLCV/fee tables (schemaManager) and dead metricsService helpers — all unread after the cutover to user_pool_*. - Docs/config/tests updated to match (README, AGENTS, example.env, validation, resilience, root, test mocks + suites). Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/CODEOWNERS | 1 - AGENTS.md | 19 +-- README.md | 33 ++-- docs/tickers-volume-cutover-validation.md | 158 +++--------------- example.env | 25 ++- src/config.ts | 6 +- src/routes/coingecko.ts | 44 +++-- src/routes/health.ts | 18 +- src/routes/market.ts | 11 +- src/routes/metrics.ts | 3 +- src/routes/root.ts | 8 +- src/schema/dune-all-volume.sql | 85 ---------- src/schema/dune-daily-buy-sell-volume.sql | 69 -------- src/schema/dune-daily-fees-volume.sql | 116 ------------- src/schema/dune-hourly-volume.sql | 88 ---------- src/schema/dune-incremental-volume.sql | 94 ----------- src/schema/dune-ten-minute-volume.sql | 119 ------------- src/schema/dune.sql | 75 --------- src/schema/v06-ohlcv-tables.sql | 157 ----------------- .../database/internal/schemaManager.ts | 141 ---------------- src/services/externalDatabaseService.ts | 138 +++++++++++++-- src/services/meteoraService.ts | 4 +- src/services/metricsService.ts | 46 ----- src/utils/resilience.ts | 8 +- src/utils/validation.ts | 4 +- tests/api.test.ts | 15 ++ tests/helpers/testApp.ts | 5 + tests/routes/health.test.ts | 4 +- tests/routes/metrics.test.ts | 2 +- tests/routes/root.test.ts | 2 +- 30 files changed, 276 insertions(+), 1222 deletions(-) delete mode 100644 src/schema/dune-all-volume.sql delete mode 100644 src/schema/dune-daily-buy-sell-volume.sql delete mode 100644 src/schema/dune-daily-fees-volume.sql delete mode 100644 src/schema/dune-hourly-volume.sql delete mode 100644 src/schema/dune-incremental-volume.sql delete mode 100644 src/schema/dune-ten-minute-volume.sql delete mode 100644 src/schema/dune.sql delete mode 100644 src/schema/v06-ohlcv-tables.sql diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 4231b5e..1c8b057 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -13,4 +13,3 @@ # Operational scripts /scripts/safe-update.ts @metanallok -/scripts/backfillV06.ts @metanallok diff --git a/AGENTS.md b/AGENTS.md index 13f346c..08f436a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,11 +12,9 @@ - **`bun test`** - Run all tests - **`bun run