Use token decimals for display amounts

.env.example

@@ -1,4 +1,8 @@
 # ─── Database ───────────────────────────────────────────────
+# Critical for mainnet: Use a durable external provider (e.g. Neon, Supabase).
+# Do NOT use Render free Postgres as it is deleted after 90 days.
+# See docs/DEPLOY_MAINNET.md for more details.
+#
 # Local:
 #   DATABASE_URL="postgresql://wraith:wraith@localhost:5432/wraith"
 #   DIRECT_DATABASE_URL="postgresql://wraith:wraith@localhost:5432/wraith"
@@ -80,6 +84,13 @@ SAC_CONTRACT_IDS_MAINNET=
 # Per-network NFT watch lists (for dual-network deployments):
 #   NFT_CONTRACT_IDS_TESTNET=
 #   NFT_CONTRACT_IDS_MAINNET=
+
+# NFT metadata enrichment is bounded so it can never stall ingest.
+# Concurrent metadata lookups per poll cycle (default 8), and the total time
+# in ms a cycle may spend on them (default 10000). Tokens not finished within
+# the budget stay uncached and are retried on a later cycle.
+# NFT_METADATA_CONCURRENCY=8
+# NFT_METADATA_BUDGET_MS=10000
 NFT_CONTRACT_IDS=
 NFT_CONTRACT_IDS_TESTNET=
 NFT_CONTRACT_IDS_MAINNET=
@@ -91,6 +102,12 @@ NFT_CONTRACT_IDS_MAINNET=
 # Max events per RPC call (Stellar RPC hard-caps at 10000)
 EVENTS_BATCH_SIZE=10000
 
+# Shard each poll window across N workers (by contract) for throughput. This
+# changes how fast a window is ingested, never what is indexed: both paths run
+# every record through the same pipeline (transfers, NFTs and their metadata,
+# LP shares, host-fn logs, account summaries). Default 1 = no sharding.
+# INGEST_WORKERS=1
+
 # Delete transfers older than N days to stay within DB free-tier limits (default: 30)
 RETENTION_DAYS=30
 

README.md

@@ -70,25 +70,27 @@ cp .env.example .env
 
 ```env
 DATABASE_URL="postgresql://wraith:wraith@localhost:5432/wraith"
+DIRECT_DATABASE_URL="postgresql://wraith:wraith@localhost:5432/wraith"
 STELLAR_NETWORK="testnet"
 # SOROBAN_RPC_URL is optional on testnet — the default public endpoint is used automatically
 SOROBAN_RPC_URL=
 
 START_LEDGER=
-CONTRACT_IDS=
+SAC_CONTRACT_IDS=
 PORT=3000
 ```
 
 **Mainnet setup** (production):
 
 ```env
 DATABASE_URL="postgresql://wraith:wraith@localhost:5432/wraith"
+DIRECT_DATABASE_URL="postgresql://wraith:wraith@localhost:5432/wraith"
 STELLAR_NETWORK="mainnet"
 # Required on mainnet — no free public Soroban RPC exists
 SOROBAN_RPC_URL="https://mainnet.stellar.validationcloud.io/v1/<YOUR_API_KEY>"
 
 # Strongly recommended on mainnet: filter to specific contracts to reduce load
-CONTRACT_IDS="CTOKEN1...,CTOKEN2..."
+SAC_CONTRACT_IDS="CTOKEN1...,CTOKEN2..."
 START_LEDGER=
 PORT=3000
 ```
@@ -98,7 +100,7 @@ PORT=3000
 ### 3. Start Postgres
 
 ```bash
-docker-compose up -d db
+docker compose up -d db
 ```
 
 ### 4. Run database migrations
@@ -107,7 +109,65 @@ docker-compose up -d db
 npx prisma migrate dev --name init
 ```
 
-### 5. Start Wraith
+### 5. Seed the database (optional)
+
+Populate the database with deterministic test data for development and testing:
+
+```bash
+npm run db:seed
+```
+
+By default, this seeds `testnet`. To seed `mainnet`:
+
+```bash
+npm run db:seed -- --network=mainnet
+```
+
+You can now query the seeded data:
+
+```bash
+curl "http://localhost:3000/transfers/incoming/GAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAWHF"
+```
+
+Expected response:
+
+```json
+{
+  "total": 2,
+  "limit": 50,
+  "offset": 0,
+  "transfers": [
+    {
+      "id": 1,
+      "contractId": "CAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD2KM",
+      "eventType": "transfer",
+      "fromAddress": "GBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBWWHF",
+      "toAddress": "GAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAWHF",
+      "amount": "10000000",
+      "displayAmount": "1.0000000",
+      "ledger": 2001,
+      "ledgerClosedAt": "2025-01-01T00:00:00.000Z",
+      "txHash": "tx-incoming-a-1",
+      "eventId": "integration-001"
+    },
+    {
+      "id": 2,
+      "contractId": "CBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBD2KM",
+      "eventType": "transfer",
+      "fromAddress": "GCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCWWHF",
+      "toAddress": "GAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAWHF",
+      "amount": "40000000",
+      "displayAmount": "4.0000000",
+      "ledger": 2004,
+      "ledgerClosedAt": "2025-02-01T00:00:00.000Z",
+      "txHash": "txhash-integration-multi",
+      "eventId": "integration-004"
+    }
+  ]
+}
+```
+
+### 6. Start Wraith
 
 ```bash
 # Development (hot reload)
@@ -120,7 +180,17 @@ npm run build && npm start
 Or run everything via Docker:
 
 ```bash
-docker-compose up --build
+# Without cache (default)
+docker compose up --build
+
+# With Redis cache enabled
+docker compose --profile cache up --build
+```
+
+To seed the database when running via Docker, run the seed command in a separate terminal after the database is up:
+
+```bash
+npm run db:seed
 ```
 
 ***
@@ -218,7 +288,8 @@ console.log(data);
 
 // Expected response
 // {
-//   "total": 42,
+//   "hasMore": true,
+//   "nextCursor": "MTIzNDU=",
 //   "limit": 10,
 //   "offset": 0,
 //   "transfers": [
@@ -239,6 +310,13 @@ console.log(data);
 // }
 ```
 
+> **Pagination and totals.** List endpoints (`/transfers/*`, `/nfts/transfers`, GraphQL `transfers`)
+> do not run a `COUNT` by default. They fetch one row past the page and return
+> `hasMore` (plus `nextCursor` to continue), which is all pagination needs. Pass
+> `?includeTotal=true` (GraphQL: `includeTotal: true`) to also get an exact
+> `total`. This runs a `COUNT` over every matching row, so it is markedly slower
+> on large result sets. Without it, `total` is absent from the response.
+
 ***
 
 ### `GET /transfers/address/:address` — All transfers (sent & received, merged)
@@ -263,7 +341,7 @@ console.log(data);
 
 // Expected response  (same shape as /transfers/incoming — adds "direction" per row)
 // {
-//   "total": 85,
+//   "hasMore": true,
 //   "limit": 20,
 //   "offset": 0,
 //   "transfers": [
@@ -331,6 +409,8 @@ console.log(data);
 
 Base URL: `http://localhost:3000`
 
+Offramp order lookups need a bearer token; see [docs/offramp-orders.md](docs/offramp-orders.md).
+
 ### Selecting a network
 
 Wraith stores testnet and mainnet rows in the same tables, discriminated by a
@@ -367,8 +447,8 @@ both chains in a single round-trip:
 
 ```graphql
 {
-  testnet: transfers(address: "GABC…", network: TESTNET) { total }
-  mainnet: transfers(address: "GABC…", network: MAINNET) { total }
+  testnet: transfers(address: "GABC…", network: TESTNET, includeTotal: true) { total hasMore }
+  mainnet: transfers(address: "GABC…", network: MAINNET, includeTotal: true) { total hasMore }
 }
 ```
 
@@ -437,8 +517,11 @@ last_indexed_ledger{network="mainnet"} 5842100
 | `ledgers_indexed_total` | counter | `network` | Ledgers the indexer has advanced through. A flat `rate()` on a network whose loop should be running means it has stalled. |
 | `transfers_stored_total` | counter | `network`, `type` | Rows persisted, split `fungible` / `nft` — one parse path can break while the other keeps working. |
 | `rpc_errors_total` | counter | `outcome` | Failed RPC attempts. `retry` counts attempts `withRetry` absorbed, `exhausted` counts calls that gave up — a degrading endpoint shows up in `retry` long before it fails a call. |
+| `events_skipped_total` | counter | `reason` | Events the decoder dropped. `unrecognised` is non-token events (normal); `malformed` is token events that failed to decode — a rising rate points at a non-standard contract or a decoder bug. |
 | `last_indexed_ledger` | gauge | `network` | Highest committed ledger. Against the chain tip, this is lag. |
 | `db_query_duration_seconds` | histogram | `operation` | Duration of instrumented DB operations, failures included. |
+| `http_requests_total` | counter | `method`, `route`, `status` | HTTP requests served. `route` is the Express route *pattern*, never the raw URL, so addresses never become label values; an unmatched request reports `route="unknown"`. |
+| `http_request_duration_seconds` | histogram | `method`, `route` | HTTP latency, 5 ms to 10 s. No `status` label — the split is rarely worth the extra series. |
 
 Standard `process_*` and `nodejs_*` metrics are exported alongside these.
 

docker-compose.yml

@@ -1,5 +1,3 @@
-version: "3.9"
-
 services:
   # ── Postgres ──────────────────────────────────────────────
   db:
@@ -20,6 +18,21 @@ services:
       timeout: 5s
       retries: 10
 
+  # ── Redis (optional, for response cache) ─────────────────
+  redis:
+    image: redis:7-alpine
+    container_name: wraith_redis
+    restart: unless-stopped
+    ports:
+      - "6379:6379"
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 5s
+      timeout: 5s
+      retries: 10
+    profiles:
+      - cache
+
   # ── Wraith Indexer + API ───────────────────────────────────
   wraith:
     build: .
@@ -28,14 +41,38 @@ services:
     depends_on:
       db:
         condition: service_healthy
+      redis:
+        condition: service_healthy
+        required: false
     environment:
       DATABASE_URL: postgresql://wraith:wraith@db:5432/wraith
-      STELLAR_RPC_URL: ${STELLAR_RPC_URL:-https://soroban-testnet.stellar.org}
+      DIRECT_DATABASE_URL: postgresql://wraith:wraith@db:5432/wraith
+      STELLAR_NETWORK: ${STELLAR_NETWORK:-testnet}
+      SOROBAN_RPC_URL: ${SOROBAN_RPC_URL:-}
+      STELLAR_RPC_URL: ${STELLAR_RPC_URL:-}
+      NETWORKS: ${NETWORKS:-}
       START_LEDGER: ${START_LEDGER:-}
       POLL_INTERVAL_MS: ${POLL_INTERVAL_MS:-6000}
+      SAC_CONTRACT_IDS: ${SAC_CONTRACT_IDS:-}
+      SAC_CONTRACT_IDS_TESTNET: ${SAC_CONTRACT_IDS_TESTNET:-}
+      SAC_CONTRACT_IDS_MAINNET: ${SAC_CONTRACT_IDS_MAINNET:-}
       CONTRACT_IDS: ${CONTRACT_IDS:-}
+      NFT_CONTRACT_IDS: ${NFT_CONTRACT_IDS:-}
+      NFT_CONTRACT_IDS_TESTNET: ${NFT_CONTRACT_IDS_TESTNET:-}
+      NFT_CONTRACT_IDS_MAINNET: ${NFT_CONTRACT_IDS_MAINNET:-}
       EVENTS_BATCH_SIZE: ${EVENTS_BATCH_SIZE:-10000}
+      RETENTION_DAYS: ${RETENTION_DAYS:-30}
       PORT: ${PORT:-3000}
+      CACHE_ENABLED: ${CACHE_ENABLED:-false}
+      REDIS_URL: ${REDIS_URL:-redis://redis:6379}
+      CACHE_KEY_PREFIX: ${CACHE_KEY_PREFIX:-wraith:cache:}
+      CACHE_TTL_POPULAR_MS: ${CACHE_TTL_POPULAR_MS:-60000}
+      CACHE_TTL_SEARCH_MS: ${CACHE_TTL_SEARCH_MS:-15000}
+      TOMBSTONE_CHECK_EVERY_CYCLES: ${TOMBSTONE_CHECK_EVERY_CYCLES:-100}
+      LP_POOL_CONTRACT_IDS: ${LP_POOL_CONTRACT_IDS:-}
+      LP_POOL_CONTRACT_IDS_TESTNET: ${LP_POOL_CONTRACT_IDS_TESTNET:-}
+      LP_POOL_CONTRACT_IDS_MAINNET: ${LP_POOL_CONTRACT_IDS_MAINNET:-}
+      SKIP_INDEXER: ${SKIP_INDEXER:-false}
     ports:
       - "${PORT:-3000}:3000"
 

docs/DEPLOY_MAINNET.md

@@ -0,0 +1,80 @@
+# Mainnet Deployment Guide
+
+Deploying Wraith on mainnet requires a more robust infrastructure setup than testnet due to the lack of free public resources and the need for high availability. 
+
+When running both testnet and mainnet indexers, it is recommended to run them side-by-side as separate deployments (with separate databases) rather than combining them, ensuring one network's issues don't impact the other. A single process *can* index both by setting `NETWORKS=testnet,mainnet` — see [DUAL_NETWORK.md](./DUAL_NETWORK.md) for the trade-offs.
+
+## 1. Database (Managed Postgres)
+
+**Critical:** Do *not* use Render's free PostgreSQL for a mainnet deployment. Render deletes free databases after 90 days, which will destroy your indexed state.
+
+- Use a durable managed Postgres provider such as **Neon** (or Supabase/AWS RDS).
+- Supply the connection string in your environment variables as `DATABASE_URL`.
+- Set `DIRECT_DATABASE_URL` as well as `DATABASE_URL`. `prisma/schema.prisma` declares `directUrl = env("DIRECT_DATABASE_URL")`, and schema sync on boot fails without it. On a pooled provider, `DATABASE_URL` is the pooler endpoint and `DIRECT_DATABASE_URL` the direct one; on a non-pooled provider both hold the same value.
+- Ensure you have a backup strategy. See the [backup and restore runbook](./backup-restore.md) for instructions on how to back up your indexed data.
+
+## 2. Soroban RPC Provider
+
+Unlike testnet, Stellar mainnet does not provide a free public Soroban RPC endpoint suitable for a production indexer.
+
+- You must configure an external provider (e.g., Validation Cloud, Ankr) or run your own self-hosted RPC.
+- Provide the endpoint URL in the `SOROBAN_RPC_URL` environment variable. 
+- *Note:* Never commit your real RPC URL containing API keys to version control. Use the Render environment secrets instead.
+
+## 3. Render Hosting & Keep-Alive
+
+Wraith can run on Render's free web services, but with two important caveats:
+
+1. **Sleep mode:** Render spins down free web services after 15 minutes of inactivity. For an indexer, sleeping means missing blocks. 
+   - **Solution:** Use a service like **UptimeRobot** to ping your deployment's `/status` endpoint every 5 minutes to prevent it from sleeping.
+2. **Usage caps:** Render has a 750 instance-hours/month free tier cap. This is roughly enough for exactly *one* always-on web service. If you are running both testnet and mainnet deployments on the free tier, one of them will run out of hours before the month ends unless you upgrade to a paid tier.
+
+## 4. Environment Variables Matrix
+
+Below is the environment configuration matrix for running Testnet vs. Mainnet side-by-side. Set these as secrets in your hosting provider.
+
+> **Set `SAC_CONTRACT_IDS` explicitly on mainnet — do not rely on the built-in default.**
+> The native XLM SAC address is derived from the network passphrase, so it differs per
+> network, and the value the code falls back to when `SAC_CONTRACT_IDS` is unset is not
+> currently correct for mainnet (tracked separately). Derive the address yourself and
+> paste the result:
+>
+> ```js
+> import { Asset, Networks } from '@stellar/stellar-sdk';
+> Asset.native().contractId(Networks.PUBLIC);   // mainnet
+> Asset.native().contractId(Networks.TESTNET);  // testnet
+> ```
+
+### Mainnet Deployment
+```env
+# Network Configuration
+STELLAR_NETWORK=mainnet
+
+# RPC Provider (Required for mainnet - use your actual provider URL with API key)
+SOROBAN_RPC_URL=https://mainnet.stellar.validationcloud.io/v1/<API_KEY>
+
+# Mainnet Native XLM SAC Contract ID
+#   = Asset.native().contractId(Networks.PUBLIC)
+SAC_CONTRACT_IDS=CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA
+
+# External Durable Database (e.g., Neon)
+DATABASE_URL=postgresql://[user]:[password]@[host]/[dbname]?sslmode=require
+DIRECT_DATABASE_URL=postgresql://[user]:[password]@[host]/[dbname]?sslmode=require
+```
+
+### Testnet Deployment
+```env
+# Network Configuration
+STELLAR_NETWORK=testnet
+
+# RPC Provider (Optional - defaults to public testnet RPC)
+SOROBAN_RPC_URL=
+
+# Testnet Native XLM SAC Contract ID
+#   = Asset.native().contractId(Networks.TESTNET)
+SAC_CONTRACT_IDS=CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC
+
+# Database
+DATABASE_URL=postgresql://[user]:[password]@[host]/[dbname]?sslmode=require
+DIRECT_DATABASE_URL=postgresql://[user]:[password]@[host]/[dbname]?sslmode=require
+```