Closes #199 - feat(escrow): reconcile against live db and soroban reads

[Emmanuel-abiola]

feat(escrow): reconcile against live db and soroban reads

Closes #199

Summary

The nightly escrow reconciliation job (src/jobs/reconcileEscrow.js) previously
reconciled hardcoded invoices (inv_1/inv_2/inv_3) against a static
mockAmounts lookup and stashed the result on global.reconciliationSummary.
It could not detect real drift between the DB fundedTotal and on-chain
LiquifactEscrow.get_escrow funded amounts.

This PR wires the job to the real Knex invoices table and the Soroban read
path used elsewhere, persists run summaries to a table, and emits a Prometheus
mismatch counter.

Changes

src/jobs/reconcileEscrow.js (rewrite)

• getInvoicesFromDb (hardcoded) → iterateInvoicesFromDb: a paginated
  db('invoices') keyset scan (ordered by id) filtered to invoices in
  linked_escrow / funded / partially_funded states and deleted_at IS NULL,
  left-joining escrow_summaries.total_funded as the DB fundedTotal.
• getOnChainFundedAmount (static mockAmounts) → readFundedAmount from
  src/services/escrowRead.js, which reads the contract funded_amount via
  callSorobanContract (shared retry + error mapping).
• global.reconciliationSummary → persisted to the new reconciliation_runs
  table; getReconciliationSummary() now reads the latest row (async).
• Mismatches now increment escrow_reconciliation_mismatches_total and emit a
  structured logger.warn carrying invoiceId, dbFundedTotal, onChainAmount.
• Worker wiring and the public function surface are preserved.

src/services/escrowRead.js

• Added readFundedAmount(invoiceId, { escrowAdapter }), reusing the existing
  INVOICE_ID_RE validation and _fetchBaseEscrowState path. Accepts either a
  full base-state object or a bare number from an adapter; non-finite values
  fall back to 0.

src/metrics.js

• Added the escrow_reconciliation_mismatches_total Prometheus counter on the
  shared registry and exported it.

migrations/20260429000000_create_reconciliation_runs.js (new)

• Creates reconciliation_runs (total, matches, mismatches, errors,
  results JSONB, reconciled_at indexed). Apply with npm run db:migrate.

src/services/health.js

• One-line change: await getReconciliationSummary() (now async). No health
  tests touch this path.

tests/reconcileEscrow.test.js (rewrite)

• 27 tests asserting match/mismatch/error classification against mocked Knex +
  Soroban; metric increment on mismatch; warning-log fields; pagination;
  persistence; and summary read-back.

docs/ops-reconcile.md

• Updated Components, Data Flow, Alerting, Metrics, and a new Persistence
  section describing the table, the metric, and the suggested alert.

Acceptance criteria

• Real paginated db('invoices') query, scoped to linked_escrow/funded states.
• Real on-chain read via escrowRead.js / callSorobanContract.
• Summaries persisted to reconciliation_runs (not global).
• Prometheus mismatch counter emitted via src/metrics.js.
• Tests assert match/mismatch/error classification against mocked Knex + Soroban.
• A mismatch increments the metric and produces a warning log with
  invoiceId, dbFundedTotal, onChainAmount.

Security notes

• Input validation: invoice IDs validated against the shared INVOICE_ID_RE
  before any Soroban call; DB page size clamped to [1, 1000].
• Auth / secrets: no secrets added; DB and Soroban config remain in env
  (DATABASE_URL, SOROBAN_*). No new endpoints introduced.
• Idempotency: reads are side-effect-free; each run appends exactly one
  summary row. Persistence failures are logged and swallowed so they never mask
  a detected mismatch (the metric + warning fire before the insert).
• Resilience: per-invoice errors are classified as error and do not abort
  the run; transient Soroban faults are retried by the existing wrapper.

Test output

PASS tests/reconcileEscrow.test.js
    ✓ classifies MATCH when DB and on-chain amounts are equal (6 ms)
    ✓ classifies MISMATCH, increments the metric, and warns with the required fields (3 ms)
    ✓ classifies ERROR when the Soroban read throws and does not touch the metric (2 ms)
    ✓ classifies ERROR for an invalid invoice id (validation failure) (1 ms)
    ✓ queries the invoices table filtered to reconcilable, non-deleted rows (2 ms)
    ✓ coerces string/null DECIMAL funded totals to finite numbers (1 ms)
    ✓ paginates: keeps fetching full pages until a short page is returned (3 ms)
    ✓ stops cleanly on an empty first page
    ✓ clamps absurd page sizes into the [1,1000] range without throwing (2 ms)
    ✓ reconciles all rows, builds an accurate summary, and persists it (3 ms)
    ✓ counts per-invoice errors without aborting the whole run (3 ms)
    ✓ still returns a summary when persistence fails (insert error is swallowed) (1 ms)
    ✓ handles an empty invoice set (3 ms)
    ✓ inserts a row mapping summary fields to columns (4 ms)
    ✓ logs and swallows insert failures
    ✓ returns null when no run has been persisted
    ✓ maps the latest row back into a summary, parsing JSON results (1 ms)
    ✓ converts a Date reconciled_at to ISO and passes through object results (4 ms)
    ✓ returns null and logs when the DB read fails (2 ms)
    ✓ returns success with a summary on a clean run (default db path)
    ✓ returns a failure result when the run throws
    ✓ enqueues a reconcile_escrow job and returns its id (1 ms)
    ✓ covers both linked_escrow and the funded SQL states
    ✓ uses the production base-state stub when no adapter is injected (1 ms)
    ✓ accepts a bare numeric adapter return
    ✓ falls back to 0 for a non-finite on-chain value (1 ms)
    ✓ throws INVALID_INVOICE_ID for a malformed id
Test Suites: 1 passed, 1 total
Tests:       27 passed, 27 total
l

Coverage (new/changed code)

File	Lines	Funcs	Branches
src/jobs/reconcileEscrow.js	100%	100%	(default-param fallbacks only)
src/services/escrowRead.js → readFundedAmount	100%	100%	100% (9/9)

Run locally:

npm test -- tests/reconcileEscrow.test.js
npm run test:coverage

Notes for reviewers

• The query includes both the SQL invoice vocabulary (funded,
  partially_funded) and the state-machine vocabulary (linked_escrow) so it
  works regardless of which status set a deployment uses; RECONCILABLE_STATUSES
  is exported for easy tuning.
• The DB-side fundedTotal comes from escrow_summaries.total_funded (the
  cached DB record of funded amount). If your environment derives fundedTotal
  differently, adjust the join in iterateInvoicesFromDb.
• Out of scope / left untouched: a pre-existing bug in escrowRead.readEscrowState
  (unreachable code after an early return) — flagged separately rather than
  fixed here to keep this PR focused on Replace mock DB and mock on-chain reads in the nightly escrow reconciliation job (src/jobs/reconcileEscrow.js) #199.

[drips-wave]

@Emmanuel-abiola Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits