Harden financial correctness and recovery boundaries

.claude/AGENTS.md

@@ -30,19 +30,32 @@ work status; it does not override repository safety rules or the current user re
   scripts unless the user explicitly requests that external action.
 - Validate on-chain and provider payloads before persistence.
 - Preserve idempotency for transaction replay and financial background jobs.
+- Treat confirmed payouts and complete manifests as terminal states.
+- The raw-balance LP publisher is retired. Do not recreate it; token-token fee
+  ALGO is operational balance, not an economic reserve.
 - Do not silently substitute zero for unavailable chain state or price data.
 
 ## Quality Gate
 
 ```bash
 pipenv verify
-pipenv sync --dev
+make sync
 make quality
 ```
 
 CI validates Compose but does not certify the image as immutable-deploy ready:
-the private Node sidecar package-auth blocker is documented in `README.md`. Keep the
-focused lint, type, and coverage ratchets honest.
+it runs the full quality gate on Python 3.12 and 3.14, builds the digest-pinned
+Python-only image, smoke-tests the non-root runtime, scans it with Trivy, and
+checks financial persistence invariants against a disposable MongoDB service.
+Keep the focused lint, type, and coverage ratchets honest.
+
+`make run` uses the production-equivalent `python app.py` entrypoint, including
+critical indexes, migrations only when `MIGRATE=true`, and configured workers.
+Use `make run-api` only for API-only hot reload.
+
+Financial projector design and known standalone-Mongo limits are documented in
+`docs/architecture/lp-projection.md`; outbound payout recovery is documented in
+`docs/architecture/outbound-asset-transfers.md`.
 
 ## Cross-Project Changes
 

.env.example

@@ -10,6 +10,7 @@ REKEYED_MNEMONIC=
 ALGOD_ADDRESS=http://algod:8080
 ALGOD_TOKEN=
 ALGO_INDEXER_ADDRESS=https://mainnet-idx.algonode.cloud
+OUTBOUND_ASSET_TRANSFER_MAX_FEE_MICROALGOS=1000
 
 # API authentication
 API_PASSWORD=
@@ -21,6 +22,11 @@ MONGODB_USERNAME=
 MONGODB_PASSWORD=
 NEW_DB_NAME=cometa-updated
 
+# Financial projectors are opt-in until their chain classifiers are verified.
+SYNC_NEW_POOLS=true
+SYNC_LIQUIDITY_POOLS=false
+SYNC_STAKING_POOLS=false
+
 # Notifications
 # Syntactically valid, non-secret local placeholder; replace in production.
 TELEGRAM_BOT_API_TOKEN=123456:test-token
@@ -32,5 +38,7 @@ FARM_CREATION_FEE=0
 FARM_FLAT_ALGO_CREATION_FEE=0
 
 # Pricing resilience
+BACKGROUND_ASSET_PRICES_UPDATE=true
+BACKGROUND_LP_PRICES_UPDATE=false
 ASSET_PRICES_TTL=120
 ASSET_PRICES_MAX_STALE=3600

.github/workflows/ci.yml

@@ -13,18 +13,23 @@ permissions:
   contents: read
 
 jobs:
-  python:
+  python-matrix:
+    name: Python ${{ matrix.python-version }}
     runs-on: ubuntu-latest
     timeout-minutes: 15
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.12", "3.14"]
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
       - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
         with:
-          python-version: "3.12"
+          python-version: ${{ matrix.python-version }}
           cache: pipenv
       - run: python -m pip install --disable-pip-version-check pipenv==2024.4.0
       - run: pipenv verify
-      - run: pipenv sync --dev
+      - run: pipenv sync --dev --python "${{ matrix.python-version }}"
       - name: Lint maintained modules
         run: make lint
       - name: Check formatting
@@ -75,3 +80,75 @@ jobs:
             --severity HIGH,CRITICAL \
             --exit-code 1 \
             "cometa-backend:${COMETA_IMAGE_TAG:-local}"
+
+  secret-scan:
+    name: Secret history
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          fetch-depth: 0
+      - name: Fetch every published ref
+        run: git fetch --force --prune origin '+refs/heads/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*'
+      - name: Scan reachable Git history
+        run: |
+          docker run --rm \
+            -v "$PWD:/repo" \
+            ghcr.io/trufflesecurity/trufflehog@sha256:59b244249d1a1aef4baa24fe73d3c931616264482580d806d77f6c74d26b3e42 \
+            git file:///repo \
+            --results=verified,unknown \
+            --fail \
+            --fail-on-scan-errors \
+            --no-update \
+            --github-actions
+
+  mongo-integration:
+    name: MongoDB financial invariants
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    services:
+      mongodb:
+        image: mongo:7.0.28@sha256:4510cf3d7050003e958745adb25d2deb3fb907430716162d9cc1a92eda2a6047
+        ports:
+          - 27017:27017
+        options: >-
+          --health-cmd "mongosh --quiet --eval 'quit(db.adminCommand(\"ping\").ok ? 0 : 2)'"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 20
+    env:
+      MONGODB_TEST_URI: mongodb://127.0.0.1:27017
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
+        with:
+          python-version: "3.12"
+          cache: pipenv
+      - run: python -m pip install --disable-pip-version-check pipenv==2024.4.0
+      - run: pipenv sync --dev --python "3.12"
+      - name: Test financial repositories against MongoDB
+        run: pipenv run pytest tests/integration -m integration -v
+
+  python:
+    name: python
+    if: ${{ always() }}
+    needs:
+      - python-matrix
+      - configuration
+      - mongo-integration
+      - secret-scan
+    runs-on: ubuntu-latest
+    timeout-minutes: 2
+    steps:
+      - name: Require Python and MongoDB checks
+        env:
+          MATRIX_RESULT: ${{ needs.python-matrix.result }}
+          CONFIGURATION_RESULT: ${{ needs.configuration.result }}
+          MONGO_RESULT: ${{ needs.mongo-integration.result }}
+          SECRET_SCAN_RESULT: ${{ needs.secret-scan.result }}
+        run: |
+          test "$MATRIX_RESULT" = "success"
+          test "$CONFIGURATION_RESULT" = "success"
+          test "$MONGO_RESULT" = "success"
+          test "$SECRET_SCAN_RESULT" = "success"

.gitignore

@@ -107,13 +107,28 @@ celerybeat.pid
 .env.*
 !.env.example
 .env.main
+*.mnemonic
+*.seed
+*.pem
+*.key
+*.p12
+*.pfx
+*.jks
+*.keystore
 .venv
 env/
 venv/
 ENV/
 env.bak/
 venv.bak/
 
+# Database exports and operator backups
+dump/
+backups/
+*.bson
+*.dump
+*.sql.gz
+
 # Spyder project settings
 .spyderproject
 .spyproject
@@ -242,3 +257,14 @@ test-results/
 test_refund.py
 check_vestige_hack.py
 sample.py
+# Retired credential-bearing operator helper; history is scrubbed separately.
+/verify_pool.sh
+
+# Local operator scratch files; never publish from this checkout.
+/analyze_logs.sh
+/download_and_analyze_logs.sh
+/log_analyzer.py
+/docs/impeccable-analysis.md
+/docs/pagination-plan.md
+/scripts/recover_contracts.py
+/scripts/restore_missing.py

BOARD.md

@@ -1,23 +1,42 @@
 # Cometa Backend — Task Board
 
-> Last updated: 2026-07-18
+> Last updated: 2026-07-19
 
 ## Conventions
 
 - **ID format**: `CB-NNN` (sequential, never reuse)
 - **Statuses**: `todo` | `in_progress` | `blocked` | `done`
 - **Priorities**: `critical` | `high` | `medium` | `low`
 - **Tags**: `security` | `backend` | `infra` | `dx` | `arch` | `perf`
-- Next available ID: **CB-078**
+- Next available ID: **CB-097**
 
 ## Active
 
 | ID | Task | Status | Priority | Tags | Definition of done |
 | --- | --- | --- | --- | --- | --- |
 | CB-073 | Public repository hardening | in_progress | critical | security, dx | Credentials rotated, history sanitized, secret protection enabled, clean-clone scan passes |
-| CB-074 | Atomic event projection | todo | critical | backend, arch | Crash-safe inbox/projector with duplicate, replay, and recovery tests |
+| CB-074 | Atomic event projection | in_progress | critical | backend, arch | LP projector is crash-safe; replace the disabled legacy staking projector with verified grouped events and recovery tests |
 | CB-075 | Isolate transaction signing | todo | high | security, arch | Read-only API boundary; authenticated policy-limited signing service |
 | CB-076 | Async persistence boundary | todo | high | backend, perf | Storage outages cannot block the event loop; timeouts and readiness covered |
+| CB-078 | Replay-safe outbound asset payouts | done | critical | security, backend, arch | Exact allocations, immutable airdrop manifests, persisted signed intents, on-chain reconciliation, and regression tests |
+| CB-079 | Crash-safe LP projection | done | critical | backend, arch | Decimal128 balances, ordered per-state CAS cursor, fenced round checkpoint, snapshot guards, and crash/concurrency tests |
+| CB-080 | Financial read-model regressions | done | medium | backend | Reward decimals use the reward asset and request ordering never mutates cached contract state |
+| CB-081 | Fence expired sync workers | done | critical | backend, arch | A worker cannot commit a financial round after its lease expires, with unit and real-Mongo regressions |
+| CB-082 | Verify Mongo financial invariants | done | high | backend, infra | CI proves CAS replay, marker repair, BSON promotion, uniqueness, and lease fencing against a disposable pinned MongoDB |
+| CB-083 | Refresh public engineering docs | done | medium | dx, arch | Runtime commands, Python support, architecture boundaries, API shapes, and cross-project contract are current |
+| CB-084 | Disable unverified LP pricing | done | critical | backend, arch | Raw-account-balance publisher is removed; startup purges its legacy rows and every stored-price read rejects them |
+| CB-085 | Remove repository credibility drift | done | medium | dx, arch | Public claims match verified behavior, local sync selects Python 3.12, and the unrelated EVM sample is removed |
+| CB-086 | Reconcile legacy lottery payouts | done | critical | security, backend | Pre-intent lottery draws fail closed until manual reconciliation; new draws use durable payout states |
+| CB-087 | Preserve terminal payout states | done | high | backend, arch | Confirmed transfer intents and complete airdrop manifests cannot regress under stale concurrent workers |
+| CB-088 | Separate LP ledger from pricing | done | critical | backend, arch | Raw balances never publish prices; fees are replay-safe events and operational ALGO is isolated from reserves |
+| CB-089 | Migrate canonical asset supply | done | high | backend, arch | Financial reads trust only Indexer-provenanced base units, migrate atomically, and fail closed on duplicate asset IDs |
+| CB-090 | Claim staking lottery entitlements atomically | done | critical | backend, arch | Concurrent and crash-recovery paths converge on one draw generation in real MongoDB |
+| CB-091 | Bound outbound signer fees and network | done | critical | security, backend | Suggested and persisted transactions enforce a configured fee ceiling and canonical genesis before network I/O |
+| CB-092 | Validate Algorand numeric boundaries | done | high | backend, arch | Indexer events and snapshots reject coercion, negatives, duplicates, and uint64 overflow before persistence |
+| CB-093 | Reserve one-of-one lottery inventory | todo | high | backend, arch | A re-enabled lottery atomically reserves NFT inventory and reconciles release/finalization |
+| CB-094 | Replace retired LP pricing with verified adapters | todo | high | backend, arch | DEX-specific app state proves economic reserves; donation and excess-balance adversarial tests pass |
+| CB-095 | Reject poisoned price chronology | done | high | backend, arch | Future-dated quotes fail before persistence and a valid quote replaces legacy future timestamps |
+| CB-096 | Enforce standalone financial indexes | done | high | backend, arch | Operator airdrops and LP discovery require unique immutable business keys before concurrent upserts |
 
 ## Completed milestones
 
@@ -27,8 +46,9 @@
 | Precision-safe pricing | done | Decimal observations, provenance, freshness policy, guarded legacy boundary |
 | Provider resilience | done | Typed fallback errors, bounded stale data, retry classification, circuit breaker |
 | Replay identity | done | Deterministic nested event IDs and collection-level uniqueness constraints |
+| LP financial ledger | done | Marker-gap recovery, uint64-safe BSON operations, full-block preflight, snapshot coverage guards, and fenced round CAS |
 | Container baseline | done | Digest-pinned Alpine base, multi-stage non-root runtime, healthcheck, image exclusions, Trivy CI gate |
-| API hardening | done | Fail-closed header authentication, trusted hosts, explicit CORS policy, bounded LP/asset/wallet requests |
+| API request hardening | done | Fail-closed configured header checks, trusted hosts, explicit CORS policy, bounded selectors and wallet expansion |
 | Native Reach decoding | done | Versioned global/local codecs, exact-width integers, deterministic layout tests, no private npm runtime |
 | Legacy runtime removal | done | CB-077: Node/Reach sidecar and production source bind mount removed |
 

CLAUDE.md

@@ -4,7 +4,7 @@ Backend for Cometa — an Algorand DeFi platform handling liquidity pools, token
 
 ## Stack
 
-- **Language**: Python 3.12, Pipenv
+- **Language**: Python 3.12 production runtime; Python 3.14 compatibility CI; Pipenv
 - **Framework**: FastAPI + Uvicorn
 - **Database**: MongoDB (pymongo)
 - **Cache**: process-local TTL caches (Redis migration is roadmap work)
@@ -16,28 +16,30 @@ Backend for Cometa — an Algorand DeFi platform handling liquidity pools, token
 ## Project Structure
 
 ```
-app.py              — FastAPI application, routes, startup
+app.py              — FastAPI composition, routes, startup, and workers
 env.py              — Settings via pydantic-settings (from .env)
-api/                — Core API: background tasks, DB models, stats, swaps, wallets, NFT lottery
-blockchain/         — Algorand node/indexer interaction utilities
-core/               — Shared core logic
-dexes/              — DEX integrations (HumbleSwap, Vestige, etc.)
-flex/               — Flex module (API + data)
+api/                — Product API, background work, wallets, and disabled lottery surface
+blockchain/         — Algorand node/indexer adapters
+core/               — Shared authentication, persistence, and resilience
+dexes/              — DEX-specific integrations
+flex/application/   — Financial use-case orchestration
+flex/domain/        — Pure allocation, pricing, projection, and identity rules
+flex/db/            — Mongo models, BSON codecs, repositories, and indexes
+flex/tools/         — Operator tools such as manifest-driven airdrops
 bot/                — Telegram bot logic
-farcaster/          — Farcaster integration
-scripts/            — Deployment & management shell scripts
-airdrop/            — Airdrop tooling
-marketplaces/       — NFT marketplace integrations
-metapunks/          — MetaPunks-specific logic
+scripts/            — Deployment and management shell scripts
+tests/              — Unit tests plus opt-in real-service integration tests
+docs/               — Architecture decisions, operations, and audit reports
 ```
 
 ## Key Commands
 
 ```bash
 # Local development
 pipenv verify
-pipenv sync --dev
-pipenv run uvicorn app:app --reload --port 8000
+make sync
+make run       # production-equivalent startup, including indexes/workers
+make run-api   # API-only Uvicorn reload; no migrations or workers
 
 # Quality gate
 make quality
@@ -57,9 +59,22 @@ scripts/redeploy.sh     # pull + rebuild + restart the backend service
 - Background tasks in `api/background.py` — use exponential backoff for retries
 - Decode only explicitly supported Reach contract versions in `flex/blockchain/contract_state.py`
 - Route registration and process orchestration stay in `app.py`; Flex routes live in `flex/api.py`
-- MongoDB models in `api/db_model.py`
+- Legacy contract persistence models live in `core/db/model.py`; `api/db_model.py`
+  contains only the public contract-type enum. Maintained Flex models and
+  repositories live under `flex/db/`.
 - Asset prices use process-local TTL caches — see `dexes/` for provider calls
 - Preserve financial values as `Decimal` or integer base units until an explicit compatibility boundary
+- Persist maintained Flex financial `uint64` fields through the BSON codecs in
+  `flex/db/bson.py`; do not copy legacy int64/float compatibility shapes
+- Outbound transfers must persist immutable signed intent before broadcast and reconcile on-chain before completion
+- Legacy lottery draws without a matching durable operation remain `reconciliation_required`
+- LP events must enter through the complete-round preflight and `MongoLpProjectionRepository`
+- Keep token-token fee funding in `operational_algo_balance_micros`; never mix it into economic reserves
+- Trust `total_supply_micros` only with `total_supply_source=indexer`; otherwise
+  refetch and persist both fields atomically before a financial supply read
+- Keep `SYNC_STAKING_POOLS=false` until full Algorand application-group validation is implemented
+- Raw LP account balances never publish prices; `BACKGROUND_LP_PRICES_UPDATE`
+  is a retired compatibility setting and cannot restore the removed publisher
 - New pricing and transaction invariants belong in pure modules under `flex/domain/`
 - Run the strict mypy target before changing `core/circuit_breaker.py` or `flex/domain/`
 
@@ -91,8 +106,8 @@ See parent `~/dev/cometa/CLAUDE.md` for the canonical API contract table. That f
 
 ## Testing
 
-Tests are organized by boundary under `tests/unit/`. Run the same fast checks used
-by CI before committing:
+Fast tests are organized by boundary under `tests/unit/`; real-service checks live
+under `tests/integration/`. Run the same local quality gate used by CI before committing:
 
 ```bash
 make sync
@@ -103,7 +118,10 @@ make quality
 - Use `httpx.AsyncClient` with ASGI transport for endpoint integration tests
 - Keep domain tests pure and deterministic; inject clocks and provider functions
 - Priority: event replay/idempotency, price freshness, contract CRUD, authorization
+- Exercise crash boundaries, concurrent replay, BSON `uint64` limits, and
+  Indexer-lag cutovers for financial projections
 - Use a dedicated test database only for integration tests; never point tests at production
+- Set `MONGODB_TEST_URI` only to run the opt-in MongoDB integration suite
 - Every bug fix should include a regression test when practical
 
 ## Commit Discipline
@@ -118,9 +136,10 @@ make quality
 
 Tasks in `BOARD.md`. Format: pantheon.
 
-## Available MCP Tools
+## Optional Diagnostic Tools
 
-- **MongoDB MCP** — direct database queries in Claude sessions. Use for debugging: `db.contracts.find({active: true})`, inspecting collections, verifying data integrity. Connection: `mongodb://localhost:27017/cometa`
-- **Algorand MCP** — on-chain state verification, account info, asset lookups on mainnet
-- **Vestige MCP** — DEX price data, pool states, trading pairs for Algorand DeFi
-- **Codex MCP** — second-opinion code review via GPT-5.x. Run `review` after writing significant code
+MongoDB, Algorand, and DEX inspection connectors may be available in some agent
+sessions. Treat them as optional and read-only by default. Derive database and
+network targets from the active environment; never assume localhost is a safe
+database and never sign or broadcast a transaction without explicit task-scoped
+authorization.