diff --git a/BETA.md b/BETA.md index aebca13..557da57 100644 --- a/BETA.md +++ b/BETA.md @@ -47,7 +47,7 @@ You're ready when the frontend logs `Starting nginx`. ## 4. Your first bot (5 minutes, no exchange account needed) -1. Go to **Algorithms** and click **Load example strategy** (or *Import* and pick a file from the `examples/` folder in the repo — `Supertrend_Trend_1d.apex.json` is a good start). +1. Go to **Algorithms** and click **Load example strategy** (or *Import* and pick a file from the `examples/` folder in the repo — `Supertrend_Trend_1d.apex.json` is a good start; `Supertrend_LongShort_Perp_1d.apex.json` is the long/short perpetual version). 2. Leave the bot in **Paper** mode with *Run Historical Backtest* enabled. 3. Click **Start**. The engine downloads historical candles (a few minutes the first time) and runs a full backtest. Open the **Console** on the bot card to watch it work. 4. Explore the results in **Trades** (equity curve, drawdown, positions) and on the **Chart** (buy/sell markers). @@ -77,7 +77,20 @@ The repo contains `STRATEGY_CONTEXT.md`. Paste that file into any capable AI ass - **`Max order value` is mandatory for live bots** — the app refuses to start a live bot without this hard cap per order. Set it low. - Set **Max drawdown** (e.g. 10–15%): by default the bot then automatically closes its positions and stops if the equity curve drops that far from its peak. You can switch **On max drawdown** to *Block new entries* instead (exits keep working, no forced liquidation) — if you do, also set **Max capital loss %** as the hard stop, since blocking entries alone does not cap losses on open positions. - Fill in your exchange's real **fee** (e.g. 0.1%) in the trade settings — backtests without fees are misleadingly optimistic. -- After any backend restart the bot reconciles its open positions with your exchange balances before going live and stops with an error if they don't match — still glance at the exchange yourself before letting it continue. +- After any backend restart the bot reconciles its open positions with your exchange balances (or, on perpetual swaps, the exchange's position list) before going live and stops with an error if they don't match or cannot be checked — still glance at the exchange yourself before letting it continue. +- **`Max order value` is a cap on the quote notional** (e.g. USDT for `BTC/USDT`), not on the margin you put up — at 5× leverage a 500 cap lets the bot risk a 100 margin. It applies in backtest and forward test too, so changing it counts as a new strategy variant. + +### Perpetual swaps: known limitations of the simulation + +Backtest and forward test model leverage, margin, fees, slippage, liquidation, funding and the exchange's maintenance-margin tiers, with these caveats: + +- **Funding history** — the bot fetches the funding-rate history when it starts and stores it next to the candles, but exchanges only serve a limited window (OKX ≈ 3 months, Binance years). Older parts of a backtest run without funding; the console and the summary (`funding: simulated | partial | no data`) say so. Paper/live positions are charged by the exchange itself and not booked by ApexAlgo. +- **Maintenance-margin tiers** — fetched once a week per pair; without them (exchange unreachable, no tier endpoint) the flat 0.5% applies (`mmr_source` in the summary). +- **Cross margin** — simulated as one account: `backtest_capital` is the whole margin wallet, a breach liquidates every open position of the bot and empties that pool (`profit_pct` can go below −100). On the exchange the *real* wallet — including balances of other bots and keys — is what gets liquidated. +- **Spot margin / borrowing** — not supported; spot bots only ever spend the cash they hold. +- **Hedge mode** — not supported; keep the account in one-way position mode. The bot never holds a long and a short on the same pair at once, and reconciliation nets both sides into one number, so hedged positions would be misread. + +Whatever you do on the exchange yourself (manual trades, changing the leverage, switching position mode) is invisible to the bot until the next restart. ### If you suspect a leak or anything weird diff --git a/README.md b/README.md index 8a47e1d..3b45bb1 100644 --- a/README.md +++ b/README.md @@ -74,9 +74,11 @@ https://github.com/user-attachments/assets/25d03926-1e88-4749-88d4-b97b14ff9c2b - **Async Event-Driven Architecture** — FastAPI backend with concurrent bot management via asyncio - **Multi-Exchange Market Data** — universal REST polling via CCXT; each `(exchange, symbol, timeframe)` gets its own independent polling stream - **CCXT Integration** — exchange-agnostic order execution with market precision handling +- **Perpetual swaps with leverage, long and short** — a bot trades spot or perpetuals (1–10×, isolated or cross margin), **linear** (stablecoin-settled, `BTC/USDT:USDT`) or **inverse** (coin-margined, `BTC/USD:BTC`); the kind is a property of the symbol, not of the key. On perpetuals the strategy can open shorts (`SHORT` / `COVER` action blocks) with mirrored stop-loss/take-profit rules; backtest, forward test and live share the same margin/liquidation model, and spot bots are untouched +- **One cash currency per bot** — every pair of a whitelist must be quoted (spot) or settled (swap) in the same currency (validator-enforced), so a bot can run in USDT, USDC, EUR, USD or BTC. Money is always shown in that currency and never summed across currencies ### Multi-Exchange Support -- **7 Exchanges out of the box** — OKX, Binance, Bitvavo, Coinbase, Crypto.com, Kraken, KuCoin +- **13 Exchanges out of the box** — OKX, Binance, Bitvavo, Coinbase, Crypto.com, Kraken, KuCoin, Bybit, Gate, Bitget, MEXC, HTX, BingX; perpetual swaps on nine of them (linear and inverse contracts, BingX linear only; Binance COIN-M runs through ccxt's `binancecoinm` class) - **Isolated data streams** — bots on different exchanges poll independently and store candles separately; no cross-exchange data mixing - **Exchange Registry** — centralized `exchange_registry.py` handles per-exchange config (OKX EU hostname, passphrase exchanges, sandbox modes) - **Automatic migration** — existing databases are upgraded non-destructively on startup; all historical data is preserved @@ -93,7 +95,7 @@ https://github.com/user-attachments/assets/25d03926-1e88-4749-88d4-b97b14ff9c2b - **Incremental Backfill Commits** — candle data is committed to the database after each exchange batch, not all at once; eliminates startup race conditions when multiple bots start simultaneously - **Indicator Fingerprinting** — bots sharing the same indicator configuration reuse computed results via MD5-based fingerprint keys, avoiding redundant pandas_ta calls in the live processing loop - **Evaluator Memoization** — `resolve_node()` caches resolved Series per evaluation cycle so diamond-shaped node graphs don't recompute shared indicator nodes -- **Drawdown Caching** — drawdown is tracked per `(bot, mode_group)` with separate backtest and live caches. Lazy-initialized from DB, updated incrementally on position close +- **Drawdown Caching** — drawdown is tracked per `(bot, mode_group)` with separate `backtest`, `forward` (forward test) and `live` (paper + live) caches, so a simulated forward loss never counts against the real-money curve. Lazy-initialized from DB, updated incrementally on every closed leg - **Backfill Lock** — `_backfilling_bots` set prevents live processing from creating duplicate signals while a bot is mid-backfill - **Signal Deduplication** — unique constraint on `(bot_name, symbol, timestamp)` with `INSERT OR IGNORE` prevents duplicate signals on bot restart - **Incremental Signal Polling** — ChartEngine uses `since_id` to fetch only new signals after initial load, reducing per-poll payload from thousands of rows to near-zero @@ -110,8 +112,9 @@ https://github.com/user-attachments/assets/25d03926-1e88-4749-88d4-b97b14ff9c2b - **Container Resource Limits** — docker-compose sets CPU/memory caps per service to prevent resource starvation ### Backtesting -- **Shared Capital Pool** — `backtest_capital` is a single pool shared across all whitelist pairs. When BTC uses $140, only the remainder is available for ETH/SOL/XRP. Capital is locked on position open and sale proceeds returned on close -- **Dynamic Trade Sizing** — trade amounts are calculated from running equity, not static starting capital. As capital depletes, position sizes shrink proportionally. Trading halts when equity reaches zero +- **Shared Capital Pool** — `backtest_capital` is a single pool in the bot's cash currency, shared across all whitelist pairs (`engine/capital.py`, `CapitalPools`). When BTC uses 140 of it, only the remainder is available for ETH/SOL/XRP. An entry locks margin plus the entry fee; on close the proceeds (spot) or margin + PnL − exit fee (derivatives) return to the pool +- **Dynamic Trade Sizing** — `percentage` sizing is a percentage of the **free cash** left in the pool (not of mark-to-market equity), so sizes compound across pairs and shrink as capital is locked. A `fixed` amount is a cash amount in the bot's currency. `max_order_value` (a quote-currency notional) caps every entry in the backtest, forward test and live alike and is therefore part of the strategy fingerprint. Trading halts when the pool reaches zero +- **Honest returns** — `profit_pct` is the return on the capital a trade had locked (margin + entry fee), in the backtest, forward test, live tick and manual close alike; liquidation is modelled in the backtest and forward test and detected live - **Capital Depletion Guard** — before opening any position, the engine verifies sufficient capital. No phantom-money trades - **Post-Backtest Drawdown Gate** — backtest always runs to completion; max drawdown is evaluated over all closed positions afterward. If the threshold is exceeded, the bot is stopped and not allowed to go live - **Vectorized Historical Evaluation** — fast backtest over configurable lookback periods with numpy-backed arrays @@ -125,10 +128,10 @@ https://github.com/user-attachments/assets/25d03926-1e88-4749-88d4-b97b14ff9c2b ### Risk Management - **Tiered Take Profit / Stop Loss** — multiple TP/SL levels with percentage or fixed close amounts - **ATR & Trailing Stops** — dynamic stop-loss adjustment based on price action -- **Trade Cooldown** — configurable max entries per N candles +- **Trade Cooldown** — configurable max entries per N candles, counted per symbol and in candles (the window starts at the N-th most recent stored candle, in the backtest and live alike) - **Position Limits** — per-pair or global max concurrent positions -- **Max Drawdown Guard** — evaluated after full backtest to gate live entry; during live trading, checked after every closed position. Default action `close_all` closes every position and stops the bot; opt-in `block_entries` pauses new entries until drawdown recovers below half the limit — or the bot has been flat for a configurable cooldown (default 7 days), after which the peak resets — while exits keep running (simulated identically in the backtest). A separate **Max Capital Loss** guard (loss of starting capital) is the hard stop and follows the same action: close everything immediately, or wind down (no new entries, exits finish, then stop) -- **Max Order Value Guard** — rejects live orders exceeding a configurable USD limit +- **Max Drawdown Guard** — evaluated after full backtest to gate live entry; during live trading, checked mark-to-market on every candle (forward-test and real-money curves are tracked separately). Default action `close_all` closes every position and stops the bot; opt-in `block_entries` pauses new entries until drawdown recovers below half the limit — or the bot has been flat for a configurable cooldown (default 7 days), after which the peak resets — while exits keep running (simulated identically in the backtest). A separate **Max Capital Loss** guard (loss of starting capital) is the hard stop and follows the same action: close everything immediately, or wind down (no new entries, exits finish, then stop) +- **Max Order Value Guard** — caps every entry to a configurable quote-currency notional (USD on `BTC/USD:BTC`, EUR on `BTC/EUR`); mandatory for live bots, applied in the backtest and forward test too ### Order Safety - **Order Fill Validation** — verifies exchange order status after every CCXT call @@ -144,8 +147,9 @@ https://github.com/user-attachments/assets/25d03926-1e88-4749-88d4-b97b14ff9c2b ### Analytics - **Equity Curve** — inline SVG cumulative P&L chart over time (no external chart library) -- **Buy & Hold Comparison** — per-symbol strategy return vs. passive buy-and-hold; strategy % is `total_pnl / backtest_capital * 100` using the same capital base as B&H for fair comparison; reference price anchored to the true first entry across all positions (open and closed) +- **Buy & Hold Comparison** — per-symbol strategy return vs. passive buy-and-hold; strategy % is `total_pnl / backtest_capital * 100` using the same capital base as B&H for fair comparison; reference price anchored to the true first entry across all positions (open and closed); live prices come from the bot's own data exchange - **8-Metric Stats Strip** — Net P&L, Win Rate, Profit Factor, Max Drawdown (percentage of peak equity using backtest_capital), Avg Hold Time, Total Fees, Return/Risk ratio +- **Per-currency money** — every amount is printed in the currency it is denominated in (`1,234.50 USDT`, `€1,234.50`, `0.0213 BTC`) and a currency filter scopes the view; with several currencies (or several modes) in view the money tiles switch to per-currency chips instead of summing. Open positions, the ledger and the execution log carry a Side column (long/short) and label orders `BUY / SELL / SHORT / COVER`; a short is force-closed with "Buy to cover" - **Avg Hold Time** — computed from entry order timestamps as fallback for backtests where position `created_at` reflects wall-clock run time rather than the candle entry time - **Exchange Filter** — filter all analytics sections by exchange, bot, symbol, or execution mode - **Real-Time Charts** — TradingView lightweight-charts with indicator overlays on correct axis scales @@ -229,7 +233,7 @@ docker compose build && docker compose up -d ### Tests and lint -The backend test suite runs against a throw-away SQLite file (your `data/` database is never touched) and covers the exit-rule table, a full live tick against a mocked exchange, backtest/live parity, the routers, and **golden backtests**: every example strategy is run on deterministic synthetic candles and its order stream is compared with `tests/golden/*.json`, so an engine change can never silently alter a strategy's trades. +The backend test suite (244 tests) runs against a throw-away SQLite file (your `data/` database is never touched) and covers the exit-rule table, a full live tick against a mocked exchange, swaps and shorts, contract economics (`test_contracts.py`), a **parity suite** (`test_parity_modes.py`: the same candles through backtest, forward test and a live mock, long/short, 1×/5×), **property tests** with hypothesis (`test_properties.py`: PnL sign, liquidation monotone in leverage, sizing never exceeds the pool), the routers, and **golden backtests**: every example strategy (plus a synthetic long/short swap fixture) is run on deterministic synthetic candles and its order stream is compared with `tests/golden/*.json`, so an engine change can never silently alter a strategy's trades. ```bash pip install -r requirements-dev.txt @@ -366,9 +370,12 @@ ApexAlgo/ │ │ ├── backtest.py # Chronological multi-pair simulation against one capital pool (pyramiding, MTM drawdown) │ │ ├── live_cycle.py # One live candle: entries, exits, signal batch, close-all │ │ ├── startup.py # Bot start: data wait/backfill, backtest, drawdown gate, reconciliation, go-live -│ │ ├── exits.py # SL/TP/trailing/ATR/multi-tier exit rule table (shared by backtest and live) -│ │ ├── risk.py # Drawdown / capital-loss tracking, unrealized PnL -│ │ ├── sizing.py # Trade sizing, capital pools, config fingerprints +│ │ ├── exits.py # SL/TP/trailing/ATR/multi-tier exit rule table, one direction-parameterised body for longs and shorts +│ │ ├── contracts.py # ContractSpec: spot / linear / inverse economics (notional, margin, PnL, liquidation, fees) in the cash currency +│ │ ├── capital.py # CapitalPools: per-currency cash/locked/start pool (one currency per bot) +│ │ ├── pnl.py # Side-aware seam: direction, price_pnl, liquidation_price, locked_capital, order sides +│ │ ├── risk.py # Drawdown / capital-loss tracking per (bot, backtest|forward|live group), unrealized PnL +│ │ ├── sizing.py # Trade sizing, max_order_value cap, forward pool, config fingerprints │ │ ├── broker.py # ccxt plumbing: order reconciliation, cancel, min-notional, wallet checks │ │ ├── candle_poller.py # Universal multi-exchange REST polling with incremental backfill │ │ ├── evaluator.py # Node graph resolver using pandas_ta (memoized per evaluation cycle) @@ -388,6 +395,8 @@ ApexAlgo/ │ └── src/ │ ├── api/ # Axios client (same-origin by default), error humanizer │ ├── theme.js # Light/dark theme state + token access +│ ├── utils/money.js # fmtMoney(value, ccy), per-currency sums, cash currency / contract kind of symbols and rows +│ ├── utils/orders.js # Side-aware order helpers (isShortOpen, isCover, orderAction, modeTag) │ ├── examples/ # One-click loader; imports the strategies from the root examples/ │ └── components/ │ ├── Builder/ # Visual strategy editor (BotBuilder, CustomNodes, indicatorConfig) @@ -413,13 +422,13 @@ ApexAlgo/ ├── examples/ # Verified importable strategies (.apex.json) — results in STRATEGY_CONTEXT.md §4.9 ├── scripts/ │ └── verify_examples.py # Re-runs the examples on real exchange candles in a temp DB (reproduces §4.9) -├── tests/ # pytest suite: exit rules, live tick with exchange mock, parity, routers, golden backtests -│ └── golden/ # Pinned order streams of the example strategies on synthetic candles +├── tests/ # pytest suite: exit rules, live tick with exchange mock, swaps/shorts, contracts, parity, hypothesis properties, routers, golden backtests +│ └── golden/ # Pinned order streams of the example strategies (+ a synthetic swap long/short fixture) on synthetic candles ├── .github/workflows/ci.yml # pytest + ruff + npm lint/build on push and PR ├── docker-compose.yml # Two services; ports on 127.0.0.1 by default (BIND_ADDR opt-in) ├── data/ # Database, .env, SSL certs (gitignored) ├── requirements.txt -├── requirements-dev.txt # pytest, ruff, httpx (tests + lint) +├── requirements-dev.txt # pytest, ruff, httpx, hypothesis (tests + lint) ├── BETA.md # Beta tester guide (setup, safety rules, troubleshooting) └── STRATEGY_CONTEXT.md # AI context: builder reference + .apex.json import schema ``` @@ -517,17 +526,50 @@ Click **Duplicate** on any stopped bot card to create a clone with `(copy)` appe ## Supported Exchanges -| Exchange | Passphrase | Sandbox | Notes | -| :--- | :--- | :--- | :--- | -| OKX | Yes | Yes | EU hostname (eea.okx.com) | -| Binance | No | Yes | | -| Bitvavo | No | No | EU exchange | -| Coinbase | No | No | | -| Crypto.com | No | No | | -| Kraken | No | No | | -| KuCoin | Yes | Yes | | +| Exchange | Passphrase | Spot sandbox | Perpetual swaps | Contract kinds | Notes | +| :--- | :--- | :--- | :--- | :--- | :--- | +| OKX | Yes | Yes | up to 10× (sandbox) | linear, inverse | EU hostname (eea.okx.com); net (one-way) position mode | +| Binance | No | Yes | up to 10× (sandbox) | linear, inverse | USDⓈ-M and COIN-M futures (COIN-M via ccxt `binancecoinm`), one-way position mode | +| Bitvavo | No | No | — | — | EU exchange | +| Coinbase | No | No | — | — | | +| Crypto.com | No | Yes | — | — | UAT environment | +| Kraken | No | No | up to 5× (demo) | linear, inverse | Kraken Futures uses its own keys (futures.kraken.com / demo-futures.kraken.com) | +| KuCoin | Yes | No | up to 10× | linear, inverse | KuCoin Futures uses its own keys (Futures → API management) | +| Bybit | No | Yes | up to 10× (testnet) | linear, inverse | Unified trading account, one-way position mode | +| Gate | No | Yes | up to 10× (testnet) | linear, inverse | | +| Bitget | Yes | Yes | up to 10× (demo) | linear, inverse | USDT-M futures, one-way position mode | +| MEXC | No | No | — | — | Futures API is closed to the public | +| HTX | No | No | up to 10× | linear, inverse | | +| BingX | No | Yes | up to 10× (demo) | linear | USDT-margined perpetuals, one-way position mode | -Adding support for any other CCXT-compatible exchange requires only adding it to the frontend dropdowns and `SUPPORTED_EXCHANGES` in `exchange_registry.py`. +Adding another CCXT-compatible exchange is one `ExchangeSpec` entry in `backend/core/exchange_registry.py`; the key form, the data manager and the builder read the list from `GET /api/keys/exchanges`. `GET /api/data/symbols/{exchange}?market_type=swap` returns the tradeable symbols plus a `markets` map with `{kind, base, quote, settle, contract_size, cash_currency}` per symbol. + +### Cash currency + +Every bot keeps its books in exactly one **cash currency**: the quote of a spot pair (`BTC/EUR` → EUR) or the settle currency of a perpetual (`BTC/USDT:USDT` → USDT, `BTC/USD:BTC` → BTC). All pairs of a whitelist must share it (the validator rejects `BTC/USDT` + `ETH/BTC`), `backtest_capital` is read in it, and positions, orders, the backtest summary and the bot summary all carry `cash_currency` so the UI can print `1,234.50 USDT`, `€1,234.50` or `0.0213 BTC` — amounts are never summed across currencies. Orders additionally record `fee_currency` / `fee_cash` (what the exchange actually charged; `fee` is always converted to the cash currency), and `GET /api/trades/stats` groups its money fields in `by_currency` (the top-level fields carry the single-currency case; with several currencies in view they are `null`). Wallet valuation on the key page stays a display-only USD estimate (`valuation_currency: "USD"`). + +### Perpetual swaps + +An API key is bound to one market: save a second key with **Market: Perpetual swaps** for derivatives (the same exchange login, with futures/derivatives trade permission; Kraken and KuCoin issue separate futures keys). A bot on a swap key trades the `BASE/QUOTE:SETTLE` symbols with the leverage and margin mode set in the builder's Exchange Routing block (1–10×, isolated by default). Two contract kinds exist, decided per symbol (the whitelist shows a `linear` / `inverse` chip): + +- **Linear** (`BTC/USDT:USDT`): settled in the quote stablecoin; `amount` is a base amount, PnL = `±(exit − entry) × amount` — the same arithmetic as spot. +- **Inverse** (`BTC/USD:BTC`, coin-margined): margin and PnL are in the base coin; `amount` is a number of contracts worth `contract_size` USD each, PnL = `±contracts × contract_size × (1/entry − 1/exit)`. Positions store `contract_kind` / `contract_size` so a later contract-size change on the exchange never rewrites history. + +The engine confirms leverage and margin mode on the exchange before the first order and refuses to start when that fails; on start-up, open swap positions are reconciled against `fetch_positions` instead of the wallet, and a position the exchange no longer holds is booked as a liquidation rather than managed blind. A reconciliation that cannot be performed stops a real-money bot (a sandbox bot only warns). + +Economics are identical across backtest, forward test and live: an entry locks `notional / leverage` plus the entry fee on the notional, PnL is on the full notional, `profit_pct` is measured on that locked capital. The liquidation level is `entry × (1 − (1 − 0.5%) / leverage)` for a linear long, `entry × (1 + (1 − 0.5%) / leverage)` for a linear short (a 1× long has none, a 1× short liquidates at +99.5%), and `entry × lev / (lev + 1 − 0.5%)` / `entry × lev / (lev − 1 + 0.5%)` on inverse contracts. Stop-losses and take-profits are evaluated first; only when nothing fills — or the fill would lie beyond the liquidation price — is the layer liquidated for its margin plus the entry fee already paid, without exit fee (`liquidation` in the exit list, `liquidations` in the backtest summary). The backtest and the forward test apply this rule on every candle; live, a liquidation is booked when the candle range reaches the level or when a rejected reduce-only close is followed by an empty `fetch_positions`. The 0.5% is the fallback: when the exchange's maintenance-margin tiers are stored (`leverage_tiers`, fetched at bot start, refreshed weekly) each position uses the rate of its bracket (`mmr_source: "tiers"` in the summary). Funding is simulated from the stored funding-rate history (`funding_rates`, fetched at bot start for the backtest window and topped up by the poller): every settlement between a position's open and close is charged on its notional at that candle's close — a long pays a positive rate, a short receives it — booked into `profit_abs`/`funding_paid` (summary `funding: simulated|partial|no data`, `funding_paid`, `funding_events`). With `margin_mode: cross` there is no level per position: the bot's whole pool backs every open position and a breach of the account's maintenance margin liquidates all of them and empties the pool. Paper/live positions are charged and liquidated by the exchange itself. `Max Order Value` caps the **quote notional** of an entry (margin × leverage) on every market kind. Exchanges without a swap testnet (KuCoin, HTX) can only forward test or trade real money. + +### Shorts + +On a perpetual market the strategy graph can also open shorts. The action block's direction has four options: **BUY** (open long), **SELL** (close long), **SHORT** (open short) and **COVER** (close short). A SHORT block carries its own TP/SL ports and entry size (`trade_settings.short`, falling back to the long entry settings when the block is not connected), COVER sizes like SELL (`trade_settings.cover`). Both are refused on a spot bot by the validator — there is no silent no-op. + +Rules of the road: + +- Long and short never coexist on one pair: a BUY while a short is open (or a SHORT while a long is open) is ignored with an INFO line in the bot console; when BUY and SHORT fire on the same candle the BUY wins. Pyramiding (`Max Positions`) counts every layer on the pair regardless of side. +- Exit rules are mirrored: a short stop-loss sits *above* the entry and is hit by the candle high, a take-profit *below* and is hit by the low; trailing and ATR rules anchor to the lowest price reached. A gap through a level fills at the open, as for longs. +- PnL is `(entry − exit) × amount` minus fees on a linear contract (the inverse formula above on coin-margined ones), slippage works against the trade (sold below the close on entry, bought above it on cover). A short is liquidated when the candle high reaches `entry × (1 + (1 − 0.5%) / leverage)` — even at 1×, where that is a +99.5% move — losing its margin. +- Live orders: a short opens with a market **sell** (not reduce-only) and closes with a reduce-only **buy**; start-up reconciliation compares the side of every open position with the exchange. The backtest summary adds `long_trades` / `short_trades`, the Analytics *Trades* tile splits PnL per side, and the chart marks shorts (`S-SH` / `T-SHORT`) and covers (`S-CV` / `T-COVER`). +- A ready-made long/short template ships in `examples/Supertrend_LongShort_Perp_1d.apex.json` (also under *Load example strategy*): long while the daily Supertrend points up, short while it points down, 1×, BTC+ETH USDT perps — +50.8% / 29.1% max DD over Dec 2023 → Sep 2026 with 26 long and 23 short trades; the parameter neighbours and the 2× version are in STRATEGY_CONTEXT.md §4.9. ### Historical data per exchange @@ -557,7 +599,7 @@ Found a security vulnerability? Please **do not** open a public issue — follow - The web UI never embeds or stores the master key: you enter it once in the login screen and receive an `HttpOnly`, `SameSite=Strict`, `Secure` session cookie that JavaScript cannot read; sessions end on backend restart or log-out. The browser talks to a single origin (nginx proxies `/api` to the backend) - Ports bind to `127.0.0.1` by default; LAN access is an explicit opt-in (`BIND_ADDR=0.0.0.0`) - The backend container runs as a non-root user; `.env` is created with restrictive permissions and excluded from version control -- Live order execution requires a `max_order_value` safety cap, sizes against the verified exchange balance, and reconciles every order fill (`fetch_order`) before booking +- Live order execution requires a `max_order_value` safety cap (quote notional per entry), sizes against the verified exchange balance in the bot's cash currency, and reconciles every order fill (`fetch_order`) before booking - Live sizing is wallet-based: each bot deploys up to `live_allocation_pct` of the exchange wallet (free balance + positions already open on that key), so multiple bots can share one API key by splitting the percentage - Sandbox-flagged keys refuse to run on exchanges without a real testnet - Swagger/OpenAPI docs are disabled by default; a Content-Security-Policy is set on the web UI diff --git a/STRATEGY_CONTEXT.md b/STRATEGY_CONTEXT.md index 2a2bcff..eb42a50 100644 --- a/STRATEGY_CONTEXT.md +++ b/STRATEGY_CONTEXT.md @@ -2,17 +2,17 @@ > Paste this file into any AI assistant (or point it at the raw GitHub URL) and ask for a strategy. Everything it produces will then be **directly importable** into ApexAlgo (Bot Manager → Import) and, more importantly, will be designed around how the engine actually trades — not around indicator folklore. -**If you are the AI reading this:** you are designing a real trading system that will run on real money. Sections 1–3 tell you how the engine executes; section 4 is the design playbook you must follow; section 5 is the exact file format; section 6 has three vetted templates to start from. Work through the checklist at the end of section 4 before you output anything. +**If you are the AI reading this:** you are designing a real trading system that will run on real money. Sections 1–3 tell you how the engine executes; section 4 is the design playbook you must follow; section 5 is the exact file format; section 6 has four vetted templates to start from (three long-only spot, one long/short perpetual). Work through the checklist at the end of section 4 before you output anything. --- ## 1. What ApexAlgo is — and how it executes -ApexAlgo is a no-code, **long-only spot** trading bot. A strategy is a graph of nodes: +ApexAlgo is a no-code trading bot for **spot** markets and **perpetual swaps** (leverage 1–10×) — linear contracts settled in a stablecoin (`BTC/USDT:USDT`) or inverse, coin-margined contracts settled in the base coin (`BTC/USD:BTC`). Every bot keeps its books in one **cash currency**: the quote of its spot pairs (USDT, USDC, EUR, …) or the settle currency of its perpetuals; all pairs of a bot must share it, and `backtest_capital`, `fixed` sizes, PnL and the guards are all in that currency. The default and the vetted templates are long-only spot; on a perpetual the graph may also short. A strategy is a graph of nodes: ``` -Indicator / Price Data ──> Condition ──> Logic Gate (optional) ──> Action (BUY / SELL) - └──> Take Profit / Stop Loss nodes (attached to the BUY action) +Indicator / Price Data ──> Condition ──> Logic Gate (optional) ──> Action (BUY / SELL, and on perps SHORT / COVER) + └──> Take Profit / Stop Loss nodes (attached to the BUY or SHORT action) ``` The engine evaluates the whole graph once per **closed candle**, per pair. Knowing exactly what happens on that tick is what separates strategies that look good from strategies that make money: @@ -21,15 +21,17 @@ The engine evaluates the whole graph once per **closed candle**, per pair. Knowi |---|---| | **Evaluation moment** | On the close of each candle. `offset: 0` on a price node means *the candle that just closed*, not a still-forming candle. | | **Entry fill** | At the close price of the signal candle, plus `slippage` %. Fee is charged on the notional. | -| **Direction** | Long only. `BUY` opens, `SELL` closes. There is no shorting and no leverage. | +| **Direction** | `BUY` opens a long, `SELL` closes it. On a perpetual market (`market_type: swap`) `SHORT` opens a short and `COVER` closes it; long and short never coexist on a pair (the conflicting signal is ignored, BUY wins when both fire). Spot bots cannot short (validator error). Leverage only on swaps (`leverage`, default 1). | | **Positions per pair** | **Pyramiding up to `max_positions`.** A BUY signal while a position is already open on that pair opens another layer until the limit is reached (`per_pair` = per symbol, `global` = across all pairs of the bot). Each layer has its own SL/TP levels (anchored to its own entry) and is exited independently; a strategy **SELL signal flattens every open layer on that pair**. A layer opened on a candle is not exit-checked until the next candle. `max_positions: 1` gives the classic one-position-per-pair behaviour. Backtest and live apply the same rule. | | **Re-entry** | The moment a position closes, the next candle whose entry condition is true opens a new one. A *state* condition (e.g. `RSI < 30`) that stays true for 10 candles will therefore re-enter immediately after every exit — see §4.2. | -| **Exit checks** | Every candle while a position is open, in this order: **stop-losses** (against the candle low) → **take-profits** (against high/low) → **strategy SELL signal** (at close). Only one group fires per candle; a hit stop-loss suppresses take-profits on the same candle. | -| **Trailing anchor** | Trailing levels use the highest high reached **before** the current candle, so one candle cannot both raise the trail and trigger it against its own low. | +| **Exit checks** | Every candle while a position is open, in this order: **stop-losses** (against the candle low) → **take-profits** (against high/low) → **strategy SELL signal** (at close). Only one group fires per candle; a hit stop-loss suppresses take-profits on the same candle. For a short every level is mirrored: the stop sits above the entry and is hit by the high, the target below and is hit by the low, the strategy exit is the COVER signal. | +| **Trailing anchor** | Trailing levels use the highest high (lowest low for a short) reached **before** the current candle, so one candle cannot both raise the trail and trigger it against its own low. | +| **Leverage (swaps)** | An entry locks `notional / leverage` plus the entry fee on the notional; PnL is on the full notional. `profit_pct` is the return on that locked capital (margin + entry fee) — at 5× a 2% price move is a ~10% `profit_pct`. Linear contracts: PnL = `±(exit − entry) × amount`. Inverse contracts (`BTC/USD:BTC`): `amount` is contracts of `contract_size` USD, PnL = `±contracts × contract_size × (1/entry − 1/exit)` in the base coin. Funding settlements stored for the pair are charged on open positions (a long pays a positive rate, a short receives it; `funding_paid` per position, `funding` status in the summary). | +| **Liquidation (swaps)** | Linear: a long is liquidated when the candle low reaches `entry × (1 − (1 − 0.5%) / leverage)`, a short when the high reaches `entry × (1 + (1 − 0.5%) / leverage)`; a 1× long never liquidates, a 1× short liquidates at +99.5%. Inverse: `entry × lev / (lev + 1 − 0.5%)` (long) and `entry × lev / (lev − 1 + 0.5%)` (short). Order on a candle that reaches the level: SL/TP are evaluated **first**; a stop that fills before the level (e.g. at the open) is a normal exit. Only when no exit fills, or the fill would be at or beyond the liquidation price, is the layer liquidated — the margin and the entry fee are lost, no exit fee. Same rule in backtest, forward test and live. The 0.5% is replaced by the pair's exchange tier rate when tiers are stored (`mmr_source`). With `margin_mode: cross` the whole bot pool backs every position: nothing liquidates per position, but when the account's equity falls to its total maintenance margin every open position is liquidated and the pool is gone. | | **Gaps** | If a candle opens beyond the trigger, the fill is at the open (worse for stops, better for targets). | | **Partial exits** | `close_amount_value` is a % of the *original* position. Each TP/SL tier fires once. A 100% tier closes whatever is left. | -| **Sizing** | `amount_type: percentage` = % of the **currently available cash** in the shared pool (all pairs of the bot share one pool). Capital is locked while a position is open, so 50% sizing on two pairs = fully invested. `fixed` = fixed USD. | -| **Cooldown** | `cooldown_trades` new entries per `cooldown_candles` candles, per bot (all pairs). | +| **Sizing** | `amount_type: percentage` = % of the **free cash remaining** in the shared pool (all pairs of the bot share one pool; free = not locked in open positions, *not* mark-to-market equity). Sizes compound: with 50% on two pairs the first entry takes 50% of the pool, the second 50% of what is left (25%), so the bot is never fully invested by percentage sizing alone. Entries are clamped so margin + entry fee fit the pool and to `max_order_value` (quote notional). `fixed` = a fixed amount in the bot's cash currency. | +| **Cooldown** | `cooldown_trades` new entries per `cooldown_candles` candles, counted **per symbol** and in candles (the window is the last N stored candles of that pair, in the backtest and live alike). | | **Warm-up** | Any indicator that is still NaN makes its condition *false*, never true. A 200-EMA eats the first 200 candles of the lookback. | | **Backtest ≙ live** | Paper/live trading runs the same node graph and the same exit rules on the same closed candles. What differs is fills (real order book) and sizing (see `live_allocation_pct`). | | **Guards** | `max_drawdown` (peak-to-trough on mark-to-market equity) and `max_capital_loss` (loss of starting capital) stop or wind down the bot — in the backtest *and* live. A backtest that breaches the guard prevents the bot from going live. | @@ -41,7 +43,7 @@ The engine evaluates the whole graph once per **closed candle**, per pair. Knowi ### 2.1 Configuration nodes (one of each) **Main Configuration** — name, timeframe (must be supported by the exchange, §5.3), `max_positions` + scope, cooldown, `max_drawdown`, `max_capital_loss`, `drawdown_action`, `drawdown_cooldown_days`, `max_order_value`, `live_allocation_pct`, execution mode. -**Asset Whitelist** — pairs as `BASE/QUOTE`, e.g. `BTC/USDC, ETH/USDC`. All pairs share one capital pool. +**Asset Whitelist** — pairs as `BASE/QUOTE`, e.g. `BTC/USDC, ETH/USDC` (perpetuals as `BASE/QUOTE:SETTLE`). All pairs share one capital pool and must settle in the same cash currency (`BTC/USDT` + `ETH/BTC` is rejected). **Backtest Engine** — run on start, start capital, lookback (candles). **Exchange Routing** — with an API key: exchange + sandbox/live derived from the key. Without: pick a *data exchange*; the bot forward-tests locally (no orders). Supported: OKX, Binance, Bitvavo, Coinbase, Crypto.com, Kraken, KuCoin. @@ -123,7 +125,7 @@ The backend enforces this allowlist — never invent a method name. ### 2.6 Action node and risk nodes -BUY action: order type, sizing, fee, slippage, and the attached TP/SL tiers. SELL action: closes the position (partially with `amount_value < 100`). +BUY action: order type, sizing, fee, slippage, and the attached TP/SL tiers. SELL action: closes the position (partially with `amount_value < 100`). On a perpetual market the action's direction can also be SHORT (opens a short; same fields and TP/SL ports as BUY, stored as `trade_settings.short`) or COVER (closes the short, `trade_settings.cover`). The table below is written for a long; for a short every level is mirrored (stop above entry hit by the high, target below hit by the low, trailing anchored to the lowest low). | TP/SL `type` | Stop-loss meaning | Take-profit meaning | |---|---|---| @@ -208,7 +210,7 @@ Aim for **reward:risk ≥ 1.5** on fixed targets, and let trailing stops handle ### 4.6 Position sizing and portfolio guards - **`amount_value` 25–50%** per trade for 2–4 pairs, so the pool can hold several positions; 100% on a single pair is acceptable for a pure trend follower with a trailing stop. -- Always set **`max_drawdown`** with **`drawdown_action: "block_entries"`** and **`max_capital_loss` 30–40**. Size the drawdown limit to the strategy: fully-invested daily trend following on BTC/ETH runs 25–32% drawdowns in normal bear phases (buy & hold ran 59% in the same window), so set **30–35** there; 15–20 is right only for low-exposure or short-holding strategies. A limit below the strategy's natural drawdown makes the backtest pause entries for weeks (`entries_blocked_days` in the summary shows this) and the live bot will do the same. +- Always set **`max_drawdown`** with **`drawdown_action: "block_entries"`** and **`max_capital_loss` 30–40**. Size the drawdown limit to the strategy: daily trend following on BTC/ETH at 50% sizing (≈ 75% invested when both are open — the second entry is 50% of the remaining cash) runs 25–32% drawdowns in normal bear phases (buy & hold ran 59% in the same window), so set **30–35** there; 15–20 is right only for low-exposure or short-holding strategies. A limit below the strategy's natural drawdown makes the backtest pause entries for weeks (`entries_blocked_days` in the summary shows this) and the live bot will do the same. - `close_all` is right only for strategies whose exits are *not* trend-following. - Add a light **cooldown** (e.g. `cooldown_trades: 1, cooldown_candles: 3–6`) to trend systems to stop whipsaw re-entries around a flat moving average. - Multi-pair: 2–3 liquid majors (`BTC`, `ETH`, optionally `SOL`). Correlation is high, so treat them as one bet when choosing `amount_value`. Adding SOL to the daily templates lowered the return in our tests; more pairs ≠ more diversification here. @@ -235,7 +237,7 @@ Aim for **reward:risk ≥ 1.5** on fixed targets, and let trailing stops handle | `==` on a float line (`rsi == 30`) | never exactly equal | `cross_above` / `>=` | | `increasing_for` without a number in `right` | defaults to 2, probably not what you meant | put N (e.g. `3`) in `right` | | `ichimoku` output 4 (chikou) | look-ahead — disabled, always false | use 0–3 | -| Mixing `symbol` quotes (`BTC/USDT` + `ETH/USDC`) | separate balances live | one quote currency per bot | +| Mixing cash currencies (`BTC/USDT` + `ETH/BTC`) | one pool cannot hold two currencies — the validator rejects it | one quote (spot) / settle (swap) currency per bot | | `4h` on Coinbase, `3m` on Kraken | unsupported timeframe → import error | see §5.3 | | `max_drawdown: 0` | nothing stops a broken bot | 15–30 + `block_entries` + `max_capital_loss` | @@ -250,6 +252,24 @@ Windows: 1d = 1 000 candles (Dec 2023 → Sep 2026, includes the 2024–25 bull | `Supertrend_Trend_1d` (BTC+ETH, 50%, `max_positions` 2 global) | 1d | 26 | 42 | +26.0% | 32.0% | BTC +73%, ETH +4% | | `Donchian_Breakout_1d` (BTC+ETH, 50%, `max_positions` 2 global) | 1d | 20 | 60 | +19.9% | 26.3% | BTC +73%, ETH +4% | | `EMA_Cross_4h` (BTC+ETH+SOL, 33%, `max_positions` 3 global) | 4h | 153 | 37 | +14.2% | 26.3% | BTC +75%, ETH +7%, SOL −5% | +| `Supertrend_LongShort_Perp_1d` (BTC+ETH USDT perps, 30%, 1×, `max_positions` 2 global) — measured 25 Sep 2026, window 30 Dec 2023 → 24 Sep 2026, fee 0.05% | 1d | 49 (26 long / 23 short) | 37 | +50.8% | 29.1% | BTC +100%, ETH +17% | + +**Long/short on perpetuals (measured 25 Sep 2026, Binance USDT-settled perps, taker fee 0.05%, window 30 Dec 2023 → 24 Sep 2026):** the same Supertrend(10, 3) flip, but *short while the direction is down* instead of flat. Entries are the trend **state** (`st_dir > 0` / `< 0`), exits the flip: a long and a short never coexist on a pair, so on the flip candle the old side is closed and the new side opens on the next candle from the state signal; with the flip as the only exit the state re-enters exactly once per trend. A 20% disaster stop replaces the 15% trail — a trailing stop on the state entry re-enters after every stop-out and turned the 1× result into −0.3% at 89 trades. + +| Variant (BTC+ETH, `max_positions` 2 global) | Lev | Trades | Win % | Return | Max DD | +|---|---|---|---|---|---| +| Supertrend(10, 3) state long/short, SL 20%, 30% size | 1× | 49 | 37 | **+50.8%** | 29.1% | +| same, 40% size | 1× | 49 | 35 | +52.7% | 31.2% | +| same, 50% size | 1× | 48 | 35 | +59.2% | 35.7% | +| same, BTC+ETH+SOL at 33% | 1× | 78 | 33 | +38.5% | 32.5% | +| Neighbours (10, 3.5) / (12, 3) / (14, 3) / (7, 3) / (10, 2.5), 40% | 1× | 42–75 | 30–38 | +33 / +22 / +46 / +14 / +12% | 31–34% | +| Supertrend(14, 4) state long/short | 1× | 34 | 41 | +11.5% | 36.2% | +| Supertrend(10, 3) state long/short, 40% | **2×** | 46 | 37 | +180% | 35.7% | +| every neighbour above at 2× — (10, 2.5), (10, 3.5), (12, 3), (14, 4) | 2× | 10–18 | 14–22 | **−30 to −39%** (stopped by `max_capital_loss`) | 33–46% | +| EMA 21/55 state long/short, ATR 3× stop, 4h and 1d | 1–2× | 16–182 | 10–34 | −33 to −41% (1d 21/55 at 1× the only positive: +8.9%) | 32–38% | +| Supertrend(10, 3) state long/short on 4h, SL 15% | 2× | 43 | 23 | −35% | 31.8% | + +Read the 2× rows correctly: the shipped parameters happen to survive 2× leverage and every neighbour blows through the capital-loss stop — that is path luck, not an edge, which is why the template ships at **1×**. Leverage doubles the trade-level swings, and a 35% max-drawdown guard sized for 1× stops the bot before the good trend arrives. The shorts do not add return in this window (the 1d bull run rewards the long side; a long-only Supertrend gives +26% at 50% size) — they cut the time flat and the 2026 drawdown, which is what makes the smaller 30% size hold up. Funding is simulated where the exchange still serves the history (§1) — on OKX only the last ~3 months of a multi-year backtest carry it; at 1× and multi-week holds it is a few percent a year against whichever side is crowded. Original measurements (engine v1, one position per pair): @@ -288,7 +308,7 @@ Take-aways: (1) simple trend following with an opposite-signal exit works across 4. SL/TP sized for the timeframe (§4.4); reward:risk ≥ 1.5 if both are fixed. 5. `backtest_lookback` covers several regimes and the exchange has that history; longest indicator length ≤ lookback / 10. 6. `is_sandbox: true`, `api_execution: false`, `api_key_name: null`, `max_order_value` present. -7. After the JSON, tell the user in 3–5 lines: what regime it targets, what the failure mode is, what to look for in the backtest (trade count, PF, DD vs. B&H), and that it must forward-test first. If the request cannot be expressed (shorting, multi-timeframe in one graph, time-of-day rules), say so instead of approximating. +7. After the JSON, tell the user in 3–5 lines: what regime it targets, what the failure mode is, what to look for in the backtest (trade count, PF, DD vs. B&H), and that it must forward-test first. If the request cannot be expressed (shorting on a spot market, multi-timeframe in one graph, time-of-day rules), say so instead of approximating. --- @@ -341,21 +361,25 @@ Output exactly **one** valid JSON document in a fenced code block. Use only meth | Field | Type | Description | |---|---|---| | `symbol` | string | Primary symbol (first in whitelist) | -| `symbols` | string[] | All pairs; one quote currency per bot | +| `symbols` | string[] | All pairs; one cash currency per bot (same quote on spot, same settle on swaps — validator error otherwise) | | `timeframe` | string | Must be supported by `data_exchange` (§5.3) | | `max_positions` | int ≥ 1 | Open positions (layers) allowed; > 1 enables pyramiding on repeated BUY signals (§2) | | `max_positions_scope` | `per_pair` / `global` | `per_pair` = limit per symbol, `global` = limit across all pairs of the bot | -| `cooldown_trades` / `cooldown_candles` | int | Max new entries per window (0 = off) | +| `cooldown_trades` / `cooldown_candles` | int | Max new entries per window of N candles, per symbol (0 = off). Whole numbers; `"5.0"` is coerced with a warning | | `max_drawdown` | % | Peak-to-trough on mark-to-market equity, checked after the backtest and after every closed live position. 0 = off | | `drawdown_action` | `close_all` / `block_entries` | `close_all`: close everything and stop (a backtest breach prevents go-live). `block_entries`: skip new entries until drawdown < half the limit, or — once flat — until `drawdown_cooldown_days` passed; then the peak resets. Exits keep working. Simulated in the backtest too | | `drawdown_cooldown_days` | 0–365 | Flat time before entries resume under `block_entries` (default 7) | | `max_capital_loss` | % | Hard stop on loss of starting capital, independent of drawdown. With `block_entries` the bot winds down (no entries, exits finish, then stops). Required > 0 for live bots using `block_entries` | -| `max_order_value` | USD | Cap per live order (0 = off; required > 0 for live) | -| `live_allocation_pct` | 1–100 | Paper/live: share of the exchange wallet (free quote + deployed by all bots on the key) this bot may deploy; split it between bots sharing a key. Snapshot at go-live becomes `live_starting_capital`, the base for live guards | +| `max_order_value` | quote notional | Cap on the notional of one entry in the pair's **quote** currency (USDC on `BTC/USDC`, USD on `BTC/USD:BTC`; on swaps notional = margin × leverage). Applied in the backtest, forward test and live alike, so it is part of the strategy (changing it counts as a new variant). 0 = off; required > 0 for live | +| `live_allocation_pct` | 1–100 | Paper/live: share of the exchange wallet (free cash currency + deployed by all bots on the key) this bot may deploy; split it between bots sharing a key. Snapshot at go-live becomes `live_starting_capital`, the base for live guards | | `api_execution` | bool | `true` = orders via API key | -| `backtest_on_start` / `backtest_capital` / `backtest_lookback` | bool / USD / candles | Backtest settings | +| `backtest_on_start` / `backtest_capital` / `backtest_lookback` | bool / cash amount / candles | Backtest settings; `backtest_capital` is in the bot's cash currency (USDC for `BTC/USDC`, BTC for `BTC/USD:BTC`) | | `api_key_name` | string/null | Saved key name (null = forward test) | -| `data_exchange` | string | `okx` `binance` `bitvavo` `coinbase` `cryptocom` `kraken` `kucoin` | +| `data_exchange` | string | `okx` `binance` `bitvavo` `coinbase` `cryptocom` `kraken` `kucoin` `bybit` `gateio` `bitget` `mexc` `htx` `bingx` | +| `market_type` | `spot` / `swap` | Default `spot`. `swap` = perpetuals; symbols then use the `BASE/QUOTE:SETTLE` form — linear `BTC/USDT:USDT` (settled in the quote stablecoin) or inverse `BTC/USD:BTC` (coin-margined, settled in the base; BingX has no inverse contracts) — and the key must be a swap key. The kind is decided per symbol. A spot bot rejects `:SETTLE` symbols, a swap bot requires them. Not every exchange offers swaps (§5.3) | +| `leverage` | 1–10 (int) | Swaps only (spot must be 1; Kraken max 5). Position notional = margin × leverage; `max_order_value` caps the notional. Warning above 3× (liquidation ≈ `(1 − 0.5%) / leverage` from entry — a wider stop never fires) | +| `margin_mode` | `isolated` / `cross` | Swaps only, default `isolated`. Sent to the exchange before the first order. `isolated`: each position has its own liquidation level; `cross`: the whole bot pool backs every position and one account-level breach liquidates all of them (see §1) | +| `short_node` / `cover_node` | string/null | Node ids for the SHORT and COVER signals (like `entry_node`/`exit_node`). Only on `swap` (validator error on spot); omit for long-only bots. A short without `cover_node`, SL or TP only warns | ### 5.3 Exchange timeframes and history @@ -369,6 +393,8 @@ Output exactly **one** valid JSON document in a fenced code block. Use only meth | KuCoin | `1m 3m 5m 15m 30m 1h 2h 4h 6h 8h 12h 1d 1w` | full | | Crypto.com | `1m 5m 15m 30m 1h 2h 4h 6h 12h 1d 1w` | full since listing | +Bybit, Gate, Bitget, MEXC, HTX and BingX were added later; their timeframes come from `GET /api/data/timeframes/{exchange}` (the builder filters the dropdown). Perpetual swaps (`market_type: swap`) are available on OKX, Binance, Kraken, KuCoin, Bybit, Gate, Bitget, HTX and BingX — not on Bitvavo, Coinbase, Crypto.com or MEXC. All of them serve linear contracts; inverse (coin-margined) contracts on all but BingX. + For long daily backtests prefer Binance or Coinbase data; for Kraken use ≤ 720 candles. ### 5.4 Trade settings @@ -392,7 +418,9 @@ For long daily backtests prefer Binance or Coinbase data; for Kraken use ≤ 720 } ``` -`fee`/`slippage` are percentages (`0.1` = 0.1%). `amount_type` `percentage` (% of available cash) or `fixed` (USD). TP/SL semantics in §2.6. +`fee`/`slippage` are percentages (`0.1` = 0.1%). `amount_type` `percentage` (% of the free cash left in the pool, see §1 Sizing) or `fixed` (an amount in the bot's cash currency). On swaps the amount is the margin; the notional is `leverage` times bigger. TP/SL semantics in §2.6. + +Shorts (swap only): add `"short"` (same shape as `entry`, its TP/SL are the short's levels) and `"cover"` (same shape as `exit`) next to them and set `short_node`/`cover_node`. When `short` is omitted the short leg reuses `entry`; when `cover` is omitted it reuses `exit`. ### 5.5 Node definitions @@ -413,7 +441,7 @@ For long daily backtests prefer Binance or Coinbase data; for Kraken use ≤ 720 ## 6. Vetted templates -Three complete files, identical to `examples/*.apex.json` in the repository, all imported and backtested in the real engine (results in §4.9, re-verified on v2.0.0 with `scripts/verify_examples.py`; `tests/golden/` pins their order stream on synthetic data so an engine change can never silently alter them). Adapt pairs, fee and sizes; keep the structure. Each is deliberately minimal — every addition we tried made them worse. +Four complete files, identical to `examples/*.apex.json` in the repository, all imported and backtested in the real engine (results in §4.9, re-verified with `scripts/verify_examples.py`; `tests/golden/` pins their order stream on synthetic data so an engine change can never silently alter them). Three are long-only spot, the fourth is the long/short perpetual version of the first. Adapt pairs, fee and sizes; keep the structure. Each is deliberately minimal — every addition we tried made them worse. ### 6.1 Supertrend trend follower — 1d, flip in / flip out, 15% disaster trail @@ -591,6 +619,142 @@ Backtest Dec 2023 → Sep 2026, BTC+ETH+SOL at 33%: **+15.3%, max DD 26.3%, 153 Roughly one trade per pair per week. The ATR 3× trailing stop (≈ 4–7% on 4h) closes most trades — 134 of 153 exits were the trail, at +0.9% average — while the death cross only catches slow rolls. Lower return than the daily systems but shallower single-trade losses and faster feedback for a forward test. Run at least 6 000 candles; a 2 000-candle window covers one regime only. +### 6.4 Supertrend long/short on perpetuals — 1d, 1×, long in uptrend, short in downtrend + +Backtest 30 Dec 2023 → 24 Sep 2026, BTC+ETH USDT-settled perps at 30%: **+50.8%, max DD 29.1%, 49 trades (26 long / 23 short), 37% win rate**, 0 liquidations, entries never blocked (buy & hold: BTC +100%, ETH +17%). Neighbours at 1× all positive (+12 to +46%); at 2× only these exact parameters survive, so it ships at 1× (§4.9). + +```json +{ + "apex_version": "1.0", + "exported_at": "2026-09-25T12:00:00Z", + "bot": { + "name": "Supertrend Long/Short Perp 1d", + "is_sandbox": true, + "strategy": "node_evaluator", + "settings": { + "symbol": "BTC/USDT:USDT", + "symbols": [ + "BTC/USDT:USDT", + "ETH/USDT:USDT" + ], + "timeframe": "1d", + "market_type": "swap", + "leverage": 1, + "margin_mode": "isolated", + "max_positions": 2, + "max_positions_scope": "global", + "cooldown_trades": 1, + "cooldown_candles": 5, + "max_drawdown": 35, + "drawdown_action": "block_entries", + "drawdown_cooldown_days": 14, + "max_capital_loss": 40, + "max_order_value": 1000, + "live_allocation_pct": 100, + "api_execution": false, + "backtest_on_start": true, + "backtest_capital": 1000, + "backtest_lookback": 1000, + "api_key_name": null, + "data_exchange": "binance", + "trade_settings": { + "entry": { + "order_type": "market", + "amount_type": "percentage", + "amount_value": 30, + "fee": 0.05, + "slippage": 0.05, + "take_profits": [], + "stop_losses": [ + { + "type": "percentage", + "value": 20.0, + "close_amount_type": "percentage", + "close_amount_value": 100 + } + ] + }, + "exit": { + "order_type": "market", + "amount_type": "percentage", + "amount_value": 100, + "fee": 0.05, + "slippage": 0.05 + }, + "short": { + "order_type": "market", + "amount_type": "percentage", + "amount_value": 30, + "fee": 0.05, + "slippage": 0.05, + "take_profits": [], + "stop_losses": [ + { + "type": "percentage", + "value": 20.0, + "close_amount_type": "percentage", + "close_amount_value": 100 + } + ] + }, + "cover": { + "order_type": "market", + "amount_type": "percentage", + "amount_value": 100, + "fee": 0.05, + "slippage": 0.05 + } + }, + "nodes": { + "st_dir": { + "class": "indicator", + "method": "supertrend", + "params": { + "length": 10, + "multiplier": 3.0 + }, + "output_idx": 1 + }, + "trend_up": { + "class": "condition", + "left": "st_dir", + "operator": ">", + "right": 0 + }, + "trend_down": { + "class": "condition", + "left": "st_dir", + "operator": "<", + "right": 0 + }, + "flip_up": { + "class": "condition", + "left": "st_dir", + "operator": "cross_above", + "right": 0 + }, + "flip_down": { + "class": "condition", + "left": "st_dir", + "operator": "cross_below", + "right": 0 + } + }, + "ui_layout": { + "nodes": [], + "edges": [] + }, + "entry_node": "trend_up", + "exit_node": "flip_down", + "short_node": "trend_down", + "cover_node": "flip_up" + } + } +} +``` + +The same flip as §6.1, made two-sided: `trend_up` (`st_dir > 0`) is the BUY, `flip_down` the SELL, `trend_down` the SHORT and `flip_up` the COVER. The entries are states rather than crosses on purpose — a long and a short never coexist on a pair, so on the flip candle the old side is closed and the new side opens one candle later from the state; with the flip as the only strategy exit the state fires once per trend, and the 20% stop is a disaster stop only (a trailing stop here would re-enter after every stop-out — tested, −0.3%). `market_type: swap` with `:USDT` symbols, `leverage: 1`, `margin_mode: isolated`, taker fee 0.05% on both legs. Runs as a forward test without a key; live needs a key bound to the swap market (§5). Do not raise the leverage without re-running the backtest: the 2× rows in §4.9 show why. + --- ## 7. Example prompt diff --git a/backend/core/database.py b/backend/core/database.py index ed9e362..3e48ae8 100644 --- a/backend/core/database.py +++ b/backend/core/database.py @@ -49,6 +49,35 @@ def run_migrations(): if 'exchange' not in pos_cols: conn.execute(text("ALTER TABLE positions ADD COLUMN exchange TEXT DEFAULT 'okx'")) logger.info("Migration: added 'exchange' to positions") + # Derivatives support: existing rows are spot at 1x + if 'market_type' not in pos_cols: + conn.execute(text("ALTER TABLE positions ADD COLUMN market_type TEXT DEFAULT 'spot'")) + logger.info("Migration: added 'market_type' to positions") + if 'leverage' not in pos_cols: + conn.execute(text("ALTER TABLE positions ADD COLUMN leverage REAL DEFAULT 1")) + logger.info("Migration: added 'leverage' to positions") + if 'contracts' not in pos_cols: + conn.execute(text("ALTER TABLE positions ADD COLUMN contracts REAL")) + logger.info("Migration: added 'contracts' to positions") + # Currency-aware books: the currency the position is funded in + # and the contract kind/size its `amount` is expressed in + _backfill_ccy = 'cash_currency' not in pos_cols + if _backfill_ccy: + conn.execute(text("ALTER TABLE positions ADD COLUMN cash_currency TEXT")) + logger.info("Migration: added 'cash_currency' to positions") + if 'contract_kind' not in pos_cols: + conn.execute(text("ALTER TABLE positions ADD COLUMN contract_kind TEXT")) + logger.info("Migration: added 'contract_kind' to positions") + if 'contract_size' not in pos_cols: + conn.execute(text("ALTER TABLE positions ADD COLUMN contract_size REAL")) + logger.info("Migration: added 'contract_size' to positions") + if 'funding_paid' not in pos_cols: + conn.execute(text("ALTER TABLE positions ADD COLUMN funding_paid REAL")) + logger.info("Migration: added 'funding_paid' to positions") + if 'funding_until' not in pos_cols: + conn.execute(text("ALTER TABLE positions ADD COLUMN funding_until DATETIME")) + logger.info("Migration: added 'funding_until' to positions") + _backfill_position_currency(conn) # ── orders ─────────────────────────────────────────────────────────────── if "orders" in existing_tables: @@ -56,6 +85,28 @@ def run_migrations(): if 'exchange' not in ord_cols: conn.execute(text("ALTER TABLE orders ADD COLUMN exchange TEXT DEFAULT 'okx'")) logger.info("Migration: added 'exchange' to orders") + if 'market_type' not in ord_cols: + conn.execute(text("ALTER TABLE orders ADD COLUMN market_type TEXT DEFAULT 'spot'")) + logger.info("Migration: added 'market_type' to orders") + if 'reduce_only' not in ord_cols: + conn.execute(text("ALTER TABLE orders ADD COLUMN reduce_only INTEGER DEFAULT 0")) + logger.info("Migration: added 'reduce_only' to orders") + # `fee` stays the fee in the position's cash currency (what the + # books use); `fee_currency`/`fee_cash` record what the exchange + # actually charged, when known + if 'fee_currency' not in ord_cols: + conn.execute(text("ALTER TABLE orders ADD COLUMN fee_currency TEXT")) + logger.info("Migration: added 'fee_currency' to orders") + if 'fee_cash' not in ord_cols: + conn.execute(text("ALTER TABLE orders ADD COLUMN fee_cash REAL")) + logger.info("Migration: added 'fee_cash' to orders") + + # ── exchange_keys ──────────────────────────────────────────────────────── + if "exchange_keys" in existing_tables: + key_cols = {c['name'] for c in inspector.get_columns('exchange_keys')} + if 'market_type' not in key_cols: + conn.execute(text("ALTER TABLE exchange_keys ADD COLUMN market_type TEXT DEFAULT 'spot'")) + logger.info("Migration: added 'market_type' to exchange_keys") # ── candles ────────────────────────────────────────────────────────────── # The unique constraint must include 'exchange'. SQLite cannot alter @@ -190,6 +241,28 @@ def run_migrations(): _sweep_orphaned_bot_rows(conn, existing_tables) +def _backfill_position_currency(conn) -> None: + """Fill `positions.cash_currency` / `contract_kind` from the symbol for + rows that predate the columns (spot → quote, ``:SETTLE`` → settle; + ``BTC/USD:BTC`` → inverse). Idempotent: only NULL rows are touched.""" + from backend.engine.contracts import spec_from_symbol + rows = conn.execute(text( + "SELECT DISTINCT symbol FROM positions WHERE symbol IS NOT NULL AND (cash_currency IS NULL OR contract_kind IS NULL)" + )).fetchall() + for (symbol,) in rows: + try: + spec = spec_from_symbol(symbol) + except Exception: + continue + conn.execute(text( + "UPDATE positions SET cash_currency = COALESCE(cash_currency, :ccy), contract_kind = COALESCE(contract_kind, :kind) " + "WHERE symbol = :sym AND (cash_currency IS NULL OR contract_kind IS NULL)" + ), {"ccy": spec.cash_currency, "kind": spec.kind, "sym": symbol}) + if rows: + conn.commit() + logger.info("Migration: backfilled cash_currency/contract_kind on positions for %d symbol(s)", len(rows)) + + def _sweep_orphaned_bot_rows(conn, existing_tables) -> None: """Delete per-bot rows whose bot no longer exists. diff --git a/backend/core/exchange_registry.py b/backend/core/exchange_registry.py index ffa7ea5..21e2c07 100644 --- a/backend/core/exchange_registry.py +++ b/backend/core/exchange_registry.py @@ -1,30 +1,184 @@ import logging import threading import time +from dataclasses import dataclass, field import ccxt logger = logging.getLogger("apexalgo.exchange_registry") -# All exchanges supported by ApexAlgo. Key = ccxt exchange ID, Value = display name. -SUPPORTED_EXCHANGES: dict[str, str] = { - "okx": "OKX", - "binance": "Binance", - "bitvavo": "Bitvavo", - "coinbase": "Coinbase", - "cryptocom": "Crypto.com", - "kraken": "Kraken", - "kucoin": "KuCoin", -} -# Exchanges that authenticate via api_key + api_secret + passphrase. -# All others use only api_key + api_secret. -_PASSPHRASE_EXCHANGES: frozenset[str] = frozenset({"okx", "kucoin"}) +@dataclass(frozen=True) +class MarketCaps: + """What ApexAlgo may do on one market type of an exchange.""" + ccxt_id: str | None = None # separate ccxt class for this market type (kucoinfutures) + max_leverage: int = 1 # cap the validator/UI enforce (1 = no leverage) + leverage_in_order: bool = False # exchange takes leverage per order instead of set_leverage + note: str | None = None # shown in the key form / go-live checklist + # Contract kinds this market type serves → ccxt class override per kind + # (None = the market type's own class). Spot is always {"spot": None}; + # swaps list "linear" (stablecoin-settled) and/or "inverse" (coin-margined). + kinds: tuple = (("spot", None),) + + @property + def kind_ids(self) -> dict: + return dict(self.kinds) + + def supports_kind(self, kind: str) -> bool: + return str(kind or "").lower() in self.kind_ids + + +SPOT = MarketCaps() +LINEAR_ONLY = (("linear", None),) +LINEAR_AND_INVERSE = (("linear", None), ("inverse", None)) + -# Per-exchange static configuration applied at instantiation. -_EXCHANGE_CONFIG: dict[str, dict] = { - "okx": {"hostname": "eea.okx.com"}, +def _swap(**kw) -> MarketCaps: + return MarketCaps(max_leverage=kw.pop("max_leverage", 10), kinds=kw.pop("kinds", LINEAR_AND_INVERSE), **kw) + + +@dataclass(frozen=True) +class ExchangeSpec: + """Everything ApexAlgo knows about one exchange, in one place. + + Adding an exchange = one entry in `EXCHANGES`; the routers, the key form + and the builder derive their lists from here (via `GET /api/keys/exchanges`). + ccxt itself is id-agnostic, so nothing engine-side needs to change. + `markets` lists the market types a key/bot may use: every exchange has + "spot"; "swap" (perpetuals, long and short) only where ccxt can place, + reduce and list positions. `MarketCaps.kinds` says which contract kinds + the swap market serves — linear (stablecoin-settled) and/or inverse + (coin-margined) — and which ccxt class serves each (Binance keeps its + coin-margined futures behind a separate class, `binancecoinm`). + """ + name: str # display name + needs_passphrase: bool = False # api_key + secret + passphrase + ccxt_config: dict = field(default_factory=dict) # extra ccxt constructor config + keys_url: str | None = None # where to create API keys + sandbox_note: str | None = None # how to get testnet keys, if any + candle_limit_note: str | None = None # shown when the exchange caps OHLCV history + markets: dict = field(default_factory=lambda: {"spot": SPOT}) # market type -> MarketCaps + + +# All exchanges supported by ApexAlgo, keyed by ccxt exchange ID. +EXCHANGES: dict[str, ExchangeSpec] = { + "okx": ExchangeSpec( + "OKX", needs_passphrase=True, ccxt_config={"hostname": "eea.okx.com"}, + keys_url="https://www.okx.com/account/my-api", + sandbox_note="Demo trading keys are created under Trade → Demo trading → API.", + markets={"spot": SPOT, "swap": _swap(note="Account must be in net (one-way) position mode.")}), + "binance": ExchangeSpec( + "Binance", keys_url="https://www.binance.com/en/my/settings/api-management", + sandbox_note="Spot testnet keys come from testnet.binance.vision, futures testnet keys from testnet.binancefuture.com (separate accounts).", + markets={"spot": SPOT, "swap": _swap(note="USDⓈ-M and COIN-M futures; account must be in one-way position mode.", + kinds=(("linear", None), ("inverse", "binancecoinm")))}), + "bitvavo": ExchangeSpec("Bitvavo", keys_url="https://account.bitvavo.com/user/api"), + "coinbase": ExchangeSpec("Coinbase", keys_url="https://www.coinbase.com/settings/api"), + "cryptocom": ExchangeSpec( + "Crypto.com", keys_url="https://crypto.com/exchange/user/settings/api-management", + sandbox_note="UAT sandbox keys are issued via the Crypto.com Exchange UAT environment."), + "kraken": ExchangeSpec( + "Kraken", keys_url="https://www.kraken.com/u/security/api", + candle_limit_note="Kraken only serves its most recent 720 candles per timeframe", + markets={"spot": SPOT, "swap": _swap(ccxt_id="krakenfutures", max_leverage=5, + note="Kraken Futures uses its own API keys (futures.kraken.com); demo keys from demo-futures.kraken.com.")}), + "kucoin": ExchangeSpec( + "KuCoin", needs_passphrase=True, keys_url="https://www.kucoin.com/account/api", + markets={"spot": SPOT, "swap": _swap(ccxt_id="kucoinfutures", leverage_in_order=True, + note="KuCoin Futures uses its own API keys (Futures → API management).")}), + "bybit": ExchangeSpec( + "Bybit", keys_url="https://www.bybit.com/app/user/api-management", + sandbox_note="Testnet keys are created on testnet.bybit.com (separate account).", + markets={"spot": SPOT, "swap": _swap(note="Unified trading account, one-way position mode.")}), + "gateio": ExchangeSpec( + "Gate", keys_url="https://www.gate.io/myaccount/api_key_manage", + sandbox_note="Testnet keys are created on testnet.gate.io (separate account).", + markets={"spot": SPOT, "swap": _swap()}), + "bitget": ExchangeSpec( + "Bitget", needs_passphrase=True, keys_url="https://www.bitget.com/account/newapi", + sandbox_note="Demo-trading keys are created under Demo Trading → API on bitget.com.", + markets={"spot": SPOT, "swap": _swap(note="USDT-M futures, one-way position mode.")}), + "mexc": ExchangeSpec( + "MEXC", keys_url="https://www.mexc.com/user/openapi", + sandbox_note="MEXC has no public testnet — keys can only be used for real trading."), + # spot only: MEXC's futures order API is closed to the public ("under maintenance") + "htx": ExchangeSpec( + "HTX", keys_url="https://www.htx.com/en-us/apikey/", + sandbox_note="HTX has no public testnet — keys can only be used for real trading.", + markets={"spot": SPOT, "swap": _swap()}), + "bingx": ExchangeSpec( + "BingX", keys_url="https://bingx.com/en/account/api/", + sandbox_note="VST demo-trading keys are created under Demo Trading → API on bingx.com.", + markets={"spot": SPOT, "swap": _swap(note="Perpetual futures (USDT-margined), one-way position mode.", kinds=LINEAR_ONLY)}), } +# Backwards-compatible views used by the routers: {id: display name} and the +# set of exchanges that authenticate with a passphrase. +SUPPORTED_EXCHANGES: dict[str, str] = {ex_id: spec.name for ex_id, spec in EXCHANGES.items()} +_PASSPHRASE_EXCHANGES: frozenset[str] = frozenset(ex_id for ex_id, spec in EXCHANGES.items() if spec.needs_passphrase) + + +def exchange_spec(exchange_id: str) -> ExchangeSpec | None: + """Spec for an exchange id (case-insensitive); None when unsupported.""" + return EXCHANGES.get(str(exchange_id or "").lower()) + + +def market_caps(exchange_id: str, market_type: str = "spot") -> MarketCaps | None: + """Capabilities of one market type on an exchange; None when the exchange + is unknown or does not offer that market type.""" + spec = exchange_spec(exchange_id) + if spec is None: + return None + return spec.markets.get(str(market_type or "spot").lower()) + + +def ccxt_id_for(exchange_id: str, market_type: str = "spot", kind: str | None = None) -> str: + """The ccxt class that serves this market type (kucoin swap = + kucoinfutures) and, within a swap market, this contract kind (binance + inverse = binancecoinm).""" + caps = market_caps(exchange_id, market_type) + if caps is None: + return str(exchange_id).lower() + if kind: + override = caps.kind_ids.get(str(kind).lower()) + if override: + return override + return caps.ccxt_id if caps.ccxt_id else str(exchange_id).lower() + + +def supported_kinds(exchange_id: str, market_type: str = "swap") -> tuple: + """Contract kinds (``linear``/``inverse``) the exchange's market type serves.""" + caps = market_caps(exchange_id, market_type) + return tuple(caps.kind_ids) if caps else () + + +def kind_of_symbol(symbol: str) -> str: + """Contract kind implied by a unified symbol (see `contracts.spec_from_symbol`).""" + from backend.engine.contracts import spec_from_symbol + return spec_from_symbol(symbol).kind + + +_sandbox_cache: dict[tuple[str, str], bool] = {} + + +def exchange_has_sandbox(exchange_id: str, market_type: str = "spot") -> bool: + """Whether ccxt can put this exchange (market type) in sandbox/testnet + mode. Probed on a throwaway instance (no network): some exchanges use a + test URL, others a demo-trading header (Bitget), and ccxt raises for the + rest.""" + exchange_id = str(exchange_id or "").lower() + market_type = str(market_type or "spot").lower() + key = (exchange_id, market_type) + if key not in _sandbox_cache: + ok = False + try: + if market_caps(exchange_id, market_type) is not None: + getattr(ccxt, ccxt_id_for(exchange_id, market_type))().set_sandbox_mode(True) + ok = True + except Exception: + pass + _sandbox_cache[key] = ok + return _sandbox_cache[key] + def build_exchange( exchange_id: str, @@ -32,6 +186,8 @@ def build_exchange( api_secret: str | None = None, passphrase: str | None = None, sandbox: bool = False, + market_type: str = "spot", + kind: str | None = None, ) -> ccxt.Exchange: """ Instantiate and return a configured CCXT exchange. @@ -43,19 +199,41 @@ def build_exchange( api_key / api_secret : str, optional Omit for unauthenticated (public market data) connections. passphrase : str, optional - Only applied for exchanges that require it (OKX, KuCoin). + Only applied for exchanges whose spec has `needs_passphrase`. sandbox : bool Enable exchange sandbox / testnet mode when supported. + market_type : str + "spot" (default, unchanged behaviour) or "swap": sets ccxt's + `defaultType` and picks the derivatives class where the exchange has + one. Raises ValueError when the spec has no such market type. + kind : str, optional + Contract kind within a swap market ("linear"/"inverse"): picks the + per-kind ccxt class (binance inverse → binancecoinm). Raises + ValueError when the market type does not serve that kind. """ exchange_id = exchange_id.lower() + market_type = str(market_type or "spot").lower() if exchange_id not in SUPPORTED_EXCHANGES: raise ValueError( f"Exchange '{exchange_id}' is not supported. " f"Supported: {', '.join(SUPPORTED_EXCHANGES)}" ) + spec = EXCHANGES[exchange_id] + caps = spec.markets.get(market_type) + if caps is None: + raise ValueError( + f"Exchange '{exchange_id}' has no '{market_type}' market in ApexAlgo. " + f"Available: {', '.join(spec.markets)}" + ) + + if kind and market_type != "spot" and not caps.supports_kind(kind): + raise ValueError( + f"Exchange '{exchange_id}' has no {kind} {market_type} contracts in ApexAlgo. " + f"Available: {', '.join(caps.kind_ids)}" + ) - cls = getattr(ccxt, exchange_id, None) + cls = getattr(ccxt, ccxt_id_for(exchange_id, market_type, kind if market_type != "spot" else None), None) if cls is None: raise RuntimeError( f"Exchange '{exchange_id}' was not found in the installed CCXT version. " @@ -63,13 +241,16 @@ def build_exchange( ) config: dict = {"enableRateLimit": True} - config.update(_EXCHANGE_CONFIG.get(exchange_id, {})) + config.update(spec.ccxt_config) + # Pin ccxt's defaultType so balance/position calls address the right + # account: ccxt's own default is not "spot" everywhere (bybit → swap) + config["options"] = {**config.get("options", {}), "defaultType": market_type} if api_key: config["apiKey"] = api_key config["secret"] = api_secret - if passphrase and exchange_id in _PASSPHRASE_EXCHANGES: + if passphrase and spec.needs_passphrase: config["password"] = passphrase exchange = cls(config) @@ -86,6 +267,20 @@ def build_exchange( return exchange +def key_market_type(key_record) -> str: + """Market type an ExchangeKey record is bound to (pre-phase-2 rows: spot).""" + return str(getattr(key_record, "market_type", None) or "spot").lower() + + +def build_exchange_for_symbol(exchange_id: str, symbol: str) -> ccxt.Exchange: + """Public (unauthenticated) instance for market data on `symbol`: the + unified symbol says which market it lives on (``BTC/USDT:USDT`` → swap), + so candle/ticker callers need no market-type plumbing of their own.""" + from backend.engine.symbols import market_type_of + mt = market_type_of(symbol) + return build_exchange(exchange_id, market_type=mt, kind=kind_of_symbol(symbol) if mt != "spot" else None) + + # Cache for exchange timeframes (loaded once per exchange, reused) _timeframe_cache: dict[str, dict[str, str]] = {} @@ -108,80 +303,194 @@ def get_exchange_timeframes(exchange_id: str) -> dict[str, str]: return tf_map -# Active spot symbols per exchange, refreshed after _MARKETS_TTL seconds so a -# newly listed pair shows up without a restart -_markets_cache: dict[str, tuple[float, list[str]]] = {} +# Active markets per (exchange, market type), refreshed after _MARKETS_TTL +# seconds so a newly listed pair shows up without a restart. Value: +# {symbol: {kind, base, quote, settle, contract_size}} — every swap kind the +# exchange serves is merged in (binance linear + binancecoinm inverse). +_markets_cache: dict[tuple[str, str], tuple[float, dict[str, dict]]] = {} _MARKETS_TTL = 3600 - - -def get_exchange_symbols(exchange_id: str) -> list[str]: - """Sorted list of tradeable spot symbols ("BTC/USDT") on the exchange. - - Empty when the exchange cannot be reached — callers must treat that as - "unknown", not "nothing is tradeable".""" +# ContractSpec per (exchange, symbol): filled from loaded markets (public +# lists above, authenticated instances in the broker) and never fetched on +# demand — the engine falls back to the symbol's own shape when it is empty. +_spec_cache: dict[tuple[str, str], object] = {} +_spec_lock = threading.Lock() + + +def _market_info(spec) -> dict: + return {"kind": spec.kind, "base": spec.base, "quote": spec.quote, "settle": spec.settle, + "contract_size": spec.contract_size, "cash_currency": spec.cash_currency} + + +def _remember_spec(exchange_id: str, spec) -> None: + with _spec_lock: + _spec_cache[(str(exchange_id or "").lower(), spec.symbol)] = spec + + +def contract_spec(exchange_id: str, symbol: str, market: dict | None = None): + """`ContractSpec` for `symbol` on `exchange_id` without touching the + network: from `market` (a ccxt market dict, cached for later callers), + else from the cache filled by `get_exchange_markets`/`remember_markets`, + else derived from the symbol alone (contract size 1).""" + from backend.engine.contracts import spec_from_market, spec_from_symbol + from backend.engine.symbols import normalize + ex = str(exchange_id or "").lower() + sym = normalize(symbol) + if market: + spec = spec_from_market(market, sym) + _remember_spec(ex, spec) + return spec + with _spec_lock: + hit = _spec_cache.get((ex, sym)) + return hit if hit is not None else spec_from_symbol(sym) + + +def remember_markets(exchange_id: str, markets: dict) -> None: + """Seed the spec cache from a loaded ccxt market map (e.g. an + authenticated instance's `markets`) so backtests know contract sizes.""" + from backend.engine.contracts import spec_from_market + for sym, m in (markets or {}).items(): + try: + _remember_spec(exchange_id, spec_from_market(m, sym)) + except Exception: + continue + + +def _load_markets_for(exchange_id: str, market_type: str) -> dict[str, dict]: + """{symbol: info} for every active market of `market_type`, across all + contract kinds the registry lists for it. Raises when nothing could be loaded.""" + from backend.engine.contracts import spec_from_market + out: dict[str, dict] = {} + errors = [] + kinds = [None] if market_type == "spot" else list(supported_kinds(exchange_id, market_type)) + # Kinds that share one ccxt class are loaded once + seen_classes: set[str] = set() + for kind in kinds: + cls_id = ccxt_id_for(exchange_id, market_type, kind) + if cls_id in seen_classes: + continue + seen_classes.add(cls_id) + try: + exchange = build_exchange(exchange_id, market_type=market_type, kind=kind) + markets = exchange.load_markets() + except Exception as exc: + errors.append(exc) + continue + for sym, m in markets.items(): + if m.get("active", True) is False: + continue + if market_type == "spot": + if not m.get("spot", True): + continue + else: + if m.get("type") != market_type: + continue + spec = spec_from_market(m, sym) + if spec.kind not in supported_kinds(exchange_id, market_type): + continue + spec = spec_from_market(m, sym) + _remember_spec(exchange_id, spec) + out[sym] = _market_info(spec) + if not out and errors: + raise errors[0] + return out + + +def get_exchange_markets(exchange_id: str, market_type: str = "spot") -> dict[str, dict]: + """{symbol: {kind, base, quote, settle, contract_size, cash_currency}} of + the tradeable markets: spot pairs, or for `market_type="swap"` the active + linear and inverse perpetuals the registry lists (dated futures left + out). Empty when the exchange cannot be reached — callers must treat + that as "unknown", not "nothing is tradeable".""" exchange_id = exchange_id.lower() - hit = _markets_cache.get(exchange_id) + market_type = str(market_type or "spot").lower() + cache_key = (exchange_id, market_type) + hit = _markets_cache.get(cache_key) if hit and time.monotonic() - hit[0] < _MARKETS_TTL: return hit[1] try: - exchange = build_exchange(exchange_id) - markets = exchange.load_markets() - symbols = sorted( - sym for sym, m in markets.items() - if m.get("spot", True) and m.get("active", True) is not False - ) + markets = _load_markets_for(exchange_id, market_type) except Exception as exc: - logger.warning("Failed to load markets for '%s': %s", exchange_id, exc) - return hit[1] if hit else [] - _markets_cache[exchange_id] = (time.monotonic(), symbols) - return symbols + logger.warning("Failed to load %s markets for '%s': %s", market_type, exchange_id, exc) + return hit[1] if hit else {} + _markets_cache[cache_key] = (time.monotonic(), markets) + return markets + + +def get_exchange_symbols(exchange_id: str, market_type: str = "spot") -> list[str]: + """Sorted symbols of `get_exchange_markets` (linear AND inverse swaps).""" + return sorted(get_exchange_markets(exchange_id, market_type)) -def build_exchange_from_key(key_record) -> ccxt.Exchange: +def build_exchange_from_key(key_record, kind: str | None = None) -> ccxt.Exchange: """ Convenience wrapper: build an authenticated exchange instance directly from a decrypted ExchangeKey record. Always a fresh instance — - use `get_authenticated_exchange` for the shared, cached one. + use `get_authenticated_exchange` for the shared, cached one. `kind` + picks the contract kind's class on a swap key (binance inverse). """ from backend.core.encryption import decrypt_data + mt = key_market_type(key_record) return build_exchange( exchange_id=key_record.exchange, api_key=decrypt_data(key_record.api_key), api_secret=decrypt_data(key_record.api_secret), passphrase=decrypt_data(key_record.passphrase) if key_record.passphrase else None, sandbox=key_record.is_sandbox, + market_type=mt, + kind=kind if mt != "spot" else None, ) +def key_kind_for_symbol(key_record, symbol: str | None) -> str | None: + """Contract kind an authenticated instance for `symbol` needs on this + key: None on spot keys, else the kind the symbol implies (linear when + no symbol is given).""" + if key_market_type(key_record) == "spot": + return None + return kind_of_symbol(symbol) if symbol else "linear" + + # Authenticated instances shared by the engine and the routers, keyed by -# (exchange, key name, sandbox). Building one per bot per candle re-decrypted -# the credentials and re-downloaded the market list every tick; here markets -# are loaded once and the instance is rebuilt after _AUTH_TTL seconds so a -# stale session or rate-limit bookkeeping never sticks around for good. -# Sync ccxt instances serialize nothing themselves; order paths are already -# serialized per bot, and concurrent REST reads on one instance are safe. -_auth_cache: dict[tuple[str, str, bool], tuple[float, ccxt.Exchange]] = {} +# (exchange, key name, sandbox, market type, contract kind). Building one per bot per candle +# re-decrypted the credentials and re-downloaded the market list every tick; +# here markets are loaded once and the instance is rebuilt after _AUTH_TTL +# seconds so a stale session or rate-limit bookkeeping never sticks around +# for good. Sync ccxt instances serialize nothing themselves; order paths are +# already serialized per bot, and concurrent REST reads on one instance are safe. +_auth_cache: dict[tuple[str, str, bool, str, str | None], tuple[float, ccxt.Exchange]] = {} _auth_lock = threading.Lock() _AUTH_TTL = 3600 -def _auth_key(key_record) -> tuple[str, str, bool]: - return (str(key_record.exchange).lower(), str(key_record.name), bool(key_record.is_sandbox)) +def _auth_key(key_record, kind: str | None = None) -> tuple[str, str, bool, str, str | None]: + mt = key_market_type(key_record) + if mt == "spot": + kind = None + else: + kind = str(kind or "linear").lower() + # Kinds served by the same ccxt class share one instance + kind = ccxt_id_for(key_record.exchange, mt, kind) + return (str(key_record.exchange).lower(), str(key_record.name), + bool(key_record.is_sandbox), mt, kind) -def get_authenticated_exchange(key_record, *, load_markets: bool = True) -> ccxt.Exchange: +def get_authenticated_exchange(key_record, *, load_markets: bool = True, symbol: str | None = None) -> ccxt.Exchange: """Cached authenticated instance for an ExchangeKey record (1 h TTL). - Markets are loaded on first use, so callers' `load_markets()` is a no-op.""" - key = _auth_key(key_record) + Markets are loaded on first use, so callers' `load_markets()` is a no-op. + On a swap key `symbol` selects the contract kind's instance + (``BTC/USD:BTC`` on binance → binancecoinm); default linear.""" + kind = key_kind_for_symbol(key_record, symbol) + key = _auth_key(key_record, kind) now = time.monotonic() with _auth_lock: hit = _auth_cache.get(key) cached = hit[1] if hit and now - hit[0] < _AUTH_TTL else None - instance = cached or build_exchange_from_key(key_record) + instance = cached or build_exchange_from_key(key_record, kind=kind) # A status poll may have built the instance without markets; the order # path needs them, so load lazily on whichever call asks first if load_markets and not instance.markets: instance.load_markets() + remember_markets(key_record.exchange, instance.markets) if cached is None: with _auth_lock: _auth_cache[key] = (now, instance) diff --git a/backend/engine/backtest.py b/backend/engine/backtest.py index c6fb962..edfdf11 100644 --- a/backend/engine/backtest.py +++ b/backend/engine/backtest.py @@ -13,14 +13,40 @@ from backend.models.orders import Order from backend.models.positions import Position from backend.engine.sizing import ( - _num, _int, _naive_utc, sim_frictions, calculate_trade_amount, _record_config_run, - backtest_pin, combined_fingerprint, slice_key, + _num, _int, _naive_utc, sim_frictions, calculate_trade_amount, max_affordable_amount, _record_config_run, + backtest_pin, combined_fingerprint, slice_key, cap_by_max_order_value, ) +from backend.engine.symbols import DEFAULT_MARGIN_MODE, leverage_for, market_type_for +from backend.engine import funding, pnl, tiers +from backend.engine.capital import CapitalPools +from backend.engine.contracts import spec_for from backend.core import bot_log_buffer as blb logger = logging.getLogger("apexalgo.bot_manager") _BT_COMMIT_EVERY = 500 # timeline steps between backtest commits +# Derivatives (v2.3): isolated positions are liquidated once the candle range +# reaches `spec.liquidation_price` at the maintenance-margin rate of their +# exchange tier (`tiers.mmr_for`, flat `MAINTENANCE_MARGIN` without tiers); +# cross-margin bots are liquidated as one account (`_cross_breached`); +# funding settlements stored in `funding_rates` are charged on open +# positions (`funding.payment`), marked at the candle close. + + +def _cap_by_max_order_value(trade_amount, price, settings, spec=None): + """Backtest/forward mirror of the live `max_order_value` clamp: the + quote notional of one entry never exceeds the cap (see `sizing.cap_by_max_order_value`).""" + return cap_by_max_order_value(trade_amount, price, settings, spec) + + +def bot_cash_currency(sym_contexts_or_symbols, exchange_name=None) -> str | None: + """The one cash currency a bot's whitelist is funded in (the validator + refuses mixed lists; the first symbol decides for legacy bots).""" + for c in sym_contexts_or_symbols: + sym = c["symbol"] if isinstance(c, dict) else c + if sym: + return spec_for(exchange_name, sym).cash_currency + return None @dataclass @@ -35,6 +61,42 @@ class SimResult: block_count: int dd_detail: dict timeline: list = field(default_factory=list) + liquidations: int = 0 + shorts: int = 0 + cash_currency: str | None = None + console: list = field(default_factory=list) # (level, msg) lines to push after the commit + funding_paid: float = 0.0 # net funding booked (cash currency, negative = paid) + funding_events: int = 0 # settlements charged on open positions + funding_info: dict = field(default_factory=dict) # `funding.coverage` per symbol + mmr_source: str | None = None # "tiers" | "flat" + margin_mode: str | None = None + + +def _cross_breached(sym_contexts, cash, leverage, current_ctx, current_high, current_low): + """Cross-margin account check on one candle: the account equity — cash + plus every open position's margin and PnL, the current symbol marked at + its adverse extreme (long: low, short: high), the others at their last + close — against the sum of the maintenance margins. Returns the mark + price per symbol when the account is liquidated, else None.""" + equity = cash + maint = 0.0 + marks = {} + any_open = False + for c2 in sym_contexts: + spec = c2["spec"] + for p in c2["open_positions"]: + any_open = True + side = p.side or "long" + if c2 is current_ctx: + mark = current_high if side == "short" else current_low + else: + mark = c2["last_close"] or p.entry_price + marks[c2["symbol"]] = mark + equity += spec.margin(p.amount, p.entry_price, leverage) + spec.pnl_cash(side, p.amount, p.entry_price, mark) + maint += spec.notional_cash(p.amount, mark) * tiers.mmr_for(c2.get("tiers"), tiers.tier_size(spec, p.amount, p.entry_price)) + if not any_open or equity > maint: + return None + return marks def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, position_states, states_lock, on_progress, check_abort): @@ -42,9 +104,19 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, and book backtest positions/orders on `db`. `check_exits` is the engine's exit evaluator; `on_progress(detail, progress)` reports to the runtime strip; `check_abort()` raises when the user stopped the bot.""" - # Shared capital pool across ALL symbols for this bot + # Console lines are buffered and pushed by the caller after the commit: + # `blb.push` opens a second SQLite connection, which would wait on this + # session's uncommitted rows for the full busy timeout + console = [] + # Every symbol is priced through its ContractSpec (spot / linear / + # inverse); the bot's books are kept in one cash currency (validator) + for ctx in sym_contexts: + ctx["spec"] = spec_for(exchange_name, ctx["symbol"]) + bt_ccy = bot_cash_currency(sym_contexts, exchange_name) or "USDT" + # Shared capital pool across ALL symbols for this bot, in `bt_ccy` bt_starting_capital = _num(bot.settings.get("backtest_capital"), 1000) - bt_equity = bt_starting_capital # Available cash (not locked in positions) + pools = CapitalPools.for_bot(bt_ccy, bt_starting_capital) + bt_equity = bt_starting_capital # Available cash (not locked in positions) — mirrors pools.cash(bt_ccy) bt_peak_equity = bt_starting_capital bt_max_dd = 0.0 # peak-to-trough on the mark-to-market equity curve bt_max_loss = 0.0 # worst loss of principal vs. starting capital (%) @@ -73,6 +145,64 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, cooldown_trades = _int(bot.settings.get("cooldown_trades"), 0) cooldown_candles = _int(bot.settings.get("cooldown_candles"), 0) + # Derivatives (phase 2): only the margin (notional / leverage) plus the + # entry fee leaves the pool on open; on close the margin comes back with + # the PnL. Every formula below keeps a literal spot branch so a spot bot + # simulates byte-for-byte as before. + bt_market_type = market_type_for(bot.settings) + bt_derivative = bt_market_type != "spot" + bt_leverage = leverage_for(bot.settings, bt_market_type) if bt_derivative else 1.0 + bt_liquidations = 0 + # Cross margin (v2.3): one account-level liquidation instead of a level + # per position; funding and the tiered maintenance margin come from the + # stored market data of every symbol (loaded below) + bt_margin_mode = (bot.settings.get("margin_mode") or DEFAULT_MARGIN_MODE) if bt_derivative else None + bt_cross = bt_margin_mode == "cross" + bt_funding_paid = 0.0 + bt_funding_events = 0 + bt_funding_info = {} + bt_mmr_source = None + # Extra Order columns for derivative fills (empty on spot → same rows as before) + _bt_open_extra = {"market_type": bt_market_type} if bt_derivative else {} + _bt_close_extra = {"market_type": bt_market_type, "reduce_only": 1} if bt_derivative else {} + _bt_extra_cols = {"cash_currency": bt_ccy} + if bt_derivative: + _tiers_syms = [] + for ctx in sym_contexts: + df_ts = ctx["df"]["timestamp"] + ctx["funding"] = funding.load(db, exchange_name, ctx["symbol"], df_ts.iloc[0], df_ts.iloc[-1]) if len(df_ts) else [] + ctx["tiers"] = tiers.load(db, exchange_name, ctx["symbol"]) + ctx["funding_total"] = 0.0 + bt_funding_info[ctx["symbol"]] = funding.coverage(ctx["funding"], None, None) + if ctx["tiers"]: + _tiers_syms.append(ctx["symbol"]) + if ctx["funding"]: + console.append(("INFO", f"{ctx['symbol']}: {len(ctx['funding'])} funding settlements stored " + f"({ctx['funding'][0][0]:%Y-%m-%d} → {ctx['funding'][-1][0]:%Y-%m-%d}) — charged on open positions")) + else: + console.append(("WARN", f"{ctx['symbol']}: no funding-rate data stored for this window — funding not simulated")) + bt_mmr_source = "tiers" if _tiers_syms and len(_tiers_syms) == len(sym_contexts) else "flat" + _has_inverse = any(c["spec"].is_inverse for c in sym_contexts) + if bt_cross: + _liq_note = "cross margin — liquidated as one account when the equity falls to the maintenance margin" + elif _has_inverse: + _liq_note = "inverse contracts liquidate on the coin-margined curve" + else: + _liq_note = f"isolated liquidation at about {100 * (1 - pnl.MAINTENANCE_MARGIN) / bt_leverage:.1f}% adverse move" + _mmr_note = (f"maintenance margin from the exchange tiers of {', '.join(_tiers_syms)}" if bt_mmr_source == "tiers" + else f"flat {100 * pnl.MAINTENANCE_MARGIN:g}% maintenance margin (no exchange tiers stored)") + console.append(("INFO", f"Backtest on {bt_market_type} at {bt_leverage:g}x {bt_margin_mode} in {bt_ccy}: {_liq_note}, {_mmr_note}")) + # Shorts (phase 3): a `short` signal opens a short layer, `cover` flattens + # the pair's shorts; both only exist when the strategy has the nodes and + # the bot runs on a derivative market (the validator rejects them on + # spot). Long and short never coexist on one pair — the conflicting + # signal is ignored (INFO once per pair). Frictions come from the + # `short`/`cover` legs, falling back to entry/exit. + bt_shorts_enabled = bt_derivative and any(c.get("short_arr") is not None for c in sym_contexts) + bt_s_entry_fee, bt_s_exit_fee, bt_s_entry_slippage, bt_s_exit_slippage = sim_frictions(bot.settings, "short") + bt_short_count = 0 + _conflict_logged = set() + # ── Merged chronological execution across all symbols ── timeline = [] for ci, ctx in enumerate(sym_contexts): @@ -112,11 +242,36 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, is_buy = bool(ctx["entry_arr"][index]) is_sell = bool(ctx["exit_arr"][index]) + is_short = bool(ctx["short_arr"][index]) if ctx.get("short_arr") is not None else False + is_cover = bool(ctx["cover_arr"][index]) if ctx.get("cover_arr") is not None else False atr_arr = ctx["atr_arr"] current_atr = float(atr_arr[index]) if atr_arr is not None and not pd.isna(atr_arr[index]) else 0.0 if run_backtest and (ctx["last_bt_ts"] is None or ts > ctx["last_bt_ts"]): open_list = ctx["open_positions"] + spec = ctx["spec"] + + # Funding: every settlement since the position's last charge up + # to this candle close, on its notional at the close (the + # forward tick applies the same rule from the same table) + if bt_derivative and ctx.get("funding") and open_list: + for open_bt_pos in open_list: + _since = open_bt_pos.funding_until or open_bt_pos.created_at + for _f_ts, _rate in funding.settlements(ctx["funding"], _since, ts): + _pay = funding.payment(spec, open_bt_pos.side or "long", open_bt_pos.amount, current_price, _rate) + pools.charge(bt_ccy, _pay) + open_bt_pos.profit_abs = (open_bt_pos.profit_abs or 0.0) + _pay + open_bt_pos.funding_paid = (open_bt_pos.funding_paid or 0.0) + _pay + open_bt_pos.funding_until = _f_ts + _orig = ctx["original_amount"].get(open_bt_pos.id) or open_bt_pos.amount + _locked = spec.locked_capital(_orig, open_bt_pos.entry_price, bt_leverage, + bt_s_entry_fee if open_bt_pos.side == "short" else bt_entry_fee) + if _locked > 0: + open_bt_pos.profit_pct = (open_bt_pos.profit_pct or 0.0) + 100.0 * _pay / _locked + ctx["funding_total"] += _pay + bt_funding_paid += _pay + bt_funding_events += 1 + bt_equity = pools.cash(bt_ccy) # Cooldown check: block entry if too many trades occurred within the cooldown window can_buy_cooldown = True @@ -134,9 +289,19 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, else: slot_free = sum(len(c2["open_positions"]) for c2 in sym_contexts) < max_pos just_opened = None + # Side already open on this pair (None when flat) — a long never + # stacks on a short and vice versa + pair_side = (open_list[0].side or "long") if open_list else None + if is_buy and pair_side == "short" and symbol not in _conflict_logged: + _conflict_logged.add(symbol) + console.append(("INFO", f"{symbol}: BUY signal ignored while a short is open (long and short never coexist on a pair)")) # Capital depletion halt / drawdown block: no new entries, exits keep running - if is_buy and slot_free and can_buy_cooldown and bt_equity > 0 and not bt_entries_blocked: - trade_amount = calculate_trade_amount(current_price, bot.settings, current_equity=bt_equity) + if is_buy and pair_side != "short" and slot_free and can_buy_cooldown and bt_equity > 0 and not bt_entries_blocked: + if bt_derivative: + trade_amount = calculate_trade_amount(current_price, bot.settings, current_equity=bt_equity, leverage=bt_leverage, spec=spec) + else: + trade_amount = calculate_trade_amount(current_price, bot.settings, current_equity=bt_equity, spec=spec) + trade_amount = _cap_by_max_order_value(trade_amount, current_price, bot.settings, spec) if trade_amount is not None: bt_entry_price = current_price * (1 + bt_entry_slippage) # Percentage sizing spends a share of equity; cap the @@ -144,19 +309,54 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, # (100% sizing would otherwise always exceed it) _entry_cfg = bot.settings.get("trade_settings", {}).get("entry", {}) if _entry_cfg.get("amount_type", "percentage") != "fixed": - max_affordable = bt_equity / (bt_entry_price * (1 + bt_entry_fee)) + # Margin + fee on the notional must fit the pool + max_affordable = max_affordable_amount(bt_equity, bt_entry_price, bt_leverage if bt_derivative else 1, bt_entry_fee, spec=spec) trade_amount = min(trade_amount, max_affordable) - investment_cost = bt_entry_price * trade_amount - total_cost = investment_cost * (1 + bt_entry_fee) + # What leaves the pool: margin + entry fee, in bt_ccy + total_cost = spec.locked_capital(trade_amount, bt_entry_price, bt_leverage if bt_derivative else 1, bt_entry_fee) if trade_amount > 0 and total_cost <= bt_equity + 1e-9: ctx["trade_entry_indices"].append(index) - bt_equity = max(bt_equity - total_cost, 0.0) # Lock capital + entry fee - just_opened = Position(exchange=exchange_name, bot_name=bot.name, symbol=symbol, mode="backtest", status="open", side="long", entry_price=bt_entry_price, amount=trade_amount, created_at=_naive_utc(ts)) + pools.lock(bt_ccy, total_cost) # Lock capital + entry fee + bt_equity = pools.cash(bt_ccy) + just_opened = Position(exchange=exchange_name, bot_name=bot.name, symbol=symbol, mode="backtest", status="open", side="long", entry_price=bt_entry_price, amount=trade_amount, created_at=_naive_utc(ts), + contract_kind=spec.kind, contract_size=spec.contract_size, **_bt_extra_cols) + if bt_derivative: + just_opened.market_type = bt_market_type + just_opened.leverage = bt_leverage + just_opened.funding_until = _naive_utc(ts) db.add(just_opened) db.flush() - db.add(Order(position_id=just_opened.id, exchange=exchange_name, bot_name=bot.name, mode="backtest", symbol=symbol, side="buy", order_type="market", price=bt_entry_price, amount=trade_amount, timestamp=_naive_utc(ts), status="filled", fee=investment_cost * bt_entry_fee)) + db.add(Order(position_id=just_opened.id, exchange=exchange_name, bot_name=bot.name, mode="backtest", symbol=symbol, side="buy", order_type="market", price=bt_entry_price, amount=trade_amount, timestamp=_naive_utc(ts), status="filled", fee=spec.fee_cash(trade_amount, bt_entry_price, bt_entry_fee), fee_currency=bt_ccy, **_bt_open_extra)) open_list.append(just_opened) ctx["original_amount"][just_opened.id] = trade_amount + elif bt_shorts_enabled and is_short and not is_buy: + if pair_side == "long": + if symbol not in _conflict_logged: + _conflict_logged.add(symbol) + console.append(("INFO", f"{symbol}: SHORT signal ignored while a long is open (long and short never coexist on a pair)")) + elif slot_free and can_buy_cooldown and bt_equity > 0 and not bt_entries_blocked: + trade_amount = calculate_trade_amount(current_price, bot.settings, current_equity=bt_equity, leverage=bt_leverage, side="short", spec=spec) + trade_amount = _cap_by_max_order_value(trade_amount, current_price, bot.settings, spec) + if trade_amount is not None: + # Slippage works against the seller: a short fills below the close + bt_entry_price = current_price * (1 - bt_s_entry_slippage) + _short_cfg = pnl.entry_cfg(bot.settings.get("trade_settings", {}), "short") + if _short_cfg.get("amount_type", "percentage") != "fixed": + max_affordable = max_affordable_amount(bt_equity, bt_entry_price, bt_leverage, bt_s_entry_fee, spec=spec) + trade_amount = min(trade_amount, max_affordable) + total_cost = spec.locked_capital(trade_amount, bt_entry_price, bt_leverage, bt_s_entry_fee) + if trade_amount > 0 and total_cost <= bt_equity + 1e-9: + ctx["trade_entry_indices"].append(index) + pools.lock(bt_ccy, total_cost) # Lock margin + entry fee + bt_equity = pools.cash(bt_ccy) + just_opened = Position(exchange=exchange_name, bot_name=bot.name, symbol=symbol, mode="backtest", status="open", side="short", entry_price=bt_entry_price, amount=trade_amount, created_at=_naive_utc(ts), market_type=bt_market_type, leverage=bt_leverage, + contract_kind=spec.kind, contract_size=spec.contract_size, funding_until=_naive_utc(ts), **_bt_extra_cols) + db.add(just_opened) + db.flush() + db.add(Order(position_id=just_opened.id, exchange=exchange_name, bot_name=bot.name, mode="backtest", symbol=symbol, side="sell", order_type="market", price=bt_entry_price, amount=trade_amount, timestamp=_naive_utc(ts), status="filled", fee=spec.fee_cash(trade_amount, bt_entry_price, bt_s_entry_fee), fee_currency=bt_ccy, **_bt_open_extra)) + open_list.append(just_opened) + ctx["original_amount"][just_opened.id] = trade_amount + bt_short_count += 1 # Every position carries its own SL/TP/trailing state; a SELL # signal reaches each of them, so it flattens the whole pair. @@ -165,7 +365,26 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, for open_bt_pos in list(open_list): if open_bt_pos is just_opened: continue - exit_events = check_exits(open_bt_pos, current_price, current_high, current_low, is_sell, bot.settings, current_atr, row_open=current_open) + pos_side = open_bt_pos.side or "long" + pos_short = pos_side == "short" + if bt_derivative and not bt_cross: + # Isolated: the level of this position at its tier's rate + _mmr = tiers.mmr_for(ctx.get("tiers"), tiers.tier_size(spec, open_bt_pos.amount, open_bt_pos.entry_price)) + liq_price = spec.liquidation_price(pos_side, open_bt_pos.entry_price, bt_leverage, mmr=_mmr) + liq_hit = pnl.liquidated(pos_side, liq_price, current_high, current_low) + else: + liq_price, liq_hit = None, False + # The regular exits are evaluated first: a stop that fills + # before the price reaches the liquidation level (e.g. an + # SL at the open) is a normal exit. Only fills at or beyond + # the liquidation price are impossible — those layers, and + # whatever is still open after the exits, are liquidated. + if pos_short: + exit_events = check_exits(open_bt_pos, current_price, current_high, current_low, is_cover, bot.settings, current_atr, row_open=current_open, side="short") + else: + exit_events = check_exits(open_bt_pos, current_price, current_high, current_low, is_sell, bot.settings, current_atr, row_open=current_open) + if liq_hit: + exit_events = [ev for ev in exit_events if not pnl.liquidated(pos_side, liq_price, ev['price'], ev['price'])] for ev in exit_events: if open_bt_pos.status == "closed": break @@ -180,16 +399,24 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, close_qty = min(close_qty, open_bt_pos.amount) if close_qty <= 0: continue - actual_price = ev['price'] * (1 - bt_exit_slippage) - db.add(Order(position_id=open_bt_pos.id, exchange=exchange_name, bot_name=bot.name, mode="backtest", symbol=symbol, side="sell", order_type="market", price=actual_price, amount=close_qty, timestamp=_naive_utc(ts), status="filled", fee=actual_price * close_qty * bt_exit_fee)) - - entry_cost = open_bt_pos.entry_price * close_qty * (1 + bt_entry_fee) - exit_proceeds = actual_price * close_qty * (1 - bt_exit_fee) - realized_pnl = exit_proceeds - entry_cost + _lev = bt_leverage if bt_derivative else 1 + if pos_short: + # Covering buys back above the trigger; fees on both legs' notional + actual_price = ev['price'] * (1 + bt_s_exit_slippage) + _fee_in, _fee_out = bt_s_entry_fee, bt_s_exit_fee + else: + actual_price = ev['price'] * (1 - bt_exit_slippage) + _fee_in, _fee_out = bt_entry_fee, bt_exit_fee + db.add(Order(position_id=open_bt_pos.id, exchange=exchange_name, bot_name=bot.name, mode="backtest", symbol=symbol, side=pnl.close_order_side(pos_side), order_type="market", price=actual_price, amount=close_qty, timestamp=_naive_utc(ts), status="filled", fee=spec.fee_cash(close_qty, actual_price, _fee_out), fee_currency=bt_ccy, **_bt_close_extra)) + # What was locked for this slice (margin + entry fee), what + # comes back (proceeds on spot; margin + PnL − exit fee on + # derivatives) and the net result — all in bt_ccy + entry_cost = spec.locked_capital(close_qty, open_bt_pos.entry_price, _lev, _fee_in) + returned = spec.close_return(pos_side, close_qty, open_bt_pos.entry_price, actual_price, _lev, _fee_out) + realized_pnl = spec.realized_pnl(pos_side, close_qty, open_bt_pos.entry_price, actual_price, _lev, _fee_in, _fee_out) open_bt_pos.profit_abs = (open_bt_pos.profit_abs or 0.0) + realized_pnl - - # Return sale proceeds to capital pool - bt_equity += exit_proceeds + pools.release(bt_ccy, returned, entry_cost) + bt_equity = pools.cash(bt_ccy) # Weighted profit_pct: accumulate based on portion of original position closed (fee-adjusted) original_amount = ctx["original_amount"].get(open_bt_pos.id) @@ -203,7 +430,7 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, position_states[open_bt_pos.id]['triggered_exits'].add(ev['id']) open_bt_pos.triggered_exits = list(position_states[open_bt_pos.id]['triggered_exits']) - if close_qty >= open_bt_pos.amount - 0.00001: + if close_qty >= open_bt_pos.amount - pnl.close_epsilon(original_amount or open_bt_pos.amount): open_bt_pos.status = "closed" open_bt_pos.closed_at = _naive_utc(ts) with states_lock: @@ -213,10 +440,70 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, else: open_bt_pos.amount -= close_qty + if liq_hit and open_bt_pos.status != "closed": + # Liquidated: the whole margin is gone, nothing returns + # to the pool (the entry fee was paid on open and counts + # against the trade; no exit fee is charged) + remaining_qty = open_bt_pos.amount + _liq_entry_fee = bt_s_entry_fee if pos_short else bt_entry_fee + liq_loss = spec.liquidation_loss(remaining_qty, open_bt_pos.entry_price, bt_leverage, _liq_entry_fee) + pools.forget(bt_ccy, liq_loss) + db.add(Order(position_id=open_bt_pos.id, exchange=exchange_name, bot_name=bot.name, mode="backtest", symbol=symbol, side=pnl.close_order_side(pos_side), order_type="market", price=liq_price, amount=remaining_qty, timestamp=_naive_utc(ts), status="filled", fee=0.0, fee_currency=bt_ccy, **_bt_close_extra)) + open_bt_pos.profit_abs = (open_bt_pos.profit_abs or 0.0) - liq_loss + original_amount = ctx["original_amount"].get(open_bt_pos.id) + if original_amount and original_amount > 0: + open_bt_pos.profit_pct = (open_bt_pos.profit_pct or 0.0) - 100.0 * (remaining_qty / original_amount) + with states_lock: + st = position_states.pop(open_bt_pos.id, None) + triggered = set((st or {}).get('triggered_exits') or ()) | set(open_bt_pos.triggered_exits or ()) + open_bt_pos.triggered_exits = sorted(str(t) for t in triggered) + ["liquidation"] + open_bt_pos.status = "closed" + open_bt_pos.closed_at = _naive_utc(ts) + open_list.remove(open_bt_pos) + ctx["original_amount"].pop(open_bt_pos.id, None) + bt_liquidations += 1 + + # Cross margin: after the candle's exits, the account as a whole + # against its maintenance margin. A breach liquidates every open + # position of the bot (all symbols) and empties the wallet; the + # free cash lost on top of the margins is attributed to the + # positions in proportion to their margin, so profit_pct can go + # below −100 and the losses add up to the equity that vanished. + if bt_cross: + _marks = _cross_breached(sym_contexts, pools.cash(bt_ccy), bt_leverage, ctx, current_high, current_low) + if _marks is not None: + _total_margin = sum(c2["spec"].margin(p.amount, p.entry_price, bt_leverage) for c2 in sym_contexts for p in c2["open_positions"]) + _cash_lost = pools.drain(bt_ccy) + for c2 in sym_contexts: + _spec2 = c2["spec"] + for _p in list(c2["open_positions"]): + _side2 = _p.side or "long" + _mark = _marks.get(c2["symbol"], _p.entry_price) + _remaining = _p.amount + _margin_i = _spec2.margin(_remaining, _p.entry_price, bt_leverage) + _fee_i = _spec2.fee_cash(_remaining, _p.entry_price, bt_s_entry_fee if _side2 == "short" else bt_entry_fee) + _loss = _margin_i + _fee_i + (_cash_lost * _margin_i / _total_margin if _total_margin > 0 else 0.0) + db.add(Order(position_id=_p.id, exchange=exchange_name, bot_name=bot.name, mode="backtest", symbol=c2["symbol"], side=pnl.close_order_side(_side2), order_type="market", price=_mark, amount=_remaining, timestamp=_naive_utc(ts), status="filled", fee=0.0, fee_currency=bt_ccy, **_bt_close_extra)) + _p.profit_abs = (_p.profit_abs or 0.0) - _loss + _orig = c2["original_amount"].get(_p.id) + if _orig and _orig > 0 and (_margin_i + _fee_i) > 0: + _p.profit_pct = (_p.profit_pct or 0.0) - 100.0 * (_loss / (_margin_i + _fee_i)) * (_remaining / _orig) + with states_lock: + _st = position_states.pop(_p.id, None) + _trig = set((_st or {}).get('triggered_exits') or ()) | set(_p.triggered_exits or ()) + _p.triggered_exits = sorted(str(t) for t in _trig) + ["liquidation"] + _p.status = "closed" + _p.closed_at = _naive_utc(ts) + c2["open_positions"].remove(_p) + c2["original_amount"].pop(_p.id, None) + bt_liquidations += 1 + bt_equity = pools.cash(bt_ccy) + console.append(("WARN", f"{ts:%Y-%m-%d %H:%M}: cross-margin liquidation on {symbol} — account equity fell to the maintenance margin, all positions closed and the wallet ({_cash_lost:,.2f} {bt_ccy} free cash) lost")) + if _naive_utc(ts) not in ctx["existing_timestamps"]: indicators = { col: float(row[col]) for col in ctx["indicator_cols"] if not pd.isna(row[col]) } if indicators: - action_str = "buy" if is_buy else ("sell" if is_sell else "neutral") + action_str = "buy" if is_buy else ("short" if is_short else ("sell" if is_sell else ("cover" if is_cover else "neutral"))) ctx["new_signals"].append(Signal(candle_id=int(row['id']), symbol=symbol, timestamp=ts, bot_name=bot.name, name="STRATEGY_TICK", action=action_str, extra_data=indicators)) # Mark-to-market equity curve: cash + open positions at their last close @@ -224,7 +511,7 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, open_value = 0.0 for c2 in sym_contexts: if c2["last_close"]: - open_value += sum(p2.amount for p2 in c2["open_positions"]) * c2["last_close"] + open_value += c2["spec"].mark_value(c2["open_positions"], c2["last_close"], bt_leverage if bt_derivative else 1) equity_now = bt_equity + open_value if equity_now > bt_peak_equity: bt_peak_equity = equity_now @@ -279,13 +566,16 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, last_price = float(df_s.iloc[-1]['close']) last_ts = df_s.iloc[-1]['timestamp'] if last_ts.tzinfo is None: last_ts = last_ts.replace(tzinfo=timezone.utc) + spec = ctx["spec"] + _lev = bt_leverage if bt_derivative else 1 for open_bt_pos in list(ctx["open_positions"]): remaining_qty = open_bt_pos.amount + pos_side = open_bt_pos.side or "long" + pos_short = pos_side == "short" + _fee_in, _fee_out = (bt_s_entry_fee, bt_s_exit_fee) if pos_short else (bt_entry_fee, bt_exit_fee) - entry_cost = open_bt_pos.entry_price * remaining_qty * (1 + bt_entry_fee) - exit_proceeds = last_price * remaining_qty * (1 - bt_exit_fee) - final_pnl = exit_proceeds - entry_cost - + entry_cost = spec.locked_capital(remaining_qty, open_bt_pos.entry_price, _lev, _fee_in) + final_pnl = spec.realized_pnl(pos_side, remaining_qty, open_bt_pos.entry_price, last_price, _lev, _fee_in, _fee_out) open_bt_pos.profit_abs = (open_bt_pos.profit_abs or 0.0) + final_pnl original_amount = ctx["original_amount"].get(open_bt_pos.id) if original_amount and original_amount > 0: @@ -295,8 +585,9 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, open_bt_pos.status = "closed" open_bt_pos.closed_at = _naive_utc(last_ts) - bt_equity += exit_proceeds # Return proceeds to capital pool - db.add(Order(position_id=open_bt_pos.id, exchange=exchange_name, bot_name=bot.name, mode="backtest", symbol=ctx["symbol"], side="sell", order_type="market", price=last_price, amount=remaining_qty, timestamp=_naive_utc(last_ts), status="filled", fee=last_price * remaining_qty * bt_exit_fee)) + pools.release(bt_ccy, spec.close_return(pos_side, remaining_qty, open_bt_pos.entry_price, last_price, _lev, _fee_out), entry_cost) + bt_equity = pools.cash(bt_ccy) + db.add(Order(position_id=open_bt_pos.id, exchange=exchange_name, bot_name=bot.name, mode="backtest", symbol=ctx["symbol"], side=pnl.close_order_side(pos_side), order_type="market", price=last_price, amount=remaining_qty, timestamp=_naive_utc(last_ts), status="filled", fee=spec.fee_cash(remaining_qty, last_price, _fee_out), fee_currency=bt_ccy, **_bt_close_extra)) with states_lock: position_states.pop(open_bt_pos.id, None) @@ -321,7 +612,10 @@ def simulate(db, bot, sym_contexts, exchange_name, run_backtest, *, check_exits, return SimResult( starting_capital=bt_starting_capital, equity=bt_equity, max_dd=bt_max_dd, max_loss=bt_max_loss, dd_action=bt_dd_action, dd_cooldown_secs=bt_dd_cooldown_secs, blocked_secs=bt_blocked_secs, - block_count=bt_block_count, dd_detail=bt_dd_detail, timeline=timeline, + block_count=bt_block_count, dd_detail=bt_dd_detail, timeline=timeline, liquidations=bt_liquidations, + shorts=bt_short_count, cash_currency=bt_ccy, console=console, + funding_paid=bt_funding_paid, funding_events=bt_funding_events, funding_info=bt_funding_info, + mmr_source=bt_mmr_source, margin_mode=bt_margin_mode, ) @@ -394,9 +688,44 @@ def build_summary(db, bot, res: SimResult, sym_contexts, exchange_name=None) -> "data_changed": data_changed, "finished_at": datetime.now(timezone.utc).isoformat(), } + # The currency every money figure above is in (backtest_capital, net_pnl, + # the drawdown peak/trough): the whitelist's cash currency + ccy = res.cash_currency or bot_cash_currency(sym_contexts, exchange_name) + if ccy: + summary["cash_currency"] = ccy + market_type = market_type_for(bot.settings) + if market_type != "spot": + summary.update({ + "market_type": market_type, + "leverage": leverage_for(bot.settings, market_type), + "margin_mode": res.margin_mode or (bot.settings.get("margin_mode") or DEFAULT_MARGIN_MODE), + "liquidations": res.liquidations, + # Funding: "simulated" when settlements were stored for every + # symbol of the window, "partial"/"no data" otherwise; the net + # amount booked is inside net_pnl already + "funding": _funding_status(res.funding_info), + "funding_paid": round(res.funding_paid, 8), + "funding_events": res.funding_events, + "funding_by_symbol": res.funding_info, + "mmr_source": res.mmr_source or "flat", + }) + if res.shorts: + # Closed short trades (buy & hold above stays the long reference on purpose) + short_closed = db.query(Position.id).filter( + Position.bot_name == bot.name, Position.mode == "backtest", Position.status == "closed", Position.side == "short" + ).count() + summary["short_trades"] = short_closed + summary["long_trades"] = len(pnls) - short_closed return summary +def _funding_status(info: dict) -> str: + states = {v.get("funding") for v in (info or {}).values()} + if not states or states == {"no data"}: + return "no data" + return "simulated" if states == {"simulated"} else "partial" + + def gate_stop_reason(bot, res: SimResult): """Enforce max_capital_loss / max_drawdown on the simulated curve before the bot may go live. Returns the stop reason, or None when it may proceed @@ -414,8 +743,9 @@ def gate_stop_reason(bot, res: SimResult): if max_drawdown_pct > 0 and res.max_dd >= max_drawdown_pct: _d = res.dd_detail _fmt = lambda t: t.strftime('%Y-%m-%d') if t is not None else '?' - detail = (f"peak {_fmt(_d['peak_ts'])} ${_d['peak_eq']:,.0f} -> trough {_fmt(_d['trough_ts'])} " - f"${_d['trough_eq']:,.0f}, {_d['open_at_trough']} open position(s)") + _ccy = res.cash_currency or "" + detail = (f"peak {_fmt(_d['peak_ts'])} {_d['peak_eq']:,.2f} {_ccy} -> trough {_fmt(_d['trough_ts'])} " + f"{_d['trough_eq']:,.2f} {_ccy}, {_d['open_at_trough']} open position(s)") if res.dd_action == "block_entries": # Informative only: the same rule already paused entries # inside the simulation, so the numbers reflect it diff --git a/backend/engine/bot_manager.py b/backend/engine/bot_manager.py index 4464cf3..06d2259 100644 --- a/backend/engine/bot_manager.py +++ b/backend/engine/bot_manager.py @@ -145,8 +145,28 @@ async def _handle_candle_close(self, exchange: str, symbol: str, timeframe: str, # ── Exchange plumbing lives in engine/broker.py ── # `_get_ccxt_instance` and `_reconcile_order` stay real methods: tests # monkeypatch them per instance to inject a fake exchange - def _get_ccxt_instance(self, api_key_record: ExchangeKey): - return broker.get_ccxt_instance(api_key_record) + def _get_ccxt_instance(self, api_key_record: ExchangeKey, symbol=None): + return broker.get_ccxt_instance(api_key_record, symbol) + + @staticmethod + def _needs_own_instance(api_key_record: ExchangeKey, symbol) -> bool: + """True when `symbol` is served by a different ccxt class than the + key's default one (binance inverse -> binancecoinm).""" + from backend.core.exchange_registry import ccxt_id_for, key_kind_for_symbol, key_market_type + if not symbol or key_market_type(api_key_record) != "swap": + return False + kind = key_kind_for_symbol(api_key_record, symbol) + ex = api_key_record.exchange + return ccxt_id_for(ex, "swap", kind) != ccxt_id_for(ex, "swap") + + def _ccxt_for(self, api_key_record: ExchangeKey, symbol=None): + """Client for `symbol`: the key's shared instance unless the contract + kind lives on its own ccxt class. Tests that monkeypatch + `_get_ccxt_instance` with a one-argument fake keep working because + the symbol is only passed when it actually matters.""" + if self._needs_own_instance(api_key_record, symbol): + return self._get_ccxt_instance(api_key_record, symbol) + return self._get_ccxt_instance(api_key_record) def _reconcile_order(self, ccxt_inst, order, ccxt_symbol, attempts=5, delay=1.0): return broker.reconcile_order(ccxt_inst, order, ccxt_symbol, attempts, delay) @@ -156,6 +176,7 @@ def _cancel_unfilled_order(self, ccxt_inst, order_id, ccxt_symbol): _below_market_minimum = staticmethod(broker.below_market_minimum) _fee_in_quote = staticmethod(broker.fee_in_quote) + _fee_cash = staticmethod(broker.fee_cash) # Sizing helpers live in engine/sizing.py; kept as attributes so callers # and tests keep addressing them through the manager @@ -167,6 +188,12 @@ def _cancel_unfilled_order(self, ccxt_inst, order_id, ccxt_symbol): _wallet_held = staticmethod(broker.wallet_held) _reconcile_positions_with_wallet = staticmethod(broker.reconcile_positions_with_wallet) + # Derivatives (phase 2): position reconciliation via fetch_positions, + # leverage/margin-mode confirmation and contract conversion + _reconcile_positions_with_exchange = staticmethod(broker.reconcile_positions_with_exchange) + _ensure_leverage = staticmethod(broker.ensure_leverage) + _to_contracts = staticmethod(broker.to_contracts) + _from_contracts = staticmethod(broker.from_contracts) def _get_live_capital(self, ccxt_inst, api_key_record, ccxt_symbol, bot_name, ttl=30): return broker.get_live_capital(self._balance_cache, ccxt_inst, api_key_record, ccxt_symbol, bot_name, ttl) @@ -174,9 +201,9 @@ def _get_live_capital(self, ccxt_inst, api_key_record, ccxt_symbol, bot_name, tt def _close_all_open_positions(self, bot, db, key_records): live_cycle.close_all_open_positions(self, bot, db, key_records) - def _check_exits(self, open_position, row_close, row_high, row_low, is_sell_signal, bot_settings, current_atr=0.0, row_open=None): + def _check_exits(self, open_position, row_close, row_high, row_low, is_sell_signal, bot_settings, current_atr=0.0, row_open=None, side="long"): return exits.check_exits(self.position_states, self._position_states_lock, open_position, - row_close, row_high, row_low, is_sell_signal, bot_settings, current_atr, row_open) + row_close, row_high, row_low, is_sell_signal, bot_settings, current_atr, row_open, side=side) async def _startup_backfill(self): def get_active_bot_ids(): @@ -220,8 +247,8 @@ async def _process_bots(self, exchange: str, symbol: str, timeframe: str, candle def mark_deleted(self, bot_name: str): """Mark a bot as deleted so _process_bots skips it.""" self._deleted_bots.add(bot_name) - self._drawdown_cache.pop((bot_name, "live"), None) - self._drawdown_cache.pop((bot_name, "backtest"), None) + for _grp in risk.MODES_BY_GROUP: + self._drawdown_cache.pop((bot_name, _grp), None) # Purge position states for this bot to prevent memory accumulation try: db = SessionLocal() @@ -237,8 +264,8 @@ def reset_bot_state(self, bot_name: str): """Drop every in-memory cache the engine keeps for a bot (cache wipe): drawdown state, entry block, position states of positions that no longer exist. Open real positions keep their state.""" - self._drawdown_cache.pop((bot_name, "live"), None) - self._drawdown_cache.pop((bot_name, "backtest"), None) + for _grp in risk.MODES_BY_GROUP: + self._drawdown_cache.pop((bot_name, _grp), None) self._entries_blocked.discard(bot_name) self._balance_cache.clear() try: diff --git a/backend/engine/broker.py b/backend/engine/broker.py index cec80db..e575978 100644 --- a/backend/engine/broker.py +++ b/backend/engine/broker.py @@ -3,18 +3,23 @@ Everything here talks to a ccxt instance and nothing here touches engine state, so tests swap the instance for a fake at `BotManager._get_ccxt_instance`.""" import logging +import threading import time from backend.models.exchange_keys import ExchangeKey from backend.models.positions import Position -from backend.core.exchange_registry import get_authenticated_exchange +from backend.core.exchange_registry import get_authenticated_exchange, market_caps from backend.core import bot_log_buffer as blb +from backend.engine.contracts import spec_for_instance +from backend.engine.symbols import base_of, cash_currency, is_derivative, normalize logger = logging.getLogger("apexalgo.bot_manager") -def get_ccxt_instance(api_key_record: ExchangeKey): - """Shared, market-loaded instance from the registry (1 h TTL).""" - return get_authenticated_exchange(api_key_record) +def get_ccxt_instance(api_key_record: ExchangeKey, symbol: str | None = None): + """Shared, market-loaded instance from the registry (1 h TTL). `symbol` + picks the ccxt class for the contract kind on exchanges that serve + inverse contracts from a separate class (binance -> binancecoinm).""" + return get_authenticated_exchange(api_key_record, symbol=symbol) def reconcile_order(ccxt_inst, order, ccxt_symbol, attempts=5, delay=1.0): """Market orders often report status open/None on creation even though they @@ -56,38 +61,92 @@ def cancel_unfilled_order(ccxt_inst, order_id, ccxt_symbol): logger.warning("cancel_order %s failed: %s", order_id, exc) return False +def contract_size(ccxt_inst, ccxt_symbol) -> float: + """Units per contract from the market (1 for spot and for most linear + perpetuals; e.g. 0.001 BTC on kucoinfutures, 100 USD on BTC/USD:BTC). + Unknown metadata counts as 1.""" + if not is_derivative(ccxt_symbol): + return 1.0 + return spec_for_instance(ccxt_inst, ccxt_symbol).contract_size + + +def to_contracts(ccxt_inst, ccxt_symbol, qty) -> float: + """Position size (base units on spot/linear, contracts on inverse) → the + `amount` ccxt expects in create_order. Identity on spot and inverse.""" + return spec_for_instance(ccxt_inst, ccxt_symbol).to_contracts(qty) + + +def from_contracts(ccxt_inst, ccxt_symbol, contracts) -> float: + return spec_for_instance(ccxt_inst, ccxt_symbol).from_contracts(contracts) + + def below_market_minimum(ccxt_inst, ccxt_symbol, amount, price): """Return a human-readable violation string when an order would fall below the exchange's minimum amount/cost limits, else None. Missing - limit metadata is treated as no restriction.""" + limit metadata is treated as no restriction. `amount` is the position + size (base units on spot/linear, contracts on inverse); the exchange's + amount limit is in contracts and its cost limit in the quote currency.""" try: + spec = spec_for_instance(ccxt_inst, ccxt_symbol) limits = (ccxt_inst.market(ccxt_symbol) or {}).get("limits") or {} min_amount = (limits.get("amount") or {}).get("min") min_cost = (limits.get("cost") or {}).get("min") - if min_amount is not None and amount < float(min_amount): - return f"amount {amount} below exchange minimum {float(min_amount)}" + if min_amount is not None: + contracts = spec.to_contracts(amount) + if contracts < float(min_amount): + unit = " contracts" if contracts != amount or spec.is_inverse else "" + return f"amount {contracts:g}{unit} below exchange minimum {float(min_amount)}{unit}" if min_cost is not None and price: - order_value = amount * float(price) + order_value = spec.notional_quote(amount, float(price)) if order_value < float(min_cost): - return f"order ${order_value:.2f} below exchange minimum ${float(min_cost):.2f}" + return f"order {order_value:.2f} {spec.quote} below exchange minimum {float(min_cost):.2f} {spec.quote}" except Exception: return None return None -def fee_in_quote(fee_info, ccxt_symbol, price): - """CCXT fee cost can be denominated in base currency (typical for buys); - convert to quote so it can be netted against PnL.""" +def fee_cash(fee_info, ccxt_symbol, price, ccxt_inst=None): + """CCXT fee cost converted to the pair's cash currency (quote on spot, + settle on derivatives) so it can be netted against PnL. Base-denominated + fees (typical for spot buys) use the fill price, quote-denominated fees + on an inverse contract are divided by it; a third currency (BNB, KCS, MX + discounts) is converted via `fetch_ticker("/")` on `ccxt_inst` + when given. When no rate can be found the fee is booked as 0 with a + warning — never as if it were cash (a 0.001 BNB fee is not 0.001 USDT).""" if not fee_info: return 0.0 try: cost = float(fee_info.get("cost", 0) or 0) except (TypeError, ValueError): return 0.0 + if not cost: + return 0.0 currency = fee_info.get("currency") - base = ccxt_symbol.split('/')[0] if '/' in ccxt_symbol else None - if currency and base and currency.upper() == base.upper() and price: - return cost * float(price) - return cost + if not currency or '/' not in ccxt_symbol: + return cost + spec = spec_for_instance(ccxt_inst, ccxt_symbol) if ccxt_inst is not None else None + if spec is None: + from backend.engine.contracts import spec_from_symbol + spec = spec_from_symbol(ccxt_symbol) + converted = spec.fee_cash_from_fill(fee_info, price) + if converted is not None: + return converted + ccy = str(currency).upper() + cash = spec.cash_currency + rate = None + if ccxt_inst is not None: + try: + ticker = ccxt_inst.fetch_ticker(f"{ccy}/{cash}") + rate = float(ticker.get("last") or ticker.get("close") or 0) or None + except Exception as exc: + logger.warning("fee_cash: no %s/%s rate for the fee on %s: %s", ccy, cash, ccxt_symbol, exc) + if rate: + return cost * rate + logger.warning("fee_cash: fee of %s %s on %s could not be converted to %s — booked as 0", cost, ccy, ccxt_symbol, cash) + return 0.0 + + +# Pre-sprint-D name, kept for the BotManager seam and older callers +fee_in_quote = fee_cash def wallet_held(balance: dict, token: str) -> float: """free + used of `token` from a ccxt fetch_balance() result.""" @@ -112,8 +171,8 @@ def reconcile_positions_with_wallet(db, bot, ccxt_inst, balance: dict, mode: str ids: dict = {} tol: dict = {} for pos in open_pos: - sym = str(pos.symbol).replace('-', '/').upper() - base = sym.split('/')[0] + sym = normalize(pos.symbol) + base = base_of(sym) expected[base] = expected.get(base, 0.0) + float(pos.amount or 0) ids.setdefault(base, []).append(pos.id) step = 0.0 @@ -132,12 +191,154 @@ def reconcile_positions_with_wallet(db, bot, ccxt_inst, balance: dict, mode: str problems.append(f"Position #{','.join(map(str, ids[base]))}: DB holds {want:g} {base} but exchange holds {held:g} {base}") return problems +def reconcile_positions_with_exchange(db, bot, ccxt_inst, mode: str): + """Derivatives counterpart of `reconcile_positions_with_wallet`: a + perpetual position holds no base coin, so compare the open `mode` + positions in the DB against `fetch_positions()` on the exchange. Returns + mismatch descriptions (empty when consistent): the exchange must hold at + least the DB amount (within two precision steps) on the same side.""" + open_pos = db.query(Position).filter( + Position.bot_name == bot.name, Position.status == "open", Position.mode == mode, + ).order_by(Position.id).all() + if not open_pos: + return [] + expected: dict = {} + ids: dict = {} + tol: dict = {} + for pos in open_pos: + sym = normalize(pos.symbol) + # Signed like the exchange side below: shorts count negative + _signed = -float(pos.amount or 0) if (pos.side or "long") == "short" else float(pos.amount or 0) + expected[sym] = expected.get(sym, 0.0) + _signed + ids.setdefault(sym, []).append(pos.id) + step = 0.0 + try: + prec = (ccxt_inst.market(sym).get("precision") or {}).get("amount") + if prec is not None: + prec = float(prec) + step = prec if prec < 1 else 10.0 ** (-prec) + except Exception: + pass + # Tolerance in position units: two precision steps (contracts) + tol[sym] = max(tol.get(sym, 0.0), spec_for_instance(ccxt_inst, sym).from_contracts(2 * step)) + held: dict = {} + for p in ccxt_inst.fetch_positions(list(expected)) or []: + sym = normalize(p.get("symbol")) + if sym not in expected: + continue + contracts = float(p.get("contracts") or 0) + if contracts <= 0: + continue + side = str(p.get("side") or "long").lower() + qty = spec_for_instance(ccxt_inst, sym).from_contracts(contracts) + held[sym] = held.get(sym, 0.0) + (qty if side == "long" else -qty) + problems = [] + for sym, want in expected.items(): + have = held.get(sym, 0.0) + tag = f"Position #{','.join(map(str, ids[sym]))}" + if want < 0: + # DB short: the exchange must hold at least as large a short + if have > 0: + problems.append(f"{tag}: DB holds a short of {-want:g} {sym} but the exchange holds a long") + elif -have + tol[sym] + 1e-12 < -want: + problems.append(f"{tag}: DB holds a short of {-want:g} {sym} but exchange position is {-have:g}") + elif have < 0: + problems.append(f"{tag}: DB holds a long of {want:g} {sym} but the exchange holds a short") + elif have + tol[sym] + 1e-12 < want: + problems.append(f"{tag}: DB holds {want:g} {sym} but exchange position is {have:g}") + return problems + + +# (key name, symbol, leverage, margin mode) already applied on the exchange +# in this process — set_leverage is a real API call, do it once, not per tick +_leverage_applied: set = set() +_leverage_lock = threading.Lock() + + +def invalidate_leverage_cache(key_name=None): + """Forget which (key, symbol, leverage, margin mode) combinations were + confirmed on the exchange — called when a key is saved or deleted so a + re-created key with the same name is set up again.""" + with _leverage_lock: + if key_name is None: + _leverage_applied.clear() + else: + for tag in [t for t in _leverage_applied if t[0] == str(key_name)]: + _leverage_applied.discard(tag) + + +def exchange_position_gone(ccxt_inst, ccxt_symbol, side="long"): + """True when `fetch_positions([symbol])` shows no open contracts on + `side` — the position was liquidated or closed outside the bot. Raises + when the exchange cannot be asked (the caller decides what to do).""" + sym = normalize(ccxt_symbol) + for p in ccxt_inst.fetch_positions([sym]) or []: + if normalize(p.get("symbol")) != sym: + continue + if float(p.get("contracts") or 0) <= 0: + continue + if str(p.get("side") or "long").lower() == side: + return False + return True + + +def ensure_leverage(ccxt_inst, api_key_record, ccxt_symbol, leverage, margin_mode, bot_name): + """Put the exchange's per-symbol leverage and margin mode in line with the + bot before it trades a derivative. Exchanges without one of the calls + (ccxt NotSupported) are skipped with an INFO line; 'leverage not + modified' style answers are fine. Any other failure raises — a bot must + not trade at an unknown leverage. No-op on spot.""" + if not is_derivative(ccxt_symbol): + return + lev = int(float(leverage or 1)) + mode = str(margin_mode or "isolated").lower() + tag = (str(api_key_record.name), normalize(ccxt_symbol), lev, mode) + with _leverage_lock: + if tag in _leverage_applied: + return + caps = market_caps(api_key_record.exchange, "swap") + import ccxt as _ccxt + if getattr(ccxt_inst, "has", {}).get("setMarginMode"): + try: + ccxt_inst.set_margin_mode(mode, ccxt_symbol, params={"leverage": lev}) + except _ccxt.NotSupported: + blb.push(bot_name, "INFO", f"{ccxt_symbol}: exchange does not support setting margin mode via API — set it to {mode} on the exchange") + except Exception as exc: + if not _benign_leverage_error(exc): + raise RuntimeError(f"set_margin_mode({mode}) failed for {ccxt_symbol}: {exc}") from exc + else: + blb.push(bot_name, "INFO", f"{ccxt_symbol}: exchange does not support setting margin mode via API — set it to {mode} on the exchange") + if caps is not None and caps.leverage_in_order: + blb.push(bot_name, "INFO", f"{ccxt_symbol}: leverage {lev}x is sent with each order") + elif getattr(ccxt_inst, "has", {}).get("setLeverage"): + try: + ccxt_inst.set_leverage(lev, ccxt_symbol, params={"marginMode": mode}) + except _ccxt.NotSupported: + raise RuntimeError(f"Exchange cannot set leverage for {ccxt_symbol} via API") + except Exception as exc: + if not _benign_leverage_error(exc): + raise RuntimeError(f"set_leverage({lev}) failed for {ccxt_symbol}: {exc}") from exc + else: + raise RuntimeError(f"Exchange cannot set leverage for {ccxt_symbol} via API") + blb.push(bot_name, "INFO", f"{ccxt_symbol}: leverage {lev}x ({mode}) confirmed on the exchange") + with _leverage_lock: + _leverage_applied.add(tag) + + +def _benign_leverage_error(exc) -> bool: + """'Already set' answers exchanges return as errors.""" + msg = str(exc).lower() + return any(t in msg for t in ("not modified", "no need to change", "already", "same as", "110043", "59000")) + + def get_live_capital(balance_cache, ccxt_inst, api_key_record, ccxt_symbol, bot_name, ttl=30): - """Free quote-currency balance on the exchange, cached briefly to spare - rate limits. Returns None when the balance cannot be determined so the - caller can fall back to the configured capital. `balance_cache` maps - (key_name, quote) -> (fetched_at, free).""" - quote = ccxt_symbol.split('/')[-1] + """Free balance of the pair's cash currency on the exchange (quote on + spot — USDT, EUR, BTC —, settle on derivatives — USDT on linear, BTC on + BTC/USD:BTC), cached briefly to spare rate limits. Returns None when the + balance cannot be determined so the caller can fall back to the + configured capital. `balance_cache` maps + (key_name, cash_currency) -> (fetched_at, free).""" + quote = cash_currency(ccxt_symbol) cache_key = (api_key_record.name, quote) now = time.monotonic() cached = balance_cache.get(cache_key) diff --git a/backend/engine/candle_poller.py b/backend/engine/candle_poller.py index 3a3a54b..d8b0ad9 100644 --- a/backend/engine/candle_poller.py +++ b/backend/engine/candle_poller.py @@ -9,7 +9,9 @@ from sqlalchemy.orm import Session from backend.core.database import SessionLocal -from backend.core.exchange_registry import build_exchange +from backend.core.exchange_registry import build_exchange_for_symbol, exchange_spec +from backend.engine.symbols import is_derivative, market_type_of +from backend.engine import funding from backend.core.events import event_bus from backend.models.bots import BotConfig from backend.models.candles import Candle @@ -36,7 +38,7 @@ def __init__(self): self.running = False self.needs_reconnect = False self._poll_tasks: list[asyncio.Task] = [] - self._exchange_cache: dict[str, tuple[float, object, threading.Lock]] = {} # exchange_id → (created_at, ccxt instance, lock) + self._exchange_cache: dict[tuple[str, str], tuple[float, object, threading.Lock]] = {} # (exchange_id, market type) → (created_at, ccxt instance, lock) self._exchange_cache_ttl = 3600 # rebuild exchange instances after 1 hour # Last closed candle ts per (exchange, symbol, timeframe), kept across # reconnects so a restarted poll task does not re-publish the same candle. @@ -261,7 +263,7 @@ def _backfill_one_symbol( lookback_limit: int, ): # Fresh instance per thread to avoid shared rate-limit state - exchange = build_exchange(exchange_name) + exchange = build_exchange_for_symbol(exchange_name, symbol) # Validate timeframe before attempting fetch try: @@ -455,7 +457,8 @@ def _backfill_one_symbol( if oldest_have is not None and oldest_have > requested_start + 2 * tf_ms and known_start != oldest_have: self._listing_start[(exchange_name, symbol, timeframe)] = oldest_have available = (now_ms - oldest_have) // tf_ms - cap_note = " (Kraken only serves its most recent 720 candles per timeframe)" if exchange_name == "kraken" else "" + spec = exchange_spec(exchange_name) + cap_note = f" ({spec.candle_limit_note})" if spec and spec.candle_limit_note else "" logger.warning( "Back-fill: %s/%s/%s — exchange has no data before %s%s; %d of the requested %d candles are available. " "Use a larger timeframe or another data exchange for a longer backtest.", @@ -514,7 +517,7 @@ async def _poll_symbol(self, exchange_name: str, symbol: str, timeframe: str): are not epoch-aligned (1w/1M) fall back to a fixed interval of max(10s, min(60s, tf_seconds / 4)). """ - exchange, ex_lock = self._get_public_exchange(exchange_name) + exchange, ex_lock = self._get_public_exchange(exchange_name, symbol) # Validate timeframe def _load_markets(): @@ -564,7 +567,7 @@ def _next_sleep(found_new: bool) -> float: def _fetch(limit): # Re-resolve from the cache every call so the TTL rebuild takes effect, # and hold the per-exchange lock: sync CCXT instances are not thread-safe. - inst, lock = self._get_public_exchange(exchange_name) + inst, lock = self._get_public_exchange(exchange_name, symbol) with lock: return inst.fetch_ohlcv(symbol, timeframe, None, limit) @@ -657,6 +660,16 @@ def db_op(): saved = await asyncio.to_thread(db_op) if not saved: return + if is_derivative(symbol): + # Top up the stored funding settlements before the bots see the + # candle (throttled to once an hour per symbol; errors logged) + def funding_op(): + db: Session = SessionLocal() + try: + funding.refresh_recent(db, exchange_name, symbol) + finally: + db.close() + await asyncio.to_thread(funding_op) await event_bus.publish("CANDLE_CLOSED", { "exchange": exchange_name, "symbol": symbol, @@ -687,18 +700,20 @@ async def _listen_for_bot_changes(self): # Internal helpers # ───────────────────────────────────────────────────────────────────────── - def _get_public_exchange(self, exchange_name: str): + def _get_public_exchange(self, exchange_name: str, symbol: str = ""): """Return (instance, lock) for a cached unauthenticated exchange used for - public market data. Callers must hold the lock around every fetch since - sync CCXT instances are not thread-safe.""" + public market data. Spot and swap symbols get separate instances (the + market type is read off the symbol). Callers must hold the lock around + every fetch since sync CCXT instances are not thread-safe.""" + cache_key = (exchange_name, market_type_of(symbol)) now = time.monotonic() - cached = self._exchange_cache.get(exchange_name) + cached = self._exchange_cache.get(cache_key) if cached and (now - cached[0]) < self._exchange_cache_ttl: return cached[1], cached[2] - instance = build_exchange(exchange_name) + instance = build_exchange_for_symbol(exchange_name, symbol) # Keep the existing lock across rebuilds so in-flight fetches stay serialized lock = cached[2] if cached else threading.Lock() - self._exchange_cache[exchange_name] = (now, instance, lock) + self._exchange_cache[cache_key] = (now, instance, lock) return instance, lock diff --git a/backend/engine/capital.py b/backend/engine/capital.py new file mode 100644 index 0000000..fd73fcb --- /dev/null +++ b/backend/engine/capital.py @@ -0,0 +1,91 @@ +"""Per-currency capital pools. + +A bot's money lives in exactly one cash currency per bot (the validator +refuses whitelists that mix USDT- and BTC-settled pairs), but the pool is +keyed by currency anyway so that a EUR spot bot, a USDT linear bot and a +BTC-margined inverse bot all keep their books in their own unit and no code +path ever adds a BTC amount to a USDT amount. + +`backtest_capital` in the settings stays a scalar; it is interpreted in the +bot's cash currency (`CapitalPools.for_bot`). + +Spot margin is deliberately not implemented: `lock` never lets `cash` go +negative (the backtest halts entries at zero, live sizing is capped by the +verified balance). A spot-margin implementation would relax that floor +here — and nowhere else. +""" +from dataclasses import dataclass, field + + +@dataclass +class Pool: + currency: str + start: float = 0.0 + cash: float = 0.0 # free, not locked in open positions + locked: float = 0.0 # margin + entry fees of open positions (informational) + + +@dataclass +class CapitalPools: + pools: dict = field(default_factory=dict) + + @classmethod + def for_bot(cls, currency: str, start: float) -> "CapitalPools": + """A bot's single pool: `backtest_capital` in its cash currency.""" + p = cls() + p.pools[currency] = Pool(currency=currency, start=float(start), cash=float(start)) + return p + + def pool(self, currency: str) -> Pool: + cur = str(currency or "").upper() + if cur not in self.pools: + self.pools[cur] = Pool(currency=cur) + return self.pools[cur] + + def cash(self, currency: str) -> float: + return self.pool(currency).cash + + def start(self, currency: str) -> float: + return self.pool(currency).start + + def lock(self, currency: str, amount: float) -> None: + """Take `amount` (margin + fee) out of the free cash. Floors at zero: + no spot margin (see module docstring).""" + p = self.pool(currency) + p.cash = max(p.cash - amount, 0.0) + p.locked += amount + + def release(self, currency: str, amount: float, locked: float | None = None) -> None: + """Return `amount` (margin + PnL − exit fee) to the free cash and + forget `locked` (what the closed layer had taken out).""" + p = self.pool(currency) + p.cash += amount + if locked is not None: + p.locked = max(p.locked - locked, 0.0) + + def forget(self, currency: str, locked: float) -> None: + """A layer's locked capital is gone for good (liquidation).""" + p = self.pool(currency) + p.locked = max(p.locked - locked, 0.0) + + def charge(self, currency: str, amount: float) -> None: + """Book a cash flow that is not a fill: a funding payment (negative) + or receipt (positive). The free cash may go below zero — the + exchange takes funding from the margin balance, and a negative + balance blocks new entries exactly like a depleted pool.""" + self.pool(currency).cash += float(amount or 0.0) + + def drain(self, currency: str) -> float: + """Cross-margin liquidation: the whole wallet is gone. Returns the + free cash that was lost on top of the positions' margins.""" + p = self.pool(currency) + lost = max(p.cash, 0.0) + p.cash = 0.0 + p.locked = 0.0 + return lost + + def currencies(self) -> list: + return list(self.pools) + + def single_currency(self) -> str | None: + return next(iter(self.pools)) if len(self.pools) == 1 else None diff --git a/backend/engine/contracts.py b/backend/engine/contracts.py new file mode 100644 index 0000000..0dd5c74 --- /dev/null +++ b/backend/engine/contracts.py @@ -0,0 +1,324 @@ +"""Contract economics: what a position of ``qty`` on a symbol is worth, what +it costs to open, what it pays out and in which currency. + +Three kinds of instrument exist in ApexAlgo: + +``spot`` + ``BTC/EUR``: ``qty`` is a base amount, everything is priced and settled + in the quote currency (EUR). +``linear`` + ``BTC/USDT:USDT``: a perpetual settled in its quote (stablecoin) currency. + ``qty`` is a base amount (the exchange counts contracts of + ``contract_size`` base units; the broker converts). PnL in the settle + currency is ``qty * (exit - entry)``, exactly like spot. +``inverse`` + ``BTC/USD:BTC``: a coin-margined perpetual. ``qty`` is a number of + contracts worth ``contract_size`` *quote* units (USD) each, margin and + PnL are in the base coin (BTC) and the PnL is + ``contracts * size * (1/entry - 1/exit)`` — non-linear in price. + +Every money path of the engine (backtest, forward, paper, live) goes through +a ``ContractSpec`` so that an amount of money never travels as a bare float +without a known currency (``cash_currency``). The spot/linear formulas are +kept literally identical to the pre-inverse engine so existing bots and the +golden backtests stay byte-for-byte the same. + +``qty`` semantics are the spec's: base units for spot/linear, contracts for +inverse. ``Position.amount`` stores exactly that, ``Position.contract_kind`` +and ``Position.contract_size`` say how to read it back. +""" +from dataclasses import dataclass + +from backend.engine.symbols import base_of, is_derivative, normalize, quote_of, settle_of + +KINDS = ("spot", "linear", "inverse") + +# Flat maintenance margin ratio: the fallback of the liquidation formulas +# when no exchange tier is stored for the pair (`engine/tiers.mmr_for`); +# the backtest summary reports which one applied as `mmr_source` +MAINTENANCE_MARGIN = 0.005 + + +def _direction(side): + return -1 if side == "short" else 1 + + +@dataclass(frozen=True) +class ContractSpec: + symbol: str + kind: str # "spot" | "linear" | "inverse" + base: str + quote: str + settle: str | None # None on spot + contract_size: float = 1.0 # linear: base units per contract; inverse: quote units per contract + + # ── identity ──────────────────────────────────────────────────────── + @property + def cash_currency(self) -> str: + """Currency the position is funded in and PnL is paid in.""" + return self.settle or self.quote + + @property + def is_inverse(self) -> bool: + return self.kind == "inverse" + + @property + def is_spot(self) -> bool: + return self.kind == "spot" + + @property + def market_type(self) -> str: + return "spot" if self.is_spot else "swap" + + # ── notional ──────────────────────────────────────────────────────── + def notional_cash(self, qty, price) -> float: + """Value of `qty` at `price` in the cash currency.""" + qty = float(qty or 0.0) + price = float(price or 0.0) + if self.is_inverse: + return qty * self.contract_size / price if price else 0.0 + return price * qty + + def notional_quote(self, qty, price) -> float: + """Value of `qty` at `price` in the quote currency (USD on + ``BTC/USD:BTC``); this is what `max_order_value` caps.""" + qty = float(qty or 0.0) + if self.is_inverse: + return qty * self.contract_size + return float(price or 0.0) * qty + + def base_amount(self, qty, price) -> float: + """`qty` expressed in base units (coins).""" + if self.is_inverse: + return self.notional_cash(qty, price) + return float(qty or 0.0) + + # ── sizing ────────────────────────────────────────────────────────── + def qty_for_cash(self, cash, price, leverage=1) -> float: + """Position size (spec units) that `cash` buys as margin at + `leverage`. Spot/linear keep the historical `cash / price * lev` + evaluation order.""" + price = float(price or 0.0) + if price <= 0: + return 0.0 + lev = float(leverage or 1) + if self.is_inverse: + qty = float(cash) * price / self.contract_size + else: + qty = float(cash) / price + if lev != 1.0: + qty *= lev + return qty + + # Plan name; identical to qty_for_cash (contracts on inverse, base otherwise) + contracts_for_cash = qty_for_cash + + def amount_for_cash(self, cash, price, leverage=1) -> float: + """Base amount (coins) that `cash` buys — spot/linear compatibility.""" + return self.base_amount(self.qty_for_cash(cash, price, leverage), price) + + def qty_for_quote_notional(self, quote_notional, price) -> float: + """Size whose quote notional is `quote_notional` (max_order_value cap).""" + price = float(price or 0.0) + if self.is_inverse: + return float(quote_notional) / self.contract_size + return float(quote_notional) / price if price else 0.0 + + def max_affordable_qty(self, cash, price, leverage=1, fee=0.0) -> float: + """Largest size whose locked capital (margin + entry fee on the + notional) fits in `cash`; spot reduces to `cash / (price * (1 + fee))`.""" + lev = max(float(leverage or 1), 1.0) + fee = float(fee or 0.0) + price = float(price or 0.0) + if self.is_inverse: + if price <= 0: + return 0.0 + # notional_cash = qty*size/price → qty = cash*price / (size*(1/lev+fee)) + return float(cash) * price / (self.contract_size * (1 / lev + fee)) + return float(cash) / (price * (1 / lev + fee)) + + # ── money in / out ────────────────────────────────────────────────── + def margin(self, qty, entry_price, leverage=1) -> float: + """Cash locked as margin (the full notional on spot / 1x).""" + lev = max(float(leverage or 1), 1.0) + n = self.notional_cash(qty, entry_price) + return n if lev == 1.0 and self.is_spot else n / lev + + def fee_cash(self, qty, price, fee_rate) -> float: + """Fee on the notional of `qty` at `price`, in the cash currency.""" + return self.notional_cash(qty, price) * float(fee_rate or 0.0) + + def locked_capital(self, qty, entry_price, leverage=1, fee=0.0) -> float: + """What leaves the pool on open: margin plus the entry fee on the + notional. Spot keeps ``notional * (1 + fee)`` literally.""" + n = self.notional_cash(qty, entry_price) + fee = float(fee or 0.0) + if self.is_spot: + return n * (1 + fee) + lev = max(float(leverage or 1), 1.0) + return n / lev + n * fee + + def pnl_cash(self, side, qty, entry_price, exit_price) -> float: + """Gross PnL in the cash currency before fees.""" + qty = float(qty or 0.0) + entry_price = float(entry_price or 0.0) + exit_price = float(exit_price or 0.0) + d = _direction(side) + if self.is_inverse: + if not entry_price or not exit_price: + return 0.0 + return d * qty * self.contract_size * (1 / entry_price - 1 / exit_price) + return d * (exit_price - entry_price) * qty + + def close_return(self, side, qty, entry_price, exit_price, leverage=1, fee=0.0) -> float: + """What comes back to the pool when `qty` closes at `exit_price`: + the sale proceeds net of fee on spot; margin + PnL − exit fee on + derivatives.""" + if self.is_spot: + return float(exit_price) * float(qty) * (1 - float(fee or 0.0)) + return self.margin(qty, entry_price, leverage) + self.pnl_cash(side, qty, entry_price, exit_price) \ + - self.fee_cash(qty, exit_price, fee) + + def realized_pnl(self, side, qty, entry_price, exit_price, leverage=1, entry_fee=0.0, exit_fee=0.0) -> float: + """Net PnL of closing `qty`: fees on both legs included.""" + if self.is_spot or side != "short": + # Long legs: proceeds/return minus what was locked (spot-identical form) + return self.close_return(side, qty, entry_price, exit_price, leverage, exit_fee) \ + - self.locked_capital(qty, entry_price, leverage, entry_fee) + return self.pnl_cash(side, qty, entry_price, exit_price) \ + - self.fee_cash(qty, entry_price, entry_fee) - self.fee_cash(qty, exit_price, exit_fee) + + def mark_value(self, positions, price, leverage=1) -> float: + """Mark-to-market value of open `positions` (objects with + `side/entry_price/amount`) at `price`, in the cash currency.""" + if self.is_spot: + return sum(float(p.amount or 0.0) for p in positions) * float(price) + return sum(self.margin(p.amount, p.entry_price, leverage) + + self.pnl_cash(p.side, p.amount, p.entry_price, price) for p in positions) + + # ── liquidation ───────────────────────────────────────────────────── + def liquidation_price(self, side, entry_price, leverage, mmr=MAINTENANCE_MARGIN): + """Price at which the margin is exhausted up to the maintenance + margin. Linear: ``entry*(1 ∓ (1-mmr)/lev)``; a 1x linear/spot long has + no liquidation level (None). Inverse long: ``entry*lev/(lev+1-mmr)``, + inverse short: ``entry*lev/(lev-1+mmr)``.""" + if not entry_price: + return None + entry_price = float(entry_price) + lev = max(float(leverage or 1), 1.0) + short = side == "short" + if self.is_inverse: + return entry_price * lev / (lev - 1 + mmr) if short else entry_price * lev / (lev + 1 - mmr) + if lev <= 1 and not short: + return None + move = (1 - mmr) / lev + return entry_price * (1 + move) if short else entry_price * (1 - move) + + def liquidation_loss(self, qty, entry_price, leverage=1, entry_fee=0.0) -> float: + """Cash lost on liquidation: the whole margin plus the entry fee + already paid (no exit fee is charged).""" + return self.margin(qty, entry_price, leverage) + self.fee_cash(qty, entry_price, entry_fee) + + # ── exchange plumbing ─────────────────────────────────────────────── + def to_contracts(self, qty) -> float: + """Exchange order amount: linear contracts of `contract_size` base + units, contracts as-is on inverse, base amount on spot.""" + qty = float(qty or 0.0) + if self.kind == "linear" and self.contract_size and self.contract_size != 1.0: + return qty / self.contract_size + return qty + + def from_contracts(self, contracts) -> float: + """Inverse of `to_contracts`.""" + contracts = float(contracts or 0.0) + if self.kind == "linear" and self.contract_size and self.contract_size != 1.0: + return contracts * self.contract_size + return contracts + + def fee_cash_from_fill(self, fee, price): + """Convert a ccxt fee dict (`{cost, currency}`) into the cash + currency at `price`; None when it cannot be valued.""" + if not isinstance(fee, dict): + return None + cost = fee.get("cost") + if cost is None: + return None + try: + cost = float(cost) + except (TypeError, ValueError): + return None + cur = str(fee.get("currency") or "").upper() + cash = self.cash_currency + if not cur or cur == cash: + return cost + price = float(price or 0.0) + if price <= 0: + return None + if cur == self.base and cash == self.quote: + return cost * price + if cur == self.quote and cash == self.base: + return cost / price + return None + + +def spec_from_symbol(symbol, contract_size=None) -> ContractSpec: + """Fallback spec from the unified symbol alone: spot without ``:``, + linear when the settle equals the quote, inverse when it equals the + base. `contract_size` defaults to 1 (one base unit / one quote unit).""" + sym = normalize(symbol) + base, quote, settle = base_of(sym), quote_of(sym), settle_of(sym) + if not is_derivative(sym): + kind = "spot" + elif settle == base: + kind = "inverse" + else: + kind = "linear" + size = float(contract_size) if contract_size else 1.0 + return ContractSpec(symbol=sym, kind=kind, base=base, quote=quote, settle=settle, contract_size=size) + + +def spec_from_market(market: dict, symbol=None) -> ContractSpec: + """Spec from a ccxt market dict (`market()` / `load_markets()` entry).""" + if not market: + return spec_from_symbol(symbol) + sym = normalize(market.get("symbol") or symbol) + base = str(market.get("base") or base_of(sym)).upper() + quote = str(market.get("quote") or quote_of(sym)).upper() + settle = market.get("settle") + settle = str(settle).upper() if settle else settle_of(sym) + if market.get("spot") or not (market.get("contract") or settle): + kind = "spot" + elif market.get("inverse") or (settle and settle == base and not market.get("linear")): + kind = "inverse" + else: + kind = "linear" + size = market.get("contractSize") + try: + size = float(size) if size else 1.0 + except (TypeError, ValueError): + size = 1.0 + return ContractSpec(symbol=sym, kind=kind, base=base, quote=quote, + settle=settle if kind != "spot" else None, contract_size=size) + + +def spec_for(exchange_id, symbol, market=None) -> ContractSpec: + """Spec for `symbol` on `exchange_id`: from the given market dict, else + from the registry's cached markets, else from the symbol alone. Never + touches the network.""" + from backend.core.exchange_registry import contract_spec + return contract_spec(exchange_id, symbol, market=market) + + +def spec_for_instance(exchange, symbol) -> ContractSpec: + """Spec from a ccxt instance whose markets are loaded (falls back to the + symbol when the market is unknown).""" + market = None + try: + markets = getattr(exchange, "markets", None) or {} + market = markets.get(symbol) if isinstance(markets, dict) else None + if market is None and hasattr(exchange, "market"): + market = exchange.market(symbol) + except Exception: + market = None + ex_id = getattr(exchange, "id", None) + return spec_for(ex_id, symbol, market=market) if ex_id else spec_from_market(market, symbol) diff --git a/backend/engine/exits.py b/backend/engine/exits.py index 07d73a0..5b8ba05 100644 --- a/backend/engine/exits.py +++ b/backend/engine/exits.py @@ -1,6 +1,15 @@ """Exit rules shared by the backtest and the live tick: stop losses, take -profits and the strategy sell signal, evaluated on one candle.""" +profits and the strategy sell/cover signal, evaluated on one candle. + +One direction-parameterised body serves both sides. With `dir = +1` (long) +every formula reduces literally to the original spot code; `dir = -1` (short) +mirrors it: a stop loss sits ABOVE the entry and is hit by the candle high, a +take profit BELOW and is hit by the low, trailing levels anchor to the lowest +price reached so far (stored in the same `highest_price` slot — opposite +extreme) and the strategy exit is the `cover` signal sized by +`trade_settings.cover`.""" import logging +from backend.engine import pnl from backend.engine.sizing import _num logger = logging.getLogger("apexalgo.bot_manager") @@ -8,70 +17,113 @@ VALID_EXIT_TYPES = {'percentage', 'trailing', 'atr', 'fixed'} -def check_exits(position_states, states_lock, open_position, row_close, row_high, row_low, is_sell_signal, bot_settings, current_atr=0.0, row_open=None): - """Stop-loss / take-profit / strategy-sell events for one position on one - candle. `position_states` (keyed by position id) carries the trailing peak - and the exits already fired; `states_lock` guards it across threads.""" +def _better(dir, a, b): + """The fill that favours the position: the higher price for a long, the + lower for a short.""" + return max(a, b) if dir > 0 else min(a, b) + + +def _worse(dir, a, b): + """The fill that goes against the position (gap-through-trigger fills).""" + return min(a, b) if dir > 0 else max(a, b) + + +def _reached_favorable(dir, price, level): + """`price` is at or beyond `level` in the profitable direction.""" + return price >= level if dir > 0 else price <= level + + +def _reached_adverse(dir, price, level): + """`price` is at or beyond `level` in the losing direction.""" + return price <= level if dir > 0 else price >= level + + +def _adverse_level(dir, base, pct): + """`base` moved `pct` percent against the position.""" + return base * (1 - dir * (pct / 100)) + + +def _favorable_level(dir, base, pct): + """`base` moved `pct` percent in favour of the position.""" + return base * (1 + dir * (pct / 100)) + + +def _parse_rule(rule, i, kind, prefix): + """(value, type) of one SL/TP rule, or None when it must be skipped.""" + try: + val = float(rule.get('value', 0)) + except (ValueError, TypeError): + return None + if val <= 0: + return None + rtype = rule.get('type', '') + if rtype not in VALID_EXIT_TYPES: + logger.warning("Invalid %s type '%s' for %s_%d, skipping", kind, rtype, prefix, i) + return None + return val, rtype + + +def check_exits(position_states, states_lock, open_position, row_close, row_high, row_low, is_sell_signal, bot_settings, current_atr=0.0, row_open=None, side="long"): + """Stop-loss / take-profit / strategy-exit events for one position on one + candle. `position_states` (keyed by position id) carries the trailing + extreme and the exits already fired; `states_lock` guards it across + threads. `side="short"` evaluates the mirrored rule set; `is_sell_signal` + is then the cover signal.""" + dir = pnl.direction(side) trade_settings = bot_settings.get("trade_settings", {}) - entry_settings = trade_settings.get("entry", {}) + entry_settings = pnl.entry_cfg(trade_settings, side) events = [] if row_open is None: row_open = row_close + # The candle extreme that moves against / in favour of the position + adverse = row_low if dir > 0 else row_high + favorable = row_high if dir > 0 else row_low with states_lock: state = position_states.get(open_position.id) if not state or state.get('entry_price') != open_position.entry_price: # Restore persisted state from DB, or initialize fresh - persisted_highest = open_position.highest_price or open_position.entry_price + persisted_extreme = open_position.highest_price or open_position.entry_price persisted_exits = set(open_position.triggered_exits or []) state = { 'entry_price': open_position.entry_price, - 'highest_price': persisted_highest, + 'highest_price': persisted_extreme, # lowest price reached for a short 'triggered_exits': persisted_exits } position_states[open_position.id] = state - # Trailing levels anchor to the peak reached BEFORE this candle; the - # current candle's high is folded in afterwards so one candle cannot - # both raise the trail and trigger it against its own low. - prev_highest = state['highest_price'] + # Trailing levels anchor to the extreme reached BEFORE this candle; the + # current candle's extreme is folded in afterwards so one candle cannot + # both move the trail and trigger it against its own range. + prev_extreme = state['highest_price'] triggered_exits = set(state['triggered_exits']) sl_hit = False for i, sl in enumerate(entry_settings.get("stop_losses", [])): sl_id = f"sl_{i}" if sl_id in triggered_exits: continue - - try: - sl_val = float(sl.get('value', 0)) - except (ValueError, TypeError): - continue - - if sl_val <= 0: continue - - sl_type = sl.get('type', '') - if sl_type not in VALID_EXIT_TYPES: - logger.warning("Invalid stop_loss type '%s' for sl_%d, skipping", sl_type, i) - continue + parsed = _parse_rule(sl, i, "stop_loss", "sl") + if parsed is None: continue + sl_val, sl_type = parsed if sl_type == 'percentage': - trigger_price = open_position.entry_price * (1 - (sl_val/100)) + trigger_price = _adverse_level(dir, open_position.entry_price, sl_val) elif sl_type == 'trailing': - trigger_price = prev_highest * (1 - (sl_val/100)) + trigger_price = _adverse_level(dir, prev_extreme, sl_val) elif sl_type == 'atr': if not (current_atr > 0): continue - trigger_price = prev_highest - (sl_val * current_atr) + trigger_price = prev_extreme - dir * (sl_val * current_atr) else: trigger_price = sl_val - if row_low <= trigger_price: + if _reached_adverse(dir, adverse, trigger_price): sl_close_type = sl.get('close_amount_type', 'percentage') sl_close_val = float(sl.get('close_amount_value', 100)) events.append({ 'qty_pct': sl_close_val, 'close_amount_type': sl_close_type, 'reason': "stop_loss", - # A gap below the trigger fills at the open, not the trigger - 'price': min(trigger_price, row_open), + # A gap through the trigger fills at the (worse) open, not the trigger + 'price': _worse(dir, trigger_price, row_open), 'id': sl_id }) sl_hit = True @@ -81,47 +133,40 @@ def check_exits(position_states, states_lock, open_position, row_close, row_high for i, tp in enumerate(entry_settings.get("take_profits", [])): tp_id = f"tp_{i}" if tp_id in triggered_exits: continue - - try: - tp_val = float(tp.get('value', 0)) - except (ValueError, TypeError): - continue - - if tp_val <= 0: continue - - tp_type = tp.get('type', '') - if tp_type not in VALID_EXIT_TYPES: - logger.warning("Invalid take_profit type '%s' for tp_%d, skipping", tp_type, i) - continue + parsed = _parse_rule(tp, i, "take_profit", "tp") + if parsed is None: continue + tp_val, tp_type = parsed tp_close_type = tp.get('close_amount_type', 'percentage') tp_close_val = float(tp.get('close_amount_value', 100)) if tp_type == 'percentage': - t_price = open_position.entry_price * (1 + (tp_val/100)) - if row_high >= t_price: - # A gap above the target fills at the (better) open price - tps.append({'id': tp_id, 'price': max(t_price, row_open), 'pct': tp_close_val, 'close_amount_type': tp_close_type}) + t_price = _favorable_level(dir, open_position.entry_price, tp_val) + if _reached_favorable(dir, favorable, t_price): + # A gap through the target fills at the (better) open price + tps.append({'id': tp_id, 'price': _better(dir, t_price, row_open), 'pct': tp_close_val, 'close_amount_type': tp_close_type}) elif tp_type == 'trailing': - # Trailing TP: price must first rise above entry by tp_val%, then - # we close when price drops tp_val% from the highest price reached. - activation_price = open_position.entry_price * (1 + (tp_val / 100)) - if prev_highest >= activation_price: - # Once activated, trail below the peak - t_price = prev_highest * (1 - (tp_val / 100)) - if row_low <= t_price: - tps.append({'id': tp_id, 'price': min(t_price, row_open), 'pct': tp_close_val, 'close_amount_type': tp_close_type}) + # Trailing TP: price must first move tp_val% in favour of the + # position, then we close when it retraces tp_val% from the + # extreme reached. + activation_price = _favorable_level(dir, open_position.entry_price, tp_val) + if _reached_favorable(dir, prev_extreme, activation_price): + # Once activated, trail behind the extreme + t_price = _adverse_level(dir, prev_extreme, tp_val) + if _reached_adverse(dir, adverse, t_price): + tps.append({'id': tp_id, 'price': _worse(dir, t_price, row_open), 'pct': tp_close_val, 'close_amount_type': tp_close_type}) elif tp_type == 'atr': if not (current_atr > 0): continue - t_price = prev_highest - (tp_val * current_atr) - if row_low <= t_price: - tps.append({'id': tp_id, 'price': min(t_price, row_open), 'pct': tp_close_val, 'close_amount_type': tp_close_type}) + t_price = prev_extreme - dir * (tp_val * current_atr) + if _reached_adverse(dir, adverse, t_price): + tps.append({'id': tp_id, 'price': _worse(dir, t_price, row_open), 'pct': tp_close_val, 'close_amount_type': tp_close_type}) else: t_price = tp_val - if row_high >= t_price: - tps.append({'id': tp_id, 'price': max(t_price, row_open), 'pct': tp_close_val, 'close_amount_type': tp_close_type}) + if _reached_favorable(dir, favorable, t_price): + tps.append({'id': tp_id, 'price': _better(dir, t_price, row_open), 'pct': tp_close_val, 'close_amount_type': tp_close_type}) - tps = sorted(tps, key=lambda x: x['price'], reverse=True) + # Best fill first: highest price for a long, lowest for a short + tps = sorted(tps, key=lambda x: dir * x['price'], reverse=True) for tp in tps: events.append({ @@ -133,17 +178,17 @@ def check_exits(position_states, states_lock, open_position, row_close, row_high }) if not events and is_sell_signal: - exit_settings = trade_settings.get("exit", {}) + exit_settings = pnl.exit_cfg(trade_settings, side) pct_to_close = _num(exit_settings.get('amount_value'), 100) if exit_settings.get('amount_type') == 'percentage' else 100 events.append({ 'qty_pct': pct_to_close, 'reason': "strategy", 'price': row_close, - 'id': 'strategy_sell' + 'id': 'strategy_cover' if dir < 0 else 'strategy_sell' }) with states_lock: - state['highest_price'] = max(state['highest_price'], row_high) + state['highest_price'] = _better(dir, state['highest_price'], favorable) # Persist state back to DB for crash recovery open_position.highest_price = state['highest_price'] diff --git a/backend/engine/funding.py b/backend/engine/funding.py new file mode 100644 index 0000000..836f5a5 --- /dev/null +++ b/backend/engine/funding.py @@ -0,0 +1,175 @@ +"""Funding-rate simulation for perpetual swaps. + +Every funding settlement of a perpetual moves money between longs and +shorts: at each settlement a position pays ``rate * notional`` (a long pays +when the rate is positive, a short receives it; negative rates reverse the +flow). The backtest and the forward test book these payments on their open +positions from the stored ``funding_rates`` rows; paper/live positions are +charged by the exchange itself and are not booked here. + +Data path: ``ensure_history`` fetches what the exchange still has +(``fetch_funding_rate_history``, paginated) for the part of the window that +is not stored yet, ``load`` returns the sorted events of a window, and +``settlements`` picks the ones a position has not been charged for +(``funding_until < ts <= candle_ts``). Stored rows are never overwritten. +""" +import logging +import time +from datetime import datetime, timedelta, timezone + +from sqlalchemy import text + +from backend.engine.sizing import _naive_utc +from backend.models.market_data import FundingRate + +logger = logging.getLogger("apexalgo.funding") + +_PAGE = 1000 +_MAX_PAGES = 200 # safety valve on the pagination loop +# The poller refreshes a symbol's funding at most this often (seconds) +REFRESH_EVERY = 3600 +_last_refresh: dict[tuple, float] = {} + + +def _ms(ts) -> int: + ts = _naive_utc(ts) + return int(ts.replace(tzinfo=timezone.utc).timestamp() * 1000) + + +def _from_ms(ms) -> datetime: + return datetime.fromtimestamp(int(ms) / 1000, tz=timezone.utc).replace(tzinfo=None) + + +def fetch_history(exchange_id: str, symbol: str, since_ms: int, until_ms: int) -> list: + """`[(timestamp_ms, rate)]` from the exchange for `[since_ms, until_ms]`. + Exchanges keep a limited history (OKX ~3 months, Binance years); what + they no longer serve is simply not returned. Raises on network errors.""" + from backend.core.exchange_registry import build_exchange_for_symbol + exchange = build_exchange_for_symbol(exchange_id, symbol) + if not exchange.has.get("fetchFundingRateHistory"): + return [] + exchange.load_markets() + out = [] + since = int(since_ms) + seen = set() + for _ in range(_MAX_PAGES): + rows = exchange.fetch_funding_rate_history(symbol, since=since, limit=_PAGE) or [] + fresh = [] + for r in rows: + ts = r.get("timestamp") + rate = r.get("fundingRate") + if ts is None or rate is None or ts in seen: + continue + seen.add(ts) + fresh.append((int(ts), float(rate))) + if not fresh: + break + fresh.sort() + out.extend(t for t in fresh if since_ms <= t[0] <= until_ms) + last = fresh[-1][0] + if last >= until_ms or len(rows) < 2 or last <= since: + break + since = last + 1 + out.sort() + return out + + +def store(db, exchange_id: str, symbol: str, rows: list) -> int: + """Insert `[(timestamp_ms, rate)]`, ignoring settlements already stored.""" + if not rows: + return 0 + now = _naive_utc(datetime.now(timezone.utc)) + n = 0 + for ts_ms, rate in rows: + res = db.execute(text( + "INSERT OR IGNORE INTO funding_rates (exchange, symbol, timestamp, rate, fetched_at) " + "VALUES (:ex, :sym, :ts, :rate, :now)" + ), {"ex": exchange_id, "sym": symbol, "ts": _from_ms(ts_ms), "rate": float(rate), "now": now}) + n += res.rowcount or 0 + db.commit() + return n + + +def stored_range(db, exchange_id: str, symbol: str): + """(first, last, count) of the stored settlements for the symbol.""" + from sqlalchemy import func + row = db.query(func.min(FundingRate.timestamp), func.max(FundingRate.timestamp), func.count(FundingRate.id)).filter( + FundingRate.exchange == exchange_id, FundingRate.symbol == symbol).first() + return (row[0], row[1], int(row[2] or 0)) if row else (None, None, 0) + + +def ensure_history(db, exchange_id: str, symbol: str, start, end) -> int: + """Fetch and store the settlements of `[start, end]` that are not stored + yet (only the head and/or tail beyond the stored range are requested). + Returns the number of new rows; raises on exchange errors so the caller + can report them.""" + start, end = _naive_utc(start), _naive_utc(end) + first, last, _ = stored_range(db, exchange_id, symbol) + new = 0 + if first is None or last is None: + new += store(db, exchange_id, symbol, fetch_history(exchange_id, symbol, _ms(start), _ms(end))) + return new + if start < first: + new += store(db, exchange_id, symbol, fetch_history(exchange_id, symbol, _ms(start), _ms(first) - 1)) + if end > last: + new += store(db, exchange_id, symbol, fetch_history(exchange_id, symbol, _ms(last) + 1, _ms(end))) + return new + + +def refresh_recent(db, exchange_id: str, symbol: str, lookback_days: int = 7, force: bool = False) -> int: + """Poller hook: top up the stored settlements to now, at most once per + `REFRESH_EVERY` seconds per symbol. Errors are logged, never raised.""" + key = (exchange_id, symbol) + now = time.monotonic() + if not force and now - _last_refresh.get(key, -1e9) < REFRESH_EVERY: + return 0 + _last_refresh[key] = now + try: + end = datetime.now(timezone.utc).replace(tzinfo=None) + return ensure_history(db, exchange_id, symbol, end - timedelta(days=lookback_days), end) + except Exception as exc: + logger.warning("Funding refresh failed for %s %s: %s", exchange_id, symbol, exc) + return 0 + + +def load(db, exchange_id: str, symbol: str, start=None, end=None) -> list: + """Sorted `[(timestamp, rate)]` (naive UTC) of the stored settlements in + `[start, end]`; the whole history when no bounds are given.""" + q = db.query(FundingRate.timestamp, FundingRate.rate).filter( + FundingRate.exchange == exchange_id, FundingRate.symbol == symbol) + if start is not None: + q = q.filter(FundingRate.timestamp >= _naive_utc(start)) + if end is not None: + q = q.filter(FundingRate.timestamp <= _naive_utc(end)) + return [(_naive_utc(ts), float(rate)) for ts, rate in q.order_by(FundingRate.timestamp.asc()).all()] + + +def settlements(events: list, after, until) -> list: + """The events with `after < ts <= until` (both naive UTC).""" + if not events: + return [] + after = _naive_utc(after) if after is not None else None + until = _naive_utc(until) + return [(ts, r) for ts, r in events if (after is None or ts > after) and ts <= until] + + +def payment(spec, side, qty, price, rate) -> float: + """Cash flow of one settlement for a position of `qty` at `price`: + ``-direction * rate * notional`` — positive when the position receives + funding. A long pays a positive rate, a short receives it.""" + from backend.engine.pnl import direction + return -direction(side) * float(rate) * spec.notional_cash(qty, price) + + +def coverage(events: list, start, end) -> dict: + """What the stored settlements cover of a backtest window — reported in + the summary so a run on a partial funding history is recognisable.""" + if not events: + return {"funding": "no data", "funding_events": 0} + first, last = events[0][0], events[-1][0] + return { + "funding": "simulated", + "funding_events": len(events), + "funding_from": first.isoformat(), + "funding_to": last.isoformat(), + } diff --git a/backend/engine/live_cycle.py b/backend/engine/live_cycle.py index c901a0b..d6d0dfa 100644 --- a/backend/engine/live_cycle.py +++ b/backend/engine/live_cycle.py @@ -9,7 +9,7 @@ from collections import defaultdict import pandas as pd import ccxt -from sqlalchemy import text, func +from sqlalchemy import text, func, or_, and_ from sqlalchemy.orm import selectinload from backend.core.database import SessionLocal from backend.models.bots import BotConfig @@ -18,12 +18,206 @@ from backend.models.positions import Position from backend.models.exchange_keys import ExchangeKey from backend.engine.evaluator import NodeEvaluator -from backend.engine.sizing import _indicator_fingerprint, _num, _int, _tf_seconds, _naive_utc +from backend.engine.sizing import _indicator_fingerprint, _num, _int, _tf_seconds, _naive_utc, cap_by_max_order_value, position_spec +from backend.engine.symbols import DEFAULT_MARGIN_MODE, base_of, is_derivative, leverage_for, market_type_for, normalize +from backend.engine import funding, tiers +from backend.engine import broker, pnl, risk +from backend.engine.contracts import spec_for, spec_for_instance +from backend.core.exchange_registry import market_caps from backend.core import bot_log_buffer as blb logger = logging.getLogger("apexalgo.bot_manager") +# ── Derivative order plumbing ───────────────────────────────────────────── +# Spot orders keep their literal create_market_buy/sell_order calls; the +# helpers below only run for `BASE/QUOTE:SETTLE` symbols, where ccxt wants +# the amount in contracts, closes must be reduce-only, and a few exchanges +# take the leverage per order. + +def _spec(ccxt_inst, exchange, ccxt_symbol): + """ContractSpec for a symbol: from the exchange instance's market when + there is one, else the registry cache / the symbol's own shape.""" + if ccxt_inst is not None: + return spec_for_instance(ccxt_inst, ccxt_symbol) + return spec_for(exchange, ccxt_symbol) + + +def _pos_spec(pos): + """ContractSpec a stored position was booked with.""" + return position_spec(pos.symbol, pos.contract_kind, pos.contract_size, pos.exchange) + + +def _precise_amount(ccxt_inst, ccxt_symbol, qty): + """Round a size to the market's precision. On derivatives the precision + is defined in contracts, so convert, round, convert back.""" + if not is_derivative(ccxt_symbol): + return float(ccxt_inst.amount_to_precision(ccxt_symbol, qty)) + spec = spec_for_instance(ccxt_inst, ccxt_symbol) + contracts = float(ccxt_inst.amount_to_precision(ccxt_symbol, spec.to_contracts(qty))) + return spec.from_contracts(contracts) + + +def _place_market_order(ccxt_inst, api_key_record, ccxt_symbol, side, qty, *, reduce_only, leverage=1): + """Market order of `qty` (base units on spot/linear, contracts on + inverse). Spot: the pre-existing ccxt shortcuts. Derivatives: + `create_order` in contracts, reduce-only on closes and, where the + exchange wants it, the leverage in the params.""" + if not is_derivative(ccxt_symbol): + if side == "buy": + return ccxt_inst.create_market_buy_order(ccxt_symbol, qty) + return ccxt_inst.create_market_sell_order(ccxt_symbol, qty) + contracts = spec_for_instance(ccxt_inst, ccxt_symbol).to_contracts(qty) + params = {} + if reduce_only: + params["reduceOnly"] = True + caps = market_caps(getattr(api_key_record, "exchange", ""), "swap") + if caps is not None and caps.leverage_in_order: + params["leverage"] = int(float(leverage or 1)) + return ccxt_inst.create_order(ccxt_symbol, "market", side, contracts, None, params) + + +def _filled_base(ccxt_inst, ccxt_symbol, ex_order) -> float: + """`filled` of a ccxt order in the position's units (contracts x + contract size on linear perpetuals, contracts as-is on inverse).""" + filled = float(ex_order.get("filled") or 0) + if filled > 0 and is_derivative(ccxt_symbol): + return spec_for_instance(ccxt_inst, ccxt_symbol).from_contracts(filled) + return filled + + +def _fee_cols(fee_info, cash_ccy): + """`fee_currency`/`fee_cash` Order columns from a ccxt fee dict (what + the exchange charged, before conversion into the cash currency).""" + if isinstance(fee_info, dict) and fee_info.get("cost") is not None: + try: + return {"fee_currency": str(fee_info.get("currency") or cash_ccy).upper(), "fee_cash": float(fee_info.get("cost"))} + except (TypeError, ValueError): + pass + return {"fee_currency": cash_ccy} + + +def _entry_fee_total(pos, open_side): + """Entry fees paid on the position's opening fills (cash currency).""" + return sum((o.fee or 0.0) for o in (pos.orders or []) if o.side == open_side and o.status == "filled") + + +def _original_amount(pos, open_side): + """Size the position was opened with (sum of its opening fills).""" + return sum(o.amount for o in (pos.orders or []) if o.side == open_side and o.status == "filled") or pos.amount + + +def _book_liquidation(engine, db, bot, pos, price, ts, dd_group, note, extra_loss=0.0): + """Close `pos` as liquidated: the remaining margin and its share of the + entry fee are lost, nothing comes back to the pool, no exit fee (same + booking as the backtest). Adds the reduce-only close order at `price` + and updates the drawdown state. `extra_loss` is the share of the free + cash a cross-margin liquidation takes on top of the margin, so the + booked losses add up to the equity that vanished.""" + pos_side = pos.side or "long" + open_side = pnl.open_order_side(pos_side) + remaining = float(pos.amount or 0.0) + original = _original_amount(pos, open_side) + lev = max(float(pos.leverage or 1), 1.0) + spec = _pos_spec(pos) + fee_portion = _entry_fee_total(pos, open_side) * (remaining / original) if original > 0 else 0.0 + own_loss = spec.margin(remaining, pos.entry_price or 0.0, lev) + fee_portion + loss = own_loss + float(extra_loss or 0.0) + close_cols = {"market_type": pos.market_type or "swap", "reduce_only": 1, "fee_currency": spec.cash_currency} + db.add(Order(position_id=pos.id, exchange=pos.exchange, bot_name=bot.name, mode=pos.mode, symbol=pos.symbol, + side=pnl.close_order_side(pos_side), order_type="market", price=price, amount=remaining, + timestamp=ts, exchange_order_id=f"liq_{uuid.uuid4().hex[:8]}", status="filled", fee=0.0, **close_cols)) + pos.profit_abs = (pos.profit_abs or 0.0) - loss + if original > 0 and own_loss > 0: + pos.profit_pct = (pos.profit_pct or 0.0) - 100.0 * (loss / own_loss) * (remaining / original) + with engine._position_states_lock: + st = engine.position_states.pop(pos.id, None) + triggered = set((st or {}).get("triggered_exits") or ()) | set(pos.triggered_exits or ()) + pos.triggered_exits = sorted(str(t) for t in triggered) + ["liquidation"] + pos.status = "closed" + pos.closed_at = ts + engine._update_drawdown(bot.name, dd_group, -loss) + # Commit before the console line: the log buffer writes on its own + # connection and would otherwise wait on this session's rows + db.commit() + logger.warning("%s LIQUIDATION %s #%d: %s @ %s (loss %.2f %s) — %s", pos.mode.upper(), pos.symbol, pos.id, remaining, price, loss, spec.cash_currency, note) + blb.push(bot.name, "WARN", f"{pos.mode.upper()} LIQUIDATION {pos.symbol}: {remaining:g} @ {price:g} (−{loss:.2f} {spec.cash_currency}) — {note}") + return loss + + +def _apply_forward_funding(engine, db, bot, positions, exchange, ccxt_symbol, price, candle_ts, dd_group, events_cache): + """Forward test: charge every stored funding settlement a position has + not been charged for yet (`funding_until < ts <= candle_ts`) on its + notional at `price` — the backtest rule, from the same table. Returns + the net amount booked.""" + key = (exchange, ccxt_symbol) + if key not in events_cache: + events_cache[key] = funding.load(db, exchange, ccxt_symbol) + events = events_cache[key] + if not events: + return 0.0 + total = 0.0 + for pos in positions: + if pos.status != "open": + continue + pos_side = pos.side or "long" + spec = _pos_spec(pos) + open_side = pnl.open_order_side(pos_side) + original = _original_amount(pos, open_side) + lev = max(float(pos.leverage or 1), 1.0) + locked = spec.margin(original, pos.entry_price or 0.0, lev) + _entry_fee_total(pos, open_side) if original > 0 else 0.0 + booked = 0.0 + for f_ts, rate in funding.settlements(events, pos.funding_until or pos.created_at, candle_ts): + pay = funding.payment(spec, pos_side, pos.amount or 0.0, price, rate) + pos.profit_abs = (pos.profit_abs or 0.0) + pay + pos.funding_paid = (pos.funding_paid or 0.0) + pay + if locked > 0: + pos.profit_pct = (pos.profit_pct or 0.0) + 100.0 * pay / locked + pos.funding_until = f_ts + booked += pay + if booked: + engine._update_drawdown(bot.name, dd_group, booked) + total += booked + blb.push(bot.name, "INFO", f"FORWARD_TEST funding {pos.symbol} #{pos.id}: {booked:+.4f} {spec.cash_currency}") + return total + + +def _forward_cross_breached(db, engine, bot, positions, exchange, timeframe, ccxt_symbol, current_high, current_low, candle_ts, close_cache): + """Cross-margin account check of a forward-test bot on this candle: + `forward_pool` free cash plus every open position's margin and PnL — + the current symbol at its adverse extreme, the others at their stored + close at or before the candle — against the sum of the maintenance + margins. Returns `(marks, cash)` when liquidated, else None.""" + if not positions: + return None + ccy = _pos_spec(positions[0]).cash_currency + cash = engine._forward_pool(db, bot, ccy) + equity = cash + maint = 0.0 + marks = {} + for p in positions: + side = p.side or "long" + spec = _pos_spec(p) + lev = max(float(p.leverage or 1), 1.0) + if normalize(p.symbol) == ccxt_symbol: + mark = current_high if side == "short" else current_low + else: + key = (p.exchange or exchange, p.symbol) + if key not in close_cache: + row = db.query(Candle.close).filter( + Candle.exchange == key[0], Candle.symbol == p.symbol, Candle.timeframe == timeframe, + Candle.timestamp <= candle_ts).order_by(Candle.timestamp.desc()).first() + close_cache[key] = float(row[0]) if row and row[0] is not None else None + mark = close_cache[key] if close_cache[key] is not None else (p.entry_price or 0.0) + marks[p.id] = mark + equity += spec.margin(p.amount or 0.0, p.entry_price or 0.0, lev) + spec.pnl_cash(side, p.amount or 0.0, p.entry_price or 0.0, mark) + maint += spec.notional_cash(p.amount or 0.0, mark) * tiers.mmr_for( + tiers.load(db, p.exchange or exchange, normalize(p.symbol)), tiers.tier_size(spec, p.amount or 0.0, p.entry_price or 0.0)) + if equity > maint: + return None + return marks, max(cash, 0.0) + + def close_all_open_positions(engine, bot, db, key_records): """Market-close every open non-backtest position for a bot before it is force-stopped. A stopped bot no longer evaluates SL/TP, so leaving live @@ -47,6 +241,21 @@ def close_all_open_positions(engine, bot, db, key_records): ccxt_inst.load_markets() except Exception as exc: logger.error("Could not build exchange client to close positions for '%s': %s", bot.name, exc, exc_info=True) + _ccxt_by_symbol = {} + + def _client_for(sym): + # Inverse contracts on a separate ccxt class (binancecoinm) need their own instance + if sym not in _ccxt_by_symbol: + inst = ccxt_inst + if api_key_record and engine._needs_own_instance(api_key_record, sym): + try: + inst = engine._get_ccxt_instance(api_key_record, sym) + inst.load_markets() + except Exception as exc: + logger.error("Could not build exchange client for %s: %s", sym, exc, exc_info=True) + inst = None + _ccxt_by_symbol[sym] = inst + return _ccxt_by_symbol[sym] now_ts = _naive_utc(datetime.now(timezone.utc)) timeframe = bot.settings.get("timeframe") @@ -61,75 +270,120 @@ def close_all_open_positions(engine, bot, db, key_records): actual_price = close_price actual_fee = 0.0 order_id = f"local_{int(time.time())}_{uuid.uuid4().hex[:8]}" - ccxt_symbol = pos.symbol.replace('-', '/').upper() - + ccxt_symbol = normalize(pos.symbol) + spec = _pos_spec(pos) + _close_cols = {"market_type": "swap", "reduce_only": 1} if is_derivative(ccxt_symbol) else {} + _close_cols["fee_currency"] = spec.cash_currency + # A short is flattened with a reduce-only buy; PnL flips sign + pos_side = pos.side or "long" + _pos_open_side, _pos_close_side = pnl.open_order_side(pos_side), pnl.close_order_side(pos_side) + + if pos.mode == "forward_test": + # Backtest exit economics: slippage against the fill, fee on the proceeds + _, exit_fee_pct, _, exit_slip = engine._sim_frictions(bot.settings, pos_side) + actual_price = close_price * (1 + exit_slip) if pos_side == "short" else close_price * (1 - exit_slip) + actual_fee = spec.fee_cash(close_qty, actual_price, exit_fee_pct) if pos.mode in ("paper", "live"): + ccxt_inst = _client_for(ccxt_symbol) if ccxt_inst is None: logger.error("Cannot close %s position on %s for '%s': no exchange client. Position left open.", pos.mode, pos.symbol, bot.name) blb.push(bot.name, "ERROR", f"Could not close {pos.mode} position on {pos.symbol}: exchange unavailable — close it manually!") continue try: - sell_qty = float(ccxt_inst.amount_to_precision(ccxt_symbol, close_qty)) + sell_qty = _precise_amount(ccxt_inst, ccxt_symbol, close_qty) if sell_qty <= 0: continue - ex_order = ccxt_inst.create_market_sell_order(ccxt_symbol, sell_qty) + ex_order = _place_market_order(ccxt_inst, api_key_record, ccxt_symbol, _pos_close_side, sell_qty, reduce_only=True, leverage=pos.leverage) ex_order = engine._reconcile_order(ccxt_inst, ex_order, ccxt_symbol) - filled_qty = float(ex_order.get("filled") or 0) + filled_qty = _filled_base(ccxt_inst, ccxt_symbol, ex_order) if filled_qty <= 0 and ex_order.get("status") != "closed": - db.add(Order(position_id=pos.id, exchange=pos.exchange, bot_name=bot.name, mode=pos.mode, symbol=pos.symbol, side="sell", order_type="market", price=close_price, amount=sell_qty, timestamp=now_ts, exchange_order_id=ex_order.get("id"), status="canceled")) + db.add(Order(position_id=pos.id, exchange=pos.exchange, bot_name=bot.name, mode=pos.mode, symbol=pos.symbol, side=_pos_close_side, order_type="market", price=close_price, amount=sell_qty, timestamp=now_ts, exchange_order_id=ex_order.get("id"), status="canceled", **_close_cols)) db.commit() blb.push(bot.name, "ERROR", f"Forced close on {pos.symbol} did not fill; position left open — close it manually!") continue close_qty = filled_qty if filled_qty > 0 else sell_qty actual_price = ex_order.get("average") or ex_order.get("price") or close_price order_id = ex_order.get("id") or order_id - actual_fee = engine._fee_in_quote(ex_order.get("fee"), ccxt_symbol, actual_price) + actual_fee = engine._fee_in_quote(ex_order.get("fee"), ccxt_symbol, actual_price, ccxt_inst) + _close_cols.update(_fee_cols(ex_order.get("fee"), spec.cash_currency)) except Exception as exc: logger.error("Forced close failed for '%s' on %s: %s", bot.name, pos.symbol, exc, exc_info=True) blb.push(bot.name, "ERROR", f"Forced close failed on {pos.symbol}: {exc} — close it manually!") continue - realized_pnl = (actual_price - pos.entry_price) * close_qty - actual_fee + # Same booking as the tick's exit: proportional entry fee + exit fee, + # profit_pct on the capital the leg had locked (margin + entry fee) + original_amount = _original_amount(pos, _pos_open_side) + entry_fee_portion = _entry_fee_total(pos, _pos_open_side) * (close_qty / original_amount) if original_amount > 0 else 0.0 + realized_pnl = pnl.price_pnl(pos_side, pos.entry_price, actual_price, close_qty, spec=spec) - entry_fee_portion - actual_fee pos.profit_abs = (pos.profit_abs or 0.0) + realized_pnl - if pos.entry_price and pos.amount: - pos.profit_pct = (pos.profit_pct or 0.0) + ((actual_price - pos.entry_price) / pos.entry_price) * 100 * (close_qty / pos.amount) + if pos.entry_price and original_amount > 0: + locked = pnl.locked_capital(pos.entry_price, close_qty, pos.leverage or 1, spec=spec) + entry_fee_portion + pos.profit_pct = (pos.profit_pct or 0.0) + ((realized_pnl / locked) * 100 if locked > 0 else 0.0) * (close_qty / original_amount) + engine._update_drawdown(bot.name, risk.mode_group_for(pos.mode), realized_pnl) - db.add(Order(position_id=pos.id, exchange=pos.exchange, bot_name=bot.name, mode=pos.mode, symbol=pos.symbol, side="sell", order_type="market", price=actual_price, amount=close_qty, timestamp=now_ts, exchange_order_id=order_id, status="filled", fee=actual_fee)) + db.add(Order(position_id=pos.id, exchange=pos.exchange, bot_name=bot.name, mode=pos.mode, symbol=pos.symbol, side=_pos_close_side, order_type="market", price=actual_price, amount=close_qty, timestamp=now_ts, exchange_order_id=order_id, status="filled", fee=actual_fee, **_close_cols)) - if close_qty >= pos.amount - 0.00001: + if close_qty >= pos.amount - pnl.close_epsilon(original_amount): pos.status = "closed" pos.closed_at = now_ts with engine._position_states_lock: engine.position_states.pop(pos.id, None) else: pos.amount -= close_qty - blb.push(bot.name, "WARN", f"Partial forced close on {pos.symbol}: {close_qty} sold, {pos.amount} still open — close it manually!") + blb.push(bot.name, "WARN", f"Partial forced close on {pos.symbol}: {close_qty} {'covered' if pos_side == 'short' else 'sold'}, {pos.amount} still open — close it manually!") - if pos.mode in ("paper", "live"): - db.commit() + # Committed before the console line (the log buffer writes on its own connection) + db.commit() - logger.info("Forced close [%s] %s: %s @ %s (PnL %+.2f)", pos.mode, pos.symbol, close_qty, actual_price, realized_pnl) - blb.push(bot.name, "INFO", f"Forced close [{pos.mode}] {pos.symbol}: {close_qty} @ {actual_price} (PnL {realized_pnl:+.2f})") + logger.info("Forced close [%s] %s: %s @ %s (PnL %+.2f %s)", pos.mode, pos.symbol, close_qty, actual_price, realized_pnl, spec.cash_currency) + blb.push(bot.name, "INFO", f"Forced close [{pos.mode}] {pos.symbol}: {close_qty} @ {actual_price} (PnL {realized_pnl:+.2f} {spec.cash_currency})") def maybe_open_position(engine, db, bot, exchange, symbol, mode, api_key_record, get_ccxt, is_buy, entries_blocked, bot_positions, open_count, max_pos, can_buy_cooldown, - current_price, latest_time): + current_price, latest_time, side="long"): """Entry leg of one live tick. Returns the freshly opened Position, or None when no entry was made (no signal, blocked, skipped, or failed). Kept separate from the exit loop on purpose: bailing out of the entry - must never skip the SL/TP evaluation of the positions already open.""" - ccxt_symbol = symbol.replace('-', '/').upper() + must never skip the SL/TP evaluation of the positions already open. + `side="short"` opens a short (derivatives only): the same sizing and + safety path with a non-reduce-only SELL, slippage against the seller and + the `short` leg of trade_settings; `is_buy` is then the short signal.""" + ccxt_symbol = normalize(symbol) + # Derivatives: leverage from the bot (the linked key fixes the market + # type); every spot bot goes through the unchanged branches below + _market_type = market_type_for(bot.settings, api_key_record) + _derivative = is_derivative(ccxt_symbol) + _leverage = leverage_for(bot.settings, _market_type) if _derivative else 1.0 + # Contract economics (spot / linear / inverse) and the cash currency + # every money figure below is in + spec = _spec(get_ccxt() if (api_key_record and mode in ("paper", "live")) else None, exchange, ccxt_symbol) + _quote = spec.cash_currency + _open_cols = {"market_type": _market_type} if _derivative else {} + _open_cols["fee_currency"] = _quote + _short = side == "short" + _label = "SHORT" if _short else "BUY" + _open_side = pnl.open_order_side(side) + _entry_cfg = pnl.entry_cfg(bot.settings.get("trade_settings", {}), side) + if _short and not _derivative: + # The validator rejects short nodes on spot; never sell what isn't held + if is_buy: + blb.push(bot.name, "ERROR", f"SHORT signal on {symbol} ignored — shorts need a perpetual (swap) market") + return None if is_buy and entries_blocked: - blb.push(bot.name, "INFO", f"BUY signal on {symbol} skipped — entries blocked by max drawdown") + blb.push(bot.name, "INFO", f"{_label} signal on {symbol} skipped — entries blocked by max drawdown") elif is_buy and open_count >= max_pos: # Pyramiding cap, same rule as the backtest (per_pair: this symbol, # global: the whole portfolio) - blb.push(bot.name, "INFO", f"BUY signal on {symbol} skipped — max_positions ({max_pos}) reached") + blb.push(bot.name, "INFO", f"{_label} signal on {symbol} skipped — max_positions ({max_pos}) reached") elif is_buy and can_buy_cooldown: - trade_amount = engine._calculate_trade_amount(current_price, bot.settings) + if _derivative: + trade_amount = engine._calculate_trade_amount(current_price, bot.settings, leverage=_leverage, side=side, spec=spec) + else: + trade_amount = engine._calculate_trade_amount(current_price, bot.settings, spec=spec) if trade_amount is None: - logger.warning("Skipping buy for %s: invalid trade amount", symbol) + logger.warning("Skipping %s for %s: invalid trade amount", _label.lower(), symbol) else: try: actual_price = current_price @@ -141,22 +395,26 @@ def maybe_open_position(engine, db, bot, exchange, symbol, mode, api_key_record, # pool, pay entry slippage and fee. A forward test that # trades frictionless on the original capital would # confirm a strategy the backtest never ran. - _quote = ccxt_symbol.split('/')[-1] pool = engine._forward_pool(db, bot, _quote) if pool <= 0: - blb.push(bot.name, "WARN", f"BUY signal on {symbol} skipped — forward-test pool depleted ({pool:,.2f} {_quote})") + blb.push(bot.name, "WARN", f"{_label} signal on {symbol} skipped — forward-test pool depleted ({pool:,.2f} {_quote})") return None - entry_fee_pct, _, entry_slip, _ = engine._sim_frictions(bot.settings) - trade_amount = engine._calculate_trade_amount(current_price, bot.settings, current_equity=pool) + entry_fee_pct, _, entry_slip, _ = engine._sim_frictions(bot.settings, side) + _lev = _leverage if _derivative else 1 + # Margin (notional / leverage) + fee on the notional must fit the pool + trade_amount = engine._calculate_trade_amount(current_price, bot.settings, current_equity=pool, leverage=_lev, side=side, spec=spec) if trade_amount is None: return None - actual_price = current_price * (1 + entry_slip) - if bot.settings.get("trade_settings", {}).get("entry", {}).get("amount_type", "percentage") != "fixed": - trade_amount = min(trade_amount, pool / (actual_price * (1 + entry_fee_pct))) - if trade_amount <= 0 or actual_price * trade_amount * (1 + entry_fee_pct) > pool + 1e-9: - blb.push(bot.name, "WARN", f"BUY signal on {symbol} skipped — entry does not fit the forward-test pool ({pool:,.2f} {_quote})") + # Slippage works against the taker: a short fills below the close + actual_price = current_price * (1 - entry_slip) if _short else current_price * (1 + entry_slip) + if _entry_cfg.get("amount_type", "percentage") != "fixed": + trade_amount = min(trade_amount, spec.max_affordable_qty(pool, actual_price, _lev, entry_fee_pct)) + if trade_amount <= 0 or spec.locked_capital(trade_amount, actual_price, _lev, entry_fee_pct) > pool + 1e-9: + blb.push(bot.name, "WARN", f"{_label} signal on {symbol} skipped — entry does not fit the forward-test pool ({pool:,.2f} {_quote})") return None - buy_fee = actual_price * trade_amount * entry_fee_pct + # Same safety cap as live and the backtest (quote notional) + trade_amount = cap_by_max_order_value(trade_amount, current_price, bot.settings, spec) + buy_fee = spec.fee_cash(trade_amount, actual_price, entry_fee_pct) elif mode in ["paper", "live"] and api_key_record: ccxt_inst = get_ccxt() @@ -164,17 +422,20 @@ def maybe_open_position(engine, db, bot, exchange, symbol, mode, api_key_record, # BUY for a candle that already produced one existing_buy = db.query(Order.id).filter( Order.bot_name == bot.name, Order.symbol == symbol, - Order.mode == mode, Order.side == "buy", + Order.mode == mode, Order.side == _open_side, Order.timestamp == latest_time, + # Closes are reduce-only: a cover on the same candle is a + # buy, a long's exit a sell — neither is an entry + func.coalesce(Order.reduce_only, 0) == 0, ).first() if existing_buy: - logger.info("%s BUY for %s @ %s already recorded, skipping duplicate entry", mode.upper(), symbol, latest_time) + logger.info("%s %s for %s @ %s already recorded, skipping duplicate entry", mode.upper(), _label, symbol, latest_time) return None # Demo accounts list fewer pairs than the public market # the backtest ran on — an order there can only fail if ccxt_inst.markets and ccxt_symbol not in ccxt_inst.markets: - logger.warning("%s BUY skipped: %s is not listed on %s for key '%s'", mode.upper(), ccxt_symbol, api_key_record.exchange, api_key_record.name) - blb.push(bot.name, "WARN", f"BUY signal on {symbol} skipped — not listed on {api_key_record.exchange.upper()}{' demo' if api_key_record.is_sandbox else ''} for key '{api_key_record.name}'") + logger.warning("%s %s skipped: %s is not listed on %s for key '%s'", mode.upper(), _label, ccxt_symbol, api_key_record.exchange, api_key_record.name) + blb.push(bot.name, "WARN", f"{_label} signal on {symbol} skipped — not listed on {api_key_record.exchange.upper()}{' demo' if api_key_record.is_sandbox else ''} for key '{api_key_record.name}'") return None # Size trades from the wallet: this bot's share @@ -188,73 +449,88 @@ def maybe_open_position(engine, db, bot, exchange, symbol, mode, api_key_record, blb.push(bot.name, "WARN", "Skipping entry: could not verify exchange balance") return None sizing_capital = _num(bot.settings.get("backtest_capital"), 1000) - logger.info("Bot '%s': sandbox balance unavailable, sizing paper entry from backtest capital $%.2f", bot.name, sizing_capital) + logger.info("Bot '%s': sandbox balance unavailable, sizing paper entry from backtest capital %.2f %s", bot.name, sizing_capital, _quote) else: - _quote = ccxt_symbol.split('/')[-1] pool, wallet_total, bot_total = engine._live_allocation(db, bot, _quote, free_balance) if pool <= 0: - blb.push(bot.name, "WARN", f"BUY signal on {symbol} skipped — allocation fully deployed ({bot_total:,.0f} {_quote} = {_num(bot.settings.get('live_allocation_pct'), 100):.0f}% of wallet {wallet_total:,.0f})") + blb.push(bot.name, "WARN", f"{_label} signal on {symbol} skipped — allocation fully deployed ({bot_total:,.0f} {_quote} = {_num(bot.settings.get('live_allocation_pct'), 100):.0f}% of wallet {wallet_total:,.0f})") return None sizing_capital = min(free_balance, pool) - logger.info("Bot '%s': sizing %s entry from $%.2f (free=$%.2f, pool remaining=$%.2f of allocation $%.2f, wallet=$%.2f)", - bot.name, mode, sizing_capital, free_balance, pool, bot_total, wallet_total) - trade_amount = engine._calculate_trade_amount(current_price, bot.settings, current_equity=sizing_capital) + logger.info("Bot '%s': sizing %s entry from %.2f %s (free=%.2f, pool remaining=%.2f of allocation %.2f, wallet=%.2f)", + bot.name, mode, sizing_capital, _quote, free_balance, pool, bot_total, wallet_total) + if _derivative: + # sizing_capital is margin; the order is leverage x bigger + trade_amount = engine._calculate_trade_amount(current_price, bot.settings, current_equity=sizing_capital, leverage=_leverage, side=side, spec=spec) + else: + trade_amount = engine._calculate_trade_amount(current_price, bot.settings, current_equity=sizing_capital, spec=spec) if trade_amount is None: - logger.warning("Skipping buy for %s: no capital available to size trade", symbol) + logger.warning("Skipping %s for %s: no capital available to size trade", _label.lower(), symbol) return None - trade_amount = float(ccxt_inst.amount_to_precision(ccxt_symbol, trade_amount)) + if _entry_cfg.get("amount_type", "percentage") != "fixed": + # Percentage sizing must leave room for the taker fee + # (same clamp as the backtest/forward pool): 100% of + # the free balance is otherwise an InsufficientFunds + _fee_pct = engine._sim_frictions(bot.settings, side)[0] + trade_amount = min(trade_amount, spec.max_affordable_qty(sizing_capital, current_price, _leverage if _derivative else 1, _fee_pct)) + trade_amount = _precise_amount(ccxt_inst, ccxt_symbol, trade_amount) if trade_amount <= 0: logger.warning("Trade amount rounded to zero for %s after precision, skipping", ccxt_symbol) return None min_violation = engine._below_market_minimum(ccxt_inst, ccxt_symbol, trade_amount, current_price) if min_violation: - logger.warning("%s BUY skipped for %s: %s", mode.upper(), symbol, min_violation) + logger.warning("%s %s skipped for %s: %s", mode.upper(), _label, symbol, min_violation) blb.push(bot.name, "WARN", f"{min_violation} — increase trade size") return None # Safety cap: clamp the order to max_order_value instead # of dropping it — a silently skipped entry is a # strategy that never trades, and the user can't see why - max_order_usd = _num(bot.settings.get("max_order_value"), 0) - if max_order_usd > 0: - order_value_usd = trade_amount * current_price - if order_value_usd > max_order_usd: - capped = float(ccxt_inst.amount_to_precision(ccxt_symbol, max_order_usd / current_price)) - logger.warning("SAFETY: BUY order $%.2f exceeds max_order_value $%.2f for %s — capped to %s", order_value_usd, max_order_usd, symbol, capped) - blb.push(bot.name, "WARN", f"BUY on {symbol} capped by max_order_value: {order_value_usd:,.0f} → {max_order_usd:,.0f} {ccxt_symbol.split('/')[-1]} (raise max_order_value or lower the entry amount to match the backtest)") + # `max_order_value` is a quote-currency notional (USD on BTC/USD:BTC, EUR on BTC/EUR) + max_order_quote = _num(bot.settings.get("max_order_value"), 0) + if max_order_quote > 0: + order_value_quote = spec.notional_quote(trade_amount, current_price) + if order_value_quote > max_order_quote: + capped = _precise_amount(ccxt_inst, ccxt_symbol, spec.qty_for_quote_notional(max_order_quote, current_price)) + logger.warning("SAFETY: %s order %.2f %s exceeds max_order_value %.2f %s for %s — capped to %s", _label, order_value_quote, spec.quote, max_order_quote, spec.quote, symbol, capped) + blb.push(bot.name, "WARN", f"{_label} on {symbol} capped by max_order_value: {order_value_quote:,.0f} → {max_order_quote:,.0f} {spec.quote} (raise max_order_value or lower the entry amount to match the backtest)") trade_amount = capped min_violation = engine._below_market_minimum(ccxt_inst, ccxt_symbol, trade_amount, current_price) if trade_amount <= 0 or min_violation: blb.push(bot.name, "WARN", f"Capped order on {symbol} is below the exchange minimum ({min_violation or 'zero amount'}) — entry skipped") return None - okx_order = ccxt_inst.create_market_buy_order(ccxt_symbol, trade_amount) - logger.info("%s BUY response: id=%s status=%s filled=%s avg=%s fee=%s", - mode.upper(), okx_order.get("id"), okx_order.get("status"), + if _derivative: + # Leverage/margin mode must be confirmed on the exchange + # before the first contract is bought (raises on failure) + engine._ensure_leverage(ccxt_inst, api_key_record, ccxt_symbol, _leverage, bot.settings.get("margin_mode"), bot.name) + okx_order = _place_market_order(ccxt_inst, api_key_record, ccxt_symbol, _open_side, trade_amount, reduce_only=False, leverage=_leverage) + logger.info("%s %s response: id=%s status=%s filled=%s avg=%s fee=%s", + mode.upper(), _label, okx_order.get("id"), okx_order.get("status"), okx_order.get("filled"), okx_order.get("average"), okx_order.get("fee")) okx_order = engine._reconcile_order(ccxt_inst, okx_order, ccxt_symbol) - filled_qty = float(okx_order.get("filled") or 0) + filled_qty = _filled_base(ccxt_inst, ccxt_symbol, okx_order) if filled_qty <= 0 and okx_order.get("status") != "closed": if engine._cancel_unfilled_order(ccxt_inst, okx_order.get("id"), ccxt_symbol): - logger.warning("%s BUY unfilled (status=%s), canceled on exchange.", mode.upper(), okx_order.get("status")) - db.add(Order(exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side="buy", order_type="market", price=current_price, amount=trade_amount, timestamp=latest_time, exchange_order_id=okx_order.get("id"), status="canceled")) + logger.warning("%s %s unfilled (status=%s), canceled on exchange.", mode.upper(), _label, okx_order.get("status")) + db.add(Order(exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side=_open_side, order_type="market", price=current_price, amount=trade_amount, timestamp=latest_time, exchange_order_id=okx_order.get("id"), status="canceled", **_open_cols)) db.commit() return None # Cancel did not go through: the order may have filled # after the last poll. Book the position conservatively # at the requested amount so it stays tracked. - logger.error("%s BUY state unknown for %s (id=%s) — booking position at requested amount, verify on the exchange", mode.upper(), symbol, okx_order.get("id")) - blb.push(bot.name, "ERROR", f"BUY order state unknown on {symbol} — position booked at requested amount/last price, verify manually on the exchange") + logger.error("%s %s state unknown for %s (id=%s) — booking position at requested amount, verify on the exchange", mode.upper(), _label, symbol, okx_order.get("id")) + blb.push(bot.name, "ERROR", f"{_label} order state unknown on {symbol} — position booked at requested amount/last price, verify manually on the exchange") # Book the position for what actually filled, even # when the exchange still reports the order as open if filled_qty > 0: trade_amount = filled_qty actual_price = okx_order.get("average") or okx_order.get("price") or current_price order_id = okx_order.get("id") - buy_fee = engine._fee_in_quote(okx_order.get("fee"), ccxt_symbol, actual_price) + buy_fee = engine._fee_in_quote(okx_order.get("fee"), ccxt_symbol, actual_price, ccxt_inst) + _open_cols.update(_fee_cols(okx_order.get("fee"), _quote)) # A fee charged in base currency comes out of the bought # amount itself; only the net amount is actually held fee_info = okx_order.get("fee") or {} - base_ccy = ccxt_symbol.split('/')[0] - if fee_info.get("currency") and str(fee_info["currency"]).upper() == base_ccy.upper(): + base_ccy = base_of(ccxt_symbol) + if not _derivative and fee_info.get("currency") and str(fee_info["currency"]).upper() == base_ccy.upper(): try: base_fee_cost = float(fee_info.get("cost") or 0) except (TypeError, ValueError): @@ -263,35 +539,45 @@ def maybe_open_position(engine, db, bot, exchange, symbol, mode, api_key_record, net_amount = float(ccxt_inst.amount_to_precision(ccxt_symbol, max(trade_amount - base_fee_cost, 0))) if net_amount > 0: trade_amount = net_amount - engine._balance_cache.pop((api_key_record.name, ccxt_symbol.split('/')[-1]), None) + engine._balance_cache.pop((api_key_record.name, _quote), None) # Position created after successful exchange order - open_position = Position(exchange=exchange, bot_name=bot.name, symbol=symbol, mode=mode, status="open", side="long", entry_price=actual_price, amount=trade_amount) + open_position = Position(exchange=exchange, bot_name=bot.name, symbol=symbol, mode=mode, status="open", side=side, entry_price=actual_price, amount=trade_amount, + cash_currency=_quote, contract_kind=spec.kind, contract_size=spec.contract_size) + if _derivative: + open_position.market_type = _market_type + open_position.leverage = _leverage + # Funding is charged from the candle that opened the + # position (created_at is wall-clock, the candle may be older) + open_position.funding_until = latest_time + if mode in ["paper", "live"]: + open_position.contracts = spec.to_contracts(trade_amount) db.add(open_position) db.flush() - db.add(Order(position_id=open_position.id, exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side="buy", order_type="market", price=actual_price, amount=trade_amount, timestamp=latest_time, exchange_order_id=order_id, status="filled", fee=buy_fee)) - if mode in ["paper", "live"]: - # A real exchange fill must be persisted immediately — - # a later rollback may not erase the record of it - db.commit() - logger.info("%s BUY Filled @ %s", mode.upper(), actual_price) - blb.push(bot.name, "INFO", f"{mode.upper()} BUY {symbol} @ {actual_price}") + db.add(Order(position_id=open_position.id, exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side=_open_side, order_type="market", price=actual_price, amount=trade_amount, timestamp=latest_time, exchange_order_id=order_id, status="filled", fee=buy_fee, **_open_cols)) + # A real exchange fill must be persisted immediately — a later + # rollback may not erase the record of it. Forward fills commit + # here too: the console line below writes on its own connection + # and would otherwise wait on this session's rows + db.commit() + logger.info("%s %s Filled @ %s", mode.upper(), _label, actual_price) + blb.push(bot.name, "INFO", f"{mode.upper()} {_label} {symbol} @ {actual_price}") return open_position except ccxt.InsufficientFunds as e: - logger.warning("%s BUY rejected (insufficient funds): %s", mode.upper(), e) - blb.push(bot.name, "WARN", f"{mode.upper()} BUY rejected: insufficient funds") - db.add(Order(exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side="buy", order_type="market", price=current_price, amount=trade_amount, timestamp=latest_time, status="rejected")) + logger.warning("%s %s rejected (insufficient funds): %s", mode.upper(), _label, e) + blb.push(bot.name, "WARN", f"{mode.upper()} {_label} rejected: insufficient funds") + db.add(Order(exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side=_open_side, order_type="market", price=current_price, amount=trade_amount, timestamp=latest_time, status="rejected", **_open_cols)) if mode in ["paper", "live"]: db.commit() except Exception as e: # The order never reached the exchange (or the exchange # refused it): "rejected", not "canceled" — a cancel # implies an order that existed - logger.error("%s BUY failed for %s: %s: %s", mode.upper(), symbol, type(e).__name__, e, exc_info=True) - blb.push(bot.name, "ERROR", f"{mode.upper()} BUY {symbol} rejected — {type(e).__name__}: {str(e)[:200]}") - db.add(Order(exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side="buy", order_type="market", price=current_price, amount=trade_amount, timestamp=latest_time, status="rejected")) + logger.error("%s %s failed for %s: %s: %s", mode.upper(), _label, symbol, type(e).__name__, e, exc_info=True) + blb.push(bot.name, "ERROR", f"{mode.upper()} {_label} {symbol} rejected — {type(e).__name__}: {str(e)[:200]}") + db.add(Order(exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side=_open_side, order_type="market", price=current_price, amount=trade_amount, timestamp=latest_time, status="rejected", **_open_cols)) if mode in ["paper", "live"]: db.commit() return None @@ -331,7 +617,7 @@ def process_tick(engine, exchange: str, symbol: str, timeframe: str, candle_ts=N if not matching_bots: return - max_lookback = max([int(b.settings.get("backtest_lookback", 150)) for b in matching_bots], default=150) + max_lookback = max([_int(b.settings.get("backtest_lookback"), 150) for b in matching_bots], default=150) query = db.query(Candle.id, Candle.timestamp, Candle.open, Candle.high, Candle.low, Candle.close, Candle.volume).filter( Candle.exchange == exchange, Candle.symbol == symbol, Candle.timeframe == timeframe @@ -369,17 +655,26 @@ def process_tick(engine, exchange: str, symbol: str, timeframe: str, candle_ts=N ).all() if matching_bot_names else [] _positions_by_bot_mode = defaultdict(list) + _funding_cache = {} # (exchange, symbol) → stored funding events, one query per tick for p in _all_open_positions: _positions_by_bot_mode[(p.bot_name, p.mode)].append(p) - # Pre-load cooldown buy counts for all bots in one query - tf_seconds = 60 - if timeframe.endswith('m'): tf_seconds = int(timeframe[:-1]) * 60 - elif timeframe.endswith('h'): tf_seconds = int(timeframe[:-1]) * 3600 - elif timeframe.endswith('d'): tf_seconds = int(timeframe[:-1]) * 86400 + # Pre-load cooldown buy counts for all bots in one query. The window + # is counted in candles like the backtest (`index - idx < + # cooldown_candles`): it starts at the N-th most recent stored candle + # at or before the one being processed, not at wall-clock now minus + # N x timeframe — a backlog replay or a gap would otherwise measure + # a different window than the simulation. + tf_seconds = _tf_seconds(timeframe) + _latest_ts = _naive_utc(df['timestamp'].iloc[-1]) + + def _cooldown_window_start(n_candles): + if n_candles <= len(df): + return _naive_utc(df['timestamp'].iloc[len(df) - n_candles]) + return _latest_ts - timedelta(seconds=(n_candles - 1) * tf_seconds) _cooldown_counts = {} - cooldown_bots = [b for b in matching_bots if int(b.settings.get("cooldown_trades", 0)) > 0 and int(b.settings.get("cooldown_candles", 0)) > 0] + cooldown_bots = [b for b in matching_bots if _int(b.settings.get("cooldown_trades"), 0) > 0 and _int(b.settings.get("cooldown_candles"), 0) > 0] if cooldown_bots: # Only executed buys in the bot's own mode count toward cooldown — # backtest history must not block live entries @@ -391,26 +686,24 @@ def process_tick(engine, exchange: str, symbol: str, timeframe: str, candle_ts=N if kr: m = "paper" if kr.is_sandbox else "live" _bot_modes[b.name] = m - now_utc = datetime.now(timezone.utc) _bot_windows = { - b.name: now_utc - timedelta(seconds=int(b.settings.get("cooldown_candles", 0)) * tf_seconds) + b.name: _cooldown_window_start(_int(b.settings.get("cooldown_candles"), 0)) for b in cooldown_bots } - max_cooldown_candles = max(int(b.settings.get("cooldown_candles", 0)) for b in cooldown_bots) - min_threshold = _naive_utc(now_utc - timedelta(seconds=max_cooldown_candles * tf_seconds)) + min_threshold = min(_bot_windows.values()) + # Opening orders: buys, plus the non-reduce-only sells that open a short _recent_buys = db.query(Order.bot_name, Order.mode, Order.timestamp).filter( Order.bot_name.in_([b.name for b in cooldown_bots]), Order.symbol == symbol, - Order.side == "buy", + or_(Order.side == "buy", and_(Order.side == "sell", Order.market_type == "swap", Order.reduce_only == 0)), Order.status == "filled", - Order.timestamp > min_threshold + Order.timestamp >= min_threshold, + Order.timestamp <= _latest_ts, ).all() for _bn, _om, _ots in _recent_buys: if _om != _bot_modes.get(_bn): continue - if _ots.tzinfo is None: - _ots = _ots.replace(tzinfo=timezone.utc) - if _ots > _bot_windows[_bn]: + if _naive_utc(_ots) >= _bot_windows[_bn]: _cooldown_counts[_bn] = _cooldown_counts.get(_bn, 0) + 1 # Collect all signal inserts for a single batch commit @@ -427,335 +720,440 @@ def process_tick(engine, exchange: str, symbol: str, timeframe: str, candle_ts=N # bot run in parallel worker threads, and the entry gate, # allocation pool and drawdown state are per bot with engine._bot_locks[bot.name]: - # The batch snapshot above was taken before the lock: a - # tick for another symbol of this bot may have opened or - # closed a position meanwhile, so refresh this bot's slice - for _k in [k for k in _positions_by_bot_mode if k[0] == bot.name]: - del _positions_by_bot_mode[_k] - for _p in db.query(Position).options(selectinload(Position.orders)).filter( - Position.bot_name == bot.name, Position.status == "open").all(): - _positions_by_bot_mode[(bot.name, _p.mode)].append(_p) - # Risk guards on the realized live equity curve: - # - max_capital_loss: loss of principal → close all + stop (any action) - # - max_drawdown + close_all (default): close all + stop - # - max_drawdown + block_entries: pause entries, keep exits, - # resume below half the limit (hysteresis) - max_drawdown_pct = _num(bot.settings.get("max_drawdown"), 0) - max_capital_loss_pct = _num(bot.settings.get("max_capital_loss"), 0) - dd_action = bot.settings.get("drawdown_action", "close_all") - if max_drawdown_pct > 0 or max_capital_loss_pct > 0: - live_capital = _num(bot.settings.get("live_starting_capital"), 0) or _num(bot.settings.get("backtest_capital"), 1000) - _reset_raw = bot.settings.get("drawdown_peak_reset_at") - try: - _peak_reset_at = _naive_utc(datetime.fromisoformat(_reset_raw)) if _reset_raw else None - except (ValueError, TypeError): - _peak_reset_at = None - dd_state = engine._get_drawdown(bot.name, db, mode_group="live", starting_capital=live_capital, peak_reset_at=_peak_reset_at) - _open_real = [p for k, v in _positions_by_bot_mode.items() if k[0] == bot.name and k[1] != "backtest" for p in v] - # Mark-to-market like the backtest: open losses count - # before they are realized, so a stop fires on the - # same curve the backtest limit was tested on - _unrealized = engine._unrealized_pnl(db, _open_real, exchange, timeframe, _last_close_cache) - dd_now, loss_now = engine._dd_now(dd_state, _unrealized) - - stop_reason = None - _open_any = len(_open_real) - if max_capital_loss_pct > 0 and loss_now >= max_capital_loss_pct: - if dd_action == "block_entries" and _open_any > 0: - # Wind down: no new entries, exits keep running, - # the bot stops on the tick it turns flat. Loss of - # principal never recovers without trades, so - # unlike the drawdown block there is no resume. - if bot.name not in engine._entries_blocked: - engine._entries_blocked.add(bot.name) - logger.warning("Bot '%s' capital loss %.2f%% > %.2f%% — winding down (entries blocked, stops when flat)", bot.name, loss_now, max_capital_loss_pct) - blb.push(bot.name, "WARN", f"Capital loss {loss_now:.1f}% > {max_capital_loss_pct:.0f}% — winding down: entries blocked, {_open_any} open position(s) keep their exits, bot stops when flat") + # One failing bot must not kill the tick for the others: + # their exits still have to be evaluated on this candle + try: + # The batch snapshot above was taken before the lock: a + # tick for another symbol of this bot may have opened or + # closed a position meanwhile, so refresh this bot's slice + for _k in [k for k in _positions_by_bot_mode if k[0] == bot.name]: + del _positions_by_bot_mode[_k] + for _p in db.query(Position).options(selectinload(Position.orders)).filter( + Position.bot_name == bot.name, Position.status == "open").all(): + _positions_by_bot_mode[(bot.name, _p.mode)].append(_p) + is_api_exec = bot.settings.get("api_execution", False) + has_key = bool(bot.settings.get("api_key_name")) + mode = "forward_test" + api_key_record = None + if is_api_exec and has_key: + api_key_record = key_records.get(bot.settings.get("api_key_name")) + if api_key_record: + mode = "paper" if api_key_record.is_sandbox else "live" else: - blb.push(bot.name, "WARN", f"Capital loss {loss_now:.1f}% > {max_capital_loss_pct:.0f}% — bot stopped") - stop_reason = f"Capital loss {loss_now:.1f}% hit the {max_capital_loss_pct:.0f}% limit — " + ("bot flat, stopped" if _open_any == 0 else "positions closed") - elif max_drawdown_pct > 0 and dd_action != "block_entries" and dd_state["max_dd"] >= max_drawdown_pct: - blb.push(bot.name, "WARN", f"Max drawdown hit ({dd_state['max_dd']:.2f}% >= {max_drawdown_pct:.2f}%), auto-stopping") - stop_reason = f"Live drawdown {dd_state['max_dd']:.1f}% hit the {max_drawdown_pct:.0f}% limit — positions closed" - - if stop_reason: - logger.warning("Bot '%s': %s", bot.name, stop_reason) - blb.push(bot.name, "WARN", "Closing all open positions before stopping — a stopped bot no longer manages SL/TP") - close_all_open_positions(engine, bot, db, key_records) - engine._engine_stop(bot, db, stop_reason) - engine._drawdown_cache.pop((bot.name, "live"), None) - engine._drawdown_cache.pop((bot.name, "backtest"), None) - engine._entries_blocked.discard(bot.name) - db.commit() - continue - - _capital_wind_down = max_capital_loss_pct > 0 and loss_now >= max_capital_loss_pct - if max_drawdown_pct > 0 and dd_action == "block_entries" and not _capital_wind_down: - # Block on breach; release when drawdown recovers below - # half the limit, or once the bot has been flat for the - # cooldown — realized equity cannot recover without - # trades, so the peak is reset (persisted in settings so - # a restart doesn't re-block) and a new campaign starts. - # max_capital_loss stays the absolute stop across campaigns. - _cooldown_days = _num(bot.settings.get("drawdown_cooldown_days"), 7) - if bot.name not in engine._entries_blocked and dd_now >= max_drawdown_pct: - engine._entries_blocked.add(bot.name) - logger.warning("Bot '%s' drawdown %.2f%% > %.2f%% — new entries blocked", bot.name, dd_now, max_drawdown_pct) - blb.push(bot.name, "WARN", f"Max drawdown {dd_now:.1f}% > {max_drawdown_pct:.0f}% — new entries blocked (open positions keep their exits; resumes below {max_drawdown_pct * 0.5:.1f}% or after {_cooldown_days:.0f}d flat)") - elif bot.name in engine._entries_blocked: - if dd_now < max_drawdown_pct * 0.5: - engine._entries_blocked.discard(bot.name) - blb.push(bot.name, "INFO", f"Drawdown recovered to {dd_now:.1f}% — new entries allowed again") - elif _open_any == 0: - _last_close = db.query(func.max(Position.closed_at)).filter( - Position.bot_name == bot.name, Position.status == "closed", - Position.mode.in_(["forward_test", "paper", "live"])).scalar() - _now_ts = _naive_utc(datetime.now(timezone.utc)) - _flat_secs = (_now_ts - _last_close).total_seconds() if _last_close is not None else float("inf") - if _flat_secs >= _cooldown_days * 86400: + logger.warning("Bot '%s': api_key_name='%s' not found. Running as forward_test.", bot.name, bot.settings.get("api_key_name")) + blb.push(bot.name, "WARN", f"API key '{bot.settings.get('api_key_name')}' not found, running as forward_test") + + # Drawdown state is per mode group: a forward test never + # shares a curve with real money (risk.MODES_BY_GROUP) + _dd_group = risk.mode_group_for(mode) + + # Risk guards on the realized live equity curve: + # - max_capital_loss: loss of principal → close all + stop (any action) + # - max_drawdown + close_all (default): close all + stop + # - max_drawdown + block_entries: pause entries, keep exits, + # resume below half the limit (hysteresis) + max_drawdown_pct = _num(bot.settings.get("max_drawdown"), 0) + max_capital_loss_pct = _num(bot.settings.get("max_capital_loss"), 0) + dd_action = bot.settings.get("drawdown_action", "close_all") + if max_drawdown_pct > 0 or max_capital_loss_pct > 0: + live_capital = _num(bot.settings.get("live_starting_capital"), 0) or _num(bot.settings.get("backtest_capital"), 1000) + _reset_raw = bot.settings.get("drawdown_peak_reset_at") + try: + _peak_reset_at = _naive_utc(datetime.fromisoformat(_reset_raw)) if _reset_raw else None + except (ValueError, TypeError): + _peak_reset_at = None + dd_state = engine._get_drawdown(bot.name, db, mode_group=_dd_group, starting_capital=live_capital, peak_reset_at=_peak_reset_at) + _open_real = [p for k, v in _positions_by_bot_mode.items() if k[0] == bot.name and k[1] in risk.MODES_BY_GROUP[_dd_group] for p in v] + # Mark-to-market like the backtest: open losses count + # before they are realized, so a stop fires on the + # same curve the backtest limit was tested on. Marked + # at the candle being processed, not the newest row, + # so a backlog replay measures the curve at its own time + _unrealized = engine._unrealized_pnl(db, _open_real, exchange, timeframe, _last_close_cache, candle_ts=_latest_ts) + dd_now, loss_now = engine._dd_now(dd_state, _unrealized) + + stop_reason = None + _open_any = len(_open_real) + if max_capital_loss_pct > 0 and loss_now >= max_capital_loss_pct: + if dd_action == "block_entries" and _open_any > 0: + # Wind down: no new entries, exits keep running, + # the bot stops on the tick it turns flat. Loss of + # principal never recovers without trades, so + # unlike the drawdown block there is no resume. + if bot.name not in engine._entries_blocked: + engine._entries_blocked.add(bot.name) + logger.warning("Bot '%s' capital loss %.2f%% > %.2f%% — winding down (entries blocked, stops when flat)", bot.name, loss_now, max_capital_loss_pct) + blb.push(bot.name, "WARN", f"Capital loss {loss_now:.1f}% > {max_capital_loss_pct:.0f}% — winding down: entries blocked, {_open_any} open position(s) keep their exits, bot stops when flat") + else: + blb.push(bot.name, "WARN", f"Capital loss {loss_now:.1f}% > {max_capital_loss_pct:.0f}% — bot stopped") + stop_reason = f"Capital loss {loss_now:.1f}% hit the {max_capital_loss_pct:.0f}% limit — " + ("bot flat, stopped" if _open_any == 0 else "positions closed") + elif max_drawdown_pct > 0 and dd_action != "block_entries" and dd_state["max_dd"] >= max_drawdown_pct: + blb.push(bot.name, "WARN", f"Max drawdown hit ({dd_state['max_dd']:.2f}% >= {max_drawdown_pct:.2f}%), auto-stopping") + stop_reason = f"Live drawdown {dd_state['max_dd']:.1f}% hit the {max_drawdown_pct:.0f}% limit — positions closed" + + if stop_reason: + logger.warning("Bot '%s': %s", bot.name, stop_reason) + blb.push(bot.name, "WARN", "Closing all open positions before stopping — a stopped bot no longer manages SL/TP") + close_all_open_positions(engine, bot, db, key_records) + engine._engine_stop(bot, db, stop_reason) + engine._drawdown_cache.pop((bot.name, _dd_group), None) + engine._drawdown_cache.pop((bot.name, "backtest"), None) + engine._entries_blocked.discard(bot.name) + db.commit() + continue + + _capital_wind_down = max_capital_loss_pct > 0 and loss_now >= max_capital_loss_pct + if max_drawdown_pct > 0 and dd_action == "block_entries" and not _capital_wind_down: + # Block on breach; release when drawdown recovers below + # half the limit, or once the bot has been flat for the + # cooldown — realized equity cannot recover without + # trades, so the peak is reset (persisted in settings so + # a restart doesn't re-block) and a new campaign starts. + # max_capital_loss stays the absolute stop across campaigns. + _cooldown_days = _num(bot.settings.get("drawdown_cooldown_days"), 7) + if bot.name not in engine._entries_blocked and dd_now >= max_drawdown_pct: + engine._entries_blocked.add(bot.name) + logger.warning("Bot '%s' drawdown %.2f%% > %.2f%% — new entries blocked", bot.name, dd_now, max_drawdown_pct) + blb.push(bot.name, "WARN", f"Max drawdown {dd_now:.1f}% > {max_drawdown_pct:.0f}% — new entries blocked (open positions keep their exits; resumes below {max_drawdown_pct * 0.5:.1f}% or after {_cooldown_days:.0f}d flat)") + elif bot.name in engine._entries_blocked: + if dd_now < max_drawdown_pct * 0.5: engine._entries_blocked.discard(bot.name) - bot.settings = {**bot.settings, "drawdown_peak_reset_at": _now_ts.isoformat()} - engine._drawdown_cache.pop((bot.name, "live"), None) - db.commit() - blb.push(bot.name, "INFO", f"Flat for {_cooldown_days:.0f} days at {dd_now:.1f}% drawdown — peak reset, new entries allowed again (capital-loss guard remains)") - entries_blocked = bot.name in engine._entries_blocked - - # Reuse indicator computation across bots with identical indicator configs - fp = _indicator_fingerprint(bot.settings) - if fp not in indicator_cache: - eval_tmp = NodeEvaluator(bot.settings) - eval_tmp.df = df.copy() - eval_tmp._calculate_indicators() - indicator_cache[fp] = eval_tmp.df - - evaluator = NodeEvaluator(bot.settings) - evaluator.df = indicator_cache[fp] - - latest_index = len(evaluator.df) - 1 - latest_row = evaluator.df.iloc[latest_index] - latest_time = _naive_utc(latest_row['timestamp']) - - current_price = float(latest_row['close']) - current_open = float(latest_row['open']) - current_high = float(latest_row['high']) - current_low = float(latest_row['low']) - - current_atr = float(evaluator.df['atr'].iloc[latest_index]) if 'atr' in evaluator.df.columns and not pd.isna(evaluator.df['atr'].iloc[latest_index]) else 0.0 - - entry_series = evaluator.resolve_node(bot.settings.get("entry_node")) if bot.settings.get("entry_node") else pd.Series(False, index=evaluator.df.index) - exit_series = evaluator.resolve_node(bot.settings.get("exit_node")) if bot.settings.get("exit_node") else pd.Series(False, index=evaluator.df.index) - - is_buy = bool(entry_series.iloc[-1]) - is_sell = bool(exit_series.iloc[-1]) - - tick_action = "BUY signal" if is_buy else ("SELL signal" if is_sell else "no signal") - blb.push(bot.name, "INFO", f"Tick {symbol} {timeframe} | close {current_price} | {tick_action}") - try: - _tf_s = _tf_seconds(timeframe) - _next = datetime.fromtimestamp(((int(time.time()) // _tf_s) + 1) * _tf_s, tz=timezone.utc) - _prev_rt = engine.get_runtime(bot.name) or {} - engine.set_runtime(bot.name, "live", f"Last tick {symbol} @ {current_price:g} — {tick_action}", - mode=_prev_rt.get("mode"), next_close=_next.isoformat(), last_tick_at=datetime.now(timezone.utc).isoformat()) - except Exception as e: - logger.debug("runtime status update skipped for %s: %s", bot.name, e) - - is_api_exec = bot.settings.get("api_execution", False) - has_key = bool(bot.settings.get("api_key_name")) - mode = "forward_test" - api_key_record = None - - if is_api_exec and has_key: - api_key_record = key_records.get(bot.settings.get("api_key_name")) - if api_key_record: - mode = "paper" if api_key_record.is_sandbox else "live" + blb.push(bot.name, "INFO", f"Drawdown recovered to {dd_now:.1f}% — new entries allowed again") + elif _open_any == 0: + _last_close = db.query(func.max(Position.closed_at)).filter( + Position.bot_name == bot.name, Position.status == "closed", + Position.mode.in_(risk.MODES_BY_GROUP[_dd_group])).scalar() + _now_ts = _naive_utc(datetime.now(timezone.utc)) + _flat_secs = (_now_ts - _last_close).total_seconds() if _last_close is not None else float("inf") + if _flat_secs >= _cooldown_days * 86400: + engine._entries_blocked.discard(bot.name) + bot.settings = {**bot.settings, "drawdown_peak_reset_at": _now_ts.isoformat()} + engine._drawdown_cache.pop((bot.name, _dd_group), None) + db.commit() + blb.push(bot.name, "INFO", f"Flat for {_cooldown_days:.0f} days at {dd_now:.1f}% drawdown — peak reset, new entries allowed again (capital-loss guard remains)") + entries_blocked = bot.name in engine._entries_blocked + + # Reuse indicator computation across bots with identical indicator configs + fp = _indicator_fingerprint(bot.settings) + if fp not in indicator_cache: + eval_tmp = NodeEvaluator(bot.settings) + eval_tmp.df = df.copy() + eval_tmp._calculate_indicators() + indicator_cache[fp] = eval_tmp.df + + evaluator = NodeEvaluator(bot.settings) + evaluator.df = indicator_cache[fp] + + latest_index = len(evaluator.df) - 1 + latest_row = evaluator.df.iloc[latest_index] + latest_time = _naive_utc(latest_row['timestamp']) + + current_price = float(latest_row['close']) + current_open = float(latest_row['open']) + current_high = float(latest_row['high']) + current_low = float(latest_row['low']) + + current_atr = float(evaluator.df['atr'].iloc[latest_index]) if 'atr' in evaluator.df.columns and not pd.isna(evaluator.df['atr'].iloc[latest_index]) else 0.0 + + entry_series = evaluator.resolve_node(bot.settings.get("entry_node")) if bot.settings.get("entry_node") else pd.Series(False, index=evaluator.df.index) + exit_series = evaluator.resolve_node(bot.settings.get("exit_node")) if bot.settings.get("exit_node") else pd.Series(False, index=evaluator.df.index) + + is_buy = bool(entry_series.iloc[-1]) + is_sell = bool(exit_series.iloc[-1]) + # Shorts (phase 3): only strategies with the nodes on a perpetual market + _short_node, _cover_node = bot.settings.get("short_node"), bot.settings.get("cover_node") + _shorts_enabled = bool(_short_node) and is_derivative(normalize(symbol)) + is_short = bool(evaluator.resolve_node(_short_node).iloc[-1]) if _shorts_enabled else False + is_cover = bool(evaluator.resolve_node(_cover_node).iloc[-1]) if _shorts_enabled and _cover_node else False + + tick_action = "BUY signal" if is_buy else ("SHORT signal" if is_short else ("SELL signal" if is_sell else ("COVER signal" if is_cover else "no signal"))) + blb.push(bot.name, "INFO", f"Tick {symbol} {timeframe} | close {current_price} | {tick_action}") + try: + _tf_s = _tf_seconds(timeframe) + _next = datetime.fromtimestamp(((int(time.time()) // _tf_s) + 1) * _tf_s, tz=timezone.utc) + _prev_rt = engine.get_runtime(bot.name) or {} + engine.set_runtime(bot.name, "live", f"Last tick {symbol} @ {current_price:g} — {tick_action}", + mode=_prev_rt.get("mode"), next_close=_next.isoformat(), last_tick_at=datetime.now(timezone.utc).isoformat()) + except Exception as e: + logger.debug("runtime status update skipped for %s: %s", bot.name, e) + + # Cache exchange instance per bot cycle to avoid repeated connections + _cached_ccxt = None + def get_ccxt(): + nonlocal _cached_ccxt + if _cached_ccxt is None and api_key_record: + _cached_ccxt = engine._ccxt_for(api_key_record, symbol) + _cached_ccxt.load_markets() + return _cached_ccxt + + max_pos = _int(bot.settings.get("max_positions"), 1) + scope = bot.settings.get("max_positions_scope", "per_pair") + + # Use pre-loaded positions instead of per-bot DB query + bot_positions = [p for p in _positions_by_bot_mode.get((bot.name, mode), []) if p.symbol == symbol] + if scope == "per_pair": + open_count = len(bot_positions) else: - logger.warning("Bot '%s': api_key_name='%s' not found. Running as forward_test.", bot.name, bot.settings.get("api_key_name")) - blb.push(bot.name, "WARN", f"API key '{bot.settings.get('api_key_name')}' not found, running as forward_test") - - # Cache exchange instance per bot cycle to avoid repeated connections - _cached_ccxt = None - def get_ccxt(): - nonlocal _cached_ccxt - if _cached_ccxt is None and api_key_record: - _cached_ccxt = engine._get_ccxt_instance(api_key_record) - _cached_ccxt.load_markets() - return _cached_ccxt - - max_pos = _int(bot.settings.get("max_positions"), 1) - scope = bot.settings.get("max_positions_scope", "per_pair") - - # Use pre-loaded positions instead of per-bot DB query - bot_positions = [p for p in _positions_by_bot_mode.get((bot.name, mode), []) if p.symbol == symbol] - if scope == "per_pair": - open_count = len(bot_positions) - else: - # Global scope: all symbols, but only the bot's own mode — a - # position left open by the backtest must not take a live slot - open_count = len(_positions_by_bot_mode.get((bot.name, mode), [])) - - ccxt_symbol = symbol.replace('-', '/').upper() - just_opened_ids = set() - - # Cooldown check using pre-loaded counts - cooldown_trades = _int(bot.settings.get("cooldown_trades"), 0) - cooldown_candles = _int(bot.settings.get("cooldown_candles"), 0) - - can_buy_cooldown = True - if cooldown_trades > 0 and cooldown_candles > 0: - recent_buys = _cooldown_counts.get(bot.name, 0) - if recent_buys >= cooldown_trades: - can_buy_cooldown = False - - opened = maybe_open_position(engine, - db, bot, exchange, symbol, mode, api_key_record, get_ccxt, - is_buy, entries_blocked, bot_positions, open_count, max_pos, can_buy_cooldown, - current_price, latest_time) - if opened is not None: - just_opened_ids.add(opened.id) - - # Use pre-loaded positions (already includes orders via selectinload) - active_positions = bot_positions - - for pos in active_positions: - if not bot.is_active: break - if pos.id in just_opened_ids: continue - - exit_events = engine._check_exits(pos, current_price, current_high, current_low, is_sell, bot.settings, current_atr, row_open=current_open) - - # Track original amount for weighted profit_pct - # Use the sum of all buy orders as the original position size - pos_original_amount = sum( - o.amount for o in (pos.orders or []) if o.side == "buy" and o.status == "filled" - ) or pos.amount - - for ev in exit_events: - if pos.amount <= 0: break - - if ev.get('close_amount_type') == 'fixed': - close_qty = min(ev['qty_pct'], pos.amount) + # Global scope: all symbols, but only the bot's own mode — a + # position left open by the backtest must not take a live slot + open_count = len(_positions_by_bot_mode.get((bot.name, mode), [])) + + ccxt_symbol = normalize(symbol) + _close_cols = {"market_type": "swap", "reduce_only": 1} if is_derivative(ccxt_symbol) else {} + just_opened_ids = set() + + # Cooldown check using pre-loaded counts + cooldown_trades = _int(bot.settings.get("cooldown_trades"), 0) + cooldown_candles = _int(bot.settings.get("cooldown_candles"), 0) + + can_buy_cooldown = True + if cooldown_trades > 0 and cooldown_candles > 0: + recent_buys = _cooldown_counts.get(bot.name, 0) + if recent_buys >= cooldown_trades: + can_buy_cooldown = False + + # Long and short never coexist on one pair: the conflicting + # signal is ignored (same rule as the backtest) + pair_side = (bot_positions[0].side or "long") if bot_positions else None + _try_buy = is_buy + if is_buy and pair_side == "short": + blb.push(bot.name, "INFO", f"BUY signal on {symbol} ignored while a short is open (long and short never coexist on a pair)") + _try_buy = False + opened = maybe_open_position(engine, + db, bot, exchange, symbol, mode, api_key_record, get_ccxt, + _try_buy, entries_blocked, bot_positions, open_count, max_pos, can_buy_cooldown, + current_price, latest_time) + if opened is None and is_short and not is_buy and _shorts_enabled: + if pair_side == "long": + blb.push(bot.name, "INFO", f"SHORT signal on {symbol} ignored while a long is open (long and short never coexist on a pair)") else: - # Percentage of the original size (parity with the backtest) - close_qty = pos_original_amount * (ev['qty_pct'] / 100) - close_qty = min(close_qty, pos.amount) - if close_qty <= 0: continue - - try: - actual_price = ev['price'] - order_id = f"local_{int(latest_time.timestamp())}_{uuid.uuid4().hex[:8]}" - - actual_fee = 0.0 - if mode == "forward_test": - # Backtest exit economics: slippage against - # the fill, fee on the proceeds - _, exit_fee_pct, _, exit_slip = engine._sim_frictions(bot.settings) - actual_price = ev['price'] * (1 - exit_slip) - actual_fee = actual_price * close_qty * exit_fee_pct - elif mode in ["paper", "live"] and api_key_record: - ccxt_inst = get_ccxt() - close_qty = float(ccxt_inst.amount_to_precision(ccxt_symbol, close_qty)) - if close_qty <= 0: - logger.warning("Sell amount rounded to zero for %s after precision, skipping", ccxt_symbol) - continue - min_violation = engine._below_market_minimum(ccxt_inst, ccxt_symbol, close_qty, ev['price']) - if min_violation: - if engine._below_market_minimum(ccxt_inst, ccxt_symbol, pos.amount, ev['price']): - # The whole remainder can never be sold on the - # exchange; close the position administratively - # instead of retrying a doomed sell forever - logger.warning("%s position remainder on %s unsellable (%s) — closing administratively", mode.upper(), symbol, min_violation) - blb.push(bot.name, "WARN", f"Position remainder on {symbol} below exchange minimum ({min_violation}) — closed administratively, dust remains on the exchange") - pos.status = "closed" - pos.closed_at = latest_time - engine._update_drawdown(bot.name, "live", pos.profit_abs) - with engine._position_states_lock: - engine.position_states.pop(pos.id, None) + opened = maybe_open_position(engine, + db, bot, exchange, symbol, mode, api_key_record, get_ccxt, + is_short, entries_blocked, bot_positions, open_count, max_pos, can_buy_cooldown, + current_price, latest_time, side="short") + if opened is not None: + just_opened_ids.add(opened.id) + + # Use pre-loaded positions (already includes orders via selectinload) + active_positions = bot_positions + + # Forward test on a perpetual: funding on the positions + # that were open before this candle, the margin mode + # decides between a level per position and one account + _fwd_swap = mode == "forward_test" and is_derivative(ccxt_symbol) + _fwd_cross = _fwd_swap and (bot.settings.get("margin_mode") or DEFAULT_MARGIN_MODE) == "cross" + if _fwd_swap: + _apply_forward_funding(engine, db, bot, [p for p in active_positions if p.id not in just_opened_ids], + exchange, ccxt_symbol, current_price, latest_time, _dd_group, _funding_cache) + _sym_tiers = tiers.load(db, exchange, ccxt_symbol) + + for pos in active_positions: + if not bot.is_active: break + if pos.id in just_opened_ids: continue + + # A short is closed by the cover signal with mirrored SL/TP, + # via reduce-only buys; its opening orders are sells + pos_side = pos.side or "long" + pos_short = pos_side == "short" + _pos_open_side, _pos_close_side = pnl.open_order_side(pos_side), pnl.close_order_side(pos_side) + _close_label = "COVER" if pos_short else "SELL" + pos_spec = _pos_spec(pos) + _pos_close_cols = {**_close_cols, "fee_currency": pos_spec.cash_currency} + exit_events = engine._check_exits(pos, current_price, current_high, current_low, is_cover if pos_short else is_sell, bot.settings, current_atr, row_open=current_open, side=pos_side) + # Forward test: the same price rule as the backtest. + # Exits that fill before the liquidation level are + # regular exits; whatever is still open afterwards + # (or would have filled beyond it) is liquidated. + _liq_price, _liq_hit = None, False + if _fwd_swap and not _fwd_cross: + _mmr = tiers.mmr_for(_sym_tiers, tiers.tier_size(pos_spec, pos.amount, pos.entry_price)) + _liq_price = pos_spec.liquidation_price(pos_side, pos.entry_price, pos.leverage or 1, mmr=_mmr) + _liq_hit = pnl.liquidated(pos_side, _liq_price, current_high, current_low) + if _liq_hit: + exit_events = [ev for ev in exit_events if not pnl.liquidated(pos_side, _liq_price, ev['price'], ev['price'])] + + # Track original amount for weighted profit_pct + # Use the sum of all opening orders as the original position size + pos_original_amount = _original_amount(pos, _pos_open_side) + + for ev in exit_events: + if pos.amount <= 0: break + + if ev.get('close_amount_type') == 'fixed': + close_qty = min(ev['qty_pct'], pos.amount) + else: + # Percentage of the original size (parity with the backtest) + close_qty = pos_original_amount * (ev['qty_pct'] / 100) + close_qty = min(close_qty, pos.amount) + if close_qty <= 0: continue + + try: + actual_price = ev['price'] + order_id = f"local_{int(latest_time.timestamp())}_{uuid.uuid4().hex[:8]}" + + actual_fee = 0.0 + if mode == "forward_test": + # Backtest exit economics: slippage against + # the fill, fee on the proceeds + _, exit_fee_pct, _, exit_slip = engine._sim_frictions(bot.settings, pos_side) + actual_price = ev['price'] * (1 + exit_slip) if pos_short else ev['price'] * (1 - exit_slip) + actual_fee = pos_spec.fee_cash(close_qty, actual_price, exit_fee_pct) + elif mode in ["paper", "live"] and api_key_record: + ccxt_inst = get_ccxt() + close_qty = _precise_amount(ccxt_inst, ccxt_symbol, close_qty) + if close_qty <= 0: + logger.warning("Sell amount rounded to zero for %s after precision, skipping", ccxt_symbol) + continue + min_violation = engine._below_market_minimum(ccxt_inst, ccxt_symbol, close_qty, ev['price']) + if min_violation: + if engine._below_market_minimum(ccxt_inst, ccxt_symbol, pos.amount, ev['price']): + # The whole remainder can never be sold on the + # exchange; close the position administratively + # instead of retrying a doomed sell forever + logger.warning("%s position remainder on %s unsellable (%s) — closing administratively", mode.upper(), symbol, min_violation) + blb.push(bot.name, "WARN", f"Position remainder on {symbol} below exchange minimum ({min_violation}) — closed administratively, dust remains on the exchange") + pos.status = "closed" + pos.closed_at = latest_time + engine._update_drawdown(bot.name, _dd_group, 0.0) # partial legs were booked as they closed + with engine._position_states_lock: + engine.position_states.pop(pos.id, None) + db.commit() + break + logger.warning("%s %s skipped for %s: %s", mode.upper(), _close_label, symbol, min_violation) + blb.push(bot.name, "WARN", f"{_close_label.capitalize()} on {symbol} skipped: {min_violation}") + continue + okx_order = _place_market_order(ccxt_inst, api_key_record, ccxt_symbol, _pos_close_side, close_qty, reduce_only=True, leverage=pos.leverage) + logger.info("%s %s response: id=%s status=%s filled=%s avg=%s fee=%s", + mode.upper(), _close_label, okx_order.get("id"), okx_order.get("status"), + okx_order.get("filled"), okx_order.get("average"), okx_order.get("fee")) + okx_order = engine._reconcile_order(ccxt_inst, okx_order, ccxt_symbol) + filled_qty = _filled_base(ccxt_inst, ccxt_symbol, okx_order) + if filled_qty <= 0 and okx_order.get("status") != "closed": + if engine._cancel_unfilled_order(ccxt_inst, okx_order.get("id"), ccxt_symbol): + logger.warning("%s %s unfilled (status=%s), canceled on exchange.", mode.upper(), _close_label, okx_order.get("status")) + db.add(Order(position_id=pos.id, exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side=_pos_close_side, order_type="market", price=ev['price'], amount=close_qty, timestamp=latest_time, exchange_order_id=okx_order.get("id"), status="canceled", **_close_cols)) + db.commit() + continue + # Cancel did not go through: the sell may still fill on + # the exchange. Keep the position amount untouched and + # stop the bot — a second sell here could double-sell. + logger.error("%s %s state unknown for %s (id=%s) — stopping bot '%s'", mode.upper(), _close_label, symbol, okx_order.get("id"), bot.name) + blb.push(bot.name, "ERROR", f"Order state unknown on {symbol} — verify manually on the exchange before restarting") + db.add(Order(position_id=pos.id, exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side=_pos_close_side, order_type="market", price=ev['price'], amount=close_qty, timestamp=latest_time, exchange_order_id=okx_order.get("id"), status="unknown", **_close_cols)) + engine._engine_stop(bot, db, f"{_close_label.capitalize()} order state unknown on {symbol} — verify on the exchange before restarting") db.commit() break - logger.warning("%s SELL skipped for %s: %s", mode.upper(), symbol, min_violation) - blb.push(bot.name, "WARN", f"Sell on {symbol} skipped: {min_violation}") - continue - okx_order = ccxt_inst.create_market_sell_order(ccxt_symbol, close_qty) - logger.info("%s SELL response: id=%s status=%s filled=%s avg=%s fee=%s", - mode.upper(), okx_order.get("id"), okx_order.get("status"), - okx_order.get("filled"), okx_order.get("average"), okx_order.get("fee")) - okx_order = engine._reconcile_order(ccxt_inst, okx_order, ccxt_symbol) - filled_qty = float(okx_order.get("filled") or 0) - if filled_qty <= 0 and okx_order.get("status") != "closed": - if engine._cancel_unfilled_order(ccxt_inst, okx_order.get("id"), ccxt_symbol): - logger.warning("%s SELL unfilled (status=%s), canceled on exchange.", mode.upper(), okx_order.get("status")) - db.add(Order(position_id=pos.id, exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side="sell", order_type="market", price=ev['price'], amount=close_qty, timestamp=latest_time, exchange_order_id=okx_order.get("id"), status="canceled")) - db.commit() - continue - # Cancel did not go through: the sell may still fill on - # the exchange. Keep the position amount untouched and - # stop the bot — a second sell here could double-sell. - logger.error("%s SELL state unknown for %s (id=%s) — stopping bot '%s'", mode.upper(), symbol, okx_order.get("id"), bot.name) - blb.push(bot.name, "ERROR", f"Order state unknown on {symbol} — verify manually on the exchange before restarting") - db.add(Order(position_id=pos.id, exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side="sell", order_type="market", price=ev['price'], amount=close_qty, timestamp=latest_time, exchange_order_id=okx_order.get("id"), status="unknown")) - engine._engine_stop(bot, db, f"Sell order state unknown on {symbol} — verify on the exchange before restarting") - db.commit() - break - # Book only what actually sold so a partial fill - # reduces the position pro rata instead of being - # retried for the full amount later - if filled_qty > 0: - close_qty = min(filled_qty, close_qty) - actual_price = okx_order.get("average") or okx_order.get("price") or ev['price'] - order_id = okx_order.get("id") - actual_fee = engine._fee_in_quote(okx_order.get("fee"), ccxt_symbol, actual_price) - engine._balance_cache.pop((api_key_record.name, ccxt_symbol.split('/')[-1]), None) - - db.add(Order(position_id=pos.id, exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side="sell", order_type="market", price=actual_price, amount=close_qty, timestamp=latest_time, exchange_order_id=order_id, status="filled", fee=actual_fee)) - - # Fee-adjusted P&L: subtract proportional entry fee + exit fee - total_buy_fees = sum((o.fee or 0.0) for o in (pos.orders or []) if o.side == "buy" and o.status == "filled") - entry_fee_portion = total_buy_fees * (close_qty / pos_original_amount) if pos_original_amount > 0 else 0.0 - realized_pnl = (actual_price - pos.entry_price) * close_qty - entry_fee_portion - actual_fee - pos.profit_abs = (pos.profit_abs or 0.0) + realized_pnl - - # Weighted profit_pct: fee-adjusted, based on portion of original position - if pos_original_amount > 0: - entry_cost_for_qty = pos.entry_price * close_qty + entry_fee_portion - portion_pct = (realized_pnl / entry_cost_for_qty) * 100 if entry_cost_for_qty > 0 else 0.0 - weight = close_qty / pos_original_amount - pos.profit_pct = (pos.profit_pct or 0.0) + (portion_pct * weight) - - with engine._position_states_lock: - if pos.id in engine.position_states: - engine.position_states[pos.id]['triggered_exits'].add(ev['id']) - pos.triggered_exits = list(engine.position_states[pos.id]['triggered_exits']) - - if close_qty >= pos.amount - 0.00001: - pos.status = "closed" - pos.closed_at = latest_time - engine._update_drawdown(bot.name, "live", pos.profit_abs) + # Book only what actually sold so a partial fill + # reduces the position pro rata instead of being + # retried for the full amount later + if filled_qty > 0: + close_qty = min(filled_qty, close_qty) + actual_price = okx_order.get("average") or okx_order.get("price") or ev['price'] + order_id = okx_order.get("id") + actual_fee = engine._fee_in_quote(okx_order.get("fee"), ccxt_symbol, actual_price, ccxt_inst) + _pos_close_cols.update(_fee_cols(okx_order.get("fee"), pos_spec.cash_currency)) + engine._balance_cache.pop((api_key_record.name, pos_spec.cash_currency), None) + + db.add(Order(position_id=pos.id, exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side=_pos_close_side, order_type="market", price=actual_price, amount=close_qty, timestamp=latest_time, exchange_order_id=order_id, status="filled", fee=actual_fee, **_pos_close_cols)) + + # Fee-adjusted P&L: subtract proportional entry fee + exit fee + total_buy_fees = _entry_fee_total(pos, _pos_open_side) + entry_fee_portion = total_buy_fees * (close_qty / pos_original_amount) if pos_original_amount > 0 else 0.0 + realized_pnl = pnl.price_pnl(pos_side, pos.entry_price, actual_price, close_qty, spec=pos_spec) - entry_fee_portion - actual_fee + pos.profit_abs = (pos.profit_abs or 0.0) + realized_pnl + + # Weighted profit_pct on the capital the leg had + # locked (margin + entry fee) — the backtest basis + if pos_original_amount > 0: + entry_cost_for_qty = pnl.locked_capital(pos.entry_price, close_qty, pos.leverage or 1, spec=pos_spec) + entry_fee_portion + portion_pct = (realized_pnl / entry_cost_for_qty) * 100 if entry_cost_for_qty > 0 else 0.0 + weight = close_qty / pos_original_amount + pos.profit_pct = (pos.profit_pct or 0.0) + (portion_pct * weight) + with engine._position_states_lock: - engine.position_states.pop(pos.id, None) - else: - pos.amount -= close_qty + if pos.id in engine.position_states: + engine.position_states[pos.id]['triggered_exits'].add(ev['id']) + pos.triggered_exits = list(engine.position_states[pos.id]['triggered_exits']) + + # Every closed leg moves the realized curve, partial or not + engine._update_drawdown(bot.name, _dd_group, realized_pnl) + if close_qty >= pos.amount - pnl.close_epsilon(pos_original_amount): + pos.status = "closed" + pos.closed_at = latest_time + with engine._position_states_lock: + engine.position_states.pop(pos.id, None) + else: + pos.amount -= close_qty - if mode in ["paper", "live"]: # A real exchange fill must be persisted immediately — - # a later rollback may not erase the record of it + # a later rollback may not erase the record of it; + # forward fills commit here too so the console line + # (own connection) never waits on this session db.commit() - logger.info("%s SELL (%s) Filled @ %s", mode.upper(), ev['reason'], actual_price) - blb.push(bot.name, "INFO", f"{mode.upper()} SELL [{ev['reason']}] {symbol} @ {actual_price}") - except Exception as e: - logger.error("%s SELL failed: %s", mode.upper(), e, exc_info=True) - blb.push(bot.name, "ERROR", f"{mode.upper()} SELL failed: {e}") - db.add(Order(position_id=pos.id, exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side="sell", order_type="market", price=current_price, amount=close_qty, timestamp=latest_time, status="rejected")) - if mode in ["paper", "live"]: - db.commit() - - standard_cols = ['id', 'timestamp', 'open', 'high', 'low', 'close', 'volume', 'atr'] - indicators = { col: float(latest_row[col]) for col in evaluator.df.columns if col not in standard_cols and not pd.isna(latest_row[col]) } - - if indicators: - action_str = "buy" if is_buy else ("sell" if is_sell else "neutral") - live_ts = latest_time - if hasattr(live_ts, 'to_pydatetime'): - live_ts = live_ts.to_pydatetime() - _pending_signals.append({"cid": int(latest_row['id']), "sym": symbol, "ts": str(live_ts), "bn": bot.name, "nm": "STRATEGY_TICK", "act": action_str, "ed": json.dumps(indicators)}) + logger.info("%s %s (%s) Filled @ %s", mode.upper(), _close_label, ev['reason'], actual_price) + blb.push(bot.name, "INFO", f"{mode.upper()} {_close_label} [{ev['reason']}] {symbol} @ {actual_price}") + except Exception as e: + logger.error("%s %s failed: %s", mode.upper(), _close_label, e, exc_info=True) + blb.push(bot.name, "ERROR", f"{mode.upper()} {_close_label} failed: {e}") + db.add(Order(position_id=pos.id, exchange=exchange, bot_name=bot.name, mode=mode, symbol=symbol, side=_pos_close_side, order_type="market", price=current_price, amount=close_qty, timestamp=latest_time, status="rejected", **_close_cols)) + if mode in ["paper", "live"]: + db.commit() + # A rejected reduce-only close on a perpetual + # usually means the exchange no longer holds + # the position (liquidated or closed by + # hand): ask, and stop retrying forever + if is_derivative(ccxt_symbol) and api_key_record and pos.status == "open": + try: + if broker.exchange_position_gone(get_ccxt(), ccxt_symbol, pos_side): + _book_liquidation(engine, db, bot, pos, current_price, latest_time, _dd_group, + "exchange reports no open position after a rejected close — booked as liquidation, check the exchange") + db.commit() + break + except Exception as probe_exc: + logger.warning("Could not verify %s position on the exchange after a rejected close: %s", symbol, probe_exc) + + if _liq_hit and pos.status == "open": + _book_liquidation(engine, db, bot, pos, _liq_price, latest_time, _dd_group, + f"candle range reached the liquidation level {_liq_price:g} at {pos.leverage or 1:g}x") + + # Cross margin: the whole account after this candle's + # exits (every symbol of the bot); a breach liquidates + # all open forward positions and the free cash is lost + # with them (same booking as the backtest) + if _fwd_cross: + _all_open = [p for p in _positions_by_bot_mode.get((bot.name, mode), []) if p.status == "open" and p.id not in just_opened_ids] + _breach = _forward_cross_breached(db, engine, bot, _all_open, exchange, timeframe, ccxt_symbol, + current_high, current_low, latest_time, _last_close_cache) + if _breach is not None: + _marks, _cash_lost = _breach + _total_margin = sum(_pos_spec(p).margin(p.amount or 0.0, p.entry_price or 0.0, max(float(p.leverage or 1), 1.0)) for p in _all_open) + for p in _all_open: + _m_i = _pos_spec(p).margin(p.amount or 0.0, p.entry_price or 0.0, max(float(p.leverage or 1), 1.0)) + _extra = _cash_lost * _m_i / _total_margin if _total_margin > 0 else 0.0 + _book_liquidation(engine, db, bot, p, _marks.get(p.id, p.entry_price), latest_time, _dd_group, + f"cross-margin account liquidation on {symbol} — equity fell to the maintenance margin, wallet lost", extra_loss=_extra) + blb.push(bot.name, "WARN", f"FORWARD_TEST cross-margin liquidation: {len(_all_open)} position(s) closed, {_cash_lost:,.2f} free cash lost") + + standard_cols = ['id', 'timestamp', 'open', 'high', 'low', 'close', 'volume', 'atr'] + indicators = { col: float(latest_row[col]) for col in evaluator.df.columns if col not in standard_cols and not pd.isna(latest_row[col]) } + + if indicators: + action_str = "buy" if is_buy else ("short" if is_short else ("sell" if is_sell else ("cover" if is_cover else "neutral"))) + live_ts = latest_time + if hasattr(live_ts, 'to_pydatetime'): + live_ts = live_ts.to_pydatetime() + _pending_signals.append({"cid": int(latest_row['id']), "sym": symbol, "ts": str(live_ts), "bn": bot.name, "nm": "STRATEGY_TICK", "act": action_str, "ed": json.dumps(indicators)}) + # Persist this bot's forward fills before the next bot + # runs: a failure there rolls back only its own work + db.commit() + except Exception as _bot_exc: + logger.error("Bot '%s' failed on %s %s @ %s: %s", bot.name, symbol, timeframe, candle_ts, _bot_exc, exc_info=True) + blb.push(bot.name, "ERROR", f"Tick {symbol} {timeframe} failed: {_bot_exc}") + try: + db.rollback() + except Exception: + pass # ── Single batch commit for all signals and position/order changes ── if _pending_signals: diff --git a/backend/engine/pnl.py b/backend/engine/pnl.py new file mode 100644 index 0000000..2254124 --- /dev/null +++ b/backend/engine/pnl.py @@ -0,0 +1,103 @@ +"""Side-aware trade economics shared by the backtest, the live tick, the +drawdown guard and the trades router. A long is `+1`, a short `-1`; every +formula for the long side reduces to the pre-shorts spot code so long-only +bots stay byte-identical.""" + +from backend.engine import contracts as _contracts + +# Flat maintenance margin ratio — the fallback when no exchange tier is +# stored for the pair (`engine/tiers`); the validator's leverage warning +# quotes this estimate. Lives in `contracts`; re-exported for the callers/ +# tests that read it here. +MAINTENANCE_MARGIN = _contracts.MAINTENANCE_MARGIN + +SIDES = ("long", "short") + + +def direction(side): + """+1 for a long (or unknown/legacy `None`), -1 for a short.""" + return -1 if side == "short" else 1 + + +def is_short(side): + return side == "short" + + +def price_pnl(side, entry_price, exit_price, amount, spec=None): + """Gross PnL in the cash currency before fees: what the position gained + between `entry_price` and `exit_price` on `amount` units. Linear/spot + when no `spec` is given; a `ContractSpec` routes inverse contracts + through their own formula.""" + if spec is not None: + return spec.pnl_cash(side, amount, entry_price, exit_price) + return direction(side) * (exit_price - entry_price) * amount + + +def liquidation_price(side, entry_price, leverage, spec=None): + """Price where the margin is exhausted up to the maintenance margin: + `entry*(1-(1-MMR)/lev)` for a long, `entry*(1+(1-MMR)/lev)` for a short. + A 1x long can lose at most its margin (None: no liquidation level); a 1x + short is liquidated once the price roughly doubles (`entry*(1+(1-MMR))`). + With a `spec` the contract kind decides (inverse has its own curve).""" + if spec is not None: + return spec.liquidation_price(side, entry_price, leverage) + if not entry_price: + return None + lev = max(float(leverage or 1), 1.0) + if lev <= 1 and not is_short(side): + return None + move = (1 - MAINTENANCE_MARGIN) / lev + return entry_price * (1 + move) if is_short(side) else entry_price * (1 - move) + + +def locked_capital(entry_price, amount, leverage=1, fee=0.0, spec=None): + """Capital taken out of the pool to open `amount` units at `entry_price`: + margin (`notional/leverage`) plus the entry fee on the full notional. + Reduces to `entry*qty*(1+fee)` on spot (leverage 1). This is the single + basis of `profit_pct` in the backtest, the live/forward tick and the + manual close. With a `spec` the contract kind decides the notional.""" + if spec is not None and spec.is_inverse: + return spec.locked_capital(amount, entry_price, leverage, fee) + notional = float(entry_price or 0.0) * float(amount or 0.0) + lev = max(float(leverage or 1), 1.0) + return notional / lev + notional * float(fee or 0.0) + + +def close_epsilon(original_amount): + """Tolerance below which a remaining amount counts as fully closed: + relative to the original size so dust on low-priced coins (SHIB-sized + quantities) still closes, with an absolute floor for tiny positions.""" + return max(1e-9, float(original_amount or 0.0) * 1e-6) + + +def liquidated(side, liq_price, row_high, row_low): + """Whether one candle's range touched the liquidation level.""" + if liq_price is None: + return False + return row_high >= liq_price if is_short(side) else row_low <= liq_price + + +def entry_cfg(trade_settings, side): + """Opening-leg config for `side`: `trade_settings.short` (falling back to + `entry`) for a short, `trade_settings.entry` for a long.""" + if is_short(side): + return trade_settings.get("short") or trade_settings.get("entry", {}) + return trade_settings.get("entry", {}) + + +def exit_cfg(trade_settings, side): + """Closing-leg config for `side`: `trade_settings.cover` (falling back to + `exit`) for a short, `trade_settings.exit` for a long.""" + if is_short(side): + return trade_settings.get("cover") or trade_settings.get("exit", {}) + return trade_settings.get("exit", {}) + + +def open_order_side(side): + """Exchange order side that opens a position of `side`.""" + return "sell" if is_short(side) else "buy" + + +def close_order_side(side): + """Exchange order side that closes a position of `side`.""" + return "buy" if is_short(side) else "sell" diff --git a/backend/engine/risk.py b/backend/engine/risk.py index 97c5612..a23b26b 100644 --- a/backend/engine/risk.py +++ b/backend/engine/risk.py @@ -3,10 +3,28 @@ from closed positions; `unrealized_pnl` marks the open positions to the latest stored close so the gates measure the same curve as the backtest.""" import threading +from sqlalchemy import func +from backend.engine import pnl +from backend.engine.sizing import position_spec from backend.models.candles import Candle from backend.models.positions import Position +MODES_BY_GROUP = { + "backtest": ("backtest",), + "forward": ("forward_test",), + "live": ("paper", "live"), +} + + +def mode_group_for(mode): + """Drawdown group a position/bot mode belongs to.""" + for group, modes in MODES_BY_GROUP.items(): + if mode in modes: + return group + return "live" + + class DrawdownTracker: def __init__(self): self.cache = {} # (bot_name, mode_group) -> {starting_capital, peak_equity, running_pnl, max_dd} @@ -16,7 +34,9 @@ def __init__(self): def get(self, bot_name, db, mode_group="live", starting_capital=1000.0, peak_reset_at=None): """Return cached drawdown state, lazy-initializing from DB on first access. - mode_group: "backtest" for backtest-only, "live" for forward_test/paper/live. + mode_group: "backtest" for backtest-only, "forward" for forward_test, + "live" for paper/live (see `mode_group_for`) — a simulated forward + loss never counts against the real-money curve. starting_capital: wallet size used as equity base for percentage calculation. peak_reset_at: naive-UTC datetime; closes before it only move the equity base (block_entries cooldown started a new drawdown campaign there).""" @@ -27,11 +47,14 @@ def get(self, bot_name, db, mode_group="live", starting_capital=1000.0, peak_res query = db.query(Position.profit_abs, Position.closed_at).filter( Position.bot_name == bot_name, Position.status == "closed" ) - if mode_group == "backtest": - query = query.filter(Position.mode == "backtest") - else: - query = query.filter(Position.mode.in_(["forward_test", "paper", "live"])) + query = query.filter(Position.mode.in_(MODES_BY_GROUP.get(mode_group, MODES_BY_GROUP["live"]))) closed = query.order_by(Position.closed_at).all() + # Partial exits book their PnL on the still-open position as they + # fill (and `update` the state per leg), so the realized curve + # includes them before the last leg closes + open_realized = db.query(func.coalesce(func.sum(Position.profit_abs), 0.0)).filter( + Position.bot_name == bot_name, Position.status == "open", + Position.mode.in_(MODES_BY_GROUP.get(mode_group, MODES_BY_GROUP["live"]))).scalar() or 0.0 running = 0.0 peak_equity = starting_capital dd = 0.0 @@ -44,12 +67,15 @@ def get(self, bot_name, db, mode_group="live", starting_capital=1000.0, peak_res peak_equity = max(peak_equity, equity) if peak_equity > 0: dd = max(dd, ((peak_equity - equity) / peak_equity) * 100) + running += float(open_realized) + peak_equity = max(peak_equity, starting_capital + running) with self.lock: self.cache.setdefault(cache_key, {"starting_capital": starting_capital, "peak_equity": peak_equity, "running_pnl": running, "max_dd": dd}) return self.cache[cache_key] def update(self, bot_name, mode_group, profit_abs): - """Incrementally update drawdown cache when a position closes.""" + """Incrementally update the drawdown state with the realized PnL of + one closed leg (partial or final).""" cache_key = (bot_name, mode_group) with self.lock: s = self.cache.get(cache_key) @@ -80,19 +106,25 @@ def now(self, state, unrealized=0.0): return max(dd, 0.0), max(loss, 0.0) -def unrealized_pnl(db, positions, exchange, timeframe, close_cache): - """Open PnL of `positions` at each symbol's latest stored close (one - Candle query per symbol per tick, memoized in `close_cache`).""" +def unrealized_pnl(db, positions, exchange, timeframe, close_cache, candle_ts=None): + """Open PnL of `positions` at each symbol's stored close at or before + `candle_ts` (the latest close when None) — one Candle query per symbol + per tick, memoized in `close_cache`. Windowing at `candle_ts` keeps a + backlog replay from marking old positions to today's price.""" total = 0.0 for p in positions: key = (p.exchange or exchange, p.symbol) if key not in close_cache: - row = db.query(Candle.close).filter( + q = db.query(Candle.close).filter( Candle.exchange == key[0], Candle.symbol == p.symbol, Candle.timeframe == timeframe - ).order_by(Candle.timestamp.desc()).first() + ) + if candle_ts is not None: + q = q.filter(Candle.timestamp <= candle_ts) + row = q.order_by(Candle.timestamp.desc()).first() close_cache[key] = float(row[0]) if row and row[0] is not None else None last = close_cache[key] if last is None: continue - total += (last - (p.entry_price or 0.0)) * (p.amount or 0.0) + spec = position_spec(p.symbol, p.contract_kind, p.contract_size, p.exchange or exchange) + total += pnl.price_pnl(p.side, p.entry_price or 0.0, last, p.amount or 0.0, spec=spec) return total diff --git a/backend/engine/settings_validator.py b/backend/engine/settings_validator.py index 892131b..ff479c9 100644 --- a/backend/engine/settings_validator.py +++ b/backend/engine/settings_validator.py @@ -2,11 +2,17 @@ import re from datetime import datetime -from backend.core.exchange_registry import get_exchange_timeframes +from backend.core.exchange_registry import exchange_spec, get_exchange_timeframes, market_caps, supported_kinds from backend.engine.indicator_registry import get_spec +from backend.engine.symbols import DEFAULT_MARGIN_MODE, DEFAULT_MARKET_TYPE, MARGIN_MODES, MARKET_TYPES, base_of, cash_currency, is_derivative, settle_of +from backend.engine.pnl import MAINTENANCE_MARGIN logger = logging.getLogger("apexalgo.settings_validator") -SYMBOL_PATTERN = re.compile(r'^[A-Z0-9]+/[A-Z0-9]+$') +# BASE/QUOTE for spot, BASE/QUOTE:SETTLE for perpetual swaps +SYMBOL_PATTERN = re.compile(r'^[A-Z0-9]+/[A-Z0-9]+(:[A-Z0-9]+)?$') +# Above this the liquidation sits close enough to the entry that a normal +# stop loss barely gets a chance — worth a warning, not a refusal +LEVERAGE_WARN_ABOVE = 3 MAX_PRICE_OFFSET = 500 VALID_EXIT_TYPES = {'percentage', 'trailing', 'atr', 'fixed'} VALID_AMOUNT_TYPES = {'percentage', 'fixed'} @@ -23,7 +29,7 @@ LOOKBACK_MARGIN = 50 -def validate_bot_settings(settings: dict, exchange_id: str | None = None) -> dict: +def validate_bot_settings(settings: dict, exchange_id: str | None = None, key_market_type: str | None = None) -> dict: """Validate bot settings and return errors and warnings. Parameters @@ -33,6 +39,9 @@ def validate_bot_settings(settings: dict, exchange_id: str | None = None) -> dic exchange_id : str, optional Resolved exchange ID for timeframe validation. Falls back to settings['data_exchange'] or 'okx' if not provided. + key_market_type : str, optional + Market type of the linked API key; a key is bound to one market, + so a bot on another market type is refused. Returns dict with 'errors' (list of blocking issues) and 'warnings' (list of non-blocking issues). @@ -40,32 +49,94 @@ def validate_bot_settings(settings: dict, exchange_id: str | None = None) -> dic errors = [] warnings = [] nodes = settings.get("nodes", {}) + eid = exchange_id or settings.get("data_exchange", "okx") + + # Market type (phase 2): spot or perpetual swap; absent = spot for every + # pre-existing bot. Symbol form, leverage and margin mode depend on it. + market_type = str(settings.get("market_type") or DEFAULT_MARKET_TYPE).strip().lower() + if market_type not in MARKET_TYPES: + errors.append(f"Invalid market_type '{settings.get('market_type')}'. Use one of: {', '.join(MARKET_TYPES)}.") + market_type = DEFAULT_MARKET_TYPE + elif "market_type" in settings: + settings["market_type"] = market_type + if key_market_type and str(key_market_type).lower() != market_type: + errors.append(f"API key '{settings.get('api_key_name')}' is a {key_market_type} key but the bot trades {market_type}. " + f"Link a {market_type} key or change the bot's market type.") + if exchange_spec(eid) is not None and market_caps(eid, market_type) is None: + errors.append(f"{exchange_spec(eid).name} has no '{market_type}' market in ApexAlgo.") + derivative = market_type != "spot" + + def _check_symbol(raw) -> str: + norm = str(raw).strip().upper().replace('-', '/') + if not SYMBOL_PATTERN.match(norm): + errors.append(f"Invalid symbol '{raw}'. Use BASE/QUOTE, e.g. BTC/USDC" + (" (BASE/QUOTE:SETTLE for swaps, e.g. BTC/USDT:USDT)." if derivative else ".")) + elif derivative and not is_derivative(norm): + errors.append(f"Symbol '{raw}' is a spot pair; a swap bot needs the BASE/QUOTE:SETTLE form, e.g. {norm}:{norm.split('/')[-1]}.") + elif not derivative and is_derivative(norm): + errors.append(f"Symbol '{raw}' is a perpetual swap; a spot bot needs BASE/QUOTE, e.g. {norm.split(':')[0]} (or set market_type to 'swap').") + elif is_derivative(norm) and settle_of(norm) == base_of(norm) and exchange_spec(eid) is not None \ + and "inverse" not in supported_kinds(eid, "swap"): + errors.append(f"Symbol '{raw}' is an inverse (coin-margined) contract, which {exchange_spec(eid).name} does not offer in ApexAlgo — " + f"use the linear contract, e.g. {base_of(norm)}/USDT:USDT.") + return norm # Symbols — normalize and validate BASE/QUOTE format, write back normalized values symbols = settings.get("symbols", []) if symbols: - normalized_symbols = [] - for sym in symbols: - norm = str(sym).strip().upper().replace('-', '/') - if not SYMBOL_PATTERN.match(norm): - errors.append(f"Invalid symbol '{sym}'. Use BASE/QUOTE, e.g. BTC/USDC.") - normalized_symbols.append(norm) - settings["symbols"] = normalized_symbols + settings["symbols"] = [_check_symbol(sym) for sym in symbols] else: if settings.get("symbol"): warnings.append("Using single 'symbol' field; consider using 'symbols' list.") else: errors.append("No trading symbols configured.") if settings.get("symbol"): - norm = str(settings["symbol"]).strip().upper().replace('-', '/') - if not SYMBOL_PATTERN.match(norm): - errors.append(f"Invalid symbol '{settings['symbol']}'. Use BASE/QUOTE, e.g. BTC/USDC.") - settings["symbol"] = norm + settings["symbol"] = _check_symbol(settings["symbol"]) + + # One cash currency per bot: the capital pool, PnL and drawdown are all + # summed in one unit — BTC/USDT and ETH/BTC in one whitelist would add + # USDT to BTC + _pairs = [s for s in ([*settings.get("symbols", [])] or ([settings["symbol"]] if settings.get("symbol") else [])) + if SYMBOL_PATTERN.match(str(s))] + _cash = sorted({cash_currency(s) for s in _pairs}) + if len(_cash) > 1: + errors.append(f"Whitelist mixes cash currencies ({', '.join(_cash)}): all pairs of a bot must be quoted/settled in the same currency.") + + # Leverage and margin mode only mean something on a swap + lev_raw = settings.get("leverage") + if lev_raw not in (None, ""): + try: + lev = float(lev_raw) + except (ValueError, TypeError): + lev = None + errors.append(f"leverage '{lev_raw}' is not a valid number.") + if lev is not None: + if lev != int(lev) or lev < 1: + errors.append("leverage must be a whole number >= 1.") + elif derivative: + caps = market_caps(eid, market_type) + if caps is not None and lev > caps.max_leverage: + errors.append(f"leverage {int(lev)}x exceeds the {caps.max_leverage}x ApexAlgo allows on {exchange_spec(eid).name} swaps.") + elif lev > LEVERAGE_WARN_ABOVE: + warnings.append(f"leverage {int(lev)}x: the estimated liquidation sits about {100 * (1 - MAINTENANCE_MARGIN) / lev:.1f}% " + "from the entry — a stop loss that is not tighter than that never fires.") + elif lev > 1: + errors.append(f"leverage {int(lev)}x has no effect on a spot bot; set it to 1 or switch market_type to 'swap'.") + mm_raw = settings.get("margin_mode") + if mm_raw not in (None, ""): + mm = str(mm_raw).strip().lower() + if mm not in MARGIN_MODES: + errors.append(f"Invalid margin_mode '{mm_raw}'. Use one of: {', '.join(MARGIN_MODES)}.") + else: + settings["margin_mode"] = mm + if mm == "cross" and derivative: + warnings.append("margin_mode 'cross': a liquidation takes the whole margin wallet, not just the position's margin. " + "The simulation treats backtest_capital as that wallet; on the exchange other balances in the account are at risk too.") + if derivative and not settings.get("margin_mode"): + settings["margin_mode"] = DEFAULT_MARGIN_MODE # Timeframe — validated against the exchange's supported timeframes tf = settings.get("timeframe") if tf: - eid = exchange_id or settings.get("data_exchange", "okx") supported = get_exchange_timeframes(eid) if supported and tf not in supported: errors.append(f"Exchange '{eid}' does not support timeframe '{tf}'. Supported: {', '.join(sorted(supported.keys()))}") @@ -77,11 +148,30 @@ def validate_bot_settings(settings: dict, exchange_id: str | None = None) -> dic errors.append(f"entry_node '{entry_node}' not found in nodes.") if exit_node and exit_node not in nodes: errors.append(f"exit_node '{exit_node}' not found in nodes.") - if not entry_node and not exit_node: + if not entry_node and not exit_node and not settings.get("short_node"): warnings.append("No entry_node or exit_node configured. Bot will not generate signals.") if exit_node and not entry_node: warnings.append("exit_node is configured without an entry_node; bot will never buy.") + # Short/cover node references (phase 3): derivatives only — a spot + # account has nothing to sell short, so this is an error, not a no-op + short_node = settings.get("short_node") + cover_node = settings.get("cover_node") + if short_node and short_node not in nodes: + errors.append(f"short_node '{short_node}' not found in nodes.") + if cover_node and cover_node not in nodes: + errors.append(f"cover_node '{cover_node}' not found in nodes.") + if (short_node or cover_node) and not derivative: + errors.append("Shorts need a perpetual market: remove the 'Open short' / 'Close short' actions or set market_type to 'swap'.") + if short_node and not cover_node: + _short_cfg = (settings.get("trade_settings") or {}).get("short") or {} + if not (_short_cfg.get("stop_losses") or _short_cfg.get("take_profits")): + warnings.append("short_node is configured without a cover_node, stop loss or take profit; shorts would only close on liquidation or a forced stop.") + if cover_node and not short_node: + warnings.append("cover_node is configured without a short_node; bot will never open a short.") + if not entry_node and short_node: + warnings.append("Short-only strategy: the bot never goes long.") + # Max positions max_pos = settings.get("max_positions", 1) if isinstance(max_pos, (int, float)) and max_pos < 1: @@ -188,6 +278,24 @@ def validate_bot_settings(settings: dict, exchange_id: str | None = None) -> dic for cyc in _find_cycles(nodes): errors.append(f"Node graph has a cycle: {' -> '.join(cyc)}. A node cannot depend on itself.") + # Integer settings the engine reads with sizing._int: coerce here so a + # "150.0" from a form is stored as 150, and refuse anything non-numeric + for _key, _min in (("backtest_lookback", 20), ("cooldown_trades", 0), ("cooldown_candles", 0)): + _raw = settings.get(_key) + if _raw in (None, ""): + continue + try: + _val = float(_raw) + except (ValueError, TypeError): + errors.append(f"{_key} '{_raw}' is not a valid whole number.") + continue + if _val < _min: + errors.append(f"{_key} must be at least {_min}.") + continue + if _val != int(_val) or not isinstance(_raw, int) or isinstance(_raw, bool): + warnings.append(f"{_key} '{_raw}' stored as {int(_val)}.") + settings[_key] = int(_val) + # Backtest lookback must cover the longest indicator warm-up longest = _longest_indicator_length(nodes) try: @@ -214,47 +322,14 @@ def validate_bot_settings(settings: dict, exchange_id: str | None = None) -> dic # Trade settings trade_settings = settings.get("trade_settings", {}) + # The `short` leg (phase 3) has the same shape as `entry` and is checked + # the same way, with a "Short " prefix so messages point at the right node entry_ts = trade_settings.get("entry", {}) - - # Entry amount amount_type = entry_ts.get("amount_type", "percentage") - if amount_type not in VALID_AMOUNT_TYPES: - errors.append(f"Invalid entry amount_type '{amount_type}'.") amount_value = entry_ts.get("amount_value") - if amount_value is not None: - try: - if float(amount_value) <= 0: - warnings.append("Entry amount_value is <= 0.") - except (ValueError, TypeError): - errors.append(f"Entry amount_value '{amount_value}' is not a valid number.") - - # Stop losses - for i, sl in enumerate(entry_ts.get("stop_losses", [])): - sl_type = sl.get("type", "") - if sl_type not in VALID_EXIT_TYPES: - errors.append(f"Stop loss #{i}: invalid type '{sl_type}'.") - try: - if float(sl.get("value", 0)) <= 0: - warnings.append(f"Stop loss #{i}: value is <= 0.") - except (ValueError, TypeError): - errors.append(f"Stop loss #{i}: value is not a valid number.") - cat = sl.get("close_amount_type", "percentage") - if cat not in VALID_CLOSE_AMOUNT_TYPES: - errors.append(f"Stop loss #{i}: invalid close_amount_type '{cat}'.") - - # Take profits - for i, tp in enumerate(entry_ts.get("take_profits", [])): - tp_type = tp.get("type", "") - if tp_type not in VALID_EXIT_TYPES: - errors.append(f"Take profit #{i}: invalid type '{tp_type}'.") - try: - if float(tp.get("value", 0)) <= 0: - warnings.append(f"Take profit #{i}: value is <= 0.") - except (ValueError, TypeError): - errors.append(f"Take profit #{i}: value is not a valid number.") - cat = tp.get("close_amount_type", "percentage") - if cat not in VALID_CLOSE_AMOUNT_TYPES: - errors.append(f"Take profit #{i}: invalid close_amount_type '{cat}'.") + _validate_entry_leg(entry_ts, "", errors, warnings) + if isinstance(trade_settings.get("short"), dict) and trade_settings["short"]: + _validate_entry_leg(trade_settings["short"], "Short ", errors, warnings) # Live allocation: share of the exchange wallet this bot may deploy if settings.get("live_allocation_pct") not in (None, ""): @@ -275,7 +350,7 @@ def validate_bot_settings(settings: dict, exchange_id: str | None = None) -> dic except (ValueError, TypeError): max_order_value = 0 if max_order_value <= 0: - errors.append("Live execution requires max_order_value > 0 as a safety cap.") + errors.append("Live execution requires max_order_value > 0 as a safety cap" + (" (on the notional, i.e. margin x leverage)." if derivative else ".")) else: # The engine clamps every live entry to the cap, so a cap below # the configured size silently turns the strategy into a smaller @@ -285,11 +360,14 @@ def validate_bot_settings(settings: dict, exchange_id: str | None = None) -> dic planned = float(amount_value or 0) if amount_type == "percentage": planned = capital * planned / 100.0 + if derivative: + # The cap is on the notional; the entry is margin x leverage + planned *= max(float(settings.get("leverage") or 1), 1.0) if planned > max_order_value > 0: warnings.append( f"max_order_value ({max_order_value:,.0f}) is below the planned entry size " - f"({planned:,.0f}): live entries will be capped to {max_order_value:,.0f}, " - "so live sizing differs from the backtest. Raise the cap or lower the entry amount." + f"({planned:,.0f}): entries are capped to {max_order_value:,.0f} in the backtest, " + "forward test and live alike (the cap is part of the strategy). Raise the cap or lower the entry amount." ) except (ValueError, TypeError): pass @@ -374,3 +452,47 @@ def _longest_indicator_length(nodes: dict) -> int: except (ValueError, TypeError): pass return longest + + +def _validate_entry_leg(entry_ts: dict, leg: str, errors: list, warnings: list) -> None: + """Amount + stop-loss/take-profit checks for one opening leg + (`trade_settings.entry` or `.short`); `leg` prefixes the messages.""" + # Entry amount + amount_type = entry_ts.get("amount_type", "percentage") + if amount_type not in VALID_AMOUNT_TYPES: + errors.append(f"Invalid {leg.lower()}entry amount_type '{amount_type}'.") + amount_value = entry_ts.get("amount_value") + if amount_value is not None: + try: + if float(amount_value) <= 0: + warnings.append(f"{leg}Entry amount_value is <= 0." if leg else "Entry amount_value is <= 0.") + except (ValueError, TypeError): + errors.append(f"{leg}Entry amount_value '{amount_value}' is not a valid number." if leg else f"Entry amount_value '{amount_value}' is not a valid number.") + + # Stop losses + for i, sl in enumerate(entry_ts.get("stop_losses", [])): + sl_type = sl.get("type", "") + if sl_type not in VALID_EXIT_TYPES: + errors.append(f"{leg}Stop loss #{i}: invalid type '{sl_type}'.") + try: + if float(sl.get("value", 0)) <= 0: + warnings.append(f"{leg}Stop loss #{i}: value is <= 0.") + except (ValueError, TypeError): + errors.append(f"{leg}Stop loss #{i}: value is not a valid number.") + cat = sl.get("close_amount_type", "percentage") + if cat not in VALID_CLOSE_AMOUNT_TYPES: + errors.append(f"{leg}Stop loss #{i}: invalid close_amount_type '{cat}'.") + + # Take profits + for i, tp in enumerate(entry_ts.get("take_profits", [])): + tp_type = tp.get("type", "") + if tp_type not in VALID_EXIT_TYPES: + errors.append(f"{leg}Take profit #{i}: invalid type '{tp_type}'.") + try: + if float(tp.get("value", 0)) <= 0: + warnings.append(f"{leg}Take profit #{i}: value is <= 0.") + except (ValueError, TypeError): + errors.append(f"{leg}Take profit #{i}: value is not a valid number.") + cat = tp.get("close_amount_type", "percentage") + if cat not in VALID_CLOSE_AMOUNT_TYPES: + errors.append(f"{leg}Take profit #{i}: invalid close_amount_type '{cat}'.") diff --git a/backend/engine/sizing.py b/backend/engine/sizing.py index 76628ca..eaa4845 100644 --- a/backend/engine/sizing.py +++ b/backend/engine/sizing.py @@ -7,6 +7,10 @@ from sqlalchemy import text, func from backend.models.bots import BotConfig from backend.models.positions import Position +from backend.models.orders import Order +from backend.engine import pnl +from backend.engine.contracts import ContractSpec, spec_for, spec_from_symbol +from backend.engine.symbols import cash_currency logger = logging.getLogger("apexalgo.bot_manager") @@ -61,21 +65,45 @@ def _naive_utc(ts): # Settings that do not change what the strategy does on the data: layout, # order routing, live sizing, the pinned backtest window (which data, not # what the strategy does on it) and the engine's own runtime bookkeeping. +# `max_order_value` is *not* here: it caps the backtest/forward sizing too, +# so it changes the trades and counts as a variant. _NON_STRATEGY_KEYS = frozenset({ - "ui_layout", "api_execution", "api_key_name", "live_allocation_pct", "max_order_value", + "ui_layout", "api_execution", "api_key_name", "live_allocation_pct", "backtest_on_start", "backtest_from", "backtest_to", "last_backtest_summary", "last_backtest_max_drawdown", "last_stop_reason", "drawdown_peak_reset_at", "live_starting_capital", }) +# Phase-2 (derivatives) settings at their default are dropped from the +# fingerprint: a spot bot saved by a newer UI must hash exactly like the +# same bot saved before these keys existed, or every variant counter bumps. +_DEFAULT_MARKET_KEYS = {"market_type": "spot", "leverage": 1, "margin_mode": "isolated", + # phase 3: absent short/cover nodes hash like a pre-shorts bot + "short_node": None, "cover_node": None} + + def _config_fingerprint(settings: dict) -> str: """Stable hash of the strategy-relevant part of a bot's settings, used to count how many distinct variants have been backtested.""" relevant = {k: v for k, v in (settings or {}).items() if k not in _NON_STRATEGY_KEYS} + for k, default in _DEFAULT_MARKET_KEYS.items(): + if k in relevant and _is_default_market_setting(relevant[k], default): + del relevant[k] return md5(json.dumps(relevant, sort_keys=True, default=str).encode()).hexdigest() +def _is_default_market_setting(value, default) -> bool: + if value in (None, ""): + return True + if isinstance(default, (int, float)): + try: + return float(value) == float(default) + except (TypeError, ValueError): + return False + return str(value).lower() == default + + def data_fingerprint(df) -> str: """sha256 of the raw OHLCV rows a backtest walked (sorted by timestamp). `repr` on the floats makes it byte-stable across runs, so two runs on the @@ -144,28 +172,50 @@ def _record_config_run(db, bot_name: str, settings: dict, slice_hash: str = "", return total, on_slice -def sim_frictions(settings): +def sim_frictions(settings, side="long"): """(entry_fee, exit_fee, entry_slippage, exit_slippage) as fractions from trade_settings — the frictions the backtest applies to every - simulated fill. Exit fee falls back to the entry fee when unset.""" + simulated fill. Exit fee falls back to the entry fee when unset. For a + short the `short`/`cover` legs are read (falling back to entry/exit).""" ts = settings.get("trade_settings", {}) or {} - entry_fee = _num(ts.get("entry", {}).get("fee"), 0) / 100 - raw_exit_fee = ts.get("exit", {}).get("fee") + open_cfg = pnl.entry_cfg(ts, side) + close_cfg = pnl.exit_cfg(ts, side) + entry_fee = _num(open_cfg.get("fee"), 0) / 100 + raw_exit_fee = close_cfg.get("fee") exit_fee = _num(raw_exit_fee, entry_fee * 100) / 100 if raw_exit_fee not in (None, "") else entry_fee - entry_slip = _num(ts.get("entry", {}).get("slippage"), 0) / 100 - exit_slip = _num(ts.get("exit", {}).get("slippage"), 0) / 100 + entry_slip = _num(open_cfg.get("slippage"), 0) / 100 + exit_slip = _num(close_cfg.get("slippage"), 0) / 100 return entry_fee, exit_fee, entry_slip, exit_slip +def position_spec(symbol, contract_kind=None, contract_size=None, exchange=None): + """`ContractSpec` for a stored position: the kind/size it was booked + with (so a later contract-size change on the exchange never rewrites + history), else the registry's cached market, else the symbol's shape.""" + if contract_kind: + shape = spec_from_symbol(symbol, contract_size) + return ContractSpec(symbol=shape.symbol, kind=str(contract_kind), base=shape.base, quote=shape.quote, + settle=shape.settle, contract_size=shape.contract_size) + return spec_for(exchange, symbol) + + def deployed_capital(db, bot_names, quote, modes=("paper", "live")): - """Quote-currency cost (entry price x amount) of the open positions of - the given bots in the given modes, in pairs quoted in `quote`.""" + """Cash locked by the open positions of the given bots in the given + modes, in pairs funded in `quote` (the cash currency): entry price x + amount on spot, the margin (notional / leverage) on derivatives.""" if not bot_names: return 0.0 - rows = db.query(Position.entry_price, Position.amount, Position.symbol).filter( + rows = db.query(Position.entry_price, Position.amount, Position.symbol, Position.leverage, + Position.contract_kind, Position.contract_size, Position.exchange).filter( Position.bot_name.in_(list(bot_names)), Position.status == "open", Position.mode.in_(list(modes))).all() - return sum((r[0] or 0.0) * (r[1] or 0.0) for r in rows if (r[2] or "").replace('-', '/').upper().endswith('/' + quote)) + total = 0.0 + for entry, amount, sym, lev, kind, size, exchange in rows: + spec = position_spec(sym, kind, size, exchange) + if spec.cash_currency != quote: + continue + total += spec.margin(amount, entry, lev if (lev and lev > 1) else 1) + return total def forward_pool(db, bot, quote): @@ -178,7 +228,21 @@ def forward_pool(db, bot, quote): Position.bot_name == bot.name, Position.status == "closed", Position.mode == "forward_test").scalar() or 0.0 deployed = deployed_capital(db, [bot.name], quote, modes=("forward_test",)) - return _num(bot.settings.get("backtest_capital"), 1000) + float(realized) - deployed + # The entry fee left the pool on open as well (bt_equity -= cost + fee) + open_ids = [pid for (pid, sym) in db.query(Position.id, Position.symbol).filter( + Position.bot_name == bot.name, Position.status == "open", + Position.mode == "forward_test").all() if cash_currency(sym) == quote] + entry_fees = 0.0 + open_funding = 0.0 + if open_ids: + entry_fees = db.query(func.coalesce(func.sum(Order.fee), 0.0)).filter( + Order.position_id.in_(open_ids), Order.mode == "forward_test", + func.coalesce(Order.reduce_only, 0) == 0).scalar() or 0.0 + # Funding already charged on the open positions moved the pool's + # cash too (closed positions carry it inside profit_abs) + open_funding = db.query(func.coalesce(func.sum(Position.funding_paid), 0.0)).filter( + Position.id.in_(open_ids)).scalar() or 0.0 + return _num(bot.settings.get("backtest_capital"), 1000) + float(realized) - deployed - float(entry_fees) + float(open_funding) def live_allocation(db, bot, quote, free_balance): @@ -198,12 +262,55 @@ def live_allocation(db, bot, quote, free_balance): return max(bot_total - deployed_bot, 0.0), wallet_total, bot_total -def calculate_trade_amount(current_price, bot_settings, current_equity=None): +def max_affordable_amount(equity, entry_price, leverage=1, fee=0.0, spec=None): + """Largest size whose locked capital (margin plus entry fee on the + notional) fits in `equity` at `entry_price`. Used to clamp percentage + sizing so slippage + fee never push an entry past the pool; reduces to + `equity / (price * (1 + fee))` on spot. Contracts on an inverse `spec`.""" + if spec is not None and spec.is_inverse: + return spec.max_affordable_qty(equity, entry_price, leverage, fee) + lev = max(float(leverage or 1), 1.0) + return equity / (entry_price * (1 / lev + float(fee or 0.0))) + + +def cap_by_max_order_value(trade_amount, price, settings, spec=None): + """Backtest/forward/live mirror of the `max_order_value` clamp: the + *quote* notional of one entry (USD on ``BTC/USD:BTC``, EUR on + ``BTC/EUR``) never exceeds the cap. `max_order_value` is defined in the + pair's quote currency on every market kind.""" + cap = _num(settings.get("max_order_value"), 0) + if trade_amount is None or cap <= 0 or not price or price <= 0: + return trade_amount + if spec is not None and spec.is_inverse: + if spec.notional_quote(trade_amount, price) > cap: + return spec.qty_for_quote_notional(cap, price) + return trade_amount + if trade_amount * price > cap: + return cap / price + return trade_amount + + +def calculate_trade_amount(current_price, bot_settings, current_equity=None, leverage=1, side="long", spec=None): + """Position size to buy (or sell short): base amount on spot/linear, + contracts on an inverse `spec`. `amount_value` is what the bot puts up + (in its cash currency): a `percentage` of the *free cash* in its pool + (`current_equity` = cash not locked in open positions, not the + mark-to-market equity — so a second layer at 50% is 50% of what is left, + not of the account), or a `fixed` cash amount; on derivatives that is the + margin, so the notional — and the returned size — is `leverage` times + bigger. Spot callers never pass leverage and get the pre-existing sizing; + a short reads `trade_settings.short` (fallback `entry`).""" if not current_price or current_price <= 0: logger.warning("Invalid current_price (%s), cannot calculate trade amount", current_price) return None + try: + leverage = float(leverage or 1) + except (TypeError, ValueError): + leverage = 1.0 + if leverage < 1: + leverage = 1.0 - entry_settings = bot_settings.get("trade_settings", {}).get("entry", {}) + entry_settings = pnl.entry_cfg(bot_settings.get("trade_settings", {}), side) amount_type = entry_settings.get("amount_type", "percentage") raw_val = entry_settings.get("amount_value") @@ -213,12 +320,16 @@ def calculate_trade_amount(current_price, bot_settings, current_equity=None): amount_value = 100.0 if amount_type == "fixed": - trade_amount = amount_value / current_price - return max(trade_amount, 0.0001) + investment = amount_value else: capital = current_equity if current_equity is not None else _num(bot_settings.get("backtest_capital"), 1000) if capital <= 0: return None investment = capital * (amount_value / 100) + if spec is not None and spec.is_inverse: + trade_amount = spec.qty_for_cash(investment, current_price, leverage) + else: trade_amount = investment / current_price - return max(trade_amount, 0.0001) + if leverage != 1.0: + trade_amount *= leverage + return max(trade_amount, 0.0001) diff --git a/backend/engine/startup.py b/backend/engine/startup.py index 13a0f7e..b119d8b 100644 --- a/backend/engine/startup.py +++ b/backend/engine/startup.py @@ -16,8 +16,11 @@ from backend.models.positions import Position from backend.models.exchange_keys import ExchangeKey from backend.engine.evaluator import NodeEvaluator -from backend.engine import backtest +from backend.engine import backtest, broker, funding, live_cycle, risk, tiers +from backend.engine.contracts import MAINTENANCE_MARGIN +from sqlalchemy.orm import selectinload from backend.engine.sizing import _num, _int, _tf_seconds, _naive_utc, backtest_pin, data_fingerprint +from backend.engine.symbols import DEFAULT_MARGIN_MODE, base_of, cash_currency, is_derivative, leverage_for, market_type_for from backend.core.exchange_registry import get_exchange_timeframes from backend.core import bot_log_buffer as blb @@ -130,10 +133,7 @@ def _check_abort(): for symbol in symbols: _window = f"pinned {pin_from:%Y-%m-%d} → {pin_to:%Y-%m-%d}" if pinned else f"lookback={lookback_limit}" blb.push(bot.name, "INFO", f"Starting: {symbol} | {timeframe} | {live_mode} | {_window}") - tf_seconds = 60 - if timeframe.endswith('m'): tf_seconds = int(timeframe[:-1]) * 60 - elif timeframe.endswith('h'): tf_seconds = int(timeframe[:-1]) * 3600 - elif timeframe.endswith('d'): tf_seconds = int(timeframe[:-1]) * 86400 + tf_seconds = _tf_seconds(timeframe) # Use fresh sessions for all polling queries. The main `db` session starts an # implicit SQLite transaction on its first read (line above), so any subsequent @@ -281,6 +281,11 @@ def _latest_candle_ts(): entry_series = evaluator.resolve_node(bot.settings.get("entry_node")) if bot.settings.get("entry_node") else pd.Series(False, index=evaluator.df.index) exit_series = evaluator.resolve_node(exit_node) if exit_node else pd.Series(False, index=evaluator.df.index) + # Shorts (phase 3): only when the strategy has the nodes; None keeps + # the simulate loop on its long-only path + short_node, cover_node = bot.settings.get("short_node"), bot.settings.get("cover_node") + short_series = evaluator.resolve_node(short_node) if short_node else None + cover_series = evaluator.resolve_node(cover_node) if cover_node else None # Pre-extract numpy arrays once — avoids O(n) .iloc index lookups inside the loop _standard_cols = {'id', 'timestamp', 'open', 'high', 'low', 'close', 'volume', 'atr'} @@ -289,6 +294,8 @@ def _latest_candle_ts(): "df": evaluator.df, "entry_arr": entry_series.values, "exit_arr": exit_series.values, + "short_arr": short_series.values if short_series is not None else None, + "cover_arr": cover_series.values if cover_series is not None else None, "atr_arr": evaluator.df['atr'].values if 'atr' in evaluator.df.columns else None, "indicator_cols": [c for c in evaluator.df.columns if c not in _standard_cols], "existing_timestamps": existing_timestamps, @@ -317,18 +324,48 @@ def _latest_candle_ts(): if not still_active(engine, bot_id, _run_token): raise StoppedByUser() + # Perpetuals: the funding settlements of the backtest window and the + # exchange's maintenance-margin tiers, stored next to the candles. + # Best effort — the simulation reports what it could not get. + if any(is_derivative(c["symbol"]) for c in sym_contexts): + engine.set_runtime(bot.name, "fetching", "Funding rates and margin tiers") + for c in sym_contexts: + if not is_derivative(c["symbol"]) or len(c["df"]) == 0: + continue + _ts = c["df"]["timestamp"] + try: + _new = funding.ensure_history(db, exchange_name, c["symbol"], _ts.iloc[0], _ts.iloc[-1]) + if _new: + blb.push(bot.name, "INFO", f"{c['symbol']}: stored {_new} new funding settlement(s) from {exchange_name.upper()}") + except Exception as _exc: + logger.warning("Bot '%s': funding history for %s failed: %s", bot.name, c["symbol"], _exc) + blb.push(bot.name, "WARN", f"{c['symbol']}: could not fetch funding history ({_exc}) — using what is stored") + try: + tiers.ensure(db, exchange_name, c["symbol"]) + except Exception as _exc: + logger.warning("Bot '%s': leverage tiers for %s failed: %s", bot.name, c["symbol"], _exc) + blb.push(bot.name, "WARN", f"{c['symbol']}: could not fetch the maintenance-margin tiers ({_exc}) — flat {100 * MAINTENANCE_MARGIN:g}% assumed") + if not still_active(engine, bot_id, _run_token): + raise StoppedByUser() + res = backtest.simulate( db, bot, sym_contexts, exchange_name, run_backtest, check_exits=engine._check_exits, position_states=engine.position_states, states_lock=engine._position_states_lock, on_progress=lambda detail, progress=None: engine.set_runtime(bot.name, "backtesting", detail, progress), check_abort=_check_abort) bt_equity = res.equity + _ccy = res.cash_currency or (cash_currency(sym_contexts[0]["symbol"]) if sym_contexts else "USDT") + # Console lines the simulation buffered: pushed only now, after its + # commit — the log buffer writes on a second connection and would + # otherwise wait on the uncommitted backtest rows + for _lvl, _msg in res.console: + blb.push(bot.name, _lvl, _msg) for ctx in sym_contexts: trade_count = len(ctx["trade_entry_indices"]) if run_backtest else 0 - logger.info("Backfill complete: '%s' on %s | mode=%s | %d candles | %d trades | equity=$%.2f", bot.name, ctx["symbol"], live_mode.upper(), len(ctx["df"]), trade_count, bt_equity) + logger.info("Backfill complete: '%s' on %s | mode=%s | %d candles | %d trades | equity=%.2f %s", bot.name, ctx["symbol"], live_mode.upper(), len(ctx["df"]), trade_count, bt_equity, _ccy) if run_backtest: - blb.push(bot.name, "INFO", f"Backtest complete: {ctx['symbol']} | {len(ctx['df'])} candles | {trade_count} trades | equity=${bt_equity:.2f}") + blb.push(bot.name, "INFO", f"Backtest complete: {ctx['symbol']} | {len(ctx['df'])} candles | {trade_count} trades | equity={bt_equity:.2f} {_ccy}") else: blb.push(bot.name, "INFO", f"Ready: {ctx['symbol']} | {len(ctx['df'])} candles | mode={live_mode.upper()}") @@ -366,9 +403,9 @@ def _latest_candle_ts(): # as the base for live drawdown / capital-loss percentages. if live_mode in ("paper", "live"): try: - _ccxt = engine._get_ccxt_instance(api_key) - _bal = _ccxt.fetch_balance() _pairs = [str(s).replace('-', '/').upper() for s in symbols] + _ccxt = engine._ccxt_for(api_key, _pairs[0] if _pairs else None) + _bal = _ccxt.fetch_balance() # The backtest ran on public (production) candles; a demo # account or another region can list fewer pairs, and an # order on a missing one can only fail @@ -377,14 +414,15 @@ def _latest_candle_ts(): blb.push(bot.name, "WARN", f"Not listed on {api_key.exchange.upper()}{' demo' if api_key.is_sandbox else ''} for key '{api_key.name}': {', '.join(_missing)} — {live_mode} entries on these pairs will be skipped") _tokens = [] for _p in _pairs: - for _t in _p.split('/'): + # Derivatives hold no base coin: only the settle currency matters + for _t in ([cash_currency(_p)] if is_derivative(_p) else [base_of(_p), cash_currency(_p)]): if _t not in _tokens: _tokens.append(_t) def _free(t): v = _bal.get(t) return float((v or {}).get("free") or 0) if isinstance(v, dict) else float((_bal.get("free") or {}).get(t) or 0) _parts = [f"{_free(t):,.4f}".rstrip('0').rstrip('.') + f" {t}" for t in _tokens] - _quote = _pairs[0].split('/')[-1] if _pairs else "USDT" + _quote = cash_currency(_pairs[0]) if _pairs else "USDT" _pool, _wallet_total, _bot_total = engine._live_allocation(db, bot, _quote, _free(_quote)) _pct = _num(bot.settings.get("live_allocation_pct"), 100) blb.push(bot.name, "INFO", f"Wallet '{api_key.name}' ({live_mode}): {', '.join(_parts)} free — this bot: {_pct:.0f}% = {_bot_total:,.2f} {_quote} ({_pool:,.2f} still deployable)") @@ -404,19 +442,74 @@ def _free(t): # Startup reconciliation: open DB positions must still be backed # by the exchange balance, otherwise exits would be sent for - # coins that are no longer there - if _bal is not None: + # coins that are no longer there. A reconciliation that cannot + # be performed (no balance, exchange error) is a WARN for a + # sandbox and a refusal to go live for real money: an + # unverified live bot must not manage positions blind. + _open_real = db.query(Position).filter( + Position.bot_name == bot.name, Position.status == "open", Position.mode == live_mode).count() + _recon_error = None + _problems = [] + if _bal is None: + _recon_error = "wallet balance unavailable" + else: try: - _problems = engine._reconcile_positions_with_wallet(db, bot, _ccxt, _bal, live_mode) + if market_type_for(bot.settings, api_key) != "spot": + # A perp position is not a coin in the wallet: a + # position the exchange no longer holds was + # liquidated (or closed by hand) while the bot was + # down — book it as such instead of managing a ghost + for _pos in db.query(Position).options(selectinload(Position.orders)).filter( + Position.bot_name == bot.name, Position.status == "open", Position.mode == live_mode).all(): + if broker.exchange_position_gone(_ccxt, _pos.symbol, _pos.side or "long"): + _last = db.query(Candle.close).filter( + Candle.exchange == _pos.exchange, Candle.symbol == _pos.symbol, Candle.timeframe == timeframe + ).order_by(Candle.timestamp.desc()).first() + live_cycle._book_liquidation(engine, db, bot, _pos, float(_last[0]) if _last else _pos.entry_price, + _naive_utc(datetime.now(timezone.utc)), risk.mode_group_for(live_mode), + "exchange holds no such position at startup — booked as liquidation, check the exchange") + db.commit() + # Then compare what is left against the exchange's open positions + _problems = engine._reconcile_positions_with_exchange(db, bot, _ccxt, live_mode) + else: + _problems = engine._reconcile_positions_with_wallet(db, bot, _ccxt, _bal, live_mode) except Exception as _exc: logger.warning("Bot '%s': position reconciliation failed: %s", bot.name, _exc) - _problems = [] - if _problems: - for _p in _problems: - blb.push(bot.name, "ERROR", f"{_p} — reconcile manually (close or delete the position in Analytics) before starting") - engine._engine_stop(bot, db, f"{_problems[0]} — reconcile manually") + _recon_error = str(_exc) + if _recon_error is not None: + if live_mode == "live": + blb.push(bot.name, "ERROR", f"Could not reconcile positions with {api_key.exchange.upper()} ({_recon_error}) — refusing to go live unverified") + engine._engine_stop(bot, db, f"Position reconciliation failed: {_recon_error} — fix the key/connection and restart") + db.commit() + return + blb.push(bot.name, "WARN", f"Could not reconcile positions with the exchange ({_recon_error}) — {_open_real} open {live_mode} position(s) unverified") + if _problems: + for _p in _problems: + blb.push(bot.name, "ERROR", f"{_p} — reconcile manually (close or delete the position in Analytics) before starting") + engine._engine_stop(bot, db, f"{_problems[0]} — reconcile manually") + db.commit() + return + + # Derivatives: confirm leverage and margin mode on the exchange + # for every pair before the first order — a bot that cannot set + # what it backtested with must not trade + _mt = market_type_for(bot.settings, api_key) + if _mt != "spot": + _lev = leverage_for(bot.settings, _mt) + _mm = bot.settings.get("margin_mode") or DEFAULT_MARGIN_MODE + try: + for _p in [str(s).replace('-', '/').upper() for s in symbols]: + _ccxt = engine._ccxt_for(api_key, _p) + if _ccxt.markets and _p not in _ccxt.markets: + continue + engine._ensure_leverage(_ccxt, api_key, _p, _lev, _mm, bot.name) + except Exception as _exc: + logger.error("Bot '%s': could not apply leverage: %s", bot.name, _exc) + blb.push(bot.name, "ERROR", f"Could not set {_lev:g}x {_mm} on {api_key.exchange.upper()}: {_exc}") + engine._engine_stop(bot, db, f"Could not set {_lev:g}x {_mm} leverage on the exchange: {_exc}") db.commit() return + blb.push(bot.name, "INFO", f"Perpetual swap ({live_mode}): {_lev:g}x {_mm} confirmed on {api_key.exchange.upper()} for {len(symbols)} pair(s) — funding and liquidation are settled by the exchange") # Make the backtest→live handover visible in the console: the next # tick only arrives when the current candle closes on the exchange diff --git a/backend/engine/symbols.py b/backend/engine/symbols.py new file mode 100644 index 0000000..d26afc5 --- /dev/null +++ b/backend/engine/symbols.py @@ -0,0 +1,73 @@ +"""Unified-symbol helpers shared by the engine and the routers. + +ccxt symbols are ``BASE/QUOTE`` for spot and ``BASE/QUOTE:SETTLE`` for +derivatives (``BTC/USDT:USDT`` = the USDT-margined BTC perpetual). Every +place that used to split on ``/`` goes through here, so a derivative symbol +never yields ``"USDT:USDT"`` as its quote currency. For spot symbols the +results are exactly what the old ``split('/')`` produced. +""" + +MARKET_TYPES = ("spot", "swap") +DEFAULT_MARKET_TYPE = "spot" +DEFAULT_MARGIN_MODE = "isolated" +MARGIN_MODES = ("isolated", "cross") + + +def normalize(symbol) -> str: + """``btc-usdt`` / ``BTC-USDT:USDT`` → ``BTC/USDT`` / ``BTC/USDT:USDT``.""" + return str(symbol or "").replace('-', '/').upper() + + +def is_derivative(symbol) -> bool: + """True for the ``BASE/QUOTE:SETTLE`` form.""" + return ':' in str(symbol or "") + + +def market_type_of(symbol) -> str: + return "swap" if is_derivative(symbol) else "spot" + + +def base_of(symbol) -> str: + return normalize(symbol).split('/')[0] + + +def quote_of(symbol) -> str: + """Quote currency: ``BTC/USDT`` → ``USDT``, ``BTC/USDT:USDT`` → ``USDT``.""" + sym = normalize(symbol) + rest = sym.split('/')[-1] if '/' in sym else sym + return rest.split(':')[0] + + +def settle_of(symbol): + """Settlement currency of a derivative symbol, None for spot.""" + sym = normalize(symbol) + return sym.split(':')[-1] if ':' in sym else None + + +def cash_currency(symbol) -> str: + """The currency a position is funded in and PnL is paid in: the settle + currency for derivatives, the quote currency for spot.""" + return settle_of(symbol) or quote_of(symbol) + + +def market_type_for(settings: dict, key_record=None) -> str: + """Market type a bot trades on. A linked API key is bound to one market + type and wins (the validator refuses a mismatch on save); otherwise the + bot's own setting, defaulting to spot for every pre-existing bot.""" + if key_record is not None: + kt = getattr(key_record, "market_type", None) + if kt: + return str(kt).lower() + return str((settings or {}).get("market_type") or DEFAULT_MARKET_TYPE).lower() + + +def leverage_for(settings: dict, market_type: str | None = None) -> float: + """Configured leverage; always 1 on spot whatever the setting says.""" + mt = market_type or market_type_for(settings) + if mt == "spot": + return 1.0 + try: + lev = float((settings or {}).get("leverage") or 1) + except (TypeError, ValueError): + lev = 1.0 + return lev if lev >= 1 else 1.0 diff --git a/backend/engine/tiers.py b/backend/engine/tiers.py new file mode 100644 index 0000000..5fa8c3d --- /dev/null +++ b/backend/engine/tiers.py @@ -0,0 +1,103 @@ +"""Tiered maintenance margin for perpetual swaps. + +Exchanges keep a larger maintenance margin for larger positions: a list of +brackets (``fetch_market_leverage_tiers``) each with a size range, a +maintenance-margin rate and a leverage cap. The liquidation price of a +position depends on the rate of its bracket, so the simulation looks its +rate up here instead of using the flat ``MAINTENANCE_MARGIN`` — which +remains the fallback when the exchange serves no tiers (spot exchanges, +network errors, tests). + +Size is the quote notional of the position on linear contracts and the +contract count on inverse ones (the Binance COIN-M / OKX convention ccxt +normalises the brackets to). Tiers are stored per (exchange, symbol) and +refreshed by the bot startup at most once a week. +""" +import logging +from datetime import datetime, timedelta, timezone + +from sqlalchemy import func + +from backend.engine.contracts import MAINTENANCE_MARGIN +from backend.engine.sizing import _naive_utc +from backend.models.market_data import LeverageTier + +logger = logging.getLogger("apexalgo.tiers") + +MAX_AGE = timedelta(days=7) + + +def fetch(exchange_id: str, symbol: str) -> list: + """`[(tier, min_size, max_size|None, mmr, max_leverage|None)]` from the + exchange; empty when it has no tier endpoint. Raises on network errors.""" + from backend.core.exchange_registry import build_exchange_for_symbol + exchange = build_exchange_for_symbol(exchange_id, symbol) + if not exchange.has.get("fetchMarketLeverageTiers") and not exchange.has.get("fetchLeverageTiers"): + return [] + exchange.load_markets() + if exchange.has.get("fetchMarketLeverageTiers"): + rows = exchange.fetch_market_leverage_tiers(symbol) or [] + else: + rows = (exchange.fetch_leverage_tiers([symbol]) or {}).get(symbol) or [] + out = [] + for i, r in enumerate(rows): + mmr = r.get("maintenanceMarginRate") + if mmr is None: + continue + lo = r.get("minNotional") + hi = r.get("maxNotional") + lev = r.get("maxLeverage") + out.append((int(r.get("tier") or (i + 1)), float(lo or 0.0), float(hi) if hi else None, + float(mmr), float(lev) if lev else None)) + out.sort(key=lambda t: t[1]) + return out + + +def store(db, exchange_id: str, symbol: str, rows: list) -> None: + """Replace the stored brackets of the symbol with `rows`.""" + if not rows: + return + now = _naive_utc(datetime.now(timezone.utc)) + db.query(LeverageTier).filter(LeverageTier.exchange == exchange_id, LeverageTier.symbol == symbol).delete() + for tier, lo, hi, mmr, lev in rows: + db.add(LeverageTier(exchange=exchange_id, symbol=symbol, tier=tier, min_size=lo, max_size=hi, + mmr=mmr, max_leverage=lev, fetched_at=now)) + db.commit() + + +def ensure(db, exchange_id: str, symbol: str, max_age: timedelta = MAX_AGE) -> bool: + """Fetch the brackets when none are stored or the stored ones are older + than `max_age`. Returns True when tiers are available afterwards; raises + on exchange errors (the caller decides whether that is a WARN).""" + newest = db.query(func.max(LeverageTier.fetched_at)).filter( + LeverageTier.exchange == exchange_id, LeverageTier.symbol == symbol).scalar() + now = _naive_utc(datetime.now(timezone.utc)) + if newest is not None and now - _naive_utc(newest) < max_age: + return True + rows = fetch(exchange_id, symbol) + store(db, exchange_id, symbol, rows) + return bool(rows) or newest is not None + + +def load(db, exchange_id: str, symbol: str) -> list: + """Sorted `[(min_size, max_size|None, mmr)]` of the stored brackets.""" + rows = db.query(LeverageTier.min_size, LeverageTier.max_size, LeverageTier.mmr).filter( + LeverageTier.exchange == exchange_id, LeverageTier.symbol == symbol).order_by(LeverageTier.min_size.asc()).all() + return [(float(lo or 0.0), float(hi) if hi is not None else None, float(mmr)) for lo, hi, mmr in rows] + + +def mmr_for(tiers: list, size: float) -> float: + """Maintenance-margin rate of the bracket `size` falls in; the flat + `MAINTENANCE_MARGIN` without tiers.""" + if not tiers: + return MAINTENANCE_MARGIN + size = float(size or 0.0) + for lo, hi, mmr in tiers: + if size >= lo and (hi is None or size < hi): + return mmr + return tiers[-1][2] if size >= tiers[-1][0] else tiers[0][2] + + +def tier_size(spec, qty, price) -> float: + """The size the brackets are keyed on for `qty` of this contract.""" + return float(qty or 0.0) if spec.is_inverse else spec.notional_quote(qty, price) diff --git a/backend/main.py b/backend/main.py index 12e426c..b749cfe 100644 --- a/backend/main.py +++ b/backend/main.py @@ -26,13 +26,14 @@ from backend.models.bots import BotConfig from backend.models.bot_logs import BotLog from backend.models.bot_config_runs import BotConfigRun +from backend.models.market_data import FundingRate, LeverageTier # Import the routers from backend.routers import auth, keys, data, bots, trades, indicators # Import the background services from backend.engine.candle_poller import candle_poller from backend.engine.bot_manager import bot_manager -from backend.core.exchange_registry import build_exchange +from backend.core.exchange_registry import build_exchange_for_symbol # Create database tables and run migrations for existing DBs Base.metadata.create_all(bind=engine) @@ -63,7 +64,7 @@ async def lifespan(app: FastAPI): enable_docs = os.getenv("ENABLE_DOCS", "0") == "1" app = FastAPI( title="ApexAlgo Engine API", - version="2.2.1", + version="2.3.0", swagger_ui_init_oauth={"clientId": "test"}, lifespan=lifespan, docs_url="/docs" if enable_docs else None, @@ -105,8 +106,8 @@ def read_root(): @app.get("/api/price/{symbol}", dependencies=[Depends(verify_api_key)]) def get_price(symbol: str, exchange: str = Query(default="okx")): try: - exch = build_exchange(exchange.lower()) formatted_symbol = symbol.replace('-', '/').upper() + exch = build_exchange_for_symbol(exchange.lower(), formatted_symbol) ticker = exch.fetch_ticker(formatted_symbol) return { "exchange": exchange.upper(), diff --git a/backend/models/exchange_keys.py b/backend/models/exchange_keys.py index ddd43c3..1933dfa 100644 --- a/backend/models/exchange_keys.py +++ b/backend/models/exchange_keys.py @@ -12,4 +12,7 @@ class ExchangeKey(Base): api_secret = Column(String, nullable=False) passphrase = Column(String, nullable=False) is_sandbox = Column(Boolean, default=True) + # One key = one market type ("spot" | "swap"): spot and perpetual + # accounts are separate ccxt instances (and on some exchanges separate keys) + market_type = Column(String, default="spot") created_at = Column(DateTime, default=lambda: datetime.now(timezone.utc)) diff --git a/backend/models/market_data.py b/backend/models/market_data.py new file mode 100644 index 0000000..2f3a735 --- /dev/null +++ b/backend/models/market_data.py @@ -0,0 +1,45 @@ +"""Derivatives market data next to the candles: funding rates and the +exchange's maintenance-margin tiers. Both are stored per (exchange, symbol) +like candles and never overwritten by the poller, so a backtest that ran +with them stays reproducible.""" +from datetime import datetime, timezone +from sqlalchemy import Column, DateTime, Float, Integer, String, UniqueConstraint +from backend.core.database import Base + + +def _utcnow(): + return datetime.now(timezone.utc).replace(tzinfo=None) + + +class FundingRate(Base): + """One funding settlement of a perpetual: `rate` is the fraction of the + notional paid at `timestamp` (positive: longs pay shorts).""" + __tablename__ = "funding_rates" + __table_args__ = (UniqueConstraint("exchange", "symbol", "timestamp", name="uq_funding_rates_ex_sym_ts"),) + + id = Column(Integer, primary_key=True, autoincrement=True) + exchange = Column(String, nullable=False, index=True) + symbol = Column(String, nullable=False, index=True) + timestamp = Column(DateTime, nullable=False, index=True) + rate = Column(Float, nullable=False) + fetched_at = Column(DateTime, nullable=True, default=_utcnow) + + +class LeverageTier(Base): + """One maintenance-margin bracket of a perpetual (`fetch_market_leverage_tiers`): + positions whose size lies in [min_size, max_size) keep `mmr` of their + notional as maintenance margin. Size is the quote notional on linear + contracts and the contract count on inverse ones (Binance/OKX convention). + Refreshed by the bot startup at most once a week.""" + __tablename__ = "leverage_tiers" + __table_args__ = (UniqueConstraint("exchange", "symbol", "tier", name="uq_leverage_tiers_ex_sym_tier"),) + + id = Column(Integer, primary_key=True, autoincrement=True) + exchange = Column(String, nullable=False, index=True) + symbol = Column(String, nullable=False, index=True) + tier = Column(Integer, nullable=False) + min_size = Column(Float, nullable=False, default=0.0) + max_size = Column(Float, nullable=True) # None: open-ended top bracket + mmr = Column(Float, nullable=False) + max_leverage = Column(Float, nullable=True) + fetched_at = Column(DateTime, nullable=False, default=_utcnow) diff --git a/backend/models/orders.py b/backend/models/orders.py index 9e4e0ec..ded9de7 100644 --- a/backend/models/orders.py +++ b/backend/models/orders.py @@ -22,9 +22,13 @@ class Order(Base): price = Column(Float) amount = Column(Float) - fee = Column(Float, nullable=True) + fee = Column(Float, nullable=True) # in the position's cash currency + fee_currency = Column(String, nullable=True) # what the exchange charged it in (live), when known + fee_cash = Column(Float, nullable=True) # the raw charged amount in `fee_currency` status = Column(String, default="open") # "open", "filled", "canceled", "rejected" + market_type = Column(String, default="spot") # "spot" | "swap" + reduce_only = Column(Integer, default=0) # 1 for derivative close orders timestamp = Column(DateTime, default=lambda: datetime.now(timezone.utc)) diff --git a/backend/models/positions.py b/backend/models/positions.py index 938e7c4..c468e55 100644 --- a/backend/models/positions.py +++ b/backend/models/positions.py @@ -27,6 +27,23 @@ class Position(Base): closed_at = Column(DateTime, nullable=True) highest_price = Column(Float, nullable=True) + # Derivatives: "swap" positions carry leverage (margin = entry x amount / + # leverage) and the ccxt contract count behind `amount` (base units) + market_type = Column(String, default="spot") + leverage = Column(Float, default=1.0) + contracts = Column(Float, nullable=True) + # Currency-aware books (sprint D): the currency `profit_abs` and the + # margin are in, and how to read `amount` back (base units on + # spot/linear, contracts of `contract_size` quote units on inverse) + cash_currency = Column(String, nullable=True) + contract_kind = Column(String, nullable=True) # "spot" | "linear" | "inverse" + contract_size = Column(Float, nullable=True) + # Funding (v2.3): sum of the simulated funding payments booked on this + # position (cash currency, negative = paid; already inside profit_abs) + # and the timestamp of the last settlement applied, so every funding + # event is charged exactly once (backtest and forward test only) + funding_paid = Column(Float, nullable=True) + funding_until = Column(DateTime, nullable=True) triggered_exits = Column(JSON, nullable=True, default=list) orders = relationship("Order", back_populates="position", cascade="all, delete-orphan") diff --git a/backend/routers/bots.py b/backend/routers/bots.py index db58f5a..8d479e1 100644 --- a/backend/routers/bots.py +++ b/backend/routers/bots.py @@ -21,8 +21,9 @@ from backend.engine.settings_validator import validate_bot_settings from backend.engine.bot_manager import bot_manager, _config_fingerprint from backend.engine.sizing import backtest_pin +from backend.engine.symbols import cash_currency as _cash_currency_of from backend.engine.data_verify import verify_window -from backend.core.exchange_registry import build_exchange +from backend.core.exchange_registry import build_exchange_for_symbol from backend.core import bot_log_buffer as blb from backend.models.bot_logs import BotLog from backend.models.bot_config_runs import BotConfigRun @@ -39,6 +40,17 @@ def _resolve_exchange(settings: dict, db: Session) -> str: return settings.get("data_exchange", "okx") +def _resolve_key_market_type(settings: dict, db: Session) -> str | None: + """Market type of the linked API key (None when no key is linked); a key + is bound to one market, so the validator refuses a bot on another.""" + api_key_name = settings.get("api_key_name") + if api_key_name: + key = db.query(ExchangeKey).filter(ExchangeKey.name == api_key_name).first() + if key: + return getattr(key, "market_type", None) or "spot" + return None + + def _candle_exchange(settings: dict, key_exchange_by_name: dict) -> str: """The exchange this bot's candles (and therefore its signals) come from — the same rule as engine startup: an API key that actually routes orders @@ -113,6 +125,22 @@ def _execution_mode(settings: dict, sandbox_by_key: dict) -> str: return "forward_test" +def _bot_cash_currency(settings) -> str | None: + """Cash currency of a bot from its whitelist (the persisted backtest + summary's `cash_currency` wins when present: it was computed with the + exchange's market data).""" + if not settings: + return None + summary = settings.get("last_backtest_summary") or {} + if summary.get("cash_currency"): + return summary["cash_currency"] + for sym in list(settings.get("symbols") or []) + ([settings["symbol"]] if settings.get("symbol") else []): + norm = str(sym or "").replace("-", "/").upper() + if "/" in norm: + return _cash_currency_of(norm) + return None + + @router.get("/summary") def get_bots_summary(db: Session = Depends(get_db)): """Lightweight bot list for polling — excludes full settings/node graph.""" @@ -138,6 +166,9 @@ def get_bots_summary(db: Session = Depends(get_db)): "api_key_name": b.settings.get("api_key_name") if b.settings else None, "backtest_on_start": b.settings.get("backtest_on_start", False) if b.settings else False, "backtest_capital": b.settings.get("backtest_capital", 1000) if b.settings else 1000, + # Unit `backtest_capital`, PnL and the pool are in: the quote + # of the whitelist (settle on swaps) — never a hard-coded USD + "cash_currency": _bot_cash_currency(b.settings), "backtest_from": b.settings.get("backtest_from") if b.settings else None, "backtest_to": b.settings.get("backtest_to") if b.settings else None, # Needed by chart-open and the Data Vault live-guard: the @@ -150,6 +181,9 @@ def get_bots_summary(db: Session = Depends(get_db)): # Sizing fields for the Analytics capital-allocation panel "max_positions": b.settings.get("max_positions", 1) if b.settings else 1, "max_order_value": b.settings.get("max_order_value") if b.settings else None, + # Phase 2: swap bots carry leverage (chips on card/Analytics/Home) + "market_type": (b.settings.get("market_type") or "spot") if b.settings else "spot", + "leverage": b.settings.get("leverage", 1) if b.settings else 1, "live_allocation_pct": b.settings.get("live_allocation_pct", 100) if b.settings else 100, "live_starting_capital": b.settings.get("live_starting_capital") if b.settings else None, "entry_amount_type": (b.settings.get("trade_settings") or {}).get("entry", {}).get("amount_type", "percentage") if b.settings else "percentage", @@ -218,7 +252,7 @@ def create_bot(bot_in: BotCreate, db: Session = Depends(get_db)): raise HTTPException(status_code=400, detail="A bot with this name already exists.") resolved_exchange = _resolve_exchange(bot_in.settings, db) - validation = validate_bot_settings(bot_in.settings, exchange_id=resolved_exchange) + validation = validate_bot_settings(bot_in.settings, exchange_id=resolved_exchange, key_market_type=_resolve_key_market_type(bot_in.settings, db)) if validation["errors"]: raise HTTPException(status_code=400, detail={"validation_errors": validation["errors"]}) @@ -264,7 +298,7 @@ def import_bot(payload: dict = Body(...), db: Session = Depends(get_db)): if not isinstance(bot_settings, dict): raise HTTPException(status_code=400, detail="Invalid file: bot settings must be an object.") resolved_exchange = _resolve_exchange(bot_settings, db) - validation = validate_bot_settings(bot_settings, exchange_id=resolved_exchange) + validation = validate_bot_settings(bot_settings, exchange_id=resolved_exchange, key_market_type=_resolve_key_market_type(bot_settings, db)) if validation["errors"]: raise HTTPException(status_code=400, detail={"validation_errors": validation["errors"]}) @@ -322,7 +356,7 @@ def update_bot(bot_id: int, background_tasks: BackgroundTasks, bot_data: dict = current_settings[key] = value resolved_exchange = _resolve_exchange(current_settings, db) - validation = validate_bot_settings(current_settings, exchange_id=resolved_exchange) + validation = validate_bot_settings(current_settings, exchange_id=resolved_exchange, key_market_type=_resolve_key_market_type(current_settings, db)) if validation["errors"]: raise HTTPException(status_code=400, detail={"validation_errors": validation["errors"]}) validation_warnings = validation["warnings"] @@ -663,9 +697,8 @@ async def verify_bot_data(bot_id: int, accept: bool = Query(default=False), db: def _run(): _db = SessionLocal() try: - exch = build_exchange(exchange_id) - return [verify_window(_db, exch, exchange_id, str(s).replace('-', '/').upper(), timeframe, start, end, accept=accept) - for s in symbols] + return [verify_window(_db, build_exchange_for_symbol(exchange_id, sym), exchange_id, sym, timeframe, start, end, accept=accept) + for sym in (str(s).replace('-', '/').upper() for s in symbols)] finally: _db.close() try: diff --git a/backend/routers/data.py b/backend/routers/data.py index b2c4eb3..9cc24fa 100644 --- a/backend/routers/data.py +++ b/backend/routers/data.py @@ -12,7 +12,7 @@ from backend.core.database import get_db, SessionLocal from backend.models.candles import Candle from backend.core.security import verify_api_key -from backend.core.exchange_registry import build_exchange, SUPPORTED_EXCHANGES, get_exchange_timeframes, get_exchange_symbols +from backend.core.exchange_registry import build_exchange_for_symbol, SUPPORTED_EXCHANGES, get_exchange_timeframes, get_exchange_markets, market_caps from backend.engine.data_verify import verify_window logger = logging.getLogger("apexalgo.data") @@ -39,15 +39,23 @@ async def get_timeframes(exchange_id: str): @router.get("/symbols/{exchange_id}") -async def get_symbols(exchange_id: str): - """Tradeable spot symbols on an exchange — the builder validates the - whitelist against this. ``symbols`` is empty (and ``known`` false) when the - exchange could not be reached, so callers do not reject every pair.""" +async def get_symbols(exchange_id: str, market_type: str = Query(default="spot")): + """Tradeable symbols on an exchange (spot pairs, or linear *and* inverse + perpetuals with ``?market_type=swap``) — the builder validates the + whitelist against this. ``markets`` carries per symbol + ``{kind, base, quote, settle, contract_size, cash_currency}`` so the UI + can show what unit a pair trades in. ``symbols`` is empty (and ``known`` + false) when the exchange could not be reached, so callers do not reject + every pair.""" exchange_id = exchange_id.lower() + market_type = (market_type or "spot").lower() if exchange_id not in SUPPORTED_EXCHANGES: raise HTTPException(status_code=400, detail=f"Unknown exchange '{exchange_id}'.") - symbols = await asyncio.to_thread(get_exchange_symbols, exchange_id) - return {"exchange": exchange_id, "symbols": symbols, "known": bool(symbols)} + if market_caps(exchange_id, market_type) is None: + raise HTTPException(status_code=400, detail=f"{SUPPORTED_EXCHANGES[exchange_id]} has no '{market_type}' market in ApexAlgo.") + markets = await asyncio.to_thread(get_exchange_markets, exchange_id, market_type) + symbols = sorted(markets) + return {"exchange": exchange_id, "market_type": market_type, "symbols": symbols, "known": bool(symbols), "markets": markets} class HistoricalDataFetch(BaseModel): @@ -82,7 +90,7 @@ async def verify_stored_candles(req: VerifyRequest): def _run(): db = SessionLocal() try: - return verify_window(db, build_exchange(exchange_id), exchange_id, symbol, req.timeframe, + return verify_window(db, build_exchange_for_symbol(exchange_id, symbol), exchange_id, symbol, req.timeframe, req.start_date, req.end_date, accept=req.accept) finally: db.close() @@ -127,7 +135,7 @@ async def fetch_historical_data( def _fetch_and_save_data(formatted_symbol: str, exchange_id: str, req: HistoricalDataFetch): - exch = build_exchange(exchange_id) + exch = build_exchange_for_symbol(exchange_id, formatted_symbol) start_ts = int(req.start_date.timestamp() * 1000) end_ts = int(req.end_date.timestamp() * 1000) @@ -368,7 +376,7 @@ def get_market_info(symbol: str, exchange: str = "okx"): if cached and (now - cached[0]) < _TICKER_TTL: return cached[1] - exch = build_exchange(exchange_id) + exch = build_exchange_for_symbol(exchange_id, formatted_symbol) ticker = exch.fetch_ticker(formatted_symbol) result = { "symbol": formatted_symbol, diff --git a/backend/routers/keys.py b/backend/routers/keys.py index 58cfdd6..94d3a55 100644 --- a/backend/routers/keys.py +++ b/backend/routers/keys.py @@ -14,7 +14,12 @@ from backend.models.bots import BotConfig from backend.core.security import verify_api_key from backend.core.encryption import encrypt_data -from backend.core.exchange_registry import build_exchange, get_authenticated_exchange, invalidate_authenticated_exchange, SUPPORTED_EXCHANGES +from backend.core.exchange_registry import ( + build_exchange, get_authenticated_exchange, invalidate_authenticated_exchange, + exchange_has_sandbox, EXCHANGES, SUPPORTED_EXCHANGES, market_caps, key_market_type, +) +from backend.engine.symbols import DEFAULT_MARKET_TYPE, MARKET_TYPES +from backend.engine.broker import invalidate_leverage_cache logger = logging.getLogger("apexalgo.keys") @@ -31,39 +36,34 @@ class ExchangeKeyCreate(BaseModel): api_secret: str passphrase: str = "" is_sandbox: bool = True - - -# Where to create API keys + what the user must switch off. Shown in the UI -# next to the form so nobody has to leave the app to figure this out. -_EXCHANGE_GUIDE = { - "okx": {"keys_url": "https://www.okx.com/account/my-api", "sandbox_note": "Demo trading keys are created under Trade → Demo trading → API."}, - "binance": {"keys_url": "https://www.binance.com/en/my/settings/api-management", "sandbox_note": "Spot testnet keys come from testnet.binance.vision (separate account)."}, - "bitvavo": {"keys_url": "https://account.bitvavo.com/user/api", "sandbox_note": None}, - "coinbase": {"keys_url": "https://www.coinbase.com/settings/api", "sandbox_note": None}, - "cryptocom": {"keys_url": "https://crypto.com/exchange/user/settings/api-management", "sandbox_note": "UAT sandbox keys are issued via the Crypto.com Exchange UAT environment."}, - "kraken": {"keys_url": "https://www.kraken.com/u/security/api", "sandbox_note": None}, - "kucoin": {"keys_url": "https://www.kucoin.com/account/api", "sandbox_note": None}, -} + # A key is bound to one market: spot keys and swap keys are separate records + market_type: str = DEFAULT_MARKET_TYPE @router.get("/exchanges") def list_exchanges(): - """Static capabilities per supported exchange for the connection form.""" + """Capabilities per supported exchange (from the registry spec) for the + connection form, the data manager and the builder. `has_sandbox` is + probed from ccxt so it tracks the installed version. `markets` lists the + market types ApexAlgo can trade on the exchange with their limits.""" out = [] - for ex_id, name in SUPPORTED_EXCHANGES.items(): - has_sandbox = False - try: - has_sandbox = bool(getattr(ccxt, ex_id)().urls.get("test")) - except Exception: - pass - guide = _EXCHANGE_GUIDE.get(ex_id, {}) + for ex_id, spec in EXCHANGES.items(): out.append({ "id": ex_id, - "name": name, - "needs_passphrase": ex_id in ("okx", "kucoin"), - "has_sandbox": has_sandbox, - "keys_url": guide.get("keys_url"), - "sandbox_note": guide.get("sandbox_note"), + "name": spec.name, + "needs_passphrase": spec.needs_passphrase, + "has_sandbox": exchange_has_sandbox(ex_id), + "keys_url": spec.keys_url, + "sandbox_note": spec.sandbox_note, + "markets": { + mt: { + "has_sandbox": exchange_has_sandbox(ex_id, mt), + "max_leverage": caps.max_leverage, + "leverage_in_order": caps.leverage_in_order, + "note": caps.note, + } + for mt, caps in spec.markets.items() + }, }) return out @@ -73,6 +73,19 @@ def save_exchange_keys(req: ExchangeKeyCreate, db: Session = Depends(get_db)): exchange_id = req.exchange.lower() if exchange_id not in SUPPORTED_EXCHANGES: raise HTTPException(status_code=400, detail=f"Unsupported exchange '{exchange_id}'.") + market_type = (req.market_type or DEFAULT_MARKET_TYPE).strip().lower() + if market_type not in MARKET_TYPES: + raise HTTPException(status_code=400, detail=f"Invalid market_type '{req.market_type}'. Use one of: {', '.join(MARKET_TYPES)}.") + if market_caps(exchange_id, market_type) is None: + raise HTTPException(status_code=400, detail=f"{SUPPORTED_EXCHANGES[exchange_id]} has no '{market_type}' market in ApexAlgo.") + # Bots keep trading the market their key was verified on: re-saving a + # key under another market type is refused while bots still reference it + existing = db.query(ExchangeKey).filter(ExchangeKey.name == req.name).first() + if existing and (getattr(existing, "market_type", None) or DEFAULT_MARKET_TYPE) != market_type: + linked = [b_name for b_name, b_settings in db.query(BotConfig.name, BotConfig.settings).all() + if (b_settings or {}).get("api_key_name") == req.name] + if linked: + raise HTTPException(status_code=409, detail=f"Key '{req.name}' is a {existing.market_type or DEFAULT_MARKET_TYPE} key linked to {', '.join(linked)}; save the {market_type} key under a new name.") try: test_exchange = build_exchange( exchange_id, @@ -80,6 +93,7 @@ def save_exchange_keys(req: ExchangeKeyCreate, db: Session = Depends(get_db)): api_secret=req.api_secret, passphrase=req.passphrase or None, sandbox=req.is_sandbox, + market_type=market_type, ) test_exchange.fetch_balance() except Exception as e: @@ -91,14 +105,13 @@ def save_exchange_keys(req: ExchangeKeyCreate, db: Session = Depends(get_db)): enc_secret = encrypt_data(req.api_secret) enc_passphrase = encrypt_data(req.passphrase) - existing = db.query(ExchangeKey).filter(ExchangeKey.name == req.name).first() - if existing: existing.api_key = enc_key existing.api_secret = enc_secret existing.passphrase = enc_passphrase existing.is_sandbox = req.is_sandbox existing.exchange = req.exchange + existing.market_type = market_type else: new_key = ExchangeKey( name=req.name, @@ -106,12 +119,14 @@ def save_exchange_keys(req: ExchangeKeyCreate, db: Session = Depends(get_db)): api_key=enc_key, api_secret=enc_secret, passphrase=enc_passphrase, - is_sandbox=req.is_sandbox + is_sandbox=req.is_sandbox, + market_type=market_type, ) db.add(new_key) db.commit() invalidate_authenticated_exchange(req.name) # replaced credentials must not linger in the registry + invalidate_leverage_cache(req.name) # leverage/margin mode must be confirmed again on the new key return {"message": f"Exchange key '{req.name}' verified and saved securely."} except Exception as e: logger.error("Database error saving key '%s': %s", req.name, e) @@ -163,6 +178,7 @@ def get_exchange_keys_status(db: Session = Depends(get_db)): "name": k.name, "exchange": k.exchange, "is_sandbox": k.is_sandbox, + "market_type": getattr(k, "market_type", None) or DEFAULT_MARKET_TYPE, "is_active": is_active, "error_msg": error_msg, "latency_ms": latency_ms, @@ -193,12 +209,17 @@ def get_key_balance(key_name: str, db: Session = Depends(get_db)): } # Best-effort USD valuation so the wallet shows one total. Stables - # count as 1; everything else is priced via a direct USD-quoted market. + # count as 1; everything else is priced via a direct USD-quoted spot + # market. A swap key's futures class lists no spot pairs, so the + # prices come from a public spot instance of the same exchange. total_usd = 0.0 unpriced = [] try: stables = {"USDT", "USDC", "USD", "DAI", "TUSD", "FDUSD", "BUSD", "PYUSD"} - exchange.load_markets() + pricer = exchange + if key_market_type(key_record) != "spot": + pricer = build_exchange(key_record.exchange, sandbox=False, market_type="spot") + pricer.load_markets() wanted = {} for coin, data in active_balances.items(): if coin in stables: @@ -206,19 +227,19 @@ def get_key_balance(key_name: str, db: Session = Depends(get_db)): continue for quote in ("USDT", "USD", "USDC"): sym = f"{coin}/{quote}" - if sym in exchange.markets: + if sym in pricer.markets: wanted[sym] = coin break else: unpriced.append(coin) if wanted: try: - tickers = exchange.fetch_tickers(list(wanted.keys())) + tickers = pricer.fetch_tickers(list(wanted.keys())) except Exception: tickers = {} for sym in wanted: try: - tickers[sym] = exchange.fetch_ticker(sym) + tickers[sym] = pricer.fetch_ticker(sym) except Exception: pass for sym, coin in wanted.items(): @@ -231,7 +252,8 @@ def get_key_balance(key_name: str, db: Session = Depends(get_db)): except Exception as exc: logger.debug("USD valuation skipped for '%s': %s", key_name, type(exc).__name__) - return {"name": key_name, "balances": active_balances, "total_usd": round(total_usd, 2), "unpriced": sorted(set(unpriced))} + return {"name": key_name, "balances": active_balances, "total_usd": round(total_usd, 2), + "valuation_currency": "USD", "unpriced": sorted(set(unpriced))} except Exception as e: logger.warning("Failed to fetch balance for '%s': %s", key_name, type(e).__name__) raise HTTPException(status_code=400, detail="Failed to fetch balance from exchange.") @@ -260,6 +282,7 @@ def delete_exchange_keys(key_name: str, db: Session = Depends(get_db)): db.delete(key_record) db.commit() invalidate_authenticated_exchange(key_name) + invalidate_leverage_cache(key_name) return {"message": f"Key '{key_name}' deleted successfully."} SWAP_MAX_NOTIONAL = float(os.environ.get("SWAP_MAX_NOTIONAL", "5000")) # in the market's quote currency @@ -304,6 +327,8 @@ def execute_quick_swap(name: str, payload: SwapRequest, db: Session = Depends(ge return JSONResponse(status_code=404, content={"detail": f"API Wallet '{name}' not found"}) exchange = get_authenticated_exchange(key_record) + ex_id = str(key_record.exchange or "").lower() + ex_name = SUPPORTED_EXCHANGES.get(ex_id, ex_id) from_asset = payload.from_asset to_asset = payload.to_asset @@ -349,7 +374,7 @@ def execute_quick_swap(name: str, payload: SwapRequest, db: Session = Depends(ge fetched_order = exchange.fetch_order(order['id'], order['symbol']) if fetched_order['status'] == 'canceled': - return JSONResponse(status_code=400, content={"detail": f"OKX canceled the order. Reason: Zero liquidity for {order['symbol']} on the Testnet."}) + return JSONResponse(status_code=400, content={"detail": f"{ex_name} canceled the order. Reason: Zero liquidity for {order['symbol']} on the testnet."}) if fetched_order['status'] == 'open': exchange.cancel_order(order['id'], order['symbol']) return JSONResponse(status_code=400, content={"detail": f"Order stuck. No volume for {order['symbol']} on the Sandbox. Order auto-canceled to prevent stuck balance."}) @@ -358,7 +383,7 @@ def execute_quick_swap(name: str, payload: SwapRequest, db: Session = Depends(ge except ccxt.ExchangeError as e: error_msg = str(e) - if "51155" in error_msg or "compliance" in error_msg.lower(): + if ex_id == "okx" and ("51155" in error_msg or "compliance" in error_msg.lower()): return JSONResponse(status_code=400, content={"detail": "European Compliance Error (MiCA): You cannot trade USDT on OKX in Europe. Please swap to USDC or EUR instead."}) return JSONResponse(status_code=400, content={"detail": "Exchange rejected the order. Please check your assets and try again."}) except ccxt.InsufficientFunds: diff --git a/backend/routers/trades.py b/backend/routers/trades.py index 05935be..148df23 100644 --- a/backend/routers/trades.py +++ b/backend/routers/trades.py @@ -18,6 +18,11 @@ from backend.core.security import verify_api_key from backend.core.exchange_registry import get_authenticated_exchange from backend.engine.bot_manager import bot_manager +from backend.engine.symbols import cash_currency, is_derivative +from backend.engine import pnl +from backend.engine.contracts import spec_from_symbol +from backend.engine.sizing import position_spec +from backend.core.exchange_registry import market_caps logger = logging.getLogger("apexalgo.trades") @@ -85,7 +90,8 @@ def get_positions( Position.id, Position.exchange, Position.bot_name, Position.symbol, Position.mode, Position.status, Position.side, Position.entry_price, Position.amount, Position.profit_abs, Position.profit_pct, - Position.created_at, Position.closed_at + Position.created_at, Position.closed_at, Position.market_type, Position.leverage, + Position.cash_currency, Position.contract_kind, Position.contract_size ) if symbol: formatted_symbol = symbol.replace('-', '/').upper() @@ -110,7 +116,13 @@ def get_positions( "mode": r[4], "status": r[5], "side": r[6], "entry_price": r[7], "amount": r[8], "profit_abs": r[9], "profit_pct": r[10], "created_at": r[11].isoformat() if r[11] else None, - "closed_at": r[12].isoformat() if r[12] else None} + "closed_at": r[12].isoformat() if r[12] else None, + "market_type": r[13] or "spot", "leverage": r[14] or 1, + # Money unit of entry/PnL (quote on spot, settle on swaps) and how + # `amount` reads: base units (spot/linear) or contracts (inverse) + "cash_currency": r[15] or cash_currency(r[3]), + "contract_kind": r[16] or spec_from_symbol(r[3]).kind, + "contract_size": r[17] if r[17] is not None else spec_from_symbol(r[3]).contract_size} for r in query.all() ] @@ -127,8 +139,10 @@ def get_orders( query = db.query( Order.id, Order.position_id, Order.exchange, Order.bot_name, Order.mode, Order.symbol, Order.side, Order.order_type, - Order.price, Order.amount, Order.fee, Order.status, Order.timestamp - ) + Order.price, Order.amount, Order.fee, Order.status, Order.timestamp, + Order.market_type, Order.reduce_only, Order.fee_currency, Order.fee_cash, + Position.cash_currency, Position.contract_kind + ).outerjoin(Position, Position.id == Order.position_id) if symbol: formatted_symbol = symbol.replace('-', '/').upper() query = query.filter(Order.symbol == formatted_symbol) @@ -142,7 +156,14 @@ def get_orders( {"id": r[0], "position_id": r[1], "exchange": r[2], "bot_name": r[3], "mode": r[4], "symbol": r[5], "side": r[6], "order_type": r[7], "price": r[8], "amount": r[9], "fee": r[10], "status": r[11], - "timestamp": r[12].isoformat() if r[12] else None} + "timestamp": r[12].isoformat() if r[12] else None, + "market_type": r[13] or "spot", "reduce_only": int(r[14] or 0), + # `fee` is always in `cash_currency`; `fee_currency`/`fee_cash` are + # what the exchange actually charged (before conversion) + "cash_currency": r[17] or cash_currency(r[5]), + "contract_kind": r[18] or spec_from_symbol(r[5]).kind, + "fee_currency": r[15] or r[17] or cash_currency(r[5]), + "fee_cash": r[16]} for r in query.all() ] @@ -162,8 +183,58 @@ def get_trade_stats( closed = query.all() if not closed: - return {"netPnl": 0, "winRate": 0, "wins": 0, "losses": 0, "total": 0, "profitFactor": 0, "maxDDpct": 0, "avgHoldMs": 0, "sharpe": 0, "totalFees": 0, "avgWin": 0, "avgLoss": 0} + return {"netPnl": 0, "winRate": 0, "wins": 0, "losses": 0, "total": 0, "profitFactor": 0, "maxDDpct": 0, "avgHoldMs": 0, "sharpe": 0, "totalFees": 0, "avgWin": 0, "avgLoss": 0, + "by_currency": {}, "cash_currency": None} + + # Money never crosses currencies: the stats are computed per cash + # currency (`by_currency`), and the top-level fields carry the + # single-currency case unchanged. With several currencies in view the + # top-level PnL fields are None — the caller must use `by_currency`. + by_ccy: dict = {} + for p in closed: + by_ccy.setdefault(p.cash_currency or cash_currency(p.symbol), []).append(p) + by_currency = {ccy: _stats_for(db, rows) for ccy, rows in sorted(by_ccy.items())} + if len(by_currency) == 1: + ccy, stats = next(iter(by_currency.items())) + return {**stats, "by_currency": by_currency, "cash_currency": ccy} + money_fields = ("netPnl", "profitFactor", "maxDDpct", "totalFees", "avgWin", "avgLoss") + counts = { + "wins": sum(s["wins"] for s in by_currency.values()), + "losses": sum(s["losses"] for s in by_currency.values()), + "total": sum(s["total"] for s in by_currency.values()), + } + counts["winRate"] = counts["wins"] / counts["total"] * 100 if counts["total"] else 0 + hold = [s["avgHoldMs"] * s["total"] for s in by_currency.values()] + counts["avgHoldMs"] = sum(hold) / counts["total"] if counts["total"] else 0 + counts["sharpe"] = _sharpe(closed) + counts["bySide"] = _by_side(closed) + return {**{f: None for f in money_fields}, **counts, "by_currency": by_currency, "cash_currency": None} + + +def _sharpe(closed): + # Simplified Sharpe on per-trade returns (percentages are unit-free) + returns = [p.profit_pct or 0 for p in closed] + mean_ret = sum(returns) / len(returns) if returns else 0 + variance = sum((r - mean_ret) ** 2 for r in returns) / (len(returns) - 1) if len(returns) > 1 else 0 + stddev = math.sqrt(variance) if variance > 0 else 0 + return mean_ret / stddev if stddev > 0 else 0 + +def _by_side(closed): + # Per-side breakdown (shorts exist on perpetual markets only) + return { + side: { + "total": len(rows), + "wins": sum(1 for p in rows if (p.profit_abs or 0) > 0), + "netPnl": sum(p.profit_abs or 0 for p in rows), + } + for side, rows in (("long", [p for p in closed if p.side != "short"]), ("short", [p for p in closed if p.side == "short"])) + if rows + } + + +def _stats_for(db: Session, closed): + """Trade statistics of closed positions that share one cash currency.""" wins = [p for p in closed if (p.profit_abs or 0) > 0] losses = [p for p in closed if (p.profit_abs or 0) <= 0] gross_profit = sum(p.profit_abs or 0 for p in wins) @@ -182,7 +253,9 @@ def get_trade_stats( bc = db.query(_BC.settings).filter(_BC.name == bn).first() if bc and bc[0]: bot_capitals.append(float(bc[0].get("backtest_capital", 1000))) - starting_capital = max(bot_capitals) if bot_capitals else 1000.0 + # Capital basis = the sum of the distinct bots' pools in view (each bot + # trades its own pool; the curve below is their combined realized PnL) + starting_capital = sum(bot_capitals) if bot_capitals else 1000.0 equity = starting_capital peak_eq = starting_capital @@ -200,12 +273,7 @@ def get_trade_stats( if diff > 0: hold_times.append(diff) avg_hold_ms = sum(hold_times) / len(hold_times) if hold_times else 0 - # Simplified Sharpe - returns = [p.profit_pct or 0 for p in closed] - mean_ret = sum(returns) / len(returns) if returns else 0 - variance = sum((r - mean_ret) ** 2 for r in returns) / (len(returns) - 1) if len(returns) > 1 else 0 - stddev = math.sqrt(variance) if variance > 0 else 0 - sharpe = mean_ret / stddev if stddev > 0 else 0 + sharpe = _sharpe(closed) # Total fees pos_ids = [p.id for p in closed] @@ -227,6 +295,7 @@ def get_trade_stats( "totalFees": total_fees, "avgWin": gross_profit / len(wins) if wins else 0, "avgLoss": gross_loss / len(losses) if losses else 0, + "bySide": _by_side(closed), } @@ -306,14 +375,28 @@ def _execute_live_close(pos: Position, db: Session): raise HTTPException(status_code=400, detail=f"API key '{key_name}' no longer exists; cannot close a live position on the exchange.") ccxt_symbol = pos.symbol.replace('-', '/').upper() - close_side = "sell" if pos.side == "long" else "buy" + close_side = pnl.close_order_side(pos.side or "long") # legacy rows without a side are longs + derivative = is_derivative(ccxt_symbol) try: - exchange = get_authenticated_exchange(key_record) - close_qty = float(exchange.amount_to_precision(ccxt_symbol, pos.amount)) + exchange = get_authenticated_exchange(key_record, symbol=ccxt_symbol) + if derivative: + # Derivatives: amount in contracts, reduce-only so a stale record + # can never open the opposite position + close_qty = float(exchange.amount_to_precision(ccxt_symbol, bot_manager._to_contracts(exchange, ccxt_symbol, pos.amount))) + else: + close_qty = float(exchange.amount_to_precision(ccxt_symbol, pos.amount)) if close_qty <= 0: raise HTTPException(status_code=400, detail="Position amount rounds to zero at exchange precision; cannot place a close order.") - exch_order = exchange.create_order(ccxt_symbol, "market", close_side, close_qty) + if derivative: + params = {"reduceOnly": True} + caps = market_caps(key_record.exchange, "swap") + if caps is not None and caps.leverage_in_order: + # Same as the engine's entry/exit path (kucoinfutures wants it per order) + params["leverage"] = int(float(pos.leverage or 1)) + exch_order = exchange.create_order(ccxt_symbol, "market", close_side, close_qty, None, params) + else: + exch_order = exchange.create_order(ccxt_symbol, "market", close_side, close_qty) except HTTPException: raise except Exception as e: @@ -326,6 +409,8 @@ def _execute_live_close(pos: Position, db: Session): order_id = exch_order.get("id") order_status = exch_order.get("status") filled_amount = float(exch_order.get("filled") or 0) + if derivative and filled_amount > 0: + filled_amount = bot_manager._from_contracts(exchange, ccxt_symbol, filled_amount) if filled_amount <= 0: if order_status not in ("closed", "canceled", "rejected", "expired") and order_id: @@ -353,8 +438,8 @@ def _execute_live_close(pos: Position, db: Session): latest_candle = db.query(Candle).filter(Candle.symbol == pos.symbol, Candle.exchange == (pos.exchange or "okx")).order_by(Candle.timestamp.desc()).first() fill_price = latest_candle.close if latest_candle else pos.entry_price - exit_fee = bot_manager._fee_in_quote(exch_order.get("fee"), ccxt_symbol, fill_price) - return float(fill_price), filled_amount, exit_fee, order_id + exit_fee = bot_manager._fee_cash(exch_order.get("fee"), ccxt_symbol, fill_price, exchange) + return float(fill_price), filled_amount, exit_fee, order_id, exch_order.get("fee") def _record_unfilled_close_order(db: Session, pos: Position, close_side: str, order_id, order_status): @@ -379,6 +464,8 @@ def _record_unfilled_close_order(db: Session, pos: Position, close_side: str, or timestamp=datetime.now(timezone.utc), exchange_order_id=order_id, status=order_status or "unknown", + market_type=pos.market_type or "spot", + reduce_only=1 if is_derivative(pos.symbol) else 0, )) db.commit() except Exception as exc: @@ -391,6 +478,14 @@ def close_position_now(pos: Position, db: Session) -> float: paper = real orders on the exchange sandbox) are closed on the exchange; forward_test at the last candle close. Raises HTTPException on failure and leaves the position open. Shared by the force-close route and bot deletion.""" + # Serialize with the engine's tick for this bot: the open→closing update + # below guards the record, the lock guards the exchange (no second sell + # while a tick is mid-order on the same position) + with bot_manager._bot_locks[pos.bot_name]: + return _close_position_locked(pos, db) + + +def _close_position_locked(pos: Position, db: Session) -> float: # Atomically mark as closing to prevent a double-close race with the engine rows_updated = db.query(Position).filter( Position.id == pos.id, @@ -405,30 +500,46 @@ def close_position_now(pos: Position, db: Session) -> float: exit_fee = 0.0 exchange_order_id = None close_qty = pos.amount + spec = position_spec(pos.symbol, pos.contract_kind, pos.contract_size, pos.exchange) + fee_cols = {"fee_currency": spec.cash_currency} + pos_side = pos.side or "long" # legacy rows without a side are longs if pos.mode in ("live", "paper"): - close_price, close_qty, exit_fee, exchange_order_id = _execute_live_close(pos, db) + close_price, close_qty, exit_fee, exchange_order_id, _fee_info = _execute_live_close(pos, db) + if isinstance(_fee_info, dict) and _fee_info.get("cost") is not None: + fee_cols = {"fee_currency": str(_fee_info.get("currency") or spec.cash_currency).upper(), "fee_cash": float(_fee_info["cost"])} else: - # Simulated modes: use the most recent candle close price + # Simulated modes: the most recent candle close, with the + # backtest's exit frictions on forward-test positions so a manual + # close books like an engine exit would latest_candle = db.query(Candle).filter(Candle.symbol == pos.symbol, Candle.exchange == (pos.exchange or "okx")).order_by(Candle.timestamp.desc()).first() close_price = latest_candle.close if latest_candle else pos.entry_price + if pos.mode == "forward_test": + bot = db.query(BotConfig).filter(BotConfig.name == pos.bot_name).first() + if bot is not None: + _, exit_fee_pct, _, exit_slip = bot_manager._sim_frictions(bot.settings or {}, pos_side) + close_price = close_price * (1 + exit_slip) if pos_side == "short" else close_price * (1 - exit_slip) + exit_fee = spec.fee_cash(close_qty, close_price, exit_fee_pct) # Fee-adjusted P&L accumulated on top of earlier partial exits - filled_buys = [o for o in (pos.orders or []) if o.side == "buy" and o.status == "filled"] + # Opening orders: buys for a long, the (non-reduce-only) sells for a short + _open_side = pnl.open_order_side(pos_side) + filled_buys = [o for o in (pos.orders or []) if o.side == _open_side and o.status == "filled"] original_amount = sum((o.amount or 0.0) for o in filled_buys) or pos.amount or close_qty total_buy_fees = sum((o.fee or 0.0) for o in filled_buys) entry_fee_portion = total_buy_fees * (close_qty / original_amount) if original_amount > 0 else 0.0 - if pos.side == "long": - realized_pnl = (close_price - pos.entry_price) * close_qty - entry_fee_portion - exit_fee - else: - realized_pnl = (pos.entry_price - close_price) * close_qty - entry_fee_portion - exit_fee + realized_pnl = pnl.price_pnl(pos_side, pos.entry_price, close_price, close_qty, spec=spec) - entry_fee_portion - exit_fee pos.status = "closed" pos.closed_at = datetime.now(timezone.utc) pos.profit_abs = (pos.profit_abs or 0.0) + realized_pnl - entry_value = (pos.entry_price or 0.0) * original_amount - pos.profit_pct = (pos.profit_abs / entry_value) * 100 if entry_value > 0 else 0.0 + # profit_pct on the capital this leg had locked (margin + entry fee), + # added to what earlier partial exits already booked — same basis as + # the backtest and the engine's exits + locked = pnl.locked_capital(pos.entry_price or 0.0, close_qty, pos.leverage or 1, spec=spec) + entry_fee_portion + leg_pct = (realized_pnl / locked) * 100 if locked > 0 else 0.0 + pos.profit_pct = (pos.profit_pct or 0.0) + leg_pct * (close_qty / original_amount if original_amount > 0 else 1.0) close_order = Order( position_id=pos.id, @@ -436,14 +547,17 @@ def close_position_now(pos: Position, db: Session) -> float: bot_name=pos.bot_name, mode=pos.mode, symbol=pos.symbol, - side="sell" if pos.side == "long" else "buy", + side=pnl.close_order_side(pos_side), order_type="market", price=close_price, amount=close_qty, fee=exit_fee, timestamp=datetime.now(timezone.utc), exchange_order_id=exchange_order_id, - status="filled" + status="filled", + market_type=pos.market_type or "spot", + reduce_only=1 if is_derivative(pos.symbol) else 0, + **fee_cols, ) db.add(close_order) db.commit() @@ -469,7 +583,7 @@ def force_close_position(position_id: int, db: Session = Depends(get_db)): close_price = close_position_now(pos, db) except HTTPException as e: return JSONResponse(status_code=e.status_code, content={"detail": e.detail}) - return {"status": "success", "message": f"Position forcefully closed at ${close_price:.2f}"} + return {"status": "success", "message": f"Position forcefully closed at {close_price:.2f} {spec_from_symbol(pos.symbol).quote}"} @router.get("/export") def export_trades_csv(mode: str = "live"): diff --git a/examples/Supertrend_LongShort_Perp_1d.apex.json b/examples/Supertrend_LongShort_Perp_1d.apex.json new file mode 100644 index 0000000..432fc8a --- /dev/null +++ b/examples/Supertrend_LongShort_Perp_1d.apex.json @@ -0,0 +1,127 @@ +{ + "apex_version": "1.0", + "exported_at": "2026-09-25T12:00:00Z", + "bot": { + "name": "Supertrend Long/Short Perp 1d", + "is_sandbox": true, + "strategy": "node_evaluator", + "settings": { + "symbol": "BTC/USDT:USDT", + "symbols": [ + "BTC/USDT:USDT", + "ETH/USDT:USDT" + ], + "timeframe": "1d", + "market_type": "swap", + "leverage": 1, + "margin_mode": "isolated", + "max_positions": 2, + "max_positions_scope": "global", + "cooldown_trades": 1, + "cooldown_candles": 5, + "max_drawdown": 35, + "drawdown_action": "block_entries", + "drawdown_cooldown_days": 14, + "max_capital_loss": 40, + "max_order_value": 1000, + "live_allocation_pct": 100, + "api_execution": false, + "backtest_on_start": true, + "backtest_capital": 1000, + "backtest_lookback": 1000, + "api_key_name": null, + "data_exchange": "binance", + "trade_settings": { + "entry": { + "order_type": "market", + "amount_type": "percentage", + "amount_value": 30, + "fee": 0.05, + "slippage": 0.05, + "take_profits": [], + "stop_losses": [ + { + "type": "percentage", + "value": 20.0, + "close_amount_type": "percentage", + "close_amount_value": 100 + } + ] + }, + "exit": { + "order_type": "market", + "amount_type": "percentage", + "amount_value": 100, + "fee": 0.05, + "slippage": 0.05 + }, + "short": { + "order_type": "market", + "amount_type": "percentage", + "amount_value": 30, + "fee": 0.05, + "slippage": 0.05, + "take_profits": [], + "stop_losses": [ + { + "type": "percentage", + "value": 20.0, + "close_amount_type": "percentage", + "close_amount_value": 100 + } + ] + }, + "cover": { + "order_type": "market", + "amount_type": "percentage", + "amount_value": 100, + "fee": 0.05, + "slippage": 0.05 + } + }, + "nodes": { + "st_dir": { + "class": "indicator", + "method": "supertrend", + "params": { + "length": 10, + "multiplier": 3.0 + }, + "output_idx": 1 + }, + "trend_up": { + "class": "condition", + "left": "st_dir", + "operator": ">", + "right": 0 + }, + "trend_down": { + "class": "condition", + "left": "st_dir", + "operator": "<", + "right": 0 + }, + "flip_up": { + "class": "condition", + "left": "st_dir", + "operator": "cross_above", + "right": 0 + }, + "flip_down": { + "class": "condition", + "left": "st_dir", + "operator": "cross_below", + "right": 0 + } + }, + "ui_layout": { + "nodes": [], + "edges": [] + }, + "entry_node": "trend_up", + "exit_node": "flip_down", + "short_node": "trend_down", + "cover_node": "flip_up" + } + } +} diff --git a/frontend/package-lock.json b/frontend/package-lock.json index ae797f9..eeefb53 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "frontend", - "version": "2.2.1", + "version": "2.3.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "frontend", - "version": "2.2.1", + "version": "2.3.0", "dependencies": { "@fontsource-variable/inter": "^5.3.0", "@fontsource/jetbrains-mono": "^5.3.0", diff --git a/frontend/package.json b/frontend/package.json index 91e65a7..575a46f 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,7 +1,7 @@ { "name": "frontend", "private": true, - "version": "2.2.1", + "version": "2.3.0", "license": "AGPL-3.0-only", "type": "module", "scripts": { diff --git a/frontend/src/App.jsx b/frontend/src/App.jsx index 35b617d..e1c25e7 100644 --- a/frontend/src/App.jsx +++ b/frontend/src/App.jsx @@ -6,8 +6,9 @@ import BotManagerUI from './components/BotManagerUI'; import TradeManager from './components/TradeManager'; import Home from './components/Home'; import ApiKeyGate from './components/ApiKeyGate'; -import Toaster from './components/ui/Toast'; +import Toaster, { toast } from './components/ui/Toast'; import ConfirmDialogHost from './components/ui/ConfirmDialog'; +import { humanizeApiError } from './api/errors'; import { Skeleton } from './components/ui/Skeleton'; import { apiClient, checkSession, logout, migrateLegacyKey } from './api/client'; @@ -159,8 +160,11 @@ export default function App() { try { const res = await apiClient.get(`/api/bots/by-id/${botSummary.id}`); setEditingBot(res.data); - } catch { - setEditingBot(botSummary); + } catch (err) { + // The summary has no nodes/edges — opening it would show an + // empty canvas that "Save" could persist over the real strategy. + toast.error(`Could not load "${botSummary.name || 'bot'}": ${humanizeApiError(err)}`); + return; } } else { setEditingBot(null); diff --git a/frontend/src/DESIGN.md b/frontend/src/DESIGN.md index 0a0bed2..2729274 100644 --- a/frontend/src/DESIGN.md +++ b/frontend/src/DESIGN.md @@ -118,8 +118,13 @@ Color props are **semantic only**: `accent|info|success|danger|purple|neutral`. ```jsx - setCap(e.target.value)} - error={capError} hint="Backtest starting equity" /> + setCap(e.target.value)} + error={capError} hint="Backtest starting equity — cash currency of the whitelist" /> + +// Money: never a bare "$"/"USD". Format through utils/money.js with the currency +// the row is denominated in; sum per currency, never across. +fmtMoney(1234.5, 'USDT') // '1,234.50 USDT' fmtMoney(0.0213, 'BTC') // '0.0213 BTC' +fmtByCurrency(sumByCurrency(positions, p => p.profit_abs)) // '12.30 USDT · 0.001 BTC' Running diff --git a/frontend/src/api/exchanges.js b/frontend/src/api/exchanges.js new file mode 100644 index 0000000..c8032d7 --- /dev/null +++ b/frontend/src/api/exchanges.js @@ -0,0 +1,78 @@ +// Supported exchanges, loaded once from the backend registry +// (GET /api/keys/exchanges). backend/core/exchange_registry.py is the single +// source of truth; the static fallback below only bridges the first render +// and an unreachable backend, so an exchange added there shows up everywhere +// (key form, data manager, builder) without touching the frontend. +// +// Shape of each entry: +// { id, name, needs_passphrase, has_sandbox, keys_url, sandbox_note, +// markets: { spot: {has_sandbox, max_leverage, leverage_in_order, note}, swap?: {...} } } +import { useEffect, useSyncExternalStore } from 'react'; +import { apiClient } from './client'; + +// Market types per exchange mirror backend ExchangeSpec.markets (spot always; +// swap = perpetual swaps where ApexAlgo supports them — linear or inverse per +// symbol, see markets[symbol].kind from /api/data/symbols) +const SPOT = { spot: { has_sandbox: false, max_leverage: 1 } }; +const fb = (id, name, needs_passphrase, has_sandbox, swap) => ({ + id, name, needs_passphrase, has_sandbox, + markets: swap ? { spot: { has_sandbox, max_leverage: 1 }, swap: { has_sandbox: swap.has_sandbox, max_leverage: swap.max_leverage || 10 } } : { spot: { ...SPOT.spot, has_sandbox } }, +}); +export const FALLBACK_EXCHANGES = [ + fb('okx', 'OKX', true, true, { has_sandbox: true }), + fb('binance', 'Binance', false, true, { has_sandbox: true }), + fb('bitvavo', 'Bitvavo', false, false), + fb('coinbase', 'Coinbase', false, false), + fb('cryptocom', 'Crypto.com', false, true), + fb('kraken', 'Kraken', false, false, { has_sandbox: true, max_leverage: 5 }), + fb('kucoin', 'KuCoin', true, false, { has_sandbox: false }), + fb('bybit', 'Bybit', false, true, { has_sandbox: true }), + fb('gateio', 'Gate', false, true, { has_sandbox: true }), + fb('bitget', 'Bitget', true, true, { has_sandbox: true }), + fb('mexc', 'MEXC', false, false), + fb('htx', 'HTX', false, false, { has_sandbox: false }), + fb('bingx', 'BingX', false, true, { has_sandbox: true }), +]; + +let cache = null; +let inflight = null; +const listeners = new Set(); +const notify = () => listeners.forEach((fn) => fn()); + +/** Fetches the list once; concurrent callers share the request. Failed loads are not cached. */ +export function loadExchanges() { + if (cache) return Promise.resolve(cache); + if (!inflight) { + inflight = apiClient.get('/api/keys/exchanges') + .then((res) => { + if (Array.isArray(res.data) && res.data.length) cache = res.data; + return cache || FALLBACK_EXCHANGES; + }) + .finally(() => { inflight = null; notify(); }); + } + return inflight; +} + +const subscribe = (fn) => { listeners.add(fn); return () => listeners.delete(fn); }; +const snapshot = () => cache || FALLBACK_EXCHANGES; + +/** React hook: the exchange list (fallback until loaded, then the backend registry). */ +export function useExchanges() { + const exchanges = useSyncExternalStore(subscribe, snapshot); + useEffect(() => { if (!cache) loadExchanges().catch(() => {}); }, []); + return exchanges; +} + +/** Display name for an exchange id ('okx' → 'OKX'); falls back to the upper-cased id. */ +export function exchangeName(id) { + const hit = snapshot().find((e) => e.id === id); + return hit ? hit.name : String(id || '').toUpperCase(); +} + +/** Market capabilities of `exchangeId` for `marketType` ('spot' | 'swap'); null when unsupported. */ +export function marketCaps(exchangeId, marketType = 'spot') { + const hit = snapshot().find((e) => e.id === exchangeId); + if (!hit) return null; + const markets = hit.markets || { spot: { has_sandbox: !!hit.has_sandbox, max_leverage: 1 } }; + return markets[marketType] || null; +} diff --git a/frontend/src/components/BotManagerUI.jsx b/frontend/src/components/BotManagerUI.jsx index 4f350b1..a6cdea3 100644 --- a/frontend/src/components/BotManagerUI.jsx +++ b/frontend/src/components/BotManagerUI.jsx @@ -11,6 +11,7 @@ import EmptyState from './ui/EmptyState'; import { toast } from './ui/Toast'; import { confirmDialog } from './ui/ConfirmDialog'; import BotConsole from './BotConsole'; +import { contractKindOf, fmtMoney, symbolParts } from '../utils/money'; /* ── Inline icons (stroke 1.8) ── */ const IconEdit = ( @@ -238,6 +239,22 @@ function BacktestResult({ bot, updateBotConfig, verifyData, verifying }) { historical candles differ from the previous run on this slice

)} + {/* Row 3 (perpetuals): what the simulation could model of funding and margin */} + {sm.market_type && sm.market_type !== 'spot' && (() => { + const fundingOk = sm.funding === 'simulated'; + const fundingText = sm.funding === 'simulated' + ? `Funding ${fmtMoney(sm.funding_paid, sm.cash_currency, { sign: true })} over ${sm.funding_events} settlement${sm.funding_events === 1 ? '' : 's'}` + : sm.funding === 'partial' + ? `Funding partial: ${fmtMoney(sm.funding_paid, sm.cash_currency, { sign: true })} — not every pair had rates stored` + : 'Funding not simulated — no funding rates stored for this range'; + const mmrText = sm.mmr_source === 'tiers' ? `${exchange} margin tiers` : 'flat 0.5% maintenance margin'; + return ( + + {fundingText} · {mmrText}{sm.liquidations > 0 ? ` · ${sm.liquidations} liquidation${sm.liquidations === 1 ? '' : 's'}` : ''} + + ); + })()} ); } @@ -276,6 +293,16 @@ const BotCard = memo(function BotCard({ bot, index, busyAction, togglingBot, ope : Stopped} {isBacktestOn && + Backtest} + {bot.settings?.market_type === 'swap' && (() => { + // Inverse pairs (settle = base) keep margin and PnL in the base coin + const inverse = assignedPairs.filter(p => contractKindOf(p) === 'inverse'); + const bases = [...new Set(inverse.map(p => symbolParts(p).base).filter(Boolean))]; + return ( + + Perps {Number(bot.settings?.leverage) || 1}×{inverse.length ? ' · inverse' : ''} + + ); + })()} {/* Metrics row */} diff --git a/frontend/src/components/Builder/BotBuilder.jsx b/frontend/src/components/Builder/BotBuilder.jsx index 08fd38c..f1d6fb3 100644 --- a/frontend/src/components/Builder/BotBuilder.jsx +++ b/frontend/src/components/Builder/BotBuilder.jsx @@ -1,5 +1,5 @@ import React, { useState, useCallback, useRef, useEffect } from 'react'; -import ReactFlow, { MiniMap, Controls, Background, useNodesState, useEdgesState, addEdge, ReactFlowProvider } from 'reactflow'; +import ReactFlow, { MiniMap, Controls, Background, useNodesState, useEdgesState, useNodesInitialized, addEdge, ReactFlowProvider } from 'reactflow'; import 'reactflow/dist/style.css'; import { BotConfigNode, WhitelistNode, BacktestNode, ApiKeyNode, IndicatorNode, ConditionNode, LogicNode, StopLossNode, TakeProfitNode, ActionNode, PriceDataNode } from './CustomNodes'; import { apiClient } from '../../api/client'; @@ -10,6 +10,8 @@ import { getToken } from '../../theme'; import Button from '../ui/Button'; import { toast } from '../ui/Toast'; import { confirmDialog } from '../ui/ConfirmDialog'; +import { marketCaps } from '../../api/exchanges'; +import { whitelistCurrency } from '../../utils/money'; const nodeTypes = { botConfig: BotConfigNode, @@ -52,7 +54,7 @@ function rebuildLayoutFromSettings(settings, updateNodeData, deleteNode) { config: { w: 340, h: 1100 }, // 12 fields incl. guards + live allocation; cooldown row is conditional whitelist: { w: 320, h: 200 }, backtest: { w: 320, h: 220 }, - apiKey: { w: 320, h: 240 }, + apiKey: { w: 320, h: 340 }, // grows with market type / leverage / margin mode indicator: { w: 270, h: 200 }, // base; grows with params priceData: { w: 240, h: 190 }, condition: { w: 280, h: 200 }, @@ -105,7 +107,10 @@ function rebuildLayoutFromSettings(settings, updateNodeData, deleteNode) { nodes.push({ id: 'rebuilt_apikey', type: 'apiKey', position: { x: ctxX, y: ctxY }, data: { onChange: updateNodeData, onDelete: deleteNode, apiKeyName: settings.api_key_name || null, - dataExchange: settings.data_exchange || 'okx' } }); + dataExchange: settings.data_exchange || 'okx', + marketType: settings.market_type || 'spot', + leverage: settings.leverage ?? 1, + marginMode: settings.margin_mode || 'isolated' } }); } // ── AREA 2: Strategy logic ── @@ -122,6 +127,8 @@ function rebuildLayoutFromSettings(settings, updateNodeData, deleteNode) { } if (settings.entry_node) collectDeps(settings.entry_node); if (settings.exit_node) collectDeps(settings.exit_node); + if (settings.short_node) collectDeps(settings.short_node); + if (settings.cover_node) collectDeps(settings.cover_node); for (const nid of Object.keys(settingsNodes)) if (!visited.has(nid)) collectDeps(nid); // Column x-positions: each column starts after previous column's width + generous gap @@ -225,13 +232,102 @@ function rebuildLayoutFromSettings(settings, updateNodeData, deleteNode) { if (settings.exit_node && settingsNodes[settings.exit_node]) edges.push({ id: `e_${settings.exit_node}_exit`, source: settings.exit_node, target: exitId, targetHandle: 'logic', animated: true, style: edgeStyle }); + // ── Short leg (phase 3) — only when the bot actually has short nodes, so + // long-only bots rebuild exactly as before ── + if (settings.short_node || settings.cover_node) { + const shortTs = settings.trade_settings?.short || entryTs; + const shortId = 'rebuilt_short'; + let shortY = strategyY + 2 * (SIZE.action.h + COL_GAP); + nodes.push({ id: shortId, type: 'action', position: { x: C4_X, y: shortY }, + data: { onChange: updateNodeData, onDelete: deleteNode, actionType: 'short', + orderType: shortTs.order_type || 'market', amountType: shortTs.amount_type || 'percentage', + amountValue: shortTs.amount_value ?? 100, fee: shortTs.fee ?? 0.1, + slippage: shortTs.slippage ?? 0.05 } }); + if (settings.short_node && settingsNodes[settings.short_node]) + edges.push({ id: `e_${settings.short_node}_short`, source: settings.short_node, target: shortId, targetHandle: 'logic', animated: true, style: edgeStyle }); + let stpslY = Math.max(tpslY, shortY); + (shortTs.take_profits || []).forEach((tp, i) => { + const tpId = `rebuilt_short_tp_${i}`; + nodes.push({ id: tpId, type: 'takeProfit', position: { x: C5_X, y: stpslY }, + data: { onChange: updateNodeData, onDelete: deleteNode, + triggerType: tp.type || 'percentage', triggerValue: tp.value ?? '', + closeType: tp.close_amount_type || 'percentage', closeValue: tp.close_amount_value ?? 100 } }); + edges.push({ id: `e_short_${tpId}`, source: shortId, sourceHandle: 'tp', target: tpId, animated: true, style: edgeStyle }); + stpslY += SIZE.tp.h + GAP + 10; + }); + (shortTs.stop_losses || []).forEach((sl, i) => { + const slId = `rebuilt_short_sl_${i}`; + nodes.push({ id: slId, type: 'stopLoss', position: { x: C5_X, y: stpslY }, + data: { onChange: updateNodeData, onDelete: deleteNode, + triggerType: sl.type || 'percentage', triggerValue: sl.value ?? '', + closeType: sl.close_amount_type || 'percentage', closeValue: sl.close_amount_value ?? 100 } }); + edges.push({ id: `e_short_${slId}`, source: shortId, sourceHandle: 'sl', target: slId, animated: true, style: edgeStyle }); + stpslY += SIZE.sl.h + GAP + 10; + }); + const coverTs = settings.trade_settings?.cover || exitTs; + const coverId = 'rebuilt_cover'; + shortY += SIZE.action.h + COL_GAP; + nodes.push({ id: coverId, type: 'action', position: { x: C4_X, y: shortY }, + data: { onChange: updateNodeData, onDelete: deleteNode, actionType: 'cover', + orderType: coverTs.order_type || 'market', amountType: coverTs.amount_type || 'percentage', + amountValue: coverTs.amount_value ?? 100, fee: coverTs.fee ?? 0.1, + slippage: coverTs.slippage ?? 0.05 } }); + if (settings.cover_node && settingsNodes[settings.cover_node]) + edges.push({ id: `e_${settings.cover_node}_cover`, source: settings.cover_node, target: coverId, targetHandle: 'logic', animated: true, style: edgeStyle }); + } + return { nodes, edges }; } +// The SIZE table above is a guess; the real heights depend on content (param +// count, hint texts, leverage fields) and drift whenever a node gains a row. +// Once ReactFlow has measured the nodes, push every column apart from the +// top so no block overlaps the one above it. Only vertical shifts, only +// downwards, so the hand-made column layout stays recognisable. +const RELAYOUT_GAP = 60; +function resolveColumnOverlaps(nodes) { + const columns = new Map(); + for (const n of nodes) { + const x = Math.round(n.position.x); + if (!columns.has(x)) columns.set(x, []); + columns.get(x).push(n); + } + const shifted = new Map(); + for (const col of columns.values()) { + col.sort((a, b) => a.position.y - b.position.y); + let bottom = -Infinity; + for (const n of col) { + const h = n.height || 0; + let y = n.position.y; + if (y < bottom + RELAYOUT_GAP) y = bottom + RELAYOUT_GAP; + if (y !== n.position.y) shifted.set(n.id, y); + bottom = y + h; + } + } + if (shifted.size === 0) return null; + return nodes.map(n => shifted.has(n.id) ? { ...n, position: { ...n.position, y: shifted.get(n.id) } } : n); +} + // Stable fingerprint of what Save would persist: node identity, position and // user-editable data (runtime keys pushed in by effects are ignored), plus // edge topology. Used to decide whether closing would lose work. const TRANSIENT_DATA_KEYS = new Set(['onChange', 'onDelete', 'availableKeys', 'knownSymbols', 'liveContext', 'supportedTimeframes']); +// Builder-pushed hint on action blocks (which market the routing key trades); +// stripped like the other transients — the key/settings remain the source. +TRANSIENT_DATA_KEYS.add('liveMarketType'); +// Builder-pushed cash-currency picture of the whitelist (code, inverse pairs) +// for the capital/order-cap labels; derived from the market list, not saved. +TRANSIENT_DATA_KEYS.add('marketCtx'); +// ReactFlow bookkeeping on the node object itself (selection, measured size, +// drag state). Never persisted and never part of the dirty snapshot: a bot +// reopened from ui_layout must compare equal until the user edits something. +const TRANSIENT_NODE_KEYS = ['selected', 'width', 'height', 'dragging', 'measured', 'positionAbsolute']; +const stripNodeTransients = (n) => { + const safe = { ...n, data: { ...(n.data || {}) } }; + for (const k of TRANSIENT_NODE_KEYS) delete safe[k]; + for (const k of TRANSIENT_DATA_KEYS) delete safe.data[k]; + return safe; +}; const graphSnapshot = (nodes, edges) => JSON.stringify({ nodes: nodes.map(n => ({ id: n.id, type: n.type, @@ -255,6 +351,10 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { const [knownSymbols, setKnownSymbols] = useState(null); const initRef = useRef(false); + // Set when the layout was generated (rebuilt from settings or the new-bot + // default) rather than restored from ui_layout: fix overlaps once measured. + const relayoutRef = useRef(false); + const nodesInitialized = useNodesInitialized(); const updateNodeData = useCallback((id, field, value) => { let safeValue = value; @@ -294,9 +394,19 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { const hasLayout = editingBot && editingBot.settings.ui_layout && editingBot.settings.ui_layout.nodes && editingBot.settings.ui_layout.nodes.length > 0; if (hasLayout) { - const restoredNodes = editingBot.settings.ui_layout.nodes.map(n => ({ - ...n, data: { ...n.data, onChange: updateNodeData, onDelete: deleteNode } - })); + const restoredNodes = editingBot.settings.ui_layout.nodes.map(n => { + const node = stripNodeTransients(n); + node.data.onChange = updateNodeData; + node.data.onDelete = deleteNode; + // The persisted layout may carry a stale name (duplicate/import + // rename): the bot record is the truth. + if (node.type === 'botConfig') node.data.botName = editingBot.name; + // Same correction ApiKeyNode would apply on mount — do it here so + // the restored graph is already settled when the baseline is taken. + if (node.type === 'apiKey' && !node.data.apiKeyName && node.data.marketType === 'swap' + && !marketCaps(node.data.dataExchange || 'okx', 'swap')) node.data.marketType = 'spot'; + return node; + }); setNodes(restoredNodes); setEdges(editingBot.settings.ui_layout.edges || []); initRef.current = true; @@ -309,14 +419,16 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { setNodes(rebuilt.nodes); setEdges(rebuilt.edges); initRef.current = true; + relayoutRef.current = true; } else { setNodes([ { id: getId(), type: 'botConfig', position: { x: 50, y: 50 }, data: { onChange: updateNodeData, onDelete: deleteNode, botName: editingBot ? editingBot.name : 'Apex Strategy Alpha', timeframe: editingBot?.settings?.timeframe || '1m', executionMode: 'paper', maxPositions: 1, maxPositionsScope: 'per_pair', cooldownTrades: 0, cooldownCandles: 0 } }, { id: getId(), type: 'whitelist', position: { x: 470, y: 50 }, data: { onChange: updateNodeData, onDelete: deleteNode, pairs: editingBot?.settings?.symbols?.join(', ') || editingBot?.settings?.symbol || DEFAULT_PAIR } }, { id: getId(), type: 'backtest', position: { x: 470, y: 320 }, data: { onChange: updateNodeData, onDelete: deleteNode, runOnStart: true, capital: 1000, lookback: 150 } }, - { id: getId(), type: 'apiKey', position: { x: 470, y: 610 }, data: { onChange: updateNodeData, onDelete: deleteNode, apiKeyName: null, dataExchange: 'okx' } } + { id: getId(), type: 'apiKey', position: { x: 470, y: 610 }, data: { onChange: updateNodeData, onDelete: deleteNode, apiKeyName: null, dataExchange: 'okx', marketType: 'spot', leverage: 1, marginMode: 'isolated' } } ]); initRef.current = true; + relayoutRef.current = true; } }, [editingBot, updateNodeData, deleteNode, setNodes, setEdges]); @@ -334,6 +446,11 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { const activeApiKeyNode = nodes.find(n => n.type === 'apiKey'); const activeApiKeyName = activeApiKeyNode?.data.apiKeyName; const activeDataExchange = activeApiKeyNode?.data.dataExchange; + // Market type: the linked key decides (a key is bound to one market), + // otherwise the routing block's own choice. Swap symbols are a separate + // market list (BASE/QUOTE:SETTLE). + const activeKeyMarketType = activeApiKeyName ? (availableKeys?.find(k => k.name === activeApiKeyName)?.market_type || null) : null; + const activeMarketType = activeKeyMarketType || activeApiKeyNode?.data.marketType || 'spot'; useEffect(() => { if (!initRef.current) return; let exchange = activeDataExchange || 'okx'; @@ -345,11 +462,12 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { setSupportedTimeframes(res.data.timeframes); }).catch(() => setSupportedTimeframes(null)); let cancelled = false; - apiClient.get(`/api/data/symbols/${exchange}`).then(res => { - if (!cancelled) setKnownSymbols({ exchange, symbols: res.data.symbols || [], known: !!res.data.known }); - }).catch(() => { if (!cancelled) setKnownSymbols({ exchange, symbols: [], known: false }); }); + apiClient.get(`/api/data/symbols/${exchange}`, { params: { market_type: activeMarketType } }).then(res => { + // `markets` (per-symbol kind/settle/quote) is additive — older backends only send `symbols` + if (!cancelled) setKnownSymbols({ exchange, marketType: activeMarketType, symbols: res.data.symbols || [], known: !!res.data.known, markets: res.data.markets || {} }); + }).catch(() => { if (!cancelled) setKnownSymbols({ exchange, marketType: activeMarketType, symbols: [], known: false, markets: {} }); }); return () => { cancelled = true; }; - }, [activeApiKeyName, activeDataExchange, availableKeys]); + }, [activeApiKeyName, activeDataExchange, activeMarketType, availableKeys]); // Pass supported timeframes to config node useEffect(() => { @@ -378,16 +496,48 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { const liveKeySandbox = !!activeKeyRecord?.is_sandbox; const entryActionNode = nodes.find(n => n.type === 'action' && n.data.actionType === 'buy'); const entryFee = entryActionNode ? (entryActionNode.data.fee === '' || entryActionNode.data.fee === undefined ? 0.1 : Number(entryActionNode.data.fee)) : null; + const liveLeverage = activeMarketType === 'swap' ? (Number(activeApiKeyNode?.data.leverage) || 1) : 1; + const liveMarginMode = activeApiKeyNode?.data.marginMode || 'isolated'; + // Short leg presence for the go-live checklist: a SHORT block, and whether + // it has any way out (COVER block or a stop-loss on its SL port). + const shortActionNode = nodes.find(n => n.type === 'action' && n.data.actionType === 'short'); + const hasShort = !!shortActionNode; + const hasCover = nodes.some(n => n.type === 'action' && n.data.actionType === 'cover'); + const hasShortStop = !!shortActionNode && edges.some(e => e.source === shortActionNode.id && e.sourceHandle === 'sl'); useEffect(() => { if (!initRef.current) return; - const liveContext = { keyName: liveKeyName, exchange: liveKeyExchange, isSandbox: liveKeySandbox, entryFee }; + const liveContext = { keyName: liveKeyName, exchange: liveKeyExchange, isSandbox: liveKeySandbox, entryFee, + marketType: activeMarketType, leverage: liveLeverage, marginMode: liveMarginMode, + hasShort, hasCover, hasShortStop }; setNodes(nds => nds.map(n => { + if (n.type === 'action') { + // Action blocks only need the market type (SHORT/COVER are swap-only) + if (n.data.liveMarketType === activeMarketType) return n; + return { ...n, data: { ...n.data, liveMarketType: activeMarketType } }; + } if (n.type !== 'botConfig') return n; const cur = n.data.liveContext; - if (cur && cur.keyName === liveContext.keyName && cur.exchange === liveContext.exchange && cur.isSandbox === liveContext.isSandbox && cur.entryFee === liveContext.entryFee) return n; + if (cur && cur.keyName === liveContext.keyName && cur.exchange === liveContext.exchange && cur.isSandbox === liveContext.isSandbox && cur.entryFee === liveContext.entryFee + && cur.marketType === liveContext.marketType && cur.leverage === liveContext.leverage && cur.marginMode === liveContext.marginMode + && cur.hasShort === hasShort && cur.hasCover === hasCover && cur.hasShortStop === hasShortStop) return n; return { ...n, data: { ...n.data, liveContext } }; })); - }, [liveKeyName, liveKeyExchange, liveKeySandbox, entryFee, setNodes]); + }, [liveKeyName, liveKeyExchange, liveKeySandbox, entryFee, activeMarketType, liveLeverage, liveMarginMode, hasShort, hasCover, hasShortStop, setNodes]); + + // Cash currency of the whitelist (from the exchange's market list, else the + // symbol form) for the capital / order-cap labels and the inverse note. + // Serialised so the effect only fires when the picture actually changes. + const whitelistPairsText = nodes.find(n => n.type === 'whitelist')?.data.pairs; + const marketCtxKey = JSON.stringify(whitelistCurrency(knownSymbols?.markets, parsePairs(whitelistPairsText !== undefined ? whitelistPairsText : DEFAULT_PAIR))); + useEffect(() => { + if (!initRef.current) return; + const marketCtx = JSON.parse(marketCtxKey); + setNodes(nds => nds.map(n => { + if (n.type !== 'botConfig' && n.type !== 'backtest' && n.type !== 'apiKey' && n.type !== 'action') return n; + if (JSON.stringify(n.data.marketCtx) === marketCtxKey) return n; + return { ...n, data: { ...n.data, marketCtx } }; + })); + }, [marketCtxKey, setNodes]); // Esc asks to close (same dirty-check as the buttons); a browser reload with // unsaved work gets the native "leave page?" prompt. @@ -433,7 +583,7 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { const getDefaultData = useCallback((type) => { const defaultData = { onChange: updateNodeData, onDelete: deleteNode }; - if (type === 'apiKey') defaultData.availableKeys = availableKeys; + if (type === 'apiKey') { defaultData.availableKeys = availableKeys; defaultData.apiKeyName = null; defaultData.dataExchange = 'okx'; defaultData.marketType = 'spot'; defaultData.leverage = 1; defaultData.marginMode = 'isolated'; } if (type === 'indicator') { defaultData.indicator = 'rsi'; defaultData.params = {length: 14}; defaultData.outputIdx = 0; } if (type === 'priceData') { defaultData.priceType = 'close'; defaultData.offset = 0; } if (type === 'condition') { defaultData.operator = '>'; defaultData.rightValue = ''; } @@ -521,21 +671,48 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { }, [nodes, edges]); const isDirty = () => baselineRef.current !== null && graphSnapshot(nodes, edges) !== baselineRef.current; + // Generated layouts: once every node has a measured height, separate the + // columns and retake the baseline — a layout correction is not user work. + // Keeps running while the graph is untouched (nodes grow when keys and + // market lists arrive); the first user edit ends it. + useEffect(() => { + if (!relayoutRef.current || !nodesInitialized || nodes.length === 0) return; + if (nodes.some(n => !n.height)) return; + if (baselineRef.current !== null && graphSnapshot(nodes, edges) !== baselineRef.current) { + relayoutRef.current = false; + return; + } + const fixed = resolveColumnOverlaps(nodes); + if (!fixed) return; + setNodes(fixed); + baselineRef.current = graphSnapshot(fixed, edges); + if (reactFlowInstance) requestAnimationFrame(() => reactFlowInstance.fitView({ padding: 0.1 })); + }, [nodesInitialized, nodes, edges, setNodes, reactFlowInstance]); + + // Re-entrancy guard: a second Close/Esc while the "Unsaved changes" dialog is + // open must not fire another confirmDialog (which would resolve the first + // one as "cancelled" and leave the user with a dialog that does nothing). + const closingRef = useRef(false); const requestClose = async () => { - if (saving) return; + if (saving || closingRef.current) return; if (!isDirty()) return closeBuilder(); - const choice = await confirmDialog({ - type: 'warning', - title: 'Unsaved changes', - message: editingBot - ? `"${editingBot.name}" has changes that are not saved. Save & close keeps them; Discard throws them away and the bot stays as it was.` - : 'This strategy has not been saved yet. Save & close keeps it; Discard throws the whole draft away.', - confirmText: 'Save & close', - secondaryText: 'Discard', - cancelText: 'Keep editing', - }); - if (choice === true) await handleSaveAndCompile(); - else if (choice === 'secondary') closeBuilder(); + closingRef.current = true; + try { + const choice = await confirmDialog({ + type: 'warning', + title: 'Unsaved changes', + message: editingBot + ? `"${editingBot.name}" has changes that are not saved. Save & close keeps them; Discard throws them away and the bot stays as it was.` + : 'This strategy has not been saved yet. Save & close keeps it; Discard throws the whole draft away.', + confirmText: 'Save & close', + secondaryText: 'Discard', + cancelText: 'Keep editing', + }); + if (choice === true) await handleSaveAndCompile(); + else if (choice === 'secondary') closeBuilder(); + } finally { + closingRef.current = false; + } }; requestCloseRef.current = requestClose; @@ -549,13 +726,25 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { const apiKeyName = apiKeyNode?.data.apiKeyName || null; const apiKeyRecord = apiKeyName ? availableKeys?.find(k => k.name === apiKeyName) : null; const dataExchange = apiKeyRecord?.exchange || apiKeyNode?.data.dataExchange || 'okx'; + // The key's market wins; leverage/margin mode only matter on swaps + // (the backend strips the defaults from the config fingerprint) + const marketType = apiKeyRecord?.market_type || apiKeyNode?.data.marketType || 'spot'; + const leverage = marketType === 'swap' ? (parseInt(apiKeyNode?.data.leverage) || 1) : 1; + const marginMode = marketType === 'swap' ? (apiKeyNode?.data.marginMode || 'isolated') : 'isolated'; if (!configNode) return showError("Missing 'Main Configuration' block."); if (!whitelistNode) return showError("Missing 'Asset Whitelist' block."); const symbolsList = parsePairs(whitelistNode.data.pairs); if (symbolsList.length === 0) return showError("Whitelist must contain at least one pair."); - if (knownSymbols?.known && knownSymbols.exchange === dataExchange) { + if (marketType === 'swap') { + const spotPairs = symbolsList.filter(sym => !sym.includes(':')); + if (spotPairs.length) return showError(`Perpetual swaps use the BASE/QUOTE:SETTLE form (e.g. BTC/USDT:USDT for a linear, BTC/USD:BTC for an inverse contract); fix ${spotPairs.join(', ')} in the whitelist.`); + } else { + const swapPairs = symbolsList.filter(sym => sym.includes(':')); + if (swapPairs.length) return showError(`${swapPairs.join(', ')} are perpetual swaps; a spot bot needs BASE/QUOTE pairs (or pick a perps market in the Exchange Routing block).`); + } + if (knownSymbols?.known && knownSymbols.exchange === dataExchange && (knownSymbols.marketType || 'spot') === marketType) { const listed = new Set(knownSymbols.symbols); const unknown = symbolsList.filter(sym => !listed.has(sym)); if (unknown.length) return showError(`Not listed on ${dataExchange.toUpperCase()}: ${unknown.join(', ')}. Fix the whitelist before saving.`); @@ -563,16 +752,9 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { const wantsExchange = configNode.data.executionMode === 'exchange'; if (wantsExchange && !apiKeyRecord) return showError("Exchange orders need an API key — select one in the Exchange Routing block or switch Execution back to forward test."); - // Strip non-serialisable function refs and runtime keys before persisting - const uiNodesSafe = nodes.map(n => { - const safeNode = { ...n, data: { ...n.data } }; - delete safeNode.data.onChange; - delete safeNode.data.onDelete; - delete safeNode.data.availableKeys; - delete safeNode.data.knownSymbols; - delete safeNode.data.liveContext; - return safeNode; - }); + // Strip non-serialisable function refs, runtime keys and ReactFlow + // bookkeeping (selected/width/height/…) before persisting + const uiNodesSafe = nodes.map(stripNodeTransients); const payload = { name: configNode.data.botName || "Untitled Algorithm", @@ -598,6 +780,9 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { backtest_lookback: backtestNode ? (backtestNode.data.lookback || 150) : 150, api_key_name: apiKeyName, data_exchange: dataExchange, + market_type: marketType, + leverage, + margin_mode: marginMode, trade_settings: {}, nodes: {}, ui_layout: { @@ -644,15 +829,33 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { }; } - if (exitNode) { - payload.settings.trade_settings.exit = { - order_type: exitNode.data.orderType || 'market', - amount_type: exitNode.data.amountType || 'percentage', - amount_value: exitNode.data.amountValue !== undefined && exitNode.data.amountValue !== "" ? exitNode.data.amountValue : 100, - fee: exitNode.data.fee !== undefined && exitNode.data.fee !== "" ? exitNode.data.fee : 0.1, - slippage: exitNode.data.slippage !== undefined && exitNode.data.slippage !== "" ? exitNode.data.slippage : 0.05 + const closingLeg = (node) => ({ + order_type: node.data.orderType || 'market', + amount_type: node.data.amountType || 'percentage', + amount_value: node.data.amountValue !== undefined && node.data.amountValue !== "" ? node.data.amountValue : 100, + fee: node.data.fee !== undefined && node.data.fee !== "" ? node.data.fee : 0.1, + slippage: node.data.slippage !== undefined && node.data.slippage !== "" ? node.data.slippage : 0.05 + }); + if (exitNode) payload.settings.trade_settings.exit = closingLeg(exitNode); + + // Short leg (phase 3): serialized only when a short/cover action exists + // on the canvas — a long-only bot keeps its exact settings shape (and + // config fingerprint). Short TP/SL hang off the short node's ports. + const shortNode = nodes.find(n => n.type === 'action' && n.data.actionType === 'short'); + const coverNode = nodes.find(n => n.type === 'action' && n.data.actionType === 'cover'); + if (shortNode) { + const ruleFromEdge = (e) => { + const n = nodes.find(nd => nd.id === e.target); + if (!n) return null; + return { type: n.data.triggerType, value: parseFloat(n.data.triggerValue) || 0, close_amount_type: n.data.closeType, close_amount_value: parseFloat(n.data.closeValue) || 100 }; + }; + payload.settings.trade_settings.short = { + ...closingLeg(shortNode), + take_profits: edges.filter(e => e.source === shortNode.id && e.sourceHandle === 'tp').map(ruleFromEdge).filter(Boolean), + stop_losses: edges.filter(e => e.source === shortNode.id && e.sourceHandle === 'sl').map(ruleFromEdge).filter(Boolean), }; } + if (coverNode) payload.settings.trade_settings.cover = closingLeg(coverNode); const traverse = (targetId) => { const incomingEdge = edges.find(e => e.target === targetId && (e.targetHandle === 'logic' || e.targetHandle === 'left' || e.targetHandle === 'in1' || !e.targetHandle)); @@ -730,8 +933,16 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { if (entryNode) payload.settings.entry_node = traverse(entryNode.id); if (exitNode) payload.settings.exit_node = traverse(exitNode.id); + if (shortNode) payload.settings.short_node = traverse(shortNode.id); + if (coverNode) payload.settings.cover_node = traverse(coverNode.id); + // PUT merges settings, so a short leg removed from the canvas must be + // cleared explicitly (null = engine default, fingerprint-neutral) + if (editingBot?.settings) { + if (!shortNode && editingBot.settings.short_node) payload.settings.short_node = null; + if (!coverNode && editingBot.settings.cover_node) payload.settings.cover_node = null; + } - const hasLogic = !!payload.settings.entry_node; + const hasLogic = !!payload.settings.entry_node || !!payload.settings.short_node; const res = editingBot ? await apiClient.put(`/api/bots/${editingBot.id}`, { name: payload.name, settings: payload.settings }) @@ -743,7 +954,7 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { if (hasLogic) { toast.success(editingBot ? 'Algorithm configuration updated.' : 'Algorithm successfully compiled & deployed.'); } else { - toast.warn('Draft saved without logic — the engine will ignore it until you connect an Entry signal.'); + toast.warn('Draft saved without logic — the engine will ignore it until you connect an Entry (or Short) signal.'); } baselineRef.current = graphSnapshot(nodes, edges); closeBuilder(); @@ -784,7 +995,7 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => { { type: 'stopLoss', label: 'Stop Loss (Risk)', cls: 'border-danger/50 text-danger hover:bg-danger/10' }, ]}, { title: '4. Execution', items: [ - { type: 'action', label: 'Entry / Exit Actions (Buy · Sell)', cls: 'border-text/20 text-text hover:bg-text/10' }, + { type: 'action', label: 'Entry / Exit Actions (Buy · Sell · Short · Cover)', cls: 'border-text/20 text-text hover:bg-text/10' }, ]}, ]; @@ -861,7 +1072,7 @@ const BotBuilderFlow = ({ closeBuilder, editingBot }) => {

Build: Indicator → Condition → Entry Action.

-

Drag or click blocks from the toolbox.

+

Drag or click blocks from the toolbox. Actions: Buy · Sell (spot and perps), Short · Cover (perps only).

)} diff --git a/frontend/src/components/Builder/CustomNodes.jsx b/frontend/src/components/Builder/CustomNodes.jsx index ac6d4af..015ef8c 100644 --- a/frontend/src/components/Builder/CustomNodes.jsx +++ b/frontend/src/components/Builder/CustomNodes.jsx @@ -1,7 +1,20 @@ import React from 'react'; import { Handle, Position } from 'reactflow'; import { useIndicators } from './indicatorConfig'; +import { useExchanges, marketCaps } from '../../api/exchanges'; import { DEFAULT_PAIR, parsePairs } from './pairs'; +import { fmtMoney, marketKind, marketCurrency } from '../../utils/money'; + +// Cash-currency picture of the whitelist, pushed in by the builder as +// `data.marketCtx` ({ cashCurrency, cashCurrencies, quoteCurrency, hasInverse, +// inverseBases }); 'mixed' = pairs settle in different currencies. +const ccyLabel = (code, fallback = 'cash currency of the whitelist') => + (!code ? fallback : code === 'mixed' ? 'mixed currencies' : code); +const CcyChip = ({ code, title }) => ( + + {code === 'mixed' ? 'mixed' : code} + +); // All known timeframes with display labels const ALL_TIMEFRAMES = [ @@ -99,9 +112,12 @@ export const BotConfigNode = ({ id, data }) => ( )}
- + data.onChange(id, 'maxOrderValue', e.target.value === "" ? "" : parseFloat(e.target.value))} /> - Safety guard: rejects live orders exceeding this USD value + Safety guard: rejects live orders whose notional (size × price, in the pair's quote currency{data.marketCtx?.quoteCurrency && data.marketCtx.quoteCurrency !== 'mixed' ? ` — ${data.marketCtx.quoteCurrency}` : ''}) exceeds this value
@@ -113,10 +129,16 @@ export const BotConfigNode = ({ id, data }) => (
); +// Isolated-margin liquidation distance the engine models (engine/pnl.py): +// entry × (1 ∓ (1 − MMR) / leverage) with MMR = 0.5 % → ≈ −19.9 % at 5×. +const LIQ_MMR = 0.005; +const liquidationPct = (lev) => Number((((1 - LIQ_MMR) / Math.max(1, Number(lev) || 1)) * 100).toFixed(1)); + // Execution mode + go-live checklist. `data.liveContext` is pushed in by the -// builder: { keyName, exchange, isSandbox, entryFee } — null key = no -// exchange orders possible, so the "exchange" option is disabled rather than -// silently saving a bot that would fall back to forward test at start. +// builder: { keyName, exchange, isSandbox, entryFee, marketType, leverage, +// marginMode } — null key = no exchange orders possible, so the "exchange" +// option is disabled rather than silently saving a bot that would fall back +// to forward test at start. const ExecutionModeSection = ({ id, data }) => { const ctx = data.liveContext || {}; const hasKey = !!ctx.keyName; @@ -124,11 +146,24 @@ const ExecutionModeSection = ({ id, data }) => { const wantsExchange = mode === 'exchange'; const isLiveMoney = wantsExchange && hasKey && !ctx.isSandbox; const cap = Number(data.maxOrderValue) || 0; + const mctx = data.marketCtx || {}; + // The cap is a quote notional; null when the whitelist quotes in several currencies + const quoteCcy = mctx.quoteCurrency && mctx.quoteCurrency !== 'mixed' ? mctx.quoteCurrency : null; const fee = Number(ctx.entryFee); + const isSwap = ctx.marketType === 'swap'; + const lev = isSwap ? (Number(ctx.leverage) || 1) : 1; + // Liquidation distance as the engine models it: (1 − MMR) / leverage with + // MMR 0.5 %, e.g. ≈ −19.9 % at 5× (not the naive 100 % / leverage) + const liqPct = liquidationPct(lev); const checks = [ { ok: hasKey, label: hasKey ? `Key "${ctx.keyName}" on ${String(ctx.exchange || '').toUpperCase()} (${ctx.isSandbox ? 'sandbox → paper' : 'real → live'})` : 'Select an API key in the Exchange Routing block' }, - { ok: cap > 0, label: cap > 0 ? `Max order value $${cap} caps every order` : 'Set Max Order Value USD above (required for live orders)' }, + { ok: cap > 0, label: cap > 0 ? `Max order value ${fmtMoney(cap, quoteCcy, { digits: 0 })}${quoteCcy ? '' : ' (quote notional)'} caps every order${isSwap ? ' (notional = margin × leverage)' : ''}` : `Set the max order value above (required for live orders${isSwap ? '; caps the notional, i.e. margin × leverage' : ''})` }, { ok: fee > 0, label: fee > 0 ? `Entry fee ${fee}% modelled` : 'Entry fee is 0% — the backtest ignores what the exchange will charge' }, + ...(isSwap ? [{ ok: lev <= 3, label: `Perpetuals ${lev}× ${ctx.marginMode || 'isolated'} · liquidation ≈ −${liqPct}% from entry · funding simulated from stored rates${ctx.marginMode === 'cross' ? ' · cross: a liquidation takes the whole pool' : ''}${mctx.hasInverse ? ` · inverse: margin and PnL in ${mctx.inverseBases.join('/')}` : ''}` }] : []), + ...(ctx.hasShort ? [ + { ok: isSwap, label: isSwap ? 'Short entries — routed through a swap key' : 'Short entries — requires a swap (perpetuals) key; this key trades spot' }, + { ok: !!(ctx.hasCover || ctx.hasShortStop), label: (ctx.hasCover || ctx.hasShortStop) ? 'Short leg has an exit (COVER block or stop-loss)' : 'Short leg has no COVER block and no stop-loss — a short could only be closed by liquidation' }, + ] : []), ]; return (
@@ -170,6 +205,16 @@ export const WhitelistNode = ({ id, data }) => { const listed = market?.known ? new Set(market.symbols) : null; const unknown = listed ? pairs.filter(p => !listed.has(p)) : []; const exch = String(market?.exchange || '').toUpperCase(); + // Perpetual swaps use ccxt's BASE/QUOTE:SETTLE form; a spot pair on a swap + // bot (or vice versa) is flagged before the validator would reject it + const isSwap = market?.marketType === 'swap'; + const wrongForm = market ? pairs.filter(p => p.includes(':') !== isSwap) : []; + // Per-pair contract kind and cash currency: from the exchange's market list + // when the API sent it, else derived from the symbol form + const markets = market?.markets; + const kindOf = (p) => marketKind(markets, p); + const ccyOf = (p) => marketCurrency(markets, p); + const ccys = [...new Set(pairs.map(ccyOf).filter(Boolean))]; return (
@@ -180,7 +225,7 @@ export const WhitelistNode = ({ id, data }) => {