diff --git a/agents/market_making_fly/AGENT.md b/agents/market_making_fly/AGENT.md new file mode 100644 index 000000000..a27f14c2d --- /dev/null +++ b/agents/market_making_fly/AGENT.md @@ -0,0 +1,151 @@ +--- +name: Market Making Fly +description: Market maker whose regime, spread width and reference-price lean are decoded + from a simulated fly connectome watching the chart, with P&L fed back as dopamine. + Operates pmm_mister on any CLOB spot or perp market, including Hyperliquid HIP-3. +agent_key: claude-acp:sonnet +tools: +- get_prices +- get_portfolio_overview +- list_executors +- get_executor +- get_performance_report +- manage_controllers +- manage_bots +- search_history +- manage_routines +- manage_agents +- manage_strategies +- control_agent +- trading_agent_journal_read +- trading_agent_journal_write +- manage_memory +- manage_skill +- run_code +when_to_consult: When the user asks what the fly sees or thinks about a market, + why it widened or leaned, whether it is learning, or wants the fly market maker + deployed, started, stopped or rotated — use consult for questions and delegate for a + deployment. +server_required: true +server_name: '' +created_by: 456181693 +created_at: '2026-09-12T00:00:00+00:00' +--- + +# Market Making Fly + +You operate a market maker whose **discretion belongs to a fly**. A simulation of the +MaleCNS v1.0 connectome (166,700 neurons, 25.6 M connections, vendored from stonkfly) +is shown a 320×180 OHLCV chart of each market. Its spike counts are decoded into a +**posture** — regime, spread multiplier, reference-price lean — which a fixed mapping +turns into a `pmm_mister` config. The combined P&L of the fly's bots is pulsed back +into its dopamine cells (PAM11 reward, PPL101 aversive) and a candidate memory rule +adjusts KC→MBON synapses. This all runs deterministically in the `fly_brain` routine. + +## The one rule above all others + +While a fly run is live on a pair, **the fly quotes and you operate.** You deploy, +start, stop, rotate, and report. You never set spreads, skew, or regime from your own +analysis, and you never "correct" the fly's posture. If you believe the fly is wrong, +you stop it and say why; you do not out-vote it. The guard inside `fly_brain` (fee +floor, loss stop, loss-rate breaker, apply cooldown, closed books, collateral) can +veto or halt, and it never substitutes a posture either. + +## What you handle +- Deploying the fly market maker end-to-end on `n_markets` (1–3) pairs on any CLOB + venue, spot or perp (`fly_mm_deploy`); the count comes from the strategy config or + the task, default 3 +- Starting `fly_brain` in shadow, then live; stopping it; rotating a market slot +- Reading `fly_status` and explaining a posture in market-making terms +- Reporting the run with `fly_report`, and the numbers with `fly_status` +- Saying plainly what the fly has and has not demonstrated + +## What you do not handle +- Choosing spreads, skew or regime yourself while the fly runs +- Venues without a central limit order book — the fly reads a book and quotes + two sides, so an AMM or a swap route is Market Making Expert's or the LP + agent's job, not yours +- Claims that the fly "understands" the market or has learned to trade — it has not + been shown to; see the caveats below + +## Two modes + +**Consulted:** "what does the fly see on DRAM", "why did it widen", "is it learning". +Run `fly_status` (and `fly_chart` for the picture), answer in key: value lines, quote +the numbers (trend_z, arousal_z, gate, regime, spread ×, lean, changed edges), and +repeat the caveats. Do not deploy. + +**Delegated:** read `fly_mm_deploy` and follow it end-to-end — scanner → `n_markets` picks → +neutral configs → deploy with a loss cap → start `fly_brain` in shadow — then verify +with `fly_report`. Switch to live only when the task says so. + +``` +manage_skill(action="read", name="fly_mm_deploy") +``` + +## Routines +| Routine | Use | +|---|---| +| `fly_setup` | `action=prepare` downloads and compiles the connectome into this agent's home (once per install, needs `uv sync --extra fly`); `verify`; `bench` | +| `mm_market_scanner` | Rank any venue's markets for market making: spread against that venue's round-trip fee, book depth, drift. Works on HIP-3 too | +| `fly_chart` | Render the exact frame for a pair (what the fly sees) | +| `fly_brain` | The loop (continuous). `mode=shadow|live`, `pairs`, `picked_ranges_bps`, `run_name` | +| `fly_status` | Latest posture per pair, last observation, guard state, memory stats | +| `fly_report` | The run dashboard: the orbitable fly, the book, the neuron strip, the decision log, and the frame the fly last saw | + +Naming is derived from the **whole** pair, so two markets on one token never collide: +`SOL-USDT` → bot `sol-usdt-fly`, config `sol_usdt_fly_mm`; `XYZ:ORCL-USD` → +`xyz-orcl-usd-fly` / `xyz_orcl_usd_fly_mm`. `fly_brain` reads P&L from exactly those +names, so deploy with them. + +## Spot or perp — settled by the connector, and it matters + +A `_perpetual` suffix means perp; anything else is spot. Two things follow, both +enforced in code rather than left to judgment: + +- **Leverage** applies only to a perp. On spot it must be 1, and `position_mode` + is not sent at all. +- **The fee floor** is derived from the venue's maker fee, and spot fees run three + to five times perp fees on the same exchange. Binance perp is 2 bp a side, so the + take-profit floor is 8.8 bp; Binance spot is 7.5 bp a side, so the floor is 33 bp. + A take-profit that earns comfortably on a perp loses money on spot, and it loses + it silently — the bot fills happily and bleeds the difference. An unknown venue + defaults to a deliberately wide 10 bp spot / 2.5 bp perp. **Pass the exchange's + real maker fee as `maker_fee_bps` whenever you know it.** + +## How to read a posture +- `regime`: pause > volatile > trending_up/down > quiet > ranging, from z-scored channels +- `trend_z`: DNp20 right-minus-left firing vs the pair's rolling baseline; sign = lean + direction, gate (DNpe017 spike) required for a trending call +- `arousal_z`: descending-neuron population rate vs baseline; sets `spread ×` + (`1 + 0.5·z`, clipped 0.6–2.5); `z ≥ 2.5` pulls quotes (kill switch) +- `lean`: `±1 bp per z`, capped at 3 bp and at half the first spread level; applied as + asymmetric buy/sell spreads around mid +- `warm=False` for a pair's first 10 observations: neutral posture while the baseline forms +- Applies happen only on a regime change, a spread-× move ≥ 0.15 or a lean move ≥ 0.5 bp, + and at most once per 5 minutes per pair + +## HIP-3 facts you must keep when the venue is Hyperliquid HIP-3 +- Uppercase pair with issuer prefix (`XYZ:DRAM-USD`); lowercase → zero orders +- Unified collateral: available USD from `get_portfolio_overview(["hyperliquid_perpetual"])` +- Many markets close off-hours; the fly holds while a book is closed and stops the bot + after 5 closed ticks +- All-in maker fee ≈ 1.3 bp/side; the take-profit floor in code is `max(4 bp, 2.2 × round trip)` +- Tight bands `target 0.4 / min 0.3 / max 0.5`, `global_stop_loss 0.02`, leverage ≤ 5 + +## Caveats you repeat when asked +- The decoder is an engineered readout of spike counts, not a discovery of market-making + neurons; a persistent circuit bias would be a persistent lean, which baseline-centring + reduces but does not remove +- Dopamine pulses report P&L change between two observations, not credit for the last + posture; synapses also change from endogenous activity +- No profitable learning has been demonstrated; the guard and the controller's stop + loss bound the loss, not the fly +- Full detail: `agents/market_making_fly/README.md` + +## Memory & Skills +Check `manage_memory` and `manage_skill` before answering; update them when the user +corrects you or a pattern repeats. + +## Response format +Key: value lines, recommendation first, numbers quoted from `fly_status`. diff --git a/agents/market_making_fly/README.md b/agents/market_making_fly/README.md new file mode 100644 index 000000000..4a132c351 --- /dev/null +++ b/agents/market_making_fly/README.md @@ -0,0 +1,682 @@ +# Market Making Fly + +A Condor agent whose market-making discretion — regime, spread width, how much +to commit, which way to lean — is decoded from a simulation of the MaleCNS v1.0 +fly connectome watching a chart, with the bots' P&L fed back as dopamine. + +Status: **implemented**. This file is the agent's reference: the design, the +decisions behind it, and §15 for what has and has not been demonstrated. +Written 2026-09-12, evidence added 2026-09-13. + +## 0. Install + +``` +uv sync --extra fly # pyarrow, for the MaleCNS feather files +manage_routines(action="run", name="fly_setup", config={"action": "prepare"}) +``` + +`pyarrow` is 122 MB and nothing else in Condor reads Arrow, so it is an extra +rather than a core dependency — a Condor installed without it is a normal +Condor with this one agent unavailable, and the two entry points that need it +say so with the command rather than an ImportError. `prepare` downloads ~1.1 GB +of connectome and compiles the kernel; it is a once-per-install step. + +## 1. Summary + +Market Making Fly is a new Condor agent modeled on Market Making Expert, but the +discretionary part of market making — *what regime are we in, how wide should +we quote, which way should the reference price lean* — is not decided by an LLM +or by indicator thresholds. It is decoded from the spiking activity of a +simulation of the **MaleCNS v1.0 fly connectome** (166,700 neurons, 25.6 M +directed connections) that is shown a rendered **OHLCV candlestick chart** of the +market, exactly the way [stonkfly](https://github.com/nftechie/stonkfly) shows +a fly a line chart of BTC-USDC and reads buy/sell/hold out of its descending +neurons. + +The strategy P&L is fed back into the fly the same way stonkfly does it: a +positive change pulses the 15 **PAM11** reward dopamine cells, a negative change +pulses the 2 **PPL101** aversive dopamine cells, and the candidate memory rule +adjusts the 7,835 existing **KC → MBON07 / MBON11** synapses. + +Everything that touches money stays deterministic and outside the fly, in the +stonkfly spirit: the connectome **proposes** a quoting posture, a fixed mapping +turns the posture into a `pmm_mister` config, a **guard can veto but never +substitute**, and Hummingbot's controller executes. The Condor LLM agent is the +*operator* — it picks the market, deploys the bot, starts and stops the fly, +reports, and explains — it never overrides the fly's posture with its own view. + +The trading universe is **any CLOB market hummingbot-api serves, spot or perp**. +Hyperliquid HIP-3 was the first venue and keeps two special cases: its order +book comes from Hyperliquid's own endpoint (hummingbot-api's 500s on HIP-3 +pairs), and its scanner ranks the xyz issuer's markets. Everything venue- +dependent — spot versus perp, whether leverage applies, and what a round trip +costs — lives in `flybrain/venue.py`; the decoder, the chart and the dopamine +feedback are venue-agnostic, because a candle chart is a candle chart. + +FLM (the "Fly Language Model") was the first candidate and is **not** used: +it is a frozen 1.2 B-parameter chat model whose next-token scores get a small +correction from the connectome, and its own paper reports the fly readout does +not beat a matched non-fly control. Stonkfly's direct spiking-simulation route +puts the actual connectome in the decision path, which is what you asked for. + +## 2. What is reused from stonkfly, what is replaced + +| Stonkfly piece | MM Fly | Notes | +|---|---|---| +| `neural/` package: graph import, C++ LIF kernel, `MemoryBrain`, `VisualMemoryBrain`, `circuit` (PAM11/PPL101/KC/MBON), `rule` (anti-Hebbian memory), checksum locks | **Vendored unchanged** into `agents/market_making_fly/flybrain/neural/` | MIT. Only the data-directory constant changes. Keeping it byte-identical keeps its lock/provenance chain valid. | +| `data.py` prepare / verify (1.1 GB MaleCNS download, checksum, compile `graph.npz`) | Vendored, exposed as the `fly_setup` routine (`action=prepare|verify|bench`) and `python agents/market_making_fly/flybrain/__main__.py …` | Data lives in the agent's own git-ignored home. | +| `display.market_frame` (320×180 line chart) | **Replaced** by `agents/market_making_fly/flybrain/chart.py`: OHLCV candlesticks + volume | Same canvas size and light palette so the retinal projection is unchanged. | +| `Decoder` (DNp20 R−L → BUY/SELL/HOLD) | **Replaced** by `agents/market_making_fly/flybrain/decoder.py`: DNp20 R−L → trend/skew, a population-rate channel → arousal/spread, gate on DNpe017 | Still a fixed, engineered readout of spike counts. | +| `reinforcement.py` (equity delta vs anchor, deadband) | Same rule, equity = controller `realized + unrealized − fees` | Filtered by the bot/controller, so other bots on the account do not leak in. | +| `risk.Guard` / `Veto` | `agents/market_making_fly/flybrain/guard.py`: fee floor, loss stop, loss-rate breaker, apply cooldown, market-open, collateral, price-move tolerance | Veto keeps the previous config. It never invents a different posture. | +| `broker.py`, `actions.py`, `ledger.py` (Coinbase FOK orders, SQLite intents) | **Dropped** | Execution is Hummingbot `pmm_mister`; the fly never places orders itself. | +| `cli.py` run loop, 2-slot checkpoints, `events.jsonl`, `latest.json`, `latest-input.png`, provenance hash | Re-implemented as the `fly_brain` continuous routine + `agents/market_making_fly/flybrain/run_state.py` | Same durability pattern. | + +## 3. Pipeline + +``` + every interval_sec (default 60 s) + ┌────────────────────────────────────────────────────────────────────────────┐ + │ 1. fetch : last N candles (HIP-3 pair) + live l2Book bid/ask │ + │ 2. render : chart.py → 320×180 RGB uint8 (OHLCV + volume + bid/ask) │ + │ 3. reward : Δ(controller net P&L) vs last observation → reward|aversive|none│ + │ 4. observe: worker process — VisualMemoryBrain.rgb_step for neural_ms │ + │ (default 500 ms neural time), dopamine pulse if reward/aversive │ + │ 5. decode : spike counts → {trend_z, arousal_z, gate} → regime, spread_mult,│ + │ shift_bps (posture) │ + │ 6. map : posture → pmm_mister config (HIP-3 base config × posture) │ + │ 7. guard : vetoes (fee floor, loss stop, cooldown, market closed, …) │ + │ 8. apply : if live mode and posture changed materially → update_config │ + │ 9. persist: checkpoint (2 slots), events.jsonl, latest.json, PNG, LiveReport│ + └────────────────────────────────────────────────────────────────────────────┘ + ▲ │ + Condor LLM agent (operator) ─ deploy / stop / report / explain ┘ +``` + +Ordering follows stonkfly: reinforcement for the *previous* observation is +delivered at the start of the *next* neural window; the checkpoint and +accounting anchor are committed **before** anything is applied to the bot. + +## 4. What the fly sees — chart specification + +`agents/market_making_fly/flybrain/chart.py: market_frame(pair, candles, bid, ask) -> np.uint8[180,320,3]` + +* Canvas 320×180, light background `(235,240,249)`, dark header bar with the + pair name — identical to stonkfly. Stonkfly found the dark chart produced no + Kenyon-cell spikes and switched to light; we keep that. +* **Candles**: last `n_candles = 72` of `interval = 5m` (6 h window) drawn 4 px + wide in the 294 px plot span. Up candle body **blue `(0,101,183)`**, down + body **red `(197,37,78)`**, 1 px wicks in the body color. Blue/red is + stonkfly's palette: the mapped R8p cells read the blue channel and R8y the + green channel, R1–R6 read luminance, so colour is not decoration here — it is + the only chromatic input the fly gets. +* **Volume**: bars in a bottom strip (rows 140–158), grey-blue, scaled to the + window maximum. +* **Bid / ask**: two short horizontal ticks at the right edge at the bid and ask + price, plus stonkfly's `BID x ASK y` footer text. +* Price axis: window `[min low, max high]` padded 12 % top and bottom, with a + floor of 0.2 % of mean price so a flat market does not blow up to full scale + (stonkfly's rule). +* **Not drawn**: the bot's own quotes, inventory, P&L, funding, or anything + from the account. Stonkfly deliberately excludes portfolio state from the + picture; P&L reaches the fly only through dopamine. + +The frame is saved as `latest-input.png` every observation, so you can see what +the fly saw when it made a call. `fly_chart` is also a one-shot routine, so you +can render the chart for any pair from chat and inspect it. + +Candles come from Hummingbot (`client.market_data.get_candles_last_days` on +`hyperliquid_perpetual`), which already serves HIP-3 pairs. The live book comes +from Hyperliquid `l2Book` directly with the `xyz:TOKEN` coin form (the +hummingbot-api `order_book` endpoint 500s on HIP-3 — documented in the HIP-3 +operator playbook). + +## 5. The neural substrate + +* `agents/market_making_fly/flybrain/neural/` = stonkfly's `neural/` package plus `data.py`, vendored + with a `THIRD_PARTY.md` notice (MIT). Files: `brain.py`, `state.py`, + `circuit.py`, `rule.py`, `visual.py`, `sensory.py`, `transmitters.py`, + `connectome.py`, `prepare.py`, `kernel.cpp`, and the four `*.lock.json` + checksum files. The only edit is `common.DATA`, which resolves to Condor's + data dir instead of `./data`. +* Integration parameters stay stonkfly's: 0.1 ms timestep, 500 ms neural time + per observation in 10 ms bins, 200 ms / 20 mV dopamine pulse, KC rest −60 mV + with adaptation, R8 → aMe12 sign correction, memory rule gain 0.001, 1 s + eligibility traces, 1,800 s memory decay, efficacy bounds 0.1–2×. +* **Data**: `fly_setup` (`action=prepare`) downloads ~1.1 GB (annotations + edges + feather), verifies SHA-256 against the locks, compiles `graph.npz` + (166,700 × 25,582,938) and builds the C++ kernel with `c++ -O3`. Location: + `.condor/agents/market_making_fly/data/` (the agent's writable home, git-ignored), + overridable with `CONDOR_FLY_DATA`. `fly_setup` `action=verify` re-checks. +* **Process model**: the simulation is CPU-bound C++ called through ctypes and + must not run on Condor's event loop. The `fly_brain` routine owns a + `ProcessPoolExecutor(max_workers=1, spawn)` whose initializer loads + `VisualMemoryBrain` once; each tick submits `observe(frame, reinforcement)` + and awaits the future. Frames are 172 KB, results are small dicts. + Checkpoint/restore run in the same worker. One worker per running fly + instance; a per-run lock file prevents two instances on one run directory + (stonkfly's `worker.lock`). +* **Compute budget**: stonkfly does not publish a per-observation timing. + `fly_setup` `action=bench` will run three observations on a synthetic frame and print + `compute_seconds`; if it exceeds ~40 % of `interval_sec`, lower `neural_ms` + (it is a config field). Memory: the graph is ~300 MB resident, checkpoints + ~100 MB each × 2 slots. + +## 6. Decoder — spikes → quoting posture + +`agents/market_making_fly/flybrain/decoder.py`. Engineered and fixed, like stonkfly's — it reads only +spike counts, and the cell IDs it reads are written to the audit log. + +Raw channels per observation (`seconds = neural_ms / 1000`): + +| Channel | Cells | Measurement | +|---|---|---| +| `trend_hz` | DNp20 left vs right (stonkfly's BUY/SELL cells) | mean right rate − mean left rate | +| `arousal_hz` | descending-neuron population (`superclass == "descending"` in `graph.npz`, minus the DNp20/DNpe017 readouts) | mean firing rate | +| `gate` | DNpe017 | spike count ≥ 1 | + +Baseline-centering (new, addresses a problem stonkfly documented): stonkfly's +six-observation run proposed BUY every time — a persistent turning bias of the +circuit became persistent buying. In market making that would become a +persistent skew. So the decoder keeps a running mean/std of `trend_hz` and +`arousal_hz` over the last `baseline_window = 60` observations (persisted in +`state.json`) and works in z-scores. The first `warmup = 10` observations emit +the neutral posture while the baseline forms. `center_bias` is a config flag +(default on) so a raw, uncentred run is still possible as a control. + +Regime, in precedence order: + +| Regime | Rule | +|---|---| +| `pause` | `arousal_z ≥ 2.5` | +| `volatile` | `arousal_z ≥ 1.0` | +| `trending_up` | `gate` and `trend_z ≥ 1.0` | +| `trending_down` | `gate` and `trend_z ≤ −1.0` | +| `quiet` | `arousal_z ≤ −1.0` | +| `ranging` | otherwise, and during warm-up | + +Continuous outputs: + +* `spread_mult = clip(1 + 0.5 · arousal_z, 0.6, 2.5)` — the fly agitated by a + jumpy chart widens, a calm fly tightens. +* `shift_bps = clip(1.0 · trend_z, −max_shift_bps, +max_shift_bps)` with + `max_shift_bps = 3` and a hard cap of 50 % of the first-level spread — the + reference price leans with the fly's turning direction. + +Hysteresis so the bot is not re-configured on noise: a new posture is only +applied if the regime changed, or `|Δspread_mult| ≥ 0.15`, or +`|Δshift_bps| ≥ 0.5`, and at least `min_apply_interval_sec = 300` have passed +since the last apply. Otherwise the tick is `HOLD`. The fly is still observed and +reinforced every tick; only the application is rate-limited (stonkfly's order +cooldown, transposed). + +Thresholds (1.0 / 2.5 z, 0.5 gain, 3 bp) are declared model choices with no +fitted basis. They are config fields. + +## 7. Posture → `pmm_mister` config + +`agents/market_making_fly/flybrain/posture.py: build_config(base, posture, fees) -> dict`. The base +config is the HIP-3 operator's bounded defaults, the posture multiplies and +shifts it: + +* Base spreads from the scanner's picked spread `S` bp: level 1 + `max(2, S/2)`, level 2 `S+1` (HIP-3 playbook). Then + `buy = base · spread_mult − shift`, `sell = base · spread_mult + shift`, + each floored at the market's own maker fee, so a two-sided round trip at + the floor breaks even (1.3 bp on a HIP-3 perp, 7.5 on a spot book). +* `take_profit = max(0.0004, level-1 buy spread)` — and never below + `2.2 × round-trip maker fee` (HIP-3 all-in ~1.3 bp/side → 2.6 bp; floor 5.7 bp + wins). Market Making Expert's rule "TP must exceed round-trip fees" becomes a + hard floor in code rather than advice. +* Timing per regime, from Market Making Expert's table: quiet + `refresh 20 / cooldown 30`, ranging `30 / 60`, trending `30 / 60`, volatile + `60 / 120`. +* Fixed by the HIP-3 playbook, not touched by the fly: `target_base_pct 0.4, + min 0.3, max 0.5`, `portfolio_allocation 0.2`, `max_active_executors_by_level + 2`, `leverage ≤ 5` and ≤ market max, `open/take_profit_order_type 3` + (LIMIT_MAKER), `global_sl_enabled true, global_stop_loss 0.02`, uppercase + `XYZ:TOKEN-USD` pair. +* `pause` regime → `manual_kill_switch = true` (quotes pulled, position kept); + leaving `pause` → `false`. +* Optional (off by default): nudge `target_base_pct` by ±0.05 in a trending + regime. Off because the HIP-3 rules say tight bands are what stopped the SPCX + and DRAM losses. + +Applied through `manage_bots(action="update_config", …, confirm_override=true)` +on the live bot **and** `manage_controllers(action="upsert")` on the saved +config, both layers, per the deploy playbook. + +## 8. Guard — vetoes only + +`agents/market_making_fly/flybrain/guard.py`. Every rule is deterministic; a veto means "keep the +previous config", a halt means "stop the bot and stop the fly until a human +passes `resume_reviewed=true`". None of them chooses a different posture. + +| Check | Effect | +|---|---| +| Market closed (live book missing a side) — HIP-3 equities close off-hours | veto (HOLD); after `closed_ticks_to_stop` consecutive closed ticks, STOP bot, leave fly observing | +| Available USD on the unified Hyperliquid account < required margin | veto | +| Any spread inside the market's maker fee, TP below fee floor, leverage above cap | veto (should be unreachable after `posture.py` floors; this is the belt to those braces) | +| Apply cooldown (`min_apply_interval_sec`) or daily apply cap (`max_applies_per_day = 48`) | veto | +| Mid moved more than `apply_price_tolerance = 0.5 %` between observation and apply | veto (stonkfly's fresh-book check) | +| `total_net ≤ −max_loss_quote` (default 4 % of `total_amount_quote`) | **halt**: STOP bot, position closed by the controller's `global_stop_loss` / market-close, fly halted | +| HIP-3 loss-rate breaker: `total_net / volume ≤ −5 bp` or no new session high for 25 ticks | **halt** with market-close, exactly the HIP-3 playbook's mandatory rule | +| `update_config` failed 3 consecutive ticks | **halt** for review | +| STOP file in the run directory | halt (stonkfly's `touch runs/live/STOP`) | + +`total_net` always includes unrealized P&L — the HIP-3 playbook's rule. + +## 9. Reinforcement — P&L → dopamine + +`agents/market_making_fly/flybrain/reinforcement.py`, stonkfly's function with a different equity +source: + +* `equity_t = realized_pnl + unrealized_pnl − fees` for **this bot's** + `pmm_mister` controller, read from `manage_bots(action="status")` + performance (`realized_pnl_quote` + `unrealized_pnl_quote`), so nothing else on the + account can reward or punish the fly. +* `delta = equity_t − anchor` where `anchor` is the previous observation's + equity. `delta ≥ +deadband` → `reward` (PAM11 ×15), `≤ −deadband` → + `aversive` (PPL101 ×2), else `none`. Deadband default = 1 bp of + `total_amount_quote` (stonkfly used 0.01 on $100, the same 1 bp). +* Binary pulse, 200 ms, 20 mV-equivalent, delivered during the first 200 ms of + the next 500 ms window; not proportional to the amount; tiny changes are not + accumulated. All as stonkfly. +* The memory rule and plastic edge set are stonkfly's, untouched: 7,835 + KC→MBON07/MBON11 edges, baseline-centred anti-Hebbian rule fed by actual + KC and DAN spike rates. +* `learning = true` by default; `frozen = true` freezes efficacies for a + control run. In **shadow** mode with no bot deployed, reinforcement is `none` + every tick (no P&L exists); with a bot deployed under the fly's name it is + the bot's P&L even though the fly's postures are not being applied — stated + plainly in the status report because that signal has nothing to do with the + fly's own proposals. + +What this is and is not: it is feedback about the controller's marked value +between two observations, including fees; it is not evidence that the last +posture caused it. Stonkfly's validation found plastic edges changing *before* +the first external pulse — endogenous dopamine activity drives the rule too. We +report `changed_edges`, `mean_efficacy`, `reward_spikes`, `aversive_spikes` +every tick and make no claim beyond them. + +## 10. Trading universe — any CLOB market + +**Venue-independent by default (2026-09-13).** `flybrain/venue.py` holds the +three things that differ between venues, all of which bear on money: + +* **spot or perp**, decided by the connector name (`_perpetual` means perp). + Leverage applies only to a perp; on spot it must be 1 and `position_mode` is + not sent. A connector declared as the type it is not is refused rather than + silently reinterpreted. +* **the maker fee**, from which the take-profit floor follows. Spot fees run + three to five times perp fees on the same exchange, so the floor moves with + it: Binance perp 2 bp a side gives an 8.8 bp floor, Binance spot 7.5 bp gives + 33 bp. An unknown venue defaults deliberately wide (10 bp spot, 2.5 bp perp), + because a floor set too low loses money silently while one set too high only + costs fills. `maker_fee_bps` overrides it per deployment. +* **the top of book**, which comes from hummingbot-api for every connector it + serves. HIP-3 pairs are the one exception: that endpoint 500s on them, so + they fall back to Hyperliquid's public `l2Book`. + +Pair grammar is `BASE-QUOTE` or, on HIP-3, `ISSUER:TOKEN-QUOTE`. Names derive +from the **whole** pair (`SOL-USDT` → `sol-usdt-fly`, `XYZ:ORCL-USD` → +`xyz-orcl-usd-fly`), because naming a bot after its base token alone makes +`BTC-USDT` and `BTC-USDC` the same bot — the fly would read one book's P&L +while updating the other's config. + +The decoder, the chart and the dopamine feedback are unchanged and venue- +agnostic: a candle chart is a candle chart. + +### Hyperliquid HIP-3, the first venue + +* Connector `hyperliquid_perpetual`, issuer `xyz`, pairs `XYZ:TOKEN-USD` + (uppercase; lowercase → KeyError → zero orders). +* Market selection is the `mm_market_scanner` routine, which ranks any venue's + markets on the same criteria (superseding the HIP-3-only scanner this agent + used to carry): volume, spread-vs-fee, daily drift, `l2Book` depth filter, + `TOP PICK`. Selection is the operator's job (deterministic routine + LLM + reading it), not the fly's — the fly never chooses which market it is shown, + as in stonkfly's fixed round-robin. +* Venue facts carried over from the HIP-3 operator: unified collateral (read + from `get_portfolio_overview(["hyperliquid_perpetual"])`, not the per-dex + clearinghouse state), isolated margin, trading hours / empty books, ~1.3 + bp/side all-in maker fee including the fixed Hummingbot builder fee, tight + inventory bands, `global_stop_loss 0.02`. +* **Three markets, one brain (decided 2026-09-12).** The fly quotes three + HIP-3 markets in parallel and learning is shared: a single + `VisualMemoryBrain` is shown the three charts in a fixed round-robin + (stonkfly's multi-asset schedule — the network never chooses which market it + sees), one pair per tick, so each pair is observed every `3 × interval_sec`. + Reinforcement is the change in the **combined** net P&L of the three + controllers since the previous observation, delivered while whichever chart + is up — feedback about the fly's whole book, as stonkfly's is about the whole + portfolio. Baselines and postures are **per pair** (keyed by pair in + `state.json`); the KC→MBON weights are shared and persist in one checkpoint + lineage. Rotating a market swaps the pair in the list and resets that pair's + baseline only; the brain and its learned efficacies carry on. +* Rotation: only when flat and the current market is closed, trending against + us, or dominated — re-run the scanner and redeploy that slot. + +## 11. Condor integration — files + +``` +agents/market_making_fly/ + flybrain/ # everything below lives inside the agent; nothing in condor/ changes + __init__.py + neural/ # vendored stonkfly.neural + THIRD_PARTY.md (MIT) + data.py # vendored prepare/verify, Condor data dir + chart.py # OHLCV → 320×180 RGB + decoder.py # counts → channels, z-scores, regime, posture + posture.py # posture → pmm_mister config with floors + guard.py # vetoes / halts + reinforcement.py # equity delta → reward|aversive|none + worker.py # ProcessPoolExecutor target: load brain, observe, checkpoint, restore + run_state.py # run dir: state.json, events.jsonl, latest.json, PNG, 2-slot checkpoints, provenance, lock, STOP + __main__.py # prepare | verify | bench, runnable by path + +agents/market_making_fly/ + AGENT.md # operator brain (LLM) + routines/ + fly_chart.py # one-shot: render + report the frame for a pair + fly_brain.py # CONTINUOUS: the loop in §3; modes shadow|live, learning|frozen, fixture|market + fly_status.py # one-shot: latest posture, channels, z-scores, spikes, memory stats, P&L, vetoes, halt reason + fly_report.py # one-shot: the run dashboard (fly, book, neurons, decisions, sensory frame) + mm_market_scanner.py # ranks any venue: spread vs its round-trip fee, depth, drift + skills/ + fly_mm_deploy/SKILL.md # deploy playbook (adapted from pmm_mister_deploy: scanner → base config → deploy → start fly_brain) + fly_decoder/SKILL.md # how to read fly_status and the decoder + strategies/mm_operator/strategy.md # thin loop: keep bot + fly alive, surface halts, rotate when flat + +agents/market_making_fly/tests/ # conftest puts the agent dir on sys.path + test_fly_chart.py test_fly_decoder.py test_fly_posture.py + test_fly_guard.py test_fly_reinforcement.py + test_fly_full_graph.py # opt-in, CONDOR_FLY_FULL_TEST=1, needs prepared data + +README.md # this file: design, decisions, and what is claimed +``` + +Run state per fly instance: +`.condor/agents/market_making_fly/fly//` with `state.json` +(baseline stats, anchor, tick, last applied posture/config, halt reason), +`events.jsonl` (one row per tick: quote, frame hash, spike summary, channels, +posture, config diff, guard result), `latest.json`, `latest-input.png`, +`brain-0.npz` / `brain-1.npz`, `provenance.json` (settings, dataset hashes, +circuit report, decoder description, source hashes — a changed protocol refuses +to resume into an old run dir, as stonkfly does), `worker.lock`, optional `STOP`. + +### The agent (LLM) — what it does and does not do + +`AGENT.md` keeps Market Making Expert's domain knowledge (regimes, spread +calibration, inventory, fee rule, `pmm_mister` parameter guide) so it can +*explain* what the fly did in market-making terms, and adds one rule above all +others: + +> While a fly run is live on a pair, the fly quotes and you operate. You deploy, +> start, stop, rotate, and report. You never set spreads, skew, or regime from +> your own analysis, and you never "correct" the fly's posture. If you believe +> the fly is wrong, you stop it and say why; you do not out-vote it. + +Two modes, like Market Making Expert: + +* **Consulted**: "what does the fly see on DRAM right now", "why did it widen", + "is it learning anything" → runs `fly_status` (and `fly_chart`), answers in + key: value lines, quotes the numbers, repeats the caveats in §15. +* **Delegated / loop**: `fly_mm_deploy` skill end-to-end: scanner → TOP PICK → + base config built by `posture.build_config`, which *is* Market Making + Expert's balanced profile everywhere the fly does not decide (see §15) → + deploy with `max_global_drawdown_quote` → start `fly_brain` + (shadow first unless told live) → verify with `fly_report`. The loop + strategy ticks every 5 min: confirms bot and fly instance alive, surfaces + halts/vetoes, rotates when flat and the market is closed. + +Tools list = Market Making Expert's plus `manage_routines` for the fly +routines. `server_name` left empty, `agent_key` copied from Market Making +Expert. + +### Modes + +| Mode | Observes | Applies config | Reinforcement | Use | +|---|---|---|---|---| +| `shadow` (default) | real chart | no — logs what it *would* apply | bot P&L if a bot is deployed, else none | first runs, watch the posture stream against `market_analyzer` | +| `live` | real chart | yes | bot P&L | after shadow looks sane | +| `frozen` flag | any | any | pulses delivered, weights frozen | control run | +| `fixture` flag | synthetic sine candles | never | none | offline plumbing test, `fast=true` skips wall waits | + +## 12. Deploy lifecycle + +1. `fly_setup` with `action=prepare` once per install (1.1 GB, several minutes, needs `c++`). +2. Operator agent runs the scanner, deploys `pmm_mister` on the TOP PICK with + the neutral (ranging) posture config. +3. `manage_routines(action="start", name="fly_brain", agent="market_making_fly", + config={"trading_pair": "XYZ:DRAM-USD", "bot_name": "dram-mm", + "config_name": "dram_mm_live", "mode": "shadow", "run_name": "dram-2026-09-12"})`. +4. Watch `fly_status` / the LiveReport for an hour; compare the fly's regime + stream with `market_analyzer` for the same window (report only). +5. Restart with `mode: "live"` on the same run directory (baseline and memory + carry over). +6. Halts require `resume_reviewed: true` on restart, and a financial halt cannot + be cleared that way at all (stonkfly's rule) — a new run directory is needed. + +## 13. Observability + +* `fly_brain` maintains one `LiveReport`: latest chart PNG, regime timeline + (last 100 ticks), `trend_z` / `arousal_z` sparklines, spike KPIs, memory + KPIs (`changed_edges`, `mean_efficacy`), P&L and reinforcement history, + applied-config history, vetoes/halts. +* `fly_status` gives the same as text for consults and the Telegram loop. +* The agent journals every apply and halt with the posture and the guard + verdict. `events.jsonl` is the audit trail; each row carries the frame hash, + spike-count hash, and checkpoint hash so a decision can be tied to exactly + what the fly saw and what state it was in. + +## 14. Testing and validation plan + +Unit (no data, run in CI): + +* `chart.py`: deterministic frame for fixed candles (hash), shape/dtype, up + candle pixels are blue and down red, bid/ask ticks land at the right rows, + flat-market scale floor. +* `decoder.py`: synthetic counts → expected regime for each row of the table; + warm-up emits ranging; centring removes a constant bias; hysteresis blocks + small changes and passes regime changes. +* `posture.py`: fee floor beats spread; shift never pushes a side below + the fee floor; pause sets the kill switch; timing table per regime; pair + is uppercase. +* `guard.py`: every row of the table in §8, halt vs veto, `resume_reviewed` + semantics, financial halt not clearable. +* `reinforcement.py`: stonkfly's four cases. +* `run_state.py`: checkpoint slot alternation, provenance mismatch refuses + resume, lock file. + +Opt-in with prepared data (`CONDOR_FLY_FULL_TEST=1`): stonkfly's full-graph +test transposed — a white frame activates KCs, reward and aversive pulses spike +their DAN cells, eligible edges change, frozen stays identical, checkpoint +round-trips; plus a rendered real chart produces non-zero KC activity (this is +the check stonkfly says is the biggest open modelling question). + +Runs, in order: `fixture + fast + steps=6` offline; `shadow` on a live HIP-3 +pair for at least 60 observations (one baseline window); `live` at +`total_amount_quote = 100` USD; only then normal size. + +Not planned in this pass, and required before any learning claim: held-out +chronological replay, shuffled-reinforcement control, exposure/cash baselines, +retention after reset. The design leaves room for them (`frozen`, fixture +market, run directories) but does not deliver them. + +## 15. What this does and does not claim + +Copied in spirit from stonkfly's `docs/model.md`, because the same limits hold: + +* The connectome supplies anatomy; the LIF cells, transmitter sign proxies, + RGB-to-photoreceptor mapping, compressed clock (500 ms neural per 60 s wall) + and dopamine assignments are engineered choices, not fly physiology. +* The decoder is an interface we chose. "Trend from DNp20, arousal from + descending neurons" is not a discovery of market-making neurons. A persistent + circuit bias becomes a persistent skew; baseline-centring reduces but does + not remove that. +* Candle colours were chosen to hit the R8 channels; that is a display adapter. +* P&L pulses are feedback about the controller's value, not credit assignment. + Weight changes do not show the fly learned to quote. +* No profitable learning is demonstrated by anything in this design. The guard + and the controller's stop-loss are what bound the loss, not the fly. + +### What the replay controls established, 2026-09-13 + +`fly_replay` walks a pinned candle series with the same brain, decoder and +geometry, changing one setting per variant. Two markets, four contiguous +windows each, 327 ticks per window, every variant on its own freshly seeded +brain: + +| market | against `live` | pooled per-tick | pooled t | windows won | +|---|---|---|---|---| +| XYZ:DRAM-USD (1.3 bp fee, 13.7 bp bars) | shuffled reinforcement | +0.00035 | +0.17 | **2 / 4** | +| | frozen plasticity | +0.00136 | +0.65 | 3 / 4 | +| ZEC-USD (2.5 bp fee, 41.3 bp bars) | shuffled reinforcement | −0.00501 | −0.80 | **2 / 4** | +| | frozen plasticity | −0.00813 | −1.11 | 1 / 4 | + +**The fly with real P&L reinforcement beats the fly whose reinforcement sign is +random in exactly half the windows, on both markets.** That is a coin flip. It +is the control this design has always said it lacked, it exists now, and it +does not separate them — on two fee families, in a regime where the strategy +made money (DRAM, all variants +1.7 to +1.8) and one where it lost (ZEC, all +variants −12.5 to −14.6). Freezing the memory rule outright does not separate +them either, and on ZEC the frozen fly was ahead. + +The fill model still assumes a touched price was ours, and eight windows of two +markets is not the last word. But the earlier single-window figures this +replaces (t = +0.24 against shuffled) were one slice of one market, and the +multi-window result is the one to quote: a reader can judge "2 of 4 windows" +without trusting the statistic at all. + +What this does *not* say is that the market maker does not work. On DRAM's +volatile stretch every variant earned, which is the geometry earning — quote +levels placed against how far the market travels. The connectome's contribution +to that is what is unmeasurable. + +Two measurement lessons are worth more than the numbers: + +* An earlier statistic ran on the equity curves rather than their increments. + A curve is cumulative, so it returned |t| of 27 to 63 for variants a few + percent apart — a measure of when two runs diverged, not of whether they earn + differently. +* The replay's open-position cap was a number chosen in the harness. It bound + before the strategy did, and manufactured |t| of 1.5 to 2.0 across four runs + that collapsed to 0.1–1.0 once the cap came from `pmm_mister`'s own limit. + Every "suggestive" result before that fix should be read as withdrawn. + +### What the levers were worth + +* **Quote levels off the bar range, not the touch** (kept). Level 1 sat at half + the observed spread — 2 bp where a typical bar ranged 9.9 — so every + multiplier the fly could express stayed inside what a normal bar covers, and + five variants returned identical fills. At half the range, `widen` fills 18 + where `live` fills 38. +* **An exit that scales with the market** (kept) — it was the fee floor whether + a market moved 2 bp a bar or 20. +* **The fly moving that exit** (reverted). Halved the round trips, 35 against + 68, and earned nothing for them in either window. +* **A gated trend taking a side off the book** (kept, unproven). Fired on 13 of + 327 ticks and finished +0.22 ahead of the two-sided control at t = +0.67. + +## 16. Decisions + +Settled 2026-09-12: 1 vendor, 2 deterministic apply, 3 descending neurons, +4 baseline-centred, 5 72 × 5 m, 6 net P&L incl. unrealized, 7 **three markets +in parallel on one shared brain** (see §10), 8 shadow default, 9 +`.condor/agents/market_making_fly/data`, 10 no target-base nudge. The original options are kept +below for the record. + +1. **Vendor stonkfly's neural package into `agents/market_making_fly/flybrain/neural/` (recommended)** + vs `pip install git+…stonkfly`. Installing pulls `coinbase-agentkit` and + `coinbase-advanced-py` into Condor for no use; vendoring adds ~1,400 lines + + the kernel + `pyarrow` as a new optional dependency (extra `fly`). +2. **Deterministic apply (recommended)** — the `fly_brain` routine applies the + config itself and the LLM only operates — vs the LLM loop reading the + posture and applying it each tick. The second puts an LLM back between the + fly and the bot, which is what stonkfly's "no LLM trading policy" rule + avoids, and costs tokens every minute. +3. **Arousal population = descending neurons (recommended)** vs whole-network + mean rate vs Kenyon cells. All three are engineered; descending neurons are + the motor-output side stonkfly already reads from and are labelled in + `graph.npz` without the annotations file. +4. **Baseline-centre the channels (recommended, flag)** vs raw rates. Raw is + stonkfly's choice and is the honest one for an experiment; centred is the + one that will not sit skewed one way for hours because of circuit bias. +5. **Chart window 72 × 5 m (recommended)** vs 100 × 1 m (stonkfly's density) + vs 1 h candles. 5 m matches the regime cadence the config timing table + assumes; 1 m is what the fly could realistically react to at 60 s ticks. +6. **Reinforcement source = controller net P&L incl. unrealized, 1 bp deadband + (recommended)** vs realized-only. The HIP-3 rules insist on unrealized. +7. **Fresh run directory per market (recommended)** vs carrying learned + weights across markets. +8. **Default mode shadow (recommended)**; live is explicit. +9. **Data dir `/fly/data` (recommended)** vs `~/.condor/fly`. +10. Whether the target-base nudge in trending regimes (§7, off by default) + should exist at all. + +## 17. Implementation phases + +| Phase | Deliverable | Verifies | +|---|---|---| +| 1 | `agents/market_making_fly/flybrain/neural` vendored, `data.py`, `fly_setup` routine, `pyarrow` dep | `prepare` completes on this Mac, `verify` passes, `bench` prints compute time | +| 2 | `chart.py` + tests, `fly_chart` routine | rendered DRAM chart in a report | +| 3 | `decoder.py`, `posture.py`, `guard.py`, `reinforcement.py` + tests | unit suite green | +| 4 | `worker.py`, `run_state.py`, `fly_brain` routine, fixture mode | `fixture + fast + steps=6` run end-to-end with checkpoint resume | +| 5 | `AGENT.md`, skills, strategy, `fly_status`, copies of scanner/dashboard | consult works; shadow run on a live HIP-3 pair | +| 6 | live mode at small size | your call, after reviewing shadow output | + +Rough effort: phases 1–4 are the bulk; 5 is mostly adaptation of Market Making +Expert text; 6 is operation, not code. + +## 18. Implementation status (2026-09-12) + +Phases 1–5 are implemented; phase 6 (live at small size) is the operator's call. + +| Piece | Where | State | +|---|---|---| +| Vendored neural package, data prepare/verify, kernel | `agents/market_making_fly/flybrain/neural/`, `flybrain/data.py`, `fly_setup` routine | done; dataset prepared at `.condor/agents/market_making_fly/data` (1.6 GB), verified, kernel built | +| Chart, decoder, posture, guard, reinforcement, naming, market, run state, worker | `agents/market_making_fly/flybrain/*.py` | done, 77 unit tests green (`uv run pytest agents/market_making_fly/tests`) | +| Loop routine | `agents/market_making_fly/routines/fly_brain.py` | done; 14-observation fixture run end to end (checkpoints, events, live report, halt path) | +| `fly_chart`, `fly_status`, copied scanner/dashboard | `agents/market_making_fly/routines/` | done, load through routine discovery | +| Agent brain, deploy playbook, decoder skill, operator strategy | `agents/market_making_fly/{AGENT.md,skills,strategies}` | done | +| Opt-in full-graph test | `agents/market_making_fly/tests/test_fly_full_graph.py` (`CONDOR_FLY_FULL_TEST=1`) | done | +| Doctor row | — | not done; use `fly_setup` `action=verify` | + +Deviations from the text above: the descending-neuron superclass label in +`graph.npz` is `descending_neuron` (1,314 cells); the whole implementation is +contained in `agents/market_making_fly/` (package `flybrain`, put on `sys.path` +by the routines) so no core Condor module changes — the only repo-level edit is +the `pyarrow` dependency; setup is the `fly_setup` routine rather than a CLI; +checkpoints are ~7 MB compressed, not 100 MB. Condor re-executes a routine +file when it changes but keeps imported modules cached, so after editing +anything under `flybrain/` restart Condor before starting `fly_brain`. + +### First measurements + +* Bench: brain loads in ~1 s; one 500 ms neural window costs 2–5 s of compute on + this Mac. A 60 s wall interval leaves ample room. +* DNp20 fires 26–46 Hz on a chart with a consistent right-minus-left surplus of + +2 to +12 Hz in every observation — stonkfly's persistent bias, now measured. + Baseline-centring is what keeps it from becoming a permanent upward lean. +* A rendered chart activates 10–17 Kenyon cells per window at first (stonkfly's + chart gave 11–16). An aversive pulse produced 13–30 PPL101 spikes, a reward + pulse ~270 PAM11 spikes, and 5 KC→MBON edges changed from endogenous activity + before any pulse, as stonkfly also reported. +* In the fixture run the network flipped, at the ninth observation, into a + persistent high-activity state (total spikes 388 k → 609 k per window, KC + spikes 12 → 4,000+, descending rate 2.5 → 8 Hz, 2,900 plastic edges + changed). A control run with reinforcement disabled reproduced the flip at + the same tick with the same numbers, so it is driven by the visual input + sequence, not by the dopamine pulses. It is what the arousal channel is meant + to read, but it means "volatile" can persist until the rolling baseline + absorbs the new level (up to one window). Stonkfly's warning that display + sensitivity is the largest open modelling question applies here in full. + +## 19. Housekeeping from this session + +* FLM was cloned, installed, and its downloads started at `~/flm` (4.3 GB) + before the change of direction. The install was stopped. Delete with + `rm -rf ~/flm ~/flm-setup.sh ~/flm-setup.log` when you like; nothing in this + design uses it. +* The scaffold copy `agents/market_making_fly/` from the first attempt was + removed; the tree is clean apart from this document. diff --git a/agents/market_making_fly/flybrain/__init__.py b/agents/market_making_fly/flybrain/__init__.py new file mode 100644 index 000000000..b08bd6771 --- /dev/null +++ b/agents/market_making_fly/flybrain/__init__.py @@ -0,0 +1,6 @@ +"""Market Making Fly — a fly-connectome simulation that proposes a quoting posture. + +The neural substrate (``flybrain.neural``) is vendored from stonkfly; the +chart, decoder, posture mapping, guard, reinforcement and worker are Condor's. +See ``agents/market_making_fly/README.md``. +""" diff --git a/agents/market_making_fly/flybrain/__main__.py b/agents/market_making_fly/flybrain/__main__.py new file mode 100644 index 000000000..f77220866 --- /dev/null +++ b/agents/market_making_fly/flybrain/__main__.py @@ -0,0 +1,76 @@ +"""``uv run python agents/market_making_fly/flybrain/__main__.py prepare | verify | bench``. + +``prepare`` downloads the MaleCNS v1.0 release files (~1.1 GB), verifies their +checksums, compiles the retained graph and builds the C++ kernel. ``verify`` +re-checks the prepared arrays. ``bench`` loads the brain and times a few +observations on a synthetic frame so ``fly_brain``'s ``neural_ms`` can be sized +against its wall interval. +""" + +from __future__ import annotations + +import argparse +import json +import sys +import time +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(prog="python -m flybrain", description=__doc__) + sub = parser.add_subparsers(dest="command", required=True) + sub.add_parser("prepare", help="download, verify and compile the connectome") + sub.add_parser("verify", help="re-check the prepared connectome arrays") + bench = sub.add_parser("bench", help="time observations on a synthetic frame") + bench.add_argument("--observations", type=int, default=3) + bench.add_argument("--neural-ms", type=float, default=500.0) + args = parser.parse_args(argv) + + from flybrain.neural.common import DATA + + if args.command == "prepare": + from flybrain.data import prepare + + print(f"Data directory: {DATA}", flush=True) + prepare() + from flybrain.neural.brain import build + + print(json.dumps({"kernel": build()["model"], "data": str(DATA)})) + return 0 + if args.command == "verify": + from flybrain.data import verify + + print(json.dumps({**verify(), "data": str(DATA)})) + return 0 + if args.command == "bench": + import numpy as np + from flybrain.worker import FlyBrain + + started = time.perf_counter() + brain = FlyBrain(learning=True) + load = time.perf_counter() - started + frame = np.full((180, 320, 3), 235, np.uint8) + rows = [] + for _ in range(args.observations): + result = brain.observe(frame, "none", neural_ms=args.neural_ms) + rows.append( + { + "compute_seconds": round(result["compute_seconds"], 3), + "total_spikes": result["total_spikes"], + "kc_spikes": result["kc_spikes"], + } + ) + print( + json.dumps({"load_seconds": round(load, 2), "observations": rows}, indent=2) + ) + return 0 + parser.error("unknown command") + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/agents/market_making_fly/flybrain/brainviz.py b/agents/market_making_fly/flybrain/brainviz.py new file mode 100644 index 000000000..416fc897c --- /dev/null +++ b/agents/market_making_fly/flybrain/brainviz.py @@ -0,0 +1,265 @@ +"""The connectome, drawn, and coloured by what just fired. + +The anatomy comes from :mod:`flybrain.cloud` — real soma coordinates from the +MaleCNS release, not a layout invented to look plausible. The colour comes from +the spike counts the last observation actually produced, so the mushroom body +lighting up is the mushroom body having fired, not an illustration of the idea. +""" + +from __future__ import annotations + +import numpy as np +from flybrain.cloud import GROUPS, load, orient +from flybrain.fly3d import GROUND, LIMB + +# Silent slate through to a hot filament: a neuron that did nothing should +# recede into the anatomy, and one that fired hard should be the brightest +# thing on the panel. +ACTIVITY_SCALE = [ + [0.0, "#1b2536"], + [0.12, "#2f4a6d"], + [0.35, "#4f8fd0"], + [0.62, "#ff9a54"], + [0.85, "#ffd166"], + [1.0, "#fff6e0"], +] + +# Used only when a run has no activity snapshot: colour by what a cell is +# rather than by what it did. +GROUP_COLORS = { + "mushroom body": "#ff9a54", + "dopamine": "#ff4d6d", + "memory output": "#ffd166", + "readout": "#c9f24d", + "descending": "#7ee0ff", + "visual": "#5aa0ff", + "other": "#55637d", +} + + +def brain_figure( + activity: list[int] | None = None, + title: str = "", + height: int = 460, +): + """The somata of the retained graph, orbitable, coloured by firing.""" + import plotly.graph_objects as go + + cloud = load() + points = orient(cloud["xyz"]) + group = cloud["group"] + hidden = dict( + showbackground=False, + showgrid=False, + zeroline=False, + showticklabels=False, + visible=False, + ) + layout = dict( + height=height, + margin=dict(l=0, r=0, t=36 if title else 6, b=6), + paper_bgcolor=GROUND, + plot_bgcolor=GROUND, + font=dict(color="#c8d4e8", family="monospace", size=11), + title=( + dict( + text=title, x=0.02, xanchor="left", font=dict(color="#c9f24d", size=13) + ) + if title + else None + ), + scene=dict( + xaxis=hidden, + yaxis=hidden, + zaxis=hidden, + bgcolor=GROUND, + aspectmode="data", + camera=dict(eye=dict(x=0, y=-1.15, z=0.22), up=dict(x=0, y=0, z=1)), + ), + legend=dict( + orientation="h", + yanchor="top", + y=-0.02, + xanchor="center", + x=0.5, + font=dict(size=10), + ), + ) + + if activity is not None and len(activity) == len(group): + rate = np.asarray(activity, dtype=float) + firing = rate > 0 + ceiling = float(np.percentile(rate[firing], 97)) if firing.any() else 1.0 + shade = np.clip(rate / (ceiling or 1.0), 0, 1) + # Silent and firing are drawn separately on purpose. Nineteen cells in + # twenty are silent in a given window, and running them through the + # same colour scale turns the anatomy into a bright haze that buries + # the few cells that actually did something. + quiet = go.Scatter3d( + x=points[~firing, 0], + y=points[~firing, 1], + z=points[~firing, 2], + mode="markers", + marker=dict(size=1.1, color="#1d2738", opacity=0.5), + hoverinfo="skip", + name="silent", + showlegend=False, + ) + order = np.argsort(shade[firing]) # brightest drawn last + live = go.Scatter3d( + x=points[firing][order, 0], + y=points[firing][order, 1], + z=points[firing][order, 2], + mode="markers", + marker=dict( + size=2.4 + 4.2 * shade[firing][order], + color=shade[firing][order], + colorscale=ACTIVITY_SCALE, + cmin=0, + cmax=1, + opacity=0.95, + showscale=False, + ), + text=[ + f"{GROUPS[g]} · {int(c)} spikes" + for g, c in zip(group[firing][order], rate[firing][order]) + ], + hoverinfo="text", + name="firing", + showlegend=False, + ) + fig = go.Figure([quiet, live]) + fig.update_layout(showlegend=False, **layout) + return fig + + # No activity recorded for this run: show what each cell is instead. + traces = [] + for slot, name in enumerate(GROUPS): + picked = group == slot + if not picked.any(): + continue + traces.append( + go.Scatter3d( + x=points[picked, 0], + y=points[picked, 1], + z=points[picked, 2], + mode="markers", + marker=dict( + size=1.6 if name in ("visual", "other") else 3.0, + color=GROUP_COLORS[name], + opacity=0.55 if name in ("visual", "other") else 0.95, + ), + name=name, + hoverinfo="name", + ) + ) + fig = go.Figure(traces) + fig.update_layout(showlegend=True, **layout) + return fig + + +def coverage() -> tuple[int, int, int]: + """``(drawn, mapped somata, retained neurons)`` — what the picture covers.""" + cloud = load() + return len(cloud["index"]), cloud["mapped_total"], cloud["neurons_total"] + + +READOUT_TOP, READOUT_BOTTOM = 36, 64 + + +def _note_y(height: int) -> float: + """Paper-coordinate y that puts a note clear of the tick labels at any + height. 56 px: the axis numbers sit about 20 below the axis.""" + return -56 / max(1, height - READOUT_TOP - READOUT_BOTTOM) + + +def readout_figure(neural: dict, posture: dict | None = None, height: int = 460): + """The three channels the posture is decoded from, as they were measured. + + DNp20 left against DNp20 right is the whole of the lean: the posture reads + their difference, not either one. The descending population mean is the + arousal that sets spread width. The gate is a count, not a rate, so it is + stated rather than drawn — a bar of 4 next to a bar of 38 Hz would invite + a comparison that means nothing. + """ + import plotly.graph_objects as go + + left = float(neural.get("left_hz") or 0.0) + right = float(neural.get("right_hz") or 0.0) + arousal = float(neural.get("arousal_hz") or 0.0) + posture = posture or {} + labels = ["DNp20 left", "DNp20 right", "descending mean"] + values = [left, right, arousal] + colors = ["#5aa0ff", "#7ee0ff", "#ff9a54"] + + fig = go.Figure( + go.Bar( + x=values, + y=labels, + orientation="h", + marker=dict(color=colors), + text=[f"{v:.1f} Hz" for v in values], + textposition="outside", + textfont=dict(color="#c8d4e8", size=11), + hoverinfo="skip", + ) + ) + gate = neural.get("gate_spikes") + trend = right - left + fig.update_layout( + height=height, + margin=dict(l=96, r=20, t=READOUT_TOP, b=READOUT_BOTTOM), + paper_bgcolor=GROUND, + plot_bgcolor=GROUND, + font=dict(color="#c8d4e8", family="monospace", size=11), + title=dict( + text="READOUT", x=0.02, xanchor="left", font=dict(color="#c9f24d", size=13) + ), + xaxis=dict( + title="firing rate (Hz)", + gridcolor="#18202e", + zerolinecolor="#243044", + range=[0, max(values + [1]) * 1.28], + ), + yaxis=dict(gridcolor="#18202e", automargin=True), + showlegend=False, + legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), + annotations=[ + dict( + x=0, + y=_note_y(height), + xref="paper", + yref="paper", + showarrow=False, + align="left", + xanchor="left", + font=dict(color=LIMB, size=11, family="monospace"), + text=( + f"right − left = {trend:+.1f} Hz" + + ( + f" (z {posture['trend_z']:+.2f})" + if "trend_z" in posture + else "" + ) + + f"
arousal z {posture.get('arousal_z', float('nan')):+.2f}" + if "arousal_z" in posture + else f"right − left = {trend:+.1f} Hz" + ), + ), + dict( + x=1, + y=_note_y(height), + xref="paper", + yref="paper", + showarrow=False, + align="right", + xanchor="right", + font=dict( + color="#c9f24d" if gate else LIMB, size=11, family="monospace" + ), + text=f"DNpe017 gate: {gate if gate is not None else '—'} spike(s)" + + ("" if gate else " — no lean without it"), + ), + ], + ) + return fig diff --git a/agents/market_making_fly/flybrain/chart.py b/agents/market_making_fly/flybrain/chart.py new file mode 100644 index 000000000..ae90e1a4f --- /dev/null +++ b/agents/market_making_fly/flybrain/chart.py @@ -0,0 +1,150 @@ +"""Render OHLCV candles into the 320×180 RGB frame the fly looks at. + +Same canvas, light background and palette as stonkfly's ``display.market_frame`` +so the vendored retinal projection is unchanged: R1–R6 cells read luminance, +the mapped R8p cells read the blue channel and R8y the green channel. Up +candles are blue and down candles red for that reason — colour is the only +chromatic input the fly gets, not decoration. + +The frame never contains the bot's own quotes, inventory or P&L. Portfolio +value reaches the fly only through the dopamine pulse (``reinforcement.py``). +""" + +from __future__ import annotations + +import math +from collections.abc import Sequence +from typing import Any + +import numpy as np +from PIL import Image, ImageDraw + +WIDTH, HEIGHT = 320, 180 +N_CANDLES = 72 + +BACKGROUND = (235, 240, 249) +HEADER = (19, 36, 71) +HEADER_TEXT = (219, 229, 249) +GRID = (200, 212, 233) +UP = (0, 101, 183) +DOWN = (197, 37, 78) +VOLUME = (120, 140, 180) +TEXT = (28, 46, 82) +TICK = (27, 39, 81) + +PLOT_LEFT, PLOT_RIGHT = 12, 306 +PLOT_TOP, PLOT_BOTTOM = 34, 132 +VOLUME_TOP, VOLUME_BOTTOM = 138, 158 +FOOTER_Y = 165 + +_FIELDS = ("open", "high", "low", "close", "volume") + + +def _finite_positive(value: Any, name: str) -> float: + number = float(value) + if not math.isfinite(number) or number <= 0: + raise ValueError(f"{name} must be a finite positive number, got {value!r}") + return number + + +def normalize_candles(candles: Sequence[Any], n_candles: int = N_CANDLES) -> list[dict]: + """Keep the last ``n_candles`` and validate every field; raise on bad data.""" + if n_candles < 2: + raise ValueError("n_candles must be at least 2") + if not candles: + raise ValueError("No candles to render") + rows = [] + for raw in list(candles)[-n_candles:]: + if not isinstance(raw, dict): + raise ValueError(f"Candle must be a dict, got {type(raw).__name__}") + row = {} + for field in _FIELDS: + if field not in raw: + raise ValueError(f"Candle is missing {field!r}") + value = float(raw[field]) + if not math.isfinite(value) or value < 0: + raise ValueError( + f"Candle {field} must be finite and >= 0, got {value!r}" + ) + row[field] = value + for field in ("open", "high", "low", "close"): + if row[field] <= 0: + raise ValueError(f"Candle {field} must be > 0") + if row["low"] > min(row["open"], row["close"]) or row["high"] < max( + row["open"], row["close"] + ): + raise ValueError("Candle high/low do not contain open/close") + rows.append(row) + return rows + + +def price_scale(rows: Sequence[dict]) -> tuple[float, float]: + """``(lo, span)`` for the price axis: window padded 12 % each side, with a + floor of 0.2 % of the mean price so a flat market is not blown up to + full scale (stonkfly's rule).""" + lows = np.asarray([r["low"] for r in rows], dtype=float) + highs = np.asarray([r["high"] for r in rows], dtype=float) + closes = np.asarray([r["close"] for r in rows], dtype=float) + span = max(float(highs.max() - lows.min()), float(closes.mean()) * 0.002) + lo = float(lows.min()) - span * 0.12 + return lo, span * 1.24 + + +def _y(value: float, lo: float, span: float) -> float: + return PLOT_BOTTOM - (value - lo) / span * (PLOT_BOTTOM - PLOT_TOP) + + +def market_frame( + pair: str, + candles: Sequence[Any], + bid: float, + ask: float, + n_candles: int = N_CANDLES, +) -> np.ndarray: + """OHLCV chart → ``np.uint8[HEIGHT, WIDTH, 3]``.""" + bid = _finite_positive(bid, "bid") + ask = _finite_positive(ask, "ask") + if ask < bid: + raise ValueError(f"ask {ask} below bid {bid}") + rows = normalize_candles(candles, n_candles) + + im = Image.new("RGB", (WIDTH, HEIGHT), BACKGROUND) + d = ImageDraw.Draw(im) + d.rectangle((0, 0, WIDTH - 1, 27), fill=HEADER) + d.text((9, 8), str(pair), fill=HEADER_TEXT) + for x in range(PLOT_LEFT, PLOT_RIGHT + 1, 30): + d.line((x, PLOT_TOP, x, VOLUME_BOTTOM), fill=GRID) + for y in range(PLOT_TOP + 4, PLOT_BOTTOM, 24): + d.line((PLOT_LEFT - 2, y, PLOT_RIGHT + 2, y), fill=GRID) + + lo, span = price_scale(rows) + slot = (PLOT_RIGHT - PLOT_LEFT) / n_candles + body_w = max(1, int(slot) - 1) + max_volume = max(r["volume"] for r in rows) + # Right-align so the newest candle always sits at the right edge, where the + # bid/ask ticks are, regardless of how many candles the feed returned. + offset = n_candles - len(rows) + for i, r in enumerate(rows): + x0 = PLOT_LEFT + (offset + i) * slot + xc = int(x0 + slot / 2) + color = UP if r["close"] >= r["open"] else DOWN + y_high, y_low = _y(r["high"], lo, span), _y(r["low"], lo, span) + d.line((xc, y_high, xc, y_low), fill=color, width=1) + y_open, y_close = _y(r["open"], lo, span), _y(r["close"], lo, span) + top, bottom = min(y_open, y_close), max(y_open, y_close) + if bottom - top < 1: + bottom = top + 1 + d.rectangle((int(x0), top, int(x0) + body_w, bottom), fill=color) + if max_volume > 0: + h = r["volume"] / max_volume * (VOLUME_BOTTOM - VOLUME_TOP) + d.rectangle( + (int(x0), VOLUME_BOTTOM - h, int(x0) + body_w, VOLUME_BOTTOM), + fill=VOLUME, + ) + + for price in (bid, ask): + y = _y(price, lo, span) + if PLOT_TOP <= y <= PLOT_BOTTOM: + d.line((PLOT_RIGHT + 2, y, WIDTH - 2, y), fill=TICK, width=2) + d.text((9, FOOTER_Y), f"BID {bid:g} ASK {ask:g}"[:50], fill=TEXT) + return np.asarray(im, dtype=np.uint8) diff --git a/agents/market_making_fly/flybrain/cloud.py b/agents/market_making_fly/flybrain/cloud.py new file mode 100644 index 000000000..d320494a3 --- /dev/null +++ b/agents/market_making_fly/flybrain/cloud.py @@ -0,0 +1,158 @@ +"""The connectome's somata, as a point cloud the report can draw. + +The graph the fly runs on carries no coordinates — it is pure topology. The +MaleCNS annotations do: 139,662 of the 166,700 retained neurons have a soma +position, which is enough to draw the animal's actual anatomy rather than a +made-up layout. + +166,700 points would not survive a report, so this builds a deterministic +subsample once and caches it beside the graph. The sample is stratified rather +than uniform: every cell the decoder and the memory rule actually read — Kenyon +cells, the two dopamine populations, their MBONs, the DNp20/DNpe017 readouts — +is kept in full, because those are the ones worth looking at. The rest is a +seeded draw, so the same install always draws the same brain. +""" + +from __future__ import annotations + +import numpy as np +from flybrain.deps import require_pyarrow +from flybrain.neural.common import DATA, GRAPH + +CLOUD = DATA / "soma_cloud.npz" +SEED = 7301 # the interface seed, so the whole model shares one +DEFAULT_POINTS = 7000 + +# Groups are coarse on purpose: the point is where activity sits in the +# animal, not a taxonomy. Order is the colour order in the report. +GROUPS = ( + "mushroom body", # Kenyon cells — where the memory rule acts + "dopamine", # PAM11 / PPL101 — the reward and aversive populations + "memory output", # MBON07 / MBON11 + "readout", # DNp20 / DNpe017 — what the posture is decoded from + "descending", # the arousal channel + "visual", # optic lobe and visual projection — what the chart drives + "other", +) + + +def _positions(ids: np.ndarray) -> np.ndarray: + """Soma coordinates per graph index; NaN where the release has none.""" + import pyarrow.feather as feather + + table = feather.read_table( + DATA / "annotations.feather", columns=["bodyId", "somaLocation"] + ) + frame = table.to_pandas().set_index("bodyId").reindex(ids) + out = np.full((len(ids), 3), np.nan, dtype=np.float32) + for row, value in enumerate(frame.somaLocation.to_numpy()): + if value is None: + continue + try: + out[row] = np.asarray(value, dtype=np.float32)[:3] + except (TypeError, ValueError): + continue + return out + + +def _group_of(ids: np.ndarray, superclass: np.ndarray) -> np.ndarray: + """One coarse group per neuron, identified cells taking precedence.""" + import pyarrow.feather as feather + from flybrain.worker import DESCENDING_SUPERCLASS + + types = ( + feather.read_table(DATA / "annotations.feather", columns=["bodyId", "type"]) + .to_pandas() + .set_index("bodyId") + .reindex(ids) + .type.fillna("") + .to_numpy() + .astype(str) + ) + group = np.full(len(ids), GROUPS.index("other"), dtype=np.int8) + visual = np.isin( + superclass, + ["ol_intrinsic", "ol_sensory", "visual_projection", "visual_centrifugal"], + ) + group[visual] = GROUPS.index("visual") + group[superclass == DESCENDING_SUPERCLASS] = GROUPS.index("descending") + group[np.char.startswith(types, "KC")] = GROUPS.index("mushroom body") + group[np.isin(types, ["MBON07", "MBON11"])] = GROUPS.index("memory output") + group[np.isin(types, ["PAM11", "PPL101"])] = GROUPS.index("dopamine") + group[np.isin(types, ["DNp20", "DNpe017"])] = GROUPS.index("readout") + return group + + +def build(max_points: int = DEFAULT_POINTS) -> dict: + """Subsample the somata and cache the result. Idempotent.""" + require_pyarrow() + with np.load(GRAPH) as graph: + ids = graph["ids"] + superclass = graph["superclass"].astype(str) + xyz = _positions(ids) + group = _group_of(ids, superclass) + mapped = np.flatnonzero(np.isfinite(xyz).all(axis=1)) + + # Sample per group, not uniformly. Uniform would be all optic lobe — it is + # half the animal — and the circuit that actually decides anything would + # vanish. Proportional keeps the silhouette recognisable; the floors keep + # the mushroom body and the descending channel dense enough to watch; the + # handful of identified cells are always kept whole. + rng = np.random.default_rng(SEED) + floors = { + GROUPS.index("mushroom body"): 420, + GROUPS.index("descending"): 420, + } + always = [GROUPS.index(name) for name in ("dopamine", "memory output", "readout")] + picked = [mapped[np.isin(group[mapped], always)]] + budget = max_points - len(picked[0]) + pool = { + g: mapped[group[mapped] == g] for g in range(len(GROUPS)) if g not in always + } + total = sum(len(v) for v in pool.values()) or 1 + for g, members in pool.items(): + want = min( + len(members), max(floors.get(g, 0), round(budget * len(members) / total)) + ) + picked.append( + members + if want >= len(members) + else rng.choice(members, size=want, replace=False) + ) + index = np.sort(np.concatenate(picked)) + + np.savez_compressed( + CLOUD, + index=index.astype(np.int32), + xyz=xyz[index], + group=group[index], + mapped_total=np.int64(len(mapped)), + neurons_total=np.int64(len(ids)), + ) + return load() + + +def load(max_points: int = DEFAULT_POINTS) -> dict: + """The cached cloud, building it on first use.""" + if not CLOUD.exists(): + return build(max_points) + with np.load(CLOUD) as data: + return { + "index": data["index"], + "xyz": data["xyz"], + "group": data["group"], + "mapped_total": int(data["mapped_total"]), + "neurons_total": int(data["neurons_total"]), + } + + +def orient(xyz: np.ndarray) -> np.ndarray: + """Centre the cloud and put the animal the way an atlas figure does. + + MaleCNS voxel axes are (x right, y down, z front-to-back). Negating y puts + dorsal up, so the brain reads as a brain rather than as its own reflection. + """ + out = np.stack([xyz[:, 0], xyz[:, 2], -xyz[:, 1]], axis=1) + centre = np.nanmean(out, axis=0) + scale = np.nanmax(np.abs(out - centre)) + return (out - centre) / (scale or 1.0) diff --git a/agents/market_making_fly/flybrain/data.py b/agents/market_making_fly/flybrain/data.py new file mode 100644 index 000000000..9785953a9 --- /dev/null +++ b/agents/market_making_fly/flybrain/data.py @@ -0,0 +1,97 @@ +"""Fetch released MaleCNS inputs and verify both sources and prepared arrays.""" + +import hashlib +import json +import shutil +import urllib.request +from pathlib import Path + +import numpy as np + +from .neural.common import DATA, GRAPH, digest + +PACKAGE = Path(__file__).with_name("neural") + + +def sha(path): + h = hashlib.sha256() + with Path(path).open("rb") as f: + for block in iter(lambda: f.read(8 * 1024 * 1024), b""): + h.update(block) + return h.hexdigest() + + +def verify(): + lock = json.loads((PACKAGE / "sources.lock.json").read_text()) + if sha(DATA / "annotations.feather") != lock["annotations.feather"]["sha256"]: + raise RuntimeError("Annotation checksum mismatch") + expected = json.loads((PACKAGE / "arrays.lock.json").read_text()) + with np.load(GRAPH, allow_pickle=False) as a: + if set(a.files) != set(expected): + raise RuntimeError("Graph fields mismatch") + for k, h in expected.items(): + if digest(a[k]) != h: + raise RuntimeError("Graph array checksum mismatch: " + k) + if len(a["ids"]) != 166700 or len(a["post"]) != 25582938: + raise RuntimeError("Wrong retained graph") + import pyarrow.feather as f + + # Match normalized transmitter identities to the checksum-locked released file. + if not (DATA / "normalized/neurons.feather").exists(): + raise RuntimeError("Normalized neuron metadata missing") + n = f.read_table(DATA / "normalized/neurons.feather").to_pandas() + transmitter_values = json.dumps( + n.neurotransmitter.fillna("").astype(str).tolist(), separators=(",", ":") + ).encode() + nt_expected = json.loads((PACKAGE / "neurons.lock.json").read_text())[ + "neurotransmitter_values_sha256" + ] + if hashlib.sha256(transmitter_values).hexdigest() != nt_expected: + raise RuntimeError("Normalized transmitter values mismatch") + with np.load(GRAPH, allow_pickle=False) as a: + if not np.array_equal(n.source_id.to_numpy(), a["ids"]): + raise RuntimeError("Normalized neuron order mismatch") + return { + "release": "MaleCNS v1.0", + "neurons": 166700, + "directed_edges": 25582938, + "arrays_verified": True, + } + + +def prepare(reuse=None): + DATA.mkdir(parents=True, exist_ok=True) + if reuse: + root = Path(reuse) + mapping = { + root / "outputs/doom/malecns_v1/graph.npz": GRAPH, + root + / "connectome_data/malecns_v1/annotations.feather": DATA + / "annotations.feather", + root + / "connectome_data/malecns_v1/normalized/neurons.feather": DATA + / "normalized/neurons.feather", + } + for source, target in mapping.items(): + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(source, target) + else: + lock = json.loads((PACKAGE / "sources.lock.json").read_text()) + for name, info in lock.items(): + path = DATA / name + if not path.exists(): + print("Downloading", name, flush=True) + tmp = path.with_suffix(".partial") + urllib.request.urlretrieve(info["url"], tmp) + if sha(tmp) != info["sha256"]: + raise RuntimeError("Downloaded checksum mismatch: " + name) + tmp.replace(path) + if sha(path) != info["sha256"]: + raise RuntimeError("Source checksum mismatch: " + name) + shutil.copyfile(PACKAGE / "sources.lock.json", DATA / "source.lock.json") + from .neural.connectome import import_graph + from .neural.prepare import prepare as compile_graph + + import_graph() + compile_graph() + print(json.dumps(verify()), flush=True) diff --git a/agents/market_making_fly/flybrain/decoder.py b/agents/market_making_fly/flybrain/decoder.py new file mode 100644 index 000000000..c4f6897b5 --- /dev/null +++ b/agents/market_making_fly/flybrain/decoder.py @@ -0,0 +1,316 @@ +"""Decode spike counts into a quoting posture. + +An engineered, fixed readout — like stonkfly's, it reads only spike counts and +the cells it reads are logged. Nothing here is a discovery of "market-making +neurons": + +* ``trend_hz`` — mean DNp20 right rate minus mean left rate (stonkfly's + BUY/SELL cells). Sign → which way the reference price leans. +* ``arousal_hz`` — mean rate of the descending-neuron population. Higher → + tighter spreads and a larger share of the book quoted. It + moved the take-profit too until a replay showed that cost + half the round trips and earned nothing for them. +* ``gate`` — DNpe017 spikes ≥ 1, required for a trending call, and for + taking a side off the book. +* ``valence_hz`` — mean MBON07 rate minus mean MBON11 rate: approach minus + avoidance. These are the cells the KC→MBON memory rule + writes to, so this is the only channel a P&L pulse can + reach. Higher → more of the book quoted. Without it the + dopamine loop changes synapses that change nothing. +* ``kc_spikes`` — Kenyon cell drive. Not a posture: a confidence test. A scene + that barely reaches the mushroom body leaves the other + channels reading noise, so the posture is marked unconfident + and the loop holds rather than applying it. + +Channels are z-scored against a rolling per-pair baseline. Stonkfly's own run +proposed BUY six times out of six — a persistent turning bias of the circuit +became persistent buying. In market making that would be a persistent skew, so +the baseline is subtracted before thresholds are applied. ``center_bias=False`` +keeps stonkfly's raw reading of the trend channel (2 Hz = one unit) as a +control; arousal has no natural zero and is always window-normalized. +""" + +from __future__ import annotations + +import math +import statistics +from dataclasses import asdict, dataclass, field +from typing import Literal + +Regime = Literal[ + "pause", "volatile", "trending_up", "trending_down", "quiet", "ranging" +] +REGIMES: tuple[str, ...] = ( + "pause", + "volatile", + "trending_up", + "trending_down", + "quiet", + "ranging", +) + +# stonkfly's DNp20 threshold: 2 Hz difference is one unit in raw mode. +RAW_TREND_UNIT_HZ = 2.0 + + +@dataclass(frozen=True) +class DecoderSettings: + window: int = 60 + warmup: int = 10 + z_regime: float = 1.0 + z_pause: float = 2.5 + # Negative: an aroused fly quotes *tighter*, not wider. Arousal is the + # descending population's rate against its own recent average — nothing + # ties it to volatility, so neither sign is derived from anything. This one + # says an active market is one to lean into; +0.5 says it is one to back + # away from, which is the textbook answer. Both are testable against each + # other on the same market and neither has been. + spread_gain: float = -0.5 + spread_min: float = 0.6 + spread_max: float = 2.5 + # And it quotes more of the book while it is active. Bounded the same way, + # then clamped in build_config so an order never falls under the venue + # minimum or the allocation over 1. + size_gain: float = 0.5 + size_min: float = 0.6 + size_max: float = 2.5 + # What the fly has learned about scenes like this one, on the same scale as + # arousal and added to it, so the size the fly commits carries both "the + # market is active" and "this looked good last time". + valence_gain: float = 0.5 + # A scene this far below the mushroom body's own recent drive is one the + # fly effectively did not see; its z-scores are noise. + z_kc_quiet: float = -1.5 + # Past this, a gated trend stops nudging the reference price and takes a + # side away. Leaning moved the quote 2.5 bp on a market whose bars run 8; + # the failure it was meant to prevent — filling one side over and over + # into a move — needs the other side gone, not discounted. + z_side: float = 1.5 + shift_gain_bps: float = 1.0 + max_shift_bps: float = 3.0 + center_bias: bool = True + + def __post_init__(self): + if self.window < 2 or self.warmup < 1 or self.warmup > self.window: + raise ValueError("window >= 2 and 1 <= warmup <= window required") + if not 0 < self.z_regime < self.z_pause: + raise ValueError("0 < z_regime < z_pause required") + for lo, hi, what in ( + (self.spread_min, self.spread_max, "spread"), + (self.size_min, self.size_max, "size"), + ): + if not 0 < lo <= 1 <= hi: + raise ValueError(f"{what}_min <= 1 <= {what}_max required") + for name in ("shift_gain_bps", "max_shift_bps"): + value = getattr(self, name) + if not math.isfinite(value) or value <= 0: + raise ValueError(f"{name} must be finite and positive") + # The two arousal gains carry a direction, so they may be negative — + # but not zero, which would mean the channel is read and discarded. + if self.z_kc_quiet >= 0: + raise ValueError("z_kc_quiet must be negative") + if self.z_side < self.z_regime: + raise ValueError("z_side must be at least z_regime") + for name in ("spread_gain", "size_gain", "valence_gain"): + value = getattr(self, name) + if not math.isfinite(value) or value == 0: + raise ValueError(f"{name} must be finite and non-zero") + + +@dataclass(frozen=True) +class Channels: + trend_hz: float + arousal_hz: float + gate_spikes: int + valence_hz: float = 0.0 + kc_spikes: int = 0 + + def __post_init__(self): + for name in ("trend_hz", "arousal_hz", "valence_hz"): + if not math.isfinite(getattr(self, name)): + raise ValueError(f"Nonfinite channel {name}") + if self.arousal_hz < 0 or self.gate_spikes < 0 or self.kc_spikes < 0: + raise ValueError("Negative rate or spike count") + + +@dataclass +class Baseline: + """Rolling per-pair history of each channel; persisted in state.json.""" + + trend: list[float] = field(default_factory=list) + arousal: list[float] = field(default_factory=list) + valence: list[float] = field(default_factory=list) + kc: list[float] = field(default_factory=list) + + @classmethod + def from_dict(cls, data: dict | None) -> "Baseline": + if not data: + return cls() + return cls( + trend=list(data["trend"]), + arousal=list(data["arousal"]), + valence=list(data["valence"]), + kc=list(data["kc"]), + ) + + def to_dict(self) -> dict: + return { + "trend": list(self.trend), + "arousal": list(self.arousal), + "valence": list(self.valence), + "kc": list(self.kc), + } + + @property + def count(self) -> int: + return len(self.trend) + + def push(self, channels: Channels, window: int) -> None: + self.trend.append(channels.trend_hz) + self.arousal.append(channels.arousal_hz) + self.valence.append(channels.valence_hz) + self.kc.append(float(channels.kc_spikes)) + for history in (self.trend, self.arousal, self.valence, self.kc): + del history[:-window] + + +def _z(value: float, history: list[float], center: bool) -> float: + if len(history) < 2: + return 0.0 + std = statistics.pstdev(history) + if std < 1e-9: + return 0.0 + mean = statistics.fmean(history) if center else 0.0 + return (value - mean) / std + + +@dataclass(frozen=True) +class Posture: + regime: str + spread_mult: float + size_mult: float + shift_bps: float + trend_z: float + arousal_z: float + valence_z: float + gate: bool + warm: bool # False while the baseline is still forming + side: str = "both" # "buy" or "sell" when a gated trend takes one away + confident: bool = True # False when the scene never reached the mushroom body + + def to_dict(self) -> dict: + return asdict(self) + + @classmethod + def from_dict(cls, data: dict) -> "Posture": + return cls(**data) + + +NEUTRAL = Posture( + regime="ranging", + spread_mult=1.0, + size_mult=1.0, + shift_bps=0.0, + trend_z=0.0, + arousal_z=0.0, + valence_z=0.0, + gate=False, + warm=False, +) + + +def classify(trend_z: float, arousal_z: float, gate: bool, s: DecoderSettings) -> str: + """Regime in precedence order: pause > volatile > trending > quiet > ranging.""" + if arousal_z >= s.z_pause: + return "pause" + if arousal_z >= s.z_regime: + return "volatile" + if gate and trend_z >= s.z_regime: + return "trending_up" + if gate and trend_z <= -s.z_regime: + return "trending_down" + if arousal_z <= -s.z_regime: + return "quiet" + return "ranging" + + +def decode(channels: Channels, baseline: Baseline, s: DecoderSettings) -> Posture: + """Push the observation into the baseline and return the posture. + + The baseline is mutated (that is the point: it is the rolling window).""" + baseline.push(channels, s.window) + if baseline.count < s.warmup: + return NEUTRAL + if s.center_bias: + trend_z = _z(channels.trend_hz, baseline.trend, center=True) + else: + trend_z = channels.trend_hz / RAW_TREND_UNIT_HZ + arousal_z = _z(channels.arousal_hz, baseline.arousal, center=True) + valence_z = _z(channels.valence_hz, baseline.valence, center=True) + kc_z = _z(float(channels.kc_spikes), baseline.kc, center=True) + gate = channels.gate_spikes >= 1 + # No Kenyon drive at all, or far below what this pair usually produces: the + # chart did not reach the mushroom body, so every z above is measuring the + # network's own noise rather than the picture. + confident = channels.kc_spikes > 0 and kc_z > s.z_kc_quiet + regime = classify(trend_z, arousal_z, gate, s) + spread_mult = min(s.spread_max, max(s.spread_min, 1 + s.spread_gain * arousal_z)) + size_mult = min( + s.size_max, + max(s.size_min, 1 + s.size_gain * arousal_z + s.valence_gain * valence_z), + ) + shift = max(-s.max_shift_bps, min(s.max_shift_bps, s.shift_gain_bps * trend_z)) + if not gate: + shift = 0.0 # no descending gate spike, no directional lean + # A trend strong enough to act on takes the other side off the book rather + # than pricing it a little worse. Quoting with the trend is the same + # direction the lean already expressed, carried to its conclusion. + side = "both" + if gate and abs(trend_z) >= s.z_side: + side = "buy" if trend_z > 0 else "sell" + return Posture( + regime=regime, + spread_mult=round(spread_mult, 4), + size_mult=round(size_mult, 4), + shift_bps=round(shift, 3), + trend_z=round(trend_z, 4), + arousal_z=round(arousal_z, 4), + valence_z=round(valence_z, 4), + gate=gate, + warm=True, + confident=confident, + side=side, + ) + + +@dataclass(frozen=True) +class Hysteresis: + min_apply_interval_sec: float = 300.0 + spread_delta: float = 0.15 + shift_delta_bps: float = 0.5 + + +def should_apply( + previous: Posture | None, + new: Posture, + last_apply_ts: float | None, + now: float, + h: Hysteresis, +) -> tuple[bool, str]: + """Rate-limit configuration changes to material posture moves.""" + if not new.confident: + # Holding is the conservative act: the last applied config stays, and + # the reason is recorded rather than a neutral posture being written + # as though the fly had decided on one. + return False, "kenyon drive below baseline — scene not seen" + if previous is None: + return True, "first posture" + if last_apply_ts is not None and now - last_apply_ts < h.min_apply_interval_sec: + return False, "apply cooldown" + if new.regime != previous.regime: + return True, f"regime {previous.regime} -> {new.regime}" + if abs(new.spread_mult - previous.spread_mult) >= h.spread_delta: + return True, f"spread_mult {previous.spread_mult} -> {new.spread_mult}" + if abs(new.shift_bps - previous.shift_bps) >= h.shift_delta_bps: + return True, f"shift_bps {previous.shift_bps} -> {new.shift_bps}" + return False, "no material change" diff --git a/agents/market_making_fly/flybrain/deps.py b/agents/market_making_fly/flybrain/deps.py new file mode 100644 index 000000000..6bf45ffe1 --- /dev/null +++ b/agents/market_making_fly/flybrain/deps.py @@ -0,0 +1,27 @@ +"""What the fly needs that Condor core does not. + +The MaleCNS release ships as Arrow feather files, so reading it needs pyarrow — +122 MB that no other part of Condor uses. It is an optional extra rather than a +core dependency, which means a Condor installed without it is a normal Condor +and only the fly is unavailable. Saying so plainly is the whole point of this +module: an ImportError from inside a vendored connectome loader tells an +operator nothing about what to do next. +""" + +from __future__ import annotations + +INSTALL = "uv sync --extra fly" + + +def require_pyarrow() -> None: + """Raise with the command to run, rather than an ImportError three frames + deep in a feather reader.""" + try: + import pyarrow # noqa: F401 + except ImportError as missing: # pragma: no cover - depends on the install + raise RuntimeError( + "Market Making Fly reads the MaleCNS connectome from Arrow feather " + f"files, which needs pyarrow. Install the extra with `{INSTALL}` " + "and run this again. Condor's other agents do not need it, which is " + "why it is not a core dependency." + ) from missing diff --git a/agents/market_making_fly/flybrain/fly3d.py b/agents/market_making_fly/flybrain/fly3d.py new file mode 100644 index 000000000..edf82d7cd --- /dev/null +++ b/agents/market_making_fly/flybrain/fly3d.py @@ -0,0 +1,560 @@ +"""A low-poly fly, built as meshes, for the run report. + +Stonkfly's dashboard renders its fly with Three.js — WebGL into an offscreen +canvas, blitted to a 2D canvas with smoothing off for the pixel look. A Condor +report cannot carry a 586 KB ES module: ``ReportBuilder`` has no raw-HTML or +script escape hatch, and its markdown is sanitized. What it does carry is +Plotly, serialized through ``pio.to_html`` with frames intact — and a Plotly 3D +scene is natively drag-to-orbit, which is the affordance that matters here. + +So the fly is ``Mesh3d`` geometry with a wing-flap animation, orbitable by +dragging, in the same palette as the chart the fly is shown. It is decoration +with one honest job: making it obvious at a glance which run you are looking at +and whether it is alive. +""" + +from __future__ import annotations + +import numpy as np + +DESK = "#1b2432" +DESK_EDGE = "#2b3950" +BEZEL = "#151d29" +KEYBOARD = "#222c3d" +MUG = "#c5254e" +CITY = "#121a27" + +# The chart's palette, so the report and the fly's own input agree. +BODY = "#c8d4e8" +THORAX = "#8fa3c4" +EYE = "#c5254e" +WING = "#5ce0d8" +LIMB = "#4a5a78" +GROUND = "#0a0e16" +ACCENT = "#c9f24d" + +FLAP_FRAMES = 10 +FLAP_DEGREES = 34.0 + + +def _ellipsoid( + center: tuple[float, float, float], + radii: tuple[float, float, float], + n_u: int = 18, + n_v: int = 11, +) -> tuple[np.ndarray, ...]: + """A closed ellipsoid as (x, y, z, i, j, k) for ``Mesh3d``. + + Each pole is ONE vertex with a triangle fan around it. Sweeping v through + 0 and pi on a full u grid instead would place n_u coincident vertices at + each pole, and the quads there collapse to zero-area slivers — which WebGL + renders as a white sawtooth along the body, even though a static export + hides it. + """ + rings = np.linspace(0, np.pi, n_v)[1:-1] + n_rings = len(rings) + u = np.linspace(0, 2 * np.pi, n_u, endpoint=False) + uu, vv = np.meshgrid(u, rings, indexing="ij") + x = np.append( + (radii[0] * np.sin(vv) * np.cos(uu) + center[0]).ravel(), + [center[0], center[0]], + ) + y = np.append( + (radii[1] * np.sin(vv) * np.sin(uu) + center[1]).ravel(), + [center[1], center[1]], + ) + z = np.append( + (radii[2] * np.cos(vv) + center[2]).ravel(), + [center[2] + radii[2], center[2] - radii[2]], + ) + north, south = n_u * n_rings, n_u * n_rings + 1 + faces_i, faces_j, faces_k = [], [], [] + for a in range(n_u): + b = (a + 1) % n_u # wrap around the waist + faces_i += [north, south] + faces_j += [b * n_rings, a * n_rings + n_rings - 1] + faces_k += [a * n_rings, b * n_rings + n_rings - 1] + for c in range(n_rings - 1): + p0, p1 = a * n_rings + c, b * n_rings + c + p2, p3 = b * n_rings + c + 1, a * n_rings + c + 1 + faces_i += [p0, p0] + faces_j += [p1, p2] + faces_k += [p2, p3] + return ( + x, + y, + z, + np.array(faces_i), + np.array(faces_j), + np.array(faces_k), + ) + + +def _rotate_x(y: np.ndarray, z: np.ndarray, pivot: tuple[float, float], degrees: float): + """Rotate (y, z) about a pivot — the wing hinge on the thorax.""" + rad = np.radians(degrees) + cy, cz = pivot + dy, dz = y - cy, z - cz + return ( + cy + dy * np.cos(rad) - dz * np.sin(rad), + cz + dy * np.sin(rad) + dz * np.cos(rad), + ) + + +def _mesh(xyzijk, color, opacity=1.0, name="", lighting=True): + import plotly.graph_objects as go + + x, y, z, i, j, k = xyzijk + return go.Mesh3d( + x=x, + y=y, + z=z, + i=i, + j=j, + k=k, + color=color, + opacity=opacity, + name=name, + flatshading=True, # low-poly: no smoothing between faces + hoverinfo="skip", + showscale=False, + lighting=( + dict(ambient=0.66, diffuse=0.42, specular=0.10, roughness=0.90) + if lighting + else dict(ambient=1.0, diffuse=0.0, specular=0.0) + ), + lightposition=dict(x=-60, y=-140, z=180), + ) + + +def _wing(side: int, degrees: float): + """One wing, hinged on the thorax and rotated by the flap angle.""" + # Swept back along the body and narrow, the way a resting fly holds them, + # so the wings read as wings rather than as two discs over the thorax. + x, y, z, i, j, k = _ellipsoid( + center=(-0.46, side * 0.40, 0.34), radii=(0.60, 0.21, 0.025), n_u=16, n_v=9 + ) + y, z = _rotate_x(y, z, pivot=(side * 0.12, 0.24), degrees=side * degrees) + return x, y, z, i, j, k + + +def _static_parts(): + """Everything that does not move: abdomen, thorax, head, eyes, legs.""" + import plotly.graph_objects as go + + parts = [ + _mesh( + _ellipsoid((-0.72, 0, 0.02), (0.62, 0.30, 0.29), n_u=26, n_v=15), + BODY, + name="abdomen", + ), + _mesh( + _ellipsoid((0.02, 0, 0.06), (0.40, 0.33, 0.32), n_u=24, n_v=14), + THORAX, + name="thorax", + ), + _mesh( + _ellipsoid((0.56, 0, 0.10), (0.27, 0.27, 0.26), n_u=22, n_v=13), + BODY, + name="head", + ), + _mesh( + _ellipsoid((0.66, 0.19, 0.17), (0.17, 0.13, 0.17), 12, 8), EYE, name="eye" + ), + _mesh( + _ellipsoid((0.66, -0.19, 0.17), (0.17, 0.13, 0.17), 12, 8), EYE, name="eye" + ), + ] + legs_x, legs_y, legs_z = [], [], [] + for hip, reach in ((0.26, 0.30), (-0.02, 0.02), (-0.30, -0.26)): + for side in (1, -1): + legs_x += [hip, hip + reach * 0.6, hip + reach, None] + legs_y += [side * 0.24, side * 0.52, side * 0.66, None] + legs_z += [-0.12, -0.48, -0.74, None] + parts.append( + go.Scatter3d( + x=legs_x, + y=legs_y, + z=legs_z, + mode="lines", + line=dict(color=LIMB, width=5), + hoverinfo="skip", + showlegend=False, + name="legs", + ) + ) + # Antennae + parts.append( + go.Scatter3d( + x=[0.70, 0.86, None, 0.70, 0.86], + y=[0.10, 0.16, None, -0.10, -0.16], + z=[0.30, 0.46, None, 0.30, 0.46], + mode="lines", + line=dict(color=LIMB, width=4), + hoverinfo="skip", + showlegend=False, + name="antennae", + ) + ) + return parts + + +def _box(x0, x1, y0, y1, z0, z1): + """An axis-aligned box as (x, y, z, i, j, k).""" + x = np.array([x0, x1, x1, x0, x0, x1, x1, x0], dtype=float) + y = np.array([y0, y0, y1, y1, y0, y0, y1, y1], dtype=float) + z = np.array([z0, z0, z0, z0, z1, z1, z1, z1], dtype=float) + i = np.array([0, 0, 4, 4, 0, 0, 1, 1, 2, 2, 3, 3]) + j = np.array([1, 2, 5, 6, 1, 5, 2, 6, 3, 7, 0, 4]) + k = np.array([2, 3, 6, 7, 5, 4, 6, 5, 7, 6, 4, 7]) + return x, y, z, i, j, k + + +def _cylinder(cx, cy, z0, z1, radius, sides=14): + """A capped cylinder — the mug.""" + a = np.linspace(0, 2 * np.pi, sides, endpoint=False) + rim_x, rim_y = cx + radius * np.cos(a), cy + radius * np.sin(a) + x = np.concatenate([rim_x, rim_x, [cx], [cx]]) + y = np.concatenate([rim_y, rim_y, [cy], [cy]]) + z = np.concatenate([np.full(sides, z0), np.full(sides, z1), [z0], [z1]]) + low_c, high_c = 2 * sides, 2 * sides + 1 + i, j, k = [], [], [] + for a0 in range(sides): + a1 = (a0 + 1) % sides + i += [a0, a0, low_c, high_c] + j += [a1, a1 + sides, a1, a0 + sides] + k += [a1 + sides, a0 + sides, a0, a1 + sides] + return x, y, z, np.array(i), np.array(j), np.array(k) + + +def _quantize(rgb: np.ndarray, colors: int = 10): + """Map the frame onto a small palette and return (index grid, palette). + + The chart is drawn from a fixed palette — background, header, grid, up, + down, volume, text — so six colours already cover ~99 % of it and the rest + is text antialiasing. That collapses to an exact stepped colorscale, which + is how a Plotly surface can show the real chart rather than a stand-in. + """ + # int32, not int16: a squared channel difference reaches 255**2 = 65025, + # which wraps in int16 and silently assigns pixels the wrong colour — + # rendering the chart with its background and header swapped. + flat = rgb.reshape(-1, 3).astype(np.int32) + unique, counts = np.unique(flat, axis=0, return_counts=True) + palette = unique[np.argsort(-counts)[:colors]] + # Nearest palette colour per pixel, in chunks so a full frame stays cheap. + index = np.empty(len(flat), dtype=np.int32) + for start in range(0, len(flat), 8192): + chunk = flat[start : start + 8192] + d = ((chunk[:, None, :] - palette[None, :, :]) ** 2).sum(-1) + index[start : start + 8192] = d.argmin(1) + return index.reshape(rgb.shape[:2]), palette + + +def _screen( + rgb: np.ndarray, + x: float, + y0: float, + y1: float, + z0: float, + z1: float, + width: int = 104, +): + """The monitor's glass: the fly's own input frame, as flat-shaded cells. + + Not a ``Surface``: that interpolates ``surfacecolor`` between grid points, + and a midpoint between the chart's light background and its grid lines + lands in the header's slot, drawing dark bars through the chart. ``Mesh3d`` + with ``intensitymode="cell"`` colours each face outright, so every pixel is + the colour it actually is. + """ + import plotly.graph_objects as go + from PIL import Image + + height = max(2, round(width * rgb.shape[0] / rgb.shape[1])) + small = np.asarray( + Image.fromarray(rgb).resize((width, height), Image.NEAREST), dtype=np.uint8 + ) + index, palette = _quantize(small) + n = len(palette) + scale = [] + for slot, colour in enumerate(palette): + css = f"rgb({colour[0]},{colour[1]},{colour[2]})" + scale += [[slot / n, css], [(slot + 1) / n, css]] + + # Seen from the fly's side (-x looking toward +x), image column 0 is at +y + # and row 0 at the top, so both axes run backwards from the array order. + ys = np.linspace(y1, y0, width + 1) + zs = np.linspace(z1, z0, height + 1) + grid_y, grid_z = np.meshgrid(ys, zs, indexing="xy") + vy, vz = grid_y.ravel(), grid_z.ravel() + stride = width + 1 + rows, cols = np.meshgrid(np.arange(height), np.arange(width), indexing="ij") + v00 = (rows * stride + cols).ravel() + v01, v10 = v00 + 1, v00 + stride + v11 = v10 + 1 + cell = index.ravel() + faces_i = np.concatenate([v00, v00]) + faces_j = np.concatenate([v01, v11]) + faces_k = np.concatenate([v11, v10]) + intensity = np.concatenate([cell, cell]).astype(float) + # Three quarters of the chart is its background. Painting that once as a + # single quad behind the mesh, and keeping only the faces that differ from + # it, drops this from ~12,000 triangles to ~3,000 without losing a pixel. + keep = intensity != 0.0 + faces_i, faces_j, faces_k, intensity = ( + faces_i[keep], + faces_j[keep], + faces_k[keep], + intensity[keep], + ) + backdrop = _mesh( + _box(x + 0.004, x + 0.006, min(y0, y1), max(y0, y1), min(z0, z1), max(z0, z1)), + f"rgb({palette[0][0]},{palette[0][1]},{palette[0][2]})", + lighting=False, + name="screen backdrop", + ) + + detail = go.Mesh3d( + x=np.full_like(vy, x), + y=vy, + z=vz, + i=faces_i, + j=faces_j, + k=faces_k, + intensity=(intensity + 0.5) / n, + intensitymode="cell", + cmin=0, + cmax=1, + colorscale=scale, + showscale=False, + flatshading=True, + hoverinfo="skip", + lighting=dict(ambient=1.0, diffuse=0.0, specular=0.0), + name="screen", + ) + return [backdrop, detail] + + +def desk_parts(chart: np.ndarray | None = None, scale: float = 0.52): + """The desk the fly works at: surface, monitor, keyboard, mug, speakers. + + Modelled on stonkfly's scene, including its proportions — the fly is the + size of the monitor, not of a person. ``scale`` shrinks the furniture about + the desk top, which stays where the fly's legs land. The monitor is the + point of it: the fly is looking at the same frame the report shows beside + it. + """ + import plotly.graph_objects as go + + top = -0.74 # where the fly's legs land + + def box(x0, x1, y0, y1, z0, z1): + return _box( + x0 * scale, + x1 * scale, + y0 * scale, + y1 * scale, + top + (z0 - top) * scale, + top + (z1 - top) * scale, + ) + + def up(z): + return top + (z - top) * scale + + parts = [ + _mesh(box(-3.6, 2.6, -3.3, 3.3, top - 0.18, top), DESK, name="desk"), + _mesh(box(2.52, 2.6, -3.3, 3.3, top, top + 0.06), DESK_EDGE, name="desk edge"), + _mesh(box(2.16, 2.30, -0.16, 0.16, top, top + 0.46), BEZEL, name="stand"), + _mesh(box(2.00, 2.44, -0.70, 0.70, top, top + 0.07), BEZEL, name="foot"), + _mesh( + box(2.20, 2.32, -2.10, 2.10, top + 0.40, top + 2.40), BEZEL, name="bezel" + ), + _mesh(box(0.95, 1.80, -1.15, 1.15, top, top + 0.08), KEYBOARD, name="keyboard"), + _mesh( + _cylinder(1.60 * scale, -1.85 * scale, top, up(top + 0.38), 0.22 * scale), + MUG, + name="mug", + ), + _mesh(box(2.00, 2.38, -2.95, -2.40, top, top + 0.80), BEZEL, name="speaker"), + _mesh(box(2.00, 2.38, 2.40, 2.95, top, top + 0.80), BEZEL, name="speaker"), + ] + # Silhouettes behind the desk, for the room the reference has. + for n, (yy, w, h) in enumerate( + ( + (-3.6, 0.8, 3.0), + (-2.2, 0.6, 4.2), + (-0.9, 0.7, 2.6), + (0.5, 0.6, 3.8), + (1.8, 0.9, 2.8), + (3.0, 0.7, 4.0), + ) + ): + parts.append( + _mesh( + box(3.5 + 0.10 * n, 3.8 + 0.10 * n, yy, yy + w, top, top + h), + CITY, + name="city", + ) + ) + if chart is not None: + parts.extend( + _screen( + chart, + 2.19 * scale, + -2.02 * scale, + 2.02 * scale, + up(top + 0.48), + up(top + 2.32), + ) + ) + else: + parts.append( + _mesh( + box(2.17, 2.20, -2.02, 2.02, top + 0.48, top + 2.32), + "#0d1622", + name="screen off", + ) + ) + parts.append( + go.Scatter3d( + x=[0.95 * scale, 1.80 * scale, None, -3.6 * scale, -3.6 * scale], + y=[-1.15 * scale, -1.15 * scale, None, -3.3 * scale, 3.3 * scale], + z=[up(top + 0.085), up(top + 0.085), None, up(top + 0.02), up(top + 0.02)], + mode="lines", + line=dict(color=WING, width=3), + opacity=0.7, + hoverinfo="skip", + showlegend=False, + name="glow", + ) + ) + return parts + + +def fly_figure( + title: str = "", + subtitle: str = "", + height: int = 460, + chart: np.ndarray | None = None, + desk: bool = True, +): + """An orbitable low-poly fly at its desk, wings beating when you press play. + + ``chart`` is the fly's own input frame; pass it and the monitor shows the + same picture the decoder was reading. + """ + import plotly.graph_objects as go + + static = (desk_parts(chart) if desk else []) + _static_parts() + angles = [ + FLAP_DEGREES * np.sin(2 * np.pi * n / FLAP_FRAMES) for n in range(FLAP_FRAMES) + ] + wing_traces = [ + _mesh(_wing(1, angles[0]), WING, opacity=0.34, name="wing"), + _mesh(_wing(-1, angles[0]), WING, opacity=0.34, name="wing"), + ] + fig = go.Figure(data=static + wing_traces) + + wing_index = [len(static), len(static) + 1] + fig.frames = [ + go.Frame( + name=f"f{n}", + traces=wing_index, + data=[ + _mesh(_wing(1, angle), WING, opacity=0.34), + _mesh(_wing(-1, angle), WING, opacity=0.34), + ], + ) + for n, angle in enumerate(angles) + ] + + hidden = dict( + showbackground=False, + showgrid=False, + zeroline=False, + showticklabels=False, + title="", + visible=False, + ) + fig.update_layout( + height=height, + margin=dict(l=0, r=0, t=40 if title else 8, b=8), + paper_bgcolor=GROUND, + plot_bgcolor=GROUND, + font=dict(color=BODY, family="monospace", size=11), + title=( + dict(text=title, x=0.02, xanchor="left", font=dict(color=ACCENT, size=13)) + if title + else None + ), + showlegend=False, + scene=dict( + xaxis=hidden, + yaxis=hidden, + zaxis=hidden, + bgcolor=GROUND, + aspectmode="data", + # Over the fly's shoulder, so both it and what it is looking at + # are in frame — the reference's own framing. + camera=dict( + eye=dict(x=-0.06, y=-1.10, z=0.38), + center=dict(x=0.22, y=0, z=-0.16), + up=dict(x=0, y=0, z=1), + ), + annotations=( + [ + dict( + showarrow=False, + x=-0.4, + y=0, + z=-1.05, + text=subtitle, + font=dict(color=LIMB, size=11, family="monospace"), + ) + ] + if subtitle + else [] + ), + ), + updatemenus=[ + dict( + type="buttons", + showactive=False, + x=0.01, + y=0.03, + xanchor="left", + yanchor="bottom", + bgcolor="#141c28", + bordercolor=LIMB, + font=dict(color=ACCENT, size=10, family="monospace"), + buttons=[ + dict( + label="▶ BEAT WINGS", + method="animate", + args=[ + None, + dict( + frame=dict(duration=55, redraw=True), + fromcurrent=True, + transition=dict(duration=0), + mode="immediate", + ), + ], + ), + dict( + label="❚❚ PAUSE", + method="animate", + args=[ + [None], + dict( + frame=dict(duration=0, redraw=False), + mode="immediate", + ), + ], + ), + ], + ) + ], + legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), + ) + return fig diff --git a/agents/market_making_fly/flybrain/guard.py b/agents/market_making_fly/flybrain/guard.py new file mode 100644 index 000000000..78d62612f --- /dev/null +++ b/agents/market_making_fly/flybrain/guard.py @@ -0,0 +1,233 @@ +"""Deterministic checks that can veto or halt, and never choose a posture. + +A ``Veto`` means "keep the previous config this tick". A ``Halt`` means "stop +the bots and stop the fly until a human passes ``resume_reviewed``"; a +*financial* halt (loss stop, loss-rate breaker) cannot be cleared that way at +all — a new run directory is needed, exactly stonkfly's rule. +""" + +from __future__ import annotations + +import math +from dataclasses import asdict, dataclass, field +from datetime import datetime, timezone + +from flybrain.posture import MarketSpec, take_profit_floor + +BPS = 1e-4 + + +class Veto(Exception): + pass + + +class Halt(Exception): + def __init__(self, reason: str, financial: bool): + super().__init__(reason) + self.reason = reason + self.financial = financial + + +@dataclass(frozen=True) +class GuardSettings: + max_applies_per_day: int = 48 + apply_price_tolerance: float = 0.005 + max_loss_quote: float = 0.0 # 0 → 4 % of the combined total_amount_quote + max_loss_per_volume_bps: float = 5.0 + loss_no_new_high_ticks: int = 25 + min_volume_for_loss_rate: float = 100.0 + max_apply_failures: int = 3 + closed_ticks_to_stop: int = 5 + + def __post_init__(self): + if self.max_applies_per_day < 1 or self.max_apply_failures < 1: + raise ValueError("positive limits required") + if not 0 < self.apply_price_tolerance <= 0.05: + raise ValueError("apply_price_tolerance must be in (0, 0.05]") + if self.max_loss_quote < 0 or self.max_loss_per_volume_bps <= 0: + raise ValueError("loss limits must be positive") + + +@dataclass +class GuardState: + """Persisted in state.json between ticks.""" + + applies_day: str = "" + applies_today: int = 0 + last_apply: dict[str, float] = field(default_factory=dict) # per pair + consecutive_failures: int = 0 + # None until the first tick with reported P&L: an unreported book has no + # high to fall from, and a first negative figure is not a drawdown. + session_high_net: float | None = None + ticks_since_high: int = 0 + closed_ticks: dict[str, int] = field(default_factory=dict) + halted: str | None = None + halt_financial: bool = False + + @classmethod + def from_dict(cls, data: dict | None) -> "GuardState": + return cls(**data) if data else cls() + + def to_dict(self) -> dict: + return asdict(self) + + +def _day(now: float) -> str: + return datetime.fromtimestamp(now, timezone.utc).strftime("%Y-%m-%d") + + +def check_not_halted(state: GuardState) -> None: + if state.halted: + raise Halt(state.halted, state.halt_financial) + + +def resume(state: GuardState, reviewed: bool) -> None: + """Clear a transient halt after review; refuse to clear a financial one.""" + if not state.halted: + return + if state.halt_financial: + raise Halt(f"financial halt cannot be cleared by review: {state.halted}", True) + if not reviewed: + raise Halt( + f"halted, pass resume_reviewed=true after review: {state.halted}", False + ) + state.halted = None + state.consecutive_failures = 0 + + +def check_market_open( + pair: str, book_open: bool, state: GuardState, s: GuardSettings +) -> bool: + """Veto while a book is closed. Returns True when the pair has been closed + for ``closed_ticks_to_stop`` ticks, meaning its bot should be stopped.""" + if book_open: + state.closed_ticks[pair] = 0 + return False + state.closed_ticks[pair] = state.closed_ticks.get(pair, 0) + 1 + if state.closed_ticks[pair] >= s.closed_ticks_to_stop: + return True + raise Veto(f"{pair}: book closed ({state.closed_ticks[pair]} ticks)") + + +def check_collateral(available_usd: float, required_usd: float) -> None: + if not math.isfinite(available_usd) or not math.isfinite(required_usd): + raise Veto("collateral figures are not finite") + if available_usd < required_usd: + raise Veto( + f"available collateral {available_usd:.2f} < required {required_usd:.2f}" + ) + + +def _spreads(value: str) -> list[float]: + return [float(x) for x in str(value).split(",") if x.strip()] + + +# A config that sits exactly on a floor reaches this check as a serialized +# decimal, and the float it parses back to can be one ulp under the float the +# floor computes to: 1.3 bp writes as 0.00013, which is less than 1.3 * 1e-4. +# Vetoing that is the guard refusing the posture builder's own arithmetic — it +# cost a real apply on 2026-09-13, tick 18. The tolerance is relative and far +# below any width that could matter. +FLOOR_TOLERANCE = 1e-9 + + +def check_config(config: dict, spec: MarketSpec) -> None: + """Belt to ``posture.build_config``'s braces: refuse anything below the floors.""" + for key in ("buy_spreads", "sell_spreads"): + for level in _spreads(config[key]): + if level < spec.min_spread_bps * BPS * (1 - FLOOR_TOLERANCE): + raise Veto(f"{key} level {level} below {spec.min_spread_bps} bp") + if float(config["take_profit"]) < take_profit_floor(spec) * (1 - FLOOR_TOLERANCE): + raise Veto("take_profit below fee floor") + if int(config["leverage"]) > spec.leverage_cap: + raise Veto(f"leverage {config['leverage']} above cap {spec.leverage_cap}") + if config["trading_pair"] != spec.trading_pair: + raise Veto("config pair does not match market spec") + if not config.get("global_sl_enabled"): + raise Veto("global stop loss must stay enabled") + + +def check_apply_window(state: GuardState, now: float, s: GuardSettings) -> None: + if state.applies_day != _day(now): + state.applies_day = _day(now) + state.applies_today = 0 + if state.applies_today >= s.max_applies_per_day: + raise Veto("daily apply limit") + + +def check_price_move(observed_mid: float, fresh_mid: float, s: GuardSettings) -> None: + for name, value in (("observed", observed_mid), ("fresh", fresh_mid)): + if not math.isfinite(value) or value <= 0: + raise Veto(f"{name} mid is not a positive finite price: {value!r}") + if abs(fresh_mid - observed_mid) / observed_mid > s.apply_price_tolerance: + raise Veto("price moved beyond neural observation tolerance") + + +def record_apply( + state: GuardState, pair: str, now: float, ok: bool, s: GuardSettings +) -> None: + state.applies_today += 1 + state.last_apply[pair] = now + if ok: + state.consecutive_failures = 0 + return + state.consecutive_failures += 1 + if state.consecutive_failures >= s.max_apply_failures: + state.halted = ( + f"{state.consecutive_failures} consecutive config update failures" + ) + state.halt_financial = False + raise Halt(state.halted, False) + + +def rebase(state: GuardState) -> None: + """Forget the session's high-water marks after a book restarted. + + A redeployed controller reports from zero. Measured against the previous + deployment's high, it would be halted for a drawdown that never happened. + """ + state.session_high_net = None + state.ticks_since_high = 0 + + +def check_pnl( + total_net: float, + volume: float, + state: GuardState, + s: GuardSettings, + max_loss_quote: float, +) -> None: + """Loss stop and the HIP-3 loss-rate breaker. ``total_net`` must include + unrealized P&L — realized alone can look fine while the open position bleeds.""" + if not math.isfinite(total_net) or not math.isfinite(volume): + raise Veto("P&L figures are not finite") + # Call only on ticks whose P&L is reported: breakers count reported ticks, + # never silence, and the first report sets the high. + if state.session_high_net is None or total_net > state.session_high_net: + state.session_high_net = total_net + state.ticks_since_high = 0 + else: + state.ticks_since_high += 1 + if total_net <= -max_loss_quote: + state.halted = f"loss stop: net {total_net:.2f} <= -{max_loss_quote:.2f}" + state.halt_financial = True + raise Halt(state.halted, True) + if volume >= s.min_volume_for_loss_rate: + rate_bps = total_net / volume * 1e4 + if rate_bps <= -s.max_loss_per_volume_bps: + state.halted = f"loss-rate breaker: {rate_bps:.1f} bp of volume" + state.halt_financial = True + raise Halt(state.halted, True) + if ( + state.ticks_since_high >= s.loss_no_new_high_ticks + and total_net < state.session_high_net + ): + state.halted = f"no new P&L high for {state.ticks_since_high} ticks" + state.halt_financial = True + raise Halt(state.halted, True) + + +def default_max_loss(specs: list[MarketSpec], s: GuardSettings) -> float: + if s.max_loss_quote > 0: + return s.max_loss_quote + return 0.04 * sum(spec.total_amount_quote for spec in specs) diff --git a/agents/market_making_fly/flybrain/market.py b/agents/market_making_fly/flybrain/market.py new file mode 100644 index 000000000..cde758e1a --- /dev/null +++ b/agents/market_making_fly/flybrain/market.py @@ -0,0 +1,487 @@ +"""What the fly loop reads from the world, and the one thing it writes. + +``LiveMarket`` talks to Hummingbot (candles, bot performance, portfolio, config +updates, stop) and reads the top of book from hummingbot-api, which serves any +CLOB connector it supports. HIP-3 pairs are the one exception: that endpoint +500s on them, so those fall back to Hyperliquid's public ``l2Book``. +``FixtureMarket`` is a deterministic offline stand-in for plumbing tests; it +never applies anything. +""" + +from __future__ import annotations + +import math +import re +import time +from dataclasses import dataclass + +import aiohttp +from flybrain.naming import pair_names +from flybrain.posture import MarketSpec +from flybrain.reinforcement import controller_net + +HL_INFO_URL = "https://api.hyperliquid.xyz/info" +# The deploy tool appends -YYYYMMDD-HHMMSS, and a redeploy that hands the +# running instance name back in stacks another one. +_SUFFIXED = re.compile(r"(?:-\d{8}-\d{6})+") + + +@dataclass(frozen=True) +class Observation: + pair: str + candles: list[dict] + bid: float + ask: float + open: bool # both sides of the live book present + + @property + def mid(self) -> float: + return (self.bid + self.ask) / 2 + + +@dataclass(frozen=True) +class Book: + bid: float | None + ask: float | None + + @property + def open(self) -> bool: + return self.bid is not None and self.ask is not None + + +def level(entry) -> tuple[float, float]: + """One book level as ``(price, size)``. Venues disagree on the shape: + Hyperliquid sends ``{"px", "sz"}``, hummingbot-api sends ``[price, qty]``, + and some connectors send ``{"price", "quantity"}``.""" + if isinstance(entry, dict): + price = entry.get("px", entry.get("price", entry.get("Price"))) + size = entry.get( + "sz", entry.get("quantity", entry.get("size", entry.get("amount"))) + ) + return float(price), float(size) + return float(entry[0]), float(entry[1]) + + +def depth_within( + bids: list[tuple[float, float]], + asks: list[tuple[float, float]], + within_bps: float, +) -> tuple[float, float, float]: + """``(bid_notional, ask_notional, spread_bps)`` inside ``within_bps`` of mid. + + What a market maker actually needs to know about a book: how wide the touch + is, and how much is resting close enough to trade against. Levels further + out than the band are not liquidity this strategy will ever see. + """ + if not bids or not asks: + return 0.0, 0.0, 0.0 + # Sorted here rather than assumed. The touch is the best price and the walk + # stops at the first level outside the band, so one out-of-order rung from + # a connector would hide every closer level behind it — understating depth + # and rejecting a market that was eligible. Sorting a book this size costs + # nothing next to the call that fetched it. + bids = sorted(bids, key=lambda level: -level[0]) + asks = sorted(asks, key=lambda level: level[0]) + best_bid, best_ask = bids[0][0], asks[0][0] + mid = (best_bid + best_ask) / 2 + if mid <= 0 or not math.isfinite(mid): + return 0.0, 0.0, 0.0 + spread_bps = (best_ask - best_bid) / mid * 1e4 + + def side(levels, is_bid): + total = 0.0 + for price, size in levels: + offset = (mid - price) / mid * 1e4 if is_bid else (price - mid) / mid * 1e4 + if offset > within_bps: + break # now genuinely sorted; nothing beyond is closer + total += price * size + return total + + return side(bids, True), side(asks, False), spread_bps + + +async def fetch_l2_levels( + session: aiohttp.ClientSession, coin: str +) -> tuple[list[tuple[float, float]], list[tuple[float, float]]]: + """Full bid/ask ladders from Hyperliquid's public book.""" + async with session.post( + HL_INFO_URL, + json={"type": "l2Book", "coin": coin}, + timeout=aiohttp.ClientTimeout(total=10), + ) as resp: + if resp.status != 200: + raise RuntimeError(f"l2Book {coin}: HTTP {resp.status}") + book = await resp.json() + levels = book.get("levels") if isinstance(book, dict) else None + if not levels or len(levels) != 2: + raise RuntimeError(f"l2Book {coin}: unexpected payload {str(book)[:120]}") + return [level(e) for e in levels[0]], [level(e) for e in levels[1]] + + +async def fetch_l2_book(session: aiohttp.ClientSession, coin: str) -> Book: + async with session.post( + HL_INFO_URL, + json={"type": "l2Book", "coin": coin}, + timeout=aiohttp.ClientTimeout(total=10), + ) as resp: + if resp.status != 200: + raise RuntimeError(f"l2Book {coin}: HTTP {resp.status}") + book = await resp.json() + levels = book.get("levels") if isinstance(book, dict) else None + if not levels or len(levels) != 2: + raise RuntimeError(f"l2Book {coin}: unexpected payload {str(book)[:120]}") + bids, asks = levels + if not bids or not asks: + return Book(None, None) # closed / empty book, a real state not an error + bid, ask = float(bids[0]["px"]), float(asks[0]["px"]) + if not (math.isfinite(bid) and math.isfinite(ask)) or bid <= 0 or ask < bid: + raise RuntimeError( + f"l2Book {coin}: invalid top of book bid={bid!r} ask={ask!r}" + ) + return Book(bid, ask) + + +def normalize_candle_payload(result) -> list[dict]: + records = ( + result + if isinstance(result, list) + else result.get("data", result.get("candles")) + ) + if not records: + raise RuntimeError("Candle feed returned no records") + return list(records) + + +class LiveMarket: + def __init__( + self, client, connector_name: str, candle_interval: str, n_candles: int + ): + self.client = client + self.connector_name = connector_name + self.candle_interval = candle_interval + self.n_candles = n_candles + + async def levels( + self, pair: str, depth: int = 50 + ) -> tuple[list[tuple[float, float]], list[tuple[float, float]]]: + """Bid/ask ladders, from whichever source serves this market. + + hummingbot-api covers every CLOB connector it supports. HIP-3 pairs are + the exception: its order-book endpoint 500s on them, so those go + straight to Hyperliquid's public one. That rule lives here alone, so + anything reading a book — the loop, the scanner — inherits it. + """ + names = pair_names(pair) + if names.hl_coin and "hyperliquid" in self.connector_name.lower(): + async with aiohttp.ClientSession() as session: + return await fetch_l2_levels(session, names.hl_coin) + raw = await self.client.market_data.get_order_book( + self.connector_name, pair, depth=depth + ) + if not isinstance(raw, dict): + raise RuntimeError(f"{pair}: unexpected order book payload") + return ( + [level(e) for e in (raw.get("bids") or [])], + [level(e) for e in (raw.get("asks") or [])], + ) + + async def book(self, pair: str) -> Book: + """Top of book. Source selection lives in :meth:`levels`.""" + bids, asks = await self.levels(pair, depth=1) + if not bids or not asks: + return Book(None, None) # closed / empty book, a real state + bid, ask = bids[0][0], asks[0][0] + if not (math.isfinite(bid) and math.isfinite(ask)) or bid <= 0 or ask < bid: + raise RuntimeError(f"{pair}: invalid top of book bid={bid!r} ask={ask!r}") + return Book(bid, ask) + + async def observe(self, pair: str) -> Observation: + candles = normalize_candle_payload( + await self.client.market_data.get_candles( + self.connector_name, + pair, + interval=self.candle_interval, + max_records=self.n_candles, + ) + ) + book = await self.book(pair) + if not book.open: + # The chart still exists; the loop decides what a closed book means. + last = float(candles[-1]["close"]) + return Observation(pair, candles, last, last, False) + return Observation(pair, candles, book.bid, book.ask, True) + + async def fresh_mid(self, pair: str) -> float: + book = await self.book(pair) + if not book.open: + raise RuntimeError(f"{pair}: book closed at apply time") + return (book.bid + book.ask) / 2 + + async def bots(self) -> dict: + resp = await self.client.bot_orchestration.get_active_bots_status() + raw = resp if isinstance(resp, dict) else {} + data = raw.get("data", raw) + return data if isinstance(data, dict) else {} + + @staticmethod + def find_bot(bots: dict, bot_name: str) -> tuple[str | None, dict | None]: + """The deploy tool suffixes instance names with ``-YYYYMMDD-HHMMSS``, + so ``orcl-fly`` runs as ``orcl-fly-20260913-055821``, and a redeploy + that hands the instance name back stacks another suffix. Match the + exact name or any number of those suffixes, and refuse an ambiguous + match: stopping the wrong bot is worse than stopping none.""" + matches = [ + name + for name in bots + if name == bot_name + or ( + name.startswith(bot_name) and _SUFFIXED.fullmatch(name[len(bot_name) :]) + ) + ] + if len(matches) > 1: + raise RuntimeError(f"several bots match {bot_name!r}: {sorted(matches)}") + if not matches: + return None, None + bot = bots[matches[0]] + return matches[0], (bot if isinstance(bot, dict) else None) + + async def equity( + self, pairs: list[str], carry: dict[str, dict] | None = None + ) -> tuple[float, float, dict, dict]: + """Combined ``realized + unrealized`` and volume across the fly's bots. + + ``carry`` holds the last figures each bot reported. A bot that has never + reported contributes nothing (a shadow run with no bot). A bot that + reported before and is now missing from the active list — stopped, + archived, or dropped from one status response — keeps contributing its + last known figures, frozen, so its result does not vanish from the + combined book and produce a fake equity jump. Returns + ``(net, volume, per_pair, carry)``; persist ``carry`` between ticks.""" + bots = await self.bots() + carry = {k: dict(v) for k, v in (carry or {}).items()} + net = 0.0 + volume = 0.0 + per_pair: dict[str, dict] = {} + for pair in pairs: + names = pair_names(pair) + _, bot = self.find_bot(bots, names.bot_name) + if bot is None: + previous = carry.get(pair) + if previous: + net += previous["net"] + volume += previous["volume"] + per_pair[pair] = {"running": False, "carried": True, **previous} + else: + per_pair[pair] = {"running": False} + continue + perf = (bot.get("performance") or {}).get(names.config_name) + if not isinstance(perf, dict): + # The bot is up but the API has no performance report for its + # controller (no MQTT report yet, or reporting broken). Keep the + # last known figures; the loop decides what that permits. + previous = carry.get(pair) + if previous: + net += previous["net"] + volume += previous["volume"] + per_pair[pair] = { + "running": True, + "reported": False, + **(previous or {}), + } + continue + inner = perf.get("performance", perf) + pair_net = controller_net(inner) + pair_volume = float(inner.get("volume_traded", 0) or 0) + net += pair_net + volume += pair_volume + previous = carry.get(pair) + per_pair[pair] = { + "running": True, + "reported": True, + "net": pair_net, + "volume": pair_volume, + } + if previous and pair_volume < previous["volume"]: + # Volume only accumulates within a controller instance, so this + # is a fresh one: a redeploy, not a collapse in P&L. + per_pair[pair]["restarted"] = True + carry[pair] = {"net": pair_net, "volume": pair_volume} + if not math.isfinite(net) or not math.isfinite(volume): + raise RuntimeError("Nonfinite bot performance") + return net, volume, per_pair, carry + + async def available_quote(self, quote_tokens: set[str]) -> float: + """Available balance in the quote assets these books trade against. + + A perp draws margin from its collateral asset; a spot book spends the + quote outright. Either way the figure that matters is what is free in + the asset the pair is denominated in, so the caller names it rather + than this assuming a venue's collateral token. + """ + state = await self.client.portfolio.get_state() + if not isinstance(state, dict): + raise RuntimeError("Portfolio state unavailable") + total = 0.0 + seen = False + for account in state.values(): + if not isinstance(account, dict): + continue + for connector, tokens in account.items(): + if self.connector_name not in connector or not isinstance(tokens, list): + continue + seen = True + for token in tokens: + if isinstance(token, dict) and token.get("token") in quote_tokens: + units = float(token.get("units", 0) or 0) + total += float(token.get("available_units", units) or 0) + if not seen: + raise RuntimeError(f"No {self.connector_name} balances in portfolio state") + return total + + async def apply(self, pair: str, config: dict) -> None: + """Update the saved config, then the live bot's controller. + + Durable layer first: if saving fails nothing has changed on the bot and + the tick is a clean error. If the live update then fails, the saved + config is ahead of the bot, the tick is an error, ``applied`` keeps the + old posture, and the next material posture retries both — the bot is + never left running a config the run state does not know about.""" + names = pair_names(pair) + running, _ = self.find_bot(await self.bots(), names.bot_name) + if running is None: + raise RuntimeError(f"bot {names.bot_name} is not running") + # The live update endpoint requires ``id`` equal to the config name. + payload = {"id": names.config_name, **config} + await self.client.controllers.create_or_update_controller_config( + names.config_name, payload + ) + await self.client.controllers.update_bot_controller_config( + running, names.config_name, payload + ) + + async def stop_bot(self, pair: str) -> bool: + names = pair_names(pair) + running, _ = self.find_bot(await self.bots(), names.bot_name) + if running is None: + return False + await self.client.bot_orchestration.stop_and_archive_bot(running) + return True + + +class FixtureMarket: + """Deterministic sine-wave candles; equity swings so both pulses fire.""" + + BASE = {0: 60.0, 1: 120.0, 2: 30.0} + + def __init__(self, pairs: list[str], n_candles: int): + self.pairs = pairs + self.n_candles = n_candles + self.tick = 0 + + def _price(self, index: int, k: int) -> float: + base = self.BASE[index % 3] + return base * (1 + 0.02 * math.sin(k * 0.35 + index)) + + async def observe(self, pair: str) -> Observation: + index = self.pairs.index(pair) + candles = [] + for k in range(self.tick, self.tick + self.n_candles): + o, c = self._price(index, k), self._price(index, k + 1) + candles.append( + { + "timestamp": k * 300, + "open": o, + "high": max(o, c) * 1.001, + "low": min(o, c) * 0.999, + "close": c, + "volume": 100 + 50 * math.sin(k * 0.7), + } + ) + self.tick += 1 + mid = candles[-1]["close"] + return Observation(pair, candles, mid * 0.9995, mid * 1.0005, True) + + async def fresh_mid(self, pair: str) -> float: + index = self.pairs.index(pair) + return self._price(index, self.tick + self.n_candles) + + async def equity( + self, pairs: list[str], carry: dict[str, dict] | None = None + ) -> tuple[float, float, dict, dict]: + """A fixture stands in for a full book that reports. + + Every pair is marked running and reported on purpose: an offline run + exists to exercise the reward/aversive path end to end, and a book that + never reports would pin the stimulus to ``none`` (see ``pnl_is_known``). + Nothing is applied from here either way — ``apply`` refuses. + + The aggregate swings so both pulses fire, drifts up so new highs recur, + and stays within ±2 bp of volume so the loss-rate breaker is not + exercised here; each pair carries an equal share of it. + """ + net = 2.0 * math.sin(self.tick * 1.3) + 0.05 * self.tick + volume = 10_000.0 * (self.tick + 1) + share_net, share_volume = net / len(pairs), volume / len(pairs) + per_pair = { + p: { + "running": True, + "reported": True, + "fixture": True, + "net": share_net, + "volume": share_volume, + } + for p in pairs + } + carry = {p: {"net": share_net, "volume": share_volume} for p in pairs} + return net, volume, per_pair, carry + + async def available_quote(self, quote_tokens: set[str]) -> float: + return 1e9 + + async def apply(self, pair: str, config: dict) -> None: + raise RuntimeError("FixtureMarket never applies a config") + + async def stop_bot(self, pair: str) -> bool: + return False + + +def book_restarted(per_pair: dict) -> list[str]: + """Books whose reported volume went backwards since the last tick. + + Volume only accumulates within one controller instance, so a drop means a + fresh instance reporting from zero — a redeploy. Its P&L is not a loss of + the difference, and the previous deployment's high-water mark is not a + height it has fallen from. + """ + return sorted(p for p, info in per_pair.items() if info.get("restarted")) + + +def pnl_is_known(per_pair: dict) -> bool: + """True when every running book reported its P&L this tick. + + The reinforcement pulse, the equity anchor and the financial breakers all + hang off this: an unreported book is silence, not a result. No running book + at all is also silence — a shadow run with no bot deployed has nothing to + learn from. A market that stands in for a full book (the fixture) must + therefore report, or it exercises none of that path. + """ + running = [info for info in per_pair.values() if info.get("running")] + return bool(running) and all(info.get("reported") for info in running) + + +def required_collateral(specs: list[MarketSpec]) -> float: + """Quote the books could need at their inventory cap. + + On a perp that is margin, so leverage divides it. On spot leverage is 1 by + construction, and the same expression is the quote actually spent. + """ + return sum(s.total_amount_quote * s.max_base_pct / s.leverage for s in specs) + + +def quote_tokens(specs: list[MarketSpec]) -> set[str]: + """The quote assets these books are denominated in.""" + return {pair_names(s.trading_pair).quote for s in specs} + + +def now() -> float: + return time.time() diff --git a/agents/market_making_fly/flybrain/naming.py b/agents/market_making_fly/flybrain/naming.py new file mode 100644 index 000000000..52b1f546c --- /dev/null +++ b/agents/market_making_fly/flybrain/naming.py @@ -0,0 +1,85 @@ +"""Derived names for a trading pair: the bot, its controller config, its book key. + +Two pair shapes are accepted, both uppercase as Hummingbot writes them: + +* ``BASE-QUOTE`` — any CLOB spot or perp market, e.g. ``SOL-USDT`` +* ``ISSUER:TOKEN-QUOTE`` — a Hyperliquid HIP-3 market, e.g. ``XYZ:ORCL-USD`` + +The derived slug carries the **whole** pair, not just its base token. Two +markets on the same token are otherwise the same bot: ``BTC-USDT`` and +``BTC-USDC`` both quoting under one name would have the fly read one book's +P&L for the other and update the wrong controller. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass + +# Docker container names must start alphanumeric and contain only +# [a-zA-Z0-9_.-]; the slug feeds one, so it is restricted to that. +_SAFE_SLUG = re.compile(r"[a-z0-9][a-z0-9.-]*") + + +@dataclass(frozen=True) +class Names: + pair: str # SOL-USDT | XYZ:ORCL-USD + base: str # SOL | ORCL + quote: str # USDT | USD + issuer: str # "" | xyz + slug: str # sol-usdt | xyz-orcl-usd + bot_name: str # sol-usdt-fly | xyz-orcl-usd-fly + config_name: str # sol_usdt_fly_mm | xyz_orcl_usd_fly_mm + hl_coin: str # "" | xyz:ORCL (Hyperliquid l2Book key) + + +def pair_names(pair: str) -> Names: + """Parse a pair into everything derived from it, or refuse.""" + if not isinstance(pair, str) or not pair: + raise ValueError("trading pair is required") + if pair != pair.upper(): + raise ValueError(f"trading pair must be uppercase, got {pair!r}") + if "-" not in pair: + raise ValueError( + f"trading pair must look like BASE-QUOTE or ISSUER:TOKEN-QUOTE, got {pair!r}" + ) + head, quote = pair.rsplit("-", 1) + issuer = "" + base = head + if ":" in head: + issuer, base = head.split(":", 1) + if ":" in base: + raise ValueError(f"trading pair has more than one issuer prefix: {pair!r}") + if not base or not quote or (":" in head and not issuer): + raise ValueError( + f"trading pair must look like BASE-QUOTE or ISSUER:TOKEN-QUOTE, got {pair!r}" + ) + slug = pair.lower().replace(":", "-").replace("-", "-") + if not _SAFE_SLUG.fullmatch(slug): + raise ValueError(f"trading pair {pair!r} does not make a usable bot name") + return Names( + pair=pair, + base=base, + quote=quote, + issuer=issuer.lower(), + slug=slug, + bot_name=f"{slug}-fly", + config_name=f"{slug.replace('-', '_').replace('.', '_')}_fly_mm", + # Hyperliquid keys a HIP-3 book by lowercase issuer + uppercase token, + # with no quote suffix. A non-HIP-3 pair has no such key. + hl_coin=f"{issuer.lower()}:{base}" if issuer else "", + ) + + +def parse_pairs(value: str, limit: int = 3) -> list[str]: + pairs = [p.strip() for p in value.split(",") if p.strip()] + if not pairs: + raise ValueError("At least one pair is required") + if len(pairs) > limit: + raise ValueError(f"At most {limit} pairs, got {len(pairs)}") + if len(set(pairs)) != len(pairs): + raise ValueError("Duplicate pair") + slugs = [pair_names(p).slug for p in pairs] + if len(set(slugs)) != len(slugs): + raise ValueError(f"Pairs collide on their derived bot name: {sorted(slugs)}") + return pairs diff --git a/agents/market_making_fly/flybrain/neural/THIRD_PARTY.md b/agents/market_making_fly/flybrain/neural/THIRD_PARTY.md new file mode 100644 index 000000000..e86bf9716 --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/THIRD_PARTY.md @@ -0,0 +1,31 @@ +# Third-party notice + +`agents/market_making_fly/flybrain/neural/` and `agents/market_making_fly/flybrain/data.py` are vendored from +https://github.com/nftechie/stonkfly (MIT License, Copyright (c) 2026 nftechie +and DOOMFLY contributors). Only `neural/common.py` is modified: the data +directory resolves to Condor's runtime root instead of `./data`. + +The MaleCNS v1.0 connectome files it downloads retain their upstream license; +see the stonkfly repository's THIRD_PARTY.md. + +MIT License + +Copyright (c) 2026 nftechie and DOOMFLY contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/agents/market_making_fly/flybrain/neural/__init__.py b/agents/market_making_fly/flybrain/neural/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/agents/market_making_fly/flybrain/neural/arrays.lock.json b/agents/market_making_fly/flybrain/neural/arrays.lock.json new file mode 100644 index 000000000..5e855fd4e --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/arrays.lock.json @@ -0,0 +1,13 @@ +{ + "ptr": "a29aa0296125a5daa972c12d17711360d01a1209cd9cd9d7ccb1f6959224bcaf", + "post": "4cc5e857d3481f433576f35ddb8d580cc33cf8e22b6651fa03e08cde5d956d90", + "weight": "d95171c2d59cfe3ebef2e5bd52be779b8f74754d718fc8941a93b8f2921c632b", + "ids": "6b6b40c3bddf84b1281ef0b18db06927c2bf61c5f4cb32a4219ec9b7839dc2e5", + "retina": "5ec145156d998f582e5d8f08c4174e10f7f853d34745118adab8589f192d04b4", + "uv": "51f7dafd4355a06176418e22373ffcdf5bc066f75050da6dcbaaaccbad55a714", + "confidence": "4ecf882056a959ebb9053f89b766eeae7cf05aab4d65beb4d3d88a034dc7061a", + "hexes": "315ab7bd1abc1ef0307e90f8fff484020cd556b8d2d877f84773c912bcf6f2ca", + "lamina": "22d6fa8ac7aa40700d09394aea1cf35226711d2efc78bc716882920ba180e47e", + "sugar": "89b057c7b57d5e783f608815b4219b17b54c812c3fa229bcb78b1cafa30d37a1", + "superclass": "3977d971fe3ef65ce97c5c9c5d6dcf0aea893bebc55311c8c886ebe0f5756d13" +} diff --git a/agents/market_making_fly/flybrain/neural/brain.py b/agents/market_making_fly/flybrain/neural/brain.py new file mode 100644 index 000000000..421badd8d --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/brain.py @@ -0,0 +1,446 @@ +"""Full-graph candidate memory dynamics with explicit stimulation and checkpoints.""" + +import ctypes as C +import hashlib +import json +import math +import subprocess +import sys +import time +from pathlib import Path + +import numpy as np + +from .circuit import identify +from .common import DATA, GRAPH, OUT, digest, save_json +from .state import NativeBrain + +SOURCE = Path(__file__).with_name("kernel.cpp") +LIBRARY = (OUT / "physiology-v6") / ( + "libmemory.dylib" if sys.platform == "darwin" else "libmemory.so" +) +MODEL = "stonkfly-dual-compartment-v1" +from .rule import PARAMETERS as RULE_PARAMETERS + +PARAMETERS = { + **RULE_PARAMETERS, + "neural_dt_ms": 0.1, + "modulator_delivery_trace_ms": 100.0, + "kc_rest_mV": -60.0, + "kc_adaptation_jump_mV": 8.0, + "kc_adaptation_tau_ms": 200.0, + "interpretation": "Candidate KC adaptation/rest plus a baseline-centered anti-Hebbian rate-rule extension to two compartments. No fitted DAN/MBON background current; lamina bias is a display proxy. Gain, trace constants and transfer to this graph remain unvalidated assumptions.", +} + + +def build(): + sha = hashlib.sha256(SOURCE.read_bytes()).hexdigest() + metadata = LIBRARY.with_suffix(LIBRARY.suffix + ".json") + if LIBRARY.exists() and metadata.exists(): + record = json.loads(metadata.read_text()) + if ( + record["source_sha256"] == sha + and record["binary_sha256"] + == hashlib.sha256(LIBRARY.read_bytes()).hexdigest() + ): + return record + LIBRARY.parent.mkdir(parents=True, exist_ok=True) + temp = LIBRARY.with_suffix(LIBRARY.suffix + ".partial") + subprocess.run( + ["c++", "-O3", "-std=c++17", "-shared", "-fPIC", str(SOURCE), "-o", str(temp)], + check=True, + ) + temp.replace(LIBRARY) + record = { + "model": MODEL, + "source_sha256": sha, + "binary_sha256": hashlib.sha256(LIBRARY.read_bytes()).hexdigest(), + "flags": ["-O3", "-std=c++17", "-shared", "-fPIC"], + } + save_json(metadata, record) + return record + + +class MemoryBrain(NativeBrain): + def __init__( + self, + path=GRAPH, + *, + eta=0.001, + circuit=None, + modulation_mask=None, + tonic=None, + dan_baseline_hz=None, + kc_rest=-60.0, + adaptation_jump=8.0, + adaptation_tau=200.0, + ): + super().__init__(path) + self.build = build() + self.library = C.CDLL(str(LIBRARY)) + self.advance = self.library.memory_advance + self.advance.argtypes = ( + [C.c_int] + + [C.c_void_p] * 11 + + [C.c_int, C.c_float] + + [C.c_void_p] * 5 + + [C.c_void_p, C.c_void_p, C.c_void_p, C.c_void_p, C.c_int] + + [C.c_void_p] * 4 + + [ + C.c_float, + C.c_float, + C.c_float, + C.c_int, + C.c_void_p, + C.c_void_p, + C.c_void_p, + C.c_void_p, + C.c_void_p, + C.c_float, + C.c_float, + ] + ) + self.advance.restype = None + self.circuit = identify(self) if circuit is None else circuit + if not math.isfinite(kc_rest) or not -80 <= kc_rest <= -45: + raise ValueError("Invalid KC resting potential") + self.rest = np.full(self.n, -52.0, dtype=np.float32) + self.rest[self.circuit["kc"]] = kc_rest + self.v[:] = self.rest + if ( + not math.isfinite(adaptation_jump) + or adaptation_jump < 0 + or not math.isfinite(adaptation_tau) + or adaptation_tau <= 20 + ): + raise ValueError("Invalid adaptation parameters") + self.adaptation = np.zeros(self.n, dtype=np.float32) + self.adaptation_jump = float(adaptation_jump) + self.adaptation_tau = float(adaptation_tau) + if modulation_mask is None: + import pyarrow.feather as feather + + neurons = ( + feather.read_table(DATA / "normalized/neurons.feather") + .to_pandas() + .set_index("source_id") + .loc[self.ids] + ) + modulation_mask = neurons.neurotransmitter.isin( + ["dopamine", "octopamine", "serotonin"] + ).to_numpy(dtype=np.uint8) + self.modulation_mask = np.asarray(modulation_mask, dtype=np.uint8).copy() + if self.modulation_mask.shape != (self.n,) or np.any(self.modulation_mask > 1): + raise ValueError("Invalid modulation mask") + self.eta = float(eta) + if not math.isfinite(self.eta) or self.eta < 0: + raise ValueError("Finite nonnegative eta required") + self.eligibility = np.zeros(self.n, dtype=np.float64) + self.eligibility_last = np.zeros(self.n, dtype=np.int64) + self.modulation = np.zeros(self.n, dtype=np.float32) + self.modulation_last = np.zeros(self.n, dtype=np.int64) + self.baseline_plastic = self.weight[self.circuit["edges"]].copy() + self.initial_weight_sha256 = digest(self.weight) + self.fields = [ + "v", + "g", + "refractory", + "drive", + "previous_drive", + "queue", + "queue_count", + "counts", + "luminance", + "active", + "active_flag", + "nactive", + "last", + "eligibility", + "eligibility_last", + "modulation", + "modulation_last", + "adaptation", + ] + self.initial = {k: getattr(self, k).copy() for k in self.fields} + from .rule import PARAMETERS as RULE_PARAMETERS + + self.rule_parameters = RULE_PARAMETERS.copy() + self.rate_kc = np.zeros(len(self.circuit["edges"]), dtype=np.float64) + self.rate_dan = np.zeros(len(self.circuit["dan"]), dtype=np.float64) + self.memory_u = np.zeros_like(self.rate_kc) + self.memory_w = np.zeros_like(self.rate_kc) + self.tonic = ( + np.zeros(self.n, dtype=np.float32) + if tonic is None + else np.asarray(tonic, dtype=np.float32).copy() + ) + self.dan_baseline_hz = ( + np.zeros(len(self.circuit["dan"]), dtype=np.float64) + if dan_baseline_hz is None + else np.asarray(dan_baseline_hz, dtype=np.float64).copy() + ) + if self.tonic.shape != (self.n,) or not np.isfinite(self.tonic).all(): + raise ValueError("Invalid tonic current") + if ( + self.dan_baseline_hz.shape != (len(self.rate_dan),) + or not np.isfinite(self.dan_baseline_hz).all() + ): + raise ValueError("Invalid DAN baseline") + self.weights_frozen = False + for k in ["rate_kc", "rate_dan", "memory_u", "memory_w"]: + self.fields.append(k) + self.initial[k] = getattr(self, k).copy() + + def reset(self, keep_memory=False): + if keep_memory: + saved = (self.memory_u.copy(), self.memory_w.copy()) + for k, v in self.initial.items(): + getattr(self, k)[:] = v + self.cursor = 0 + self.sim_ms = 0.0 + self.total_spikes = 0 + if not keep_memory: + self.weight[self.circuit["edges"]] = self.baseline_plastic + else: + self.memory_u[:], self.memory_w[:] = saved + + def _neural_step( + self, + luminance, + duration_ms, + *, + learning=False, + stimulation=None, + lamina_bias=12.0, + ): + light = np.asarray(luminance) + if light.shape != (len(self.retina),) or not np.isfinite(light).all(): + raise ValueError("Invalid retinal input") + steps = round(duration_ms / self.dt) + if ( + not math.isfinite(duration_ms) + or steps < 1 + or not math.isfinite(lamina_bias) + ): + raise ValueError("Invalid interval/current") + self.luminance += (1 - math.exp(-steps * self.dt / 10)) * ( + np.clip(light, 0, 1) - self.luminance + ) + self.drive.fill(0) + self.drive[self.lamina] = lamina_bias + self.drive[self.retina] = 30 * self.luminance / (0.02 + self.luminance) + self.drive += self.tonic + if stimulation is not None: + pulses = stimulation if isinstance(stimulation, list) else [stimulation] + for indices, current in pulses: + ix = np.asarray(indices, dtype=np.int32) + amplitude = np.asarray(current, dtype=np.float32) + if ( + ix.ndim != 1 + or np.any(ix < 0) + or np.any(ix >= self.n) + or not np.isfinite(amplitude).all() + or amplitude.shape not in [(), ix.shape] + ): + raise ValueError("Invalid external stimulation") + self.drive[ix] += amplitude + self.counts.fill(0) + clock = np.asarray([self.cursor], dtype=np.int64) + c = self.circuit + arrays = [ + self.ptr, + self.post, + self.weight, + self.v, + self.g, + self.refractory, + self.drive, + self.previous_drive, + self.queue, + self.queue_count, + clock, + ] + start = time.perf_counter() + self.advance( + self.n, + *[x.ctypes.data for x in arrays], + steps, + self.dt, + *[ + getattr(self, k).ctypes.data + for k in ["counts", "active", "active_flag", "nactive", "last"] + ], + c["kc_mask"].ctypes.data, + c["dan_index"].ctypes.data, + self.eligibility.ctypes.data, + self.eligibility_last.ctypes.data, + len(c["edges"]), + c["edges"].ctypes.data, + c["pre"].ctypes.data, + self.baseline_plastic.ctypes.data, + c["gain"].ctypes.data, + self.eta, + PARAMETERS["trace_kc_seconds"] * 1000, + PARAMETERS["minimum_fraction"], + int(learning), + self.modulation.ctypes.data, + self.modulation_last.ctypes.data, + self.modulation_mask.ctypes.data, + self.rest.ctypes.data, + self.adaptation.ctypes.data, + self.adaptation_jump, + self.adaptation_tau, + ) + elapsed = time.perf_counter() - start + self.cursor = int(clock[0]) + self.sim_ms = self.cursor * self.dt + self.total_spikes += int(self.counts.sum()) + return self.counts.copy(), elapsed + + def step( + self, + luminance, + duration_ms, + *, + learning=False, + stimulation=None, + lamina_bias=12.0, + ): + from .rule import advance + + if not math.isfinite(duration_ms) or duration_ms <= 0: + raise ValueError("Invalid duration") + remaining = round(duration_ms / self.dt) + if remaining < 1: + raise ValueError("Duration too short") + total = np.zeros(self.n, dtype=np.int32) + wall = 0.0 + while remaining: + ticks = min(100, remaining) + interval = ticks * self.dt + # The original LTD update is disabled. Only the centered rule below + # writes candidate memory efficacies; all neural integration remains. + c, t = self._neural_step( + luminance, + interval, + learning=False, + stimulation=stimulation, + lamina_bias=lamina_bias, + ) + seconds = interval / 1000 + advance( + self.rate_kc, + self.rate_dan, + self.memory_u, + self.memory_w, + c[self.circuit["pre"]] / seconds, + c[self.circuit["dan"]] / seconds - self.dan_baseline_hz, + self.circuit["gain"], + seconds, + self.eta, + learning, + self.weights_frozen, + ) + if not self.weights_frozen: + self.weight[self.circuit["edges"]] = self.baseline_plastic * ( + 1 + self.memory_w + ) + total += c + wall += t + remaining -= ticks + self.counts[:] = total + return total, wall + + def memory(self): + w = self.weight[self.circuit["edges"]] + fraction = w / self.baseline_plastic + return { + "plastic_edges": len(w), + "changed_edges": int(np.count_nonzero(w != self.baseline_plastic)), + "mean_efficacy": float(fraction.mean()), + "minimum_efficacy": float(fraction.min()), + "sha256": digest(w), + "model": MODEL, + } + + def checkpoint(self, path): + metadata = { + "model": MODEL, + "build": self.build, + "eta": self.eta, + "parameters": PARAMETERS, + "cursor": self.cursor, + "weights_frozen": self.weights_frozen, + "total_spikes": self.total_spikes, + "graph_ids_sha256": digest(self.ids), + "graph_ptr_sha256": digest(self.ptr), + "graph_post_sha256": digest(self.post), + "plastic_edges_sha256": digest(self.circuit["edges"]), + "configuration_sha256": self.configuration_signature(), + } + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_suffix(".partial") + with temporary.open("wb") as handle: + np.savez_compressed( + handle, + metadata=json.dumps(metadata), + weight=self.weight, + **{k: getattr(self, k) for k in self.fields}, + ) + temporary.replace(path) + + def restore(self, path): + with np.load(path, allow_pickle=False) as a: + m = json.loads(str(a["metadata"])) + expected = { + "model": MODEL, + "build": self.build, + "eta": self.eta, + "parameters": PARAMETERS, + "graph_ids_sha256": digest(self.ids), + "graph_ptr_sha256": digest(self.ptr), + "graph_post_sha256": digest(self.post), + "plastic_edges_sha256": digest(self.circuit["edges"]), + "configuration_sha256": self.configuration_signature(), + } + if any(m.get(k) != v for k, v in expected.items()): + raise ValueError("Checkpoint provenance mismatch") + for k in ["weight", *self.fields]: + if ( + a[k].shape != getattr(self, k).shape + or a[k].dtype != getattr(self, k).dtype + ): + raise ValueError("Checkpoint array mismatch") + if a[k].dtype.kind == "f" and not np.isfinite(a[k]).all(): + raise ValueError("Nonfinite checkpoint state") + for k in ["weight", *self.fields]: + getattr(self, k)[:] = a[k] + self.cursor = int(m["cursor"]) + self.sim_ms = self.cursor * self.dt + self.total_spikes = int(m["total_spikes"]) + self.weights_frozen = bool(m["weights_frozen"]) + + def configuration_signature(self): + # Equal cell IDs and CSR endpoints alone do not imply equal input + # geometry, original efficacies or compartment assignment. + return { + "initial_weight": self.initial_weight_sha256, + "modulation_mask": digest(self.modulation_mask), + "rule": self.rule_parameters, + "rule_sha256": hashlib.sha256( + Path(__file__).with_name("rule.py").read_bytes() + ).hexdigest(), + "tonic": digest(self.tonic), + "dan_baseline_hz": digest(self.dan_baseline_hz), + "rest": digest(self.rest), + "adaptation_jump": self.adaptation_jump, + "adaptation_tau": self.adaptation_tau, + **{ + k: digest(getattr(self, k)) for k in ["retina", "uv", "lamina", "sugar"] + }, + **{ + k: digest(self.circuit[k]) + for k in ["pre", "gain", "kc_mask", "dan_index"] + }, + } diff --git a/agents/market_making_fly/flybrain/neural/circuit.py b/agents/market_making_fly/flybrain/neural/circuit.py new file mode 100644 index 000000000..d7a4196a7 --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/circuit.py @@ -0,0 +1,87 @@ +"""Two explicit, literature-motivated memory compartments in MaleCNS v1.0. + +PAM11 → alpha1 / MBON07 and PPL101 → gamma1pedc / MBON11. +Using the same centered rule in both is a new, unvalidated model assumption. +""" + +import numpy as np + +from .common import annotations, digest + + +def identify(brain): + a = annotations(brain.ids) + types = a.type.fillna("") + kc = np.flatnonzero(types.str.startswith("KC")).astype(np.int32) + reward = np.flatnonzero(types.eq("PAM11")).astype(np.int32) + aversive = np.flatnonzero(types.eq("PPL101")).astype(np.int32) + reward_mb = np.flatnonzero(types.eq("MBON07")).astype(np.int32) + aversive_mb = np.flatnonzero(types.eq("MBON11")).astype(np.int32) + if (len(reward), len(aversive), len(reward_mb), len(aversive_mb)) != (15, 2, 4, 2): + raise ValueError("Unexpected cell identities/counts for MaleCNS v1.0") + dan = np.r_[reward, aversive].astype(np.int32) + mb = np.r_[reward_mb, aversive_mb].astype(np.int32) + edges = np.flatnonzero(np.isin(brain.post, mb)).astype(np.int64) + pre = (np.searchsorted(brain.ptr, edges, side="right") - 1).astype(np.int32) + keep = np.isin(pre, kc) + edges = edges[keep] + pre = pre[keep] + if not len(edges) or np.any(brain.weight[edges] <= 0): + raise ValueError("Invalid reconstructed KC inputs") + gains = np.zeros((len(dan), len(edges)), dtype=np.float32) + for targets, drivers in [(reward_mb, reward), (aversive_mb, aversive)]: + for target in targets: + contact = [] + for d in drivers: + sl = slice(brain.ptr[d], brain.ptr[d + 1]) + contact.append( + float(np.abs(brain.weight[sl][brain.post[sl] == target]).sum()) + ) + contact = np.asarray(contact) + if contact.sum() <= 0: + raise ValueError("Missing direct DAN-to-MBON anatomical support") + selected = np.flatnonzero(brain.post[edges] == target) + for d, value in zip(drivers, contact / contact.sum()): + gains[np.flatnonzero(dan == d)[0], selected] = value + mask = np.zeros(brain.n, dtype=np.uint8) + mask[kc] = 1 + dan_index = np.full(brain.n, -1, dtype=np.int8) + dan_index[dan] = np.arange(len(dan)) + + def cells(ix): + return [ + { + "index": int(i), + "id": str(brain.ids[i]), + "type": str(types.iloc[i]), + "instance": str(a.instance.iloc[i]), + } + for i in ix + ] + + report = { + "release": "MaleCNS v1.0", + "neurons": brain.n, + "directed_edges": len(brain.post), + "plastic_edges": len(edges), + "plastic_edges_sha256": digest(edges), + "reward_cells": cells(reward), + "aversive_cells": cells(aversive), + "memory_outputs": cells(mb), + "selection": "All existing KC-to-MBON07/11 connections; no graph cropping or added edges.", + "gain": "Within-compartment DAN-to-MBON contact fractions; not measured receptor/dopamine kinetics.", + "validated": False, + } + return { + "kc": kc, + "mb": mb, + "dan": dan, + "reward": reward, + "aversive": aversive, + "edges": edges, + "pre": pre, + "gain": gains, + "kc_mask": mask, + "dan_index": dan_index, + "report": report, + } diff --git a/agents/market_making_fly/flybrain/neural/common.py b/agents/market_making_fly/flybrain/neural/common.py new file mode 100644 index 000000000..07bac74ed --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/common.py @@ -0,0 +1,40 @@ +"""Local verified dataset and build cache; never a dependency on another repo.""" + +import hashlib +import json +import os +from pathlib import Path + +# Condor: the connectome data lives in this agent's writable home +# (``.condor/agents/market_making_fly/data``), overridable with ``CONDOR_FLY_DATA``. +# This is the only edit to the vendored stonkfly code. +from condor.memory.paths import agent_home + +DATA = Path( + os.environ.get("CONDOR_FLY_DATA") or (agent_home("market_making_fly") / "data") +).resolve() +GRAPH = DATA / "graph.npz" +OUT = DATA / "cache" + + +def digest(array): + return hashlib.sha256(array.tobytes()).hexdigest() + + +def save_json(path, value): + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(path.suffix + ".partial") + tmp.write_text(json.dumps(value, indent=2, allow_nan=False) + "\n") + tmp.replace(path) + + +def annotations(ids): + import pyarrow.feather as f + + return ( + f.read_table(DATA / "annotations.feather") + .to_pandas() + .set_index("bodyId") + .loc[ids] + ) diff --git a/agents/market_making_fly/flybrain/neural/connectome.py b/agents/market_making_fly/flybrain/neural/connectome.py new file mode 100644 index 000000000..f7c941b38 --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/connectome.py @@ -0,0 +1,288 @@ +"""Loss-accounted import of the retained MaleCNS v1.0 graph for Stonkfly. + +This module stores topology and annotations. It does not infer missing dynamics, +neurotransmitter receptors, muscle mappings, or behavior from a wiring graph. +""" + +import argparse +import hashlib +import json +from pathlib import Path + +import numpy as np + +from .common import DATA + +REGISTRY = Path(__file__).with_name("datasets.json") + + +def exact_ids(values) -> np.ndarray: + """Never round 64-bit biological IDs through floating point or JavaScript.""" + items = np.asarray(values) + if items.dtype.kind == "f": + raise ValueError( + "Neuron IDs must be integers or decimal strings, never floats." + ) + if items.dtype.kind in "iu": + if np.any(items < 0): + raise ValueError("Neuron IDs cannot be negative.") + return items.astype(np.uint64) + text = [str(value) for value in items] + if any(not value.isascii() or not value.isdecimal() for value in text): + raise ValueError("Neuron IDs must be nonnegative decimal integers.") + return np.asarray(text, dtype=np.uint64) + + +def index_edges(ids: np.ndarray, pre, post, counts): + """Return retained edges plus a mask accounting for every excluded row.""" + if not len(ids) or np.any(ids[1:] <= ids[:-1]): + raise ValueError("Node IDs must be nonempty, unique, and sorted.") + pre, post = exact_ids(pre), exact_ids(post) + counts = np.asarray(counts) + if len(pre) != len(post) or len(pre) != len(counts): + raise ValueError("Edge columns have different lengths.") + if ( + not np.all(np.isfinite(counts)) + or np.any(counts < 1) + or np.any(counts != np.floor(counts)) + or np.any(counts > 2**32 - 1) + ): + raise ValueError("Synapse counts must be positive uint32-compatible integers.") + i, j = np.searchsorted(ids, pre), np.searchsorted(ids, post) + keep = (i < len(ids)) & (j < len(ids)) + keep &= ids[np.minimum(i, len(ids) - 1)] == pre + keep &= ids[np.minimum(j, len(ids) - 1)] == post + return ( + i[keep].astype(np.uint32), + j[keep].astype(np.uint32), + counts[keep].astype(np.uint32), + keep, + ) + + +def file_digest(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + while chunk := handle.read(8 * 1024 * 1024): + digest.update(chunk) + return digest.hexdigest() + + +def normalize_nodes(dataset_id: str, frame, nt_frame=None): + """Normalize fields while keeping the original annotations in raw files.""" + import pandas as pd + + if dataset_id == "malecns_v1": + source = exact_ids(frame.bodyId) + retain = frame.superclass.notna() & frame.superclass.astype(str).ne("") + quality = frame.statusLabel.astype(object).fillna("unknown") + superclasses, types = frame.superclass, frame.type + nonneural = frame.status.eq("Glia") + retain = retain & ~nonneural + predicted = pd.Series([None] * len(frame)) + nt_source = pd.Series(["missing"] * len(frame)) + if nt_frame is not None: + nt_frame = nt_frame.set_index("body") + if not nt_frame.index.is_unique: + raise ValueError("Duplicate male neurotransmitter IDs.") + predicted = frame.bodyId.map(nt_frame.consensus_nt) + nt_source = pd.Series( + ["source_consensus_prediction_or_ground_truth"] * len(frame) + ) + candidate_reason = np.where( + retain, "assigned_neuronal_superclass", "unresolved_object" + ) + else: + raise ValueError(f"Unsupported dataset: {dataset_id}") + if len(np.unique(source)) != len(source): + raise ValueError("Duplicate versioned neuron IDs in source annotations.") + catalog = pd.DataFrame( + { + "source_id": source, + "retained": np.asarray(retain, dtype=bool), + "object_kind": np.where( + nonneural, + "non_neuronal", + np.where(retain, "neuron_candidate", "unresolved_object"), + ), + "inclusion_reason": np.where( + nonneural, "explicit_non_neuronal_annotation", candidate_reason + ), + "quality": np.asarray(quality), + "superclass": np.asarray(superclasses), + "cell_type": np.asarray(types), + "neurotransmitter": np.asarray(predicted), + "neurotransmitter_source": np.asarray(nt_source), + } + ).sort_values("source_id", ignore_index=True) + nodes = catalog.loc[catalog.retained].reset_index(drop=True) + nodes.insert(0, "node_index", np.arange(len(nodes), dtype=np.uint32)) + return catalog, nodes + + +def import_graph(dataset_id: str = "malecns_v1") -> dict: + import pyarrow as pa + import pyarrow.feather as feather + import pyarrow.ipc as ipc + + config = json.loads(REGISTRY.read_text())["datasets"][dataset_id] + source_dir = DATA + output = source_dir / "normalized" + output.mkdir(parents=True, exist_ok=True) + lock_path = source_dir / "source.lock.json" + hashes = { + name: { + "url": url, + "bytes": (source_dir / name).stat().st_size, + "sha256": file_digest(source_dir / name), + } + for name, url in config["files"].items() + } + if lock_path.exists(): + locked = json.loads(lock_path.read_text()) + if any( + locked[name]["sha256"] != info["sha256"] for name, info in hashes.items() + ): + raise ValueError( + "Source files changed since the lock was created; use a new versioned dataset directory." + ) + else: + lock_path.write_text(json.dumps(hashes, indent=2) + "\n") + frame = feather.read_table(source_dir / "annotations.feather").to_pandas() + nt_frame = feather.read_table(source_dir / "neurotransmitters.feather").to_pandas() + catalog, nodes = normalize_nodes(dataset_id, frame, nt_frame) + feather.write_feather(catalog, output / "catalog.feather") + feather.write_feather(nodes, output / "neurons.feather") + ids = exact_ids(nodes.source_id) + np.save(output / "neuron_ids.npy", ids) + edge_columns = ("body_pre", "body_post", "weight") + reader = ipc.open_file(pa.memory_map(str(source_dir / "edges.feather"), "r")) + schema = pa.schema( + [ + ("pre_index", pa.uint32()), + ("post_index", pa.uint32()), + ("synapse_count", pa.uint32()), + ] + ) + stats = { + key: 0 + for key in [ + "source_edge_rows", + "retained_edge_rows", + "excluded_edge_rows", + "source_synaptic_contacts", + "retained_synaptic_contacts", + "excluded_synaptic_contacts", + "retained_weight_one_edges", + "retained_self_edges", + ] + } + incoming, outgoing = ( + np.zeros(len(ids), dtype=np.int64), + np.zeros(len(ids), dtype=np.int64), + ) + temporary = output / "edges.arrow.partial" + with pa.OSFile(str(temporary), "wb") as sink, ipc.new_file(sink, schema) as writer: + for number in range(reader.num_record_batches): + batch = reader.get_batch(number) + pre, post, weights = [ + batch.column(batch.schema.get_field_index(c)).to_numpy( + zero_copy_only=False + ) + for c in edge_columns + ] + i, j, count, keep = index_edges(ids, pre, post, weights) + stats["source_edge_rows"] += len(pre) + stats["retained_edge_rows"] += len(i) + stats["source_synaptic_contacts"] += int(weights.sum(dtype=np.uint64)) + stats["retained_synaptic_contacts"] += int(count.sum(dtype=np.uint64)) + stats["retained_weight_one_edges"] += int(np.count_nonzero(count == 1)) + stats["retained_self_edges"] += int(np.count_nonzero(i == j)) + np.add.at(incoming, j, count) + np.add.at(outgoing, i, count) + writer.write_batch( + pa.record_batch( + [pa.array(i), pa.array(j), pa.array(count)], schema=schema + ) + ) + temporary.replace(output / "edges.arrow") + stats["excluded_edge_rows"] = ( + stats["source_edge_rows"] - stats["retained_edge_rows"] + ) + stats["excluded_synaptic_contacts"] = ( + stats["source_synaptic_contacts"] - stats["retained_synaptic_contacts"] + ) + assert ( + int(incoming.sum()) + == int(outgoing.sum()) + == stats["retained_synaptic_contacts"] + ) + np.save(output / "incoming_synapse_counts.npy", incoming) + np.save(output / "outgoing_synapse_counts.npy", outgoing) + report = { + "dataset_id": dataset_id, + "sex": config["sex"], + "release": config["release"], + "coverage": config["coverage"], + "source_annotation_rows": len(catalog), + "retained_neuron_candidates": len(nodes), + "excluded_object_counts": catalog.loc[~catalog.retained] + .object_kind.value_counts() + .to_dict(), + "quality_counts": nodes.quality.value_counts(dropna=False).to_dict(), + "superclass_counts": nodes.superclass.fillna("unknown") + .value_counts() + .to_dict(), + "neurotransmitter_counts": nodes.neurotransmitter.fillna("missing") + .astype(str) + .value_counts() + .to_dict(), + "isolated_neurons": int(np.count_nonzero((incoming == 0) & (outgoing == 0))), + "node_policy": config["node_policy"], + "edge_policy": config["edge_policy"], + "upstream_filters": config["upstream_filters"], + "upstream_autapses_excluded": config["upstream_autapses_excluded"], + "additional_edge_strength_threshold": None, + "synaptic_weights_are_contact_counts": True, + "source_hashes": hashes, + "graph": stats, + "neural_dynamics_validated": False, + "embodied_behavior_implemented": False, + "remaining_gaps": [ + "Receptor-dependent synapse dynamics and neuromodulation", + "Calibrated retinal dynamics", + "Validated learning and controller interpretation", + ], + } + (output / "report.json").write_text(json.dumps(report, indent=2) + "\n") + public_report = DATA / "connectome-import.json" + public_report.parent.mkdir(parents=True, exist_ok=True) + public_report.write_text(json.dumps(report, indent=2) + "\n") + return report + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "dataset", choices=["malecns_v1"], nargs="?", default="malecns_v1" + ) + args = parser.parse_args() + report = import_graph(args.dataset) + print( + json.dumps( + { + k: report[k] + for k in [ + "dataset_id", + "retained_neuron_candidates", + "graph", + "neural_dynamics_validated", + ] + }, + indent=2, + ) + ) + + +if __name__ == "__main__": + main() diff --git a/agents/market_making_fly/flybrain/neural/datasets.json b/agents/market_making_fly/flybrain/neural/datasets.json new file mode 100644 index 000000000..27808f861 --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/datasets.json @@ -0,0 +1,25 @@ +{ + "source_checked": "2026-09-05", + "google_post": "https://research.google/blog/a-connectomics-milestone-mapping-the-complete-male-fruit-fly-brain/", + "datasets": { + "malecns_v1": { + "sex": "male", + "release": "MaleCNS v1.0", + "coverage": "brain_and_ventral_nerve_cord", + "source": "https://male-cns.janelia.org/download/", + "paper": "https://doi.org/10.1016/j.cell.2026.08.015", + "node_policy": "Every entry with an assigned superclass, including uncertain tbc classes; exclude explicit Glia status; no restriction to Traced status or typed cells.", + "edge_policy": "All released edges between retained entries; no additional weight threshold or removal of self-connections.", + "upstream_filters": [ + "Published pre/post synapse confidence threshold 0.5", + "Only annotated neuronal entries become simulation nodes; other segmentation objects remain in raw data" + ], + "upstream_autapses_excluded": false, + "files": { + "annotations.feather": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/body-annotations-male-cns-v1.0-minconf-0.5.feather", + "neurotransmitters.feather": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/body-neurotransmitters-male-cns-v1.0.feather", + "edges.feather": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/connectome-weights-male-cns-v1.0-minconf-0.5.feather" + } + } + } +} diff --git a/agents/market_making_fly/flybrain/neural/kernel.cpp b/agents/market_making_fly/flybrain/neural/kernel.cpp new file mode 100644 index 000000000..4b5b942ab --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/kernel.cpp @@ -0,0 +1,96 @@ +// Experimental dopamine-gated KC-to-MBON11 depression. +// Baseline integration is copied without changing its timestep/schedule. +// PPL101 edges deliver a modeled modulatory trace instead of fast excitation. +// Lazy exact subthreshold evolution. An inactive neuron is skipped only when +// its voltage AND both its instantaneous and asymptotic drive are below the +// threshold. With no incoming event it cannot fire. Every edge is retained. +#include +#include +#include +extern "C" void memory_advance( + int n,const int64_t* ptr,const int32_t* post,float* weight, + float* v,float* g,int16_t* refractory,const float* drive,float* previous_drive, + int32_t* queue,int32_t* queue_count,int64_t* clock,int steps,float dt,int32_t* counts, + int32_t* active,uint8_t* flags,int32_t* nactive,int64_t* last, + const uint8_t* kc_mask,const int8_t* dan_index,double* eligibility,int64_t* eligibility_last, + int nplastic,const int64_t* plastic_edge,const int32_t* plastic_pre, + const float* baseline_weight,const float* dan_gain,float eta,float tau_elig_ms,float floor_fraction, + int learning_enabled,float* modulation,int64_t* modulation_last,const uint8_t* modulation_mask,const float* rest, + float* adaptation,float adaptation_jump,float adaptation_tau) { + const int delay=std::lround(1.8f/dt),rfc=std::lround(2.2f/dt),slots=delay+1; + float av[1024],ag[1024],aa[1024]; + for(int i=0;i<1024;i++){av[i]=std::exp(-dt*i/20.f);ag[i]=std::exp(-dt*i/5.f);aa[i]=std::exp(-dt*i/adaptation_tau);} + auto evolve=[&](int i,int64_t now,float current){ + int64_t d=now-last[i];if(d<=0)return; + const int frozen=refractory[i]>0?refractory[i]-1:0; + const int skip=(int)(d0 && adaptation[i]>0)adaptation[i]*=skip<1024?aa[skip]:std::exp(-dt*skip/adaptation_tau); + refractory[i]=d>=refractory[i]?0:refractory[i]-d;d-=skip; + if(d>0){const float a=d<1024?av[d]:std::exp(-dt*d/20.f),b=d<1024?ag[d]:std::exp(-dt*d/5.f); + v[i]=rest[i]+(v[i]-rest[i])*a+current*(1.f-a)+g[i]*(a-b)/3.f;g[i]*=b; + if(adaptation[i]>0){const float c=d<1024?aa[d]:std::exp(-dt*d/adaptation_tau); + v[i]-=adaptation[i]*adaptation_tau/(adaptation_tau-20.f)*(c-a);adaptation[i]*=c;} + } + last[i]=now; + }; + auto awaken=[&](int i){if(!flags[i]){flags[i]=1;active[(*nactive)++]=i;}}; + // Apply changing sensory currents only after settling old-current history. + for(int i=0;i-45.f){queue[future*n+queue_count[future]++]=i;counts[i]++; + if(kc_mask[i]){ + adaptation[i]+=adaptation_jump; + eligibility[i]*=std::exp(-dt*(*clock-eligibility_last[i])/tau_elig_ms); + eligibility[i]+=1.;eligibility_last[i]=*clock; + } + } + // Convex relaxation toward drive+g(t): the bound makes this exact in + // real arithmetic, not an activity cutoff or a dropped weak connection. + const float gap=-45.f-rest[i]; + const bool can_fire=v[i]>-45.f || drive[i]>gap || drive[i]+g[i]>gap; + if(can_fire)active[kept++]=i;else flags[i]=0; + } + *nactive=kept; + for(int q=0;q=0){ + for(int p=0;plower?candidate:lower; + } + } + continue; + } + for(int64_t e=ptr[i];e 0.0100001: + raise ValueError("Rate bins must be 0--10 ms") + h = dt_seconds + p = PARAMETERS + ak = math.exp(-h / p["trace_kc_seconds"]) + ad = math.exp(-h / p["trace_dan_seconds"]) + kmid = y_kc * math.sqrt(ak) + kc_hz * (1 - math.sqrt(ak)) + dmid = y_dan * math.sqrt(ad) + dan_hz * (1 - math.sqrt(ad)) + y_kc[:] = y_kc * ak + kc_hz * (1 - ak) + y_dan[:] = y_dan * ad + dan_hz * (1 - ad) + if frozen: + return + drive = ( + eta * (kc_hz * (gain.T @ dmid) - (gain.T @ dan_hz) * kmid) + if learning + else np.zeros_like(u) + ) + tu = p["memory_decay_seconds"] + tw = p["weight_filter_seconds"] + eu = math.exp(-h / tu) + ew = math.exp(-h / tw) + c = tu / (tu - tw) * (eu - ew) + old_u = u.copy() + u[:] = old_u * eu + drive * tu * (-math.expm1(-h / tu)) + w[:] = w * ew + old_u * c + drive * tu * (-math.expm1(-h / tw) - c) + # Bounds preserve the reconstructed excitatory sign. Saturation is reported, + # not hidden or reset when an assay has an unfavorable outcome. + lo = p["minimum_fraction"] - 1 + hi = p["maximum_fraction"] - 1 + np.clip(u, lo, hi, out=u) + np.clip(w, lo, hi, out=w) diff --git a/agents/market_making_fly/flybrain/neural/sensory.py b/agents/market_making_fly/flybrain/neural/sensory.py new file mode 100644 index 000000000..d83a32d3c --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/sensory.py @@ -0,0 +1,17 @@ +"""Display brightness proxy; not a calibrated compound-eye model.""" + +import numpy as np + + +def retinal_samples(frame, uv): + frame = np.asarray(frame) + if frame.ndim != 3 or frame.shape[2] != 3 or frame.dtype != np.uint8: + raise ValueError("RGB uint8 frame required") + h, w = frame.shape[:2] + x = np.clip((uv[:, 0] * (w - 1)).astype(int), 0, w - 1) + y = np.clip((uv[:, 1] * (h - 1)).astype(int), 0, h - 1) + rgb = frame[y, x].astype(np.float32) / 255 + linear = np.where(rgb <= 0.04045, rgb / 12.92, ((rgb + 0.055) / 1.055) ** 2.4) + return (linear @ np.asarray([0.2126, 0.7152, 0.0722], dtype=np.float32)).astype( + np.float32 + ) diff --git a/agents/market_making_fly/flybrain/neural/sources.lock.json b/agents/market_making_fly/flybrain/neural/sources.lock.json new file mode 100644 index 000000000..addd65565 --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/sources.lock.json @@ -0,0 +1,17 @@ +{ + "annotations.feather": { + "url": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/body-annotations-male-cns-v1.0-minconf-0.5.feather", + "bytes": 14483314, + "sha256": "2177e246113e4cfbf1e7772ec37c6da1955ff22e8063d0b1f833101f99a9a3b2" + }, + "neurotransmitters.feather": { + "url": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/body-neurotransmitters-male-cns-v1.0.feather", + "bytes": 43282834, + "sha256": "95c9289220663abeb3409f3ad9e5a7f8a53f8093f5139d15502cd08da8879621" + }, + "edges.feather": { + "url": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/connectome-weights-male-cns-v1.0-minconf-0.5.feather", + "bytes": 1051241946, + "sha256": "e35da783d1c686b2b58b3b87cd6a403ae43bfcfba8bff28e08ef752c1a56afc1" + } +} diff --git a/agents/market_making_fly/flybrain/neural/state.py b/agents/market_making_fly/flybrain/neural/state.py new file mode 100644 index 000000000..5800a6b7b --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/state.py @@ -0,0 +1,82 @@ +"""State allocation for the retained connectome; native integration is separate.""" + +import numpy as np + + +class Brain: + def __init__(self, path, dt=0.1): + if dt != 0.1: + raise ValueError("This audited kernel supports only dt=0.1 ms.") + a = np.load(path) + for k in [ + "ptr", + "post", + "weight", + "ids", + "retina", + "uv", + "lamina", + "sugar", + "superclass", + ]: + setattr(self, k, a[k]) + n = len(self.ids) + for k, dtype in [ + ("ptr", np.int64), + ("post", np.int32), + ("weight", np.float32), + ("ids", np.int64), + ("retina", np.int32), + ("lamina", np.int32), + ("sugar", np.int32), + ]: + x = getattr(self, k) + if x.ndim != 1 or x.dtype != dtype or not x.flags.c_contiguous: + raise ValueError(f"Invalid native graph array: {k}") + if ( + n < 1 + or self.ptr.shape != (n + 1,) + or self.ptr[0] != 0 + or self.ptr[-1] != len(self.post) + or np.any(np.diff(self.ptr) < 0) + or len(self.weight) != len(self.post) + ): + raise ValueError("Invalid CSR graph") + if not np.isfinite(self.weight).all(): + raise ValueError("Nonfinite synaptic weight") + for x in [self.post, self.retina, self.lamina, self.sugar]: + if np.any(x < 0) or np.any(x >= n): + raise ValueError("Graph index out of bounds") + if ( + self.uv.shape != (len(self.retina), 2) + or not np.isfinite(self.uv).all() + or np.any(self.uv < 0) + or np.any(self.uv > 1) + ): + raise ValueError("Invalid receptor UV coordinates") + self.dt = dt + self.n = len(self.ids) + self.cursor = 0 + self.v = np.full(self.n, -52, dtype=np.float32) + self.g = np.zeros(self.n, dtype=np.float32) + self.drive = np.zeros(self.n, dtype=np.float32) + self.refractory = np.zeros(self.n, dtype=np.int16) + self.queue = np.zeros((int(round(1.8 / dt)) + 1, self.n), dtype=np.int32) + self.queue_count = np.zeros(self.queue.shape[0], dtype=np.int32) + self.counts = np.zeros(self.n, dtype=np.int32) + self.luminance = np.zeros(len(self.retina), dtype=np.float32) + self.active = np.zeros(self.n, dtype=np.int32) + self.active_flag = np.zeros(self.n, dtype=np.uint8) + initial = np.unique(np.r_[self.retina, self.lamina, self.sugar]) + self.active[: len(initial)] = initial + self.active_flag[initial] = 1 + self.nactive = np.asarray([len(initial)], dtype=np.int32) + self.total_spikes = 0 + self.sim_ms = 0 + + +class NativeBrain(Brain): + def __init__(self, path, dt=0.1): + super().__init__(path, dt) + self.previous_drive = np.zeros(self.n, dtype=np.float32) + self.last = np.full(self.n, -1, dtype=np.int64) diff --git a/agents/market_making_fly/flybrain/neural/transmitters.py b/agents/market_making_fly/flybrain/neural/transmitters.py new file mode 100644 index 000000000..995d1a800 --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/transmitters.py @@ -0,0 +1,24 @@ +"""Declared neurotransmitter-sign proxy used in DOOMFLY graph preparation.""" + +import numpy as np + + +def transmitter_signs(transmitters, ambiguous_sign=1): + """Declared coarse fast-transmission assumption; never deletes unknown edges. + + ACh +; GABA, glutamate, histamine -. A co-transmitter combination with only + one fast sign uses that sign; conflicting, missing and modulator-only cells + use the explicit sensitivity parameter. This is NOT receptor physiology. + """ + if ambiguous_sign not in (-1, 1): + raise ValueError("Ambiguous edges must remain active with sign +1 or -1.") + signs, uncertain = [], [] + for value in transmitters: + tokens = set(str(value).lower().split(",")) + fast = ({1} if "acetylcholine" in tokens else set()) | ( + {-1} if tokens & {"gaba", "glutamate", "histamine"} else set() + ) + ambiguous = len(fast) != 1 + signs.append(ambiguous_sign if ambiguous else next(iter(fast))) + uncertain.append(ambiguous) + return np.asarray(signs, dtype=np.int8), np.asarray(uncertain, dtype=bool) diff --git a/agents/market_making_fly/flybrain/neural/visual.py b/agents/market_making_fly/flybrain/neural/visual.py new file mode 100644 index 000000000..3dbae6249 --- /dev/null +++ b/agents/market_making_fly/flybrain/neural/visual.py @@ -0,0 +1,155 @@ +"""Candidate R8 display adapter and target-specific cotransmission. + +Anatomy remains exactly MaleCNS. R8->aMe12 net excitation is supported by +Xiao et al. Nature 2023, doi:10.1038/s41586-023-06681-6 (AMA includes aMe12). +The transferred type correspondence, unitary magnitude, LIF photoreceptor +proxy and RGB-to-opsin transfer are explicit modeling assumptions. No UV +channel or sensitivity is invented for R7 or untyped photoreceptors. +""" + +import math + +import numpy as np + +from .brain import MemoryBrain +from .common import annotations, digest + + +def projection(brain, a): + known = np.flatnonzero(a.type.isin(["R8p", "R8y"])) + mapped = [] + hexes = [] + confidence = [] + for i in known: + edges = np.arange(brain.ptr[i], brain.ptr[i + 1]) + targets = brain.post[edges] + valid = ( + a.assignedOlHex1.iloc[targets].notna().to_numpy() + & a.assignedOlHex2.iloc[targets].notna().to_numpy() + ) + votes = {} + for e, j in zip(edges[valid], targets[valid]): + h = (float(a.assignedOlHex1.iloc[j]), float(a.assignedOlHex2.iloc[j])) + votes[h] = votes.get(h, 0) + float(abs(brain.weight[e])) + if not votes: + continue + h = max(votes, key=votes.get) + mapped.append(i) + hexes.append(h) + confidence.append(votes[h] / sum(votes.values())) + mapped = np.asarray(mapped, dtype=np.int32) + hexes = np.asarray(hexes) + xy = np.column_stack( + [hexes[:, 0] - 0.5 * hexes[:, 1], np.sqrt(3) / 2 * hexes[:, 1]] + ) + # Fit the original display projection from the existing R1-R6 anchors so + # R8 does not get a separately stretched eye or arbitrary target position. + from .common import GRAPH + + with np.load(GRAPH) as g: + oldhex = g["hexes"] + oldxy = np.column_stack( + [oldhex[:, 0] - 0.5 * oldhex[:, 1], np.sqrt(3) / 2 * oldhex[:, 1]] + ) + uv = np.empty_like(xy) + for side in ["L", "R"]: + original = a.rootSide.iloc[brain.retina].eq(side).to_numpy() + select = a.rootSide.iloc[mapped].eq(side).to_numpy() + lo = oldxy[original].min(axis=0) + span = np.ptp(oldxy[original], axis=0) + z = (xy[select] - lo) / span + uv[select, 0] = 0.6 * z[:, 0] if side == "L" else 0.4 + 0.6 * (1 - z[:, 0]) + uv[select, 1] = 1 - z[:, 1] + if not a.rootSide.iloc[mapped].isin(["L", "R"]).all(): + raise ValueError("Unresolved eye side") + return ( + mapped, + np.clip(uv, 0, 1).astype(np.float32), + np.asarray(confidence, dtype=np.float32), + ) + + +class VisualMemoryBrain(MemoryBrain): + def __init__(self, **kwargs): + super().__init__(**kwargs) + a = annotations(self.ids) + self.r8, self.r8_uv, self.r8_confidence = projection(self, a) + self.r8_channel = np.where(a.type.iloc[self.r8].eq("R8p"), 2, 1).astype( + np.int32 + ) + self.r8_light = np.zeros(len(self.r8), dtype=np.float32) + corrected = [] + for i in np.flatnonzero(a.type.fillna("").str.startswith("R8")): + edges = np.arange(self.ptr[i], self.ptr[i + 1]) + e = edges[a.type.iloc[self.post[edges]].eq("aMe12").to_numpy()] + corrected.extend(e.tolist()) + self.weight[e] = np.abs(self.weight[e]) + self.corrected_edges = np.asarray(corrected, dtype=np.int64) + self.initial_weight_sha256 = digest(self.weight) + self.fields.append("r8_light") + self.initial["r8_light"] = self.r8_light.copy() + self.visual_report = { + "model": "r8-rgb-ame12-v1", + "mapped_R8p": int((self.r8_channel == 2).sum()), + "mapped_R8y": int((self.r8_channel == 1).sum()), + "known_unmapped": int(a.type.isin(["R8p", "R8y"]).sum() - len(self.r8)), + "projection_confidence_median": float(np.median(self.r8_confidence)), + "projection_below_80_percent": int((self.r8_confidence < 0.8).sum()), + "corrected_existing_edges": len(corrected), + "corrected_edge_sha256": digest(self.corrected_edges), + "coordinate_inference": "Modal column of all outgoing contacts to any column-annotated target; same viewport transform as R1-R6.", + "spectrum": "Linear sRGB B for R8p; G for R8y. Display proxy, not calibrated photon flux or spectral sensitivity. R7, dorsal and untyped R8 receive no invented optical drive.", + "physiology": "R8 to aMe12 net sign positive; original contact magnitudes retained. Other R8 targets retain baseline sign. Photoreceptors still use a LIF rate proxy, not graded in-vivo dynamics.", + "evidence": [ + "https://doi.org/10.1038/s41586-023-06681-6", + "https://doi.org/10.1038/s41467-024-49616-z", + "https://doi.org/10.7554/eLife.71858", + ], + "validated": False, + } + + def rgb_step(self, frame, duration_ms, **kwargs): + if duration_ms > 10: + ticks = round(duration_ms / self.dt) + total = np.zeros(self.n, dtype=np.int32) + wall = 0.0 + while ticks: + n = min(100, ticks) + c, t = self.rgb_step(frame, n * self.dt, **kwargs) + total += c + wall += t + ticks -= n + self.counts[:] = total + return total, wall + from .sensory import retinal_samples + + frame = np.asarray(frame) + if frame.ndim != 3 or frame.shape[2] != 3 or frame.dtype != np.uint8: + raise ValueError("RGB uint8 required") + h, w = frame.shape[:2] + x = np.minimum((self.r8_uv[:, 0] * (w - 1)).astype(int), w - 1) + y = np.minimum((self.r8_uv[:, 1] * (h - 1)).astype(int), h - 1) + values = frame[y, x, self.r8_channel].astype(np.float32) / 255 + values = np.where( + values <= 0.04045, values / 12.92, ((values + 0.055) / 1.055) ** 2.4 + ) + self.r8_light += ( + 1 - math.exp(-round(duration_ms / self.dt) * self.dt / 10) + ) * (values - self.r8_light) + extra = kwargs.pop("stimulation", None) + pulses = ( + [] if extra is None else list(extra) if isinstance(extra, list) else [extra] + ) + pulses.append((self.r8, 30 * self.r8_light / (0.02 + self.r8_light))) + return self.step( + retinal_samples(frame, self.uv), duration_ms, stimulation=pulses, **kwargs + ) + + def configuration_signature(self): + return { + **super().configuration_signature(), + **{ + k: digest(getattr(self, k)) + for k in ["r8", "r8_uv", "r8_channel", "corrected_edges"] + }, + } diff --git a/agents/market_making_fly/flybrain/posture.py b/agents/market_making_fly/flybrain/posture.py new file mode 100644 index 000000000..4fd4a4635 --- /dev/null +++ b/agents/market_making_fly/flybrain/posture.py @@ -0,0 +1,314 @@ +"""Turn a posture into a full ``pmm_mister`` config. + +The base is a bounded default set; the posture multiplies the spreads and leans +them. Every money-relevant floor lives here, in code: + +* no spread level closer to mid than the market's own maker fee, so a + two-sided round trip at the floor at least pays for itself; +* ``take_profit`` never below ``2.2 ×`` the round-trip maker fee, and never + below 4 bp (Market Making Expert's "TP must exceed round-trip fees" made + mandatory); +* the reference-price lean is capped at half the first-level spread. + +Inventory bands, allocation, leverage cap and the global stop loss are the +operator's, not the fly's to move. + +The spec works on any CLOB market, spot or perp. What the venue decides — +whether leverage applies at all, and what a round trip costs — comes from +``venue.py``; what the fly decides is only ever the posture. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass + +from flybrain import venue +from flybrain.decoder import REGIMES, Posture +from flybrain.naming import pair_names + +# executor_refresh_time, buy/sell cooldown — Market Making Expert's table. +TIMING: dict[str, tuple[int, int]] = { + "quiet": (20, 30), + "ranging": (30, 60), + "trending_up": (30, 60), + "trending_down": (30, 60), + "volatile": (60, 120), + "pause": (60, 120), +} +assert set(TIMING) == set(REGIMES) + +BPS = 1e-4 +# How far above an exchange minimum an order must be sized to survive the +# venue's own rounding. 20 % covers a step rounded down on a three-decimal +# market near $150 and leaves room for the lean. +ORDER_SIZE_MARGIN = 1.2 + + +@dataclass(frozen=True) +class MarketSpec: + """What the operator settles once per deployment; the fly never changes it.""" + + connector_name: str + trading_pair: str # BASE-QUOTE, or ISSUER:TOKEN-QUOTE on HIP-3 + total_amount_quote: float + range_bps: float # median candle range for this market, at the fly's interval + # Blank means "derive from the connector name"; see venue.resolve. + market_type: str = "" + # 1 on spot, where there is nothing to lever. + leverage: int = 1 + # 0 means "use the venue default"; pass the exchange's real figure when + # known, since the take-profit floor is derived from it. + maker_fee_bps: float = 0.0 + # Everything below is Market Making Expert's balanced profile, which is the + # vetted steady state for this controller. The fly's own contribution is + # the spread geometry, the fee-derived floor and the dopamine loop; there + # was no reason for the rest of the config to differ from what the expert + # deploys, and several of these were quietly the controller's defaults + # rather than a choice. + portfolio_allocation: float = 0.15 + target_base_pct: float = 0.5 + min_base_pct: float = 0.35 + max_base_pct: float = 0.65 + max_active_executors_by_level: int = 3 + min_skew: float = 1.5 + effectivization_time: int = 120 + price_distance_tolerance: float = 0.0005 + refresh_tolerance: float = 0.0005 + tolerance_scaling: float = 1.2 + global_stop_loss: float = 0.05 + leverage_cap: int = 5 + # Exchange minimum per order. pmm_mister sizes one cycle as + # total_amount_quote × portfolio_allocation, split across both sides and + # every level, so a small book needs a larger allocation to clear it. + min_order_notional: float = 10.0 + + def __post_init__(self): + pair_names(self.trading_pair) # refuses anything unparseable + resolved = venue.resolve(self.connector_name, self.market_type) + object.__setattr__(self, "market_type", resolved) + if not self.maker_fee_bps: + object.__setattr__( + self, + "maker_fee_bps", + venue.default_maker_fee_bps(self.connector_name, resolved), + ) + for name in ("total_amount_quote", "range_bps", "maker_fee_bps"): + value = getattr(self, name) + if not math.isfinite(value) or value <= 0: + raise ValueError(f"{name} must be finite and positive") + if self.is_spot: + if self.leverage != 1: + raise ValueError( + f"{self.connector_name} is a spot market: leverage must be 1, " + f"got {self.leverage}" + ) + elif not 1 <= self.leverage <= self.leverage_cap: + raise ValueError(f"leverage must be within 1..{self.leverage_cap}") + if not 0 < self.min_base_pct < self.target_base_pct < self.max_base_pct < 1: + raise ValueError("0 < min_base < target_base < max_base < 1 required") + if not 0 < self.portfolio_allocation <= 1: + raise ValueError("portfolio_allocation must be in (0, 1]") + if self.min_order_notional < 0: + raise ValueError("min_order_notional must be >= 0") + + @property + def is_spot(self) -> bool: + return self.market_type == venue.SPOT + + @property + def min_portfolio_allocation(self) -> float: + """The smallest allocation whose orders still clear the venue minimum.""" + return self.min_order_notional * ORDER_SIZE_MARGIN * 4 / self.total_amount_quote + + @property + def min_spread_bps(self) -> float: + """The closest to mid a quote may sit: one maker fee. + + A buy at −f and a sell at +f capture exactly 2f, which is the round + trip — break-even. Anything tighter loses money on every completed + pair whatever the fly decodes, so it is the one width that is not the + strategy's to choose. It was a fixed 3 bp, which is both too wide for + a HIP-3 market at 1.3 bp and too tight for a spot book at 7.5, and it + was a parameter nobody could set correctly without already knowing the + fee the spec now carries. + """ + return self.maker_fee_bps + + @property + def order_notional(self) -> float: + """Quote size of one order at neutral posture: one cycle's allocation + over two sides and two levels.""" + return self.total_amount_quote * self.portfolio_allocation / 4 + + def check_order_size(self) -> None: + """Refuse a size the exchange will reject once it has been rounded. + + An order sized to exactly the minimum does not survive the round trip + through the exchange's own precision: the controller converts the quote + size to a base amount, rounds it down to the market's step, and the + resulting notional lands *under* the minimum. XYZ:ORCL-USD quotes to + three decimals near 148, so $10.00 became $9.94 and Hyperliquid + rejected every order with "lower than minimum notional size". The + margin is what a rounded-down step can cost plus the widest lean. + """ + floor = self.min_order_notional * ORDER_SIZE_MARGIN + if self.order_notional < floor: + needed = floor * 4 / self.total_amount_quote + raise ValueError( + f"{self.trading_pair}: an order would be {self.order_notional:.2f} quote, " + f"under the {floor:.2f} needed to clear a " + f"{self.min_order_notional:.0f} minimum after rounding; raise " + f"portfolio_allocation to at least {min(1.0, needed):.2f} or " + "total_amount_quote" + ) + + +# The two geometry rules, as plain functions of what they actually depend on: +# a market's fee and its observed spread. The scanner needs them before a spec +# exists — it is deciding whether a market is worth quoting at all — and two +# copies of a money rule is one too many. +def take_profit_floor_bps(maker_fee_bps: float) -> float: + """Where a position must close to have been worth opening, in bp.""" + return max(4.0, 2.2 * 2 * maker_fee_bps) + + +# The exit was one number for every market and every posture, so a replay of +# five variants closed the same eleven round trips apiece however they quoted. +# Three quarters of a typical bar is a target the market reaches often without +# giving the whole excursion away, and it scales with the market rather than +# with the fee alone — the fee only ever sets the floor. +TP_RANGE_FRACTION = 0.75 + + +def take_profit_base_bps(maker_fee_bps: float, range_bps: float) -> float: + """The exit before the fly moves it: a fraction of the range, never under + what a round trip costs.""" + return max(take_profit_floor_bps(maker_fee_bps), TP_RANGE_FRACTION * range_bps) + + +# The outer level must stay outside the inner one, whatever the arithmetic +# says: a ladder whose second rung is inside its first fills in the wrong +# order and means nothing. +LEVEL_STEP_BPS = 1.0 + +# Quotes are placed against how far the market actually travels, not against +# how wide its touch is. The playbook derived level 1 from the observed spread +# — half of 1.75 bp on DRAM, so 2 bp — while a typical 5-minute bar there +# ranges 9.9 bp and therefore reaches about 5 bp either side of mid. Every +# level the fly could express, tight or wide, sat inside what a normal bar +# covers, so the same candles touched the same orders whatever it decided: +# five replay variants returned 23 fills and 15 round trips apiece. +# +# Half the range puts level 1 where a typical bar just reaches it, so the +# spread multiplier straddles the fill boundary instead of living inside it: +# 0.6x fills most bars, 2.5x fills few. +LEVEL_ONE_RANGE_FRACTION = 0.5 +LEVEL_STEP_RANGE_FRACTION = 0.25 + + +def base_levels_from_range(range_bps: float) -> tuple[float, float]: + """Level 1 at half a typical bar's range, level 2 a quarter-bar beyond.""" + first = max(2.0, LEVEL_ONE_RANGE_FRACTION * range_bps) + step = max(LEVEL_STEP_BPS, LEVEL_STEP_RANGE_FRACTION * range_bps) + return first, first + step + + +def take_profit_floor(spec: MarketSpec) -> float: + return round(take_profit_floor_bps(spec.maker_fee_bps) * BPS, 8) + + +def base_levels_bps(spec: MarketSpec) -> tuple[float, float]: + return base_levels_from_range(spec.range_bps) + + +def _fmt(values: list[float]) -> str: + return ",".join(f"{v:.6f}".rstrip("0").rstrip(".") for v in values) + + +def build_config(spec: MarketSpec, posture: Posture) -> dict: + if posture.regime not in TIMING: + raise ValueError(f"Unknown regime {posture.regime!r}") + spec.check_order_size() + # Arousal moves how much of the book is quoted as well as how tight: an + # active market gets leaned into on both. Clamped so a scaled-down cycle + # never sizes an order under the venue minimum, nor a scaled-up one over + # the whole book. + allocation = min( + 1.0, + max( + spec.min_portfolio_allocation, + spec.portfolio_allocation * posture.size_mult, + ), + ) + l1, l2 = base_levels_bps(spec) + levels = [l1 * posture.spread_mult, l2 * posture.spread_mult] + shift = max(-levels[0] / 2, min(levels[0] / 2, posture.shift_bps)) + buy = [max(spec.min_spread_bps, lvl - shift) for lvl in levels] + sell = [max(spec.min_spread_bps, lvl + shift) for lvl in levels] + # The exit scales with the market, never below what a round trip costs and + # never below the fly's own first level — a take-profit inside the spread + # it quotes would close for nothing. The fly does not move it: letting + # arousal widen it halved the round trips and earned nothing for them. + take_profit = max( + take_profit_floor(spec), + min(buy[0], sell[0]) * BPS, + take_profit_base_bps(spec.maker_fee_bps, spec.range_bps) * BPS, + ) + refresh, cooldown = TIMING[posture.regime] + config = { + "controller_type": "generic", + "controller_name": "pmm_mister", + "connector_name": spec.connector_name, + "trading_pair": spec.trading_pair, + "total_amount_quote": spec.total_amount_quote, + "portfolio_allocation": round(allocation, 6), + "leverage": spec.leverage, + "target_base_pct": spec.target_base_pct, + "min_base_pct": spec.min_base_pct, + "max_base_pct": spec.max_base_pct, + "buy_spreads": _fmt([b * BPS for b in buy]), + "sell_spreads": _fmt([s * BPS for s in sell]), + # A suppressed side is quoted at zero size, which pmm_mister skips by + # name ("The amount of the level is 0. Skipping."). It also normalizes + # amounts across both sides, so the surviving side's orders double — + # the same capital through half as many quotes, which is what taking a + # side off the book should mean. + "buy_amounts_pct": "0,0" if posture.side == "sell" else "1,1", + "sell_amounts_pct": "0,0" if posture.side == "buy" else "1,1", + "executor_refresh_time": refresh, + "buy_cooldown_time": cooldown, + "sell_cooldown_time": cooldown, + "take_profit": round(take_profit, 8), + "take_profit_order_type": 3, + "open_order_type": 3, + "max_active_executors_by_level": spec.max_active_executors_by_level, + # Stated rather than left to the controller's defaults, so a change in + # hummingbot cannot move the fly's risk without anyone deciding to. + "buy_position_effectivization_time": spec.effectivization_time, + "sell_position_effectivization_time": spec.effectivization_time, + "price_distance_tolerance": spec.price_distance_tolerance, + "refresh_tolerance": spec.refresh_tolerance, + "tolerance_scaling": spec.tolerance_scaling, + # The expert's third domain, which the fly had no answer to at all: + # quote the accumulating side wider until the inventory comes back. + "min_skew": spec.min_skew, + "tick_mode": False, + "global_tp_enabled": False, + "global_sl_enabled": True, + "global_stop_loss": spec.global_stop_loss, + "global_sl_activation_from": "target_base", + "global_pnl_reference": "position", + "manual_kill_switch": posture.regime == "pause", + } + if not spec.is_spot: + # Only a perpetual has one; pmm_mister skips the check on spot. + config["position_mode"] = "ONEWAY" + return config + + +def config_diff(old: dict | None, new: dict) -> dict: + """Fields whose value changed; the whole config when there was none.""" + if old is None: + return dict(new) + return {k: v for k, v in new.items() if old.get(k) != v} diff --git a/agents/market_making_fly/flybrain/reinforcement.py b/agents/market_making_fly/flybrain/reinforcement.py new file mode 100644 index 000000000..e501e5a8b --- /dev/null +++ b/agents/market_making_fly/flybrain/reinforcement.py @@ -0,0 +1,56 @@ +"""Turn a change in the fly's P&L into a dopamine pulse kind. + +Stonkfly's rule, with the equity source changed: ``equity`` is the combined net +P&L of the ``pmm_mister`` controllers the fly is quoting for — realized plus +unrealized minus fees — so nothing else on the account can reward or punish it. + +The pulse is binary above a deadband, not proportional, and tiny changes are +not accumulated. It is feedback about value between two observations; it is +not evidence that the last posture caused that change. +""" + +from __future__ import annotations + +import math +from decimal import Decimal +from typing import Literal + +Kind = Literal["reward", "aversive", "none"] + + +def D(value) -> Decimal: + if isinstance(value, bool): + raise ValueError("Boolean is not money") + number = Decimal(str(value)) + if not number.is_finite(): + raise ValueError("Nonfinite quantity") + return number + + +def reinforcement(equity, anchor, deadband) -> tuple[Kind, Decimal]: + """``(kind, delta)`` with ``delta = equity - anchor``.""" + delta = D(equity) - D(anchor) + threshold = D(deadband) + if threshold <= 0: + raise ValueError("Positive reinforcement deadband required") + if delta >= threshold: + return "reward", delta + if delta <= -threshold: + return "aversive", delta + return "none", delta + + +def controller_net(performance: dict) -> float: + """``realized + unrealized`` in quote for one controller's performance + block (the ``performance`` dict inside ``get_active_bots_status``). Fees + are already netted into Hummingbot's realized figure.""" + realized = performance.get("realized_pnl_quote") + unrealized = performance.get("unrealized_pnl_quote") + if realized is None or unrealized is None: + raise ValueError( + "Controller performance lacks realized_pnl_quote/unrealized_pnl_quote" + ) + total = float(realized) + float(unrealized) + if not math.isfinite(total): + raise ValueError("Nonfinite controller P&L") + return total diff --git a/agents/market_making_fly/flybrain/replay.py b/agents/market_making_fly/flybrain/replay.py new file mode 100644 index 000000000..3882c0855 --- /dev/null +++ b/agents/market_making_fly/flybrain/replay.py @@ -0,0 +1,421 @@ +"""Replay the fly over recorded candles, with a maker fill model for its P&L. + +Every experiment this agent needs is a comparison: does the memory rule change +anything, does real reinforcement beat shuffled, does a gain matter. Answering +any of them live costs a bot, an hour and real capital, and produces one sample +of a market that never repeats. So the same loop runs here over a fixed candle +series instead, where the only thing that differs between two runs is the +setting under test. + +What is identical to the live loop: the frame the retina sees (``market_frame`` +over the same sliding window), the brain (the same worker, the same seeded +network), the decoder, the posture geometry and the fee. What is simulated is +only the market's answer — whether a resting quote filled — and that is the +part to be honest about: + +* a quote at price ``p`` fills if the next candle trades through it. There is + no queue position, so every fill a price *touches* is assumed ours. Real + fills are a fraction of that, and the fraction is worst exactly when the + market is moving, which is when this model is most generous. +* a fill's take-profit closes when a later candle trades through it. The same + optimism applies, and inventory that never reaches its take-profit is marked + to the close, exactly as the live loop's unrealized P&L would be. +* adverse selection appears only through price: if the market runs, the model + fills the losing side and holds the position, which is the real failure mode + it does reproduce. + +So replay P&L is an upper bound, not a forecast. It is useful because the bias +is the *same* for every variant, which is what a comparison needs. +""" + +from __future__ import annotations + +import math +import random +from dataclasses import dataclass, field + +import numpy as np +from flybrain.chart import market_frame +from flybrain.decoder import ( + Baseline, + Channels, + DecoderSettings, + Hysteresis, + Posture, + decode, + should_apply, +) +from flybrain.posture import BPS, MarketSpec, build_config, take_profit_floor +from flybrain.reinforcement import reinforcement + +# A held position is marked to the candle close, like the live loop's +# unrealized P&L; a closed one has both fees taken out of it. +SIDES = ("buy", "sell") + +# Below this a window is arithmetically valid and statistically worthless: it +# would fit the retina's candle window and leave a handful of ticks to score, +# which is not a sample of anything. Refusing is better than reporting it. +MIN_TICKS_PER_WINDOW = 30 + + +@dataclass +class Lot: + """One filled quote, still open.""" + + side: str + price: float + amount: float # base units + take_profit: float # fraction from entry + + +@dataclass +class Ledger: + """Fills, closes and P&L for one replay, in quote units.""" + + realized: float = 0.0 + fees: float = 0.0 + fills: int = 0 + round_trips: int = 0 + volume: float = 0.0 + open_lots: list[Lot] = field(default_factory=list) + + @property + def inventory(self) -> float: + return sum(l.amount if l.side == "buy" else -l.amount for l in self.open_lots) + + def equity(self, mark: float) -> float: + """Realized less fees, plus the open inventory marked to ``mark``.""" + unrealized = sum( + ( + (mark - l.price) * l.amount + if l.side == "buy" + else (l.price - mark) * l.amount + ) + for l in self.open_lots + ) + return self.realized - self.fees + unrealized + + +def _levels(config: dict, key: str) -> list[float]: + return [float(x) for x in config[key].split(",")] + + +def _amounts(config: dict, side: str) -> list[float]: + """The per-level amount fractions for one side; all zero when suppressed.""" + return [float(x) for x in config[f"{side}_amounts_pct"].split(",") if float(x)] + + +def quote_prices(config: dict, mid: float) -> dict[str, list[float]]: + """Where this config rests its orders around ``mid``.""" + return { + "buy": [mid * (1 - s) for s in _levels(config, "buy_spreads")], + "sell": [mid * (1 + s) for s in _levels(config, "sell_spreads")], + } + + +def step( + ledger: Ledger, + config: dict, + spec: MarketSpec, + candle: dict, + mid: float, + max_lots: int, +) -> None: + """One candle against one config: close what reaches take-profit, then fill. + + Closes are settled before new fills so a lot cannot open and close inside + the same candle, which the data cannot support: a candle says a price was + touched, not in what order. + """ + high, low = float(candle["high"]), float(candle["low"]) + fee = spec.maker_fee_bps * BPS + + still_open: list[Lot] = [] + for lot in ledger.open_lots: + target = ( + lot.price * (1 + lot.take_profit) + if lot.side == "buy" + else lot.price * (1 - lot.take_profit) + ) + reached = high >= target if lot.side == "buy" else low <= target + if not reached: + still_open.append(lot) + continue + gross = ( + (target - lot.price) * lot.amount + if lot.side == "buy" + else (lot.price - target) * lot.amount + ) + ledger.realized += gross + ledger.fees += target * lot.amount * fee # the closing side + ledger.volume += target * lot.amount + ledger.round_trips += 1 + ledger.open_lots = still_open + + notional = float(config["total_amount_quote"]) * float( + config["portfolio_allocation"] + ) + # pmm_mister normalizes amounts across both sides, so a side quoted at zero + # size does not shrink the book — it doubles the other side's orders. + active = [s for s in SIDES if any(_amounts(config, s))] + per_order = notional / max(1, sum(len(_amounts(config, s)) for s in active)) + prices = quote_prices(config, mid) + for side in active: + for price in prices[side]: + if len(ledger.open_lots) >= max_lots: + break + touched = low <= price if side == "buy" else high >= price + if not touched or price <= 0: + continue + amount = per_order / price + ledger.open_lots.append( + Lot( + side=side, + price=price, + amount=amount, + take_profit=float(config["take_profit"]), + ) + ) + ledger.fills += 1 + ledger.fees += per_order * fee + ledger.volume += per_order + + +@dataclass +class ReplayResult: + variant: str + ticks: int + applies: int + holds: int + unconfident: int + ledger: Ledger + equity_curve: list[float] + postures: list[Posture] + + def summary(self) -> dict: + regimes: dict[str, int] = {} + for p in self.postures: + regimes[p.regime] = regimes.get(p.regime, 0) + 1 + spreads = [p.spread_mult for p in self.postures] or [0.0] + sizes = [p.size_mult for p in self.postures] or [0.0] + return { + "variant": self.variant, + "ticks": self.ticks, + "net": round(self.equity_curve[-1] if self.equity_curve else 0.0, 4), + "realized": round(self.ledger.realized - self.ledger.fees, 4), + "fees": round(self.ledger.fees, 4), + "fills": self.ledger.fills, + "round_trips": self.ledger.round_trips, + "volume": round(self.ledger.volume, 2), + "open_lots": len(self.ledger.open_lots), + "applies": self.applies, + "unconfident": self.unconfident, + "mean_spread_mult": round(sum(spreads) / len(spreads), 3), + "mean_size_mult": round(sum(sizes) / len(sizes), 3), + "one_sided": sum(1 for p in self.postures if p.side != "both"), + "regimes": regimes, + } + + +def frames( + pair: str, candles: list[dict], window: int +) -> list[tuple[np.ndarray, dict, float]]: + """``(frame, candle, mid)`` per tick, over a sliding window of candles. + + The bid/ask drawn on the frame are the previous close plus and minus half + the observed spread of that bar — the live loop draws the live book, and a + candle series has no book. The tick's *decisions* are then applied to the + following candle, so nothing is decided on a bar it can already see. + """ + out = [] + for i in range(window, len(candles) - 1): + history = candles[i - window : i] + last = float(history[-1]["close"]) + half = max( + last * 1e-5, (float(history[-1]["high"]) - float(history[-1]["low"])) / 200 + ) + frame = market_frame(pair, history, last - half, last + half, n_candles=window) + out.append((frame, candles[i], last)) + return out + + +def replay( + variant: str, + pair: str, + candles: list[dict], + spec: MarketSpec, + settings: DecoderSettings, + observe, + window: int = 72, + deadband_bps: float = 1.0, + shuffle_seed: int | None = None, + neural_ms: float = 500.0, +) -> ReplayResult: + """Walk the candles once, exactly as the live loop walks wall time. + + ``observe(frame, stimulus, neural_ms) -> dict`` is the brain, injected so a + caller can hand in a fresh worker per variant — a brain that has already + learned from one variant is not a control for the next. + + ``shuffle_seed`` replaces each pulse with a randomly signed one of the same + magnitude. The fly still gets reinforced exactly as often and as hard; only + the correspondence to its own P&L is destroyed. If that scores the same, + what the synapses hold is not about this market. + """ + # How many open positions the controller would tolerate, rather than a + # number picked here: two levels a side, each allowed its own concurrent + # executors. A cap set independently of the strategy quietly becomes the + # thing under test — with a fixed 8, widening the take-profit blocked new + # fills by leaving lots open, so the exit was measured through the cap. + max_lots = spec.max_active_executors_by_level * 2 * len(SIDES) + baseline = Baseline() + ledger = Ledger() + hysteresis = Hysteresis(min_apply_interval_sec=0.0) + rng = random.Random(shuffle_seed) if shuffle_seed is not None else None + deadband = deadband_bps * BPS * spec.total_amount_quote + + config = build_config(spec, decode(Channels(0.0, 0.0, 0), Baseline(), settings)) + previous: Posture | None = None + anchor = 0.0 + applies = holds = unconfident = 0 + curve: list[float] = [] + postures: list[Posture] = [] + + for tick, (frame, candle, mid) in enumerate(frames(pair, candles, window)): + equity = ledger.equity(mid) + kind, _ = reinforcement(equity, anchor, deadband) + if rng is not None and kind != "none": + kind = rng.choice(["reward", "aversive"]) + anchor = equity + + neural = observe(frame, kind, neural_ms) + posture = decode( + Channels( + neural["trend_hz"], + neural["arousal_hz"], + neural["gate_spikes"], + neural["valence_hz"], + neural["kc_spikes"], + ), + baseline, + settings, + ) + postures.append(posture) + if not posture.confident: + unconfident += 1 + ok, _ = should_apply(previous, posture, None, float(tick), hysteresis) + if ok: + config = build_config(spec, posture) + previous = posture + applies += 1 + else: + holds += 1 + + step(ledger, config, spec, candle, mid, max_lots) + curve.append(ledger.equity(float(candle["close"]))) + + return ReplayResult( + variant=variant, + ticks=len(curve), + applies=applies, + holds=holds, + unconfident=unconfident, + ledger=ledger, + equity_curve=curve, + postures=postures, + ) + + +def windows(candles: list[dict], count: int, span: int) -> list[list[dict]]: + """Split a series into ``count`` contiguous slices of equal length. + + One window is one sample of one market, and a sample of one is how the sign + of a result flips between two runs ninety minutes apart. Contiguous rather + than overlapping, so the slices share no candle and each is an independent + draw of the same market. + """ + if count < 1: + raise ValueError("count must be at least 1") + usable = len(candles) // count + if usable - span - 1 < MIN_TICKS_PER_WINDOW: + raise ValueError( + f"{len(candles)} candles split {count} ways leaves {usable} each, " + f"which is {max(0, usable - span - 1)} ticks after a {span}-candle " + f"window — not enough to score. A window needs at least " + f"{MIN_TICKS_PER_WINDOW}: fetch more candles or ask for fewer windows" + ) + return [candles[i * usable : (i + 1) * usable] for i in range(count)] + + +def pooled_stats(pairs: list[tuple[list[float], list[float]]]) -> dict: + """One verdict over several windows: the increments pooled, and how many + windows the first run actually led in. + + The pooled t answers "does it earn differently"; the count answers "or did + one window carry it". A result that is real should do both, and a result + that leads in two windows of four is noise however large its total. + """ + gains: list[float] = [] + led = 0 + for a, b in pairs: + n = min(len(a), len(b)) + if n < 4: + continue + gains.extend((a[i] - a[i - 1]) - (b[i] - b[i - 1]) for i in range(1, n)) + if a[n - 1] > b[n - 1]: + led += 1 + if len(gains) < 4: + return {"n": 0, "windows": len(pairs), "led": led, "mean_diff": 0.0, "t": 0.0} + mean = sum(gains) / len(gains) + var = sum((g - mean) ** 2 for g in gains) / (len(gains) - 1) + sd = math.sqrt(var) + scale = max(abs(mean), max((abs(g) for g in gains), default=0.0)) + return { + "n": len(gains), + "windows": len(pairs), + "led": led, + "mean_diff": mean, + "sd": sd, + "t": 0.0 if sd <= scale * 1e-9 else mean / (sd / math.sqrt(len(gains))), + } + + +def paired_stats(a: list[float], b: list[float]) -> dict: + """Do two runs earn differently per tick? Mean difference, sd, and t. + + On the *increments*, not the levels. An equity curve is cumulative: once + two runs separate, every later tick inherits that gap, so a t-test on + levels measures how long ago they diverged rather than whether they earn + differently. Run on levels this returned |t| of 27 to 63 for variants whose + final P&L differed by a few percent — a number that says nothing except + that a random walk is autocorrelated. + + Even on increments this is a weak instrument, and it is reported as one: + one replay of one market, with fills that assume a touched price was ours. + A |t| under 2 is not evidence of a difference; it is also not evidence of + sameness. + """ + n = min(len(a), len(b)) + if n < 4: + return {"n": 0, "mean_diff": 0.0, "sd": 0.0, "t": 0.0, "final_gap": 0.0} + gains_a = [a[i] - a[i - 1] for i in range(1, n)] + gains_b = [b[i] - b[i - 1] for i in range(1, n)] + deltas = [x - y for x, y in zip(gains_a, gains_b)] + k = len(deltas) + mean = sum(deltas) / k + var = sum((d - mean) ** 2 for d in deltas) / (k - 1) + sd = math.sqrt(var) + # A difference with no spread has no sampling variation, so t is undefined + # rather than enormous — and floating point will not produce a clean zero: + # the increments of a perfect ramp differ in the last bits, which took the + # naive expression to 4e15. Below a hair of the scale it measures, the + # answer is "read the mean, there is nothing here to test". + scale = max(abs(mean), max((abs(d) for d in deltas), default=0.0)) + degenerate = sd <= scale * 1e-9 + return { + "n": k, + "mean_diff": mean, + "sd": sd, + "t": 0.0 if degenerate else mean / (sd / math.sqrt(k)), + "final_gap": a[n - 1] - b[n - 1], + } diff --git a/agents/market_making_fly/flybrain/run_state.py b/agents/market_making_fly/flybrain/run_state.py new file mode 100644 index 000000000..5286b2cd1 --- /dev/null +++ b/agents/market_making_fly/flybrain/run_state.py @@ -0,0 +1,218 @@ +"""One fly run's durable state on disk. + +Stonkfly's pattern: a lock so two workers never share a run directory, two +alternating brain checkpoints so a crash mid-write leaves the previous one +intact, an append-only ``events.jsonl`` audit trail, ``latest.json`` and +``latest-input.png`` for "what did the fly just see and do", and a provenance +signature that refuses to resume a run under a changed protocol. +""" + +from __future__ import annotations + +import fcntl +import hashlib +import json +from pathlib import Path + +import numpy as np +from PIL import Image + +from condor.fsutil import atomic_write_bytes, atomic_write_json + +FLY_PACKAGE = Path(__file__).resolve().parent + + +def source_hashes() -> dict[str, str]: + return { + str(path.relative_to(FLY_PACKAGE)): hashlib.sha256( + path.read_bytes() + ).hexdigest() + for path in sorted(FLY_PACKAGE.rglob("*")) + if path.suffix in (".py", ".cpp") and path.is_file() + } + + +# Recorded in provenance.json for the audit trail, but not part of the +# signature a resume is checked against: a bug fix in the loop must not orphan +# a brain lineage. Settings, decoder, guard, dataset and circuit are. +# The shape of what ``state.json`` holds, and how the code reads it back. +# BUMP THIS whenever a persisted field is added, removed or reinterpreted: +# resuming an old run on code that reads its state differently is how a fixed +# bug comes back. ``session_high_net`` going from ``0.0`` to ``None`` did +# exactly that — a resumed run read its first reported figure as a drawdown +# from zero and halted. +STATE_VERSION = 1 + +# Recorded in provenance.json for the audit trail, but not signed: a bug fix +# in the loop must not orphan a brain lineage. What the code *means* by the +# persisted state is pinned by STATE_VERSION instead. +# ``state_version`` is compared directly in check_provenance, so it is kept +# out of the signature — inside it, re-signing would overwrite it and the +# comparison would never fire. +UNSIGNED_KEYS = ("source_sha256", "state_version") + +# Cadence is the operator's to change mid-lineage: it moves no money and +# reinterprets no accounting. Everything else about the deployment is signed. +# Sizing, leverage, allocation and venue all denominate the anchor, the session +# high and the carried P&L that a resume restores, so changing one while +# restoring the other half of the books is how a run halts on a mix of two +# deployments' numbers. Changing them needs a new run_name. +UNSIGNED_SETTINGS = ("interval_sec",) + + +def signature(provenance: dict) -> str: + signed = {k: v for k, v in provenance.items() if k not in UNSIGNED_KEYS} + settings = signed.get("settings") + if isinstance(settings, dict): + signed["settings"] = { + k: v for k, v in settings.items() if k not in UNSIGNED_SETTINGS + } + return hashlib.sha256( + json.dumps(signed, sort_keys=True, default=str).encode() + ).hexdigest() + + +class RunDir: + def __init__(self, root: Path): + self.root = Path(root) + self.root.mkdir(parents=True, exist_ok=True) + self.state_path = self.root / "state.json" + self.events_path = self.root / "events.jsonl" + self.latest_path = self.root / "latest.json" + self.frame_path = self.root / "latest-input.png" + self.activity_path = self.root / "latest-activity.json" + self.provenance_path = self.root / "provenance.json" + self.lock_path = self.root / "worker.lock" + self.stop_path = self.root / "STOP" + self._lock = None + + # -- exclusivity --------------------------------------------------------- + + def lock(self) -> None: + handle = self.lock_path.open("a") + try: + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + handle.close() + raise RuntimeError(f"A fly worker already owns {self.root}") + self._lock = handle + + def unlock(self) -> None: + if self._lock is not None: + fcntl.flock(self._lock, fcntl.LOCK_UN) + self._lock.close() + self._lock = None + + def stop_requested(self) -> bool: + return self.stop_path.exists() + + # -- state --------------------------------------------------------------- + + def load_state(self) -> dict: + if not self.state_path.exists(): + return {} + return json.loads(self.state_path.read_text()) + + def save_state(self, state: dict) -> None: + atomic_write_json(self.state_path, state, indent=2, default=str) + + def append_event(self, row: dict) -> None: + with self.events_path.open("a") as handle: + handle.write(json.dumps(row, allow_nan=False, default=str) + "\n") + + def write_latest(self, row: dict) -> None: + atomic_write_json(self.latest_path, row, indent=2, default=str) + + def save_activity(self, counts: list[int], tick: int, pair: str) -> None: + """Spike counts at the cloud's neurons, for the report to colour by. + + Overwritten each observation rather than appended: it is a few thousand + numbers, which would bury `events.jsonl` within an hour, and only the + latest is ever drawn. + + Stamped with the observation it came from. The file is written when the + brain runs, but the tick can still fail afterwards in the guard or the + apply — and the report then falls back to the last observation that + completed, which is a *different* one. Unstamped, a round-robin run + would draw one market's neurons beside another market's posture. + """ + atomic_write_json( + self.activity_path, {"tick": tick, "pair": pair, "counts": counts} + ) + + def load_activity( + self, tick: int | None = None, pair: str = "" + ) -> list[int] | None: + """The spike counts, but only if they belong to the observation asked + for. A mismatch returns None, which draws the anatomy instead of + somebody else's activity.""" + if not self.activity_path.exists(): + return None + saved = json.loads(self.activity_path.read_text()) + if not isinstance(saved, dict): + return None # a snapshot from before they were stamped + if tick is not None and ( + saved.get("tick") != tick or saved.get("pair") != pair + ): + return None + return saved.get("counts") + + def save_frame(self, frame: np.ndarray) -> None: + Image.fromarray(frame).save(self.frame_path) + + def recent_events(self, limit: int) -> list[dict]: + if not self.events_path.exists(): + return [] + lines = self.events_path.read_text().splitlines()[-limit:] + return [json.loads(line) for line in lines if line.strip()] + + # -- checkpoints --------------------------------------------------------- + + def checkpoint_path(self, tick: int) -> Path: + return self.root / f"brain-{tick % 2}.npz" + + def verify_checkpoint(self, info: dict) -> Path: + path = self.root / info["file"] + if not path.exists(): + raise RuntimeError( + f"Checkpoint {path.name} recorded in state.json is missing" + ) + if hashlib.sha256(path.read_bytes()).hexdigest() != info["sha256"]: + raise RuntimeError(f"Checkpoint {path.name} integrity mismatch") + return path + + # -- provenance ---------------------------------------------------------- + + def check_provenance(self, provenance: dict) -> str: + """Write the run's provenance on first start; refuse a changed one.""" + sig = signature(provenance) + if self.provenance_path.exists(): + recorded = json.loads(self.provenance_path.read_text()) + # Compared directly, never through the signature: re-signing the + # recorded provenance below rewrites whatever the current rule + # controls, which would make this check silently inert. + recorded_version = recorded.get("state_version") + if recorded_version != STATE_VERSION: + raise RuntimeError( + f"Run state is version {recorded_version}, this code reads " + f"version {STATE_VERSION}; its persisted state would be " + "reinterpreted. Use a new run_name." + ) + # Re-sign what was recorded under the current rule, so a change to + # which keys are signed does not itself refuse every existing run. + recorded_sig = signature( + {k: v for k, v in recorded.items() if k != "signature"} + ) + if recorded_sig != sig: + raise RuntimeError( + "Run protocol changed (settings, decoder, dataset or source); " + "use a new run_name or review the migration explicitly" + ) + return sig + atomic_write_json( + self.provenance_path, + {"signature": sig, "state_version": STATE_VERSION, **provenance}, + indent=2, + default=str, + ) + return sig diff --git a/agents/market_making_fly/flybrain/venue.py b/agents/market_making_fly/flybrain/venue.py new file mode 100644 index 000000000..16623df88 --- /dev/null +++ b/agents/market_making_fly/flybrain/venue.py @@ -0,0 +1,203 @@ +"""What differs between one CLOB venue and the next. + +The fly's decoder, its chart and its dopamine feedback are venue-agnostic: a +candle chart is a candle chart. Three things are not, and they all bear on +money, so they live here rather than being assumed: + +* **spot or perp** — a perp quotes on margin and carries leverage and a + position mode; a spot book quotes the inventory you actually hold. +* **the maker fee** — the take-profit floor is derived from it. Spot fees run + three to five times perp fees on the same exchange, so a take-profit that is + comfortably profitable on a perp loses money on spot. Getting this wrong is + silent: the bot fills happily and bleeds the difference. On Hyperliquid it + also differs *within* one connector by a factor of two, so that venue's fee + is fetched per market rather than assumed — see ``hyperliquid_maker_fee_bps``. +* **where the top of book comes from** — see ``market.py``. + +The fee table below is a floor-setting default, not a quote. Defaults are +deliberately **conservative** (too high costs fills, too low loses money +without saying so), and every one of them is overridable per deployment. +""" + +from __future__ import annotations + +import asyncio + +PERP_MARKERS = ("_perpetual", "_perp", "_futures") + +SPOT = "spot" +PERP = "perp" +MARKET_TYPES = (SPOT, PERP) + +# Maker fee per side, in basis points. Keys are matched as a prefix of the +# connector name, longest first, so `binance_perpetual` beats `binance`. +_MAKER_FEE_BPS: dict[tuple[str, str], float] = { + # Hyperliquid is fetched, not tabled — see hyperliquid_maker_fee_bps. These + # two are what a caller gets if that fetch is not used: the dearer of the + # venue's two families, so an unfetched fee never quotes too tight. + ("hyperliquid_perpetual", PERP): 2.5, + ("hyperliquid", SPOT): 5.0, + ("binance_perpetual", PERP): 2.0, # 0.02 % + ("binance", SPOT): 7.5, # 0.075 % + ("gate_io_perpetual", PERP): 2.0, + ("gate_io", SPOT): 9.0, + ("okx_perpetual", PERP): 2.0, + ("okx", SPOT): 8.0, + ("kucoin_perpetual", PERP): 2.0, + ("kucoin", SPOT): 10.0, + ("backpack", SPOT): 8.0, +} + +# Used when the connector is not in the table at all. A market maker on an +# unknown venue should quote too wide rather than too tight. +_FALLBACK_FEE_BPS = {PERP: 2.5, SPOT: 10.0} + + +def market_type_for(connector_name: str) -> str: + """``perp`` when the connector names itself one, else ``spot``. + + This is the same rule Market Making Expert uses, and the same one + hummingbot follows: the `_perpetual` suffix is the contract type. + """ + if not connector_name: + raise ValueError("connector_name is required") + lowered = connector_name.lower() + return PERP if any(m in lowered for m in PERP_MARKERS) else SPOT + + +def default_maker_fee_bps(connector_name: str, market_type: str) -> float: + """A conservative per-side maker fee for this venue, in bp. + + Always overridable: pass the real figure from the exchange's fee schedule + when it is known, because everything about the take-profit floor follows + from it. + """ + if market_type not in MARKET_TYPES: + raise ValueError(f"market_type must be one of {MARKET_TYPES}") + lowered = (connector_name or "").lower() + matches = [ + (key, fee) + for (key, kind), fee in _MAKER_FEE_BPS.items() + if kind == market_type and lowered.startswith(key) + ] + if not matches: + return _FALLBACK_FEE_BPS[market_type] + # Longest prefix wins: binance_perpetual is not binance. + return max(matches, key=lambda m: len(m[0]))[1] + + +def resolve(connector_name: str, market_type: str = "") -> str: + """The market type to use: an explicit one, else derived from the connector.""" + if not market_type: + return market_type_for(connector_name) + if market_type not in MARKET_TYPES: + raise ValueError( + f"market_type must be one of {MARKET_TYPES}, got {market_type!r}" + ) + derived = market_type_for(connector_name) + if market_type != derived: + # Naming a perp connector spot (or the reverse) silently changes the + # leverage rule and the fee floor, so say so rather than proceed. + raise ValueError( + f"{connector_name!r} looks like a {derived} connector but was declared " + f"{market_type!r}; check the connector name" + ) + return market_type + + +# ── Hyperliquid: the fee is published, so it is read rather than assumed ────── +# +# One connector serves two families of market, and they do not cost the same: +# +# * a core perp pays the venue's own schedule — 1.5 bp a side to a maker at the +# base tier; +# * a HIP-3 market (``ISSUER:TOKEN-QUOTE``) pays that schedule scaled by its +# deployer's own setting, and again by a tenth where the deployer has turned +# growth mode on. The XYZ dex is in growth mode, so it costs 0.3 bp a side. +# +# Everything above is fetched. One number is not: Hummingbot signs every +# Hyperliquid order with the foundation builder code, and the venue charges +# that on top of its own fee. It is a constant of the client, not of the +# market — ``FOUNDATION_BUILDER_FEE_TENTHS_BPS = 10`` in +# hummingbot/connector/derivative/hyperliquid_perpetual/hyperliquid_perpetual_constants.py +# — and it is not importable here, so it is named rather than fetched. +# +# Checked against fills: 173 maker fills on XYZ:ORCL-USD paid 1.29 bp all-in, +# against 1.5 × 2 × 0.1 + 1.0 = 1.30 bp computed. Hyperliquid reports ``fee`` +# inclusive of the builder fee, which is why one number covers both. +HL_INFO_URL = "https://api.hyperliquid.xyz/info" +HUMMINGBOT_BUILDER_FEE_BPS = 1.0 +GROWTH_MODE_FACTOR = 0.1 +# The published schedule, which is what an account with no discounts pays. +# Reading it for a real address would return that account's own rate; a +# market maker sizing a floor wants the undiscounted one. +_SCHEDULE_PROBE_ADDRESS = "0x0000000000000000000000000000000000000001" + +_hl_cache: dict[str, dict] = {} +_hl_lock = asyncio.Lock() + + +def is_hyperliquid(connector_name: str) -> bool: + return (connector_name or "").lower().startswith("hyperliquid") + + +async def _hl_info(payload: dict) -> dict: + """One Hyperliquid info call, cached for the life of the process. + + The fee schedule and a dex's deployer settings change on the order of + months (the XYZ dex last changed its scale in November 2025), so a scan of + 120 markets should not ask 120 times. + """ + import aiohttp + + key = repr(sorted(payload.items())) + async with _hl_lock: + if key not in _hl_cache: + async with aiohttp.ClientSession() as session: + async with session.post(HL_INFO_URL, json=payload) as response: + response.raise_for_status() + _hl_cache[key] = await response.json() + return _hl_cache[key] + + +async def hyperliquid_maker_fee_bps(trading_pair: str, market_type: str) -> float: + """What one maker side of ``trading_pair`` actually costs, in bp.""" + from flybrain.naming import pair_names + + if market_type not in MARKET_TYPES: + raise ValueError(f"market_type must be one of {MARKET_TYPES}") + names = pair_names(trading_pair) + schedule = (await _hl_info({"type": "userFees", "user": _SCHEDULE_PROBE_ADDRESS}))[ + "feeSchedule" + ] + base = float(schedule["spotAdd" if market_type == SPOT else "add"]) * 1e4 + if not names.issuer: + return base + HUMMINGBOT_BUILDER_FEE_BPS + + meta = await _hl_info({"type": "meta", "dex": names.issuer}) + asset = next( + (a for a in meta["universe"] if a["name"] == names.hl_coin), + None, + ) + if asset is None: + raise ValueError( + f"{names.hl_coin!r} is not listed on the {names.issuer!r} dex, so its " + "fee cannot be read" + ) + scale = float(asset["deployerFeeScale"]) + # The deployer's own multiplier, as Hyperliquid documents it: below 1 it + # adds to the venue's fee, at or above 1 it doubles the scale. + multiplier = scale * 2 if scale >= 1 else scale + 1 + if asset.get("growthMode") == "enabled": + multiplier *= GROWTH_MODE_FACTOR + return base * multiplier + HUMMINGBOT_BUILDER_FEE_BPS + + +async def maker_fee_bps( + connector_name: str, market_type: str, trading_pair: str +) -> float: + """The per-side maker fee for one market: fetched where the venue publishes + it per market, the venue default otherwise.""" + if is_hyperliquid(connector_name): + return await hyperliquid_maker_fee_bps(trading_pair, market_type) + return default_maker_fee_bps(connector_name, market_type) diff --git a/agents/market_making_fly/flybrain/worker.py b/agents/market_making_fly/flybrain/worker.py new file mode 100644 index 000000000..461d9020b --- /dev/null +++ b/agents/market_making_fly/flybrain/worker.py @@ -0,0 +1,214 @@ +"""The brain lives here, in its own process. + +``FlyBrain`` wraps stonkfly's ``VisualMemoryBrain`` and measures the three +channels the decoder reads. The module-level ``_init`` / ``_observe`` / +``_checkpoint`` / ``_restore`` functions are the ``ProcessPoolExecutor`` +targets the ``fly_brain`` routine submits, so the C++ integration never runs on +Condor's event loop. One worker holds one brain; the routine creates the pool +with ``max_workers=1`` and the ``spawn`` context. +""" + +from __future__ import annotations + +import hashlib +import math +from pathlib import Path + +import numpy as np + +# MaleCNS v1.0 superclass label of the 1,314 descending neurons in graph.npz. +DESCENDING_SUPERCLASS = "descending_neuron" + + +class FlyBrain: + def __init__( + self, + learning: bool = True, + neural_bin_ms: float = 10.0, + pulse_ms: float = 200.0, + pulse_current: float = 20.0, + ): + from flybrain.neural.common import annotations + from flybrain.neural.visual import VisualMemoryBrain + + if neural_bin_ms <= 0 or neural_bin_ms > 10: + raise ValueError("neural_bin_ms must be in (0, 10]") + if pulse_ms <= 0 or pulse_current <= 0: + raise ValueError("pulse_ms and pulse_current must be positive") + self.neural_bin_ms = neural_bin_ms + self.pulse_ms = pulse_ms + self.pulse_current = pulse_current + self.brain = VisualMemoryBrain() + self.brain.weights_frozen = not learning + a = annotations(self.brain.ids) + types = a.type.fillna("") + sides = a.somaSide.fillna("") + self.left = np.flatnonzero(types.eq("DNp20") & sides.eq("L")) + self.right = np.flatnonzero(types.eq("DNp20") & sides.eq("R")) + self.gate = np.flatnonzero(types.eq("DNpe017")) + if not len(self.left) or not len(self.right) or not len(self.gate): + raise RuntimeError("Missing annotated DNp20 / DNpe017 readout cells") + # The memory rule's own output. KC→MBON07/11 are the synapses dopamine + # moves, so their firing is the only place a P&L pulse can reach a + # decision. Read here rather than through the vendored circuit map, + # which bundles both into one `mb` array, and for the same reason the + # other readouts are read here: this is what the decoder consumes. + self.mbon_approach = np.flatnonzero(types.eq("MBON07")) + self.mbon_avoid = np.flatnonzero(types.eq("MBON11")) + if not len(self.mbon_approach) or not len(self.mbon_avoid): + raise RuntimeError("Missing annotated MBON07 / MBON11 memory outputs") + superclass = np.asarray(self.brain.superclass).astype(str) + readouts = np.concatenate([self.left, self.right, self.gate]) + descending = np.flatnonzero(superclass == DESCENDING_SUPERCLASS) + self.descending = np.setdiff1d(descending, readouts) + if not len(self.descending): + raise RuntimeError( + f"No {DESCENDING_SUPERCLASS!r} superclass in graph.npz; " + f"available: {sorted(set(superclass.tolist()))}" + ) + # The report draws the connectome's somata; carrying their spike counts + # back is what lets it colour the animal by what actually fired, rather + # than by cell class alone. A few thousand ints per observation. + from flybrain.cloud import load as load_cloud + + self.cloud_index = load_cloud()["index"].astype(np.int64) + self.cell_ids = { + "left": [str(self.brain.ids[i]) for i in self.left], + "right": [str(self.brain.ids[i]) for i in self.right], + "gate": [str(self.brain.ids[i]) for i in self.gate], + "mbon_approach": [str(self.brain.ids[i]) for i in self.mbon_approach], + "mbon_avoid": [str(self.brain.ids[i]) for i in self.mbon_avoid], + "descending_count": int(len(self.descending)), + } + + def observe( + self, frame: np.ndarray, reinforcement: str, neural_ms: float = 500.0 + ) -> dict: + if reinforcement not in ("none", "reward", "aversive"): + raise ValueError("Unknown reinforcement") + if not math.isfinite(neural_ms) or neural_ms < self.pulse_ms: + raise ValueError("neural_ms must be finite and >= pulse_ms") + frame = np.asarray(frame) + if frame.shape != (180, 320, 3) or frame.dtype != np.uint8: + raise ValueError("frame must be uint8[180, 320, 3]") + b = self.brain + counts = np.zeros(b.n, dtype=np.int32) + wall = 0.0 + remaining = round(neural_ms / b.dt) + pulse = round(self.pulse_ms / b.dt) if reinforcement != "none" else 0 + delivered = 0 + learning = not b.weights_frozen + while remaining: + n = min(remaining, round(self.neural_bin_ms / b.dt)) + if pulse: + n = min(n, pulse) + stimulus = (b.circuit[reinforcement], self.pulse_current) if pulse else None + c, elapsed = b.rgb_step( + frame, n * b.dt, learning=learning, stimulation=stimulus + ) + counts += c + wall += elapsed + remaining -= n + if pulse: + delivered += n + pulse -= n + b.counts[:] = counts + seconds = neural_ms / 1000 + left = float(np.mean(counts[self.left]) / seconds) + right = float(np.mean(counts[self.right]) / seconds) + approach = float(np.mean(counts[self.mbon_approach]) / seconds) + avoid = float(np.mean(counts[self.mbon_avoid]) / seconds) + return { + "trend_hz": right - left, + "left_hz": left, + "right_hz": right, + # What the memory rule has made of this scene: MBON07 approach + # minus MBON11 avoidance. Positive is the learned "go". + "valence_hz": approach - avoid, + "mbon_approach_hz": approach, + "mbon_avoid_hz": avoid, + "arousal_hz": float(np.mean(counts[self.descending]) / seconds), + "gate_spikes": int(counts[self.gate].sum()), + # Over the whole network, not extrapolated from the drawn sample: + # the report's cloud is stratified, so a fraction of it would not be + # a fraction of the animal. + "active_neurons": int((counts > 0).sum()), + "mean_rate_hz": float(counts.mean() / seconds), + "kc_spikes": int(counts[b.circuit["kc"]].sum()), + "reward_spikes": int(counts[b.circuit["reward"]].sum()), + "aversive_spikes": int(counts[b.circuit["aversive"]].sum()), + "total_spikes": int(counts.sum()), + "stimulus": reinforcement, + "stimulus_ms": delivered * b.dt, + # This observation's neural time, and the brain's total since it + # was seeded. Reporting the second as the first says a 500 ms + # observation ran for ten seconds once twenty of them have gone by. + "observation_ms": float(neural_ms), + "brain_ms": b.sim_ms, + "compute_seconds": wall, + "spike_sha256": hashlib.sha256(counts.tobytes()).hexdigest(), + "input_sha256": hashlib.sha256(frame.tobytes()).hexdigest(), + "memory": b.memory(), + "cell_ids": self.cell_ids, + "activity": counts[self.cloud_index].astype(int).tolist(), + } + + def save(self, path: Path) -> None: + self.brain.checkpoint(path) + + def restore(self, path: Path) -> None: + self.brain.restore(path) + + def provenance(self) -> dict: + return { + "circuit": self.brain.circuit["report"], + "vision": self.brain.visual_report, + "readout": { + "trend": "DNp20 mean right minus left rate", + "arousal": f"mean rate of superclass={DESCENDING_SUPERCLASS!r} minus readouts", + "valence": "mean MBON07 rate minus mean MBON11 rate", + "gate": "DNpe017 spike count >= 1", + "cells": self.cell_ids, + "validated": False, + }, + } + + +# ---- ProcessPoolExecutor targets (one brain per worker process) ---------- + +_BRAIN: FlyBrain | None = None + + +def _init( + learning: bool, neural_bin_ms: float, pulse_ms: float, pulse_current: float +) -> None: + global _BRAIN + _BRAIN = FlyBrain( + learning=learning, + neural_bin_ms=neural_bin_ms, + pulse_ms=pulse_ms, + pulse_current=pulse_current, + ) + + +def _brain() -> FlyBrain: + if _BRAIN is None: + raise RuntimeError("worker not initialized") + return _BRAIN + + +def _observe(frame: np.ndarray, reinforcement: str, neural_ms: float) -> dict: + return _brain().observe(frame, reinforcement, neural_ms) + + +def _checkpoint(path: str) -> str: + _brain().save(Path(path)) + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +def _restore(path: str) -> None: + _brain().restore(Path(path)) + + +def _provenance() -> dict: + return _brain().provenance() diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py new file mode 100644 index 000000000..9c531c90f --- /dev/null +++ b/agents/market_making_fly/routines/fly_brain.py @@ -0,0 +1,760 @@ +"""The fly loop: chart → connectome → posture → pmm_mister config, with P&L dopamine. + +One shared brain is shown up to three markets in round-robin — any CLOB spot or +perp market hummingbot-api serves. Each tick: + +1. fetch candles + live book for this tick's pair; +2. read the combined net P&L of the fly's bots, turn its change since the last + observation into ``reward`` / ``aversive`` / ``none``; +3. render the chart, run the neural window in the worker process (the dopamine + pulse is delivered during it), decode spike counts into a posture; +4. checkpoint the brain and commit the accounting anchor BEFORE anything is + applied; +5. map the posture to a config, pass it through the guard, apply it in ``live`` + mode (``shadow`` only records what it would have applied); +6. persist events, latest.json, the input frame, and the live report. + +The guard can veto or halt; nothing in this file chooses a posture. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) +# Condor re-executes this file when it changes but keeps imported modules +# cached: after editing anything under flybrain/, restart Condor. Purging the +# cache here is not an option — each routine would re-import its own copy and +# the spawned brain worker could no longer pickle flybrain.worker._init. + +import asyncio +import logging +import multiprocessing +import time +from concurrent.futures import ProcessPoolExecutor +from dataclasses import asdict +from pathlib import Path + +import numpy as np +import plotly.graph_objects as go +from flybrain import venue, worker +from flybrain.chart import market_frame +from flybrain.decoder import ( + Baseline, + Channels, + DecoderSettings, + Hysteresis, + Posture, + decode, + should_apply, +) +from flybrain.guard import ( + GuardSettings, + GuardState, + Halt, + Veto, + check_apply_window, + check_collateral, + check_config, + check_market_open, + check_not_halted, + check_pnl, + check_price_move, + default_max_loss, + rebase, + record_apply, + resume, +) +from flybrain.market import ( + FixtureMarket, + LiveMarket, + book_restarted, + pnl_is_known, + quote_tokens, + required_collateral, +) +from flybrain.naming import pair_names, parse_pairs +from flybrain.posture import MarketSpec, build_config, config_diff +from flybrain.reinforcement import reinforcement +from flybrain.run_state import RunDir, source_hashes +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.memory.paths import agent_home +from condor.paths import safe_id +from condor.reports import LiveReport + +logger = logging.getLogger(__name__) + +CONTINUOUS = True +CATEGORY = "Monitoring" +AGENT_SLUG = "market_making_fly" +ROUTINE_NAME = "fly_brain" +BPS = 1e-4 + + +class Config(BaseModel): + """Fly-connectome market maker: one brain, up to three HIP-3 markets, P&L dopamine.""" + + pairs: str = Field( + default="XYZ:ORCL-USD", + description="1 to 3 uppercase pairs, comma-separated — BASE-QUOTE on any CLOB venue, or ISSUER:TOKEN-QUOTE on HIP-3. This list IS the market count; each pair names its own bot ({slug}-fly) and config ({slug}_fly_mm)", + ) + picked_ranges_bps: str = Field( + default="10,10,10", + description="Scanner median candle range per pair in bp, same order as " + "pairs. Quote levels are placed against how far the market travels, not " + "against how wide its touch is", + ) + connector_name: str = Field( + default="hyperliquid_perpetual", + description="Any CLOB connector hummingbot-api serves, spot or perp (a _perpetual suffix means perp)", + ) + market_type: str = Field( + default="", + description="spot | perp; blank derives it from the connector name", + ) + maker_fee_bps: float = Field( + default=0.0, + description="Maker fee per side in bp; 0 uses a conservative venue default. The take-profit floor is derived from it, so pass the exchange's real figure when you know it", + ) + total_amount_quote: float = Field( + default=500.0, description="Capital per pair (quote)" + ) + leverage: int = Field( + default=1, + description="Leverage per pair (perp only, cap 5); must be 1 on spot", + ) + portfolio_allocation: float = Field( + default=0.2, + description="Fraction of total_amount_quote quoted per cycle; each order is total × allocation / 4 and must clear the exchange minimum (10 USD on HIP-3) — one market at 200 quote needs 0.2+", + ) + mode: str = Field( + default="shadow", description="shadow (record only) or live (apply configs)" + ) + learning: bool = Field( + default=True, description="False freezes the memory rule (control run)" + ) + run_name: str = Field( + default="fly", + description="Run directory under the agent home (letters, digits, dot, dash, underscore); new name = new brain lineage", + ) + resume_reviewed: bool = Field( + default=False, description="Clear a transient halt after review" + ) + interval_sec: int = Field( + default=60, description="Wall seconds between observations" + ) + neural_ms: float = Field( + default=500.0, description="Neural time per observation (ms)" + ) + candle_interval: str = Field( + default="5m", description="Candle interval the fly sees" + ) + n_candles: int = Field(default=72, description="Candles on the chart") + reward_deadband_bps: float = Field( + default=1.0, description="Pulse deadband, bp of combined capital" + ) + baseline_window: int = Field( + default=60, description="Observations per pair in the rolling baseline" + ) + baseline_warmup: int = Field( + default=10, description="Observations before the first non-neutral posture" + ) + center_bias: bool = Field( + default=True, description="Subtract the rolling mean of the trend channel" + ) + min_apply_interval_sec: int = Field( + default=300, description="Per-pair cooldown between config applies" + ) + max_loss_quote: float = Field( + default=0.0, description="Loss stop in quote; 0 = 4% of combined capital" + ) + fixture: bool = Field( + default=False, description="Offline synthetic market (shadow only)" + ) + fast: bool = Field( + default=False, description="Skip wall waits (fixture/shadow only)" + ) + steps: int = Field( + default=0, description="Stop after N observations; 0 runs until stopped" + ) + + +async def _specs(config: Config, pairs: list[str]) -> list[MarketSpec]: + """One spec per pair, each carrying the fee its own market charges. + + The fee is not a property of the connector: one Hyperliquid account pays + 2.5 bp a side on a core perp and 1.3 bp on a HIP-3 market in growth mode. + The take-profit floor is derived from it, so it is read per pair unless the + caller passed a figure of their own. + """ + ranges = [float(x) for x in config.picked_ranges_bps.split(",") if x.strip()] + if len(ranges) != len(pairs): + raise ValueError( + f"picked_ranges_bps has {len(ranges)} entries for {len(pairs)} pairs" + ) + market_type = venue.resolve(config.connector_name, config.market_type) + return [ + MarketSpec( + connector_name=config.connector_name, + trading_pair=pair, + total_amount_quote=config.total_amount_quote, + range_bps=market_range, + market_type=config.market_type, + leverage=config.leverage, + maker_fee_bps=( + config.maker_fee_bps + or await venue.maker_fee_bps(config.connector_name, market_type, pair) + ), + portfolio_allocation=config.portfolio_allocation, + ) + for pair, market_range in zip(pairs, ranges) + ] + + +def _frame_figure(frame: np.ndarray) -> go.Figure: + fig = go.Figure(go.Image(z=frame)) + fig.update_layout( + margin=dict(l=0, r=0, t=0, b=0), + height=360, + xaxis=dict(visible=False), + yaxis=dict(visible=False), + legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), + ) + return fig + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + chat_id = context._chat_id + if config.mode not in ("shadow", "live"): + raise ValueError("mode must be shadow or live") + if config.fixture and config.mode != "shadow": + raise ValueError("fixture runs are shadow only") + if config.fast and config.mode == "live": + raise ValueError("fast is for fixture/shadow runs only") + if config.interval_sec < 10: + raise ValueError("interval_sec must be >= 10") + pairs = parse_pairs(config.pairs) + specs = await _specs(config, pairs) + for spec in specs: + spec.check_order_size() # fail at start, not on the first apply + by_pair = {s.trading_pair: s for s in specs} + decoder_settings = DecoderSettings( + window=config.baseline_window, + warmup=config.baseline_warmup, + center_bias=config.center_bias, + ) + hysteresis = Hysteresis(min_apply_interval_sec=config.min_apply_interval_sec) + guard_settings = GuardSettings(max_loss_quote=config.max_loss_quote) + max_loss = default_max_loss(specs, guard_settings) + deadband = ( + config.reward_deadband_bps * BPS * sum(s.total_amount_quote for s in specs) + ) + + from flybrain.neural.common import DATA, GRAPH + + if not GRAPH.exists(): + raise RuntimeError( + f"Connectome not prepared at {DATA}; run the fly_setup routine with " + 'action="prepare" first (downloads ~1.1 GB, needs a C++ compiler)' + ) + run_dir = RunDir(agent_home(AGENT_SLUG) / "fly" / safe_id(config.run_name)) + run_dir.lock() + pool = ProcessPoolExecutor( + max_workers=1, + mp_context=multiprocessing.get_context("spawn"), + initializer=worker._init, + initargs=(config.learning, 10.0, 200.0, 20.0), + ) + loop = asyncio.get_running_loop() + report = LiveReport( + "Market Making Fly", + source_name=ROUTINE_NAME, + tags=["fly", "market-making", "hip3", config.mode], + auto_refresh_seconds=config.interval_sec, + ) + count = 0 + stop_reason = "stopped" + try: + # ---- state, provenance, checkpoint -------------------------------- + state = run_dir.load_state() + guard_state = GuardState.from_dict(state.get("guard")) + if guard_state.halted: + resume( + guard_state, config.resume_reviewed + ) # raises Halt when not clearable + baselines = { + p: Baseline.from_dict(state.get("baselines", {}).get(p)) for p in pairs + } + postures: dict[str, Posture | None] = { + p: Posture.from_dict(v) if (v := state.get("postures", {}).get(p)) else None + for p in pairs + } + applied: dict[str, dict | None] = { + p: state.get("applied", {}).get(p) for p in pairs + } + anchor = state.get("anchor") + tick = int(state.get("tick", 0)) + pnl_carry: dict = state.get("pnl_carry", {}) + # Shadow keeps its own per-pair apply clock so the trial shows the same + # cooldown live would, without touching the live guard's accounting. + shadow_last: dict = state.get("shadow_last", {}) + + from flybrain.data import verify + + dataset = await loop.run_in_executor(None, verify) + brain_prov = await loop.run_in_executor(pool, worker._provenance) + provenance = { + "settings": { + k: v + for k, v in config.model_dump().items() + if k + not in ( + "pairs", + "picked_ranges_bps", + "mode", + "fast", + "steps", + "resume_reviewed", + "run_name", + "fixture", + ) + }, + "decoder": asdict(decoder_settings), + "guard": asdict(guard_settings), + "dataset": dataset, + **brain_prov, + "learning_validated": False, + "source_sha256": source_hashes(), + } + run_dir.check_provenance(provenance) + if state.get("checkpoint"): + path = run_dir.verify_checkpoint(state["checkpoint"]) + await loop.run_in_executor(pool, worker._restore, str(path)) + + if config.fixture: + market = FixtureMarket(pairs, config.n_candles) + else: + from config_manager import get_client + + client = await get_client(chat_id, context=context) + if not client: + raise RuntimeError("No Hummingbot server available for this chat") + market = LiveMarket( + client, config.connector_name, config.candle_interval, config.n_candles + ) + + await context.bot.send_message( + chat_id=chat_id, + text=( + f"🪰 Fly started [{config.mode}] run={config.run_name} tick={tick} " + f"pairs={','.join(pairs)} learning={config.learning}" + ), + ) + + def persist(extra: dict | None = None) -> None: + run_dir.save_state( + { + "tick": tick, + "anchor": anchor, + "guard": guard_state.to_dict(), + "baselines": {p: b.to_dict() for p, b in baselines.items()}, + "postures": { + p: (q.to_dict() if q else None) for p, q in postures.items() + }, + "applied": applied, + "pnl_carry": pnl_carry, + "shadow_last": shadow_last, + **(extra or {}), + } + ) + + async def pace() -> None: + """Wait out the rest of the interval; every tick path ends here.""" + if config.fast: + return + until = started + config.interval_sec + while time.monotonic() < until: + if run_dir.stop_requested(): + break + await asyncio.sleep(min(1.0, until - time.monotonic())) + + # ---- the loop ------------------------------------------------------- + while not config.steps or count < config.steps: + started = time.monotonic() + if run_dir.stop_requested(): + stop_reason = "STOP file" + break + if guard_state.halted: + stop_reason = f"halted: {guard_state.halted}" + break + pair = pairs[tick % len(pairs)] + spec = by_pair[pair] + row: dict = { + "tick": tick, + "wall_time": time.time(), + "pair": pair, + "mode": config.mode, + } + try: + obs = await market.observe(pair) + equity, volume, per_pair, pnl_carry = await market.equity( + pairs, pnl_carry + ) + pnl_known = pnl_is_known(per_pair) + restarted = book_restarted(per_pair) + if restarted: + # A redeployed controller reports from zero: that is not a + # loss to punish, and its predecessor's high is not a height + # to measure it against. + rebase(guard_state) + if anchor is None or not pnl_known or restarted: + kind, delta = "none", 0.0 + else: + kind, delta = reinforcement(equity, anchor, deadband) + delta = float(delta) + row.update( + { + "quote": {"bid": obs.bid, "ask": obs.ask, "open": obs.open}, + "equity": equity, + "volume": volume, + "pnl_delta": delta, + "stimulus": kind, + "pnl_known": pnl_known, + "restarted": restarted, + "bots": per_pair, + } + ) + if pnl_known: + # Breakers judge reported P&L only; an unreported book is + # neither a loss nor a high. + check_pnl(equity, volume, guard_state, guard_settings, max_loss) + + if not obs.open: + # Nothing new for the fly to see; keep the anchor so the next + # open observation carries the whole interval's P&L change. + try: + stop = check_market_open( + pair, False, guard_state, guard_settings + ) + except Veto as veto: + row["execution"] = {"status": "CLOSED", "reason": str(veto)} + stop = False + if stop: + stopped = config.mode == "live" and await market.stop_bot(pair) + row["execution"] = { + "status": "STOP_BOT" if stopped else "CLOSED", + "reason": f"book closed {guard_state.closed_ticks[pair]} ticks", + } + tick += 1 + persist() + run_dir.append_event(row) + run_dir.write_latest(row) + count += 1 + if config.steps and count >= config.steps: + stop_reason = f"{count} steps done" + break + await pace() + continue + check_market_open(pair, True, guard_state, guard_settings) + + frame = market_frame( + pair, obs.candles, obs.bid, obs.ask, config.n_candles + ) + neural = await loop.run_in_executor( + pool, worker._observe, frame, kind, config.neural_ms + ) + posture = decode( + Channels( + neural["trend_hz"], + neural["arousal_hz"], + neural["gate_spikes"], + neural["valence_hz"], + neural["kc_spikes"], + ), + baselines[pair], + decoder_settings, + ) + proposed = build_config(spec, posture) + + # Checkpoint and anchor are committed before any apply. + ck = run_dir.checkpoint_path(tick) + sha = await loop.run_in_executor(pool, worker._checkpoint, str(ck)) + if pnl_known: + anchor = equity + tick += 1 + persist({"checkpoint": {"file": ck.name, "sha256": sha}}) + + run_dir.save_activity(neural["activity"], tick - 1, pair) + neural_row = { + k: v for k, v in neural.items() if k not in ("cell_ids", "activity") + } + row.update({"neural": neural_row, "posture": posture.to_dict()}) + + now = time.time() + last_apply_ts = ( + guard_state.last_apply.get(pair) + if config.mode == "live" + else shadow_last.get(pair) + ) + ok, why = should_apply( + postures[pair], posture, last_apply_ts, now, hysteresis + ) + diff = config_diff(applied[pair], proposed) + if not ok: + row["execution"] = {"status": "HOLD", "reason": why} + elif config.mode == "shadow": + postures[pair] = posture + shadow_last[pair] = now + row["execution"] = { + "status": "SHADOW", + "reason": why, + "would_apply": diff, + } + else: + try: + check_not_halted(guard_state) + unreported = [ + p + for p, info in per_pair.items() + if info.get("running") and not info.get("reported") + ] + if unreported: + raise Veto( + "no performance report for " + + ", ".join(unreported) + + "; the guard cannot see P&L" + ) + check_apply_window(guard_state, now, guard_settings) + check_config(proposed, spec) + check_collateral( + await market.available_quote(quote_tokens(specs)), + required_collateral(specs), + ) + check_price_move( + obs.mid, await market.fresh_mid(pair), guard_settings + ) + try: + await market.apply(pair, proposed) + except Exception as failure: + logger.exception("fly apply failed for %s", pair) + row["execution"] = { + "status": "ERROR", + "reason": repr(failure)[:300], + } + record_apply(guard_state, pair, now, False, guard_settings) + else: + record_apply(guard_state, pair, now, True, guard_settings) + postures[pair] = posture + applied[pair] = proposed + row["execution"] = { + "status": "APPLIED", + "reason": why, + "diff": diff, + } + except Veto as veto: + row["execution"] = {"status": "VETO", "reason": str(veto)} + persist() + run_dir.append_event(row) + run_dir.write_latest(row) + run_dir.save_frame(frame) + + # ---- live report -------------------------------------------- + report.clear() + b = report.builder + b.manual_order() + b.section( + "01 / WHAT THE FLY SEES", + f"{pair} — last input frame, tick {tick - 1}", + ) + b.plotly(_frame_figure(frame)) + b.section( + "02 / THIS OBSERVATION", "Spikes, channels and the decoded posture" + ) + b.kpi("Regime", posture.regime) + b.kpi("Spread ×", f"{posture.spread_mult:.2f}") + b.kpi("Size ×", f"{posture.size_mult:.2f}") + b.kpi("Valence z", f"{posture.valence_z:+.2f}") + b.kpi("Lean", f"{posture.shift_bps:+.2f} bp") + b.kpi("Trend z", f"{posture.trend_z:+.2f}") + b.kpi("Arousal z", f"{posture.arousal_z:+.2f}") + b.kpi("Gate", str(neural["gate_spikes"])) + b.kpi("Stimulus", kind) + b.kpi("P&L Δ", f"{delta:+.4f}") + b.kpi("Equity", f"{equity:+.4f}") + b.kpi("Execution", row["execution"]["status"]) + b.kpi("KC spikes", str(neural["kc_spikes"])) + b.kpi("Changed edges", str(neural["memory"]["changed_edges"])) + b.kpi("Mean efficacy", f"{neural['memory']['mean_efficacy']:.4f}") + b.kpi("Compute s", f"{neural['compute_seconds']:.1f}") + b.section( + "03 / POSTURES", "Last decided posture per pair and what is applied" + ) + b.table( + [ + { + "Pair": p, + "Regime": (postures[p].regime if postures[p] else "—"), + "Spread ×": ( + f"{postures[p].spread_mult:.2f}" if postures[p] else "—" + ), + "Lean bp": ( + f"{postures[p].shift_bps:+.2f}" if postures[p] else "—" + ), + "Baseline n": baselines[p].count, + "Applied buy": (applied[p] or {}).get("buy_spreads", "—"), + "Applied sell": (applied[p] or {}).get("sell_spreads", "—"), + "Bot": ( + "running" if per_pair.get(p, {}).get("running") else "—" + ), + } + for p in pairs + ], + [ + "Pair", + "Regime", + "Spread ×", + "Lean bp", + "Baseline n", + "Applied buy", + "Applied sell", + "Bot", + ], + ) + b.section("04 / RECENT TICKS", "Newest last") + events = run_dir.recent_events(40) + b.table( + [ + { + "Tick": e["tick"], + "Pair": e["pair"], + "Stim": e.get("stimulus", "—"), + "Δ": f"{e.get('pnl_delta', 0):+.3f}", + "Regime": (e.get("posture") or {}).get("regime", "—"), + "Spread ×": (e.get("posture") or {}).get( + "spread_mult", "—" + ), + "Lean": (e.get("posture") or {}).get("shift_bps", "—"), + "Exec": (e.get("execution") or {}).get("status", "—"), + "Reason": str((e.get("execution") or {}).get("reason", ""))[ + :60 + ], + } + for e in events + ], + [ + "Tick", + "Pair", + "Stim", + "Δ", + "Regime", + "Spread ×", + "Lean", + "Exec", + "Reason", + ], + ) + b.section("05 / GUARD", "Halts, breakers, apply budget") + b.kpi("Halted", guard_state.halted or "no") + b.kpi("Applies today", str(guard_state.applies_today)) + # None until a bot has reported P&L: the first report sets the + # high, and a zero here would read as a real high the run had + # already fallen from. + b.kpi( + "Session high", + ( + "—" + if guard_state.session_high_net is None + else f"{guard_state.session_high_net:+.4f}" + ), + ) + b.kpi("Ticks since high", str(guard_state.ticks_since_high)) + b.kpi("Loss stop", f"-{max_loss:.2f}") + b.markdown( + "_The regime, spread multiplier and lean are decoded from spike counts " + "of identified cells; the mapping is engineered and unvalidated. Dopamine " + "pulses report P&L change, not credit for the last posture. See " + "agents/market_making_fly/README.md §15._" + ) + await report.update() + + if row["execution"]["status"] in ("APPLIED", "STOP_BOT", "ERROR"): + await context.bot.send_message( + chat_id=chat_id, + text=( + f"🪰 {pair} tick {tick - 1}: {row['execution']['status']} — " + f"{posture.regime} ×{posture.spread_mult:.2f} " + f"size ×{posture.size_mult:.2f} {posture.shift_bps:+.1f}bp " + f"({row['execution']['reason']})" + )[:900], + ) + except asyncio.CancelledError: + raise + except Halt as halt: + row["execution"] = { + "status": "HALT", + "reason": halt.reason, + "financial": halt.financial, + } + persist() + run_dir.append_event(row) + run_dir.write_latest(row) + stopped = [] + if config.mode == "live": + for p in pairs: + try: + if await market.stop_bot(p): + stopped.append(pair_names(p).bot_name) + except Exception: + logger.exception("fly: stopping %s after halt failed", p) + await context.bot.send_message( + chat_id=chat_id, + text=( + f"🪰 HALT ({'financial' if halt.financial else 'transient'}): {halt.reason}. " + f"Stopped bots: {', '.join(stopped) or 'none'}. " + + ( + "A financial halt needs a new run_name." + if halt.financial + else "Restart with resume_reviewed=true after review." + ) + ), + ) + stop_reason = f"halted: {halt.reason}" + break + except Exception as failure: + logger.exception("fly tick %s failed", tick) + row["execution"] = { + "status": "TICK_ERROR", + "reason": repr(failure)[:300], + } + run_dir.append_event(row) + run_dir.write_latest(row) + count += 1 + if config.steps and count >= config.steps: + stop_reason = f"{count} steps done" + break + await pace() + except asyncio.CancelledError: + stop_reason = "cancelled" + raise + finally: + if report.report_id is not None: + report.clear() + report.builder.auto_refresh(None) + report.builder.section("FLY STOPPED", stop_reason) + report.builder.markdown(f"Run directory: `{run_dir.root}`") + await report.update() + pool.shutdown(wait=True, cancel_futures=True) + run_dir.unlock() + try: + await context.bot.send_message( + chat_id=chat_id, + text=f"🪰 Fly stopped after {count} observations: {stop_reason}", + ) + except Exception: + logger.warning("fly: final notification failed") + return f"Fly stopped after {count} observations: {stop_reason}" diff --git a/agents/market_making_fly/routines/fly_chart.py b/agents/market_making_fly/routines/fly_chart.py new file mode 100644 index 000000000..e5e4f7c0e --- /dev/null +++ b/agents/market_making_fly/routines/fly_chart.py @@ -0,0 +1,94 @@ +"""Render the exact frame the fly would see for a HIP-3 pair, and report it.""" + +from __future__ import annotations + +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + +import logging + +import aiohttp +import plotly.graph_objects as go +from flybrain.chart import market_frame, normalize_candles, price_scale +from flybrain.market import fetch_l2_book, normalize_candle_payload +from flybrain.naming import pair_names +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.reports import ReportBuilder +from config_manager import get_client + +logger = logging.getLogger(__name__) + +CATEGORY = "Market Data" + + +class Config(BaseModel): + """What the fly sees: the 320×180 OHLCV frame for one HIP-3 pair.""" + + trading_pair: str = Field( + default="XYZ:DRAM-USD", description="Uppercase HIP-3 pair" + ) + connector_name: str = Field( + default="hyperliquid_perpetual", description="Connector" + ) + candle_interval: str = Field(default="5m", description="Candle interval") + n_candles: int = Field(default=72, description="Candles on the chart") + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + client = await get_client(context._chat_id, context=context) + if not client: + return "No server available" + names = pair_names(config.trading_pair) + candles = normalize_candle_payload( + await client.market_data.get_candles( + config.connector_name, + config.trading_pair, + interval=config.candle_interval, + max_records=config.n_candles, + ) + ) + async with aiohttp.ClientSession() as session: + book = await fetch_l2_book(session, names.coin) + if not book.open: + return f"{config.trading_pair}: book is closed (no bid/ask) — nothing to render" + frame = market_frame( + config.trading_pair, candles, book.bid, book.ask, config.n_candles + ) + rows = normalize_candles(candles, config.n_candles) + lo, span = price_scale(rows) + + fig = go.Figure(go.Image(z=frame)) + fig.update_layout( + margin=dict(l=0, r=0, t=0, b=0), + height=360, + xaxis=dict(visible=False), + yaxis=dict(visible=False), + legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), + ) + builder = ReportBuilder(f"Fly view — {config.trading_pair}") + builder.source("routine", "fly_chart") + builder.tags(["fly", "chart", "hip3"]) + builder.manual_order() + builder.section("01 / FRAME", "320×180 RGB, exactly what enters the retina") + builder.plotly(fig) + builder.kpi("Candles", str(len(rows))) + builder.kpi("Interval", config.candle_interval) + builder.kpi("Bid", f"{book.bid:g}") + builder.kpi("Ask", f"{book.ask:g}") + builder.kpi("Price floor", f"{lo:g}") + builder.kpi("Price span", f"{span:g}") + builder.markdown( + "_Up candles blue, down candles red: the mapped R8p cells read blue and R8y " + "read green; R1–R6 read luminance. No quotes, inventory or P&L are drawn._" + ) + await builder.save() + return ( + f"{config.trading_pair}: {len(rows)} × {config.candle_interval} candles, " + f"bid {book.bid:g} ask {book.ask:g}, last close {rows[-1]['close']:g}" + ) diff --git a/agents/market_making_fly/routines/fly_replay.py b/agents/market_making_fly/routines/fly_replay.py new file mode 100644 index 000000000..7d1f8b2c8 --- /dev/null +++ b/agents/market_making_fly/routines/fly_replay.py @@ -0,0 +1,508 @@ +"""Run the fly over recorded candles, several ways, and compare them. + +The agent's central claim — that P&L feedback shapes what the fly does — has +never had a control, because every live run is one sample of a market that +never repeats. This runs the same candles through the same brain as many times +as there are variants, changing one setting each time: + +* ``live`` — the fly as deployed. +* ``no-memory``— the memory rule frozen. If this scores the same, the plastic + synapses are decoration. +* ``no-valence``— the memory rule still runs, but its output is disconnected + from the posture (``valence_gain`` ~ 0). Separates "the rule + does nothing" from "the rule does something the decoder does + not read". +* ``shuffled`` — reinforcement of the same frequency and magnitude, with the + sign randomised. The control the caveats have always demanded. +* ``widen`` — the old arousal direction, for the A/B that motivated the flip. +* ``two-sided``— a side threshold no trend reaches, so both sides stay on the + book. Says what taking a side away is worth. + +Each variant gets its own brain process: a network that has already learned +from one variant is not a control for the next. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + +import asyncio +import json +import logging +import multiprocessing +import time +from concurrent.futures import ProcessPoolExecutor + +import plotly.graph_objects as go +from flybrain import venue, worker +from flybrain.decoder import DecoderSettings +from flybrain.fly3d import ACCENT, BODY, GROUND, LIMB +from flybrain.naming import pair_names +from flybrain.posture import MarketSpec +from flybrain.replay import paired_stats, pooled_stats, replay, windows +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.memory.paths import agent_home +from condor.reports import ReportBuilder + +logger = logging.getLogger(__name__) + +CATEGORY = "Bot Analysis" +AGENT_SLUG = "market_making_fly" + +# A gain cannot be zero — the decoder refuses a channel it would read and +# discard — so "disconnected" is the smallest gain that rounds out of every +# posture the size multiplier can express. +OFF = 1e-9 + +VARIANTS: dict[str, dict] = { + "live": {}, + "no-memory": {"learning": False}, + "no-valence": {"valence_gain": OFF}, + "shuffled": {"shuffle": True}, + "widen": {"spread_gain": 0.5}, + "two-sided": {"z_side": 99.0}, +} + + +class Config(BaseModel): + """Replay the fly over historical candles and compare its variants.""" + + trading_pair: str = Field(default="XYZ:DRAM-USD", description="Market to replay") + connector_name: str = Field(default="hyperliquid_perpetual") + interval: str = Field(default="5m", description="Candle interval, as the fly sees") + max_records: int = Field( + default=1000, ge=200, le=5000, description="Candles to fetch" + ) + variants: str = Field( + default="live,no-memory,no-valence,shuffled,widen,two-sided", + description=f"Comma-separated, from: {', '.join(VARIANTS)}", + ) + total_amount_quote: float = Field(default=200.0) + portfolio_allocation: float = Field(default=0.3) + range_bps: float = Field( + default=0.0, description="0 measures the median candle range from the data" + ) + maker_fee_bps: float = Field(default=0.0, description="0 fetches the venue's") + leverage: int = Field(default=1) + n_candles: int = Field(default=72, description="Window the retina sees") + neural_ms: float = Field(default=500.0) + baseline_window: int = Field(default=60) + baseline_warmup: int = Field(default=10) + seed: int = Field(default=7301, description="Seed for the shuffled control") + windows: int = Field( + default=1, + ge=1, + le=8, + description="Replay every variant on this many contiguous slices of the " + "series, and judge a control by how many of them it lost. One window is " + "one sample: the sign of a result flipped between two windows on " + "2026-09-13, so a single one settles nothing either way", + ) + refresh_candles: bool = Field( + default=False, + description="Fetch a new window and pin it. Off by default: the venue " + "only serves the latest N candles, so two runs an hour apart replay " + "different markets and are not comparable — the sign of a control's " + "difference flipped between two such windows on 2026-09-13", + ) + concurrency: int = Field( + default=4, + ge=1, + le=8, + description="Variants to replay at once. Each holds its own brain (~250 MB) " + "and saturates one core; they are independent, so this is wall time " + "divided rather than work shared", + ) + + +def _settings(config: Config, overrides: dict) -> DecoderSettings: + fields = { + "window": config.baseline_window, + "warmup": config.baseline_warmup, + } + fields.update( + {k: v for k, v in overrides.items() if k not in ("learning", "shuffle")} + ) + return DecoderSettings(**fields) + + +def _curve_figure(results: list) -> go.Figure: + fig = go.Figure() + palette = [ACCENT, "#7fb2ff", "#f2a35c", "#c58cff", "#6fd3c0"] + for n, result in enumerate(results): + fig.add_trace( + go.Scatter( + x=list(range(len(result.equity_curve))), + y=result.equity_curve, + mode="lines", + name=result.variant, + line=dict(color=palette[n % len(palette)], width=2), + ) + ) + fig.update_layout( + height=366, + margin=dict(l=56, r=16, t=10, b=40), + paper_bgcolor=GROUND, + plot_bgcolor=GROUND, + font=dict(color=BODY, family="monospace", size=11), + xaxis=dict(title="tick", gridcolor="#18202e", zerolinecolor="#18202e"), + yaxis=dict( + title="net P&L (quote)", gridcolor="#18202e", zerolinecolor="#243044" + ), + legend=dict(orientation="h", yanchor="top", y=-0.18, xanchor="center", x=0.5), + ) + fig.add_hline(y=0, line=dict(color=LIMB, width=1, dash="dot")) + return fig + + +async def _candles(client, config: Config, pinned: Path) -> list[dict]: + """The series every variant replays — and every *later* run replays too. + + The venue serves only the latest N candles, so fetching each time means two + runs an hour apart are scored on different markets. That is not a detail: + between two such windows the sign of the fly's difference from its frozen + control reversed. So the first fetch is pinned to disk and reused until + someone asks for a new one, which is what makes a lever's before and after + a comparison rather than two anecdotes. + """ + if pinned.exists() and not config.refresh_candles: + saved = json.loads(pinned.read_text()) + if saved.get("interval") == config.interval and saved.get("candles"): + return saved["candles"] + for attempt in (1, 2): # a cold feed answers on the second ask + try: + raw = await client.market_data.get_candles( + config.connector_name, + config.trading_pair, + interval=config.interval, + max_records=config.max_records, + ) + rows = raw if isinstance(raw, list) else raw.get("data", raw.get("candles")) + if rows: + pinned.parent.mkdir(parents=True, exist_ok=True) + pinned.write_text( + json.dumps( + { + "pair": config.trading_pair, + "interval": config.interval, + "fetched_at": time.time(), + "candles": list(rows), + } + ) + ) + return list(rows) + except Exception: + if attempt == 2: + raise + raise RuntimeError(f"No candles for {config.trading_pair}") + + +def _range_from_candles(candles: list[dict]) -> float: + """The median bar's range in bp — the quantity the quote levels are built + from, and the same median the scanner ranks reach by.""" + import statistics + + ranges = [ + (float(c["high"]) - float(c["low"])) / float(c["close"]) * 1e4 + for c in candles + if float(c.get("close") or 0) > 0 + ] + if not ranges: + raise ValueError("No usable candles to measure a range from") + return round(statistics.median(ranges), 2) + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + from config_manager import get_client + + names = [v.strip() for v in config.variants.split(",") if v.strip()] + unknown = [v for v in names if v not in VARIANTS] + if unknown: + raise ValueError(f"Unknown variant(s) {unknown}; choose from {list(VARIANTS)}") + + client = await get_client(context._chat_id, context=context) + if not client: + return "No server available" + slug = pair_names(config.trading_pair).slug + home = agent_home(AGENT_SLUG) / "replay" + home.mkdir(parents=True, exist_ok=True) + pinned = home / f"{slug}-{config.interval}-candles.json" + candles = await _candles(client, config, pinned) + market_type = venue.market_type_for(config.connector_name) + fee = config.maker_fee_bps or await venue.maker_fee_bps( + config.connector_name, market_type, config.trading_pair + ) + market_range = config.range_bps or _range_from_candles(candles) + spec = MarketSpec( + connector_name=config.connector_name, + trading_pair=config.trading_pair, + total_amount_quote=config.total_amount_quote, + range_bps=market_range, + leverage=config.leverage, + portfolio_allocation=config.portfolio_allocation, + maker_fee_bps=fee, + ) + spec.check_order_size() + record = home / f"{slug}.json" + + loop = asyncio.get_running_loop() + gate = asyncio.Semaphore(config.concurrency) + + async def one(name: str, series: list[dict], label: str): + overrides = VARIANTS[name] + async with gate: + # A fresh process per variant: the brain is stateful, and one that + # has already learned is not a control for the next. + pool = ProcessPoolExecutor( + max_workers=1, + mp_context=multiprocessing.get_context("spawn"), + initializer=worker._init, + initargs=(overrides.get("learning", True), 10.0, 200.0, 20.0), + ) + try: + + def observe(frame, stimulus, neural_ms, _pool=pool): + return _pool.submit( + worker._observe, frame, stimulus, neural_ms + ).result() + + result = await loop.run_in_executor( + None, + lambda o=overrides, ob=observe: replay( + variant=name, + pair=config.trading_pair, + candles=series, + spec=spec, + settings=_settings(config, o), + observe=ob, + window=config.n_candles, + shuffle_seed=config.seed if o.get("shuffle") else None, + neural_ms=config.neural_ms, + ), + ) + finally: + pool.shutdown(wait=True) + await context.bot.send_message( + chat_id=context._chat_id, + text=f"🪰 replay {name}{label}: net {result.equity_curve[-1]:+.4f} " + f"over {result.ticks} ticks, {result.ledger.fills} fills", + ) + return result + + slices = windows(candles, config.windows, config.n_candles) + # Order is the caller's, not the order they finished in: the first variant + # is the baseline every control is compared against. + per_window: list[list] = [] + for index, series in enumerate(slices, 1): + label = f" w{index}" if len(slices) > 1 else "" + per_window.append( + list(await asyncio.gather(*(one(name, series, label) for name in names))) + ) + results = per_window[0] + summaries = [r.summary() for r in results] + # The brains are the expensive part and the arithmetic over their output is + # not; keeping the curves means a better statistic never costs another run. + record.write_text( + json.dumps( + { + "pair": config.trading_pair, + "interval": config.interval, + "candles": len(candles), + "range_bps": market_range, + "fee_bps": fee, + "summaries": summaries, + "curves": {r.variant: r.equity_curve for r in results}, + }, + indent=1, + ) + ) + builder = ReportBuilder(f"Fly replay — {config.trading_pair}") + builder.source("routine", "fly_replay") + builder.tags(["fly", "replay", "control", config.trading_pair]) + builder.manual_order() + builder.section( + "WHAT WAS REPLAYED", + f"{len(candles):,} {config.interval} candles of {config.trading_pair}" + + (" (freshly fetched)" if config.refresh_candles else " (the pinned window)") + + ", " + f"{results[0].ticks if results else 0} ticks after the " + f"{config.n_candles}-candle window. Every variant saw the same series and " + f"the same geometry — {market_range:.2f} bp median bar range, {fee:.2f} bp " + "maker fee — and each ran on its own freshly seeded brain. Fills assume " + "a quote the price touched was ours, so every P&L here is an upper " + "bound; the bias is identical across variants, which is what makes the " + "comparison worth reading and the absolute number not.", + ) + builder.kpi("Market", config.trading_pair) + builder.kpi("Candles", f"{len(candles):,}") + builder.kpi("Ticks", f"{results[0].ticks if results else 0:,}") + builder.kpi("Maker fee", f"{fee:.2f} bp") + + builder.section("VARIANTS", "One row per run, over identical candles") + builder.table( + [ + { + "Variant": s["variant"], + "Net P&L": f"{s['net']:+.4f}", + "Realized": f"{s['realized']:+.4f}", + "Fees": f"{s['fees']:.4f}", + "Fills": f"{s['fills']:,}", + "Round trips": f"{s['round_trips']:,}", + "Applies": f"{s['applies']:,}", + "Unconfident": f"{s['unconfident']:,}", + "Spread ×": f"{s['mean_spread_mult']:.2f}", + "Size ×": f"{s['mean_size_mult']:.2f}", + "One-sided": f"{s['one_sided']:,}", + } + for s in summaries + ], + [ + "Variant", + "Net P&L", + "Realized", + "Fees", + "Fills", + "Round trips", + "Applies", + "Unconfident", + "Spread ×", + "Size ×", + "One-sided", + ], + ) + builder.plotly(_curve_figure(results)) + + if len(results) > 1 and len(per_window) > 1: + # The verdict that matters when there is more than one window: the + # increments pooled across all of them, and how many windows the fly + # actually lost. A control it beats in two of four is noise whatever + # the total says. + builder.section( + "ACROSS WINDOWS", + f"{len(per_window)} contiguous slices of the same series, each " + "replayed by every variant on its own freshly seeded brain. " + "'Windows lost' counts the slices where the control finished ahead " + "of the deployed fly — a real difference should show in the pooled " + "increments *and* in most windows, and one that shows in only the " + "total was carried by one slice.", + ) + pooled_rows = [] + for position, name in enumerate(names[1:], start=1): + curves = [ + (window[0].equity_curve, window[position].equity_curve) + for window in per_window + ] + stats = pooled_stats(curves) + pooled_rows.append( + { + "Against": f"{names[0]} − {name}", + "Pooled per-tick": f"{stats['mean_diff']:+.5f}", + "Pooled t": f"{stats['t']:+.2f}", + "Windows won": f"{stats['led']}/{stats['windows']}", + "Reads as": ( + "distinguishable" + if abs(stats["t"]) >= 2 + and stats["led"] in (0, stats["windows"]) + else "not distinguishable" + ), + } + ) + builder.table( + pooled_rows, + ["Against", "Pooled per-tick", "Pooled t", "Windows won", "Reads as"], + ) + + if len(results) > 1: + base = results[0] + rows = [] + for other in results[1:]: + stats = paired_stats(base.equity_curve, other.equity_curve) + rows.append( + { + "Against": f"{base.variant} − {other.variant}", + "Mean per-tick earnings difference": f"{stats['mean_diff']:+.5f}", + "SD": f"{stats['sd']:.5f}", + "Final gap": f"{stats['final_gap']:+.4f}", + "t": f"{stats['t']:+.2f}", + "Reads as": ( + "distinguishable" + if abs(stats["t"]) >= 2 + else "not distinguishable" + ), + } + ) + builder.section( + "AGAINST THE CONTROLS" + (" (first window)" if len(per_window) > 1 else ""), + "How differently the deployed fly earns per tick, against each " + "control. On increments, not on the equity curves themselves: a " + "curve is cumulative, so once two runs separate every later tick " + "inherits the gap and a t on levels measures when they diverged " + "rather than whether they earn differently. One replay of one " + "market is still a weak instrument — |t| under 2 is not evidence of " + "a difference, and it is not evidence of sameness either.", + ) + builder.table( + rows, + [ + "Against", + "Mean per-tick earnings difference", + "SD", + "Final gap", + "t", + "Reads as", + ], + ) + + builder.markdown( + "_A variant that scores better here has not been shown to make money: " + "the fill model is optimistic, one market is one sample, and the " + "take-profit that never fills is marked to the close rather than to " + "what it would cost to get out. What replay can establish is the " + "negative — that a variant is **not** distinguishable from its control, " + "which is the claim this agent has never been able to test._" + ) + await builder.save() + + lines = [ + f"pair: {config.trading_pair} ({config.interval})", + f"candles: {len(candles)} in {len(per_window)} window(s), " + f"ticks each: {results[0].ticks if results else 0}", + f"median range: {market_range:.2f} bp, fee: {fee:.2f} bp", + ] + for s in summaries: + lines.append( + f"{s['variant']}: net {s['net']:+.4f}, {s['fills']} fills, " + f"{s['round_trips']} round trips, {s['applies']} applies, " + f"spread ×{s['mean_spread_mult']:.2f}, size ×{s['mean_size_mult']:.2f}, " + f"{s['one_sided']} one-sided" + ) + if len(results) > 1: + for position, name in enumerate(names[1:], start=1): + if len(per_window) > 1: + stats = pooled_stats( + [ + (window[0].equity_curve, window[position].equity_curve) + for window in per_window + ] + ) + lines.append( + f"{names[0]} vs {name}: pooled per-tick {stats['mean_diff']:+.5f}, " + f"t {stats['t']:+.2f}, won {stats['led']}/{stats['windows']} windows" + ) + else: + stats = paired_stats( + results[0].equity_curve, results[position].equity_curve + ) + lines.append( + f"{names[0]} vs {name}: mean per-tick earnings " + f"{stats['mean_diff']:+.5f}, final gap {stats['final_gap']:+.4f}, " + f"t {stats['t']:+.2f}" + ) + return "\n".join(lines) diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py new file mode 100644 index 000000000..adb7e2cb3 --- /dev/null +++ b/agents/market_making_fly/routines/fly_report.py @@ -0,0 +1,507 @@ +"""A dashboard for a running fly: the fly itself, its book, its neurons, its calls. + +Laid out after stonkfly's dashboard — the fly, the bag, the neuron strip, the +latest neural order, the decision log, and what the fly is actually looking at. +Stonkfly draws its fly with Three.js; a Condor report carries Plotly, so the fly +here is Mesh3d geometry that orbits by dragging and beats its wings on play. + +Everything except the fly is read back from the run directory and the live bot, +so the report says what happened rather than what was intended. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + +import json +import logging +import time + +import numpy as np +import plotly.graph_objects as go +from flybrain.brainviz import brain_figure, coverage, readout_figure +from flybrain.fly3d import ACCENT, BODY, GROUND, LIMB, fly_figure +from flybrain.market import LiveMarket +from flybrain.naming import pair_names +from flybrain.run_state import RunDir +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.fetchers.bot_performance import count_trade_closes +from condor.memory.paths import agent_home +from condor.paths import safe_id +from condor.reports import ReportBuilder + +logger = logging.getLogger(__name__) + +CATEGORY = "Bot Analysis" + +# Columns of the report's 12-wide grid, and the panel heights that make each +# row's two halves finish level on a wide screen. Everything collapses to full +# width under the runtime's 800px breakpoint, where heights stop mattering. +# +# A panel is its figure plus 34px of padding and border, measured rather than +# guessed. Narrow panels are also released from the stylesheet's 400px floor, +# which exists to stop a full-width chart being squashed and would otherwise +# stop the brain matching the card stack beside it. +FLY, FRAME = 6, 6 +# Six and six, not five and seven: the cards wrap by their own 230px minimum, +# so a five-column stack needs a ~1150px container before it fits two per row +# and stops towering over the brain. Six needs ~970, which covers every screen +# that is meaningfully wider than the 800px breakpoint. +CARDS, BRAIN = 6, 6 +PANEL_CHROME = 34 # the panel's own padding and border, measured +FULL_ROW = 400 # the stylesheet's floor for a full-width chart panel +# Row one is sized so the frame fills its half: 16:9 at six columns is ~335px. +ROW_ONE = 430 +# Row two matches the card stack, which is what it is: six cards at two per +# row is three rows of roughly 98px plus two 16px gaps. A card's exact height +# drifts a few pixels with the container, so this lands within ~5px rather +# than exactly — nothing about a KPI card's height is ours to set. +ROW_TWO = 3 * 98 + 2 * 16 +AGENT_SLUG = "market_making_fly" + +# How an execution status reads in the decision log. +RESULT_WORDS = { + "APPLIED": "APPLIED", + "SHADOW": "SHADOW", + "HOLD": "HOLD", + "VETO": "VETO", + "CLOSED": "BOOK CLOSED", + "STOP_BOT": "BOT STOPPED", + "ERROR": "UPDATE FAILED", + "HALT": "HALT", + "TICK_ERROR": "TICK FAILED", +} + + +class Config(BaseModel): + """Dashboard for one fly run: the fly, its book, its neurons and its calls.""" + + run_name: str = Field( + default="fly", description="Run directory under the agent home" + ) + recent: int = Field(default=12, ge=1, le=100, description="Decisions to list") + connector_name: str = Field( + default="hyperliquid_perpetual", description="Connector the bots run on" + ) + + +def _fmt(value, digits=2, plus=False) -> str: + if value is None or isinstance(value, bool): + return "—" + try: + number = float(value) + except (TypeError, ValueError): + return str(value) + return f"{number:+,.{digits}f}" if plus else f"{number:,.{digits}f}" + + +def _clock(wall_time) -> str: + if not wall_time: + return "—" + return time.strftime("%H:%M:%S", time.localtime(float(wall_time))) + + +def _read_frame(path: Path) -> np.ndarray | None: + """The exact pixels the retina last received, as saved by the loop.""" + if not path.exists(): + return None + from PIL import Image + + return np.asarray(Image.open(path).convert("RGB"), dtype=np.uint8) + + +def _frame_figure(frame: np.ndarray, height: int = 340) -> go.Figure: + """That frame, full size, as the report's sensory panel.""" + fig = go.Figure(go.Image(z=frame)) + fig.update_layout( + height=height, + margin=dict(l=0, r=0, t=6, b=6), + paper_bgcolor=GROUND, + plot_bgcolor=GROUND, + xaxis=dict(visible=False), + yaxis=dict(visible=False), + legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), + ) + return fig + + +def _pnl_figure(events: list[dict]) -> go.Figure | None: + """Equity across the ticks whose P&L was actually reported.""" + points = [ + (e["tick"], float(e["equity"])) + for e in events + if e.get("pnl_known") and e.get("equity") is not None + ] + if len(points) < 2: + return None + ticks, equity = zip(*points) + fig = go.Figure( + go.Scatter( + x=list(ticks), + y=list(equity), + mode="lines", + line=dict(color=ACCENT, width=2), + fill="tozeroy", + fillcolor="rgba(201,242,77,0.10)", + name="net P&L", + ) + ) + fig.update_layout( + height=FULL_ROW - PANEL_CHROME, + margin=dict(l=48, r=16, t=10, b=36), + paper_bgcolor=GROUND, + plot_bgcolor=GROUND, + font=dict(color=BODY, family="monospace", size=11), + xaxis=dict(title="tick", gridcolor="#18202e", zerolinecolor="#18202e"), + yaxis=dict( + title="net P&L (quote)", gridcolor="#18202e", zerolinecolor="#243044" + ), + showlegend=False, + legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), + ) + fig.add_hline(y=0, line=dict(color=LIMB, width=1, dash="dot")) + return fig + + +async def _holdings( + client, connector_name: str, pairs: list[str] +) -> tuple[list[dict], int | None, int]: + """What the fly's bots hold now, how many positions they closed, and how + many they are still holding. + + The trade count is round trips actually completed — closes that ended a + position and realized its P&L — not quotes placed, and not the quotes the + controller cancelled in order to replace them. ``None`` when no bot + reported, so the report can say "unknown" rather than "zero". + """ + market = LiveMarket(client, connector_name, "5m", 72) + bots = await market.bots() + rows: list[dict] = [] + trades: int | None = None + held = 0 + for pair in pairs: + names = pair_names(pair) + running, bot = LiveMarket.find_bot(bots, names.bot_name) + if bot is None: + rows.append({"Market": pair, "Bot": "not running", "Side": "—"}) + continue + perf = (bot.get("performance") or {}).get(names.config_name) or {} + inner = perf.get("performance", perf) if isinstance(perf, dict) else {} + closes = inner.get("close_type_counts") or {} + if isinstance(closes, dict): + # Condor's own definition of a trade, not a second copy of it: the + # live snapshot and the cumulative history already count round + # trips this way, and a report that disagreed with them would be a + # third answer to the same question. + trades = (trades or 0) + count_trade_closes(inner) + held = held + sum( + int(count or 0) + for name, count in closes.items() + if str(name).split(".")[-1] == "POSITION_HOLD" + ) + positions = inner.get("positions_summary") or [] + amount = sum( + float(p.get("amount", 0) or 0) for p in positions if isinstance(p, dict) + ) + rows.append( + { + "Market": pair, + "Bot": running, + "Side": ("LONG" if amount > 0 else "SHORT" if amount < 0 else "FLAT"), + "Position": _fmt(abs(amount), 4), + "Realized": _fmt(inner.get("realized_pnl_quote"), 4, plus=True), + "Unrealized": _fmt(inner.get("unrealized_pnl_quote"), 4, plus=True), + "Volume": _fmt(inner.get("volume_traded")), + } + ) + return rows, trades, held + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + from config_manager import get_client + + root = agent_home(AGENT_SLUG) / "fly" / safe_id(config.run_name) + if not root.exists(): + raise FileNotFoundError( + f"No fly run at {root}; start fly_brain with run_name={config.run_name!r}" + ) + run_dir = RunDir(root) + state = run_dir.load_state() + latest = ( + json.loads(run_dir.latest_path.read_text()) + if run_dir.latest_path.exists() + else {} + ) + events = run_dir.recent_events(max(config.recent, 60)) + provenance = ( + json.loads(run_dir.provenance_path.read_text()) + if run_dir.provenance_path.exists() + else {} + ) + + settings = provenance.get("settings") or {} + guard = state.get("guard", {}) + postures = state.get("postures", {}) + pairs = list(postures) + # A halt, a closed book or a failed tick records no observation, so the + # neuron panel reads back the last tick that actually ran the brain rather + # than showing dashes over a run with thirty ticks of history behind it. + observed = next( + (e for e in reversed(events) if (e.get("neural") or {}).get("total_spikes")), + latest, + ) + neural = observed.get("neural", {}) + memory = neural.get("memory", {}) + neurons = (provenance.get("circuit") or {}).get("neurons") + last_posture = observed.get("posture") or latest.get("posture") or {} + execution = latest.get("execution") or {} + stale = observed is not latest + + client = await get_client(context._chat_id, context=context) + holdings, trades, held = ( + await _holdings(client, config.connector_name, pairs) + if client + else ([], None, 0) + ) + book_net = sum( + float(info.get("net", 0) or 0) + for info in (latest.get("bots") or {}).values() + if isinstance(info, dict) + ) + book_volume = latest.get("volume") + + halted = guard.get("halted") + alive = "HALTED" if halted else "RUNNING" + + builder = ReportBuilder(f"Fly — {config.run_name}") + builder.source("routine", "fly_report") + builder.tags(["fly", "dashboard", config.run_name]) + builder.manual_order() + + # ── FLY.EXE ────────────────────────────────────────────────────────────── + sensory = _read_frame(run_dir.frame_path) + # The prose belongs under the heading, not in a box below the figures it + # describes. `section` escapes its description, so this is plain text — + # no markdown syntax, which would show as literal asterisks. + builder.section( + "WHAT THE FLY SEES", + ( + f"The 320×180 frame fed to the retina on tick {observed.get('tick')} — " + f"{settings.get('n_candles', '?')} × " + f"{settings.get('candle_interval', '?')} candles, volume and the live " + "bid/ask — shown on its monitor and again at full size beside it. This " + "is the picture the posture below was decoded from. No quotes, " + "inventory or P&L are drawn, because those reach the fly only as " + "dopamine. Drag the scene to orbit it." + if sensory is not None + else "No input frame has been recorded for this run yet." + ), + ) + # Two halves of the grid, not one figure split internally: below the + # layout's 800px breakpoint these stack on their own. + builder.plotly( + fly_figure( + title=f"FLY.EXE — {alive}", + subtitle=f"{', '.join(pairs) or 'no market'}", + chart=sensory, + height=ROW_ONE - PANEL_CHROME, + ), + width=FLY, + ) + if sensory is not None: + builder.plotly( + _frame_figure(sensory, height=ROW_ONE - PANEL_CHROME), width=FRAME + ) + + # ── NEURONS & NEURAL ORDER ─────────────────────────────────────────────── + # Six cards, not fifteen. The rest of the numbers are narrative, and the + # picture says more about where the activity sat than a card ever could. + drawn, mapped, neurons_total = coverage() + active = neural.get("active_neurons") + posture_line = ( + f"Decoded from trend z {_fmt(last_posture.get('trend_z'), 2, plus=True)} and " + f"arousal z {_fmt(last_posture.get('arousal_z'), 2, plus=True)}, learned " + f"valence z {_fmt(last_posture.get('valence_z'), 2, plus=True)}, gated on " + f"{neural.get('gate_spikes', '—')} DNpe017 spike(s)." + + ("" if last_posture.get("warm", True) else " Baseline still forming.") + ) + # A run recorded before the loop knew its own observation length has only + # the brain's lifetime; printing "— ms" for the missing half reads worse + # than not claiming it. + ran = ( + f" The observation ran {_fmt(neural['observation_ms'], 0)} ms of neural time" + if neural.get("observation_ms") + else " The observation" + ) + if neural.get("brain_ms"): + ran += ( + f" ({_fmt(float(neural['brain_ms']) / 1000, 1)} s on this brain since it " + "was seeded)" + ) + observation_line = ( + f"{ran} produced {neural.get('total_spikes', 0):,} spikes, " + f"{neural.get('kc_spikes', 0):,} of them in Kenyon cells. Stimulus " + f"{observed.get('stimulus', 'none')} — {neural.get('reward_spikes', 0)} PAM11 " + f"and {neural.get('aversive_spikes', 0)} PPL101 spikes — left " + f"{(memory.get('changed_edges') or 0):,} of " + f"{(memory.get('plastic_edges') or 0):,} plastic edges away from baseline, " + f"mean efficacy {_fmt(memory.get('mean_efficacy'), 5)}." + ) + builder.section( + "THE FLY BRAIN", + posture_line + + observation_line + + " In the scene, brighter and larger is more spikes in the window and the " + "dim haze is every cell that stayed silent; hover one for its group.", + ) + # Cards on the left, the brain on the right, the same way the fly and its + # frame sit above — five columns and seven of the runtime's twelve, both + # collapsing to full width below its 800px breakpoint. + builder.kpi( + "Active neurons", + ( + f"{active:,} · {100 * active / neurons_total:.1f}%" + if active is not None + else "—" + ), + width=CARDS, + ) + builder.kpi( + "Mean firing rate", + f"{neural['mean_rate_hz']:.1f} Hz" if neural.get("mean_rate_hz") else "—", + width=CARDS, + ) + builder.kpi("Regime", str(last_posture.get("regime", "—")).upper(), width=CARDS) + # One card, not two: both multipliers are read off the same channel, and + # the stack's height is matched to the brain beside it at six cards. + builder.kpi( + "Spread / size ×", + f"{_fmt(last_posture.get('spread_mult'))} / " + f"{_fmt(last_posture.get('size_mult'))}", + width=CARDS, + ) + builder.kpi( + "Lean", + f"{_fmt(last_posture.get('shift_bps'), 2, plus=True)} bp", + width=CARDS, + ) + builder.kpi( + "Result", + RESULT_WORDS.get(execution.get("status"), execution.get("status", "—")), + width=CARDS, + ) + builder.plotly( + brain_figure( + run_dir.load_activity(observed.get("tick"), observed.get("pair", "")), + title="NEURAL ACTIVITY", + height=ROW_TWO - PANEL_CHROME, + ), + width=BRAIN, + ) + builder.plotly(readout_figure(neural, last_posture, height=FULL_ROW - PANEL_CHROME)) + # ── DECISIONS & POSITIONS ──────────────────────────────────────────────── + # One panel: what the fly called, and what those calls left it holding. + builder.section( + "DECISIONS & POSITIONS", + f"Last {min(config.recent, len(events))} observations, newest last — and " + "what the fly's own bots are holding right now", + ) + builder.table( + [ + { + "Time": _clock(e.get("wall_time")), + "Tick": e.get("tick"), + "Market": e.get("pair"), + "Neural proposal": ( + f"{str((e.get('posture') or {}).get('regime', '—')).upper()}" + f" ×{_fmt((e.get('posture') or {}).get('spread_mult'))}" + f" {_fmt((e.get('posture') or {}).get('shift_bps'), 2, plus=True)}bp" + ), + "R−L": f"{_fmt((e.get('neural') or {}).get('trend_hz'), 1, plus=True)} Hz", + "Stimulus": e.get("stimulus", "—"), + "Result": RESULT_WORDS.get( + (e.get("execution") or {}).get("status"), + (e.get("execution") or {}).get("status", "—"), + ), + "Reason": str((e.get("execution") or {}).get("reason", ""))[:54], + } + for e in events[-config.recent :] + ], + [ + "Time", + "Tick", + "Market", + "Neural proposal", + "R−L", + "Stimulus", + "Result", + "Reason", + ], + ) + if holdings: + builder.table( + holdings, + ["Market", "Bot", "Side", "Position", "Realized", "Unrealized", "Volume"], + ) + else: + builder.markdown( + "_No live bot data — either no server is bound to this chat, or the fly is " + "running in shadow with nothing deployed._" + ) + + # ── PERFORMANCE ────────────────────────────────────────────────────────── + builder.section( + "PERFORMANCE", + ( + (f"Halted: {halted}. " if halted else "") + + f"Session high {_fmt(guard.get('session_high_net'), 4, plus=True)}, " + f"{guard.get('ticks_since_high', 0)} observation(s) since, " + f"{guard.get('applies_today', 0)} config change(s) applied today. " + + ( + f"{held} position(s) opened and still held." + if held + else "No position is open." + ) + if guard.get("session_high_net") is not None + else "No P&L has been reported yet, so the breakers have nothing to judge." + ), + ) + builder.kpi("Ticks", f"{state.get('tick', 0):,}") + builder.kpi("Net P&L", _fmt(book_net, 4, plus=True)) + builder.kpi("Round trips", f"{trades:,}" if trades is not None else "—") + builder.kpi("Volume", _fmt(book_volume)) + pnl = _pnl_figure(events) + if pnl is not None: + builder.plotly(pnl) + await builder.save() + + lines = [ + f"run: {config.run_name}", + f"state: {alive}" + (f" ({halted})" if halted else ""), + f"tick: {state.get('tick', 0)}", + f"markets: {', '.join(pairs) or '—'}", + f"net_pnl: {_fmt(book_net, 4, plus=True)}", + f"volume: {_fmt(book_volume)}", + f"last_regime: {last_posture.get('regime', '—')}" + + (f" (observed tick {observed.get('tick')})" if stale else ""), + f"last_result: {execution.get('status', '—')}", + f"latest_spikes: {neural.get('total_spikes', '—')}", + f"changed_edges: {memory.get('changed_edges', '—')}", + ] + for row in holdings: + lines.append( + f"{row['Market']}: {row.get('Side', '—')} {row.get('Position', '')} " + f"realized {row.get('Realized', '—')} volume {row.get('Volume', '—')}" + ) + lines.append( + "caveat: the readout is engineered, the pulses are value feedback rather than " + "credit, and no profitable learning is demonstrated" + ) + return "\n".join(lines) diff --git a/agents/market_making_fly/routines/fly_setup.py b/agents/market_making_fly/routines/fly_setup.py new file mode 100644 index 000000000..565499477 --- /dev/null +++ b/agents/market_making_fly/routines/fly_setup.py @@ -0,0 +1,133 @@ +"""Prepare, verify or benchmark the fly's connectome from inside the agent. + +``prepare`` downloads the MaleCNS v1.0 release files (~1.1 GB), verifies their +checksums, compiles the retained graph into this agent's home and builds the +C++ kernel (needs ``c++``). ``verify`` re-checks the prepared arrays. +``bench`` loads the brain and times a few observations so ``fly_brain``'s +``neural_ms`` can be sized against its wall interval. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + +import asyncio +import logging +import time + +import numpy as np +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.reports import ReportBuilder + +logger = logging.getLogger(__name__) + +CATEGORY = "Analysis" + + +class Config(BaseModel): + """Prepare (download + compile), verify, or bench the fly connectome.""" + + action: str = Field(default="verify", description="prepare | verify | bench") + observations: int = Field( + default=3, ge=1, le=20, description="bench: observations to time (1–20)" + ) + neural_ms: float = Field( + default=500.0, + ge=200.0, + le=5000.0, + description="bench: neural ms per observation (200–5000)", + ) + + +def _prepare() -> dict: + from flybrain.data import prepare + from flybrain.neural.brain import build + + prepare() + return {"kernel": build()["model"]} + + +def _verify() -> dict: + from flybrain.data import verify + + return verify() + + +def _bench(observations: int, neural_ms: float) -> dict: + from flybrain.worker import FlyBrain + + started = time.perf_counter() + brain = FlyBrain(learning=True) + load = time.perf_counter() - started + frame = np.full((180, 320, 3), 235, np.uint8) + rows = [] + for _ in range(observations): + result = brain.observe(frame, "none", neural_ms=neural_ms) + rows.append( + { + "compute_seconds": round(result["compute_seconds"], 3), + "total_spikes": result["total_spikes"], + "kc_spikes": result["kc_spikes"], + "arousal_hz": round(result["arousal_hz"], 3), + } + ) + return {"load_seconds": round(load, 2), "observations": rows} + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + from flybrain.deps import require_pyarrow + from flybrain.neural.common import DATA + + # Every path from here reads a feather file. Check once, at the front, so + # the answer is a command to run rather than an ImportError from inside the + # vendored loader after a 1.1 GB download. + require_pyarrow() + if config.action not in ("prepare", "verify", "bench"): + raise ValueError("action must be prepare, verify or bench") + loop = asyncio.get_running_loop() + started = time.perf_counter() + if config.action == "prepare": + result = await loop.run_in_executor(None, _prepare) + elif config.action == "verify": + result = await loop.run_in_executor(None, _verify) + else: + result = await loop.run_in_executor( + None, _bench, config.observations, config.neural_ms + ) + elapsed = time.perf_counter() - started + + builder = ReportBuilder(f"Fly setup — {config.action}") + builder.source("routine", "fly_setup") + builder.tags(["fly", "setup"]) + builder.manual_order() + builder.section("01 / RESULT", f"{config.action} in {elapsed:.1f} s") + builder.kpi("Data dir", str(DATA)) + for key, value in result.items(): + if key != "observations": + builder.kpi(key, str(value)) + if config.action == "bench": + builder.table( + result["observations"], + ["compute_seconds", "total_spikes", "kc_spikes", "arousal_hz"], + ) + await builder.save() + + lines = [ + f"action: {config.action}", + f"data_dir: {DATA}", + f"elapsed_s: {elapsed:.1f}", + ] + lines += [f"{k}: {v}" for k, v in result.items() if k != "observations"] + if config.action == "bench": + lines += [ + f"obs{i}: compute={r['compute_seconds']}s kc={r['kc_spikes']} arousal={r['arousal_hz']}Hz" + for i, r in enumerate(result["observations"]) + ] + return "\n".join(lines) diff --git a/agents/market_making_fly/routines/fly_status.py b/agents/market_making_fly/routines/fly_status.py new file mode 100644 index 000000000..8e3aba49a --- /dev/null +++ b/agents/market_making_fly/routines/fly_status.py @@ -0,0 +1,157 @@ +"""Latest state of a fly run: posture per pair, last observation, guard, memory.""" + +from __future__ import annotations + +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + +import json +import logging + +from flybrain.run_state import RunDir +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.memory.paths import agent_home +from condor.paths import safe_id +from condor.reports import ReportBuilder + +logger = logging.getLogger(__name__) + +CATEGORY = "Bot Analysis" +AGENT_SLUG = "market_making_fly" + + +class Config(BaseModel): + """Read a fly run's state.json / latest.json / recent events without touching the brain.""" + + run_name: str = Field( + default="fly", description="Run directory under the agent home" + ) + recent: int = Field(default=20, description="Recent ticks to list") + + +def _fmt(value, digits=3) -> str: + if isinstance(value, (int, float)): + return f"{value:+.{digits}f}" if isinstance(value, float) else str(value) + return "—" if value is None else str(value) + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + root = agent_home(AGENT_SLUG) / "fly" / safe_id(config.run_name) + if not root.exists(): + raise FileNotFoundError( + f"No fly run at {root}; start fly_brain with run_name={config.run_name!r}" + ) + run_dir = RunDir(root) + state = run_dir.load_state() + latest = ( + json.loads(run_dir.latest_path.read_text()) + if run_dir.latest_path.exists() + else {} + ) + events = run_dir.recent_events(config.recent) + guard = state.get("guard", {}) + postures = state.get("postures", {}) + baselines = state.get("baselines", {}) + applied = state.get("applied", {}) + neural = latest.get("neural", {}) + memory = neural.get("memory", {}) + + lines = [ + f"run: {config.run_name}", + f"tick: {state.get('tick', 0)}", + f"halted: {guard.get('halted') or 'no'}", + f"anchor_equity: {_fmt(state.get('anchor'))}", + f"session_high_net: {_fmt(guard.get('session_high_net'))}", + f"ticks_since_high: {guard.get('ticks_since_high', 0)}", + f"applies_today: {guard.get('applies_today', 0)}", + f"last_pair: {latest.get('pair', '—')}", + f"last_stimulus: {latest.get('stimulus', '—')}", + f"last_pnl_delta: {_fmt(latest.get('pnl_delta'))}", + f"last_execution: {(latest.get('execution') or {}).get('status', '—')} " + f"({(latest.get('execution') or {}).get('reason', '')})", + f"kc_spikes: {neural.get('kc_spikes', '—')}", + f"gate_spikes: {neural.get('gate_spikes', '—')}", + f"trend_hz: {_fmt(neural.get('trend_hz'))}", + f"arousal_hz: {_fmt(neural.get('arousal_hz'))}", + f"changed_edges: {memory.get('changed_edges', '—')}", + f"mean_efficacy: {_fmt(memory.get('mean_efficacy'), 4)}", + ] + for pair, posture in postures.items(): + if posture: + lines.append( + f"{pair}: {posture['regime']} spread_x={posture['spread_mult']:.2f} " + f"size_x={posture.get('size_mult', 1.0):.2f} " + f"valence_z={posture.get('valence_z', 0.0):+.2f} " + f"lean_bp={posture['shift_bps']:+.2f} trend_z={posture['trend_z']:+.2f} " + f"arousal_z={posture['arousal_z']:+.2f} baseline_n={len((baselines.get(pair) or {}).get('trend', []))}" + ) + else: + lines.append( + f"{pair}: no posture yet (baseline_n={len((baselines.get(pair) or {}).get('trend', []))})" + ) + lines.append( + "caveat: regime/spread/lean are an engineered readout of spike counts; dopamine " + "pulses report P&L change, not credit for the last posture; no learning is validated" + ) + + builder = ReportBuilder(f"Fly status — {config.run_name}") + builder.source("routine", "fly_status") + builder.tags(["fly", "status"]) + builder.manual_order() + builder.section("01 / RUN", str(root)) + builder.kpi("Tick", str(state.get("tick", 0))) + builder.kpi("Halted", guard.get("halted") or "no") + builder.kpi("Anchor equity", _fmt(state.get("anchor"))) + builder.kpi("Session high", _fmt(guard.get("session_high_net"))) + builder.kpi("Changed edges", str(memory.get("changed_edges", "—"))) + builder.section("02 / POSTURES", "Per pair") + builder.table( + [ + { + "Pair": pair, + "Regime": (p or {}).get("regime", "—"), + "Spread ×": (p or {}).get("spread_mult", "—"), + "Size ×": (p or {}).get("size_mult", "—"), + "Lean bp": (p or {}).get("shift_bps", "—"), + "Trend z": (p or {}).get("trend_z", "—"), + "Arousal z": (p or {}).get("arousal_z", "—"), + "Applied buy": (applied.get(pair) or {}).get("buy_spreads", "—"), + "Applied sell": (applied.get(pair) or {}).get("sell_spreads", "—"), + } + for pair, p in postures.items() + ], + [ + "Pair", + "Regime", + "Spread ×", + "Lean bp", + "Trend z", + "Arousal z", + "Applied buy", + "Applied sell", + ], + ) + builder.section("03 / RECENT TICKS", "Newest last") + builder.table( + [ + { + "Tick": e.get("tick"), + "Pair": e.get("pair"), + "Stim": e.get("stimulus", "—"), + "Δ": _fmt(e.get("pnl_delta")), + "Regime": (e.get("posture") or {}).get("regime", "—"), + "Exec": (e.get("execution") or {}).get("status", "—"), + "Reason": str((e.get("execution") or {}).get("reason", ""))[:60], + } + for e in events + ], + ["Tick", "Pair", "Stim", "Δ", "Regime", "Exec", "Reason"], + ) + await builder.save() + return "\n".join(lines) diff --git a/agents/market_making_fly/routines/mm_market_scanner.py b/agents/market_making_fly/routines/mm_market_scanner.py new file mode 100644 index 000000000..18cb4dc30 --- /dev/null +++ b/agents/market_making_fly/routines/mm_market_scanner.py @@ -0,0 +1,474 @@ +"""Rank the markets of any CLOB venue for market making, spot or perp. + +The question a market maker asks before quoting is not "is this liquid" but +"is the spread wide enough to clear what a round trip costs here, and is there +depth to quote into". Nothing else in the library answers it: the global +``market_scanner`` profiles volume and volatility with no spread, depth or fee +term at all, ``market_analyzer`` reads one pair's regime, and ``arb_check`` +compares one pair across venues. + +This is the HIP-3 scanner's ranking generalized. Three things change: + +* **Where the numbers come from.** That routine reads one Hyperliquid call + which returns every market in an issuer's dex. Condor's own ticker fetcher + turns out to enumerate the same markets — 285 issuer-prefixed pairs come + back for ``hyperliquid_perpetual`` alongside its native ones — so one code + path now covers HIP-3 and every other venue. Book depth comes from + ``LiveMarket.levels``, which already knows which source serves which market. +* **What "wide enough" means.** That routine hardcodes 3 bp, which is unrelated + to what trading costs. Here the threshold is a multiple of the venue's own + round-trip maker fee, and that fee differs by a factor of five between a + HIP-3 perp and a spot book. + +* **What "worth quoting" means.** Both older scanners asked whether the touch + is already wider than a round trip. That is the test for someone joining the + touch. The fly does not join it — it rests a quote away from mid and waits — + so the question is whether the market *comes to it*: does a typical candle + travel the round trip the fly must make, from mid down to its quote and back + out through the take-profit? On Hyperliquid the touch test rejected all 120 + HIP-3 markets including the one the fly was quoting profitably by hand. + +One honest limit. Reading a book costs a call, so only the top markets by +volume get read — and the busiest markets are not the most reachable. So +``prescreen`` is the knob that matters when nothing survives, and the report +says how deep it looked, and ranks the best of what it measured whether or not +anything cleared. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + +import asyncio +import logging +import math +import statistics + +from flybrain import venue +from flybrain.market import LiveMarket, depth_within +from flybrain.naming import pair_names +from flybrain.posture import base_levels_from_range, take_profit_floor_bps +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.fetchers.market_data import fetch_tickers +from condor.reports import ReportBuilder + +logger = logging.getLogger(__name__) + +CATEGORY = "Market Data" + +# Reach saturates: past three cycles in a median candle the market is moving +# faster than a maker can requote, and the extra movement is adverse selection +# rather than income. +REACH_SCORE_CAP = 3.0 + +# A day of candles at the fly's own interval: enough for a stable median range +# and for the 24 h drift, from one call per market. +CANDLES_PER_DAY = 288 + + +class Config(BaseModel): + """Rank a venue's markets for market making: spread vs fee, depth, drift.""" + + connector_name: str = Field( + default="hyperliquid_perpetual", description="Any CLOB connector, spot or perp" + ) + pairs: str = Field( + default="", + description="Comma-separated pairs to rank; blank ranks everything the venue lists", + ) + quote: str = Field( + default="", + description="Only pairs quoted in this asset, e.g. USDT (blank = any)", + ) + issuer: str = Field( + default="", + description="Only pairs with this issuer prefix, e.g. xyz for HIP-3 (blank = any)", + ) + # Every threshold below is bounded. A negative fee or a negative multiple + # does not loosen a filter, it inverts it: the floor goes below zero and + # every market that is not crossed "clears the fee", which is the one + # mistake this routine exists to prevent. + maker_fee_bps: float = Field( + default=0.0, + ge=0.0, + description="Maker fee per side in bp; 0 uses the venue default", + ) + min_range_over_cycle: float = Field( + default=1.0, + gt=0.0, + description="Require a typical candle's range to be this multiple of the " + "round trip the fly would have to travel (quote distance + take-profit). " + "1.0 means a median candle completes one cycle", + ) + candle_interval: str = Field( + default="5m", + description="Candle the range is measured on; use the fly's own interval", + ) + min_volume_usd: float = Field( + default=250_000.0, ge=0.0, description="Minimum 24h volume" + ) + max_daily_drift_pct: float = Field( + default=3.0, ge=0.0, description="Maximum 24h price drift %" + ) + min_book_depth_usd: float = Field( + default=10_000.0, + ge=0.0, + description="Minimum resting notional per side, within the band", + ) + depth_within_bps: float = Field( + default=10.0, gt=0.0, description="Band around mid for depth" + ) + prescreen: int = Field( + default=30, + ge=1, + le=120, + description="How many of the highest-volume markets to read the book of. " + "Raise it when nothing survives: the widest markets are rarely the " + "busiest, so a low prescreen sees only tight books", + ) + top_n: int = Field(default=5, ge=1, le=25, description="Markets to report") + + +def cycle_bps(range_bps: float, fee_bps: float) -> float: + """The round trip the fly must travel on this market, in bp. + + From mid down to where it would rest level 1 — half a typical bar's range, + never inside the fee — and back out through the take-profit floor. This is + the distance a market has to move for one completed pair. + """ + entry = max(base_levels_from_range(range_bps)[0], fee_bps) + return entry + take_profit_floor_bps(fee_bps) + + +async def _measure(market: LiveMarket, pair: str, config: Config, sem) -> dict: + """Spread, depth and drift for one market. Never raises: a market whose + book or candles cannot be read is reported as unreadable, not dropped + silently and not allowed to kill the scan.""" + row: dict = {"pair": pair} + async with sem: + try: + bids, asks = await market.levels(pair, depth=50) + bid_usd, ask_usd, spread_bps = depth_within( + bids, asks, config.depth_within_bps + ) + row.update( + { + "spread_bps": spread_bps, + "bid_depth_usd": bid_usd, + "ask_depth_usd": ask_usd, + "open": bool(bids and asks), + } + ) + except Exception as failure: # external feed, one market + row["error"] = repr(failure)[:80] + return row + # One day of the fly's own candle gives both numbers: the drift across + # it, and how far a typical one of them travels. The first ask for a + # market hummingbot-api has not seen subscribes a candle feed and + # returns 504 if it is not ready within 30 s. That is a cold feed, not + # a missing market: the same call answers on the second attempt. + for attempt in (1, 2): + try: + raw = await market.client.market_data.get_candles( + config.connector_name, + pair, + interval=config.candle_interval, + max_records=CANDLES_PER_DAY, + ) + rows = ( + raw + if isinstance(raw, list) + else raw.get("data", raw.get("candles")) + ) + bars = [ + (float(c["high"]), float(c["low"]), float(c["close"])) + for c in (rows or []) + if c.get("close") and float(c["close"]) > 0 + ] + if len(bars) < 2: + raise ValueError(f"only {len(bars)} usable candles") + row["drift_pct"] = abs(bars[-1][2] / bars[0][2] - 1) * 100 + # Median, not mean: one news bar should not make a quiet market + # look reachable. + row["range_bps"] = statistics.median( + (high - low) / close * 1e4 for high, low, close in bars + ) + row["candles"] = len(bars) + break + except Exception as failure: # external feed, one market + row["drift_pct"] = None + row["range_bps"] = None + row["candle_error"] = ( + str(getattr(failure, "message", "") or failure)[:70] + or repr(failure)[:70] + ) + return row + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + from config_manager import get_client + + client = await get_client(context._chat_id, context=context) + if not client: + return "No server available" + + market_type = venue.market_type_for(config.connector_name) + + # ── 1. Enumerate and screen on volume, which is one call ────────────────── + if config.pairs.strip(): + candidates = {p.strip(): None for p in config.pairs.split(",") if p.strip()} + tickers = (await fetch_tickers(client, config.connector_name)).get( + "tickers" + ) or {} + volumes = { + p: float((tickers.get(p) or {}).get("usd_volume", 0) or 0) + for p in candidates + } + else: + tickers = (await fetch_tickers(client, config.connector_name)).get( + "tickers" + ) or {} + if not tickers: + return ( + f"No tickers for {config.connector_name}; is the connector supported?" + ) + volumes = {} + for pair, row in tickers.items(): + try: + names = pair_names(pair) + except ValueError: + continue # not a shape this agent can quote + if config.quote and names.quote != config.quote.upper(): + continue + if config.issuer and names.issuer != config.issuer.lower(): + continue + volumes[pair] = float((row or {}).get("usd_volume", 0) or 0) + + listed = len(volumes) + screened = sorted( + ((p, v) for p, v in volumes.items() if v >= config.min_volume_usd), + key=lambda kv: -kv[1], + )[: config.prescreen] + if not screened: + return ( + f"{config.connector_name}: {listed} markets listed, none above " + f"${config.min_volume_usd:,.0f} of 24h volume" + ) + + # ── 2. Book and candles, only for what survived — the expensive part ───── + market = LiveMarket( + client, config.connector_name, config.candle_interval, CANDLES_PER_DAY + ) + sem = asyncio.Semaphore(6) + measured = await asyncio.gather( + *(_measure(market, pair, config, sem) for pair, _ in screened) + ) + by_pair = {m["pair"]: m for m in measured} + + # One connector can serve two fee families — a Hyperliquid core perp costs + # twice what one of its HIP-3 markets does — so each market is ranked + # against its own round trip, not the venue's average. + fees = { + pair: config.maker_fee_bps + or await venue.maker_fee_bps(config.connector_name, market_type, pair) + for pair, _ in screened + } + + rows = [] + for pair, volume in screened: + m = by_pair[pair] + fee_bps = fees[pair] + drift = m.get("drift_pct") + depth = min(m.get("bid_depth_usd", 0.0), m.get("ask_depth_usd", 0.0)) + spread = m.get("spread_bps", 0.0) + # What the fly would actually do here: rest level 1 at max(2, S/2) bp, + # never inside the fee, and close at the take-profit floor. The round + # trip it must travel is the sum — down to the quote, then back up + # through the exit. + entry_bps = max(base_levels_from_range(range_bps or 0.0)[0], fee_bps) + exit_bps = take_profit_floor_bps(fee_bps) + cycle = cycle_bps(range_bps or 0.0, fee_bps) + range_bps = m.get("range_bps") + # How far a typical candle travels against that cycle. Above 1 the + # median candle completes one; below it, the market does not come to + # the fly often enough to matter, however wide its touch looks. + reach = (range_bps / cycle) if range_bps else 0.0 + reasons = [] + if m.get("error"): + reasons.append("book unreadable") + elif not m.get("open"): + reasons.append("book closed") + if range_bps is None: + reasons.append(f"candles unreadable: {m.get('candle_error', 'none')}") + elif reach < config.min_range_over_cycle: + reasons.append( + f"range {range_bps:.1f} < {config.min_range_over_cycle:g}× the " + f"{cycle:.1f} bp cycle" + ) + if depth < config.min_book_depth_usd: + reasons.append(f"depth ${depth:,.0f}") + if drift is not None and drift > config.max_daily_drift_pct: + reasons.append(f"drift {drift:.1f}%") + rows.append( + { + "pair": pair, + "volume": volume, + "fee_bps": fee_bps, + "entry_bps": entry_bps, + "exit_bps": exit_bps, + "cycle_bps": cycle, + "range_bps": range_bps, + "reach": reach, + "spread_bps": spread, + "depth_usd": depth, + "drift_pct": drift, + "survives": not reasons, + "why_not": ", ".join(reasons), + # Reach is what earns: a market that traverses the fly's cycle + # twice as often is worth twice as much to it, up to the point + # where the movement is adverse selection rather than noise. + "score": ( + math.log(max(volume, 1)) + + 2.0 * min(reach, REACH_SCORE_CAP) + - 0.4 * (drift if drift is not None else 99.0) + ), + } + ) + survivors = sorted((r for r in rows if r["survives"]), key=lambda r: -r["score"])[ + : config.top_n + ] + # Rank the best measured markets whether or not they clear, so a scan that + # finds no survivor still says which markets came closest and by how much. + ranked = sorted(rows, key=lambda r: -r["score"])[: config.top_n] + rejected = sorted( + (r for r in rows if not r["survives"]), key=lambda r: -r["volume"] + ) + + builder = ReportBuilder(f"MM markets — {config.connector_name}") + builder.source("routine", "mm_market_scanner") + builder.tags(["market-making", "scanner", config.connector_name]) + builder.manual_order() + cheapest, dearest = min(fees.values()), max(fees.values()) + fee_text = ( + f"{cheapest:.2f} bp" + if cheapest == dearest + else f"{cheapest:.2f}–{dearest:.2f} bp" + ) + cycles = [r["cycle_bps"] for r in rows] + builder.section( + "WHETHER THE MARKET COMES TO THE FLY", + f"{market_type} venue · maker {fee_text} a side. The fly does not join " + "the touch: it rests a quote and waits. So the test is not whether the " + "spread is already wide, but whether a typical " + f"{config.candle_interval} candle travels the round trip it would have " + f"to — down to its quote and back out through the take-profit, " + f"{min(cycles):.1f}–{max(cycles):.1f} bp here — at least " + f"{config.min_range_over_cycle:g}× over.", + ) + builder.kpi("Venue", config.connector_name) + builder.kpi("Type", market_type) + builder.kpi("Maker fee", fee_text) + builder.kpi("Cycle to travel", f"{min(cycles):.1f}–{max(cycles):.1f} bp") + builder.kpi("Listed", f"{listed:,}") + builder.kpi("Book-checked", str(len(screened))) + builder.kpi("Survivors", str(len([r for r in rows if r["survives"]]))) + + builder.section( + "RANKED", + f"Best {len(ranked)} of the {len(screened)} measured, clearing or not — " + "reach is the median candle's range over the cycle the fly must travel", + ) + builder.table( + [ + { + "Pair": r["pair"], + "Clears": "yes" if r["survives"] else "no", + "24h volume": f"${r['volume']:,.0f}", + "Median range": (f"{r['range_bps']:.2f} bp" if r["range_bps"] else "—"), + "Cycle": f"{r['cycle_bps']:.2f} bp", + "Reach": f"{r['reach']:.2f}×", + "Spread": f"{r['spread_bps']:.2f} bp", + "Depth/side": f"${r['depth_usd']:,.0f}", + "Drift": ( + f"{r['drift_pct']:.2f}%" if r["drift_pct"] is not None else "—" + ), + } + for r in ranked + ] + or [{"Pair": "— nothing measured —"}], + [ + "Pair", + "Clears", + "24h volume", + "Median range", + "Cycle", + "Reach", + "Spread", + "Depth/side", + "Drift", + ], + ) + builder.section("REJECTED", "Why each screened market did not make it") + builder.table( + [ + { + "Pair": r["pair"], + "24h volume": f"${r['volume']:,.0f}", + "Spread": f"{r['spread_bps']:.2f} bp", + "Depth/side": f"${r['depth_usd']:,.0f}", + "Reason": r["why_not"], + } + for r in rejected + ] + or [{"Pair": "— none rejected —"}], + ["Pair", "24h volume", "Spread", "Depth/side", "Reason"], + ) + builder.markdown( + f"_Books were read for the {len(screened)} highest-volume markets of " + f"{listed} listed. The widest markets are rarely the busiest, so raise " + "`prescreen` when nothing survives — it is the only thing standing " + "between this scan and the rest of the venue._" + ) + builder.markdown( + "_Reach says the market reaches the fly's quote, not that the fills are " + "good ones: the same movement that fills a maker is what runs him over, " + "and a median range hides the bar that gaps through both levels. Depth " + "is filtered separately because a wide touch is usually a thin one. " + "Drift is a proxy for the inventory a maker accumulates against a trend, " + "not a forecast. Nothing here says a market is profitable._" + ) + await builder.save() + + lines = [ + f"venue: {config.connector_name} ({market_type})", + f"maker_fee_bps: {fee_text} a side", + f"listed: {listed}, book_checked: {len(screened)}, survivors: {len(survivors)}", + ] + for n, r in enumerate(ranked, 1): + lines.append( + f"{n}. {r['pair']}: reach {r['reach']:.2f}× (median " + + (f"{r['range_bps']:.2f}" if r["range_bps"] else "—") + + f" bp range vs a {r['cycle_bps']:.2f} bp cycle), spread " + f"{r['spread_bps']:.2f} bp, fee {r['fee_bps']:.2f} bp, depth " + f"${r['depth_usd']:,.0f}/side, vol ${r['volume']:,.0f}" + + ("" if r["survives"] else f" — REJECTED: {r['why_not']}") + ) + if survivors: + best = survivors[0] + lines.append( + f"TOP PICK: {best['pair']} at reach {best['reach']:.2f}×, " + f"picked_ranges_bps={best['range_bps']:.2f}" + ) + else: + lines.append( + "TOP PICK: none — no market travels the fly's own round trip often " + "enough, with depth behind it" + ) + return "\n".join(lines) diff --git a/agents/market_making_fly/skills/fly_decoder/SKILL.md b/agents/market_making_fly/skills/fly_decoder/SKILL.md new file mode 100644 index 000000000..d00761195 --- /dev/null +++ b/agents/market_making_fly/skills/fly_decoder/SKILL.md @@ -0,0 +1,85 @@ +--- +name: fly_decoder +description: How spike counts become a quoting posture, how to read fly_status, and what + each number can and cannot tell you. +when_to_use: When explaining a posture, answering "why did the fly widen/lean/pause", + "is it learning", or checking whether a run is healthy. +created: '2026-09-12T00:00:00Z' +source: agent:market_making_fly +references_routine: fly_status +--- + +# Reading the fly + +Run `manage_routines(action="run", name="fly_status", config={"run_name": ""})` +for the numbers as text, or `fly_report` for the dashboard — the same data plus the +fly itself, the decision log and the last frame it saw. Use `fly_report` when the +user wants to *look* at the run, `fly_status` when you need to quote figures. + +## Channels (per observation, 500 ms of neural time) + +| Field | Cells | Meaning | +|---|---|---| +| `trend_hz` | DNp20 right mean rate − left mean rate | lean direction; stonkfly's BUY/SELL cells | +| `arousal_hz` | mean rate of the 1,314 descending neurons (minus the readouts) | spread width **and** how much of the book is quoted | +| `gate_spikes` | DNpe017 | ≥ 1 required for a trending call and for any lean | +| `valence_hz` | mean MBON07 rate − mean MBON11 rate | approach minus avoidance — **the memory rule's own output**, and the only channel a P&L pulse can reach. Drives how much of the book is quoted | +| `kc_spikes` | Kenyon cells | the confidence test: no sparse code of the chart means every other channel is reading noise, so the posture is marked unconfident and the loop holds | +| `reward_spikes` / `aversive_spikes` | PAM11 / PPL101 | did the pulse arrive | + +## Posture + +* `trend_z`, `arousal_z`: the channel minus its rolling per-pair mean, over its std, + window 60 observations. `warm=False` for the first 10 — neutral posture. +* Regime precedence: `pause` (arousal_z ≥ 2.5) > `volatile` (≥ 1) > `trending_up/down` + (gate and |trend_z| ≥ 1) > `quiet` (arousal_z ≤ −1) > `ranging`. +* **Side**: a gated trend past |trend_z| ≥ 1.5 takes the other side off the book — + `buy` on a positive trend, `sell` on a negative one — by quoting it at zero size, + which pmm_mister skips. The surviving side's orders double, because the controller + normalizes amounts across both. +* `spread ×` = clip(1 − 0.5·arousal_z, 0.6, 2.5) — an aroused fly quotes **tighter**. + `size ×` = clip(1 + 0.5·arousal_z, 0.6, 2.5) — and quotes **more** of the book, + clamped so an order never falls under the venue minimum nor the allocation over 1. + `size ×` also carries `+0.5·valence_z`, so what the fly has learned about scenes + like this one moves the capital it commits. The exit is 0.75 × the market's median + candle range, floored by the round-trip fee and by the first level; the fly does + not move it — letting arousal widen it halved the round trips and earned nothing. + The sign on each is a choice, not a finding: arousal is a population rate against + its own average and nothing ties it to volatility. `lean` = clip(trend_z, ±3 bp), 0 + without a gate spike, and capped at half the first spread level when mapped. +* Mapping: level 1 = max(2, R/2) bp, level 2 = level 1 + R/4 (R = the market's + median candle range), times + `spread ×`, buy −lean / sell +lean, floor 3 bp; TP = max(4 bp, 2.2 × round-trip fee, + first level); timing per regime; `pause` sets `manual_kill_switch`. + +## Execution statuses in the event log + +`SHADOW` would have applied (shadow mode) · `APPLIED` live update done · `HOLD` no +material change or per-pair 5-minute cooldown · `VETO` guard refused (reason given) · +`CLOSED` book closed, fly did not observe · `STOP_BOT` closed ≥ 5 ticks · `ERROR` +update failed · `HALT` loop stopped · `TICK_ERROR` data fetch failed, loop continues. + +## What you may say + +* "The fly's arousal channel is 1.8 σ above its baseline on DRAM, so it tightened to + 0.6× and quoted 1.9× the usual share of the book." +* "No gate spike this observation, so no lean regardless of trend." +* "Changed edges rose from 0 to 312 after the first aversive pulse." + +## What you may not say + +* That the fly detected a regime: it emitted a z-score we labelled. +* That a P&L pulse taught it anything: pulses are value feedback, not credit assignment, + and edges change from endogenous dopamine activity too. The valence channel means the + memory rule can now *reach* a decision — it does not mean what it reaches is right. +* That a run with positive P&L shows the fly works: a rising market makes any long + inventory look skilled. No held-out replay or shuffled-reinforcement control exists. + +## Health checks + +* `kc_spikes` stays 0 → the chart is not activating the mushroom body. The loop now + holds instead of applying (`HOLD` with "scene not seen"); a run that does this + every tick is quoting a stale config, so report it. +* `baseline_n` not growing for a pair → that pair's book is closed or its feed fails. +* `halted` set → read the reason; financial halts need a new `run_name`. +* `compute_seconds` above ~40 % of the interval → lower `neural_ms`. diff --git a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md new file mode 100644 index 000000000..29e640330 --- /dev/null +++ b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md @@ -0,0 +1,159 @@ +--- +name: fly_mm_deploy +description: End-to-end deployment of the fly market maker on up to three markets on + any CLOB venue, spot or perp — pick the markets, deploy neutral pmm_mister bots with + the fly's naming, start fly_brain in shadow, verify, and (only when told) go live. +when_to_use: When asked to set up, deploy, launch, or restart the fly market maker, or + to rotate one of its market slots. Follow it as a delegate task with no mid-flow + confirmation. +created: '2026-09-12T00:00:00Z' +source: agent:market_making_fly +--- + +# Fly MM Deploy + +You are deploying **Market Making Fly**: one shared fly brain, up to three markets +on any CLOB venue, `pmm_mister` controllers. The fly decides posture; you set up +the plumbing. + +## Step 0 — Settle the venue + +- `connector_name`: any CLOB connector hummingbot-api serves. A `_perpetual` + suffix means perp, anything else is spot. +- **Spot**: `leverage` must be 1, and the fee floor is far wider — check + `maker_fee_bps` before anything else. +- **`maker_fee_bps`**: pass the exchange's real per-side maker fee when you know + it. Leaving it at 0 uses a conservative default, which quotes wider than + necessary rather than tighter than is profitable. + +## Step 1 — Pick the markets + +``` +manage_routines(action="run", name="mm_market_scanner", config={ + "connector_name": "", "issuer": "xyz", # issuer only on HIP-3 + "quote": "USDT", # optional, e.g. spot venues + "prescreen": 30, "top_n": 5}) +``` + +It ranks by volume, by whether a typical candle travels the round trip the fly +must make — down to its quote, back out through the take-profit — and by book +depth, then reports why every rejected market failed. +Take the top **`n_markets`** (1-3; from `[CURRENT CONFIG]` or the task, default +3) with an open book — one brain quotes them all in round-robin. Record each +`pair` and its **median candle range in bp**; that is `picked_ranges_bps`. The +quote levels are built from how far the market travels, not from how wide its +touch is: level 1 sits at half a typical bar, where the spread multiplier can +actually change whether a bar reaches it. + +**If nothing survives, raise `prescreen` before anything else.** Reading a book +costs a call, so only the busiest markets are read — and the busiest markets are +not the most reachable. On Hyperliquid the heaviest HIP-3 markets barely move +2 bp in five minutes against a 7.7 bp cycle; the first market that cleared it +sat fourth by volume. A `TOP PICK: none` line means the scan did not look far +enough, or this venue genuinely does not come to a resting quote. + +If it still finds nothing, stop and report that rather than lowering the +floor: quoting inside the fee loses money on every fill. + +## Step 2 — Collateral + +`get_portfolio_overview(["hyperliquid_perpetual"])` → available USD. Required ≈ +`Σ total_amount_quote × 0.5 / leverage` across the pairs. If short, reduce +`total_amount_quote` or drop a pair; say so in the report. + +## Step 3 — Neutral configs (the fly's starting point) + +For each pair derive the slug from the **whole** pair — `SOL-USDT` → `sol-usdt`, +`XYZ:ORCL-USD` → `xyz-orcl-usd` — then `bot_name = {slug}-fly` and +`config_name = {slug}_fly_mm` (underscores). Never name a bot after the base token +alone: `BTC-USDT` and `BTC-USDC` would collide. The neutral config is exactly what +`fly_brain` would apply for the `ranging` regime; build it with: + +```python +run_code(code=""" +import sys; sys.path.insert(0, "agents/market_making_fly") +from flybrain.posture import MarketSpec, build_config +from flybrain.decoder import NEUTRAL +spec = MarketSpec(connector_name="binance_perpetual", trading_pair="SOL-USDT", + total_amount_quote=500, range_bps=10.0, leverage=3, + portfolio_allocation=0.2) # spot: leverage=1, and check maker_fee_bps +print(build_config(spec, NEUTRAL)) +""") +``` + +`build_config` refuses a spec whose orders would fall under the exchange minimum +(10 USD on HIP-3, 5-10 USD on most spot venues): each order is `total_amount_quote × portfolio_allocation / 4`. +200 quote on one market needs `portfolio_allocation` ≥ 0.2; 100 quote needs ≥ 0.4. +Whatever value you use here, pass the **same** `portfolio_allocation` to `fly_brain` +in Step 5 — it rebuilds every config from it. + +Then save it: + +``` +manage_controllers(action="upsert", target="config", config_name="dram_fly_mm", + config_data={...printed config...}, confirm_override=True) +``` + +Do not edit the spreads, TP, bands or stop loss by hand — the floors live in code. + +## Step 4 — Deploy the bots + +One bot per pair, named exactly `{token}-fly`, with a loss cap: + +``` +manage_bots(action="deploy", bot_name="dram-fly", controllers_config=["dram_fly_mm"], + max_global_drawdown_quote=<0.04 × total_amount_quote>) +``` + +Confirm with `manage_bots(action="status")` that each bot is running with its +controller. + +## Step 5 — Start the fly in shadow + +`pairs` and `picked_ranges_bps` list exactly the `n_markets` picks, same order. +With `n_markets: 1` that is a single pair and a single range. + +``` +manage_routines(action="start", name="fly_brain", config={ + "pairs": "XYZ:DRAM-USD,XYZ:SPCX-USD,XYZ:SMSN-USD", # n_markets entries + "picked_ranges_bps": "10,8,14", # one per pair + "total_amount_quote": 500, "leverage": 3, "portfolio_allocation": 0.2, + "mode": "shadow", "run_name": "fly-2026-09-12"}) +``` + +Shadow observes, decodes and reinforces from the bots' P&L but applies nothing. Note +the instance id. After ~10 observations per pair (30 ticks) `fly_status` shows +non-neutral postures. + +## Step 6 — Verify + +``` +manage_routines(action="run", name="fly_status", config={"run_name": "fly-2026-09-12"}) +manage_routines(action="run", name="fly_report", config={"run_name": "fly-2026-09-12"}) +``` + +Report: pairs, ranges, bots running, fly tick count, first postures, any vetoes or +halts, and the caveat that the fly's learning is not validated. + +## Step 7 — Live (only when the task says so) + +Stop the shadow instance (`manage_routines(action="stop", name="")`) and +start again with `"mode": "live"` and the **same** `run_name` — the baseline and the +brain's memory carry over. Never start live on a fresh `run_name` without a shadow +period first. + +## Rotation + +When a slot's market is closed, dominated, or the operator asks: stop that bot +(`manage_bots(action="stop_bot", bot_name=...)`), re-run the scanner, deploy the new +pick with Steps 3–4, stop `fly_brain` and start it again with the updated `pairs` and +`picked_ranges_bps` and the same `run_name`. The brain keeps its memory; only the +swapped pair's baseline starts over. + +## Halts + +`fly_brain` stops itself and (in live) stops the bots on a halt. A **transient** halt +(repeated config-update failures) restarts with `"resume_reviewed": true` after you +have looked at the bot logs. A **financial** halt (loss stop, loss-rate breaker, no +new P&L high) cannot be cleared: report it, leave the bots stopped, and only redeploy +under a new `run_name` if the operator asks. diff --git a/agents/market_making_fly/strategies/mm_operator/strategy.md b/agents/market_making_fly/strategies/mm_operator/strategy.md new file mode 100644 index 000000000..989e6dbb1 --- /dev/null +++ b/agents/market_making_fly/strategies/mm_operator/strategy.md @@ -0,0 +1,71 @@ +--- +name: Fly Operator +description: Keeps the fly market maker alive on its market slots, on any CLOB venue — + bots up, fly_brain running, halts surfaced, closed markets rotated. Never sets a + posture itself. +agent_key: null +skills: [] +default_config: + frequency_sec: 300 + total_amount_quote: 500 + execution_mode: loop + run_name: fly + n_markets: 3 + mode: shadow + risk_limits: + max_position_size_quote: 600 + max_open_executors: 12 +default_trading_context: '' +created_by: 456181693 +created_at: '2026-09-12T00:00:00+00:00' +--- + +# Fly Operator + +You are Market Making Fly's operator loop. The fly (`fly_brain`) quotes; you keep +the plumbing healthy. **You never choose spreads, lean or regime.** + +## Configuration at launch + +Read these from `[CURRENT CONFIG]`: +- `n_markets` (1–3, default 3): how many markets the fly quotes at once. The + one shared brain is shown that many charts in round-robin; each pair is observed + every `n_markets × interval_sec`. Fewer markets means each one is seen more often. +- `total_amount_quote`: capital **per market**. +- `connector_name` and `market_type`: any CLOB venue, spot or perp. On spot, + `leverage` must be 1 and the take-profit floor is several times wider. +- `run_name`: the fly's run directory (brain lineage). +- `mode`: `shadow` or `live`. +- `trading_context`, if present, may name the pairs explicitly ("MM XYZ:DRAM-USD and + XYZ:SPCX-USD"); then `n_markets` is the count of those pairs. + +If the fly is not yet deployed, run the `fly_mm_deploy` skill with exactly +`n_markets` picks from the scanner. Never start `fly_brain` with more pairs than +`n_markets`, and never fewer unless the scanner has fewer open survivors — say so +in the journal when that happens. + +## Each tick + +1. `manage_routines(action="run", name="fly_status", config={"run_name": ""})` + and `manage_routines(action="list_instances")`. +2. **Is the fly running?** If no `fly_brain` instance for this run and the status is + not halted → start it again with the same `run_name` and the pairs/spreads recorded + in your journal (shadow or live, whichever it was). Journal the restart. +3. **Is it halted?** Financial halt → journal, notify, do nothing else (bots are + already stopped by the fly). Transient halt → read `manage_bots(action="logs")` for + the failing bot, journal what you found, restart `fly_brain` with + `resume_reviewed: true` only if the cause is clearly external and resolved. +4. **Are the bots up?** `manage_bots(action="status")`. A missing `{token}-fly` bot for + a pair the fly is running → redeploy it from the saved `{token}_fly_mm` config + (`fly_mm_deploy` Step 4). The fly reads P&L by those names. +5. **Closed / dead slot?** A pair with `STOP_BOT` or many `CLOSED` ticks and a flat + position → follow `fly_mm_deploy` Rotation: scanner, new pick, redeploy, restart + `fly_brain` with the updated pairs and the same `run_name`. +6. **Report** in key: value lines: tick, mode, halted, per-pair regime / spread × / + lean / bot state, changed edges, session P&L vs high, and the standing caveat that + the fly's learning is not validated. + +## Never +- Update a controller's spreads, TP, bands or leverage yourself while `fly_brain` runs. +- Start `mode=live` unless the journal or the trading context says the operator asked. +- Clear a financial halt. diff --git a/agents/market_making_fly/tests/conftest.py b/agents/market_making_fly/tests/conftest.py new file mode 100644 index 000000000..01690e384 --- /dev/null +++ b/agents/market_making_fly/tests/conftest.py @@ -0,0 +1,20 @@ +"""Make ``flybrain`` importable: it lives in the agent dir, not on the module path.""" + +import os +import sys +from pathlib import Path + +AGENT_DIR = Path(__file__).resolve().parents[1] +REPO_ROOT = AGENT_DIR.parents[1] +# This directory has no __init__.py (a package named ``tests`` would collide +# with the repo's), so pytest does not put the repo root on the path itself. +for entry in (str(REPO_ROOT), str(AGENT_DIR)): + if entry not in sys.path: + sys.path.insert(0, entry) + +# The opt-in full-graph test needs the prepared dataset from this install's +# agent home; every other test uses tmp_path. +os.environ.setdefault( + "CONDOR_FLY_DATA", + str(REPO_ROOT / ".condor" / "agents" / "market_making_fly" / "data"), +) diff --git a/agents/market_making_fly/tests/test_fly_chart.py b/agents/market_making_fly/tests/test_fly_chart.py new file mode 100644 index 000000000..45e2b72f6 --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_chart.py @@ -0,0 +1,118 @@ +"""The frame the fly sees: fixed size, right palette, no silent bad data.""" + +import hashlib + +import numpy as np +import pytest +from flybrain import chart + + +def _candles(n=72, start=100.0, step=0.5, up=True): + rows = [] + price = start + for i in range(n): + nxt = price + step if up else price - step + o, c = price, nxt + rows.append( + { + "timestamp": i, + "open": o, + "high": max(o, c) + 0.2, + "low": min(o, c) - 0.2, + "close": c, + "volume": 10 + i, + } + ) + price = nxt + return rows + + +def test_frame_shape_dtype_and_background(): + frame = chart.market_frame("XYZ:DRAM-USD", _candles(), 135.0, 135.2) + assert frame.shape == (chart.HEIGHT, chart.WIDTH, 3) + assert frame.dtype == np.uint8 + assert tuple(frame[90, 2]) == chart.BACKGROUND # left margin + assert tuple(frame[5, 200]) == chart.HEADER + + +def test_up_candles_are_blue_and_down_candles_red(): + up = chart.market_frame("P:A-USD", _candles(up=True), 135.0, 135.2) + down = chart.market_frame("P:A-USD", _candles(up=False), 64.0, 64.2) + plot_up = up[chart.PLOT_TOP : chart.PLOT_BOTTOM, chart.PLOT_LEFT : chart.PLOT_RIGHT] + plot_down = down[ + chart.PLOT_TOP : chart.PLOT_BOTTOM, chart.PLOT_LEFT : chart.PLOT_RIGHT + ] + assert (plot_up == chart.UP).all(axis=2).any() + assert not (plot_up == chart.DOWN).all(axis=2).any() + assert (plot_down == chart.DOWN).all(axis=2).any() + assert not (plot_down == chart.UP).all(axis=2).any() + + +def test_bid_ask_ticks_at_right_edge(): + rows = _candles() + frame = chart.market_frame("P:A-USD", rows, 120.0, 130.0) + lo, span = chart.price_scale(chart.normalize_candles(rows)) + for price in (120.0, 130.0): + y = int(chart._y(price, lo, span)) + strip = frame[y - 1 : y + 2, chart.PLOT_RIGHT + 3 : chart.WIDTH - 2] + assert (strip == chart.TICK).all(axis=2).any() + + +def test_volume_strip_is_drawn(): + frame = chart.market_frame("P:A-USD", _candles(), 135.0, 135.2) + strip = frame[ + chart.VOLUME_TOP : chart.VOLUME_BOTTOM + 1, chart.PLOT_LEFT : chart.PLOT_RIGHT + ] + assert (strip == chart.VOLUME).all(axis=2).any() + + +def test_deterministic(): + a = chart.market_frame("P:A-USD", _candles(), 135.0, 135.2) + b = chart.market_frame("P:A-USD", _candles(), 135.0, 135.2) + assert ( + hashlib.sha256(a.tobytes()).hexdigest() + == hashlib.sha256(b.tobytes()).hexdigest() + ) + + +def test_flat_market_scale_floor(): + rows = [ + {"open": 100, "high": 100, "low": 100, "close": 100, "volume": 1} + for _ in range(10) + ] + lo, span = chart.price_scale(chart.normalize_candles(rows)) + assert span == pytest.approx(100 * 0.002 * 1.24) + assert lo == pytest.approx(100 - 100 * 0.002 * 0.12) + + +def test_keeps_only_last_n_and_right_aligns(): + rows = _candles(n=200) + kept = chart.normalize_candles(rows, 72) + assert len(kept) == 72 and kept[-1]["close"] == rows[-1]["close"] + few = chart.market_frame("P:A-USD", _candles(n=5), 102.0, 102.2) + plot = few[chart.PLOT_TOP : chart.PLOT_BOTTOM] + colored = (plot == chart.UP).all(axis=2).any(axis=0) + assert colored[: chart.PLOT_LEFT + 200].sum() == 0 # nothing on the left + assert colored[chart.PLOT_RIGHT - 30 : chart.PLOT_RIGHT].any() + + +@pytest.mark.parametrize( + "bad", + [ + [], + [{"open": 1, "high": 2, "low": 0.5}], # missing close/volume + [{"open": 1, "high": 0.9, "low": 0.5, "close": 1, "volume": 1}], # high < open + [{"open": 0, "high": 1, "low": 0, "close": 1, "volume": 1}], # zero price + [{"open": float("nan"), "high": 1, "low": 0, "close": 1, "volume": 1}], + ], +) +def test_bad_candles_raise(bad): + with pytest.raises(ValueError): + chart.market_frame("P:A-USD", bad, 1.0, 1.1) + + +def test_bad_quotes_raise(): + with pytest.raises(ValueError): + chart.market_frame("P:A-USD", _candles(), 0, 1) + with pytest.raises(ValueError): + chart.market_frame("P:A-USD", _candles(), 2.0, 1.0) diff --git a/agents/market_making_fly/tests/test_fly_decoder.py b/agents/market_making_fly/tests/test_fly_decoder.py new file mode 100644 index 000000000..ef040e6f9 --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_decoder.py @@ -0,0 +1,209 @@ +"""Spike channels → posture: the regime table, warm-up, centring, hysteresis.""" + +import pytest +from flybrain.decoder import ( + NEUTRAL, + Baseline, + Channels, + DecoderSettings, + Hysteresis, + Posture, + classify, + decode, + should_apply, +) + +S = DecoderSettings(window=20, warmup=5) + + +def _warm(baseline, trend=0.0, arousal=10.0, n=None, jitter=True): + """Feed a calm history with a little spread so std is non-zero.""" + n = n or S.warmup + 5 + for i in range(n): + t = trend + (0.1 if jitter and i % 2 else -0.1) + a = arousal + (0.5 if jitter and i % 2 else -0.5) + decode(Channels(t, a, 0), baseline, S) + + +@pytest.mark.parametrize( + "trend_z,arousal_z,gate,expected", + [ + (0, 3.0, True, "pause"), + (5, 3.0, True, "pause"), # pause beats trending + (0, 1.5, False, "volatile"), + (2, 1.5, True, "volatile"), # volatile beats trending + (2, 0, True, "trending_up"), + (-2, 0, True, "trending_down"), + (2, 0, False, "ranging"), # no gate, no trend call + (0, -1.5, False, "quiet"), + (2, -1.5, True, "trending_up"), # trending beats quiet + (0.5, 0.5, True, "ranging"), + ], +) +def test_regime_table(trend_z, arousal_z, gate, expected): + assert classify(trend_z, arousal_z, gate, S) == expected + + +def test_warmup_emits_neutral(): + b = Baseline() + for _ in range(S.warmup - 1): + assert decode(Channels(50.0, 500.0, 5), b, S) == NEUTRAL + assert decode(Channels(50.0, 500.0, 5), b, S).warm is True + + +def test_centring_removes_constant_bias(): + b = Baseline() + _warm(b, trend=8.0) # the circuit "always turns right" + p = decode(Channels(8.0, 10.0, 3), b, S) + assert p.regime == "ranging" and abs(p.trend_z) < 1 and p.shift_bps == 0 + + +def test_raw_mode_keeps_bias(): + raw = DecoderSettings(window=20, warmup=5, center_bias=False) + b = Baseline() + for _ in range(10): + decode(Channels(8.0, 10.0, 3), b, raw) + p = decode(Channels(8.0, 10.0, 3), b, raw) + assert p.trend_z == pytest.approx(4.0) # 8 Hz / 2 Hz unit + + +def test_trend_up_leans_and_caps(): + b = Baseline() + _warm(b) + p = decode(Channels(50.0, 10.0, 2), b, S) + assert p.regime == "trending_up" and p.shift_bps == S.max_shift_bps and p.gate + + +def test_no_gate_no_lean(): + b = Baseline() + _warm(b) + p = decode(Channels(50.0, 10.0, 0), b, S) + assert p.shift_bps == 0 and p.regime == "ranging" + + +def test_arousal_tightens_sizes_up_and_pauses(): + """An aroused fly leans in: tighter quotes and more of the book. The sign + is a choice, not a finding — arousal is a population rate against its own + average and nothing ties it to volatility.""" + b = Baseline() + _warm(b) + hot = decode(Channels(0.0, 11.5, 0), b, S) + assert hot.arousal_z > 1 + assert hot.spread_mult < 1 and hot.size_mult > 1 + b2 = Baseline() + _warm(b2) + calm = decode(Channels(0.0, 8.0, 0), b2, S) + assert calm.arousal_z < 0 + assert calm.spread_mult > 1 and calm.size_mult < 1 + # the breaker still fires on the same channel, and both knobs stay clipped + b3 = Baseline() + _warm(b3) + pause = decode(Channels(0.0, 100.0, 0), b3, S) + assert pause.regime == "pause" + assert pause.spread_mult == S.spread_min and pause.size_mult == S.size_max + + +def test_baseline_window_and_roundtrip(): + b = Baseline() + _warm(b, n=50) + assert b.count == S.window + again = Baseline.from_dict(b.to_dict()) + assert again.trend == b.trend and again.arousal == b.arousal + + +def test_posture_roundtrip(): + p = Posture("quiet", 0.8, 1.0, -1.0, -0.2, -1.3, 0.0, True, True) + assert Posture.from_dict(p.to_dict()) == p + + +def test_channels_validation(): + with pytest.raises(ValueError): + Channels(float("nan"), 1.0, 0) + with pytest.raises(ValueError): + Channels(0.0, -1.0, 0) + + +def test_settings_validation(): + with pytest.raises(ValueError): + DecoderSettings(z_regime=3.0, z_pause=2.5) + with pytest.raises(ValueError): + DecoderSettings(warmup=0) + + +def test_hysteresis(): + h = Hysteresis(min_apply_interval_sec=300) + base = Posture("ranging", 1.0, 1.0, 0.0, 0, 0, 0.0, False, True) + assert should_apply(None, base, None, 1000, h)[0] + same = Posture("ranging", 1.05, 1.0, 0.2, 0, 0, 0.0, False, True) + assert not should_apply(base, same, 0, 1000, h)[0] + regime = Posture("volatile", 1.0, 1.0, 0.0, 0, 1.2, 0.0, False, True) + assert should_apply(base, regime, 0, 1000, h)[0] + assert not should_apply(base, regime, 900, 1000, h)[0] # cooldown + wider = Posture("ranging", 1.2, 1.0, 0.0, 0, 0, 0.0, False, True) + assert should_apply(base, wider, 0, 1000, h)[0] + lean = Posture("ranging", 1.0, 1.0, 0.6, 0, 0, 0.0, True, True) + assert should_apply(base, lean, 0, 1000, h)[0] + + +def test_valence_moves_the_size_the_fly_commits(): + """MBON07 minus MBON11 is what the KC→MBON memory rule writes to, so it is + the only path a P&L pulse has to a decision. Without it the dopamine loop + moved thousands of synapses and changed nothing the fly did.""" + b = Baseline() + for _ in range(S.warmup): + b.push(Channels(0.0, 8.0, 0, 0.0, 400), S.window) + good = decode(Channels(0.0, 8.0, 0, 6.0, 400), b, S) + assert good.valence_z > 1 and good.size_mult > 1 + b2 = Baseline() + for _ in range(S.warmup): + b2.push(Channels(0.0, 8.0, 0, 0.0, 400), S.window) + bad = decode(Channels(0.0, 8.0, 0, -6.0, 400), b2, S) + assert bad.valence_z < -1 and bad.size_mult < 1 + # and it moves size only — the spread is the arousal channel's + assert good.spread_mult == pytest.approx(bad.spread_mult) + + +def test_a_scene_that_never_reached_the_mushroom_body_is_not_acted_on(): + """Kenyon drive is the confidence test: with no sparse code of the chart, + every other channel is reading the network's own noise.""" + b = Baseline() + for _ in range(S.warmup): + b.push(Channels(0.0, 8.0, 0, 0.0, 400), S.window) + seen = decode(Channels(0.0, 8.0, 1, 0.0, 400), b, S) + assert seen.confident + + dark = Baseline() + for _ in range(S.warmup): + dark.push(Channels(0.0, 8.0, 0, 0.0, 400), S.window) + blind = decode(Channels(0.0, 8.0, 1, 0.0, 0), dark, S) + assert not blind.confident + ok, why = should_apply(seen, blind, None, 0.0, Hysteresis()) + assert ok is False and "kenyon" in why.lower() + + +def _varied(gate: int) -> Baseline: + """A baseline with real spread in it. A flat history has zero variance, so + the decoder scores *any* departure from it at the same 2.2 sigma however + small — which made the first version of this test assert that 2.0 Hz was a + mild move and 40 a strong one when the decoder could not tell them apart.""" + b = Baseline() + for value in (-4.0, -2.0, 0.0, 2.0, 4.0): + b.push(Channels(value, 8.0, gate, 0.0, 400), S.window) + return b + + +def test_a_side_is_only_taken_away_on_a_gated_trend(): + # a strong trend with no gate spike is still both sides: the gate is what + # separates a reading from a decision + ungated = decode(Channels(20.0, 8.0, 0, 0.0, 400), _varied(0), S) + assert ungated.trend_z > S.z_side and ungated.side == "both" + + up = decode(Channels(20.0, 8.0, 1, 0.0, 400), _varied(1), S) + assert up.trend_z > S.z_side and up.side == "buy" + + down = decode(Channels(-20.0, 8.0, 1, 0.0, 400), _varied(1), S) + assert down.trend_z < -S.z_side and down.side == "sell" + + # and a trend too weak to act on leaves both sides up + mild = decode(Channels(2.0, 8.0, 1, 0.0, 400), _varied(1), S) + assert abs(mild.trend_z) < S.z_side and mild.side == "both" diff --git a/agents/market_making_fly/tests/test_fly_full_graph.py b/agents/market_making_fly/tests/test_fly_full_graph.py new file mode 100644 index 000000000..4f7c6bdef --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_full_graph.py @@ -0,0 +1,61 @@ +"""Opt-in full-connectome checks — stonkfly's integration test transposed. + +Needs the prepared dataset (``python -m flybrain prepare``) and about a +minute: ``CONDOR_FLY_FULL_TEST=1 uv run pytest tests/test_fly_full_graph.py``. +""" + +import os + +import numpy as np +import pytest + +pytestmark = pytest.mark.skipif( + os.environ.get("CONDOR_FLY_FULL_TEST") != "1", + reason="Uses the full MaleCNS graph; set CONDOR_FLY_FULL_TEST=1", +) + + +def test_chart_reaches_kenyon_cells_and_pulses_hit_dopamine_cells(tmp_path): + from flybrain.chart import market_frame + from flybrain.data import verify + from flybrain.market import FixtureMarket + from flybrain.worker import FlyBrain + + assert verify()["neurons"] == 166700 + brain = FlyBrain(learning=True) + assert len(brain.brain.post) == 25582938 + assert len(brain.descending) > 1000 and len(brain.left) and len(brain.right) + + market = FixtureMarket(["XYZ:A-USD"], 72) + import asyncio + + obs = asyncio.run(market.observe("XYZ:A-USD")) + frame = market_frame(obs.pair, obs.candles, obs.bid, obs.ask) + first = brain.observe(frame, "none") + assert first["kc_spikes"] > 0, "the rendered chart must activate Kenyon cells" + assert first["total_spikes"] > 0 and first["stimulus_ms"] == 0 + + brain.save(tmp_path / "before.npz") + before = brain.brain.weight[brain.brain.circuit["edges"]].copy() + + reward = brain.observe(frame, "reward") + assert reward["reward_spikes"] > 0 and reward["stimulus_ms"] == 200 + assert reward["memory"]["changed_edges"] > 0 + rewarded = brain.brain.weight[brain.brain.circuit["edges"]].copy() + + brain.restore(tmp_path / "before.npz") + brain.observe(frame, "none") + assert not np.array_equal( + rewarded, brain.brain.weight[brain.brain.circuit["edges"]] + ) + + brain.restore(tmp_path / "before.npz") + brain.brain.weights_frozen = True + brain.observe(frame, "reward") + assert np.array_equal(before, brain.brain.weight[brain.brain.circuit["edges"]]) + + brain.restore(tmp_path / "before.npz") + brain.brain.weights_frozen = False + loss = brain.observe(frame, "aversive") + assert loss["aversive_spikes"] > 0 and loss["stimulus_ms"] == 200 + assert np.isfinite(brain.brain.weight).all() diff --git a/agents/market_making_fly/tests/test_fly_guard.py b/agents/market_making_fly/tests/test_fly_guard.py new file mode 100644 index 000000000..c07682deb --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_guard.py @@ -0,0 +1,173 @@ +"""Every guard rule: veto vs halt, and a financial halt that review cannot clear.""" + +import pytest +from flybrain.decoder import NEUTRAL +from flybrain.guard import ( + GuardSettings, + GuardState, + Halt, + Veto, + check_apply_window, + check_collateral, + check_config, + check_market_open, + check_not_halted, + check_pnl, + check_price_move, + default_max_loss, + record_apply, + resume, +) +from flybrain.posture import MarketSpec, build_config + +SPEC = MarketSpec("hyperliquid_perpetual", "XYZ:DRAM-USD", 500, 8.0) +S = GuardSettings() + + +def test_market_open_counts_closed_ticks_then_asks_for_stop(): + st = GuardState() + assert check_market_open("XYZ:DRAM-USD", True, st, S) is False + small = GuardSettings(closed_ticks_to_stop=3) + for _ in range(2): + with pytest.raises(Veto): + check_market_open("XYZ:DRAM-USD", False, st, small) + assert check_market_open("XYZ:DRAM-USD", False, st, small) is True + assert check_market_open("XYZ:DRAM-USD", True, st, small) is False + assert st.closed_ticks["XYZ:DRAM-USD"] == 0 + + +def test_collateral(): + check_collateral(100.0, 50.0) + with pytest.raises(Veto): + check_collateral(40.0, 50.0) + with pytest.raises(Veto): + check_collateral(float("nan"), 50.0) + + +def test_config_floors(): + good = build_config(SPEC, NEUTRAL) + check_config(good, SPEC) + for bad in ( + {"buy_spreads": "0.0001,0.0009"}, + {"take_profit": 0.0001}, + {"leverage": 20}, + {"trading_pair": "XYZ:SPCX-USD"}, + {"global_sl_enabled": False}, + ): + with pytest.raises(Veto): + check_config({**good, **bad}, SPEC) + + +def test_apply_window_resets_per_utc_day(): + st = GuardState() + small = GuardSettings(max_applies_per_day=2) + day1 = 1_700_000_000.0 + check_apply_window(st, day1, small) + record_apply(st, "P", day1, True, small) + record_apply(st, "P", day1 + 60, True, small) + with pytest.raises(Veto): + check_apply_window(st, day1 + 120, small) + check_apply_window(st, day1 + 86_400, small) + assert st.applies_today == 0 + + +def test_price_move(): + check_price_move(100.0, 100.4, S) + with pytest.raises(Veto): + check_price_move(100.0, 101.0, S) + with pytest.raises(Veto): + check_price_move(0, 1, S) + + +def test_apply_failures_halt_transiently_and_resume(): + st = GuardState() + small = GuardSettings(max_apply_failures=2) + record_apply(st, "P", 1.0, False, small) + with pytest.raises(Halt) as info: + record_apply(st, "P", 2.0, False, small) + assert info.value.financial is False + with pytest.raises(Halt): + check_not_halted(st) + with pytest.raises(Halt): + resume(st, reviewed=False) + resume(st, reviewed=True) + check_not_halted(st) + assert st.consecutive_failures == 0 + record_apply(st, "P", 3.0, True, small) + + +def test_loss_stop_is_financial_and_unclearable(): + st = GuardState() + check_pnl(1.0, 1000.0, st, S, max_loss_quote=20.0) + with pytest.raises(Halt) as info: + check_pnl(-20.0, 1000.0, st, S, max_loss_quote=20.0) + assert info.value.financial is True + with pytest.raises(Halt): + resume(st, reviewed=True) + + +def test_loss_rate_breaker_needs_volume(): + st = GuardState() + check_pnl(-0.1, 10.0, st, S, max_loss_quote=100.0) # too little volume to judge + with pytest.raises(Halt): + check_pnl(-1.0, 1000.0, st, S, max_loss_quote=100.0) # -10 bp of volume + + +def test_no_new_high_breaker(): + st = GuardState() + small = GuardSettings(loss_no_new_high_ticks=3) + check_pnl(1.0, 1000.0, st, small, 100.0) + check_pnl(0.9, 1000.0, st, small, 100.0) + check_pnl(0.9, 1000.0, st, small, 100.0) + with pytest.raises(Halt): + check_pnl(0.9, 1000.0, st, small, 100.0) + + +def test_first_reported_figure_is_the_high_not_a_drawdown(): + st = GuardState() + small = GuardSettings(loss_no_new_high_ticks=2) + check_pnl(-0.5, 100_000.0, st, small, 100.0) # first report, negative + assert st.session_high_net == -0.5 and st.ticks_since_high == 0 + check_pnl(-0.6, 100_000.0, st, small, 100.0) + with pytest.raises(Halt): + check_pnl(-0.6, 100_000.0, st, small, 100.0) + + +def test_new_high_resets_counter(): + st = GuardState() + small = GuardSettings(loss_no_new_high_ticks=3) + for net in (1.0, 0.9, 0.9, 1.1, 1.0, 1.0): + check_pnl(net, 1000.0, st, small, 100.0) + assert st.ticks_since_high == 2 + + +def test_default_max_loss_and_roundtrip(): + assert default_max_loss([SPEC, SPEC], S) == pytest.approx(40.0) + assert default_max_loss([SPEC], GuardSettings(max_loss_quote=7.0)) == 7.0 + st = GuardState(applies_today=3, closed_ticks={"X:Y-USD": 1}, halted="x") + assert GuardState.from_dict(st.to_dict()) == st + + +def test_a_config_sitting_exactly_on_the_floor_is_not_vetoed(): + """Live on 2026-09-13, tick 18: a trending posture leaned the buy side down + to the fee floor, the config serialized it as 0.00013, and the guard vetoed + it because 0.00013 < 1.3 * 1e-4 by one ulp. The guard was refusing the + posture builder's own arithmetic, and a real apply was lost.""" + from flybrain.decoder import Posture + from flybrain.posture import MarketSpec, build_config + + spec = MarketSpec( + connector_name="hyperliquid_perpetual", + trading_pair="XYZ:DRAM-USD", + total_amount_quote=200, + range_bps=4.0, + leverage=1, + portfolio_allocation=0.3, + maker_fee_bps=1.3, + ) + leaned = Posture("trending_up", 1.14, 1.0, 1.27, 2.0, 0.3, 0.0, True, True) + config = build_config(spec, leaned) + assert min(float(x) for x in config["buy_spreads"].split(",")) == pytest.approx( + spec.min_spread_bps * 1e-4 + ) + check_config(config, spec) # must not raise diff --git a/agents/market_making_fly/tests/test_fly_market.py b/agents/market_making_fly/tests/test_fly_market.py new file mode 100644 index 000000000..e622a9930 --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_market.py @@ -0,0 +1,263 @@ +"""Fixture market, book parsing, candle payload shapes, collateral requirement.""" + +import asyncio + +import pytest +from flybrain.chart import market_frame +from flybrain.market import ( + Book, + FixtureMarket, + depth_within, + normalize_candle_payload, + pnl_is_known, + required_collateral, +) +from flybrain.posture import MarketSpec + +PAIRS = ["XYZ:A-USD", "XYZ:B-USD", "XYZ:C-USD"] + + +def test_fixture_observation_renders_and_advances(): + m = FixtureMarket(PAIRS, 72) + obs = asyncio.run(m.observe("XYZ:B-USD")) + assert obs.open and obs.bid < obs.ask and len(obs.candles) == 72 + frame = market_frame(obs.pair, obs.candles, obs.bid, obs.ask) + assert frame.shape == (180, 320, 3) + again = asyncio.run(m.observe("XYZ:B-USD")) + assert again.candles[-1]["close"] != obs.candles[-1]["close"] + + +def test_fixture_equity_stays_inside_the_breaker(): + m = FixtureMarket(PAIRS, 72) + for tick in range(60): + m.tick = tick + net, volume, per_pair, carry = asyncio.run(m.equity(PAIRS)) + assert abs(net / volume) * 1e4 < 5 + assert sum(i["net"] for i in per_pair.values()) == pytest.approx(net) + assert sum(i["volume"] for i in per_pair.values()) == pytest.approx(volume) + assert set(carry) == set(PAIRS) + + +def test_fixture_reports_so_an_offline_run_exercises_the_pulses(): + """A fixture book that did not report would pin the stimulus to ``none`` + and never advance the anchor, which is the whole point of the offline run.""" + m = FixtureMarket(PAIRS, 72) + _, _, per_pair, _ = asyncio.run(m.equity(PAIRS)) + assert all(i["running"] and i["reported"] for i in per_pair.values()) + assert pnl_is_known(per_pair) + # and the swing really does cross the deadband both ways + nets = [] + for tick in range(12): + m.tick = tick + net, _, _, _ = asyncio.run(m.equity(PAIRS)) + nets.append(net) + deltas = [b - a for a, b in zip(nets, nets[1:])] + assert max(deltas) > 0.02 and min(deltas) < -0.02 + + +@pytest.mark.parametrize( + "per_pair,expected", + [ + ({}, False), # no book at all is silence, not a result + ({"a": {"running": False}}, False), + ({"a": {"running": True}}, False), # running but no report yet + ({"a": {"running": True, "reported": True}}, True), + ( + {"a": {"running": True, "reported": True}, "b": {"running": True}}, + False, # one silent book makes the combined figure unusable + ), + ({"a": {"running": True, "reported": True}, "b": {"running": False}}, True), + ], +) +def test_pnl_is_known(per_pair, expected): + assert pnl_is_known(per_pair) is expected + + +def test_fixture_never_applies(): + m = FixtureMarket(PAIRS, 72) + with pytest.raises(RuntimeError): + asyncio.run(m.apply("XYZ:A-USD", {})) + assert asyncio.run(m.stop_bot("XYZ:A-USD")) is False + + +def test_book_open(): + assert Book(1.0, 1.1).open + assert not Book(None, None).open + + +def test_candle_payload_shapes(): + rows = [{"close": 1}] + assert normalize_candle_payload(rows) == rows + assert normalize_candle_payload({"data": rows}) == rows + assert normalize_candle_payload({"candles": rows}) == rows + with pytest.raises(RuntimeError): + normalize_candle_payload([]) + with pytest.raises(RuntimeError): + normalize_candle_payload({"data": []}) + + +def test_required_collateral(): + spec = MarketSpec("hyperliquid_perpetual", "XYZ:A-USD", 500, 8.0, leverage=2) + assert required_collateral([spec, spec]) == pytest.approx(2 * 500 * 0.65 / 2) + + +class _MarketData: + """Records what the generic order-book endpoint was asked for.""" + + def __init__(self, book=None): + self.calls = [] + self.book = ( + book if book is not None else {"bids": [[99.0, 5]], "asks": [[101.0, 5]]} + ) + + async def get_order_book(self, connector, pair, depth=1): + self.calls.append((connector, pair, depth)) + return self.book + + +class _Client: + def __init__(self, book=None): + self.market_data = _MarketData(book) + + +def _live(connector, book=None): + from flybrain.market import LiveMarket + + return LiveMarket(_Client(book), connector, "5m", 72) + + +def test_a_plain_pair_reads_the_generic_order_book(): + m = _live("binance_perpetual") + book = asyncio.run(m.book("SOL-USDT")) + assert (book.bid, book.ask) == (99.0, 101.0) and book.open + assert m.client.market_data.calls == [("binance_perpetual", "SOL-USDT", 1)] + assert asyncio.run(m.fresh_mid("SOL-USDT")) == 100.0 + + +def test_an_empty_generic_book_is_closed_not_an_error(): + m = _live("binance", {"bids": [], "asks": []}) + assert not asyncio.run(m.book("SOL-USDT")).open + + +@pytest.mark.parametrize( + "book", + [ + {"bids": [[float("nan"), 1]], "asks": [[101.0, 1]]}, + {"bids": [[0.0, 1]], "asks": [[101.0, 1]]}, + {"bids": [[101.0, 1]], "asks": [[99.0, 1]]}, # crossed + ], +) +def test_a_nonsense_generic_book_raises(book): + with pytest.raises(RuntimeError): + asyncio.run(_live("binance", book).book("SOL-USDT")) + + +def test_a_hip3_pair_does_not_use_the_generic_endpoint(): + """hummingbot-api's order-book endpoint 500s on HIP-3 pairs, so those must + go to Hyperliquid's own. Proven by the generic one never being called.""" + import aiohttp + + m = _live("hyperliquid_perpetual") + try: + asyncio.run(m.book("XYZ:ORCL-USD")) + except (aiohttp.ClientError, RuntimeError, OSError): + pass # offline in CI; the point is which path was taken + assert m.client.market_data.calls == [] + + +def test_a_book_level_is_read_whatever_shape_the_venue_sends(): + from flybrain.market import level + + assert level({"px": "100.5", "sz": "2"}) == (100.5, 2.0) # Hyperliquid + assert level([100.5, 2]) == (100.5, 2.0) # hummingbot-api + assert level({"price": 100.5, "quantity": 2}) == (100.5, 2.0) + assert level({"price": 100.5, "amount": 2}) == (100.5, 2.0) + + +def test_depth_only_counts_what_is_close_enough_to_trade_against(): + """Liquidity resting far from mid is not liquidity this strategy sees.""" + from flybrain.market import depth_within + + bids = [(99.9, 10), (99.0, 100), (90.0, 1000)] # ~10bp, ~100bp, ~1000bp out + asks = [(100.1, 10), (101.0, 100), (110.0, 1000)] + bid_usd, ask_usd, spread = depth_within(bids, asks, within_bps=20) + assert spread == pytest.approx(20.0, abs=0.1) + assert bid_usd == pytest.approx(99.9 * 10) # the 99.0 level is ~100bp out + assert ask_usd == pytest.approx(100.1 * 10) + # a wider band reaches further down the ladder + wide_bid, wide_ask, _ = depth_within(bids, asks, within_bps=150) + assert wide_bid > bid_usd and wide_ask > ask_usd + # an empty side is no depth and no spread, not a crash + assert depth_within([], asks, 20) == (0.0, 0.0, 0.0) + + +def test_the_scanner_measures_the_trip_the_fly_would_actually_make(): + """Both older scanners asked whether the touch already clears a round trip, + which is the test for joining the touch. The fly rests a quote away from + mid, so what matters is the distance it must travel: down to level 1, never + inside the fee, then out through the take-profit floor.""" + import importlib.util + from pathlib import Path + + path = Path(__file__).resolve().parents[1] / "routines" / "mm_market_scanner.py" + spec = importlib.util.spec_from_file_location("mm_market_scanner", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + + # XYZ:ORCL-USD as it was on 2026-09-13: 2 bp touch, 1.3 bp all-in fee. + # Level 1 is max(2, 2/2) = 2 bp, the take-profit floor 2.2 × 2.6 = 5.72. + assert mod.cycle_bps(2.0, 1.3) == pytest.approx(2.0 + 5.72) + # A dearer market has to travel further for the same quote, because the + # exit is fee-derived: binance spot at 7.5 bp needs 33 bp of take-profit. + assert mod.cycle_bps(2.0, 7.5) == pytest.approx(7.5 + 33.0) + # and the fee is a floor on the entry too — a market quoting inside it + # cannot be quoted inside it + assert mod.cycle_bps(0.2, 7.5) == pytest.approx(7.5 + 33.0) + # A market whose median candle travels less than the cycle never completes + # one, however wide its touch looks. + assert 4.0 / mod.cycle_bps(2.0, 1.3) < 1.0 + + +def test_depth_survives_a_ladder_that_arrives_out_of_order(): + """The walk stops at the first level outside the band, so one out-of-order + rung would hide every closer level behind it — understating depth and + rejecting a market that was eligible.""" + tidy_bids = [(100.0, 5.0), (99.9, 5.0), (99.0, 5.0)] + tidy_asks = [(100.1, 5.0), (100.2, 5.0), (101.0, 5.0)] + expected = depth_within(tidy_bids, tidy_asks, within_bps=30) + + shuffled_bids = [(99.0, 5.0), (100.0, 5.0), (99.9, 5.0)] + shuffled_asks = [(101.0, 5.0), (100.1, 5.0), (100.2, 5.0)] + assert depth_within(shuffled_bids, shuffled_asks, within_bps=30) == expected + # and the touch itself is the best price, not whatever arrived first + assert depth_within(shuffled_bids, shuffled_asks, 30)[2] == pytest.approx( + (100.1 - 100.0) / 100.05 * 1e4 + ) + + +def test_the_scanner_refuses_a_threshold_that_would_invert_a_filter(): + """A negative fee or multiple does not loosen the spread floor, it puts it + below zero — after which every market that is not crossed 'clears the + fee', which is the one mistake this routine exists to prevent.""" + import importlib.util + from pathlib import Path + + import pydantic + + path = Path(__file__).resolve().parents[1] / "routines" / "mm_market_scanner.py" + spec = importlib.util.spec_from_file_location("mm_market_scanner", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + + for field, bad in ( + ("maker_fee_bps", -1.3), + ("min_range_over_cycle", -1.0), + ("min_range_over_cycle", 0.0), + ("min_volume_usd", -1.0), + ("max_daily_drift_pct", -3.0), + ("min_book_depth_usd", -10_000.0), + ("depth_within_bps", 0.0), + ): + with pytest.raises(pydantic.ValidationError): + mod.Config(**{field: bad}) + mod.Config() # the defaults are all inside their own bounds diff --git a/agents/market_making_fly/tests/test_fly_naming.py b/agents/market_making_fly/tests/test_fly_naming.py new file mode 100644 index 000000000..2e91df3a6 --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_naming.py @@ -0,0 +1,51 @@ +"""Derived names for any CLOB pair, and the pairs-list rules.""" + +import pytest +from flybrain.naming import pair_names, parse_pairs + + +def test_hip3_pair(): + n = pair_names("XYZ:ORCL-USD") + assert (n.base, n.quote, n.issuer) == ("ORCL", "USD", "xyz") + assert n.hl_coin == "xyz:ORCL" + assert (n.slug, n.bot_name, n.config_name) == ( + "xyz-orcl-usd", + "xyz-orcl-usd-fly", + "xyz_orcl_usd_fly_mm", + ) + + +def test_plain_pair(): + n = pair_names("SOL-USDT") + assert (n.base, n.quote, n.issuer) == ("SOL", "USDT", "") + assert n.hl_coin == "" # nothing to ask Hyperliquid for + assert (n.slug, n.bot_name, n.config_name) == ( + "sol-usdt", + "sol-usdt-fly", + "sol_usdt_fly_mm", + ) + + +def test_the_whole_pair_names_the_bot(): + """One token on two quotes is two markets. Naming both after the base + would point the fly at one book\'s P&L while updating the other\'s config.""" + assert pair_names("BTC-USDT").bot_name != pair_names("BTC-USDC").bot_name + assert pair_names("XYZ:ORCL-USD").bot_name != pair_names("ABC:ORCL-USD").bot_name + + +@pytest.mark.parametrize( + "bad", ["sol-usdt", "SOLUSDT", "XYZ:-USD", ":SOL-USDT", "SOL-", "-USDT", ""] +) +def test_bad_pairs(bad): + with pytest.raises(ValueError): + pair_names(bad) + + +def test_parse_pairs(): + assert parse_pairs(" SOL-USDT, BTC-USDT ") == ["SOL-USDT", "BTC-USDT"] + with pytest.raises(ValueError): + parse_pairs("") + with pytest.raises(ValueError): + parse_pairs("SOL-USDT,SOL-USDT") + with pytest.raises(ValueError): + parse_pairs("A-USDT,B-USDT,C-USDT,D-USDT") diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py new file mode 100644 index 000000000..a5ee4f812 --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -0,0 +1,306 @@ +"""Posture → pmm_mister config: floors, lean cap, timing table, pause switch.""" + +import pytest +from flybrain.decoder import NEUTRAL, Posture +from flybrain.posture import ( + BPS, + TIMING, + MarketSpec, + base_levels_bps, + build_config, + config_diff, + take_profit_floor, +) + +SPEC = MarketSpec( + connector_name="hyperliquid_perpetual", + trading_pair="XYZ:DRAM-USD", + total_amount_quote=500, + range_bps=10.0, + # What a HIP-3 market in growth mode costs all-in: 0.3 bp to the venue plus + # 1.0 bp of builder fee. The loop fetches this per market; a core + # Hyperliquid perp is 2.5 bp, which is why it is not assumed here. + maker_fee_bps=1.3, +) + + +def _spreads(value): + return [float(x) for x in value.split(",")] + + +def test_neutral_config_sits_at_half_a_bar_and_a_quarter_beyond(): + cfg = build_config(SPEC, NEUTRAL) + l1, l2 = base_levels_bps(SPEC) + assert (l1, l2) == (5.0, 7.5) # a 10 bp bar: half of it, then a quarter more + assert _spreads(cfg["buy_spreads"]) == pytest.approx([5 * BPS, 7.5 * BPS]) + assert _spreads(cfg["sell_spreads"]) == pytest.approx([5 * BPS, 7.5 * BPS]) + assert ( + cfg["controller_name"] == "pmm_mister" and cfg["controller_type"] == "generic" + ) + assert cfg["trading_pair"] == "XYZ:DRAM-USD" + assert cfg["global_sl_enabled"] is True and cfg["global_stop_loss"] == 0.05 + assert cfg["manual_kill_switch"] is False + assert (cfg["executor_refresh_time"], cfg["buy_cooldown_time"]) == TIMING["ranging"] + assert cfg["open_order_type"] == 3 and cfg["take_profit_order_type"] == 3 + + +def test_take_profit_floor_beats_fees_and_spread(): + assert take_profit_floor(SPEC) == pytest.approx(2.2 * 2 * 1.3 * BPS) + cfg = build_config(SPEC, NEUTRAL) + assert cfg["take_profit"] >= take_profit_floor(SPEC) + cheap = MarketSpec(**{**SPEC.__dict__, "maker_fee_bps": 0.1}) + assert take_profit_floor(cheap) == pytest.approx(4 * BPS) + + +def test_volatile_widens_quiet_tightens_with_floor(): + wide = build_config( + SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, 0.0, False, True) + ) + assert _spreads(wide["buy_spreads"])[0] == pytest.approx(10 * BPS) + assert (wide["executor_refresh_time"], wide["buy_cooldown_time"]) == TIMING[ + "volatile" + ] + tight = build_config( + SPEC, Posture("quiet", 0.6, 1.0, 0.0, 0, -1.5, 0.0, False, True) + ) + # 5 bp × 0.6 = 3 bp, well inside a typical bar and clear of the 1.3 bp fee + assert _spreads(tight["buy_spreads"])[0] == pytest.approx(3 * BPS) + assert (tight["executor_refresh_time"], tight["buy_cooldown_time"]) == TIMING[ + "quiet" + ] + + +def test_lean_is_asymmetric_and_capped(): + up = build_config( + SPEC, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) + ) + buy, sell = _spreads(up["buy_spreads"]), _spreads(up["sell_spreads"]) + # lean capped at half of level 1 (5 bp → 2.5 bp), clear of the fee floor + assert buy[0] == pytest.approx(2.5 * BPS) and sell[0] == pytest.approx(7.5 * BPS) + assert buy[1] == pytest.approx(5 * BPS) and sell[1] == pytest.approx(10 * BPS) + down = build_config( + SPEC, Posture("trending_down", 1.0, 1.0, -3.0, -2.0, 0, 0.0, True, True) + ) + assert _spreads(down["sell_spreads"])[0] == pytest.approx(2.5 * BPS) + assert _spreads(down["buy_spreads"])[0] == pytest.approx(7.5 * BPS) + + +def test_the_spread_floor_is_the_market_own_fee(): + """A quote at the floor breaks even: buy at −f and sell at +f capture 2f, + exactly the round trip. A fixed 3 bp was too wide for a HIP-3 perp and far + too tight for a spot book.""" + assert SPEC.min_spread_bps == pytest.approx(SPEC.maker_fee_bps) + dear = MarketSpec( + connector_name="binance", + trading_pair="SOL-USDT", + total_amount_quote=500, + range_bps=10.0, + ) + assert dear.min_spread_bps == pytest.approx(7.5) # binance spot + # a lean that would quote inside the fee is pushed back out to it + leaned = build_config( + dear, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) + ) + assert min(_spreads(leaned["buy_spreads"])) == pytest.approx(7.5 * BPS) + + +def test_pause_sets_kill_switch(): + cfg = build_config(SPEC, Posture("pause", 2.5, 1.0, 0.0, 0, 3.0, 0.0, False, True)) + assert cfg["manual_kill_switch"] is True + + +def test_every_spread_respects_min(): + for regime in TIMING: + for mult in (0.6, 1.0, 2.5): + for shift in (-3.0, 0.0, 3.0): + cfg = build_config( + SPEC, Posture(regime, mult, 1.0, shift, 0, 0, 0.0, True, True) + ) + for key in ("buy_spreads", "sell_spreads"): + assert min(_spreads(cfg[key])) >= SPEC.min_spread_bps * BPS - 1e-12 + + +def test_spec_validation(): + with pytest.raises(ValueError): # lowercase pair + MarketSpec("hyperliquid_perpetual", "xyz:dram-usd", 500, 8) + with pytest.raises(ValueError): # no quote + MarketSpec("hyperliquid_perpetual", "DRAMUSD", 500, 8) + with pytest.raises(ValueError): # leverage above the cap + MarketSpec("hyperliquid_perpetual", "XYZ:DRAM-USD", 500, 8, leverage=10) + with pytest.raises(ValueError): # no capital + MarketSpec("hyperliquid_perpetual", "XYZ:DRAM-USD", 0, 8) + + +def test_spot_and_perp_are_settled_by_the_connector(): + perp = MarketSpec("binance_perpetual", "SOL-USDT", 1000, 6.0, leverage=3) + spot = MarketSpec("binance", "SOL-USDT", 1000, 6.0) + assert (perp.market_type, spot.market_type) == ("perp", "spot") + assert not perp.is_spot and spot.is_spot + # a spot book has nothing to lever + with pytest.raises(ValueError, match="leverage must be 1"): + MarketSpec("binance", "SOL-USDT", 1000, 6.0, leverage=3) + # and a mislabelled connector is refused rather than silently reinterpreted + with pytest.raises(ValueError, match="looks like a perp"): + MarketSpec("binance_perpetual", "SOL-USDT", 1000, 6.0, market_type="spot") + + +def test_position_mode_is_a_perp_field(): + perp = build_config(MarketSpec("binance_perpetual", "SOL-USDT", 1000, 6.0), NEUTRAL) + spot = build_config(MarketSpec("binance", "SOL-USDT", 1000, 6.0), NEUTRAL) + assert perp["position_mode"] == "ONEWAY" and perp["leverage"] == 1 + assert "position_mode" not in spot + + +def test_the_fee_floor_follows_the_venue(): + """Spot fees run several times perp fees, and a take-profit that is + comfortably profitable on a perp loses money on spot.""" + perp = MarketSpec("binance_perpetual", "SOL-USDT", 1000, 6.0) + spot = MarketSpec("binance", "SOL-USDT", 1000, 6.0) + assert perp.maker_fee_bps == 2.0 and spot.maker_fee_bps == 7.5 + assert take_profit_floor(spot) > 3 * take_profit_floor(perp) + assert build_config(spot, NEUTRAL)["take_profit"] >= take_profit_floor(spot) + # an unknown venue is quoted wide, not tight + unknown = MarketSpec("some_new_dex", "SOL-USDT", 1000, 6.0) + assert unknown.maker_fee_bps == 10.0 + # and the real figure always wins + assert ( + MarketSpec("binance", "SOL-USDT", 1000, 6.0, maker_fee_bps=1.0).maker_fee_bps + == 1.0 + ) + + +def test_config_diff(): + a = build_config(SPEC, NEUTRAL) + b = build_config(SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, 0.0, False, True)) + diff = config_diff(a, b) + assert "buy_spreads" in diff and "trading_pair" not in diff + assert config_diff(None, a) == a + + +def test_an_order_sized_to_the_bare_minimum_is_refused(): + """The live failure of 2026-09-13: 200 quote at 0.2 allocation sizes each + order to exactly the 10 USD minimum, the controller rounds the base amount + down to the market's step, and Hyperliquid rejected all of them at 9.94.""" + bare = MarketSpec( + **{**SPEC.__dict__, "total_amount_quote": 200, "portfolio_allocation": 0.2} + ) + assert bare.order_notional == pytest.approx(10.0) + with pytest.raises(ValueError, match="after rounding"): + bare.check_order_size() + # and the message names an allocation that actually clears it + roomy = MarketSpec(**{**bare.__dict__, "portfolio_allocation": 0.3}) + roomy.check_order_size() + assert roomy.order_notional == pytest.approx(15.0) + + +def test_quotes_are_placed_against_how_far_the_market_travels(): + """Level 1 sat at half the *touch* — 2 bp on a market whose typical bar + ranges 9.9 — so every multiplier the fly could express stayed inside what a + normal bar covers, and five replay variants returned the same 23 fills. + Half the range puts it where a typical bar just reaches.""" + from flybrain.posture import base_levels_from_range + + assert base_levels_from_range(9.9) == (pytest.approx(4.95), pytest.approx(7.425)) + # the multiplier now straddles a typical bar's reach instead of living + # inside it: tight fills most bars, wide fills few + first, _ = base_levels_from_range(9.9) + assert first * 0.6 < 9.9 / 2 < first * 2.5 + # a dead market still gets a floor rather than a quote on top of mid + assert base_levels_from_range(0.4)[0] == 2.0 + for market_range in (0.1, 0.4, 4.0, 9.9, 40.0): + first, second = base_levels_from_range(market_range) + assert second > first, f"levels inverted at range={market_range}" + tight = build_config(MarketSpec(**{**SPEC.__dict__, "range_bps": 0.4}), NEUTRAL) + buys = _spreads(tight["buy_spreads"]) + assert buys == sorted(buys) and len(set(buys)) == 2 + + +def test_size_follows_arousal_and_stays_inside_both_limits(): + """The fly quotes more of the book when aroused, but never so little that + an order falls under the venue minimum, nor more than the whole book.""" + # 200 quote at 0.3 is a 15 order; scaled down by 0.6 it would be 9, under + # the 12 the venue needs once the base amount is rounded, so the floor + # binds before the multiplier does. + spec = MarketSpec( + **{**SPEC.__dict__, "total_amount_quote": 200, "portfolio_allocation": 0.3} + ) + hot = build_config(spec, Posture("ranging", 1.0, 2.5, 0.0, 0, 0, 0.0, False, True)) + calm = build_config(spec, Posture("ranging", 1.0, 0.6, 0.0, 0, 0, 0.0, False, True)) + assert hot["portfolio_allocation"] > spec.portfolio_allocation + assert calm["portfolio_allocation"] < spec.portfolio_allocation + assert calm["portfolio_allocation"] == pytest.approx(0.24) + assert calm["portfolio_allocation"] == pytest.approx(spec.min_portfolio_allocation) + # and a big book scaled up still cannot quote more than all of itself + big = MarketSpec(**{**SPEC.__dict__, "portfolio_allocation": 0.5}) + assert build_config(big, Posture("ranging", 1.0, 2.5, 0.0, 0, 0, 0.0, False, True))[ + "portfolio_allocation" + ] == pytest.approx(1.0) + + +def test_the_config_matches_the_experts_balanced_profile(): + """Everything that is not spread, floor or feedback is the expert's vetted + profile, and is stated rather than left to the controller's defaults.""" + cfg = build_config(SPEC, NEUTRAL) + assert cfg["target_base_pct"] == 0.5 + assert cfg["min_base_pct"] == 0.35 and cfg["max_base_pct"] == 0.65 + assert cfg["max_active_executors_by_level"] == 3 + assert cfg["min_skew"] == 1.5 + assert cfg["buy_position_effectivization_time"] == 120 + assert cfg["sell_position_effectivization_time"] == 120 + assert cfg["price_distance_tolerance"] == 0.0005 + assert cfg["refresh_tolerance"] == 0.0005 + assert cfg["tolerance_scaling"] == 1.2 + assert cfg["global_tp_enabled"] is False and cfg["global_sl_enabled"] is True + assert cfg["global_sl_activation_from"] == "target_base" + assert cfg["global_pnl_reference"] == "position" + assert cfg["tick_mode"] is False + + +def test_the_exit_scales_with_the_market_and_never_goes_under_the_fee(): + """The take-profit was the fee floor whether a market moved 2 bp a bar or + 20, so five replay variants closed the same eleven round trips. It is now + three quarters of a typical bar, with the fee floor and the fly's own first + level underneath it. + + The fly does not move it. Letting arousal widen it halved the round trips — + 35 against a fixed exit's 68 on the same candles — and earned nothing for + them, so that half of the change was reverted and this is the half kept.""" + from flybrain.posture import take_profit_base_bps, take_profit_floor_bps + + # a market that moves: the range sets the exit, not the fee + assert take_profit_base_bps(1.3, 20.0) == pytest.approx(15.0) + # a market that barely moves: the fee floor holds it up + assert take_profit_base_bps(1.3, 2.0) == pytest.approx(take_profit_floor_bps(1.3)) + + # SPEC's 10 bp range puts the exit at 7.5 bp, whatever the posture says + for posture in ( + NEUTRAL, + Posture("volatile", 1.0, 1.0, 0.0, 0, 1.5, 0.0, False, True), + Posture("quiet", 0.6, 1.0, 0.0, 0, -1.5, 0.0, False, True), + ): + assert float(build_config(SPEC, posture)["take_profit"]) == pytest.approx( + 7.5 * BPS + ) + + # and on a market that barely moves, the fee floor holds the exit up + tight = MarketSpec(**{**SPEC.__dict__, "range_bps": 2.0}) + cfg = build_config(tight, NEUTRAL) + assert float(cfg["take_profit"]) >= take_profit_floor(tight) + + +def test_a_gated_trend_takes_a_side_off_the_book(): + """Leaning moved the quote 2.5 bp on a market whose bars run 8, which is + why the fly filled five buys and no sells into a fall and held them. A + trend worth acting on removes the other side instead of discounting it.""" + up = build_config( + SPEC, + Posture("trending_up", 1.0, 1.0, 2.0, 2.0, 0, 0.0, True, True, "buy"), + ) + assert up["sell_amounts_pct"] == "0,0" and up["buy_amounts_pct"] == "1,1" + down = build_config( + SPEC, + Posture("trending_down", 1.0, 1.0, -2.0, -2.0, 0, 0.0, True, True, "sell"), + ) + assert down["buy_amounts_pct"] == "0,0" and down["sell_amounts_pct"] == "1,1" + # both sides is still the default, and the guard sees a valid config either way + assert build_config(SPEC, NEUTRAL)["buy_amounts_pct"] == "1,1" diff --git a/agents/market_making_fly/tests/test_fly_reinforcement.py b/agents/market_making_fly/tests/test_fly_reinforcement.py new file mode 100644 index 000000000..5751497a6 --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_reinforcement.py @@ -0,0 +1,32 @@ +"""P&L delta → dopamine pulse kind, and the controller net-P&L reader.""" + +from decimal import Decimal + +import pytest +from flybrain.reinforcement import controller_net, reinforcement + + +@pytest.mark.parametrize( + "equity,expected", + [("100.03", "reward"), ("99.97", "aversive"), ("100.001", "none"), ("100", "none")], +) +def test_explicit_feedback(equity, expected): + kind, delta = reinforcement(equity, "100", ".01") + assert kind == expected + assert delta == Decimal(equity) - Decimal("100") + + +def test_deadband_must_be_positive(): + with pytest.raises(ValueError): + reinforcement("1", "0", "0") + + +def test_controller_net_includes_unrealized(): + assert ( + controller_net({"realized_pnl_quote": "1.5", "unrealized_pnl_quote": -2}) + == -0.5 + ) + with pytest.raises(ValueError): + controller_net({"realized_pnl_quote": 1}) + with pytest.raises(ValueError): + controller_net({"realized_pnl_quote": float("inf"), "unrealized_pnl_quote": 0}) diff --git a/agents/market_making_fly/tests/test_fly_replay.py b/agents/market_making_fly/tests/test_fly_replay.py new file mode 100644 index 000000000..30b95593a --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_replay.py @@ -0,0 +1,139 @@ +"""The replay fill model: what it counts as a fill, a close, and a P&L.""" + +import pytest +from flybrain.decoder import NEUTRAL, DecoderSettings +from flybrain.posture import MarketSpec, build_config +from flybrain.replay import Ledger, Lot, frames, paired_stats, quote_prices, step + +SPEC = MarketSpec( + connector_name="hyperliquid_perpetual", + trading_pair="XYZ:DRAM-USD", + total_amount_quote=200, + range_bps=10.0, + leverage=1, + portfolio_allocation=0.3, + maker_fee_bps=1.3, +) +CONFIG = build_config(SPEC, NEUTRAL) + + +def _candle(high, low, close=None): + return {"high": high, "low": low, "close": close if close else (high + low) / 2} + + +def test_a_quote_the_price_never_reached_does_not_fill(): + ledger = Ledger() + mid = 100.0 + # a candle that never moves cannot touch a quote resting away from mid + step(ledger, CONFIG, SPEC, _candle(100.0, 100.0), mid, max_lots=8) + assert ledger.fills == 0 and ledger.open_lots == [] + + +def test_a_quote_the_price_traded_through_fills_and_pays_its_fee(): + ledger = Ledger() + mid = 100.0 + prices = quote_prices(CONFIG, mid) + lowest_buy = min(prices["buy"]) + step(ledger, CONFIG, SPEC, _candle(100.0, lowest_buy * 0.999), mid, max_lots=8) + assert ledger.fills == len(prices["buy"]) # both buy levels, no sell + assert all(l.side == "buy" for l in ledger.open_lots) + assert ledger.fees == pytest.approx( + ledger.fills * (200 * 0.3 / 4) * SPEC.maker_fee_bps * 1e-4 + ) + + +def test_a_lot_closes_when_its_take_profit_is_traded_through(): + ledger = Ledger() + tp = float(CONFIG["take_profit"]) + ledger.open_lots = [Lot(side="buy", price=100.0, amount=1.0, take_profit=tp)] + # max_lots=0 admits no new fill, so only the close is under test — a buy + # fills whenever the low is under its price, which no choice of mid avoids + step(ledger, CONFIG, SPEC, _candle(100.0 * (1 + tp) * 1.001, 99.999), 100.0, 0) + assert ledger.round_trips == 1 + assert ledger.realized == pytest.approx(100.0 * tp, rel=1e-3) + assert not ledger.open_lots + + +def test_equity_marks_open_inventory_and_nets_fees(): + ledger = Ledger(realized=1.0, fees=0.25) + ledger.open_lots = [Lot(side="buy", price=100.0, amount=2.0, take_profit=0.001)] + assert ledger.equity(101.0) == pytest.approx(1.0 - 0.25 + 2.0) + assert ledger.equity(99.0) == pytest.approx(1.0 - 0.25 - 2.0) + assert ledger.inventory == pytest.approx(2.0) + + +def test_a_tick_never_decides_on_a_candle_it_can_see(): + """The frame is built from candles strictly before the one it trades.""" + candles = [ + {"open": 1, "high": 1 + i, "low": 1, "close": 1 + i, "volume": 1} + for i in range(1, 12) + ] + out = frames("XYZ:DRAM-USD", candles, window=5) + assert len(out) == len(candles) - 5 - 1 + _, traded, mid = out[0] + assert traded is candles[5] + assert mid == pytest.approx(float(candles[4]["close"])) + + +def test_paired_stats_compares_earnings_not_the_gap_it_already_has(): + """A cumulative curve inherits every past difference, so a t on levels + reports how long ago two runs diverged. Run that way on the 2026-09-13 + replay it returned |t| of 27 to 63 for variants a few percent apart.""" + flat = [0.0] * 50 + assert paired_stats(flat, flat)["t"] == 0.0 + + # one run earns a steady amount more every tick: a real, detectable edge + steady = paired_stats([i * 0.01 for i in range(50)], flat) + assert steady["mean_diff"] == pytest.approx(0.01) + assert steady["t"] == 0.0 # perfectly steady: no variance to test against + + # a run that jumped once and then tracked its control earns the same after + # the jump, and must not read as an edge however wide the gap stays + jumped = [0.0] * 10 + [5.0] * 40 + stats = paired_stats(jumped, flat) + assert stats["final_gap"] == pytest.approx(5.0) + assert abs(stats["t"]) < 2, "a single old jump is not a per-tick edge" + + +def test_the_disconnected_gain_is_a_gain_the_decoder_accepts(): + """`valence_gain: 0` is refused by design — a channel read and discarded — + so the control uses the smallest gain that cannot move a posture.""" + import importlib.util + from pathlib import Path + + path = Path(__file__).resolve().parents[1] / "routines" / "fly_replay.py" + spec = importlib.util.spec_from_file_location("fly_replay", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + with pytest.raises(ValueError): + DecoderSettings(valence_gain=0.0) + settings = DecoderSettings(valence_gain=mod.OFF) + assert round(1 + settings.valence_gain * 3.0, 4) == 1.0 + + +def test_windows_are_contiguous_and_share_no_candle(): + from flybrain.replay import windows + + series = [{"close": i} for i in range(1000)] + cut = windows(series, 4, 72) + assert [len(w) for w in cut] == [250] * 4 + assert [w[0]["close"] for w in cut] == [0, 250, 500, 750] + # a split too thin to score is refused rather than silently returning stubs + with pytest.raises(ValueError, match="not enough"): + windows(series, 8, 120) + + +def test_a_result_carried_by_one_window_is_not_a_result(): + """The whole point of several windows: a control beaten in one slice and + lost in the rest should not read as an edge because the totals add up.""" + from flybrain.replay import pooled_stats + + flat = [0.0] * 60 + jump = [0.0] * 30 + [9.0] * 30 # one window, one big move, then nothing + carried = pooled_stats([(jump, flat), (flat, jump), (flat, jump), (flat, jump)]) + assert carried["led"] == 1 and carried["windows"] == 4 + + # and a steady per-tick edge in every window shows in both + steady = [i * 0.05 for i in range(60)] + real = pooled_stats([(steady, flat)] * 4) + assert real["led"] == 4 and real["mean_diff"] > 0 diff --git a/agents/market_making_fly/tests/test_fly_report.py b/agents/market_making_fly/tests/test_fly_report.py new file mode 100644 index 000000000..72dcc33fc --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_report.py @@ -0,0 +1,157 @@ +"""The run dashboard: the fly mesh, and the panels' own formatting.""" + +import importlib.util +from pathlib import Path + +import numpy as np +import pytest +from flybrain import fly3d + + +def _module(name): + path = Path(__file__).resolve().parents[1] / "routines" / f"{name}.py" + spec = importlib.util.spec_from_file_location(name, path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def test_every_mesh_face_indexes_a_real_vertex(): + """A Mesh3d with an out-of-range face renders nothing and says nothing.""" + x, y, z, i, j, k = fly3d._ellipsoid((0, 0, 0), (1, 0.5, 0.5), n_u=12, n_v=7) + assert len(x) == len(y) == len(z) == 12 * 5 + 2 # 5 rings plus two poles + assert len(i) == len(j) == len(k) + for faces in (i, j, k): + assert faces.min() >= 0 and faces.max() < len(x) + assert np.isfinite(np.concatenate([x, y, z])).all() + + +def test_no_face_collapses_to_a_sliver(): + """Coincident pole vertices make zero-area triangles, which WebGL draws as + a white sawtooth across the body even though a static export hides them.""" + x, y, z, i, j, k = fly3d._ellipsoid((0, 0, 0), (1, 0.5, 0.5), n_u=12, n_v=7) + p = np.stack([x, y, z], axis=1) + a, b, c = p[i], p[j], p[k] + area = 0.5 * np.linalg.norm(np.cross(b - a, c - a), axis=1) + assert area.min() > 1e-9, "degenerate face in the mesh" + # and each vertex is actually used, so nothing is left stranded + assert set(np.concatenate([i, j, k]).tolist()) == set(range(len(x))) + + +def test_the_wings_actually_move(): + """The flap is the animation; identical frames would be a still image.""" + up = fly3d._wing(1, fly3d.FLAP_DEGREES) + down = fly3d._wing(1, -fly3d.FLAP_DEGREES) + assert not np.allclose(up[2], down[2]) # z differs + # and the two wings are mirrored, not stacked on one side + left, right = fly3d._wing(1, 0.0), fly3d._wing(-1, 0.0) + assert left[1].mean() > 0 > right[1].mean() + + +def test_the_figure_animates_and_orbits(): + fig = fly3d.fly_figure(title="FLY.EXE", subtitle="XYZ:ORCL-USD") + assert len(fig.frames) == fly3d.FLAP_FRAMES + assert len(fig.data) >= 7 # abdomen, thorax, head, 2 eyes, legs, antennae, wings + # every frame retargets exactly the two wing traces + wing_traces = list(fig.frames[0].traces) + assert len(wing_traces) == 2 + assert all(list(f.traces) == wing_traces for f in fig.frames) + # a play button exists, and the scene is a 3D one (drag-to-orbit is native) + assert fig.layout.updatemenus and fig.layout.updatemenus[0].buttons + assert fig.layout.scene.camera.eye.x is not None + + +def test_numbers_are_formatted_or_visibly_absent(): + report = _module("fly_report") + assert report._fmt(None) == "—" + assert report._fmt(float("nan")) == "nan" + assert report._fmt(1234.5678, 2) == "1,234.57" + assert report._fmt(-0.5, 2, plus=True) == "-0.50" + assert report._fmt(0.5, 2, plus=True) == "+0.50" + assert report._clock(None) == "—" + assert ":" in report._clock(1789279456.0) + + +def test_the_pnl_curve_only_plots_reported_ticks(): + """An unreported tick has no P&L to plot; charting its stale figure would + draw a flat line that looks like a result.""" + report = _module("fly_report") + events = [ + {"tick": 1, "equity": 1.0, "pnl_known": True}, + {"tick": 2, "equity": 9.9, "pnl_known": False}, + {"tick": 3, "equity": 2.0, "pnl_known": True}, + ] + fig = report._pnl_figure(events) + assert list(fig.data[0].x) == [1, 3] and list(fig.data[0].y) == [1.0, 2.0] + assert report._pnl_figure(events[:1]) is None + + +def test_every_execution_status_has_a_word(): + report = _module("fly_report") + for status in ( + "APPLIED", + "SHADOW", + "HOLD", + "VETO", + "CLOSED", + "STOP_BOT", + "ERROR", + "HALT", + "TICK_ERROR", + ): + assert report.RESULT_WORDS[status] + + +def test_the_cloud_is_real_anatomy_and_always_holds_the_identified_circuit(): + """The graph carries no coordinates; these come from the release's soma + positions. The cells the decoder and the memory rule read are kept whole, + because a proportional sample would drop populations of four and six.""" + from flybrain.cloud import GROUPS, load + + cloud = load() + index, xyz, group = cloud["index"], cloud["xyz"], cloud["group"] + assert len(index) == len(xyz) == len(group) + assert np.isfinite(xyz).all() # every drawn point has a real position + assert len(np.unique(index)) == len(index) + assert index.max() < cloud["neurons_total"] + assert cloud["mapped_total"] < cloud["neurons_total"] # not every cell is mapped + # the tiny identified populations survive sampling + for name, expected in (("dopamine", 17), ("memory output", 6), ("readout", 4)): + assert int((group == GROUPS.index(name)).sum()) == expected + # and the silhouette is not swamped by one group + assert int((group == GROUPS.index("visual")).sum()) > 1000 + + +def test_orient_centres_and_normalizes_without_distorting(): + from flybrain.cloud import orient + + raw = np.array([[0, 0, 0], [100, 200, 300], [-100, -200, -300]], dtype=np.float32) + out = orient(raw) + assert np.abs(out).max() == pytest.approx(1.0) + assert np.allclose(out.mean(axis=0), 0, atol=1e-6) + # one scale for all axes: a per-axis scale would stretch the animal + spans = (raw.max(axis=0) - raw.min(axis=0))[[0, 2, 1]] + got = out.max(axis=0) - out.min(axis=0) + assert np.allclose(got / got[0], spans / spans[0], atol=1e-5) + + +def test_silent_neurons_are_drawn_apart_from_firing_ones(): + """Nineteen cells in twenty are silent; running them through the same + colour scale turns the anatomy into a haze that buries the live ones.""" + from flybrain.brainviz import brain_figure + from flybrain.cloud import load + + n = len(load()["index"]) + activity = [0] * n + activity[0] = 40 + activity[1] = 5 + fig = brain_figure(activity) + assert len(fig.data) == 2 + quiet, live = fig.data + assert len(quiet.x) == n - 2 and len(live.x) == 2 + assert quiet.marker.size < live.marker.size.min() # silent recede + assert live.marker.color.max() == pytest.approx(1.0) # hottest is the top colour + + # with no snapshot at all it falls back to colouring by cell class + classed = brain_figure(None) + assert len(classed.data) > 2 and classed.layout.showlegend diff --git a/agents/market_making_fly/tests/test_fly_review_fixes.py b/agents/market_making_fly/tests/test_fly_review_fixes.py new file mode 100644 index 000000000..2f7f313bb --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_review_fixes.py @@ -0,0 +1,252 @@ +"""Regressions for the first review round: nonfinite prices, vanished bots, +suffixed bot names, durable-first apply, bounded bench, validated run names.""" + +import asyncio +import math + +import pytest +from flybrain.guard import GuardSettings, Veto, check_price_move +from flybrain.market import Book, LiveMarket, book_restarted, pnl_is_known + + +class _Controllers: + def __init__(self, fail_saved=False, fail_live=False): + self.calls = [] + self.fail_saved = fail_saved + self.fail_live = fail_live + + async def create_or_update_controller_config(self, name, config): + assert config["id"] == name + self.calls.append(("saved", name)) + if self.fail_saved: + raise RuntimeError("saved failed") + + async def update_bot_controller_config(self, bot, name, config): + self.calls.append(("live", bot, name)) + if self.fail_live: + raise RuntimeError("live failed") + + +class _Orchestration: + def __init__(self, bots): + self._bots = bots + self.stopped = [] + + async def get_active_bots_status(self): + return {"data": self._bots} + + async def stop_and_archive_bot(self, name): + self.stopped.append(name) + + +class _Client: + def __init__(self, bots, **kw): + self.bot_orchestration = _Orchestration(bots) + self.controllers = _Controllers(**kw) + + +def _perf(net, volume): + return { + "performance": { + "realized_pnl_quote": net, + "unrealized_pnl_quote": 0, + "volume_traded": volume, + } + } + + +def _market(bots, **kw): + return LiveMarket(_Client(bots, **kw), "hyperliquid_perpetual", "5m", 72) + + +@pytest.mark.parametrize("bad", [float("nan"), float("inf"), 0.0, -1.0]) +def test_price_move_rejects_nonfinite(bad): + with pytest.raises(Veto): + check_price_move(100.0, bad, GuardSettings()) + with pytest.raises(Veto): + check_price_move(bad, 100.0, GuardSettings()) + + +def test_find_bot_accepts_deploy_suffix_and_refuses_ambiguity(): + bots = { + "xyz-orcl-usd-fly-20260913-055821": {}, + "xyz-dram-usd-fly": {}, + "xyz-orcl-usd-flyer": {}, + } + assert ( + LiveMarket.find_bot(bots, "xyz-orcl-usd-fly")[0] + == "xyz-orcl-usd-fly-20260913-055821" + ) + assert LiveMarket.find_bot(bots, "xyz-dram-usd-fly")[0] == "xyz-dram-usd-fly" + assert LiveMarket.find_bot(bots, "xyz-spcx-usd-fly") == (None, None) + with pytest.raises(RuntimeError): + LiveMarket.find_bot({**bots, "xyz-orcl-usd-fly": {}}, "xyz-orcl-usd-fly") + + +def test_equity_carries_a_vanished_bot(): + running = { + "xyz-orcl-usd-fly-20260913-055821": { + "performance": {"xyz_orcl_usd_fly_mm": _perf(-1.5, 400)} + } + } + net, volume, per_pair, carry = asyncio.run( + _market(running).equity(["XYZ:ORCL-USD"]) + ) + assert (net, volume) == (-1.5, 400) and per_pair["XYZ:ORCL-USD"]["reported"] + # bot disappears from the status response: its last figures stay in the book + net2, volume2, per_pair2, carry2 = asyncio.run( + _market({}).equity(["XYZ:ORCL-USD"], carry) + ) + assert (net2, volume2) == (-1.5, 400) + assert per_pair2["XYZ:ORCL-USD"] == { + "running": False, + "carried": True, + "net": -1.5, + "volume": 400, + } + assert carry2 == carry + # a bot that never reported contributes nothing + net3, _, per_pair3, _ = asyncio.run(_market({}).equity(["XYZ:DRAM-USD"])) + assert net3 == 0 and per_pair3["XYZ:DRAM-USD"] == {"running": False} + + +def test_equity_marks_a_running_bot_without_a_report(): + unreported = {"xyz-orcl-usd-fly-20260913-055821": {"performance": {}}} + net, volume, per_pair, carry = asyncio.run( + _market(unreported).equity( + ["XYZ:ORCL-USD"], {"XYZ:ORCL-USD": {"net": 2.0, "volume": 50}} + ) + ) + assert (net, volume) == (2.0, 50) + assert per_pair["XYZ:ORCL-USD"] == { + "running": True, + "reported": False, + "net": 2.0, + "volume": 50, + } + assert carry == {"XYZ:ORCL-USD": {"net": 2.0, "volume": 50}} + _, _, fresh, _ = asyncio.run(_market(unreported).equity(["XYZ:ORCL-USD"])) + assert fresh["XYZ:ORCL-USD"] == {"running": True, "reported": False} + + +def test_apply_saves_before_touching_the_live_bot(): + bots = {"xyz-orcl-usd-fly-20260913-055821": {}} + m = _market(bots) + asyncio.run(m.apply("XYZ:ORCL-USD", {"x": 1})) + assert m.client.controllers.calls == [ + ("saved", "xyz_orcl_usd_fly_mm"), + ("live", "xyz-orcl-usd-fly-20260913-055821", "xyz_orcl_usd_fly_mm"), + ] + failing = _market(bots, fail_saved=True) + with pytest.raises(RuntimeError): + asyncio.run(failing.apply("XYZ:ORCL-USD", {"x": 1})) + assert failing.client.controllers.calls == [("saved", "xyz_orcl_usd_fly_mm")] + with pytest.raises(RuntimeError): + asyncio.run(_market({}).apply("XYZ:ORCL-USD", {"x": 1})) + + +def test_stop_bot_uses_the_running_name(): + m = _market({"xyz-orcl-usd-fly-20260913-055821": {}}) + assert asyncio.run(m.stop_bot("XYZ:ORCL-USD")) is True + assert m.client.bot_orchestration.stopped == ["xyz-orcl-usd-fly-20260913-055821"] + assert asyncio.run(_market({}).stop_bot("XYZ:ORCL-USD")) is False + + +def test_book_requires_finite_positive_prices(): + assert Book(1.0, 1.1).open and not Book(None, None).open + assert not math.isfinite(float("nan")) + + +def test_setup_bench_bounds(): + import importlib.util + from pathlib import Path + + path = Path(__file__).resolve().parents[1] / "routines" / "fly_setup.py" + spec = importlib.util.spec_from_file_location("fly_setup", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + assert mod.Config(observations=20, neural_ms=200).observations == 20 + with pytest.raises(ValueError): + mod.Config(observations=0) + with pytest.raises(ValueError): + mod.Config(observations=21) + with pytest.raises(ValueError): + mod.Config(neural_ms=10) + + +def test_status_rejects_escaping_run_name(): + import importlib.util + from pathlib import Path + + from condor.paths import UnsafeIdError + + path = Path(__file__).resolve().parents[1] / "routines" / "fly_status.py" + spec = importlib.util.spec_from_file_location("fly_status", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + + class Ctx: + _chat_id = 1 + + for bad in ("../../etc", "/tmp/x", "a/b"): + with pytest.raises(UnsafeIdError): + asyncio.run(mod.run(mod.Config(run_name=bad), Ctx())) + + +def test_a_redeployed_book_is_not_a_five_dollar_loss(): + """A fresh controller instance reports from zero. Without the restart + check that reads as the whole previous deployment's P&L evaporating: a + large false aversive pulse, and a stale high the new book is measured + against.""" + from flybrain.guard import GuardSettings, GuardState, Halt, check_pnl, rebase + from flybrain.market import book_restarted + + traded = { + "xyz-orcl-usd-fly-20260913-055821": { + "performance": {"xyz_orcl_usd_fly_mm": _perf(5.0, 7000.0)} + } + } + net, _, per_pair, carry = asyncio.run(_market(traded).equity(["XYZ:ORCL-USD"])) + assert net == 5.0 and not book_restarted(per_pair) + + guard, settings = GuardState(), GuardSettings(loss_no_new_high_ticks=3) + check_pnl(net, 7000.0, guard, settings, 100.0) + assert guard.session_high_net == 5.0 + + # the operator redeploys; the new instance reports from zero + fresh = { + "xyz-orcl-usd-fly-20260913-071220": { + "performance": {"xyz_orcl_usd_fly_mm": _perf(0.0, 0.0)} + } + } + net2, _, per_pair2, _ = asyncio.run(_market(fresh).equity(["XYZ:ORCL-USD"], carry)) + assert book_restarted(per_pair2) == ["XYZ:ORCL-USD"] + + # rebased, the fresh book sets its own high instead of counting down to a halt + rebase(guard) + for _ in range(5): + check_pnl(net2, 0.0, guard, settings, 100.0) + assert guard.session_high_net == 0.0 + + # without the rebase it would have halted against the old deployment's high + stale = GuardState(session_high_net=5.0) + with pytest.raises(Halt): + for _ in range(4): + check_pnl(0.0, 0.0, stale, settings, 100.0) + + +def test_growing_volume_is_not_a_restart(): + traded = { + "xyz-orcl-usd-fly-20260913-055821": { + "performance": {"xyz_orcl_usd_fly_mm": _perf(5.0, 7000.0)} + } + } + _, _, _, carry = asyncio.run(_market(traded).equity(["XYZ:ORCL-USD"])) + more = { + "xyz-orcl-usd-fly-20260913-055821": { + "performance": {"xyz_orcl_usd_fly_mm": _perf(-2.0, 9000.0)} + } + } + _, _, per_pair, _ = asyncio.run(_market(more).equity(["XYZ:ORCL-USD"], carry)) + # P&L fell hard, but volume grew: a real loss, and it must still pulse + assert not book_restarted(per_pair) and pnl_is_known(per_pair) diff --git a/agents/market_making_fly/tests/test_fly_run_state.py b/agents/market_making_fly/tests/test_fly_run_state.py new file mode 100644 index 000000000..c92b04534 --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_run_state.py @@ -0,0 +1,123 @@ +"""Run directory: lock, state, events, checkpoint slots, provenance refusal.""" + +import json + +import numpy as np +import pytest +from flybrain.run_state import RunDir, signature, source_hashes + + +def test_lock_is_exclusive(tmp_path): + a = RunDir(tmp_path / "run") + a.lock() + b = RunDir(tmp_path / "run") + with pytest.raises(RuntimeError): + b.lock() + a.unlock() + b.lock() + b.unlock() + + +def test_state_events_latest_frame(tmp_path): + run = RunDir(tmp_path / "run") + assert run.load_state() == {} + run.save_state({"tick": 3, "anchor": 1.5}) + assert run.load_state() == {"tick": 3, "anchor": 1.5} + for i in range(5): + run.append_event({"tick": i}) + assert [e["tick"] for e in run.recent_events(3)] == [2, 3, 4] + run.write_latest({"tick": 4}) + assert json.loads(run.latest_path.read_text()) == {"tick": 4} + run.save_frame(np.zeros((180, 320, 3), np.uint8)) + assert run.frame_path.exists() + assert not run.stop_requested() + run.stop_path.touch() + assert run.stop_requested() + + +def test_checkpoint_slots_and_integrity(tmp_path): + run = RunDir(tmp_path / "run") + assert run.checkpoint_path(4).name == "brain-0.npz" + assert run.checkpoint_path(5).name == "brain-1.npz" + path = run.checkpoint_path(1) + path.write_bytes(b"brain") + import hashlib + + good = {"file": path.name, "sha256": hashlib.sha256(b"brain").hexdigest()} + assert run.verify_checkpoint(good) == path + with pytest.raises(RuntimeError): + run.verify_checkpoint({"file": path.name, "sha256": "0" * 64}) + with pytest.raises(RuntimeError): + run.verify_checkpoint({"file": "missing.npz", "sha256": "0" * 64}) + + +def test_provenance_refuses_changed_protocol(tmp_path): + run = RunDir(tmp_path / "run") + prov = {"decoder": {"window": 60}, "source_sha256": {"a.py": "1"}} + sig = run.check_provenance(prov) + assert sig == signature(prov) + assert run.check_provenance(dict(prov)) == sig + with pytest.raises(RuntimeError): + run.check_provenance({**prov, "decoder": {"window": 30}}) + # source hashes are recorded but a code change does not refuse the resume + assert run.check_provenance({**prov, "source_sha256": {"a.py": "2"}}) == sig + # a run recorded under an older signing rule is re-signed, not refused + stale = json.loads(run.provenance_path.read_text()) + stale["signature"] = "0" * 64 + run.provenance_path.write_text(json.dumps(stale)) + assert run.check_provenance(prov) == sig + + +def test_cadence_may_change_but_the_books_own_terms_may_not(tmp_path): + """A resume restores the anchor, the session high and the carried P&L. + Anything that denominates those must be signed, or a run judges one + deployment's numbers by another's thresholds.""" + run = RunDir(tmp_path / "run") + base = { + "settings": { + "neural_ms": 500, + "interval_sec": 60, + "total_amount_quote": 200, + "leverage": 3, + "portfolio_allocation": 0.2, + "connector_name": "hyperliquid_perpetual", + } + } + sig = run.check_provenance(base) + # cadence moves no money and reinterprets no accounting + slower = {"settings": {**base["settings"], "interval_sec": 300}} + assert run.check_provenance(slower) == sig + for changed in ( + {"total_amount_quote": 1000}, + {"leverage": 5}, + {"portfolio_allocation": 1.0}, + {"connector_name": "binance_perpetual"}, + {"neural_ms": 700}, + ): + with pytest.raises(RuntimeError): + run.check_provenance({"settings": {**base["settings"], **changed}}) + + +def test_state_version_is_signed(tmp_path): + """A persisted-state change must orphan the run rather than let old state + be reinterpreted: session_high_net 0.0 -> None already proved why.""" + import flybrain.run_state as run_state + + run = RunDir(tmp_path / "run") + prov = {"settings": {"neural_ms": 500}} + sig = run.check_provenance(prov) + recorded = json.loads(run.provenance_path.read_text()) + assert recorded["state_version"] == run_state.STATE_VERSION + original = run_state.STATE_VERSION + try: + run_state.STATE_VERSION = original + 1 + with pytest.raises(RuntimeError): + run.check_provenance(prov) + finally: + run_state.STATE_VERSION = original + assert run.check_provenance(prov) == sig + + +def test_source_hashes_cover_the_package(): + hashes = source_hashes() + assert "decoder.py" in hashes and "neural/kernel.cpp" in hashes diff --git a/agents/market_making_fly/tests/test_fly_venue.py b/agents/market_making_fly/tests/test_fly_venue.py new file mode 100644 index 000000000..474e74f60 --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_venue.py @@ -0,0 +1,119 @@ +"""Spot or perp, and what a round trip costs there.""" + +import asyncio + +import pytest +from flybrain import venue + + +@pytest.mark.parametrize( + "connector,expected", + [ + ("binance_perpetual", "perp"), + ("hyperliquid_perpetual", "perp"), + ("gate_io_perpetual", "perp"), + ("binance", "spot"), + ("kucoin", "spot"), + ("backpack", "spot"), + ], +) +def test_market_type_comes_from_the_connector(connector, expected): + assert venue.market_type_for(connector) == expected + + +def test_longest_prefix_wins(): + """binance_perpetual is not binance, and their fees differ by 3.75x.""" + assert venue.default_maker_fee_bps("binance_perpetual", "perp") == 2.0 + assert venue.default_maker_fee_bps("binance", "spot") == 7.5 + + +def test_an_unknown_venue_is_quoted_wide_not_tight(): + """A fee floor set too low loses money silently; too high only costs fills.""" + assert venue.default_maker_fee_bps("brand_new_dex", "spot") == 10.0 + assert venue.default_maker_fee_bps("brand_new_dex_perpetual", "perp") == 2.5 + + +def test_resolve_refuses_a_contradiction(): + assert venue.resolve("binance") == "spot" + assert venue.resolve("binance_perpetual") == "perp" + assert venue.resolve("binance", "spot") == "spot" + with pytest.raises(ValueError): + venue.resolve("binance", "perp") + with pytest.raises(ValueError): + venue.resolve("binance", "margin") + with pytest.raises(ValueError): + venue.market_type_for("") + + +def _seed_hyperliquid_fees(): + """The two Hyperliquid payloads the fee comes from, as the venue returns + them. Seeding the cache keeps the arithmetic under test and the network + out of it.""" + venue._hl_cache.clear() + venue._hl_cache[ + repr( + sorted({"type": "userFees", "user": venue._SCHEDULE_PROBE_ADDRESS}.items()) + ) + ] = {"feeSchedule": {"add": "0.00015", "cross": "0.00045", "spotAdd": "0.0004"}} + venue._hl_cache[repr(sorted({"type": "meta", "dex": "xyz"}.items()))] = { + "universe": [ + {"name": "xyz:ORCL", "deployerFeeScale": "1.0", "growthMode": "enabled"}, + {"name": "xyz:NOGROWTH", "deployerFeeScale": "1.0"}, + {"name": "xyz:HALFSCALE", "deployerFeeScale": "0.5"}, + ] + } + + +def test_one_hyperliquid_connector_charges_two_different_fees(): + """A core perp and a HIP-3 market differ by about 2x, and the floor built + on the wrong one either loses money or never fills. Measured against the + run of 2026-09-13: 173 maker fills on XYZ:ORCL-USD paid 1.29 bp all-in.""" + _seed_hyperliquid_fees() + core = asyncio.run(venue.hyperliquid_maker_fee_bps("BTC-USD", "perp")) + hip3 = asyncio.run(venue.hyperliquid_maker_fee_bps("XYZ:ORCL-USD", "perp")) + assert core == pytest.approx(1.5 + venue.HUMMINGBOT_BUILDER_FEE_BPS) + # scale 1.0 doubles the venue rate, growth mode takes a tenth of that + assert hip3 == pytest.approx(1.5 * 2 * 0.1 + venue.HUMMINGBOT_BUILDER_FEE_BPS) + assert abs(hip3 - 1.29) < 0.02, "computed fee has drifted from what was paid" + + +def test_the_deployers_own_setting_moves_the_fee_both_ways(): + """Hyperliquid's rule is not a discount: without growth mode a HIP-3 + market costs more than the core venue, not less.""" + _seed_hyperliquid_fees() + dear = asyncio.run(venue.hyperliquid_maker_fee_bps("XYZ:NOGROWTH-USD", "perp")) + half = asyncio.run(venue.hyperliquid_maker_fee_bps("XYZ:HALFSCALE-USD", "perp")) + assert dear == pytest.approx(1.5 * 2 + venue.HUMMINGBOT_BUILDER_FEE_BPS) + assert half == pytest.approx(1.5 * 1.5 + venue.HUMMINGBOT_BUILDER_FEE_BPS) + assert dear > asyncio.run(venue.hyperliquid_maker_fee_bps("BTC-USD", "perp")) + + +def test_an_unlisted_hip3_market_is_refused_not_guessed(): + _seed_hyperliquid_fees() + with pytest.raises(ValueError, match="not listed"): + asyncio.run(venue.hyperliquid_maker_fee_bps("XYZ:NOTREAL-USD", "perp")) + + +def test_the_fly_says_what_to_install_rather_than_failing_deep(): + """pyarrow is 122 MB that only this agent uses, so it is an extra. An + operator who skipped it should get the command, not an ImportError three + frames inside a vendored connectome loader.""" + import builtins + + from flybrain.deps import INSTALL, require_pyarrow + + real_import = builtins.__import__ + + def without_pyarrow(name, *args, **kwargs): + if name == "pyarrow" or name.startswith("pyarrow."): + raise ImportError("No module named 'pyarrow'") + return real_import(name, *args, **kwargs) + + builtins.__import__ = without_pyarrow + try: + with pytest.raises(RuntimeError, match=INSTALL): + require_pyarrow() + finally: + builtins.__import__ = real_import + + require_pyarrow() # installed here, so it must stay silent diff --git a/condor/reports/builder.py b/condor/reports/builder.py index f4bc2388c..3580ce444 100644 --- a/condor/reports/builder.py +++ b/condor/reports/builder.py @@ -88,8 +88,22 @@ def auto_refresh(self, seconds: int | None) -> ReportBuilder: return self def kpi( - self, label: str, value: str, delta: str | None = None, trend: str = "neutral" + self, + label: str, + value: str, + delta: str | None = None, + trend: str = "neutral", + width: int = 12, ) -> ReportBuilder: + """Add a KPI card. + + ``width`` is the span of the report's 12-column grid taken by the whole + run of consecutive cards, the first card's value winning. The default + spans the row as before; a narrower one lets the cards sit beside a + figure and, below the layout's 800px breakpoint, stack under it. The + cards keep their own auto-fitting grid inside that span, so a narrow + run simply wraps to fewer per row. + """ self._sections.append( { "type": "kpi", @@ -97,6 +111,7 @@ def kpi( "value": value, "delta": delta, "trend": trend, + "width": max(1, min(12, int(width))), } ) return self @@ -118,9 +133,15 @@ def section(self, title: str, description: str | None = None) -> ReportBuilder: ) return self - def plotly(self, fig: Any, optimize: bool = True) -> ReportBuilder: + def plotly(self, fig: Any, optimize: bool = True, width: int = 12) -> ReportBuilder: """Attach a Plotly figure. + ``width`` is the figure's span of the report's 12-column grid, the same + knob every data-bound component already takes. The default spans the + row, exactly as before. A narrower one lets two figures sit side by + side and, below the layout's 800px breakpoint, stack on their own — + which a single figure split internally cannot do. + Large figures are re-encoded for display (see :mod:`condor.reports.figure_opt`): evenly spaced x arrays collapse to ``x0``/``dx`` and dense traces move to the WebGL renderer, which keeps a @@ -139,7 +160,13 @@ def plotly(self, fig: Any, optimize: bool = True) -> ReportBuilder: ) else: content = fig.to_html(full_html=False, include_plotlyjs=False) - self._sections.append({"type": "plotly", "content": content}) + self._sections.append( + { + "type": "plotly", + "content": content, + "width": max(1, min(12, int(width))), + } + ) return self def table( @@ -565,7 +592,18 @@ def _render_sections(self) -> str: f'
{html.escape(str(kpi["value"]))}
' f"{delta_html}" ) - parts.append(f'
{"".join(cards)}
') + # A full-width run keeps the markup it has always had. A + # narrower one is wrapped in a grid item rather than given the + # span itself: `.report-grid > .kpi-bar` would beat it, and the + # inner bar still needs its own card grid. + span = kpis[0].get("width", 12) + bar = f'
{"".join(cards)}
' + parts.append( + bar + if span >= 12 + else f'
{bar}
' + ) elif section["type"] == "markdown": parts.append( '
' @@ -573,9 +611,26 @@ def _render_sections(self) -> str: ) index += 1 elif section["type"] == "plotly": - parts.append( - f'
{section["content"]}
' - ) + # A full-width figure keeps the plain section markup it has + # always had. A narrower one becomes a grid item instead, so it + # spans its columns and collapses to full width on a narrow + # screen; `.report-grid > .section` would otherwise force it + # back to the whole row. + span = section.get("width", 12) + if span < 12: + # `min-height: 400px` keeps a full-width chart from being + # squashed, but a narrow panel was placed deliberately + # beside something else and has to be free to match it. + # Inline, so it overrides the stylesheet for this panel only. + parts.append( + f'
' + f'{section["content"]}
' + ) + else: + parts.append( + f'
{section["content"]}
' + ) index += 1 elif section["type"] == "table": parts.append(self._render_table(section["columns"], section["rows"])) diff --git a/pyproject.toml b/pyproject.toml index aec2c149b..b56e6c5f4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -39,6 +39,15 @@ dependencies = [ "pywebpush>=2.5.0", ] +[project.optional-dependencies] +# Market Making Fly reads the MaleCNS release from Arrow feather files. pyarrow +# is 122 MB installed and nothing else in Condor touches it, so it is an extra +# rather than a core dependency: `uv sync --extra fly`. Every import of it in +# the agent is inside a function, and the two entry points that reach one check +# first and say this — so a Condor without the extra runs normally and the fly +# refuses to start with an instruction rather than an ImportError. +fly = ["pyarrow"] + [dependency-groups] # PEP 735 group: this is what ``uv sync --dev`` installs. Declared as an # optional-dependencies extra it was invisible to ``--dev``, so CI ran diff --git a/tests/test_report_builder.py b/tests/test_report_builder.py index d82379ed7..76b0344e0 100644 --- a/tests/test_report_builder.py +++ b/tests/test_report_builder.py @@ -409,3 +409,54 @@ def test_hydrate_is_a_no_op_for_a_pre_existing_inlined_report(reports_dir): """The 100 reports already on disk need no migration to keep working.""" legacy = "" assert rendering.hydrate(legacy) == legacy + + +def test_plotly_width_spans_the_grid_and_defaults_to_the_full_row(): + """A narrow figure has to leave the plain `.section` markup behind: + `.report-grid > .section` forces `grid-column: 1 / -1`, which would beat + `.report-component`'s span and silently keep the figure full width.""" + import plotly.graph_objects as go + + from condor.reports import ReportBuilder + + def html_for(**kwargs): + builder = ReportBuilder("t") + builder.plotly(go.Figure(go.Scatter(x=[1, 2], y=[1, 2])), **kwargs) + return builder._render_sections() + + full = html_for() + assert 'class="section plotly-chart report-panel"' in full + assert "--component-span" not in full + + half = html_for(width=6) + assert "--component-span:6" in half + # the stylesheet's 400px floor would stop a narrow panel matching whatever + # it sits beside, so it is released inline for that panel only + assert "min-height:0" in half and "min-height:0" not in full + assert 'class="section plotly-chart' not in half # or the grid rule wins + assert "report-component" in half + + # out-of-range widths are clamped rather than emitted as broken CSS + assert "--component-span" not in html_for(width=99) # clamps to the full row + assert "--component-span:1" in html_for(width=0) + + +def test_kpi_width_wraps_the_bar_so_the_grid_rule_cannot_override_it(): + """`.report-grid > .kpi-bar` forces `grid-column: 1 / -1`, so a narrow run + has to be wrapped rather than given the span itself — and the inner bar + still needs its own card grid.""" + from condor.reports import ReportBuilder + + def html_for(**kwargs): + builder = ReportBuilder("t") + builder.kpi("A", "1", **kwargs) + builder.kpi("B", "2", **kwargs) + return builder._render_sections() + + full = html_for() + assert '
' in full + assert "--component-span" not in full + + half = html_for(width=5) + assert '--component-span:5">
' in half + assert half.count('class="kpi-bar"') == 1 # one bar, wrapped once diff --git a/uv.lock b/uv.lock index e41d6e639..69e8a6d75 100644 --- a/uv.lock +++ b/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 3 +revision = 2 requires-python = ">=3.12" resolution-markers = [ "python_full_version >= '3.14' and sys_platform == 'win32'", @@ -626,6 +626,11 @@ dependencies = [ { name = "yfinance" }, ] +[package.optional-dependencies] +fly = [ + { name = "pyarrow" }, +] + [package.dev-dependencies] dev = [ { name = "black" }, @@ -651,6 +656,7 @@ requires-dist = [ { name = "pandas" }, { name = "pandas-ta", specifier = ">=0.4.71b0" }, { name = "plotly" }, + { name = "pyarrow", marker = "extra == 'fly'" }, { name = "pydantic-ai", extras = ["mcp"] }, { name = "python-dotenv" }, { name = "python-jose", extras = ["cryptography"] }, @@ -662,6 +668,7 @@ requires-dist = [ { name = "watchfiles" }, { name = "yfinance", specifier = ">=1.7.0" }, ] +provides-extras = ["fly"] [package.metadata.requires-dev] dev = [ @@ -3086,6 +3093,42 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7f/15/f9d0171e1ad863ca49e826d5afb6b50566f20dc9b4f76965096d3555ce9e/py_vapid-1.9.4-py2.py3-none-any.whl", hash = "sha256:f165a5bf90dcf966b226114f01f178f137579a09784c7f0628fa2f0a299741b6", size = 23912, upload-time = "2026-01-05T20:42:05.455Z" }, ] +[[package]] +name = "pyarrow" +version = "25.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/3d/e3/27f57f80141379d60defe6703eb50a707325706f07fedfd1312c7a751995/pyarrow-25.0.1.tar.gz", hash = "sha256:9150a83248bfed9813ea3c3af74c3856c1984d444aa28e58bf7733b9750ddf6a", size = 1201653, upload-time = "2026-08-10T12:40:53.904Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a6/e2/9ab15b88cbfac28e16419ce5439ec29234c5172cb8259301b4ba639bdec0/pyarrow-25.0.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:df961f2e7ae9cf496459259d798652c70625f6c080650d6952f8c04053c58ee9", size = 35861559, upload-time = "2026-08-10T12:38:02.567Z" }, + { url = "https://files.pythonhosted.org/packages/58/79/a0036dbe1eabe1f73127427342f1d99982584c4a2cde2651d6c93499c6f6/pyarrow-25.0.1-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:cc4aa407fde9fc660be3939e49ea31f50f3e9fec17c0ec63159f7711edd3efc9", size = 37628383, upload-time = "2026-08-10T12:38:09.083Z" }, + { url = "https://files.pythonhosted.org/packages/13/49/d93a57d375f4bf0cf82913dd6bb54acafde83dd993be2282c81ac5616cad/pyarrow-25.0.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:4340f0ba6c1d2e13f21658de1d7c662ca2545018568d0030a1e9afca159d87e3", size = 46820190, upload-time = "2026-08-10T12:38:15.458Z" }, + { url = "https://files.pythonhosted.org/packages/60/c9/711ca85d79f1ec98f29a5eae2b051e25b4ecec5de3e3c0e2d5c5dcb15664/pyarrow-25.0.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:5389cdf79447ed1515c9e31620e6e1e2302249564d603f2ad727d4f6d313e4c3", size = 50102437, upload-time = "2026-08-10T12:38:22.487Z" }, + { url = "https://files.pythonhosted.org/packages/80/53/8fb8359ff17cfb6263a1cf3ebf7caec9fe197de118719e84fcb1d0618026/pyarrow-25.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:d51592cb7561e87877c506113e7adbf1342ab579e6c21f0ef44b8ba41cb74c80", size = 49942424, upload-time = "2026-08-10T12:38:28.755Z" }, + { url = "https://files.pythonhosted.org/packages/e8/83/4e5ae02a9341571b18a6fca380ac7a58ce6ddae7ab3c060208c0a1e79f02/pyarrow-25.0.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6109c94d8b9f3b17a041daca16cacb2f651ad8f1ef70a4232c2c0f37a23da2a8", size = 53144206, upload-time = "2026-08-10T12:38:34.862Z" }, + { url = "https://files.pythonhosted.org/packages/65/ee/197cbf47e49f83e6ebeb946a5259a48a638dea27ac774db42fe78022179d/pyarrow-25.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:8858d7bfc22e3f51529aeaa4077225029724623e4595dc9eff8c793935c34140", size = 27953934, upload-time = "2026-08-10T12:38:39.808Z" }, + { url = "https://files.pythonhosted.org/packages/cc/8d/8f271a7a034c834910ec925d56fa4b29733b1380f5289419f5aaa3b02777/pyarrow-25.0.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:c7c534ec03c358a76ea3e505e74c1b6aef290af90c444dfd092dbfe23e755b85", size = 35855328, upload-time = "2026-08-10T12:38:45.489Z" }, + { url = "https://files.pythonhosted.org/packages/d2/cd/5bac242f4e841b9971d5eb94fdfe2577e2b70be983e27401e72055786037/pyarrow-25.0.1-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:dda9470024204d7bbf2042b47c6e8a0e47a3eeb8e34405882dfaea6577e0c153", size = 37622415, upload-time = "2026-08-10T12:38:51.107Z" }, + { url = "https://files.pythonhosted.org/packages/63/1f/96d03b4e1506524f7087adb0fd6b2f69f0c9c7aaff1ec36d8030082e15a5/pyarrow-25.0.1-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:44a9120ce5bd81936b8ab9a88076e3fd47c2c6838e0e43630fed83626aca81d9", size = 46813813, upload-time = "2026-08-10T12:38:57.773Z" }, + { url = "https://files.pythonhosted.org/packages/98/d6/33a411115b61dbfc16ad6ad73e71730f6fea654ee3667673bc53ab0e2fe7/pyarrow-25.0.1-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:0befcf816e45a1af33ac775a9970b749e4868a230c7372f0ae5e932bee27039f", size = 50104452, upload-time = "2026-08-10T12:39:04.579Z" }, + { url = "https://files.pythonhosted.org/packages/33/ae/b1b97c9ca87f9f9ddbb5230c798df94eccce61bd79b9b45458c69a478588/pyarrow-25.0.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3f89685964f46e4216103c75483aac0c0692a5f72212d7ca835adba5ede56ce3", size = 49951343, upload-time = "2026-08-10T12:39:11.8Z" }, + { url = "https://files.pythonhosted.org/packages/98/9e/a112df5cfd5a68cb1d9fc31cfe38c28d5aec9f10865ce37ecef2e4450873/pyarrow-25.0.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6943e2fe7954d29d84de45d29d34c8dc36ce96570e67d89aa9976e650a4a9138", size = 53144784, upload-time = "2026-08-10T12:39:20.503Z" }, + { url = "https://files.pythonhosted.org/packages/31/24/97e8bd98f1e3b07e2ba08bcdff690674fbe16d69a7d2712cc3884665e615/pyarrow-25.0.1-cp313-cp313-win_amd64.whl", hash = "sha256:31e49a7888fcdf3a835da33ae777f6bb9a866334e5a789282fc26dcf426f7f15", size = 27870159, upload-time = "2026-08-10T12:39:26.161Z" }, + { url = "https://files.pythonhosted.org/packages/36/4c/b525824ad3094076919273cd97db61fb3d78252dee76fa3b8dc8f76774aa/pyarrow-25.0.1-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:bf0b672390cdcb640d7288f96b826d71ff4e9abb254a86c89890baf51a29cee6", size = 35885255, upload-time = "2026-08-10T12:39:32.366Z" }, + { url = "https://files.pythonhosted.org/packages/08/62/448bb0e940de41aec31d1a956e63ad9c54afdf122a103cc3ab20c2a3ce33/pyarrow-25.0.1-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:38a9a4b4b9613380e200641891495a56c3d5a98a092db4a870af9975e220471d", size = 37644461, upload-time = "2026-08-10T12:39:38.142Z" }, + { url = "https://files.pythonhosted.org/packages/6e/9a/13587e38bd4806fd218f50fd13b8903fab60588a699ff0c406372e5b4043/pyarrow-25.0.1-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:0b726ad7e7b669be982b0c71c07fe4b037d654354130da79a7902a669e93a66b", size = 46877146, upload-time = "2026-08-10T12:39:43.722Z" }, + { url = "https://files.pythonhosted.org/packages/8d/61/1c5d1229fa21da4cff5365e41e57177aaac57c563c727f35419b8513d1c1/pyarrow-25.0.1-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:9171748cdf796972d85a4b60157c279913e242992e350c90c7450182a9838b2a", size = 50131616, upload-time = "2026-08-10T12:39:49.304Z" }, + { url = "https://files.pythonhosted.org/packages/43/20/291e1d65cc0b09aa19f03cf25cf51a2f5fa94b5db315178f2d254ed5cad4/pyarrow-25.0.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:b7a296aac7a71fa0886c08e155ddb6c636a50013f801f6178daafa0f9e726188", size = 50008879, upload-time = "2026-08-10T12:39:56.891Z" }, + { url = "https://files.pythonhosted.org/packages/8b/7c/1b7c9ec28e76576337e4f97b31141c9a181b89b6d1d6221e9d8205621a58/pyarrow-25.0.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:0fe7c8b6c03969b49c8c66182e4a18e3819ab92d07cfab5d8370c531b9369ef0", size = 53170864, upload-time = "2026-08-10T12:40:04.918Z" }, + { url = "https://files.pythonhosted.org/packages/b7/75/f3d789dc06011a765d14d86bda799cf72ac1d715b6a6edecaa0d73d95062/pyarrow-25.0.1-cp314-cp314-win_amd64.whl", hash = "sha256:f729cfdbd36fd99d543b67a914d2de044c84ebe45be8b34902b299b608c15c8f", size = 28620729, upload-time = "2026-08-10T12:40:51.41Z" }, + { url = "https://files.pythonhosted.org/packages/fc/05/647a8ee6f7c2662feb6921315617bc04dcd6034763fb61b1199720bf6162/pyarrow-25.0.1-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:59a2de54c0cbd954da861eee4d1d330f8e909c45b53455baef696380f2c55033", size = 36130288, upload-time = "2026-08-10T12:40:11.014Z" }, + { url = "https://files.pythonhosted.org/packages/93/f8/c9ee997554d7bea94520667dd1933f109ac1da3ee3556d2b49381e023484/pyarrow-25.0.1-cp314-cp314t-macosx_12_0_x86_64.whl", hash = "sha256:35935cd5de130aa5cf4dea052a63e6bf2e17006c35c3a468194242b9b2bf5956", size = 37762187, upload-time = "2026-08-10T12:40:16.592Z" }, + { url = "https://files.pythonhosted.org/packages/a2/08/a28c01c7fe9e96e8233ce2d13df1d402f4f999f848f51d2daacd6bb4c036/pyarrow-25.0.1-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:f3831aaa25c67a99f99dc8b05873cb9d64560390372e2aa197ce9dd4a3f06a44", size = 46888003, upload-time = "2026-08-10T12:40:23.242Z" }, + { url = "https://files.pythonhosted.org/packages/1b/b9/58612e977d28dc58c878448866838369ee8da2f1e7cc8ed2c84b952aafee/pyarrow-25.0.1-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:6a1fdfc6659b6b19022f2e50627fb5cf7156a66c46bf4299379955cbe742382a", size = 50079036, upload-time = "2026-08-10T12:40:29.169Z" }, + { url = "https://files.pythonhosted.org/packages/72/13/66e1402dcc860e1dc2760b1e0292c9a569b62b3bccab69def1b3e907d006/pyarrow-25.0.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:169d3429d5be7c752125890620f75a60776d38b0035eddae939651640822332e", size = 50040226, upload-time = "2026-08-10T12:40:35.186Z" }, + { url = "https://files.pythonhosted.org/packages/78/10/3f1a5497a7ef732ab0f03ecca3e66d89d9c0f57fdc61b4794c456b781f01/pyarrow-25.0.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:119297a6dc197e45d9c6d4415f7814a67ffa36c180d26f68c154c58067ae782d", size = 53149035, upload-time = "2026-08-10T12:40:41.454Z" }, + { url = "https://files.pythonhosted.org/packages/93/c0/37d4a7e8e2f7a6076283673d5298018ca26478b934c6ee369e10505ab32c/pyarrow-25.0.1-cp314-cp314t-win_amd64.whl", hash = "sha256:4288f27577352d608ca08553b0865e4a9b3aa14820c5d95b53337218d609835b", size = 28753071, upload-time = "2026-08-10T12:40:46.623Z" }, +] + [[package]] name = "pyasn1" version = "0.6.3"