From c11f6b238a19978be26047005f0f0f726094faa7 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sat, 12 Sep 2026 18:03:19 -0700 Subject: [PATCH 01/48] (feat) a fly connectome decides the market maker's posture on HIP-3 Market Making Fly: a Market Making Expert sibling whose regime, spread width and reference-price lean are decoded from a spiking simulation of the MaleCNS v1.0 fly connectome (vendored from nftechie/stonkfly, MIT) watching a 320x180 OHLCV chart. The combined P&L of its pmm_mister bots is pulsed into the PAM11 and PPL101 dopamine cells and stonkfly's KC->MBON memory rule runs unchanged. One shared brain observes up to three Hyperliquid HIP-3 pairs in round-robin. A fixed mapping turns the decoded posture into a pmm_mister config with the fee floor, spread floor and HIP-3 bounds in code; a guard can veto or halt and never substitutes a posture; the LLM agent only deploys, starts, stops, rotates and explains. condor/fly: vendored neural package + data prepare/verify/bench CLI (python -m condor.fly ...), chart, decoder, posture, guard, reinforcement, market adapters, run directory state, worker process. agents/market_making_fly: AGENT.md, fly_brain (continuous), fly_chart, fly_status, copied HIP-3 scanner and dashboard, fly_mm_deploy and fly_decoder skills, fly_hip3_operator strategy. docs/market_making_fly_design.md: decisions, measurements, caveats. Adds pyarrow (MaleCNS feather files). 77 unit tests plus an opt-in full-connectome test (CONDOR_FLY_FULL_TEST=1). --- agents/market_making_fly/AGENT.md | 129 ++++ .../market_making_fly/routines/fly_brain.py | 635 ++++++++++++++++++ .../market_making_fly/routines/fly_chart.py | 87 +++ .../market_making_fly/routines/fly_status.py | 146 ++++ .../routines/hip3_market_scanner.py | 325 +++++++++ .../routines/mm_dashboard.py | 388 +++++++++++ .../skills/capital_allocation/SKILL.md | 162 +++++ .../skills/fly_decoder/SKILL.md | 64 ++ .../skills/fly_mm_deploy/SKILL.md | 117 ++++ .../skills/mm_bot_report/SKILL.md | 40 ++ .../skills/pmm_config_playbook/SKILL.md | 106 +++ .../pmm_config_playbook/config_aggressive.md | 76 +++ .../pmm_config_playbook/config_balanced.md | 70 ++ .../config_conservative.md | 92 +++ .../strategies/fly_hip3_operator/strategy.md | 49 ++ condor/fly/__init__.py | 6 + condor/fly/__main__.py | 72 ++ condor/fly/chart.py | 150 +++++ condor/fly/data.py | 97 +++ condor/fly/decoder.py | 218 ++++++ condor/fly/guard.py | 209 ++++++ condor/fly/market.py | 249 +++++++ condor/fly/naming.py | 48 ++ condor/fly/neural/THIRD_PARTY.md | 31 + condor/fly/neural/__init__.py | 0 condor/fly/neural/arrays.lock.json | 13 + condor/fly/neural/brain.py | 446 ++++++++++++ condor/fly/neural/circuit.py | 87 +++ condor/fly/neural/common.py | 40 ++ condor/fly/neural/connectome.py | 288 ++++++++ condor/fly/neural/datasets.json | 25 + condor/fly/neural/kernel.cpp | 96 +++ condor/fly/neural/neurons.lock.json | 3 + condor/fly/neural/prepare.py | 139 ++++ condor/fly/neural/rule.py | 65 ++ condor/fly/neural/sensory.py | 17 + condor/fly/neural/sources.lock.json | 17 + condor/fly/neural/state.py | 82 +++ condor/fly/neural/transmitters.py | 24 + condor/fly/neural/visual.py | 155 +++++ condor/fly/posture.py | 137 ++++ condor/fly/reinforcement.py | 56 ++ condor/fly/run_state.py | 134 ++++ condor/fly/worker.py | 179 +++++ docs/market_making_fly_design.md | 571 ++++++++++++++++ pyproject.toml | 2 + tests/test_fly_chart.py | 119 ++++ tests/test_fly_decoder.py | 135 ++++ tests/test_fly_full_graph.py | 69 ++ tests/test_fly_guard.py | 139 ++++ tests/test_fly_market.py | 63 ++ tests/test_fly_naming.py | 29 + tests/test_fly_posture.py | 107 +++ tests/test_fly_reinforcement.py | 33 + tests/test_fly_run_state.py | 67 ++ uv.lock | 40 +- 56 files changed, 6942 insertions(+), 1 deletion(-) create mode 100644 agents/market_making_fly/AGENT.md create mode 100644 agents/market_making_fly/routines/fly_brain.py create mode 100644 agents/market_making_fly/routines/fly_chart.py create mode 100644 agents/market_making_fly/routines/fly_status.py create mode 100644 agents/market_making_fly/routines/hip3_market_scanner.py create mode 100644 agents/market_making_fly/routines/mm_dashboard.py create mode 100644 agents/market_making_fly/skills/capital_allocation/SKILL.md create mode 100644 agents/market_making_fly/skills/fly_decoder/SKILL.md create mode 100644 agents/market_making_fly/skills/fly_mm_deploy/SKILL.md create mode 100644 agents/market_making_fly/skills/mm_bot_report/SKILL.md create mode 100644 agents/market_making_fly/skills/pmm_config_playbook/SKILL.md create mode 100644 agents/market_making_fly/skills/pmm_config_playbook/config_aggressive.md create mode 100644 agents/market_making_fly/skills/pmm_config_playbook/config_balanced.md create mode 100644 agents/market_making_fly/skills/pmm_config_playbook/config_conservative.md create mode 100644 agents/market_making_fly/strategies/fly_hip3_operator/strategy.md create mode 100644 condor/fly/__init__.py create mode 100644 condor/fly/__main__.py create mode 100644 condor/fly/chart.py create mode 100644 condor/fly/data.py create mode 100644 condor/fly/decoder.py create mode 100644 condor/fly/guard.py create mode 100644 condor/fly/market.py create mode 100644 condor/fly/naming.py create mode 100644 condor/fly/neural/THIRD_PARTY.md create mode 100644 condor/fly/neural/__init__.py create mode 100644 condor/fly/neural/arrays.lock.json create mode 100644 condor/fly/neural/brain.py create mode 100644 condor/fly/neural/circuit.py create mode 100644 condor/fly/neural/common.py create mode 100644 condor/fly/neural/connectome.py create mode 100644 condor/fly/neural/datasets.json create mode 100644 condor/fly/neural/kernel.cpp create mode 100644 condor/fly/neural/neurons.lock.json create mode 100644 condor/fly/neural/prepare.py create mode 100644 condor/fly/neural/rule.py create mode 100644 condor/fly/neural/sensory.py create mode 100644 condor/fly/neural/sources.lock.json create mode 100644 condor/fly/neural/state.py create mode 100644 condor/fly/neural/transmitters.py create mode 100644 condor/fly/neural/visual.py create mode 100644 condor/fly/posture.py create mode 100644 condor/fly/reinforcement.py create mode 100644 condor/fly/run_state.py create mode 100644 condor/fly/worker.py create mode 100644 docs/market_making_fly_design.md create mode 100644 tests/test_fly_chart.py create mode 100644 tests/test_fly_decoder.py create mode 100644 tests/test_fly_full_graph.py create mode 100644 tests/test_fly_guard.py create mode 100644 tests/test_fly_market.py create mode 100644 tests/test_fly_naming.py create mode 100644 tests/test_fly_posture.py create mode 100644 tests/test_fly_reinforcement.py create mode 100644 tests/test_fly_run_state.py diff --git a/agents/market_making_fly/AGENT.md b/agents/market_making_fly/AGENT.md new file mode 100644 index 000000000..5cbdb281e --- /dev/null +++ b/agents/market_making_fly/AGENT.md @@ -0,0 +1,129 @@ +--- +name: Market Making Fly +description: Market maker whose regime, spread width and reference-price lean are decoded + from a simulated fly connectome watching the chart, with P&L fed back as dopamine. + Operates pmm_mister on Hyperliquid HIP-3 perps. +agent_key: claude-acp:sonnet +tools: +- get_prices +- get_portfolio_overview +- list_executors +- get_executor +- get_performance_report +- manage_controllers +- manage_bots +- search_history +- manage_routines +- manage_agents +- manage_strategies +- control_agent +- trading_agent_journal_read +- trading_agent_journal_write +- manage_memory +- manage_skill +- run_code +when_to_consult: When the user asks what the fly sees or thinks about a HIP-3 market, + why it widened or leaned, whether it is learning, or wants the fly market maker + deployed, started, stopped or rotated — use consult for questions and delegate for a + deployment. +server_required: true +server_name: '' +created_by: 456181693 +created_at: '2026-09-12T00:00:00+00:00' +--- + +# Market Making Fly + +You operate a market maker whose **discretion belongs to a fly**. A simulation of the +MaleCNS v1.0 connectome (166,700 neurons, 25.6 M connections, vendored from stonkfly) +is shown a 320×180 OHLCV chart of each market. Its spike counts are decoded into a +**posture** — regime, spread multiplier, reference-price lean — which a fixed mapping +turns into a `pmm_mister` config. The combined P&L of the fly's bots is pulsed back +into its dopamine cells (PAM11 reward, PPL101 aversive) and a candidate memory rule +adjusts KC→MBON synapses. This all runs deterministically in the `fly_brain` routine. + +## The one rule above all others + +While a fly run is live on a pair, **the fly quotes and you operate.** You deploy, +start, stop, rotate, and report. You never set spreads, skew, or regime from your own +analysis, and you never "correct" the fly's posture. If you believe the fly is wrong, +you stop it and say why; you do not out-vote it. The guard inside `fly_brain` (fee +floor, loss stop, loss-rate breaker, apply cooldown, closed books, collateral) can +veto or halt, and it never substitutes a posture either. + +## What you handle +- Deploying the fly market maker end-to-end on up to three HIP-3 pairs (`fly_mm_deploy`) +- Starting `fly_brain` in shadow, then live; stopping it; rotating a market slot +- Reading `fly_status` and explaining a posture in market-making terms +- Reporting bot health with `mm_bot_report` / `mm_dashboard` +- Saying plainly what the fly has and has not demonstrated + +## What you do not handle +- Choosing spreads, skew or regime yourself while the fly runs +- Non-HIP-3 venues (use Market Making Expert) +- Claims that the fly "understands" the market or has learned to trade — it has not + been shown to; see the caveats below + +## Two modes + +**Consulted:** "what does the fly see on DRAM", "why did it widen", "is it learning". +Run `fly_status` (and `fly_chart` for the picture), answer in key: value lines, quote +the numbers (trend_z, arousal_z, gate, regime, spread ×, lean, changed edges), and +repeat the caveats. Do not deploy. + +**Delegated:** read `fly_mm_deploy` and follow it end-to-end — scanner → three picks → +neutral configs → deploy with a loss cap → start `fly_brain` in shadow — then verify +with `mm_bot_report`. Switch to live only when the task says so. + +``` +manage_skill(action="read", name="fly_mm_deploy") +``` + +## Routines +| Routine | Use | +|---|---| +| `hip3_market_scanner` | Rank xyz HIP-3 markets; take the top picks and their spreads | +| `fly_chart` | Render the exact frame for a pair (what the fly sees) | +| `fly_brain` | The loop (continuous). `mode=shadow|live`, `pairs`, `picked_spreads_bps`, `run_name` | +| `fly_status` | Latest posture per pair, last observation, guard state, memory stats | +| `mm_dashboard`, `mm_bot_report` | Inventory, positions, P&L, errors | + +Naming is derived from the pair: `XYZ:DRAM-USD` → bot `dram-fly`, config `dram_fly_mm`. +`fly_brain` reads P&L from exactly those names, so deploy with them. + +## How to read a posture +- `regime`: pause > volatile > trending_up/down > quiet > ranging, from z-scored channels +- `trend_z`: DNp20 right-minus-left firing vs the pair's rolling baseline; sign = lean + direction, gate (DNpe017 spike) required for a trending call +- `arousal_z`: descending-neuron population rate vs baseline; sets `spread ×` + (`1 + 0.5·z`, clipped 0.6–2.5); `z ≥ 2.5` pulls quotes (kill switch) +- `lean`: `±1 bp per z`, capped at 3 bp and at half the first spread level; applied as + asymmetric buy/sell spreads around mid +- `warm=False` for a pair's first 10 observations: neutral posture while the baseline forms +- Applies happen only on a regime change, a spread-× move ≥ 0.15 or a lean move ≥ 0.5 bp, + and at most once per 5 minutes per pair + +## HIP-3 facts you must keep (from the HIP-3 operator playbook) +- Uppercase pair with issuer prefix (`XYZ:DRAM-USD`); lowercase → zero orders +- Unified collateral: available USD from `get_portfolio_overview(["hyperliquid_perpetual"])` +- Many markets close off-hours; the fly holds while a book is closed and stops the bot + after 5 closed ticks +- All-in maker fee ≈ 1.3 bp/side; the take-profit floor in code is `max(4 bp, 2.2 × round trip)` +- Tight bands `target 0.4 / min 0.3 / max 0.5`, `global_stop_loss 0.02`, leverage ≤ 5 + +## Caveats you repeat when asked +- The decoder is an engineered readout of spike counts, not a discovery of market-making + neurons; a persistent circuit bias would be a persistent lean, which baseline-centring + reduces but does not remove +- Dopamine pulses report P&L change between two observations, not credit for the last + posture; synapses also change from endogenous activity +- No profitable learning has been demonstrated; the guard and the controller's stop + loss bound the loss, not the fly +- Full detail: `docs/market_making_fly_design.md` + +## Memory & Skills +Check `manage_memory` and `manage_skill` before answering; update them when the user +corrects you or a pattern repeats. + +## Response format +Key: value lines, recommendation first, numbers quoted from `fly_status`. diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py new file mode 100644 index 000000000..87957959c --- /dev/null +++ b/agents/market_making_fly/routines/fly_brain.py @@ -0,0 +1,635 @@ +"""The fly loop: chart → connectome → posture → pmm_mister config, with P&L dopamine. + +One shared brain is shown up to three HIP-3 markets in round-robin. Each tick: + +1. fetch candles + live book for this tick's pair; +2. read the combined net P&L of the fly's bots, turn its change since the last + observation into ``reward`` / ``aversive`` / ``none``; +3. render the chart, run the neural window in the worker process (the dopamine + pulse is delivered during it), decode spike counts into a posture; +4. checkpoint the brain and commit the accounting anchor BEFORE anything is + applied; +5. map the posture to a config, pass it through the guard, apply it in ``live`` + mode (``shadow`` only records what it would have applied); +6. persist events, latest.json, the input frame, and the live report. + +The guard can veto or halt; nothing in this file chooses a posture. +""" + +from __future__ import annotations + +import asyncio +import logging +import multiprocessing +import time +from concurrent.futures import ProcessPoolExecutor +from dataclasses import asdict +from pathlib import Path + +import numpy as np +import plotly.graph_objects as go +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.fly import worker +from condor.fly.chart import market_frame +from condor.fly.decoder import ( + Baseline, + Channels, + DecoderSettings, + Hysteresis, + Posture, + decode, + should_apply, +) +from condor.fly.guard import ( + GuardSettings, + GuardState, + Halt, + Veto, + check_apply_window, + check_collateral, + check_config, + check_market_open, + check_not_halted, + check_pnl, + check_price_move, + default_max_loss, + record_apply, + resume, +) +from condor.fly.market import FixtureMarket, LiveMarket, required_collateral +from condor.fly.naming import pair_names, parse_pairs +from condor.fly.posture import MarketSpec, build_config, config_diff +from condor.fly.reinforcement import reinforcement +from condor.fly.run_state import RunDir, source_hashes +from condor.memory.paths import agent_home +from condor.reports import LiveReport + +logger = logging.getLogger(__name__) + +CONTINUOUS = True +CATEGORY = "Monitoring" +AGENT_SLUG = "market_making_fly" +ROUTINE_NAME = "fly_brain" +BPS = 1e-4 + + +class Config(BaseModel): + """Fly-connectome market maker: one brain, up to three HIP-3 markets, P&L dopamine.""" + + pairs: str = Field( + default="XYZ:DRAM-USD,XYZ:SPCX-USD,XYZ:SMSN-USD", + description="Up to 3 uppercase HIP-3 pairs, comma-separated (bot {token}-fly, config {token}_fly_mm)", + ) + picked_spreads_bps: str = Field( + default="8,8,8", + description="Scanner spread per pair in bp, same order as pairs", + ) + connector_name: str = Field( + default="hyperliquid_perpetual", description="Connector" + ) + total_amount_quote: float = Field( + default=500.0, description="Capital per pair (quote)" + ) + leverage: int = Field(default=3, description="Leverage per pair (cap 5)") + mode: str = Field( + default="shadow", description="shadow (record only) or live (apply configs)" + ) + learning: bool = Field( + default=True, description="False freezes the memory rule (control run)" + ) + run_name: str = Field( + default="fly", + description="Run directory under the agent home; new name = new brain lineage", + ) + resume_reviewed: bool = Field( + default=False, description="Clear a transient halt after review" + ) + interval_sec: int = Field( + default=60, description="Wall seconds between observations" + ) + neural_ms: float = Field( + default=500.0, description="Neural time per observation (ms)" + ) + candle_interval: str = Field( + default="5m", description="Candle interval the fly sees" + ) + n_candles: int = Field(default=72, description="Candles on the chart") + reward_deadband_bps: float = Field( + default=1.0, description="Pulse deadband, bp of combined capital" + ) + baseline_window: int = Field( + default=60, description="Observations per pair in the rolling baseline" + ) + baseline_warmup: int = Field( + default=10, description="Observations before the first non-neutral posture" + ) + center_bias: bool = Field( + default=True, description="Subtract the rolling mean of the trend channel" + ) + min_apply_interval_sec: int = Field( + default=300, description="Per-pair cooldown between config applies" + ) + max_loss_quote: float = Field( + default=0.0, description="Loss stop in quote; 0 = 4% of combined capital" + ) + fixture: bool = Field( + default=False, description="Offline synthetic market (shadow only)" + ) + fast: bool = Field( + default=False, description="Skip wall waits (fixture/shadow only)" + ) + steps: int = Field( + default=0, description="Stop after N observations; 0 runs until stopped" + ) + + +def _specs(config: Config, pairs: list[str]) -> list[MarketSpec]: + spreads = [float(x) for x in config.picked_spreads_bps.split(",") if x.strip()] + if len(spreads) != len(pairs): + raise ValueError( + f"picked_spreads_bps has {len(spreads)} entries for {len(pairs)} pairs" + ) + return [ + MarketSpec( + connector_name=config.connector_name, + trading_pair=pair, + total_amount_quote=config.total_amount_quote, + picked_spread_bps=spread, + leverage=config.leverage, + ) + for pair, spread in zip(pairs, spreads) + ] + + +def _frame_figure(frame: np.ndarray) -> go.Figure: + fig = go.Figure(go.Image(z=frame)) + fig.update_layout( + margin=dict(l=0, r=0, t=0, b=0), + height=360, + xaxis=dict(visible=False), + yaxis=dict(visible=False), + legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), + ) + return fig + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + chat_id = context._chat_id + if config.mode not in ("shadow", "live"): + raise ValueError("mode must be shadow or live") + if config.fixture and config.mode != "shadow": + raise ValueError("fixture runs are shadow only") + if config.fast and config.mode == "live": + raise ValueError("fast is for fixture/shadow runs only") + if config.interval_sec < 10: + raise ValueError("interval_sec must be >= 10") + pairs = parse_pairs(config.pairs) + specs = _specs(config, pairs) + by_pair = {s.trading_pair: s for s in specs} + decoder_settings = DecoderSettings( + window=config.baseline_window, + warmup=config.baseline_warmup, + center_bias=config.center_bias, + ) + hysteresis = Hysteresis(min_apply_interval_sec=config.min_apply_interval_sec) + guard_settings = GuardSettings(max_loss_quote=config.max_loss_quote) + max_loss = default_max_loss(specs, guard_settings) + deadband = ( + config.reward_deadband_bps * BPS * sum(s.total_amount_quote for s in specs) + ) + + run_dir = RunDir(agent_home(AGENT_SLUG) / "fly" / config.run_name) + run_dir.lock() + pool = ProcessPoolExecutor( + max_workers=1, + mp_context=multiprocessing.get_context("spawn"), + initializer=worker._init, + initargs=(config.learning, 10.0, 200.0, 20.0), + ) + loop = asyncio.get_running_loop() + report = LiveReport( + "Market Making Fly", + source_name=ROUTINE_NAME, + tags=["fly", "market-making", "hip3", config.mode], + auto_refresh_seconds=config.interval_sec, + ) + count = 0 + stop_reason = "stopped" + try: + # ---- state, provenance, checkpoint -------------------------------- + state = run_dir.load_state() + guard_state = GuardState.from_dict(state.get("guard")) + if guard_state.halted: + resume( + guard_state, config.resume_reviewed + ) # raises Halt when not clearable + baselines = { + p: Baseline.from_dict(state.get("baselines", {}).get(p)) for p in pairs + } + postures: dict[str, Posture | None] = { + p: Posture.from_dict(v) if (v := state.get("postures", {}).get(p)) else None + for p in pairs + } + applied: dict[str, dict | None] = { + p: state.get("applied", {}).get(p) for p in pairs + } + anchor = state.get("anchor") + tick = int(state.get("tick", 0)) + + from condor.fly.data import verify + + dataset = await loop.run_in_executor(None, verify) + brain_prov = await loop.run_in_executor(pool, worker._provenance) + provenance = { + "settings": { + k: v + for k, v in config.model_dump().items() + if k + not in ( + "pairs", + "picked_spreads_bps", + "mode", + "fast", + "steps", + "resume_reviewed", + "run_name", + "fixture", + ) + }, + "decoder": asdict(decoder_settings), + "guard": asdict(guard_settings), + "dataset": dataset, + **brain_prov, + "learning_validated": False, + "source_sha256": source_hashes(), + } + run_dir.check_provenance(provenance) + if state.get("checkpoint"): + path = run_dir.verify_checkpoint(state["checkpoint"]) + await loop.run_in_executor(pool, worker._restore, str(path)) + + if config.fixture: + market = FixtureMarket(pairs, config.n_candles) + else: + from config_manager import get_client + + client = await get_client(chat_id, context=context) + if not client: + raise RuntimeError("No Hummingbot server available for this chat") + market = LiveMarket( + client, config.connector_name, config.candle_interval, config.n_candles + ) + + await context.bot.send_message( + chat_id=chat_id, + text=( + f"🪰 Fly started [{config.mode}] run={config.run_name} tick={tick} " + f"pairs={','.join(pairs)} learning={config.learning}" + ), + ) + + def persist(extra: dict | None = None) -> None: + run_dir.save_state( + { + "tick": tick, + "anchor": anchor, + "guard": guard_state.to_dict(), + "baselines": {p: b.to_dict() for p, b in baselines.items()}, + "postures": { + p: (q.to_dict() if q else None) for p, q in postures.items() + }, + "applied": applied, + **(extra or {}), + } + ) + + # ---- the loop ------------------------------------------------------- + while not config.steps or count < config.steps: + started = time.monotonic() + if run_dir.stop_requested(): + stop_reason = "STOP file" + break + if guard_state.halted: + stop_reason = f"halted: {guard_state.halted}" + break + pair = pairs[tick % len(pairs)] + names = pair_names(pair) + spec = by_pair[pair] + row: dict = { + "tick": tick, + "wall_time": time.time(), + "pair": pair, + "mode": config.mode, + } + try: + obs = await market.observe(pair) + equity, volume, per_pair = await market.equity(pairs) + if anchor is None: + kind, delta = "none", 0.0 + else: + kind, delta = reinforcement(equity, anchor, deadband) + delta = float(delta) + row.update( + { + "quote": {"bid": obs.bid, "ask": obs.ask, "open": obs.open}, + "equity": equity, + "volume": volume, + "pnl_delta": delta, + "stimulus": kind, + "bots": per_pair, + } + ) + check_pnl(equity, volume, guard_state, guard_settings, max_loss) + + if not obs.open: + # Nothing new for the fly to see; keep the anchor so the next + # open observation carries the whole interval's P&L change. + try: + stop = check_market_open( + pair, False, guard_state, guard_settings + ) + except Veto as veto: + row["execution"] = {"status": "CLOSED", "reason": str(veto)} + stop = False + if stop: + stopped = config.mode == "live" and await market.stop_bot(pair) + row["execution"] = { + "status": "STOP_BOT" if stopped else "CLOSED", + "reason": f"book closed {guard_state.closed_ticks[pair]} ticks", + } + tick += 1 + persist() + run_dir.append_event(row) + run_dir.write_latest(row) + count += 1 + continue + check_market_open(pair, True, guard_state, guard_settings) + + frame = market_frame( + pair, obs.candles, obs.bid, obs.ask, config.n_candles + ) + neural = await loop.run_in_executor( + pool, worker._observe, frame, kind, config.neural_ms + ) + posture = decode( + Channels( + neural["trend_hz"], neural["arousal_hz"], neural["gate_spikes"] + ), + baselines[pair], + decoder_settings, + ) + proposed = build_config(spec, posture) + + # Checkpoint and anchor are committed before any apply. + ck = run_dir.checkpoint_path(tick) + sha = await loop.run_in_executor(pool, worker._checkpoint, str(ck)) + anchor = equity + tick += 1 + persist({"checkpoint": {"file": ck.name, "sha256": sha}}) + + neural_row = {k: v for k, v in neural.items() if k != "cell_ids"} + row.update({"neural": neural_row, "posture": posture.to_dict()}) + + now = time.time() + ok, why = should_apply( + postures[pair], + posture, + guard_state.last_apply.get(pair), + now, + hysteresis, + ) + diff = config_diff(applied[pair], proposed) + if not ok: + row["execution"] = {"status": "HOLD", "reason": why} + elif config.mode == "shadow": + postures[pair] = posture + row["execution"] = { + "status": "SHADOW", + "reason": why, + "would_apply": diff, + } + else: + try: + check_not_halted(guard_state) + check_apply_window(guard_state, now, guard_settings) + check_config(proposed, spec) + check_collateral( + await market.available_usd(), required_collateral(specs) + ) + check_price_move( + obs.mid, await market.fresh_mid(pair), guard_settings + ) + try: + await market.apply(pair, proposed) + except Exception as failure: + logger.exception("fly apply failed for %s", pair) + row["execution"] = { + "status": "ERROR", + "reason": repr(failure)[:300], + } + record_apply(guard_state, pair, now, False, guard_settings) + else: + record_apply(guard_state, pair, now, True, guard_settings) + postures[pair] = posture + applied[pair] = proposed + row["execution"] = { + "status": "APPLIED", + "reason": why, + "diff": diff, + } + except Veto as veto: + row["execution"] = {"status": "VETO", "reason": str(veto)} + persist() + run_dir.append_event(row) + run_dir.write_latest(row) + run_dir.save_frame(frame) + + # ---- live report -------------------------------------------- + report.clear() + b = report.builder + b.manual_order() + b.section( + "01 / WHAT THE FLY SEES", + f"{pair} — last input frame, tick {tick - 1}", + ) + b.plotly(_frame_figure(frame)) + b.section( + "02 / THIS OBSERVATION", "Spikes, channels and the decoded posture" + ) + b.kpi("Regime", posture.regime) + b.kpi("Spread ×", f"{posture.spread_mult:.2f}") + b.kpi("Lean", f"{posture.shift_bps:+.2f} bp") + b.kpi("Trend z", f"{posture.trend_z:+.2f}") + b.kpi("Arousal z", f"{posture.arousal_z:+.2f}") + b.kpi("Gate", str(neural["gate_spikes"])) + b.kpi("Stimulus", kind) + b.kpi("P&L Δ", f"{delta:+.4f}") + b.kpi("Equity", f"{equity:+.4f}") + b.kpi("Execution", row["execution"]["status"]) + b.kpi("KC spikes", str(neural["kc_spikes"])) + b.kpi("Changed edges", str(neural["memory"]["changed_edges"])) + b.kpi("Mean efficacy", f"{neural['memory']['mean_efficacy']:.4f}") + b.kpi("Compute s", f"{neural['compute_seconds']:.1f}") + b.section( + "03 / POSTURES", "Last decided posture per pair and what is applied" + ) + b.table( + [ + { + "Pair": p, + "Regime": (postures[p].regime if postures[p] else "—"), + "Spread ×": ( + f"{postures[p].spread_mult:.2f}" if postures[p] else "—" + ), + "Lean bp": ( + f"{postures[p].shift_bps:+.2f}" if postures[p] else "—" + ), + "Baseline n": baselines[p].count, + "Applied buy": (applied[p] or {}).get("buy_spreads", "—"), + "Applied sell": (applied[p] or {}).get("sell_spreads", "—"), + "Bot": ( + "running" if per_pair.get(p, {}).get("running") else "—" + ), + } + for p in pairs + ], + [ + "Pair", + "Regime", + "Spread ×", + "Lean bp", + "Baseline n", + "Applied buy", + "Applied sell", + "Bot", + ], + ) + b.section("04 / RECENT TICKS", "Newest last") + events = run_dir.recent_events(40) + b.table( + [ + { + "Tick": e["tick"], + "Pair": e["pair"], + "Stim": e.get("stimulus", "—"), + "Δ": f"{e.get('pnl_delta', 0):+.3f}", + "Regime": (e.get("posture") or {}).get("regime", "—"), + "Spread ×": (e.get("posture") or {}).get( + "spread_mult", "—" + ), + "Lean": (e.get("posture") or {}).get("shift_bps", "—"), + "Exec": (e.get("execution") or {}).get("status", "—"), + "Reason": str((e.get("execution") or {}).get("reason", ""))[ + :60 + ], + } + for e in events + ], + [ + "Tick", + "Pair", + "Stim", + "Δ", + "Regime", + "Spread ×", + "Lean", + "Exec", + "Reason", + ], + ) + b.section("05 / GUARD", "Halts, breakers, apply budget") + b.kpi("Halted", guard_state.halted or "no") + b.kpi("Applies today", str(guard_state.applies_today)) + b.kpi("Session high", f"{guard_state.session_high_net:+.4f}") + b.kpi("Ticks since high", str(guard_state.ticks_since_high)) + b.kpi("Loss stop", f"-{max_loss:.2f}") + b.markdown( + "_The regime, spread multiplier and lean are decoded from spike counts " + "of identified cells; the mapping is engineered and unvalidated. Dopamine " + "pulses report P&L change, not credit for the last posture. See " + "docs/market_making_fly_design.md §15._" + ) + await report.update() + + if row["execution"]["status"] in ("APPLIED", "STOP_BOT", "ERROR"): + await context.bot.send_message( + chat_id=chat_id, + text=( + f"🪰 {pair} tick {tick - 1}: {row['execution']['status']} — " + f"{posture.regime} ×{posture.spread_mult:.2f} {posture.shift_bps:+.1f}bp " + f"({row['execution']['reason']})" + )[:900], + ) + except asyncio.CancelledError: + raise + except Halt as halt: + row["execution"] = { + "status": "HALT", + "reason": halt.reason, + "financial": halt.financial, + } + persist() + run_dir.append_event(row) + run_dir.write_latest(row) + stopped = [] + if config.mode == "live": + for p in pairs: + try: + if await market.stop_bot(p): + stopped.append(pair_names(p).bot_name) + except Exception: + logger.exception("fly: stopping %s after halt failed", p) + await context.bot.send_message( + chat_id=chat_id, + text=( + f"🪰 HALT ({'financial' if halt.financial else 'transient'}): {halt.reason}. " + f"Stopped bots: {', '.join(stopped) or 'none'}. " + + ( + "A financial halt needs a new run_name." + if halt.financial + else "Restart with resume_reviewed=true after review." + ) + ), + ) + stop_reason = f"halted: {halt.reason}" + break + except Exception as failure: + logger.exception("fly tick %s failed", tick) + row["execution"] = { + "status": "TICK_ERROR", + "reason": repr(failure)[:300], + } + run_dir.append_event(row) + run_dir.write_latest(row) + count += 1 + if config.steps and count >= config.steps: + stop_reason = f"{count} steps done" + break + if not config.fast: + until = started + config.interval_sec + while time.monotonic() < until: + if run_dir.stop_requested(): + break + await asyncio.sleep(min(1.0, until - time.monotonic())) + except asyncio.CancelledError: + stop_reason = "cancelled" + raise + finally: + if report.report_id is not None: + report.clear() + report.builder.auto_refresh(None) + report.builder.section("FLY STOPPED", stop_reason) + report.builder.markdown(f"Run directory: `{run_dir.root}`") + await report.update() + pool.shutdown(wait=True, cancel_futures=True) + run_dir.unlock() + try: + await context.bot.send_message( + chat_id=chat_id, + text=f"🪰 Fly stopped after {count} observations: {stop_reason}", + ) + except Exception: + logger.warning("fly: final notification failed") + return f"Fly stopped after {count} observations: {stop_reason}" diff --git a/agents/market_making_fly/routines/fly_chart.py b/agents/market_making_fly/routines/fly_chart.py new file mode 100644 index 000000000..f0b5fa76a --- /dev/null +++ b/agents/market_making_fly/routines/fly_chart.py @@ -0,0 +1,87 @@ +"""Render the exact frame the fly would see for a HIP-3 pair, and report it.""" + +from __future__ import annotations + +import logging + +import aiohttp +import plotly.graph_objects as go +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.fly.chart import market_frame, normalize_candles, price_scale +from condor.fly.market import fetch_l2_book, normalize_candle_payload +from condor.fly.naming import pair_names +from condor.reports import ReportBuilder +from config_manager import get_client + +logger = logging.getLogger(__name__) + +CATEGORY = "Market Data" + + +class Config(BaseModel): + """What the fly sees: the 320×180 OHLCV frame for one HIP-3 pair.""" + + trading_pair: str = Field( + default="XYZ:DRAM-USD", description="Uppercase HIP-3 pair" + ) + connector_name: str = Field( + default="hyperliquid_perpetual", description="Connector" + ) + candle_interval: str = Field(default="5m", description="Candle interval") + n_candles: int = Field(default=72, description="Candles on the chart") + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + client = await get_client(context._chat_id, context=context) + if not client: + return "No server available" + names = pair_names(config.trading_pair) + candles = normalize_candle_payload( + await client.market_data.get_candles( + config.connector_name, + config.trading_pair, + interval=config.candle_interval, + max_records=config.n_candles, + ) + ) + async with aiohttp.ClientSession() as session: + book = await fetch_l2_book(session, names.coin) + if not book.open: + return f"{config.trading_pair}: book is closed (no bid/ask) — nothing to render" + frame = market_frame( + config.trading_pair, candles, book.bid, book.ask, config.n_candles + ) + rows = normalize_candles(candles, config.n_candles) + lo, span = price_scale(rows) + + fig = go.Figure(go.Image(z=frame)) + fig.update_layout( + margin=dict(l=0, r=0, t=0, b=0), + height=360, + xaxis=dict(visible=False), + yaxis=dict(visible=False), + legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), + ) + builder = ReportBuilder(f"Fly view — {config.trading_pair}") + builder.source("routine", "fly_chart") + builder.tags(["fly", "chart", "hip3"]) + builder.manual_order() + builder.section("01 / FRAME", "320×180 RGB, exactly what enters the retina") + builder.plotly(fig) + builder.kpi("Candles", str(len(rows))) + builder.kpi("Interval", config.candle_interval) + builder.kpi("Bid", f"{book.bid:g}") + builder.kpi("Ask", f"{book.ask:g}") + builder.kpi("Price floor", f"{lo:g}") + builder.kpi("Price span", f"{span:g}") + builder.markdown( + "_Up candles blue, down candles red: the mapped R8p cells read blue and R8y " + "read green; R1–R6 read luminance. No quotes, inventory or P&L are drawn._" + ) + await builder.save() + return ( + f"{config.trading_pair}: {len(rows)} × {config.candle_interval} candles, " + f"bid {book.bid:g} ask {book.ask:g}, last close {rows[-1]['close']:g}" + ) diff --git a/agents/market_making_fly/routines/fly_status.py b/agents/market_making_fly/routines/fly_status.py new file mode 100644 index 000000000..597a3f653 --- /dev/null +++ b/agents/market_making_fly/routines/fly_status.py @@ -0,0 +1,146 @@ +"""Latest state of a fly run: posture per pair, last observation, guard, memory.""" + +from __future__ import annotations + +import json +import logging + +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.fly.run_state import RunDir +from condor.memory.paths import agent_home +from condor.reports import ReportBuilder + +logger = logging.getLogger(__name__) + +CATEGORY = "Bot Analysis" +AGENT_SLUG = "market_making_fly" + + +class Config(BaseModel): + """Read a fly run's state.json / latest.json / recent events without touching the brain.""" + + run_name: str = Field( + default="fly", description="Run directory under the agent home" + ) + recent: int = Field(default=20, description="Recent ticks to list") + + +def _fmt(value, digits=3) -> str: + if isinstance(value, (int, float)): + return f"{value:+.{digits}f}" if isinstance(value, float) else str(value) + return "—" if value is None else str(value) + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + root = agent_home(AGENT_SLUG) / "fly" / config.run_name + if not root.exists(): + raise FileNotFoundError( + f"No fly run at {root}; start fly_brain with run_name={config.run_name!r}" + ) + run_dir = RunDir(root) + state = run_dir.load_state() + latest = ( + json.loads(run_dir.latest_path.read_text()) + if run_dir.latest_path.exists() + else {} + ) + events = run_dir.recent_events(config.recent) + guard = state.get("guard", {}) + postures = state.get("postures", {}) + baselines = state.get("baselines", {}) + applied = state.get("applied", {}) + neural = latest.get("neural", {}) + memory = neural.get("memory", {}) + + lines = [ + f"run: {config.run_name}", + f"tick: {state.get('tick', 0)}", + f"halted: {guard.get('halted') or 'no'}", + f"anchor_equity: {_fmt(state.get('anchor'))}", + f"session_high_net: {_fmt(guard.get('session_high_net'))}", + f"ticks_since_high: {guard.get('ticks_since_high', 0)}", + f"applies_today: {guard.get('applies_today', 0)}", + f"last_pair: {latest.get('pair', '—')}", + f"last_stimulus: {latest.get('stimulus', '—')}", + f"last_pnl_delta: {_fmt(latest.get('pnl_delta'))}", + f"last_execution: {(latest.get('execution') or {}).get('status', '—')} " + f"({(latest.get('execution') or {}).get('reason', '')})", + f"kc_spikes: {neural.get('kc_spikes', '—')}", + f"gate_spikes: {neural.get('gate_spikes', '—')}", + f"trend_hz: {_fmt(neural.get('trend_hz'))}", + f"arousal_hz: {_fmt(neural.get('arousal_hz'))}", + f"changed_edges: {memory.get('changed_edges', '—')}", + f"mean_efficacy: {_fmt(memory.get('mean_efficacy'), 4)}", + ] + for pair, posture in postures.items(): + if posture: + lines.append( + f"{pair}: {posture['regime']} spread_x={posture['spread_mult']:.2f} " + f"lean_bp={posture['shift_bps']:+.2f} trend_z={posture['trend_z']:+.2f} " + f"arousal_z={posture['arousal_z']:+.2f} baseline_n={len((baselines.get(pair) or {}).get('trend', []))}" + ) + else: + lines.append( + f"{pair}: no posture yet (baseline_n={len((baselines.get(pair) or {}).get('trend', []))})" + ) + lines.append( + "caveat: regime/spread/lean are an engineered readout of spike counts; dopamine " + "pulses report P&L change, not credit for the last posture; no learning is validated" + ) + + builder = ReportBuilder(f"Fly status — {config.run_name}") + builder.source("routine", "fly_status") + builder.tags(["fly", "status"]) + builder.manual_order() + builder.section("01 / RUN", str(root)) + builder.kpi("Tick", str(state.get("tick", 0))) + builder.kpi("Halted", guard.get("halted") or "no") + builder.kpi("Anchor equity", _fmt(state.get("anchor"))) + builder.kpi("Session high", _fmt(guard.get("session_high_net"))) + builder.kpi("Changed edges", str(memory.get("changed_edges", "—"))) + builder.section("02 / POSTURES", "Per pair") + builder.table( + [ + { + "Pair": pair, + "Regime": (p or {}).get("regime", "—"), + "Spread ×": (p or {}).get("spread_mult", "—"), + "Lean bp": (p or {}).get("shift_bps", "—"), + "Trend z": (p or {}).get("trend_z", "—"), + "Arousal z": (p or {}).get("arousal_z", "—"), + "Applied buy": (applied.get(pair) or {}).get("buy_spreads", "—"), + "Applied sell": (applied.get(pair) or {}).get("sell_spreads", "—"), + } + for pair, p in postures.items() + ], + [ + "Pair", + "Regime", + "Spread ×", + "Lean bp", + "Trend z", + "Arousal z", + "Applied buy", + "Applied sell", + ], + ) + builder.section("03 / RECENT TICKS", "Newest last") + builder.table( + [ + { + "Tick": e.get("tick"), + "Pair": e.get("pair"), + "Stim": e.get("stimulus", "—"), + "Δ": _fmt(e.get("pnl_delta")), + "Regime": (e.get("posture") or {}).get("regime", "—"), + "Exec": (e.get("execution") or {}).get("status", "—"), + "Reason": str((e.get("execution") or {}).get("reason", ""))[:60], + } + for e in events + ], + ["Tick", "Pair", "Stim", "Δ", "Regime", "Exec", "Reason"], + ) + await builder.save() + return "\n".join(lines) diff --git a/agents/market_making_fly/routines/hip3_market_scanner.py b/agents/market_making_fly/routines/hip3_market_scanner.py new file mode 100644 index 000000000..a41c85442 --- /dev/null +++ b/agents/market_making_fly/routines/hip3_market_scanner.py @@ -0,0 +1,325 @@ +"""HIP-3 market scanner — ranks xyz-issuer perps for volume-farming market-making.""" + +import asyncio +import logging +import math + +import aiohttp +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from config_manager import get_client + +logger = logging.getLogger(__name__) + +CATEGORY = "Market Data" + +HL_URL = "https://api.hyperliquid.xyz/info" + + +class Config(BaseModel): + """Scan all markets of a HIP-3 builder issuer and return a shortlist for volume-farming MM.""" + + issuer: str = Field( + default="xyz", description="HIP-3 builder issuer slug (e.g. 'xyz')" + ) + min_spread_bps: float = Field( + default=3.0, description="Minimum impact spread in bps" + ) + max_daily_drift_pct: float = Field( + default=3.0, description="Maximum daily price drift %" + ) + min_oi_notional: float = Field( + default=1_000_000.0, description="Minimum open interest in USD" + ) + min_book_depth_usd: float = Field( + default=10_000.0, + description="Min resting book notional within depth_within_bps, per side (liquidity filter)", + ) + depth_within_bps: float = Field( + default=10.0, + description="Band (bps from mid) over which book depth is measured", + ) + depth_check_top_k: int = Field( + default=12, + description="How many top-scored survivors to depth-check via l2Book (bounds API calls)", + ) + top_n: int = Field(default=5, description="Number of top markets to return") + all_in_fee_bps_roundtrip: float = Field( + default=2.6, + description="Informational: total roundtrip fee in bps (~1.3bps/side)", + ) + + +async def _fetch_book_depth(session, coin, ctx_mid, within_bps): + """Return (bid_depth_usd, ask_depth_usd) resting within `within_bps` of mid. (0,0) on failure/empty.""" + try: + async with session.post( + HL_URL, + json={"type": "l2Book", "coin": coin}, + timeout=aiohttp.ClientTimeout(total=10), + ) as resp: + if resp.status != 200: + return 0.0, 0.0 + book = await resp.json() + levels = book.get("levels") if isinstance(book, dict) else None + if not levels or len(levels) != 2 or not levels[0] or not levels[1]: + return 0.0, 0.0 # empty book = closed / illiquid + bids, asks = levels[0], levels[1] + best_bid = float(bids[0]["px"]) + best_ask = float(asks[0]["px"]) + mid = (best_bid + best_ask) / 2 or ctx_mid + if mid <= 0: + return 0.0, 0.0 + + def _side(side_levels, is_bid): + tot = 0.0 + for lvl in side_levels: + px = float(lvl["px"]) + sz = float(lvl["sz"]) + off = (mid - px) / mid * 1e4 if is_bid else (px - mid) / mid * 1e4 + if off > within_bps: + break + tot += px * sz + return tot + + return _side(bids, True), _side(asks, False) + except Exception as e: + logger.warning(f"l2Book depth fetch failed for {coin}: {e}") + return 0.0, 0.0 + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + # Client is optional — only used for report persistence, not for the scan itself. + client = await get_client(context._chat_id, context=context) + + issuer = config.issuer.lower() + issuer_upper = issuer.upper() + + # ── 1. Fetch universe + contexts from Hyperliquid public API ────────────── + payload = {"type": "metaAndAssetCtxs", "dex": issuer} + try: + async with aiohttp.ClientSession() as session: + async with session.post( + HL_URL, json=payload, timeout=aiohttp.ClientTimeout(total=10) + ) as resp: + if resp.status != 200: + return f"Hyperliquid API error: HTTP {resp.status}" + data = await resp.json() + except Exception as e: + return f"Failed to fetch Hyperliquid data: {e}" + + if not isinstance(data, list) or len(data) < 2: + return f"Unexpected API response shape: {type(data)}" + + meta, ctxs = data[0], data[1] + universe = meta.get("universe", []) + if not universe: + return f"No markets found for issuer '{issuer}'" + + # ── 2. Compute per-market metrics ───────────────────────────────────────── + markets = [] + for asset, ctx in zip(universe, ctxs): + try: + name = asset.get("name", "") # e.g. "xyz:SMSN" (l2Book coin) + pair = name.upper() + "-USD" # e.g. "XYZ:SMSN-USD" (trading_pair) + + mid_raw = ctx.get("midPx") + mark_raw = ctx.get("markPx") + prev_raw = ctx.get("prevDayPx") + impact_pxs = ctx.get("impactPxs") + + volume = float(ctx.get("dayNtlVlm", 0) or 0) + mid = float(mid_raw) if mid_raw else 0.0 + mark = float(mark_raw) if mark_raw else 0.0 + prev = float(prev_raw) if prev_raw else 0.0 + + open_market = ( + mid > 0 and isinstance(impact_pxs, list) and len(impact_pxs) == 2 + ) + + spread_bps = None + if open_market: + try: + bid_impact = float(impact_pxs[0]) + ask_impact = float(impact_pxs[1]) + spread_bps = (ask_impact - bid_impact) / mid * 1e4 + except (ValueError, TypeError, ZeroDivisionError): + open_market = False + + daily_drift_pct = abs(mark / prev - 1) * 100 if prev else 999.0 + oi_notional = float(ctx.get("openInterest", 0) or 0) * mark + + markets.append( + { + "pair": pair, + "coin": name, + "volume": volume, + "mid": mid, + "mark": mark, + "spread_bps": spread_bps, + "daily_drift_pct": daily_drift_pct, + "oi_notional": oi_notional, + "book_depth_usd": None, + "open_market": open_market, + "maxLeverage": int(asset.get("maxLeverage", 0)), + "funding": ctx.get("funding", "N/A"), + } + ) + except Exception as e: + logger.warning(f"Error processing market {asset.get('name', '?')}: {e}") + + total_markets = len(markets) + + # ── 3. Pre-filters (open / spread / drift / OI) ─────────────────────────── + prelim = [ + m + for m in markets + if ( + m["open_market"] + and m["spread_bps"] is not None + and m["spread_bps"] >= config.min_spread_bps + and m["daily_drift_pct"] <= config.max_daily_drift_pct + and m["oi_notional"] >= config.min_oi_notional + ) + ] + + # ── 4. Score and rank (before depth check) ──────────────────────────────── + for m in prelim: + m["score"] = ( + math.log(max(m["volume"], 1)) + + 0.3 * min(m["spread_bps"], 8.0) + - 0.4 * m["daily_drift_pct"] + ) + prelim.sort(key=lambda m: m["score"], reverse=True) + + # ── 5. LIQUIDITY FILTER — real book depth on the top-scored candidates ──── + # Only depth-check the top_k (bounds l2Book calls; the rest can't outrank them anyway). + candidates = prelim[: config.depth_check_top_k] + if candidates: + try: + async with aiohttp.ClientSession() as session: + depths = await asyncio.gather( + *[ + _fetch_book_depth( + session, m["coin"], m["mid"], config.depth_within_bps + ) + for m in candidates + ] + ) + for m, (bid_d, ask_d) in zip(candidates, depths): + # Require BOTH sides liquid for two-sided MM → use the weaker side. + m["book_depth_usd"] = min(bid_d, ask_d) + except Exception as e: + logger.warning(f"Depth-check batch failed: {e}") + + survivors = [ + m + for m in candidates + if m["book_depth_usd"] is not None + and m["book_depth_usd"] >= config.min_book_depth_usd + ] + survivors.sort(key=lambda m: m["score"], reverse=True) + shortlist = survivors[: config.top_n] + + # ── 6. Fallback if zero survivors ───────────────────────────────────────── + no_survivors = len(survivors) == 0 + fallback = [] + if no_survivors: + fallback = sorted(markets, key=lambda m: m["volume"], reverse=True)[:5] + + # ── 7. Build summary ────────────────────────────────────────────────────── + top_pick = shortlist[0]["pair"] if shortlist else "NONE" + + lines = [ + f"**HIP-3 Market Scanner — issuer: {issuer_upper}**", + f"Scanned: {total_markets} markets | Pre-filter pass: {len(prelim)} | Depth-checked: {len(candidates)} | Survivors: {len(survivors)} | Top-{config.top_n} shown", + f"Filters: spread >= {config.min_spread_bps}bps | drift <= {config.max_daily_drift_pct}% | OI >= ${config.min_oi_notional:,.0f} | depth >= ${config.min_book_depth_usd:,.0f}/side within {config.depth_within_bps}bps", + f"Fee context: {config.all_in_fee_bps_roundtrip}bps round-trip (~{config.all_in_fee_bps_roundtrip / 2:.2f}bps/side)", + "", + ] + + def _mrow(rank, m, note=""): + spd = f"{m['spread_bps']:.2f}" if m["spread_bps"] is not None else "N/A" + dep = ( + f"${m['book_depth_usd']:,.0f}" + if m.get("book_depth_usd") is not None + else "n/a" + ) + flag = f" [{note}]" if note else "" + score_str = f" | Score={m['score']:.3f}" if "score" in m else "" + return ( + f" {rank}. {m['pair']}: Vol=${m['volume']:,.0f} | Spread={spd}bps" + f" | Drift={m['daily_drift_pct']:.2f}% | Depth={dep}/side | OI=${m['oi_notional']:,.0f}" + f" | Lev={m['maxLeverage']}x{score_str}{flag}" + ) + + if no_survivors: + lines.append("WARNING: NONE PASSED FILTERS — top 5 by volume (informational):") + for rank, m in enumerate(fallback, 1): + lines.append(_mrow(rank, m, "NO FILTER PASS")) + else: + lines.append(f"TOP PICK: {top_pick}") + lines.append("") + for rank, m in enumerate(shortlist, 1): + lines.append(_mrow(rank, m)) + + summary = "\n".join(lines) + + # ── 8. Persistent report ────────────────────────────────────────────────── + from condor.reports import ReportBuilder + + builder = ReportBuilder(f"HIP-3 Scanner: {issuer_upper}") + builder.source("routine", "hip3_market_scanner").tags( + ["market-making", "hip3", issuer, "scanner"] + ) + builder.kpi("Markets Scanned", str(total_markets)) + builder.kpi("Survivors", str(len(survivors))) + builder.kpi("Top Pick", top_pick) + builder.kpi("Fee RT", f"{config.all_in_fee_bps_roundtrip}bps") + + display_list = shortlist if not no_survivors else fallback + note_col = "Score" if not no_survivors else "Note" + table_rows = [] + for rank, m in enumerate(display_list, 1): + table_rows.append( + { + "Rank": rank, + "Pair": m["pair"], + "24h Vol ($)": f"${m['volume']:,.0f}", + "Spread (bps)": ( + f"{m['spread_bps']:.2f}" if m["spread_bps"] is not None else "N/A" + ), + "Drift %": f"{m['daily_drift_pct']:.2f}%", + "Depth/side ($)": ( + f"${m['book_depth_usd']:,.0f}" + if m.get("book_depth_usd") is not None + else "n/a" + ), + "OI ($)": f"${m['oi_notional']:,.0f}", + "MaxLev": m["maxLeverage"], + note_col: f"{m['score']:.3f}" if "score" in m else "NO FILTER PASS", + } + ) + + if table_rows: + builder.table( + table_rows, + [ + "Rank", + "Pair", + "24h Vol ($)", + "Spread (bps)", + "Drift %", + "Depth/side ($)", + "OI ($)", + "MaxLev", + note_col, + ], + ) + + builder.markdown(summary) + builder.manual_order() + await builder.save() + + return summary diff --git a/agents/market_making_fly/routines/mm_dashboard.py b/agents/market_making_fly/routines/mm_dashboard.py new file mode 100644 index 000000000..8f53a950b --- /dev/null +++ b/agents/market_making_fly/routines/mm_dashboard.py @@ -0,0 +1,388 @@ +"""MM Dashboard: unified portfolio inventory + bot positions + controller performance. + +Consolidates three former routines into one: + • portfolio_scanner → Portfolio / inventory section (get_state + get_total_value) + • mm_bot_report → Bots / positions / PnL section (bot_orchestration.get_active_bots_status) + • bot_position_tracker → superseded (its executor-search grouping is replaced by the + strictly-richer bot_orchestration path used here) + +Bot data comes from client.bot_orchestration.get_active_bots_status(): + bots_data[bot_name]["performance"][ctrl_name]["performance"] + → realized_pnl_quote, unrealized_pnl_quote, volume_traded + → positions_summary (list of open positions with pair/side/amount/breakeven) + → close_type_counts (CloseType.XXX → count) + bots_data[bot_name]["error_logs"] → errors +""" + +import logging +from collections import Counter +from datetime import datetime, timezone + +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from config_manager import get_client + +logger = logging.getLogger(__name__) + +CATEGORY = "Bot Analysis" + +_CLOSE_LABELS = { + "TAKE_PROFIT": "TP", + "STOP_LOSS": "SL", + "TRAILING_STOP": "Trail", + "EARLY_STOP": "Early", + "POSITION_HOLD": "Hold", + "HOLD": "Hold", + "EXPIRED": "Expired", + "INSUFFICIENT_BALANCE": "Insuf$", + "FAILED": "Failed", + "UNKNOWN": "?", +} + +_ERROR_LEVELS = {"ERROR", "CRITICAL", "FATAL"} + + +def _close_label(raw: str) -> str: + """Convert 'CloseType.EARLY_STOP' → 'Early'.""" + code = raw.split(".")[-1] if "." in raw else raw + return _CLOSE_LABELS.get(code.upper(), code) + + +def _side_label(raw: str) -> str: + """Convert 'TradeType.BUY' → 'BUY'.""" + return raw.split(".")[-1] if "." in raw else raw + + +class Config(BaseModel): + """Unified MM dashboard: portfolio inventory, bot positions, PnL, and errors.""" + + connector_name: str = Field( + default="binance_perpetual", + description="Focus connector for portfolio (empty=all)", + ) + trading_pair: str = Field( + default="", description="Filter bot positions by pair (empty = all)" + ) + min_value_usd: float = Field( + default=1.0, description="Hide tokens below this USD value" + ) + include_errors: bool = Field(default=True, description="Include error log summary") + + +async def _fetch_portfolio(client, connector_name: str, min_value_usd: float): + """Scan portfolio inventory. Returns (inv_rows, total_value, error_msg, summary_lines).""" + state = None + errors = [] + try: + state = await client.portfolio.get_state() + except Exception as e: + errors.append(f"get_state: {e}") + + if not state or not isinstance(state, dict): + try: + state = await client.portfolio.get_portfolio_state() + except Exception as e: + errors.append(f"get_portfolio_state: {e}") + + if not state or not isinstance(state, dict): + return [], None, " | ".join(errors) or "No portfolio state available", [] + + inv_rows = [] + summary_lines = [] + for acct_name, acct_data in state.items(): + if not isinstance(acct_data, dict): + continue + for conn_name, tokens in acct_data.items(): + if connector_name and connector_name not in conn_name: + continue + if not isinstance(tokens, list): + continue + + significant = [ + t + for t in tokens + if isinstance(t, dict) and float(t.get("value", 0)) >= min_value_usd + ] + if not significant: + continue + + significant.sort(key=lambda t: float(t.get("value", 0)), reverse=True) + total_value = sum(float(t.get("value", 0)) for t in significant) + + summary_lines.append(f"**{acct_name} / {conn_name}**") + for t in significant: + token = t.get("token", "?") + units = float(t.get("units", 0)) + value = float(t.get("value", 0)) + available = float(t.get("available_units", units)) + pct = (value / total_value * 100) if total_value > 0 else 0 + in_use = units - available + use_flag = f" (in_use: {in_use:.4f})" if in_use > 0.001 else "" + summary_lines.append( + f" {token}: {units:,.4f} = ${value:,.2f} ({pct:.1f}%){use_flag}" + ) + inv_rows.append( + { + "Account": acct_name, + "Connector": conn_name, + "Token": token, + "Units": round(units, 4), + "Value (USD)": f"${value:,.2f}", + "Weight": f"{pct:.1f}%", + "In Use": f"{in_use:.4f}" if in_use > 0.001 else "-", + } + ) + summary_lines.append(f" Subtotal: ${total_value:,.2f}") + summary_lines.append("") + + total_val = None + try: + tv = await client.portfolio.get_total_value() + if tv: + total_val = float(tv) + except Exception: + pass + if total_val is None and inv_rows: + total_val = sum( + float(r["Value (USD)"].replace("$", "").replace(",", "")) for r in inv_rows + ) + + return inv_rows, total_val, None, summary_lines + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + client = await get_client(context._chat_id, context=context) + if not client: + return "No server available" + + # ── 1. Portfolio / Inventory ───────────────────────────────────────────── + inv_rows, total_value, portfolio_error, portfolio_summary = await _fetch_portfolio( + client, config.connector_name, config.min_value_usd + ) + + # ── 2. Bots / Positions / PnL (bot_orchestration — richer source) ──────── + bots_data: dict = {} + bots_error = None + try: + resp = await client.bot_orchestration.get_active_bots_status() + raw = resp if isinstance(resp, dict) else {} + bots_data = raw.get("data", raw) if isinstance(raw, dict) else {} + if not isinstance(bots_data, dict): + bots_data = {} + except Exception as e: + bots_error = f"Failed to fetch bot status: {e}" + + ctrl_rows = [] # per-controller perf table + pos_rows = [] # open positions table + close_totals: Counter = Counter() + total_realized = total_unrealized = total_volume = 0.0 + total_errors = 0 + error_lines = [] + + for bot_name, bot_data in bots_data.items(): + if not isinstance(bot_data, dict): + continue + + # Error logs + if config.include_errors: + errs = [ + e + for e in (bot_data.get("error_logs") or []) + if isinstance(e, dict) + and str(e.get("level_name", "")).upper() in _ERROR_LEVELS + ] + if errs: + total_errors += len(errs) + error_lines.append(f" • {bot_name}: {len(errs)} error(s)") + + perf_dict = bot_data.get("performance", {}) + if not isinstance(perf_dict, dict): + continue + + for ctrl_name, ctrl_data in perf_dict.items(): + if not isinstance(ctrl_data, dict): + continue + inner = ctrl_data.get("performance", ctrl_data) + if not isinstance(inner, dict): + continue + + # Parse open positions to get pair/connector for filtering + positions = inner.get("positions_summary") or [] + pair = "" + connector = "" + if ( + positions + and isinstance(positions, list) + and isinstance(positions[0], dict) + ): + pair = positions[0].get("trading_pair", "") + connector = positions[0].get("connector_name", "") + + # Filter by trading pair (empty = all controllers) + if config.trading_pair and config.trading_pair not in pair: + continue + + realized = float(inner.get("realized_pnl_quote", 0) or 0) + unrealized = float(inner.get("unrealized_pnl_quote", 0) or 0) + volume = float(inner.get("volume_traded", 0) or 0) + total_realized += realized + total_unrealized += unrealized + total_volume += volume + + # Close type breakdown + close_counts: dict = inner.get("close_type_counts", {}) or {} + ctrl_closes: Counter = Counter() + for raw_ct, cnt in close_counts.items(): + label = _close_label(str(raw_ct)) + ctrl_closes[label] += int(cnt) + close_totals.update(ctrl_closes) + close_str = ( + " | ".join(f"{k}:{v}" for k, v in ctrl_closes.most_common()) or "—" + ) + + ctrl_rows.append( + { + "Controller": ctrl_name, + "Pair": pair or "—", + "Realized PnL": f"${realized:,.4f}", + "Unrealized PnL": f"${unrealized:,.4f}", + "Volume": f"${volume:,.2f}", + "Closes": close_str, + } + ) + + # Open positions + for pos in positions: + if not isinstance(pos, dict): + continue + pos_pair = pos.get("trading_pair", pair) + pos_conn = pos.get("connector_name", connector) + side = _side_label(str(pos.get("side", ""))) + amount = float(pos.get("amount", 0) or 0) + bp = float(pos.get("breakeven_price", 0) or 0) + pos_unreal = float(pos.get("unrealized_pnl_quote", 0) or 0) + pos_real = float(pos.get("realized_pnl_quote", 0) or 0) + pos_rows.append( + { + "Controller": ctrl_name, + "Pair": pos_pair, + "Connector": pos_conn, + "Side": side, + "Amount": f"{amount:,.2f}", + "Breakeven": f"${bp:,.6f}", + "Unrealized PnL": f"${pos_unreal:,.4f}", + "Realized PnL": f"${pos_real:,.4f}", + } + ) + + net_pnl = total_realized + total_unrealized + + # ── Summary text ───────────────────────────────────────────────────────── + now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC") + close_summary = ( + " | ".join(f"{k}:{v}" for k, v in close_totals.most_common()) or "none" + ) + + lines = [f"**MM Dashboard** — {now}", ""] + + # Portfolio block + if portfolio_error and not inv_rows: + lines.append(f"Portfolio: {portfolio_error}") + else: + total_str = f"${total_value:,.2f}" if total_value else "unknown" + lines.append( + f"Portfolio total: {total_str} | {len(inv_rows)} asset(s) on {config.connector_name or 'all'}" + ) + if portfolio_summary: + lines.append("") + lines.extend(portfolio_summary) + + # Bots block + if bots_error: + lines.append(bots_error) + else: + lines.append(f"Controllers: {len(ctrl_rows)} | Open positions: {len(pos_rows)}") + lines.append( + f"Realized PnL: ${total_realized:,.4f} | Unrealized: ${total_unrealized:,.4f} | Net: ${net_pnl:,.4f}" + ) + lines.append(f"Volume: ${total_volume:,.2f}") + lines.append(f"Closes: {close_summary}") + + if config.include_errors and not bots_error: + if total_errors: + lines.append("") + lines.append(f"⚠️ {total_errors} error(s) across active bots:") + lines.extend(error_lines[:8]) + else: + lines.append("") + lines.append("✅ No errors in active bot logs") + + summary = "\n".join(lines) + + # ── Persistent report ───────────────────────────────────────────────────── + from condor.reports import ReportBuilder + + builder = ReportBuilder("MM Dashboard") + builder.source("routine", "mm_dashboard").tags( + ["portfolio", "bots", "positions", "market-making"] + ) + + if total_value: + builder.kpi("Portfolio Value", f"${total_value:,.2f}") + builder.kpi("Active Controllers", str(len(ctrl_rows))) + builder.kpi("Open Positions", str(len(pos_rows))) + builder.kpi("Realized PnL", f"${total_realized:,.4f}") + builder.kpi("Unrealized PnL", f"${total_unrealized:,.4f}") + builder.kpi("Net PnL", f"${net_pnl:,.4f}") + builder.kpi("Volume", f"${total_volume:,.2f}") + if config.include_errors: + builder.kpi("Errors", str(total_errors)) + + if pos_rows: + builder.table( + pos_rows, + [ + "Controller", + "Pair", + "Connector", + "Side", + "Amount", + "Breakeven", + "Unrealized PnL", + "Realized PnL", + ], + ) + if ctrl_rows: + builder.table( + ctrl_rows, + [ + "Controller", + "Pair", + "Realized PnL", + "Unrealized PnL", + "Volume", + "Closes", + ], + ) + if inv_rows: + builder.table( + inv_rows, + [ + "Account", + "Connector", + "Token", + "Units", + "Value (USD)", + "Weight", + "In Use", + ], + ) + elif portfolio_error: + builder.markdown(f"**Portfolio unavailable**: {portfolio_error}") + + builder.markdown(summary) + builder.manual_order() + await builder.save() + + return summary diff --git a/agents/market_making_fly/skills/capital_allocation/SKILL.md b/agents/market_making_fly/skills/capital_allocation/SKILL.md new file mode 100644 index 000000000..44c45c0bf --- /dev/null +++ b/agents/market_making_fly/skills/capital_allocation/SKILL.md @@ -0,0 +1,162 @@ +--- +name: capital_allocation +description: How total_amount_quote and initial_positions define a controller's isolated + capital — the budget a single pmm_mister controller trades with, and how to seed it + with base assets you already hold so each controller is independent from the wider portfolio. +when_to_use: When sizing a controller, deciding total_amount_quote, splitting one market + into several controllers, or when the user already holds the base asset (spot) and wants + to fund the strategy with existing inventory instead of buying fresh. Also read this + before answering "how much capital does this bot use" or "how do I use the BTC I already have". +source: agent:market_making_fly +--- + +# Capital Allocation: total_amount_quote & initial_positions + +This skill explains the two knobs that define **how much capital a single +controller trades with** and **which of that capital comes from assets you +already hold**. Together they make each controller's book independent from the +rest of your portfolio. + +--- + +## 1. `total_amount_quote` — the controller's budget + +`total_amount_quote` is the **total capital assigned to that one controller**, +denominated in the quote asset (e.g. USDC for a BTC-USDC market, BRL for a +BTC-BRL market). + +This is the reference amount everything else is measured against: + +- `portfolio_allocation` — fraction of `total_amount_quote` actively deployed + per iteration. +- `target_base_pct` / `min_base_pct` / `max_base_pct` — the inventory band. These + percentages are **percentages of `total_amount_quote`**, expressed in quote + value. If `total_amount_quote = 2000` and `target_base_pct = 50`, the target + base inventory is worth **$1,000** of the base asset. + +So `total_amount_quote` is the denominator. Change it and every absolute +position size, order size, and inventory band scales with it. It is the single +number that says "this controller is allowed to work with this much money." + +--- + +## 2. The default: starting fresh from quote + +By default a controller assumes it starts with **`total_amount_quote` worth of +quote asset and zero base**. It then builds up base inventory toward +`target_base_pct` by getting its buy orders filled — buying the base with quote +as the market comes to it. + +This is fine when you have plenty of quote and don't mind the controller +acquiring the base itself. Nothing extra is needed. + +--- + +## 3. `initial_positions` — seeding with assets you already hold + +If you are trading **spot and you already own the base asset**, you don't have to +make the controller buy it from scratch. You can hand existing inventory to the +controller at startup via `initial_positions`: + +```yaml +initial_positions: + - amount: 0.08328 + connector_name: binance + side: BUY + trading_pair: BTC-BRL +``` + +When you deploy through the normal flow, configs are upserted as a `config_data` +**dict** (`manage_controllers(action="upsert", target="config", ...)`), so +`initial_positions` is a **list of dicts** — the copy-paste-ready form is: + +```json +"initial_positions": [ + { + "amount": 0.08328, + "connector_name": "binance", + "side": "BUY", + "trading_pair": "BTC-BRL" + } +] +``` + +Both forms are equivalent — YAML for config files, JSON/dict for the +`manage_controllers` upsert path. + +What this means: + +- You declare **how many units of the base asset from your portfolio you want to + assign to this controller at the start** of the strategy. +- That amount goes **directly into position hold** — the controller starts + already holding this inventory as an open BUY position, instead of holding pure + quote. +- `side: BUY` marks it as a long base position that the strategy now manages + (its TP/SL and inventory logic apply to it just like a position it opened + itself). + +### You choose whether to use existing assets or not + +- **Don't assign them** → the controller starts fresh from quote. Use this when + you have enough quote to fund `total_amount_quote` on its own and you'd rather + leave your existing base untouched. You can hold assets and simply not use + them. +- **Assign them via `initial_positions`** → the controller starts with that base + inventory already in hand, counting toward its `target_base_pct` band. Use this + when you want your existing holdings to be the working inventory rather than + buying more. + +The base you assign should be consistent with the controller's budget: the quote +value of the assigned base is part of the `total_amount_quote` this controller +manages, so it counts toward the inventory band (`target/min/max_base_pct`). + +--- + +## 4. Why this matters: portfolio independence + +This is the mechanism that makes **each controller's book independent from the +overall portfolio**. Instead of one giant strategy over your whole balance, you +carve the portfolio into slices and hand each slice to its own controller with +its own `total_amount_quote` and its own seeded inventory. + +### Worked example — splitting a market into 10 controllers + +Say you hold **$10k USDC and $10k of BTC** (dollar value) and want to market-make +BTC-USDC. You can deploy **10 controllers**, each with: + +- `total_amount_quote = 2000` (2,000 USDC of budget per controller → 10 × 2,000 = + $20k total, matching your combined capital). +- A proportional slice of your existing BTC assigned via `initial_positions` — + i.e. split the BTC you hold across the 10 controllers so each starts with + ~1/10 of it as seeded base inventory. + +Now each controller runs its own isolated book on a $2,000 budget, half funded by +quote and half by the BTC you already had. The controllers don't fight over one +shared balance — each has a fixed, known slice, so their PnL, inventory bands, +and risk are measured independently. **This is how we make the strategy's capital +independent from the general portfolio.** + +--- + +## Quick reference + +| Concept | Meaning | +|---------|---------| +| `total_amount_quote` | Total capital assigned to **this one controller**, in quote units. The denominator for allocation and all base-pct bands. | +| `target/min/max_base_pct` | Inventory band as a % of `total_amount_quote`. | +| Default start | Controller assumes `total_amount_quote` in quote, 0 base — buys base itself. | +| `initial_positions` | Seed the controller with base you already hold; goes straight to position hold as a managed BUY. Optional. | +| Splitting a market | Deploy N controllers, each with `total_amount_quote = budget/N` and a proportional slice of existing base via `initial_positions` → N independent books. | + +### `initial_positions` fields + +- `amount` — units of the **base** asset to assign (e.g. `0.08328` BTC). +- `connector_name` — the connector holding the asset (e.g. `binance`). +- `side` — `BUY` for a long base position the strategy will manage. +- `trading_pair` — the controller's pair (e.g. `BTC-BRL`). + +Only applies to **spot** with existing base inventory. It's optional — assign +existing assets when you want them to be the working inventory; omit it to start +fresh from quote. + + diff --git a/agents/market_making_fly/skills/fly_decoder/SKILL.md b/agents/market_making_fly/skills/fly_decoder/SKILL.md new file mode 100644 index 000000000..40927b42d --- /dev/null +++ b/agents/market_making_fly/skills/fly_decoder/SKILL.md @@ -0,0 +1,64 @@ +--- +name: fly_decoder +description: How spike counts become a quoting posture, how to read fly_status, and what + each number can and cannot tell you. +when_to_use: When explaining a posture, answering "why did the fly widen/lean/pause", + "is it learning", or checking whether a run is healthy. +created: '2026-09-12T00:00:00Z' +source: agent:market_making_fly +references_routine: fly_status +--- + +# Reading the fly + +Run `manage_routines(action="run", name="fly_status", config={"run_name": ""})`. + +## Channels (per observation, 500 ms of neural time) + +| Field | Cells | Meaning | +|---|---|---| +| `trend_hz` | DNp20 right mean rate − left mean rate | lean direction; stonkfly's BUY/SELL cells | +| `arousal_hz` | mean rate of the 1,314 descending neurons (minus the readouts) | spread width | +| `gate_spikes` | DNpe017 | ≥ 1 required for a trending call and for any lean | +| `kc_spikes` | Kenyon cells | did the chart reach the mushroom body at all (0 = the fly saw nothing useful) | +| `reward_spikes` / `aversive_spikes` | PAM11 / PPL101 | did the pulse arrive | + +## Posture + +* `trend_z`, `arousal_z`: the channel minus its rolling per-pair mean, over its std, + window 60 observations. `warm=False` for the first 10 — neutral posture. +* Regime precedence: `pause` (arousal_z ≥ 2.5) > `volatile` (≥ 1) > `trending_up/down` + (gate and |trend_z| ≥ 1) > `quiet` (arousal_z ≤ −1) > `ranging`. +* `spread ×` = clip(1 + 0.5·arousal_z, 0.6, 2.5). `lean` = clip(trend_z, ±3 bp), 0 without + a gate spike, and capped at half the first spread level when mapped. +* Mapping: level 1 = max(2, S/2) bp, level 2 = S+1 bp (S = scanner spread), times + `spread ×`, buy −lean / sell +lean, floor 3 bp; TP = max(4 bp, 2.2 × round-trip fee, + first level); timing per regime; `pause` sets `manual_kill_switch`. + +## Execution statuses in the event log + +`SHADOW` would have applied (shadow mode) · `APPLIED` live update done · `HOLD` no +material change or per-pair 5-minute cooldown · `VETO` guard refused (reason given) · +`CLOSED` book closed, fly did not observe · `STOP_BOT` closed ≥ 5 ticks · `ERROR` +update failed · `HALT` loop stopped · `TICK_ERROR` data fetch failed, loop continues. + +## What you may say + +* "The fly's arousal channel is 1.8 σ above its baseline on DRAM, so it widened to 1.9×." +* "No gate spike this observation, so no lean regardless of trend." +* "Changed edges rose from 0 to 312 after the first aversive pulse." + +## What you may not say + +* That the fly detected a regime: it emitted a z-score we labelled. +* That a P&L pulse taught it anything: pulses are value feedback, not credit assignment, + and edges change from endogenous dopamine activity too. +* That a run with positive P&L shows the fly works: a rising market makes any long + inventory look skilled. No held-out replay or shuffled-reinforcement control exists. + +## Health checks + +* `kc_spikes` stays 0 → the chart is not activating the mushroom body; report it. +* `baseline_n` not growing for a pair → that pair's book is closed or its feed fails. +* `halted` set → read the reason; financial halts need a new `run_name`. +* `compute_seconds` above ~40 % of the interval → lower `neural_ms`. diff --git a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md new file mode 100644 index 000000000..228d4ff21 --- /dev/null +++ b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md @@ -0,0 +1,117 @@ +--- +name: fly_mm_deploy +description: End-to-end deployment of the fly market maker on up to three HIP-3 pairs — + scan, deploy neutral pmm_mister bots with the fly's naming, start fly_brain in shadow, + verify, and (only when told) switch to live. +when_to_use: When asked to set up, deploy, launch, or restart the fly market maker, or + to rotate one of its market slots. Follow it as a delegate task with no mid-flow + confirmation. +created: '2026-09-12T00:00:00Z' +source: agent:market_making_fly +--- + +# Fly MM Deploy + +You are deploying **Market Making Fly**: one shared fly brain, up to three HIP-3 +markets, `pmm_mister` controllers. The fly decides posture; you set up the plumbing. + +## Step 1 — Pick the markets + +``` +manage_routines(action="run", name="hip3_market_scanner", + config={"issuer": "xyz", "min_spread_bps": 3, "max_daily_drift_pct": 3, "top_n": 5}) +``` + +Take up to **three** survivors from the top of the ranking that have an open live +book. Record for each: `pair` (uppercase, e.g. `XYZ:DRAM-USD`) and its **spread in +bp** — this is `picked_spreads_bps`. If fewer than one survivor, stop and report. + +## Step 2 — Collateral + +`get_portfolio_overview(["hyperliquid_perpetual"])` → available USD. Required ≈ +`Σ total_amount_quote × 0.5 / leverage` across the pairs. If short, reduce +`total_amount_quote` or drop a pair; say so in the report. + +## Step 3 — Neutral configs (the fly's starting point) + +For each pair derive `token` (e.g. `DRAM`), `bot_name = {token.lower()}-fly`, +`config_name = {token.lower()}_fly_mm`. The neutral config is exactly what +`fly_brain` would apply for the `ranging` regime; build it with: + +```python +run_code(code=""" +from condor.fly.posture import MarketSpec, build_config +from condor.fly.decoder import NEUTRAL +spec = MarketSpec(connector_name="hyperliquid_perpetual", trading_pair="XYZ:DRAM-USD", + total_amount_quote=500, picked_spread_bps=8.0, leverage=3) +print(build_config(spec, NEUTRAL)) +""") +``` + +Then save it: + +``` +manage_controllers(action="upsert", target="config", config_name="dram_fly_mm", + config_data={...printed config...}, confirm_override=True) +``` + +Do not edit the spreads, TP, bands or stop loss by hand — the floors live in code. + +## Step 4 — Deploy the bots + +One bot per pair, named exactly `{token}-fly`, with a loss cap: + +``` +manage_bots(action="deploy", bot_name="dram-fly", controllers_config=["dram_fly_mm"], + max_global_drawdown_quote=<0.04 × total_amount_quote>) +``` + +Confirm with `manage_bots(action="status")` that each bot is running with its +controller. + +## Step 5 — Start the fly in shadow + +``` +manage_routines(action="start", name="fly_brain", config={ + "pairs": "XYZ:DRAM-USD,XYZ:SPCX-USD,XYZ:SMSN-USD", + "picked_spreads_bps": "8,6,10", + "total_amount_quote": 500, "leverage": 3, + "mode": "shadow", "run_name": "fly-2026-09-12"}) +``` + +Shadow observes, decodes and reinforces from the bots' P&L but applies nothing. Note +the instance id. After ~10 observations per pair (30 ticks) `fly_status` shows +non-neutral postures. + +## Step 6 — Verify + +``` +manage_routines(action="run", name="fly_status", config={"run_name": "fly-2026-09-12"}) +manage_routines(action="run", name="mm_bot_report", config={}) +``` + +Report: pairs, spreads, bots running, fly tick count, first postures, any vetoes or +halts, and the caveat that the fly's learning is not validated. + +## Step 7 — Live (only when the task says so) + +Stop the shadow instance (`manage_routines(action="stop", name="")`) and +start again with `"mode": "live"` and the **same** `run_name` — the baseline and the +brain's memory carry over. Never start live on a fresh `run_name` without a shadow +period first. + +## Rotation + +When a slot's market is closed, dominated, or the operator asks: stop that bot +(`manage_bots(action="stop_bot", bot_name=...)`), re-run the scanner, deploy the new +pick with Steps 3–4, stop `fly_brain` and start it again with the updated `pairs` and +`picked_spreads_bps` and the same `run_name`. The brain keeps its memory; only the +swapped pair's baseline starts over. + +## Halts + +`fly_brain` stops itself and (in live) stops the bots on a halt. A **transient** halt +(repeated config-update failures) restarts with `"resume_reviewed": true` after you +have looked at the bot logs. A **financial** halt (loss stop, loss-rate breaker, no +new P&L high) cannot be cleared: report it, leave the bots stopped, and only redeploy +under a new `run_name` if the operator asks. diff --git a/agents/market_making_fly/skills/mm_bot_report/SKILL.md b/agents/market_making_fly/skills/mm_bot_report/SKILL.md new file mode 100644 index 000000000..6b5c760e6 --- /dev/null +++ b/agents/market_making_fly/skills/mm_bot_report/SKILL.md @@ -0,0 +1,40 @@ +--- +name: mm_bot_report +description: 'Run the MM bot status report: running bots, open/hold-mode positions, + closed position breakdown (TP/SL/Early/Hold), PnL, volume, and error summary.' +when_to_use: When the user asks for a bot status report, "how is the bot doing", "show + me the report", "what's the PnL", "any errors", "how are positions", "closed positions + breakdown", or any general health/status check on the running MM bots. Also use + after deploying a new bot to verify it's running correctly. +created: '2026-07-02T15:46:08Z' +source: agent:market_making_fly +references_routine: mm_bot_report +--- + +## MM Bot Report + +Run the `mm_bot_report` routine — it fetches everything in one shot: + +``` +manage_routines(action="run", name="mm_bot_report", config={}) +``` + +**What it returns:** +- **Controllers** — active controller count +- **Open positions** — active executors currently placing quotes (`is_trading=True`) +- **Hold-mode** — active executors paused/holding inventory (`is_trading=False`) +- **Recent closes breakdown** — by close type: TP | SL | Early | Hold | Trail | Expired +- **PnL & Volume** — realized + unrealized PnL per controller, total volume +- **Error summary** — error count per active bot from live logs + +**Config overrides** (pass as `config={}` keys): +- `trading_pair` — filter to one pair (default: all) +- `connector_name` — filter to one connector (default: all) +- `recent_closes` — how many closed executors to analyze (default: 100) +- `include_errors` — set `false` to skip error log fetch (default: true) + +**After reading the output:** +1. Surface the KPIs (open positions, hold-mode count, top close type). +2. Flag any errors — if errors are present, note the bot name and count. For deeper log analysis run `manage_routines(action="run", name="logs_summary")` (global routine). +3. If hold-mode > 0 and user hasn't set it intentionally, flag it — positions holding inventory aren't earning spread. +4. Summarize PnL vs volume to comment on fee efficiency. diff --git a/agents/market_making_fly/skills/pmm_config_playbook/SKILL.md b/agents/market_making_fly/skills/pmm_config_playbook/SKILL.md new file mode 100644 index 000000000..b3dd2b8d2 --- /dev/null +++ b/agents/market_making_fly/skills/pmm_config_playbook/SKILL.md @@ -0,0 +1,106 @@ +--- +name: pmm_config_playbook +description: Ready-to-deploy pmm_mister config profiles (aggressive / balanced / conservative) + — full parameter coverage including spreads, effectivization times, tolerance, order + types, skew, and global TP/SL. +when_to_use: When you need a starting pmm_mister controller config and want a vetted + template instead of hand-tuning every parameter — pick a profile by regime, fetch + its template, then adapt the connector/pair/amount. +source: builtin +--- + +# pmm_mister Config Playbook + +Three vetted `pmm_mister` profiles, one per risk posture. Each profile lives in a +**companion file** — fetch only the one you need so the others never load into +context: + +``` +manage_skill(action="read_file", name="pmm_config_playbook", file="config_aggressive.md") +``` + +## Pick a profile by regime + +| Regime | Profile | File | +|-----------------------------------------|------------------|-----------------------------|\n| Quiet / low-vol ranging (ADX < 18) | **Aggressive** | `config_aggressive.md` | +| Ranging / normal (ADX < 25) | **Balanced** | `config_balanced.md` | +| Volatile / trending / uncertain | **Conservative** | `config_conservative.md` | + +- **Aggressive** — tight spreads, fast refresh, short effectivization (60s), + wide inventory bands, high allocation. Maximizes fill rate in calm markets. + Most inventory/PnL risk. +- **Balanced** — the default. Moderate spreads, 120s effectivization, standard + tolerances. Good steady-state when regime is unclear. +- **Conservative** — wide spreads, slow refresh, long effectivization (300s), + tight inventory bands, strong skew enforcement (min_skew=2.0), low + allocation/leverage, both global TP and SL active. Capital preservation in + chop/vol. + +## How to use a template + +1. Read the chosen companion file. It contains a full `config_data` block for + `manage_controllers(action="upsert", target="config")`. +2. **Always adapt** these to the actual operation before deploying: + - `connector_name`, `trading_pair` + - `total_amount_quote` (respect the strategy's risk limit) + - `leverage` (never above the strategy's cap; templates default low) +3. Deploy via the normal flow (`manage_controllers` upsert → `manage_bots` deploy). + Live retunes go through `manage_bots(action="update_config", confirm_override=true)`. + +## Key parameter reference + +Parameters most commonly tuned in real operations: + +**Inventory & allocation** +- `portfolio_allocation` — fraction of `total_amount_quote` actively deployed per iteration +- `target_base_pct` / `min_base_pct` / `max_base_pct` — inventory band; skew + kicks in when base drifts outside min/max +- `min_skew` — minimum spread multiplier applied to the heavy side when inventory + drifts; 1.0 = no minimum, 2.0 = at least 2× wider on the accumulating side +- `max_active_executors_by_level` — max concurrent open executors per level; + controls total directional exposure (fills × allocation per level) + +**Order timing** +- `executor_refresh_time` — how often (seconds) the controller checks and potentially + replaces open orders; lower = tighter to mid price but more rate limit usage +- `buy/sell_cooldown_time` — after a fill, how long to wait before placing a new order + on that side; lower = faster re-entry, more risk of accumulating at similar prices +- `buy/sell_position_effectivization_time` — how long (seconds) the per-fill + LIMIT_MAKER TP order stays on the book after a fill; when this expires, the TP + order is **removed** and the position transitions to "hold" mode managed only by + the global SL/TP layer. Lower = TP has less time to fill → positions accumulate + into hold faster. Higher = TP order stays on book longer → more chances of hitting TP. + +**Spread & refresh tolerance** +- `price_distance_tolerance` — minimum price gap required between open orders at + the same level; prevents stacking orders too close together +- `refresh_tolerance` — minimum mid-price move required to trigger a quote + refresh/replacement; lower = more responsive, higher cancel/replace churn +- `tolerance_scaling` — multiplier applied to tolerance values as the number of + active executors grows; prevents cancel-loops when multiple orders are open at + the same level + +**Order types** (3 = LIMIT_MAKER / post-only, 2 = LIMIT, 1 = MARKET) +- `open_order_type` — order type for entry orders (always use 3 unless exchange rejects) +- `take_profit_order_type` — order type for TP orders (always 3) + +**Per-fill risk** +- `take_profit` — offset from fill price where the LIMIT_MAKER TP order is placed; + must be > round-trip fees to be profitable + +**Global portfolio guardrails (position hold phase)** +- `global_tp_enabled` / `global_take_profit` — when total held position gains this %, close and restart MM +- `global_sl_enabled` / `global_stop_loss` — when total held position loses this %, close and restart MM +- `global_sl_activation_from` — inventory threshold from which global SL activates + (`"min_base"` = when base is light, `"target_base"` = from neutral) +- `global_tp_activation_from` — inventory threshold from which global TP activates +- `global_pnl_reference` — PnL basis: `"position"` (unrealized open PnL) or + `"portfolio"` (total portfolio value) +- `position_profit_protection` — blocks inventory reduction at unfavorable prices + +**Other** +- `tick_mode` — if true, controller runs only on candle ticks; false = continuous (default) + +The full regime→param decision logic lives in the `pmm_mister_operator` strategy. +These templates are starting points — every deploy still goes through normal +risk/confirmation controls. diff --git a/agents/market_making_fly/skills/pmm_config_playbook/config_aggressive.md b/agents/market_making_fly/skills/pmm_config_playbook/config_aggressive.md new file mode 100644 index 000000000..3a1fa49dc --- /dev/null +++ b/agents/market_making_fly/skills/pmm_config_playbook/config_aggressive.md @@ -0,0 +1,76 @@ +# Aggressive pmm_mister Profile + +**Use when:** quiet / low-volatility ranging market (ADX < 18, BBW < ~3%). You +want maximum fill rate and volume capture. **Highest** inventory + PnL risk — +do NOT use in trending or volatile regimes. + +Tight spreads, fast order refresh, short cooldowns and effectivization times, +wide inventory tolerance, higher allocation. + +```json +{ + "controller_type": "generic", + "controller_name": "pmm_mister", + "connector_name": "binance_perpetual", + "trading_pair": "JTO-USDT", + "total_amount_quote": 500, + "portfolio_allocation": 0.25, + "target_base_pct": 0.5, + "min_base_pct": 0.3, + "max_base_pct": 0.7, + "buy_spreads": "0.0008,0.0015", + "sell_spreads": "0.0008,0.0015", + "buy_amounts_pct": "1,1", + "sell_amounts_pct": "1,1", + "executor_refresh_time": 20, + "buy_cooldown_time": 30, + "sell_cooldown_time": 30, + "buy_position_effectivization_time": 60, + "sell_position_effectivization_time": 60, + "price_distance_tolerance": 0.0005, + "refresh_tolerance": 0.0003, + "tolerance_scaling": 1.1, + "open_order_type": 3, + "take_profit": 0.0008, + "take_profit_order_type": 3, + "leverage": 10, + "position_mode": "ONEWAY", + "position_side": "BUY", + "max_active_executors_by_level": 4, + "tick_mode": false, + "min_skew": 1.0, + "global_tp_enabled": false, + "global_sl_enabled": true, + "global_stop_loss": 0.05, + "global_sl_activation_from": "target_base", + "global_pnl_reference": "position" +} +``` + +**Parameter notes** +- `buy/sell_position_effectivization_time` (60s): The per-fill LIMIT_MAKER TP + order stays on the book for only 60s after each fill. In a quiet, low-drift + market the price barely moves, so the TP is unlikely to fill in that window — + positions quickly transition to hold mode. This is intentional: in calm + conditions we let fills accumulate into a held position rather than chasing + individual TPs. global_tp is disabled here, so held positions grow until the + global SL triggers. +- `price_distance_tolerance` (0.0005): Minimum gap required between stacked + orders at the same level. Keeps orders spread out to avoid clustering. +- `refresh_tolerance` (0.0003): Tighter than default — triggers a quote + refresh/replacement with smaller mid-price moves. More responsive in calm + markets where small moves matter. +- `tolerance_scaling` (1.1): Low multiplier — tolerance widens slowly as + executors accumulate. Stay close to mid. +- `open_order_type` / `take_profit_order_type` (3 = LIMIT_MAKER): Post-only. + Never takes liquidity. Change to 2 (LIMIT) only if the exchange rejects makers. +- `tick_mode` (false): Keep false for continuous market making. +- `min_skew` (1.0): No minimum skew enforced — spreads stay symmetric when + inventory is balanced. + +**Tuning notes** +- If fills are too one-sided, narrow the inventory band (raise `min_base_pct` / + lower `max_base_pct`) so skew kicks in sooner. +- If the market starts trending, switch to **balanced** or **conservative** — + tight two-sided spreads bleed into a trend. +- `global_sl_enabled` stays on even here: 5% hard stop is the floor. diff --git a/agents/market_making_fly/skills/pmm_config_playbook/config_balanced.md b/agents/market_making_fly/skills/pmm_config_playbook/config_balanced.md new file mode 100644 index 000000000..e18f149d0 --- /dev/null +++ b/agents/market_making_fly/skills/pmm_config_playbook/config_balanced.md @@ -0,0 +1,70 @@ +# Balanced pmm_mister Profile + +**Use when:** normal ranging market (ADX < 25, moderate BBW). This is the +**default** steady-state profile — moderate spreads, allocation and cooldowns. +Good when no regime signal is strong enough to justify aggressive or +conservative. + +```json +{ + "controller_type": "generic", + "controller_name": "pmm_mister", + "connector_name": "binance_perpetual", + "trading_pair": "JTO-USDT", + "total_amount_quote": 500, + "portfolio_allocation": 0.15, + "target_base_pct": 0.5, + "min_base_pct": 0.35, + "max_base_pct": 0.65, + "buy_spreads": "0.0012,0.0025", + "sell_spreads": "0.0012,0.0025", + "buy_amounts_pct": "1,1", + "sell_amounts_pct": "1,1", + "executor_refresh_time": 30, + "buy_cooldown_time": 60, + "sell_cooldown_time": 60, + "buy_position_effectivization_time": 120, + "sell_position_effectivization_time": 120, + "price_distance_tolerance": 0.0005, + "refresh_tolerance": 0.0005, + "tolerance_scaling": 1.2, + "open_order_type": 3, + "take_profit": 0.001, + "take_profit_order_type": 3, + "leverage": 8, + "position_mode": "ONEWAY", + "position_side": "BUY", + "max_active_executors_by_level": 3, + "tick_mode": false, + "min_skew": 1.5, + "global_tp_enabled": false, + "global_sl_enabled": true, + "global_stop_loss": 0.05, + "global_sl_activation_from": "target_base", + "global_pnl_reference": "position" +} +``` + +**Parameter notes** +- `buy/sell_position_effectivization_time` (120s): The per-fill LIMIT_MAKER TP + order stays on the book for 2 minutes after each fill. If the market moves + enough to hit the TP in that window, the position closes with a per-fill profit. + If not, the position transitions to hold mode after 120s and is managed by + the global SL layer. Balanced between giving the TP time to fill and not + leaving stale positions open indefinitely. +- `price_distance_tolerance` / `refresh_tolerance` (0.0005): Controller defaults. + Balanced refresh cadence — not too aggressive, not too slow. +- `tolerance_scaling` (1.2): Default multiplier. Tolerance widens moderately + as executors accumulate, preventing cancel-loops in ranging markets. +- `open_order_type` / `take_profit_order_type` (3 = LIMIT_MAKER): Post-only + orders. Change to 2 (LIMIT) only if the exchange rejects makers. +- `tick_mode` (false): Keep false for continuous market making. +- `min_skew` (1.5): Enforces a minimum 1.5× spread multiplier on the heavy side + when inventory drifts. Mild protection against runaway accumulation. + +**Tuning notes** +- Start here when unsure, then shift toward aggressive (calm) or conservative + (vol/trend) as the regime clarifies. +- For a mild trend, make spreads asymmetric: widen the side you don't want to + trade into (e.g. wider `sell_spreads` in an uptrend). +- Increase `min_skew` to 2.0+ if inventory keeps drifting despite the band. diff --git a/agents/market_making_fly/skills/pmm_config_playbook/config_conservative.md b/agents/market_making_fly/skills/pmm_config_playbook/config_conservative.md new file mode 100644 index 000000000..559900e48 --- /dev/null +++ b/agents/market_making_fly/skills/pmm_config_playbook/config_conservative.md @@ -0,0 +1,92 @@ +# Conservative pmm_mister Profile + +**Use when:** volatile, trending, or uncertain market (ATR expanding, BBW > ~6%, +ADX > 25, volume surge). Capital preservation first — wide spreads, slow +refresh, tight inventory bands, long effectivization times, low +allocation/leverage, and **both** global TP and SL protections enabled. + +```json +{ + "controller_type": "generic", + "controller_name": "pmm_mister", + "connector_name": "binance_perpetual", + "trading_pair": "JTO-USDT", + "total_amount_quote": 500, + "portfolio_allocation": 0.1, + "target_base_pct": 0.5, + "min_base_pct": 0.4, + "max_base_pct": 0.6, + "buy_spreads": "0.003,0.006", + "sell_spreads": "0.003,0.006", + "buy_amounts_pct": "1,1", + "sell_amounts_pct": "1,1", + "executor_refresh_time": 60, + "buy_cooldown_time": 120, + "sell_cooldown_time": 120, + "buy_position_effectivization_time": 300, + "sell_position_effectivization_time": 300, + "price_distance_tolerance": 0.001, + "refresh_tolerance": 0.001, + "tolerance_scaling": 1.3, + "open_order_type": 3, + "take_profit": 0.0015, + "take_profit_order_type": 3, + "leverage": 5, + "position_mode": "ONEWAY", + "position_side": "BUY", + "max_active_executors_by_level": 2, + "tick_mode": false, + "min_skew": 2.0, + "position_profit_protection": true, + "global_tp_enabled": true, + "global_take_profit": 0.03, + "global_tp_activation_from": "min_base", + "global_sl_enabled": true, + "global_stop_loss": 0.04, + "global_sl_activation_from": "target_base", + "global_pnl_reference": "position" +} +``` + +**Parameter notes** +- `buy/sell_position_effectivization_time` (300s): The per-fill LIMIT_MAKER TP + order stays on the book for 5 minutes after each fill. In volatile markets + with frequent wicks, this gives the TP more time to be hit — individual fills + get closed profitably before the position transitions to hold. If the TP is + not hit in 300s, the position enters hold mode and both global TP (3%) and + global SL (4%) take over risk management. +- `price_distance_tolerance` / `refresh_tolerance` (0.001): Wider than default. + Avoids over-refreshing when price is moving constantly — reduces + cancel/replace churn and fees in volatile conditions. +- `tolerance_scaling` (1.3): Higher multiplier — tolerance grows faster per + executor so the controller doesn't thrash in choppy conditions. +- `open_order_type` / `take_profit_order_type` (3 = LIMIT_MAKER): Post-only. + Never takes liquidity — critical in volatile markets to avoid adverse fills. +- `tick_mode` (false): Keep false. Tick mode reduces update frequency but adds + complexity not needed here. +- `min_skew` (2.0): Forces at least 2× spread multiplier on the accumulating + side when inventory drifts. Aggressively discourages one-sided fills in + trending conditions. +- `position_profit_protection` (true): Blocks inventory reductions at + unfavorable prices — won't dump positions into a spike. +- `global_tp_enabled` / `global_take_profit` (3%): Portfolio-level TP. When + the held position's PnL crosses +3%, the controller begins winding down. +- `global_tp_activation_from` ("min_base"): TP activates when base inventory + is at or below `min_base_pct` — when the portfolio is light on base and + already showing profit. +- `global_sl_activation_from` ("target_base"): SL activates when base inventory + is at or above `target_base_pct` — protecting against heavy accumulation + losing value. +- `global_pnl_reference` ("position"): PnL is measured against the current + open position value (unrealized). Use "portfolio" to measure against total + portfolio value instead. + +**Tuning notes** +- In extreme volatility, drop `portfolio_allocation` further or pause the bot + entirely rather than widening spreads indefinitely. +- `position_profit_protection` blocks reductions at unfavorable prices — keep + it on so the controller won't dump inventory into a spike. +- Tighter `global_stop_loss` (4%) than the other profiles: cut losers faster + when the regime is hostile. +- If wicks keep triggering the SL, increase `global_stop_loss` slightly or + widen `buy/sell_spreads` so entry prices have more buffer. diff --git a/agents/market_making_fly/strategies/fly_hip3_operator/strategy.md b/agents/market_making_fly/strategies/fly_hip3_operator/strategy.md new file mode 100644 index 000000000..a77661b40 --- /dev/null +++ b/agents/market_making_fly/strategies/fly_hip3_operator/strategy.md @@ -0,0 +1,49 @@ +--- +name: Fly HIP-3 Operator +description: Keeps the fly market maker alive on its HIP-3 slots — bots up, fly_brain + running, halts surfaced, closed markets rotated. Never sets a posture itself. +agent_key: null +skills: [] +default_config: + frequency_sec: 300 + total_amount_quote: 500 + execution_mode: loop + run_name: fly + risk_limits: + max_position_size_quote: 600 + max_open_executors: 12 +default_trading_context: '' +created_by: 456181693 +created_at: '2026-09-12T00:00:00+00:00' +--- + +# Fly HIP-3 Operator + +You are Market Making Fly's operator loop. The fly (`fly_brain`) quotes; you keep +the plumbing healthy. **You never choose spreads, lean or regime.** + +## Each tick + +1. `manage_routines(action="run", name="fly_status", config={"run_name": ""})` + and `manage_routines(action="list_instances")`. +2. **Is the fly running?** If no `fly_brain` instance for this run and the status is + not halted → start it again with the same `run_name` and the pairs/spreads recorded + in your journal (shadow or live, whichever it was). Journal the restart. +3. **Is it halted?** Financial halt → journal, notify, do nothing else (bots are + already stopped by the fly). Transient halt → read `manage_bots(action="logs")` for + the failing bot, journal what you found, restart `fly_brain` with + `resume_reviewed: true` only if the cause is clearly external and resolved. +4. **Are the bots up?** `manage_bots(action="status")`. A missing `{token}-fly` bot for + a pair the fly is running → redeploy it from the saved `{token}_fly_mm` config + (`fly_mm_deploy` Step 4). The fly reads P&L by those names. +5. **Closed / dead slot?** A pair with `STOP_BOT` or many `CLOSED` ticks and a flat + position → follow `fly_mm_deploy` Rotation: scanner, new pick, redeploy, restart + `fly_brain` with the updated pairs and the same `run_name`. +6. **Report** in key: value lines: tick, mode, halted, per-pair regime / spread × / + lean / bot state, changed edges, session P&L vs high, and the standing caveat that + the fly's learning is not validated. + +## Never +- Update a controller's spreads, TP, bands or leverage yourself while `fly_brain` runs. +- Start `mode=live` unless the journal or the trading context says the operator asked. +- Clear a financial halt. diff --git a/condor/fly/__init__.py b/condor/fly/__init__.py new file mode 100644 index 000000000..f1ab1e8bd --- /dev/null +++ b/condor/fly/__init__.py @@ -0,0 +1,6 @@ +"""Market Making Fly — a fly-connectome simulation that proposes a quoting posture. + +The neural substrate (``condor.fly.neural``) is vendored from stonkfly; the +chart, decoder, posture mapping, guard, reinforcement and worker are Condor's. +See ``docs/market_making_fly_design.md``. +""" diff --git a/condor/fly/__main__.py b/condor/fly/__main__.py new file mode 100644 index 000000000..5c3831c28 --- /dev/null +++ b/condor/fly/__main__.py @@ -0,0 +1,72 @@ +"""``uv run python -m condor.fly prepare | verify | bench``. + +``prepare`` downloads the MaleCNS v1.0 release files (~1.1 GB), verifies their +checksums, compiles the retained graph and builds the C++ kernel. ``verify`` +re-checks the prepared arrays. ``bench`` loads the brain and times a few +observations on a synthetic frame so ``fly_brain``'s ``neural_ms`` can be sized +against its wall interval. +""" + +from __future__ import annotations + +import argparse +import json +import sys +import time + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(prog="python -m condor.fly", description=__doc__) + sub = parser.add_subparsers(dest="command", required=True) + sub.add_parser("prepare", help="download, verify and compile the connectome") + sub.add_parser("verify", help="re-check the prepared connectome arrays") + bench = sub.add_parser("bench", help="time observations on a synthetic frame") + bench.add_argument("--observations", type=int, default=3) + bench.add_argument("--neural-ms", type=float, default=500.0) + args = parser.parse_args(argv) + + from condor.fly.neural.common import DATA + + if args.command == "prepare": + from condor.fly.data import prepare + + print(f"Data directory: {DATA}", flush=True) + prepare() + from condor.fly.neural.brain import build + + print(json.dumps({"kernel": build()["model"], "data": str(DATA)})) + return 0 + if args.command == "verify": + from condor.fly.data import verify + + print(json.dumps({**verify(), "data": str(DATA)})) + return 0 + if args.command == "bench": + import numpy as np + + from condor.fly.worker import FlyBrain + + started = time.perf_counter() + brain = FlyBrain(learning=True) + load = time.perf_counter() - started + frame = np.full((180, 320, 3), 235, np.uint8) + rows = [] + for _ in range(args.observations): + result = brain.observe(frame, "none", neural_ms=args.neural_ms) + rows.append( + { + "compute_seconds": round(result["compute_seconds"], 3), + "total_spikes": result["total_spikes"], + "kc_spikes": result["kc_spikes"], + } + ) + print( + json.dumps({"load_seconds": round(load, 2), "observations": rows}, indent=2) + ) + return 0 + parser.error("unknown command") + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/condor/fly/chart.py b/condor/fly/chart.py new file mode 100644 index 000000000..ae90e1a4f --- /dev/null +++ b/condor/fly/chart.py @@ -0,0 +1,150 @@ +"""Render OHLCV candles into the 320×180 RGB frame the fly looks at. + +Same canvas, light background and palette as stonkfly's ``display.market_frame`` +so the vendored retinal projection is unchanged: R1–R6 cells read luminance, +the mapped R8p cells read the blue channel and R8y the green channel. Up +candles are blue and down candles red for that reason — colour is the only +chromatic input the fly gets, not decoration. + +The frame never contains the bot's own quotes, inventory or P&L. Portfolio +value reaches the fly only through the dopamine pulse (``reinforcement.py``). +""" + +from __future__ import annotations + +import math +from collections.abc import Sequence +from typing import Any + +import numpy as np +from PIL import Image, ImageDraw + +WIDTH, HEIGHT = 320, 180 +N_CANDLES = 72 + +BACKGROUND = (235, 240, 249) +HEADER = (19, 36, 71) +HEADER_TEXT = (219, 229, 249) +GRID = (200, 212, 233) +UP = (0, 101, 183) +DOWN = (197, 37, 78) +VOLUME = (120, 140, 180) +TEXT = (28, 46, 82) +TICK = (27, 39, 81) + +PLOT_LEFT, PLOT_RIGHT = 12, 306 +PLOT_TOP, PLOT_BOTTOM = 34, 132 +VOLUME_TOP, VOLUME_BOTTOM = 138, 158 +FOOTER_Y = 165 + +_FIELDS = ("open", "high", "low", "close", "volume") + + +def _finite_positive(value: Any, name: str) -> float: + number = float(value) + if not math.isfinite(number) or number <= 0: + raise ValueError(f"{name} must be a finite positive number, got {value!r}") + return number + + +def normalize_candles(candles: Sequence[Any], n_candles: int = N_CANDLES) -> list[dict]: + """Keep the last ``n_candles`` and validate every field; raise on bad data.""" + if n_candles < 2: + raise ValueError("n_candles must be at least 2") + if not candles: + raise ValueError("No candles to render") + rows = [] + for raw in list(candles)[-n_candles:]: + if not isinstance(raw, dict): + raise ValueError(f"Candle must be a dict, got {type(raw).__name__}") + row = {} + for field in _FIELDS: + if field not in raw: + raise ValueError(f"Candle is missing {field!r}") + value = float(raw[field]) + if not math.isfinite(value) or value < 0: + raise ValueError( + f"Candle {field} must be finite and >= 0, got {value!r}" + ) + row[field] = value + for field in ("open", "high", "low", "close"): + if row[field] <= 0: + raise ValueError(f"Candle {field} must be > 0") + if row["low"] > min(row["open"], row["close"]) or row["high"] < max( + row["open"], row["close"] + ): + raise ValueError("Candle high/low do not contain open/close") + rows.append(row) + return rows + + +def price_scale(rows: Sequence[dict]) -> tuple[float, float]: + """``(lo, span)`` for the price axis: window padded 12 % each side, with a + floor of 0.2 % of the mean price so a flat market is not blown up to + full scale (stonkfly's rule).""" + lows = np.asarray([r["low"] for r in rows], dtype=float) + highs = np.asarray([r["high"] for r in rows], dtype=float) + closes = np.asarray([r["close"] for r in rows], dtype=float) + span = max(float(highs.max() - lows.min()), float(closes.mean()) * 0.002) + lo = float(lows.min()) - span * 0.12 + return lo, span * 1.24 + + +def _y(value: float, lo: float, span: float) -> float: + return PLOT_BOTTOM - (value - lo) / span * (PLOT_BOTTOM - PLOT_TOP) + + +def market_frame( + pair: str, + candles: Sequence[Any], + bid: float, + ask: float, + n_candles: int = N_CANDLES, +) -> np.ndarray: + """OHLCV chart → ``np.uint8[HEIGHT, WIDTH, 3]``.""" + bid = _finite_positive(bid, "bid") + ask = _finite_positive(ask, "ask") + if ask < bid: + raise ValueError(f"ask {ask} below bid {bid}") + rows = normalize_candles(candles, n_candles) + + im = Image.new("RGB", (WIDTH, HEIGHT), BACKGROUND) + d = ImageDraw.Draw(im) + d.rectangle((0, 0, WIDTH - 1, 27), fill=HEADER) + d.text((9, 8), str(pair), fill=HEADER_TEXT) + for x in range(PLOT_LEFT, PLOT_RIGHT + 1, 30): + d.line((x, PLOT_TOP, x, VOLUME_BOTTOM), fill=GRID) + for y in range(PLOT_TOP + 4, PLOT_BOTTOM, 24): + d.line((PLOT_LEFT - 2, y, PLOT_RIGHT + 2, y), fill=GRID) + + lo, span = price_scale(rows) + slot = (PLOT_RIGHT - PLOT_LEFT) / n_candles + body_w = max(1, int(slot) - 1) + max_volume = max(r["volume"] for r in rows) + # Right-align so the newest candle always sits at the right edge, where the + # bid/ask ticks are, regardless of how many candles the feed returned. + offset = n_candles - len(rows) + for i, r in enumerate(rows): + x0 = PLOT_LEFT + (offset + i) * slot + xc = int(x0 + slot / 2) + color = UP if r["close"] >= r["open"] else DOWN + y_high, y_low = _y(r["high"], lo, span), _y(r["low"], lo, span) + d.line((xc, y_high, xc, y_low), fill=color, width=1) + y_open, y_close = _y(r["open"], lo, span), _y(r["close"], lo, span) + top, bottom = min(y_open, y_close), max(y_open, y_close) + if bottom - top < 1: + bottom = top + 1 + d.rectangle((int(x0), top, int(x0) + body_w, bottom), fill=color) + if max_volume > 0: + h = r["volume"] / max_volume * (VOLUME_BOTTOM - VOLUME_TOP) + d.rectangle( + (int(x0), VOLUME_BOTTOM - h, int(x0) + body_w, VOLUME_BOTTOM), + fill=VOLUME, + ) + + for price in (bid, ask): + y = _y(price, lo, span) + if PLOT_TOP <= y <= PLOT_BOTTOM: + d.line((PLOT_RIGHT + 2, y, WIDTH - 2, y), fill=TICK, width=2) + d.text((9, FOOTER_Y), f"BID {bid:g} ASK {ask:g}"[:50], fill=TEXT) + return np.asarray(im, dtype=np.uint8) diff --git a/condor/fly/data.py b/condor/fly/data.py new file mode 100644 index 000000000..9785953a9 --- /dev/null +++ b/condor/fly/data.py @@ -0,0 +1,97 @@ +"""Fetch released MaleCNS inputs and verify both sources and prepared arrays.""" + +import hashlib +import json +import shutil +import urllib.request +from pathlib import Path + +import numpy as np + +from .neural.common import DATA, GRAPH, digest + +PACKAGE = Path(__file__).with_name("neural") + + +def sha(path): + h = hashlib.sha256() + with Path(path).open("rb") as f: + for block in iter(lambda: f.read(8 * 1024 * 1024), b""): + h.update(block) + return h.hexdigest() + + +def verify(): + lock = json.loads((PACKAGE / "sources.lock.json").read_text()) + if sha(DATA / "annotations.feather") != lock["annotations.feather"]["sha256"]: + raise RuntimeError("Annotation checksum mismatch") + expected = json.loads((PACKAGE / "arrays.lock.json").read_text()) + with np.load(GRAPH, allow_pickle=False) as a: + if set(a.files) != set(expected): + raise RuntimeError("Graph fields mismatch") + for k, h in expected.items(): + if digest(a[k]) != h: + raise RuntimeError("Graph array checksum mismatch: " + k) + if len(a["ids"]) != 166700 or len(a["post"]) != 25582938: + raise RuntimeError("Wrong retained graph") + import pyarrow.feather as f + + # Match normalized transmitter identities to the checksum-locked released file. + if not (DATA / "normalized/neurons.feather").exists(): + raise RuntimeError("Normalized neuron metadata missing") + n = f.read_table(DATA / "normalized/neurons.feather").to_pandas() + transmitter_values = json.dumps( + n.neurotransmitter.fillna("").astype(str).tolist(), separators=(",", ":") + ).encode() + nt_expected = json.loads((PACKAGE / "neurons.lock.json").read_text())[ + "neurotransmitter_values_sha256" + ] + if hashlib.sha256(transmitter_values).hexdigest() != nt_expected: + raise RuntimeError("Normalized transmitter values mismatch") + with np.load(GRAPH, allow_pickle=False) as a: + if not np.array_equal(n.source_id.to_numpy(), a["ids"]): + raise RuntimeError("Normalized neuron order mismatch") + return { + "release": "MaleCNS v1.0", + "neurons": 166700, + "directed_edges": 25582938, + "arrays_verified": True, + } + + +def prepare(reuse=None): + DATA.mkdir(parents=True, exist_ok=True) + if reuse: + root = Path(reuse) + mapping = { + root / "outputs/doom/malecns_v1/graph.npz": GRAPH, + root + / "connectome_data/malecns_v1/annotations.feather": DATA + / "annotations.feather", + root + / "connectome_data/malecns_v1/normalized/neurons.feather": DATA + / "normalized/neurons.feather", + } + for source, target in mapping.items(): + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(source, target) + else: + lock = json.loads((PACKAGE / "sources.lock.json").read_text()) + for name, info in lock.items(): + path = DATA / name + if not path.exists(): + print("Downloading", name, flush=True) + tmp = path.with_suffix(".partial") + urllib.request.urlretrieve(info["url"], tmp) + if sha(tmp) != info["sha256"]: + raise RuntimeError("Downloaded checksum mismatch: " + name) + tmp.replace(path) + if sha(path) != info["sha256"]: + raise RuntimeError("Source checksum mismatch: " + name) + shutil.copyfile(PACKAGE / "sources.lock.json", DATA / "source.lock.json") + from .neural.connectome import import_graph + from .neural.prepare import prepare as compile_graph + + import_graph() + compile_graph() + print(json.dumps(verify()), flush=True) diff --git a/condor/fly/decoder.py b/condor/fly/decoder.py new file mode 100644 index 000000000..e6f49224e --- /dev/null +++ b/condor/fly/decoder.py @@ -0,0 +1,218 @@ +"""Decode spike counts into a quoting posture. + +An engineered, fixed readout — like stonkfly's, it reads only spike counts and +the cells it reads are logged. Nothing here is a discovery of "market-making +neurons": + +* ``trend_hz`` — mean DNp20 right rate minus mean left rate (stonkfly's + BUY/SELL cells). Sign → which way the reference price leans. +* ``arousal_hz`` — mean rate of the descending-neuron population. Higher → + wider spreads. +* ``gate`` — DNpe017 spikes ≥ 1, required for a trending call. + +Channels are z-scored against a rolling per-pair baseline. Stonkfly's own run +proposed BUY six times out of six — a persistent turning bias of the circuit +became persistent buying. In market making that would be a persistent skew, so +the baseline is subtracted before thresholds are applied. ``center_bias=False`` +keeps stonkfly's raw reading of the trend channel (2 Hz = one unit) as a +control; arousal has no natural zero and is always window-normalized. +""" + +from __future__ import annotations + +import math +import statistics +from dataclasses import asdict, dataclass, field +from typing import Literal + +Regime = Literal[ + "pause", "volatile", "trending_up", "trending_down", "quiet", "ranging" +] +REGIMES: tuple[str, ...] = ( + "pause", + "volatile", + "trending_up", + "trending_down", + "quiet", + "ranging", +) + +# stonkfly's DNp20 threshold: 2 Hz difference is one unit in raw mode. +RAW_TREND_UNIT_HZ = 2.0 + + +@dataclass(frozen=True) +class DecoderSettings: + window: int = 60 + warmup: int = 10 + z_regime: float = 1.0 + z_pause: float = 2.5 + spread_gain: float = 0.5 + spread_min: float = 0.6 + spread_max: float = 2.5 + shift_gain_bps: float = 1.0 + max_shift_bps: float = 3.0 + center_bias: bool = True + + def __post_init__(self): + if self.window < 2 or self.warmup < 1 or self.warmup > self.window: + raise ValueError("window >= 2 and 1 <= warmup <= window required") + if not 0 < self.z_regime < self.z_pause: + raise ValueError("0 < z_regime < z_pause required") + if not 0 < self.spread_min <= 1 <= self.spread_max: + raise ValueError("spread_min <= 1 <= spread_max required") + for name in ("spread_gain", "shift_gain_bps", "max_shift_bps"): + value = getattr(self, name) + if not math.isfinite(value) or value <= 0: + raise ValueError(f"{name} must be finite and positive") + + +@dataclass(frozen=True) +class Channels: + trend_hz: float + arousal_hz: float + gate_spikes: int + + def __post_init__(self): + if not math.isfinite(self.trend_hz) or not math.isfinite(self.arousal_hz): + raise ValueError("Nonfinite channel") + if self.arousal_hz < 0 or self.gate_spikes < 0: + raise ValueError("Negative rate or spike count") + + +@dataclass +class Baseline: + """Rolling per-pair history of the two channels; persisted in state.json.""" + + trend: list[float] = field(default_factory=list) + arousal: list[float] = field(default_factory=list) + + @classmethod + def from_dict(cls, data: dict | None) -> "Baseline": + if not data: + return cls() + return cls(trend=list(data["trend"]), arousal=list(data["arousal"])) + + def to_dict(self) -> dict: + return {"trend": list(self.trend), "arousal": list(self.arousal)} + + @property + def count(self) -> int: + return len(self.trend) + + def push(self, channels: Channels, window: int) -> None: + self.trend.append(channels.trend_hz) + self.arousal.append(channels.arousal_hz) + del self.trend[:-window] + del self.arousal[:-window] + + +def _z(value: float, history: list[float], center: bool) -> float: + if len(history) < 2: + return 0.0 + std = statistics.pstdev(history) + if std < 1e-9: + return 0.0 + mean = statistics.fmean(history) if center else 0.0 + return (value - mean) / std + + +@dataclass(frozen=True) +class Posture: + regime: str + spread_mult: float + shift_bps: float + trend_z: float + arousal_z: float + gate: bool + warm: bool # False while the baseline is still forming + + def to_dict(self) -> dict: + return asdict(self) + + @classmethod + def from_dict(cls, data: dict) -> "Posture": + return cls(**data) + + +NEUTRAL = Posture( + regime="ranging", + spread_mult=1.0, + shift_bps=0.0, + trend_z=0.0, + arousal_z=0.0, + gate=False, + warm=False, +) + + +def classify(trend_z: float, arousal_z: float, gate: bool, s: DecoderSettings) -> str: + """Regime in precedence order: pause > volatile > trending > quiet > ranging.""" + if arousal_z >= s.z_pause: + return "pause" + if arousal_z >= s.z_regime: + return "volatile" + if gate and trend_z >= s.z_regime: + return "trending_up" + if gate and trend_z <= -s.z_regime: + return "trending_down" + if arousal_z <= -s.z_regime: + return "quiet" + return "ranging" + + +def decode(channels: Channels, baseline: Baseline, s: DecoderSettings) -> Posture: + """Push the observation into the baseline and return the posture. + + The baseline is mutated (that is the point: it is the rolling window).""" + baseline.push(channels, s.window) + if baseline.count < s.warmup: + return NEUTRAL + if s.center_bias: + trend_z = _z(channels.trend_hz, baseline.trend, center=True) + else: + trend_z = channels.trend_hz / RAW_TREND_UNIT_HZ + arousal_z = _z(channels.arousal_hz, baseline.arousal, center=True) + gate = channels.gate_spikes >= 1 + regime = classify(trend_z, arousal_z, gate, s) + spread_mult = min(s.spread_max, max(s.spread_min, 1 + s.spread_gain * arousal_z)) + shift = max(-s.max_shift_bps, min(s.max_shift_bps, s.shift_gain_bps * trend_z)) + if not gate: + shift = 0.0 # no descending gate spike, no directional lean + return Posture( + regime=regime, + spread_mult=round(spread_mult, 4), + shift_bps=round(shift, 3), + trend_z=round(trend_z, 4), + arousal_z=round(arousal_z, 4), + gate=gate, + warm=True, + ) + + +@dataclass(frozen=True) +class Hysteresis: + min_apply_interval_sec: float = 300.0 + spread_delta: float = 0.15 + shift_delta_bps: float = 0.5 + + +def should_apply( + previous: Posture | None, + new: Posture, + last_apply_ts: float | None, + now: float, + h: Hysteresis, +) -> tuple[bool, str]: + """Rate-limit configuration changes to material posture moves.""" + if previous is None: + return True, "first posture" + if last_apply_ts is not None and now - last_apply_ts < h.min_apply_interval_sec: + return False, "apply cooldown" + if new.regime != previous.regime: + return True, f"regime {previous.regime} -> {new.regime}" + if abs(new.spread_mult - previous.spread_mult) >= h.spread_delta: + return True, f"spread_mult {previous.spread_mult} -> {new.spread_mult}" + if abs(new.shift_bps - previous.shift_bps) >= h.shift_delta_bps: + return True, f"shift_bps {previous.shift_bps} -> {new.shift_bps}" + return False, "no material change" diff --git a/condor/fly/guard.py b/condor/fly/guard.py new file mode 100644 index 000000000..d7f81c98a --- /dev/null +++ b/condor/fly/guard.py @@ -0,0 +1,209 @@ +"""Deterministic checks that can veto or halt, and never choose a posture. + +A ``Veto`` means "keep the previous config this tick". A ``Halt`` means "stop +the bots and stop the fly until a human passes ``resume_reviewed``"; a +*financial* halt (loss stop, loss-rate breaker) cannot be cleared that way at +all — a new run directory is needed, exactly stonkfly's rule. +""" + +from __future__ import annotations + +import math +from dataclasses import asdict, dataclass, field +from datetime import datetime, timezone + +from condor.fly.posture import MarketSpec, take_profit_floor + +BPS = 1e-4 + + +class Veto(Exception): + pass + + +class Halt(Exception): + def __init__(self, reason: str, financial: bool): + super().__init__(reason) + self.reason = reason + self.financial = financial + + +@dataclass(frozen=True) +class GuardSettings: + max_applies_per_day: int = 48 + apply_price_tolerance: float = 0.005 + max_loss_quote: float = 0.0 # 0 → 4 % of the combined total_amount_quote + max_loss_per_volume_bps: float = 5.0 + loss_no_new_high_ticks: int = 25 + min_volume_for_loss_rate: float = 100.0 + max_apply_failures: int = 3 + closed_ticks_to_stop: int = 5 + + def __post_init__(self): + if self.max_applies_per_day < 1 or self.max_apply_failures < 1: + raise ValueError("positive limits required") + if not 0 < self.apply_price_tolerance <= 0.05: + raise ValueError("apply_price_tolerance must be in (0, 0.05]") + if self.max_loss_quote < 0 or self.max_loss_per_volume_bps <= 0: + raise ValueError("loss limits must be positive") + + +@dataclass +class GuardState: + """Persisted in state.json between ticks.""" + + applies_day: str = "" + applies_today: int = 0 + last_apply: dict[str, float] = field(default_factory=dict) # per pair + consecutive_failures: int = 0 + session_high_net: float = 0.0 + ticks_since_high: int = 0 + closed_ticks: dict[str, int] = field(default_factory=dict) + halted: str | None = None + halt_financial: bool = False + + @classmethod + def from_dict(cls, data: dict | None) -> "GuardState": + return cls(**data) if data else cls() + + def to_dict(self) -> dict: + return asdict(self) + + +def _day(now: float) -> str: + return datetime.fromtimestamp(now, timezone.utc).strftime("%Y-%m-%d") + + +def check_not_halted(state: GuardState) -> None: + if state.halted: + raise Halt(state.halted, state.halt_financial) + + +def resume(state: GuardState, reviewed: bool) -> None: + """Clear a transient halt after review; refuse to clear a financial one.""" + if not state.halted: + return + if state.halt_financial: + raise Halt(f"financial halt cannot be cleared by review: {state.halted}", True) + if not reviewed: + raise Halt( + f"halted, pass resume_reviewed=true after review: {state.halted}", False + ) + state.halted = None + state.consecutive_failures = 0 + + +def check_market_open( + pair: str, book_open: bool, state: GuardState, s: GuardSettings +) -> bool: + """Veto while a book is closed. Returns True when the pair has been closed + for ``closed_ticks_to_stop`` ticks, meaning its bot should be stopped.""" + if book_open: + state.closed_ticks[pair] = 0 + return False + state.closed_ticks[pair] = state.closed_ticks.get(pair, 0) + 1 + if state.closed_ticks[pair] >= s.closed_ticks_to_stop: + return True + raise Veto(f"{pair}: book closed ({state.closed_ticks[pair]} ticks)") + + +def check_collateral(available_usd: float, required_usd: float) -> None: + if not math.isfinite(available_usd) or not math.isfinite(required_usd): + raise Veto("collateral figures are not finite") + if available_usd < required_usd: + raise Veto( + f"available collateral {available_usd:.2f} < required {required_usd:.2f}" + ) + + +def _spreads(value: str) -> list[float]: + return [float(x) for x in str(value).split(",") if x.strip()] + + +def check_config(config: dict, spec: MarketSpec) -> None: + """Belt to ``posture.build_config``'s braces: refuse anything below the floors.""" + for key in ("buy_spreads", "sell_spreads"): + for level in _spreads(config[key]): + if level < spec.min_spread_bps * BPS: + raise Veto(f"{key} level {level} below {spec.min_spread_bps} bp") + if float(config["take_profit"]) < take_profit_floor(spec): + raise Veto("take_profit below fee floor") + if int(config["leverage"]) > spec.leverage_cap: + raise Veto(f"leverage {config['leverage']} above cap {spec.leverage_cap}") + if config["trading_pair"] != spec.trading_pair: + raise Veto("config pair does not match market spec") + if not config.get("global_sl_enabled"): + raise Veto("global stop loss must stay enabled") + + +def check_apply_window(state: GuardState, now: float, s: GuardSettings) -> None: + if state.applies_day != _day(now): + state.applies_day = _day(now) + state.applies_today = 0 + if state.applies_today >= s.max_applies_per_day: + raise Veto("daily apply limit") + + +def check_price_move(observed_mid: float, fresh_mid: float, s: GuardSettings) -> None: + if observed_mid <= 0 or fresh_mid <= 0: + raise Veto("non-positive mid") + if abs(fresh_mid - observed_mid) / observed_mid > s.apply_price_tolerance: + raise Veto("price moved beyond neural observation tolerance") + + +def record_apply( + state: GuardState, pair: str, now: float, ok: bool, s: GuardSettings +) -> None: + state.applies_today += 1 + state.last_apply[pair] = now + if ok: + state.consecutive_failures = 0 + return + state.consecutive_failures += 1 + if state.consecutive_failures >= s.max_apply_failures: + state.halted = ( + f"{state.consecutive_failures} consecutive config update failures" + ) + state.halt_financial = False + raise Halt(state.halted, False) + + +def check_pnl( + total_net: float, + volume: float, + state: GuardState, + s: GuardSettings, + max_loss_quote: float, +) -> None: + """Loss stop and the HIP-3 loss-rate breaker. ``total_net`` must include + unrealized P&L — realized alone can look fine while the open position bleeds.""" + if not math.isfinite(total_net) or not math.isfinite(volume): + raise Veto("P&L figures are not finite") + if total_net > state.session_high_net: + state.session_high_net = total_net + state.ticks_since_high = 0 + else: + state.ticks_since_high += 1 + if total_net <= -max_loss_quote: + state.halted = f"loss stop: net {total_net:.2f} <= -{max_loss_quote:.2f}" + state.halt_financial = True + raise Halt(state.halted, True) + if volume >= s.min_volume_for_loss_rate: + rate_bps = total_net / volume * 1e4 + if rate_bps <= -s.max_loss_per_volume_bps: + state.halted = f"loss-rate breaker: {rate_bps:.1f} bp of volume" + state.halt_financial = True + raise Halt(state.halted, True) + if ( + state.ticks_since_high >= s.loss_no_new_high_ticks + and total_net < state.session_high_net + ): + state.halted = f"no new P&L high for {state.ticks_since_high} ticks" + state.halt_financial = True + raise Halt(state.halted, True) + + +def default_max_loss(specs: list[MarketSpec], s: GuardSettings) -> float: + if s.max_loss_quote > 0: + return s.max_loss_quote + return 0.04 * sum(spec.total_amount_quote for spec in specs) diff --git a/condor/fly/market.py b/condor/fly/market.py new file mode 100644 index 000000000..76aa7579d --- /dev/null +++ b/condor/fly/market.py @@ -0,0 +1,249 @@ +"""What the fly loop reads from the world, and the one thing it writes. + +``LiveMarket`` talks to Hummingbot (candles, bot performance, portfolio, config +updates, stop) and to Hyperliquid's public ``l2Book`` for the live book — the +hummingbot-api order-book endpoint 500s on HIP-3 pairs. ``FixtureMarket`` is a +deterministic offline stand-in for plumbing tests; it never applies anything. +""" + +from __future__ import annotations + +import math +import time +from dataclasses import dataclass + +import aiohttp + +from condor.fly.naming import pair_names +from condor.fly.posture import MarketSpec +from condor.fly.reinforcement import controller_net + +HL_INFO_URL = "https://api.hyperliquid.xyz/info" +QUOTE_TOKENS = ("USD", "USDC") + + +@dataclass(frozen=True) +class Observation: + pair: str + candles: list[dict] + bid: float + ask: float + open: bool # both sides of the live book present + + @property + def mid(self) -> float: + return (self.bid + self.ask) / 2 + + +@dataclass(frozen=True) +class Book: + bid: float | None + ask: float | None + + @property + def open(self) -> bool: + return self.bid is not None and self.ask is not None + + +async def fetch_l2_book(session: aiohttp.ClientSession, coin: str) -> Book: + async with session.post( + HL_INFO_URL, + json={"type": "l2Book", "coin": coin}, + timeout=aiohttp.ClientTimeout(total=10), + ) as resp: + if resp.status != 200: + raise RuntimeError(f"l2Book {coin}: HTTP {resp.status}") + book = await resp.json() + levels = book.get("levels") if isinstance(book, dict) else None + if not levels or len(levels) != 2: + raise RuntimeError(f"l2Book {coin}: unexpected payload {str(book)[:120]}") + bids, asks = levels + if not bids or not asks: + return Book(None, None) # closed / empty book, a real state not an error + return Book(float(bids[0]["px"]), float(asks[0]["px"])) + + +def normalize_candle_payload(result) -> list[dict]: + records = ( + result + if isinstance(result, list) + else result.get("data", result.get("candles")) + ) + if not records: + raise RuntimeError("Candle feed returned no records") + return list(records) + + +class LiveMarket: + def __init__( + self, client, connector_name: str, candle_interval: str, n_candles: int + ): + self.client = client + self.connector_name = connector_name + self.candle_interval = candle_interval + self.n_candles = n_candles + + async def observe(self, pair: str) -> Observation: + names = pair_names(pair) + candles = normalize_candle_payload( + await self.client.market_data.get_candles( + self.connector_name, + pair, + interval=self.candle_interval, + max_records=self.n_candles, + ) + ) + async with aiohttp.ClientSession() as session: + book = await fetch_l2_book(session, names.coin) + if not book.open: + # The chart still exists; the loop decides what a closed book means. + last = float(candles[-1]["close"]) + return Observation(pair, candles, last, last, False) + return Observation(pair, candles, book.bid, book.ask, True) + + async def fresh_mid(self, pair: str) -> float: + async with aiohttp.ClientSession() as session: + book = await fetch_l2_book(session, pair_names(pair).coin) + if not book.open: + raise RuntimeError(f"{pair}: book closed at apply time") + return (book.bid + book.ask) / 2 + + async def bots(self) -> dict: + resp = await self.client.bot_orchestration.get_active_bots_status() + raw = resp if isinstance(resp, dict) else {} + data = raw.get("data", raw) + return data if isinstance(data, dict) else {} + + async def equity(self, pairs: list[str]) -> tuple[float, float, dict]: + """Combined ``realized + unrealized`` and volume across the fly's bots. + + A bot that is not running contributes nothing — there is no P&L to + report. Returns ``(net, volume, per_pair)``.""" + bots = await self.bots() + net = 0.0 + volume = 0.0 + per_pair: dict[str, dict] = {} + for pair in pairs: + names = pair_names(pair) + bot = bots.get(names.bot_name) + if not isinstance(bot, dict): + per_pair[pair] = {"running": False} + continue + perf = (bot.get("performance") or {}).get(names.config_name) + if not isinstance(perf, dict): + raise RuntimeError( + f"bot {names.bot_name} is running without controller {names.config_name}" + ) + inner = perf.get("performance", perf) + pair_net = controller_net(inner) + pair_volume = float(inner.get("volume_traded", 0) or 0) + net += pair_net + volume += pair_volume + per_pair[pair] = {"running": True, "net": pair_net, "volume": pair_volume} + if not math.isfinite(net) or not math.isfinite(volume): + raise RuntimeError("Nonfinite bot performance") + return net, volume, per_pair + + async def available_usd(self) -> float: + state = await self.client.portfolio.get_portfolio_state() + if not isinstance(state, dict): + raise RuntimeError("Portfolio state unavailable") + total = 0.0 + seen = False + for account in state.values(): + if not isinstance(account, dict): + continue + for connector, tokens in account.items(): + if self.connector_name not in connector or not isinstance(tokens, list): + continue + seen = True + for token in tokens: + if isinstance(token, dict) and token.get("token") in QUOTE_TOKENS: + units = float(token.get("units", 0) or 0) + total += float(token.get("available_units", units) or 0) + if not seen: + raise RuntimeError(f"No {self.connector_name} balances in portfolio state") + return total + + async def apply(self, pair: str, config: dict) -> None: + """Update the live bot's controller and the saved config, both layers.""" + names = pair_names(pair) + bots = await self.bots() + if names.bot_name not in bots: + raise RuntimeError(f"bot {names.bot_name} is not running") + await self.client.controllers.update_bot_controller_config( + names.bot_name, names.config_name, config + ) + await self.client.controllers.create_or_update_controller_config( + names.config_name, config + ) + + async def stop_bot(self, pair: str) -> bool: + names = pair_names(pair) + if names.bot_name not in await self.bots(): + return False + await self.client.bot_orchestration.stop_and_archive_bot(names.bot_name) + return True + + +class FixtureMarket: + """Deterministic sine-wave candles; equity swings so both pulses fire.""" + + BASE = {0: 60.0, 1: 120.0, 2: 30.0} + + def __init__(self, pairs: list[str], n_candles: int): + self.pairs = pairs + self.n_candles = n_candles + self.tick = 0 + + def _price(self, index: int, k: int) -> float: + base = self.BASE[index % 3] + return base * (1 + 0.02 * math.sin(k * 0.35 + index)) + + async def observe(self, pair: str) -> Observation: + index = self.pairs.index(pair) + candles = [] + for k in range(self.tick, self.tick + self.n_candles): + o, c = self._price(index, k), self._price(index, k + 1) + candles.append( + { + "timestamp": k * 300, + "open": o, + "high": max(o, c) * 1.001, + "low": min(o, c) * 0.999, + "close": c, + "volume": 100 + 50 * math.sin(k * 0.7), + } + ) + self.tick += 1 + mid = candles[-1]["close"] + return Observation(pair, candles, mid * 0.9995, mid * 1.0005, True) + + async def fresh_mid(self, pair: str) -> float: + index = self.pairs.index(pair) + return self._price(index, self.tick + self.n_candles) + + async def equity(self, pairs: list[str]) -> tuple[float, float, dict]: + # Swings so both pulses fire, drifts up so new highs recur, and stays + # within ±2 bp of volume so the loss-rate breaker is not exercised here. + net = 2.0 * math.sin(self.tick * 1.3) + 0.05 * self.tick + volume = 10_000.0 * (self.tick + 1) + return net, volume, {p: {"running": False, "fixture": True} for p in pairs} + + async def available_usd(self) -> float: + return 1e9 + + async def apply(self, pair: str, config: dict) -> None: + raise RuntimeError("FixtureMarket never applies a config") + + async def stop_bot(self, pair: str) -> bool: + return False + + +def required_collateral(specs: list[MarketSpec]) -> float: + """Margin the three books could need at their inventory cap.""" + return sum(s.total_amount_quote * s.max_base_pct / s.leverage for s in specs) + + +def now() -> float: + return time.time() diff --git a/condor/fly/naming.py b/condor/fly/naming.py new file mode 100644 index 000000000..479fee4eb --- /dev/null +++ b/condor/fly/naming.py @@ -0,0 +1,48 @@ +"""Derived names for a HIP-3 pair: the bot, its controller config, the l2Book coin.""" + +from __future__ import annotations + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class Names: + pair: str # XYZ:DRAM-USD (Hummingbot trading_pair, uppercase) + issuer: str # xyz + token: str # DRAM + coin: str # xyz:DRAM (Hyperliquid l2Book coin: lowercase issuer, uppercase token) + bot_name: str # dram-fly + config_name: str # dram_fly_mm + + +def pair_names(pair: str) -> Names: + if ":" not in pair or not pair.endswith("-USD"): + raise ValueError(f"HIP-3 pair must look like ISSUER:TOKEN-USD, got {pair!r}") + if pair != pair.upper(): + raise ValueError(f"HIP-3 pair must be uppercase, got {pair!r}") + issuer, rest = pair.split(":", 1) + token = rest[: -len("-USD")] + if not issuer or not token: + raise ValueError(f"HIP-3 pair must look like ISSUER:TOKEN-USD, got {pair!r}") + base = token.lower() + return Names( + pair=pair, + issuer=issuer.lower(), + token=token, + coin=f"{issuer.lower()}:{token}", + bot_name=f"{base}-fly", + config_name=f"{base}_fly_mm", + ) + + +def parse_pairs(value: str, limit: int = 3) -> list[str]: + pairs = [p.strip() for p in value.split(",") if p.strip()] + if not pairs: + raise ValueError("At least one pair is required") + if len(pairs) > limit: + raise ValueError(f"At most {limit} pairs, got {len(pairs)}") + if len(set(pairs)) != len(pairs): + raise ValueError("Duplicate pair") + for pair in pairs: + pair_names(pair) + return pairs diff --git a/condor/fly/neural/THIRD_PARTY.md b/condor/fly/neural/THIRD_PARTY.md new file mode 100644 index 000000000..caf7cbc2f --- /dev/null +++ b/condor/fly/neural/THIRD_PARTY.md @@ -0,0 +1,31 @@ +# Third-party notice + +`condor/fly/neural/` and `condor/fly/data.py` are vendored from +https://github.com/nftechie/stonkfly (MIT License, Copyright (c) 2026 nftechie +and DOOMFLY contributors). Only `neural/common.py` is modified: the data +directory resolves to Condor's runtime root instead of `./data`. + +The MaleCNS v1.0 connectome files it downloads retain their upstream license; +see the stonkfly repository's THIRD_PARTY.md. + +MIT License + +Copyright (c) 2026 nftechie and DOOMFLY contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/condor/fly/neural/__init__.py b/condor/fly/neural/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/condor/fly/neural/arrays.lock.json b/condor/fly/neural/arrays.lock.json new file mode 100644 index 000000000..5e855fd4e --- /dev/null +++ b/condor/fly/neural/arrays.lock.json @@ -0,0 +1,13 @@ +{ + "ptr": "a29aa0296125a5daa972c12d17711360d01a1209cd9cd9d7ccb1f6959224bcaf", + "post": "4cc5e857d3481f433576f35ddb8d580cc33cf8e22b6651fa03e08cde5d956d90", + "weight": "d95171c2d59cfe3ebef2e5bd52be779b8f74754d718fc8941a93b8f2921c632b", + "ids": "6b6b40c3bddf84b1281ef0b18db06927c2bf61c5f4cb32a4219ec9b7839dc2e5", + "retina": "5ec145156d998f582e5d8f08c4174e10f7f853d34745118adab8589f192d04b4", + "uv": "51f7dafd4355a06176418e22373ffcdf5bc066f75050da6dcbaaaccbad55a714", + "confidence": "4ecf882056a959ebb9053f89b766eeae7cf05aab4d65beb4d3d88a034dc7061a", + "hexes": "315ab7bd1abc1ef0307e90f8fff484020cd556b8d2d877f84773c912bcf6f2ca", + "lamina": "22d6fa8ac7aa40700d09394aea1cf35226711d2efc78bc716882920ba180e47e", + "sugar": "89b057c7b57d5e783f608815b4219b17b54c812c3fa229bcb78b1cafa30d37a1", + "superclass": "3977d971fe3ef65ce97c5c9c5d6dcf0aea893bebc55311c8c886ebe0f5756d13" +} diff --git a/condor/fly/neural/brain.py b/condor/fly/neural/brain.py new file mode 100644 index 000000000..421badd8d --- /dev/null +++ b/condor/fly/neural/brain.py @@ -0,0 +1,446 @@ +"""Full-graph candidate memory dynamics with explicit stimulation and checkpoints.""" + +import ctypes as C +import hashlib +import json +import math +import subprocess +import sys +import time +from pathlib import Path + +import numpy as np + +from .circuit import identify +from .common import DATA, GRAPH, OUT, digest, save_json +from .state import NativeBrain + +SOURCE = Path(__file__).with_name("kernel.cpp") +LIBRARY = (OUT / "physiology-v6") / ( + "libmemory.dylib" if sys.platform == "darwin" else "libmemory.so" +) +MODEL = "stonkfly-dual-compartment-v1" +from .rule import PARAMETERS as RULE_PARAMETERS + +PARAMETERS = { + **RULE_PARAMETERS, + "neural_dt_ms": 0.1, + "modulator_delivery_trace_ms": 100.0, + "kc_rest_mV": -60.0, + "kc_adaptation_jump_mV": 8.0, + "kc_adaptation_tau_ms": 200.0, + "interpretation": "Candidate KC adaptation/rest plus a baseline-centered anti-Hebbian rate-rule extension to two compartments. No fitted DAN/MBON background current; lamina bias is a display proxy. Gain, trace constants and transfer to this graph remain unvalidated assumptions.", +} + + +def build(): + sha = hashlib.sha256(SOURCE.read_bytes()).hexdigest() + metadata = LIBRARY.with_suffix(LIBRARY.suffix + ".json") + if LIBRARY.exists() and metadata.exists(): + record = json.loads(metadata.read_text()) + if ( + record["source_sha256"] == sha + and record["binary_sha256"] + == hashlib.sha256(LIBRARY.read_bytes()).hexdigest() + ): + return record + LIBRARY.parent.mkdir(parents=True, exist_ok=True) + temp = LIBRARY.with_suffix(LIBRARY.suffix + ".partial") + subprocess.run( + ["c++", "-O3", "-std=c++17", "-shared", "-fPIC", str(SOURCE), "-o", str(temp)], + check=True, + ) + temp.replace(LIBRARY) + record = { + "model": MODEL, + "source_sha256": sha, + "binary_sha256": hashlib.sha256(LIBRARY.read_bytes()).hexdigest(), + "flags": ["-O3", "-std=c++17", "-shared", "-fPIC"], + } + save_json(metadata, record) + return record + + +class MemoryBrain(NativeBrain): + def __init__( + self, + path=GRAPH, + *, + eta=0.001, + circuit=None, + modulation_mask=None, + tonic=None, + dan_baseline_hz=None, + kc_rest=-60.0, + adaptation_jump=8.0, + adaptation_tau=200.0, + ): + super().__init__(path) + self.build = build() + self.library = C.CDLL(str(LIBRARY)) + self.advance = self.library.memory_advance + self.advance.argtypes = ( + [C.c_int] + + [C.c_void_p] * 11 + + [C.c_int, C.c_float] + + [C.c_void_p] * 5 + + [C.c_void_p, C.c_void_p, C.c_void_p, C.c_void_p, C.c_int] + + [C.c_void_p] * 4 + + [ + C.c_float, + C.c_float, + C.c_float, + C.c_int, + C.c_void_p, + C.c_void_p, + C.c_void_p, + C.c_void_p, + C.c_void_p, + C.c_float, + C.c_float, + ] + ) + self.advance.restype = None + self.circuit = identify(self) if circuit is None else circuit + if not math.isfinite(kc_rest) or not -80 <= kc_rest <= -45: + raise ValueError("Invalid KC resting potential") + self.rest = np.full(self.n, -52.0, dtype=np.float32) + self.rest[self.circuit["kc"]] = kc_rest + self.v[:] = self.rest + if ( + not math.isfinite(adaptation_jump) + or adaptation_jump < 0 + or not math.isfinite(adaptation_tau) + or adaptation_tau <= 20 + ): + raise ValueError("Invalid adaptation parameters") + self.adaptation = np.zeros(self.n, dtype=np.float32) + self.adaptation_jump = float(adaptation_jump) + self.adaptation_tau = float(adaptation_tau) + if modulation_mask is None: + import pyarrow.feather as feather + + neurons = ( + feather.read_table(DATA / "normalized/neurons.feather") + .to_pandas() + .set_index("source_id") + .loc[self.ids] + ) + modulation_mask = neurons.neurotransmitter.isin( + ["dopamine", "octopamine", "serotonin"] + ).to_numpy(dtype=np.uint8) + self.modulation_mask = np.asarray(modulation_mask, dtype=np.uint8).copy() + if self.modulation_mask.shape != (self.n,) or np.any(self.modulation_mask > 1): + raise ValueError("Invalid modulation mask") + self.eta = float(eta) + if not math.isfinite(self.eta) or self.eta < 0: + raise ValueError("Finite nonnegative eta required") + self.eligibility = np.zeros(self.n, dtype=np.float64) + self.eligibility_last = np.zeros(self.n, dtype=np.int64) + self.modulation = np.zeros(self.n, dtype=np.float32) + self.modulation_last = np.zeros(self.n, dtype=np.int64) + self.baseline_plastic = self.weight[self.circuit["edges"]].copy() + self.initial_weight_sha256 = digest(self.weight) + self.fields = [ + "v", + "g", + "refractory", + "drive", + "previous_drive", + "queue", + "queue_count", + "counts", + "luminance", + "active", + "active_flag", + "nactive", + "last", + "eligibility", + "eligibility_last", + "modulation", + "modulation_last", + "adaptation", + ] + self.initial = {k: getattr(self, k).copy() for k in self.fields} + from .rule import PARAMETERS as RULE_PARAMETERS + + self.rule_parameters = RULE_PARAMETERS.copy() + self.rate_kc = np.zeros(len(self.circuit["edges"]), dtype=np.float64) + self.rate_dan = np.zeros(len(self.circuit["dan"]), dtype=np.float64) + self.memory_u = np.zeros_like(self.rate_kc) + self.memory_w = np.zeros_like(self.rate_kc) + self.tonic = ( + np.zeros(self.n, dtype=np.float32) + if tonic is None + else np.asarray(tonic, dtype=np.float32).copy() + ) + self.dan_baseline_hz = ( + np.zeros(len(self.circuit["dan"]), dtype=np.float64) + if dan_baseline_hz is None + else np.asarray(dan_baseline_hz, dtype=np.float64).copy() + ) + if self.tonic.shape != (self.n,) or not np.isfinite(self.tonic).all(): + raise ValueError("Invalid tonic current") + if ( + self.dan_baseline_hz.shape != (len(self.rate_dan),) + or not np.isfinite(self.dan_baseline_hz).all() + ): + raise ValueError("Invalid DAN baseline") + self.weights_frozen = False + for k in ["rate_kc", "rate_dan", "memory_u", "memory_w"]: + self.fields.append(k) + self.initial[k] = getattr(self, k).copy() + + def reset(self, keep_memory=False): + if keep_memory: + saved = (self.memory_u.copy(), self.memory_w.copy()) + for k, v in self.initial.items(): + getattr(self, k)[:] = v + self.cursor = 0 + self.sim_ms = 0.0 + self.total_spikes = 0 + if not keep_memory: + self.weight[self.circuit["edges"]] = self.baseline_plastic + else: + self.memory_u[:], self.memory_w[:] = saved + + def _neural_step( + self, + luminance, + duration_ms, + *, + learning=False, + stimulation=None, + lamina_bias=12.0, + ): + light = np.asarray(luminance) + if light.shape != (len(self.retina),) or not np.isfinite(light).all(): + raise ValueError("Invalid retinal input") + steps = round(duration_ms / self.dt) + if ( + not math.isfinite(duration_ms) + or steps < 1 + or not math.isfinite(lamina_bias) + ): + raise ValueError("Invalid interval/current") + self.luminance += (1 - math.exp(-steps * self.dt / 10)) * ( + np.clip(light, 0, 1) - self.luminance + ) + self.drive.fill(0) + self.drive[self.lamina] = lamina_bias + self.drive[self.retina] = 30 * self.luminance / (0.02 + self.luminance) + self.drive += self.tonic + if stimulation is not None: + pulses = stimulation if isinstance(stimulation, list) else [stimulation] + for indices, current in pulses: + ix = np.asarray(indices, dtype=np.int32) + amplitude = np.asarray(current, dtype=np.float32) + if ( + ix.ndim != 1 + or np.any(ix < 0) + or np.any(ix >= self.n) + or not np.isfinite(amplitude).all() + or amplitude.shape not in [(), ix.shape] + ): + raise ValueError("Invalid external stimulation") + self.drive[ix] += amplitude + self.counts.fill(0) + clock = np.asarray([self.cursor], dtype=np.int64) + c = self.circuit + arrays = [ + self.ptr, + self.post, + self.weight, + self.v, + self.g, + self.refractory, + self.drive, + self.previous_drive, + self.queue, + self.queue_count, + clock, + ] + start = time.perf_counter() + self.advance( + self.n, + *[x.ctypes.data for x in arrays], + steps, + self.dt, + *[ + getattr(self, k).ctypes.data + for k in ["counts", "active", "active_flag", "nactive", "last"] + ], + c["kc_mask"].ctypes.data, + c["dan_index"].ctypes.data, + self.eligibility.ctypes.data, + self.eligibility_last.ctypes.data, + len(c["edges"]), + c["edges"].ctypes.data, + c["pre"].ctypes.data, + self.baseline_plastic.ctypes.data, + c["gain"].ctypes.data, + self.eta, + PARAMETERS["trace_kc_seconds"] * 1000, + PARAMETERS["minimum_fraction"], + int(learning), + self.modulation.ctypes.data, + self.modulation_last.ctypes.data, + self.modulation_mask.ctypes.data, + self.rest.ctypes.data, + self.adaptation.ctypes.data, + self.adaptation_jump, + self.adaptation_tau, + ) + elapsed = time.perf_counter() - start + self.cursor = int(clock[0]) + self.sim_ms = self.cursor * self.dt + self.total_spikes += int(self.counts.sum()) + return self.counts.copy(), elapsed + + def step( + self, + luminance, + duration_ms, + *, + learning=False, + stimulation=None, + lamina_bias=12.0, + ): + from .rule import advance + + if not math.isfinite(duration_ms) or duration_ms <= 0: + raise ValueError("Invalid duration") + remaining = round(duration_ms / self.dt) + if remaining < 1: + raise ValueError("Duration too short") + total = np.zeros(self.n, dtype=np.int32) + wall = 0.0 + while remaining: + ticks = min(100, remaining) + interval = ticks * self.dt + # The original LTD update is disabled. Only the centered rule below + # writes candidate memory efficacies; all neural integration remains. + c, t = self._neural_step( + luminance, + interval, + learning=False, + stimulation=stimulation, + lamina_bias=lamina_bias, + ) + seconds = interval / 1000 + advance( + self.rate_kc, + self.rate_dan, + self.memory_u, + self.memory_w, + c[self.circuit["pre"]] / seconds, + c[self.circuit["dan"]] / seconds - self.dan_baseline_hz, + self.circuit["gain"], + seconds, + self.eta, + learning, + self.weights_frozen, + ) + if not self.weights_frozen: + self.weight[self.circuit["edges"]] = self.baseline_plastic * ( + 1 + self.memory_w + ) + total += c + wall += t + remaining -= ticks + self.counts[:] = total + return total, wall + + def memory(self): + w = self.weight[self.circuit["edges"]] + fraction = w / self.baseline_plastic + return { + "plastic_edges": len(w), + "changed_edges": int(np.count_nonzero(w != self.baseline_plastic)), + "mean_efficacy": float(fraction.mean()), + "minimum_efficacy": float(fraction.min()), + "sha256": digest(w), + "model": MODEL, + } + + def checkpoint(self, path): + metadata = { + "model": MODEL, + "build": self.build, + "eta": self.eta, + "parameters": PARAMETERS, + "cursor": self.cursor, + "weights_frozen": self.weights_frozen, + "total_spikes": self.total_spikes, + "graph_ids_sha256": digest(self.ids), + "graph_ptr_sha256": digest(self.ptr), + "graph_post_sha256": digest(self.post), + "plastic_edges_sha256": digest(self.circuit["edges"]), + "configuration_sha256": self.configuration_signature(), + } + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_suffix(".partial") + with temporary.open("wb") as handle: + np.savez_compressed( + handle, + metadata=json.dumps(metadata), + weight=self.weight, + **{k: getattr(self, k) for k in self.fields}, + ) + temporary.replace(path) + + def restore(self, path): + with np.load(path, allow_pickle=False) as a: + m = json.loads(str(a["metadata"])) + expected = { + "model": MODEL, + "build": self.build, + "eta": self.eta, + "parameters": PARAMETERS, + "graph_ids_sha256": digest(self.ids), + "graph_ptr_sha256": digest(self.ptr), + "graph_post_sha256": digest(self.post), + "plastic_edges_sha256": digest(self.circuit["edges"]), + "configuration_sha256": self.configuration_signature(), + } + if any(m.get(k) != v for k, v in expected.items()): + raise ValueError("Checkpoint provenance mismatch") + for k in ["weight", *self.fields]: + if ( + a[k].shape != getattr(self, k).shape + or a[k].dtype != getattr(self, k).dtype + ): + raise ValueError("Checkpoint array mismatch") + if a[k].dtype.kind == "f" and not np.isfinite(a[k]).all(): + raise ValueError("Nonfinite checkpoint state") + for k in ["weight", *self.fields]: + getattr(self, k)[:] = a[k] + self.cursor = int(m["cursor"]) + self.sim_ms = self.cursor * self.dt + self.total_spikes = int(m["total_spikes"]) + self.weights_frozen = bool(m["weights_frozen"]) + + def configuration_signature(self): + # Equal cell IDs and CSR endpoints alone do not imply equal input + # geometry, original efficacies or compartment assignment. + return { + "initial_weight": self.initial_weight_sha256, + "modulation_mask": digest(self.modulation_mask), + "rule": self.rule_parameters, + "rule_sha256": hashlib.sha256( + Path(__file__).with_name("rule.py").read_bytes() + ).hexdigest(), + "tonic": digest(self.tonic), + "dan_baseline_hz": digest(self.dan_baseline_hz), + "rest": digest(self.rest), + "adaptation_jump": self.adaptation_jump, + "adaptation_tau": self.adaptation_tau, + **{ + k: digest(getattr(self, k)) for k in ["retina", "uv", "lamina", "sugar"] + }, + **{ + k: digest(self.circuit[k]) + for k in ["pre", "gain", "kc_mask", "dan_index"] + }, + } diff --git a/condor/fly/neural/circuit.py b/condor/fly/neural/circuit.py new file mode 100644 index 000000000..d7a4196a7 --- /dev/null +++ b/condor/fly/neural/circuit.py @@ -0,0 +1,87 @@ +"""Two explicit, literature-motivated memory compartments in MaleCNS v1.0. + +PAM11 → alpha1 / MBON07 and PPL101 → gamma1pedc / MBON11. +Using the same centered rule in both is a new, unvalidated model assumption. +""" + +import numpy as np + +from .common import annotations, digest + + +def identify(brain): + a = annotations(brain.ids) + types = a.type.fillna("") + kc = np.flatnonzero(types.str.startswith("KC")).astype(np.int32) + reward = np.flatnonzero(types.eq("PAM11")).astype(np.int32) + aversive = np.flatnonzero(types.eq("PPL101")).astype(np.int32) + reward_mb = np.flatnonzero(types.eq("MBON07")).astype(np.int32) + aversive_mb = np.flatnonzero(types.eq("MBON11")).astype(np.int32) + if (len(reward), len(aversive), len(reward_mb), len(aversive_mb)) != (15, 2, 4, 2): + raise ValueError("Unexpected cell identities/counts for MaleCNS v1.0") + dan = np.r_[reward, aversive].astype(np.int32) + mb = np.r_[reward_mb, aversive_mb].astype(np.int32) + edges = np.flatnonzero(np.isin(brain.post, mb)).astype(np.int64) + pre = (np.searchsorted(brain.ptr, edges, side="right") - 1).astype(np.int32) + keep = np.isin(pre, kc) + edges = edges[keep] + pre = pre[keep] + if not len(edges) or np.any(brain.weight[edges] <= 0): + raise ValueError("Invalid reconstructed KC inputs") + gains = np.zeros((len(dan), len(edges)), dtype=np.float32) + for targets, drivers in [(reward_mb, reward), (aversive_mb, aversive)]: + for target in targets: + contact = [] + for d in drivers: + sl = slice(brain.ptr[d], brain.ptr[d + 1]) + contact.append( + float(np.abs(brain.weight[sl][brain.post[sl] == target]).sum()) + ) + contact = np.asarray(contact) + if contact.sum() <= 0: + raise ValueError("Missing direct DAN-to-MBON anatomical support") + selected = np.flatnonzero(brain.post[edges] == target) + for d, value in zip(drivers, contact / contact.sum()): + gains[np.flatnonzero(dan == d)[0], selected] = value + mask = np.zeros(brain.n, dtype=np.uint8) + mask[kc] = 1 + dan_index = np.full(brain.n, -1, dtype=np.int8) + dan_index[dan] = np.arange(len(dan)) + + def cells(ix): + return [ + { + "index": int(i), + "id": str(brain.ids[i]), + "type": str(types.iloc[i]), + "instance": str(a.instance.iloc[i]), + } + for i in ix + ] + + report = { + "release": "MaleCNS v1.0", + "neurons": brain.n, + "directed_edges": len(brain.post), + "plastic_edges": len(edges), + "plastic_edges_sha256": digest(edges), + "reward_cells": cells(reward), + "aversive_cells": cells(aversive), + "memory_outputs": cells(mb), + "selection": "All existing KC-to-MBON07/11 connections; no graph cropping or added edges.", + "gain": "Within-compartment DAN-to-MBON contact fractions; not measured receptor/dopamine kinetics.", + "validated": False, + } + return { + "kc": kc, + "mb": mb, + "dan": dan, + "reward": reward, + "aversive": aversive, + "edges": edges, + "pre": pre, + "gain": gains, + "kc_mask": mask, + "dan_index": dan_index, + "report": report, + } diff --git a/condor/fly/neural/common.py b/condor/fly/neural/common.py new file mode 100644 index 000000000..1d8cef7a0 --- /dev/null +++ b/condor/fly/neural/common.py @@ -0,0 +1,40 @@ +"""Local verified dataset and build cache; never a dependency on another repo.""" + +import hashlib +import json +import os +from pathlib import Path + +# Condor: the connectome data lives under this install's runtime root +# (``/.condor/fly/data``), overridable with ``CONDOR_FLY_DATA``. This is +# the only edit to the vendored stonkfly code. +from condor.paths import runtime_root + +DATA = Path( + os.environ.get("CONDOR_FLY_DATA") or (runtime_root() / "fly" / "data") +).resolve() +GRAPH = DATA / "graph.npz" +OUT = DATA / "cache" + + +def digest(array): + return hashlib.sha256(array.tobytes()).hexdigest() + + +def save_json(path, value): + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(path.suffix + ".partial") + tmp.write_text(json.dumps(value, indent=2, allow_nan=False) + "\n") + tmp.replace(path) + + +def annotations(ids): + import pyarrow.feather as f + + return ( + f.read_table(DATA / "annotations.feather") + .to_pandas() + .set_index("bodyId") + .loc[ids] + ) diff --git a/condor/fly/neural/connectome.py b/condor/fly/neural/connectome.py new file mode 100644 index 000000000..f7c941b38 --- /dev/null +++ b/condor/fly/neural/connectome.py @@ -0,0 +1,288 @@ +"""Loss-accounted import of the retained MaleCNS v1.0 graph for Stonkfly. + +This module stores topology and annotations. It does not infer missing dynamics, +neurotransmitter receptors, muscle mappings, or behavior from a wiring graph. +""" + +import argparse +import hashlib +import json +from pathlib import Path + +import numpy as np + +from .common import DATA + +REGISTRY = Path(__file__).with_name("datasets.json") + + +def exact_ids(values) -> np.ndarray: + """Never round 64-bit biological IDs through floating point or JavaScript.""" + items = np.asarray(values) + if items.dtype.kind == "f": + raise ValueError( + "Neuron IDs must be integers or decimal strings, never floats." + ) + if items.dtype.kind in "iu": + if np.any(items < 0): + raise ValueError("Neuron IDs cannot be negative.") + return items.astype(np.uint64) + text = [str(value) for value in items] + if any(not value.isascii() or not value.isdecimal() for value in text): + raise ValueError("Neuron IDs must be nonnegative decimal integers.") + return np.asarray(text, dtype=np.uint64) + + +def index_edges(ids: np.ndarray, pre, post, counts): + """Return retained edges plus a mask accounting for every excluded row.""" + if not len(ids) or np.any(ids[1:] <= ids[:-1]): + raise ValueError("Node IDs must be nonempty, unique, and sorted.") + pre, post = exact_ids(pre), exact_ids(post) + counts = np.asarray(counts) + if len(pre) != len(post) or len(pre) != len(counts): + raise ValueError("Edge columns have different lengths.") + if ( + not np.all(np.isfinite(counts)) + or np.any(counts < 1) + or np.any(counts != np.floor(counts)) + or np.any(counts > 2**32 - 1) + ): + raise ValueError("Synapse counts must be positive uint32-compatible integers.") + i, j = np.searchsorted(ids, pre), np.searchsorted(ids, post) + keep = (i < len(ids)) & (j < len(ids)) + keep &= ids[np.minimum(i, len(ids) - 1)] == pre + keep &= ids[np.minimum(j, len(ids) - 1)] == post + return ( + i[keep].astype(np.uint32), + j[keep].astype(np.uint32), + counts[keep].astype(np.uint32), + keep, + ) + + +def file_digest(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + while chunk := handle.read(8 * 1024 * 1024): + digest.update(chunk) + return digest.hexdigest() + + +def normalize_nodes(dataset_id: str, frame, nt_frame=None): + """Normalize fields while keeping the original annotations in raw files.""" + import pandas as pd + + if dataset_id == "malecns_v1": + source = exact_ids(frame.bodyId) + retain = frame.superclass.notna() & frame.superclass.astype(str).ne("") + quality = frame.statusLabel.astype(object).fillna("unknown") + superclasses, types = frame.superclass, frame.type + nonneural = frame.status.eq("Glia") + retain = retain & ~nonneural + predicted = pd.Series([None] * len(frame)) + nt_source = pd.Series(["missing"] * len(frame)) + if nt_frame is not None: + nt_frame = nt_frame.set_index("body") + if not nt_frame.index.is_unique: + raise ValueError("Duplicate male neurotransmitter IDs.") + predicted = frame.bodyId.map(nt_frame.consensus_nt) + nt_source = pd.Series( + ["source_consensus_prediction_or_ground_truth"] * len(frame) + ) + candidate_reason = np.where( + retain, "assigned_neuronal_superclass", "unresolved_object" + ) + else: + raise ValueError(f"Unsupported dataset: {dataset_id}") + if len(np.unique(source)) != len(source): + raise ValueError("Duplicate versioned neuron IDs in source annotations.") + catalog = pd.DataFrame( + { + "source_id": source, + "retained": np.asarray(retain, dtype=bool), + "object_kind": np.where( + nonneural, + "non_neuronal", + np.where(retain, "neuron_candidate", "unresolved_object"), + ), + "inclusion_reason": np.where( + nonneural, "explicit_non_neuronal_annotation", candidate_reason + ), + "quality": np.asarray(quality), + "superclass": np.asarray(superclasses), + "cell_type": np.asarray(types), + "neurotransmitter": np.asarray(predicted), + "neurotransmitter_source": np.asarray(nt_source), + } + ).sort_values("source_id", ignore_index=True) + nodes = catalog.loc[catalog.retained].reset_index(drop=True) + nodes.insert(0, "node_index", np.arange(len(nodes), dtype=np.uint32)) + return catalog, nodes + + +def import_graph(dataset_id: str = "malecns_v1") -> dict: + import pyarrow as pa + import pyarrow.feather as feather + import pyarrow.ipc as ipc + + config = json.loads(REGISTRY.read_text())["datasets"][dataset_id] + source_dir = DATA + output = source_dir / "normalized" + output.mkdir(parents=True, exist_ok=True) + lock_path = source_dir / "source.lock.json" + hashes = { + name: { + "url": url, + "bytes": (source_dir / name).stat().st_size, + "sha256": file_digest(source_dir / name), + } + for name, url in config["files"].items() + } + if lock_path.exists(): + locked = json.loads(lock_path.read_text()) + if any( + locked[name]["sha256"] != info["sha256"] for name, info in hashes.items() + ): + raise ValueError( + "Source files changed since the lock was created; use a new versioned dataset directory." + ) + else: + lock_path.write_text(json.dumps(hashes, indent=2) + "\n") + frame = feather.read_table(source_dir / "annotations.feather").to_pandas() + nt_frame = feather.read_table(source_dir / "neurotransmitters.feather").to_pandas() + catalog, nodes = normalize_nodes(dataset_id, frame, nt_frame) + feather.write_feather(catalog, output / "catalog.feather") + feather.write_feather(nodes, output / "neurons.feather") + ids = exact_ids(nodes.source_id) + np.save(output / "neuron_ids.npy", ids) + edge_columns = ("body_pre", "body_post", "weight") + reader = ipc.open_file(pa.memory_map(str(source_dir / "edges.feather"), "r")) + schema = pa.schema( + [ + ("pre_index", pa.uint32()), + ("post_index", pa.uint32()), + ("synapse_count", pa.uint32()), + ] + ) + stats = { + key: 0 + for key in [ + "source_edge_rows", + "retained_edge_rows", + "excluded_edge_rows", + "source_synaptic_contacts", + "retained_synaptic_contacts", + "excluded_synaptic_contacts", + "retained_weight_one_edges", + "retained_self_edges", + ] + } + incoming, outgoing = ( + np.zeros(len(ids), dtype=np.int64), + np.zeros(len(ids), dtype=np.int64), + ) + temporary = output / "edges.arrow.partial" + with pa.OSFile(str(temporary), "wb") as sink, ipc.new_file(sink, schema) as writer: + for number in range(reader.num_record_batches): + batch = reader.get_batch(number) + pre, post, weights = [ + batch.column(batch.schema.get_field_index(c)).to_numpy( + zero_copy_only=False + ) + for c in edge_columns + ] + i, j, count, keep = index_edges(ids, pre, post, weights) + stats["source_edge_rows"] += len(pre) + stats["retained_edge_rows"] += len(i) + stats["source_synaptic_contacts"] += int(weights.sum(dtype=np.uint64)) + stats["retained_synaptic_contacts"] += int(count.sum(dtype=np.uint64)) + stats["retained_weight_one_edges"] += int(np.count_nonzero(count == 1)) + stats["retained_self_edges"] += int(np.count_nonzero(i == j)) + np.add.at(incoming, j, count) + np.add.at(outgoing, i, count) + writer.write_batch( + pa.record_batch( + [pa.array(i), pa.array(j), pa.array(count)], schema=schema + ) + ) + temporary.replace(output / "edges.arrow") + stats["excluded_edge_rows"] = ( + stats["source_edge_rows"] - stats["retained_edge_rows"] + ) + stats["excluded_synaptic_contacts"] = ( + stats["source_synaptic_contacts"] - stats["retained_synaptic_contacts"] + ) + assert ( + int(incoming.sum()) + == int(outgoing.sum()) + == stats["retained_synaptic_contacts"] + ) + np.save(output / "incoming_synapse_counts.npy", incoming) + np.save(output / "outgoing_synapse_counts.npy", outgoing) + report = { + "dataset_id": dataset_id, + "sex": config["sex"], + "release": config["release"], + "coverage": config["coverage"], + "source_annotation_rows": len(catalog), + "retained_neuron_candidates": len(nodes), + "excluded_object_counts": catalog.loc[~catalog.retained] + .object_kind.value_counts() + .to_dict(), + "quality_counts": nodes.quality.value_counts(dropna=False).to_dict(), + "superclass_counts": nodes.superclass.fillna("unknown") + .value_counts() + .to_dict(), + "neurotransmitter_counts": nodes.neurotransmitter.fillna("missing") + .astype(str) + .value_counts() + .to_dict(), + "isolated_neurons": int(np.count_nonzero((incoming == 0) & (outgoing == 0))), + "node_policy": config["node_policy"], + "edge_policy": config["edge_policy"], + "upstream_filters": config["upstream_filters"], + "upstream_autapses_excluded": config["upstream_autapses_excluded"], + "additional_edge_strength_threshold": None, + "synaptic_weights_are_contact_counts": True, + "source_hashes": hashes, + "graph": stats, + "neural_dynamics_validated": False, + "embodied_behavior_implemented": False, + "remaining_gaps": [ + "Receptor-dependent synapse dynamics and neuromodulation", + "Calibrated retinal dynamics", + "Validated learning and controller interpretation", + ], + } + (output / "report.json").write_text(json.dumps(report, indent=2) + "\n") + public_report = DATA / "connectome-import.json" + public_report.parent.mkdir(parents=True, exist_ok=True) + public_report.write_text(json.dumps(report, indent=2) + "\n") + return report + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "dataset", choices=["malecns_v1"], nargs="?", default="malecns_v1" + ) + args = parser.parse_args() + report = import_graph(args.dataset) + print( + json.dumps( + { + k: report[k] + for k in [ + "dataset_id", + "retained_neuron_candidates", + "graph", + "neural_dynamics_validated", + ] + }, + indent=2, + ) + ) + + +if __name__ == "__main__": + main() diff --git a/condor/fly/neural/datasets.json b/condor/fly/neural/datasets.json new file mode 100644 index 000000000..27808f861 --- /dev/null +++ b/condor/fly/neural/datasets.json @@ -0,0 +1,25 @@ +{ + "source_checked": "2026-09-05", + "google_post": "https://research.google/blog/a-connectomics-milestone-mapping-the-complete-male-fruit-fly-brain/", + "datasets": { + "malecns_v1": { + "sex": "male", + "release": "MaleCNS v1.0", + "coverage": "brain_and_ventral_nerve_cord", + "source": "https://male-cns.janelia.org/download/", + "paper": "https://doi.org/10.1016/j.cell.2026.08.015", + "node_policy": "Every entry with an assigned superclass, including uncertain tbc classes; exclude explicit Glia status; no restriction to Traced status or typed cells.", + "edge_policy": "All released edges between retained entries; no additional weight threshold or removal of self-connections.", + "upstream_filters": [ + "Published pre/post synapse confidence threshold 0.5", + "Only annotated neuronal entries become simulation nodes; other segmentation objects remain in raw data" + ], + "upstream_autapses_excluded": false, + "files": { + "annotations.feather": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/body-annotations-male-cns-v1.0-minconf-0.5.feather", + "neurotransmitters.feather": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/body-neurotransmitters-male-cns-v1.0.feather", + "edges.feather": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/connectome-weights-male-cns-v1.0-minconf-0.5.feather" + } + } + } +} diff --git a/condor/fly/neural/kernel.cpp b/condor/fly/neural/kernel.cpp new file mode 100644 index 000000000..4b5b942ab --- /dev/null +++ b/condor/fly/neural/kernel.cpp @@ -0,0 +1,96 @@ +// Experimental dopamine-gated KC-to-MBON11 depression. +// Baseline integration is copied without changing its timestep/schedule. +// PPL101 edges deliver a modeled modulatory trace instead of fast excitation. +// Lazy exact subthreshold evolution. An inactive neuron is skipped only when +// its voltage AND both its instantaneous and asymptotic drive are below the +// threshold. With no incoming event it cannot fire. Every edge is retained. +#include +#include +#include +extern "C" void memory_advance( + int n,const int64_t* ptr,const int32_t* post,float* weight, + float* v,float* g,int16_t* refractory,const float* drive,float* previous_drive, + int32_t* queue,int32_t* queue_count,int64_t* clock,int steps,float dt,int32_t* counts, + int32_t* active,uint8_t* flags,int32_t* nactive,int64_t* last, + const uint8_t* kc_mask,const int8_t* dan_index,double* eligibility,int64_t* eligibility_last, + int nplastic,const int64_t* plastic_edge,const int32_t* plastic_pre, + const float* baseline_weight,const float* dan_gain,float eta,float tau_elig_ms,float floor_fraction, + int learning_enabled,float* modulation,int64_t* modulation_last,const uint8_t* modulation_mask,const float* rest, + float* adaptation,float adaptation_jump,float adaptation_tau) { + const int delay=std::lround(1.8f/dt),rfc=std::lround(2.2f/dt),slots=delay+1; + float av[1024],ag[1024],aa[1024]; + for(int i=0;i<1024;i++){av[i]=std::exp(-dt*i/20.f);ag[i]=std::exp(-dt*i/5.f);aa[i]=std::exp(-dt*i/adaptation_tau);} + auto evolve=[&](int i,int64_t now,float current){ + int64_t d=now-last[i];if(d<=0)return; + const int frozen=refractory[i]>0?refractory[i]-1:0; + const int skip=(int)(d0 && adaptation[i]>0)adaptation[i]*=skip<1024?aa[skip]:std::exp(-dt*skip/adaptation_tau); + refractory[i]=d>=refractory[i]?0:refractory[i]-d;d-=skip; + if(d>0){const float a=d<1024?av[d]:std::exp(-dt*d/20.f),b=d<1024?ag[d]:std::exp(-dt*d/5.f); + v[i]=rest[i]+(v[i]-rest[i])*a+current*(1.f-a)+g[i]*(a-b)/3.f;g[i]*=b; + if(adaptation[i]>0){const float c=d<1024?aa[d]:std::exp(-dt*d/adaptation_tau); + v[i]-=adaptation[i]*adaptation_tau/(adaptation_tau-20.f)*(c-a);adaptation[i]*=c;} + } + last[i]=now; + }; + auto awaken=[&](int i){if(!flags[i]){flags[i]=1;active[(*nactive)++]=i;}}; + // Apply changing sensory currents only after settling old-current history. + for(int i=0;i-45.f){queue[future*n+queue_count[future]++]=i;counts[i]++; + if(kc_mask[i]){ + adaptation[i]+=adaptation_jump; + eligibility[i]*=std::exp(-dt*(*clock-eligibility_last[i])/tau_elig_ms); + eligibility[i]+=1.;eligibility_last[i]=*clock; + } + } + // Convex relaxation toward drive+g(t): the bound makes this exact in + // real arithmetic, not an activity cutoff or a dropped weak connection. + const float gap=-45.f-rest[i]; + const bool can_fire=v[i]>-45.f || drive[i]>gap || drive[i]+g[i]>gap; + if(can_fire)active[kept++]=i;else flags[i]=0; + } + *nactive=kept; + for(int q=0;q=0){ + for(int p=0;plower?candidate:lower; + } + } + continue; + } + for(int64_t e=ptr[i];e 0.0100001: + raise ValueError("Rate bins must be 0--10 ms") + h = dt_seconds + p = PARAMETERS + ak = math.exp(-h / p["trace_kc_seconds"]) + ad = math.exp(-h / p["trace_dan_seconds"]) + kmid = y_kc * math.sqrt(ak) + kc_hz * (1 - math.sqrt(ak)) + dmid = y_dan * math.sqrt(ad) + dan_hz * (1 - math.sqrt(ad)) + y_kc[:] = y_kc * ak + kc_hz * (1 - ak) + y_dan[:] = y_dan * ad + dan_hz * (1 - ad) + if frozen: + return + drive = ( + eta * (kc_hz * (gain.T @ dmid) - (gain.T @ dan_hz) * kmid) + if learning + else np.zeros_like(u) + ) + tu = p["memory_decay_seconds"] + tw = p["weight_filter_seconds"] + eu = math.exp(-h / tu) + ew = math.exp(-h / tw) + c = tu / (tu - tw) * (eu - ew) + old_u = u.copy() + u[:] = old_u * eu + drive * tu * (-math.expm1(-h / tu)) + w[:] = w * ew + old_u * c + drive * tu * (-math.expm1(-h / tw) - c) + # Bounds preserve the reconstructed excitatory sign. Saturation is reported, + # not hidden or reset when an assay has an unfavorable outcome. + lo = p["minimum_fraction"] - 1 + hi = p["maximum_fraction"] - 1 + np.clip(u, lo, hi, out=u) + np.clip(w, lo, hi, out=w) diff --git a/condor/fly/neural/sensory.py b/condor/fly/neural/sensory.py new file mode 100644 index 000000000..d83a32d3c --- /dev/null +++ b/condor/fly/neural/sensory.py @@ -0,0 +1,17 @@ +"""Display brightness proxy; not a calibrated compound-eye model.""" + +import numpy as np + + +def retinal_samples(frame, uv): + frame = np.asarray(frame) + if frame.ndim != 3 or frame.shape[2] != 3 or frame.dtype != np.uint8: + raise ValueError("RGB uint8 frame required") + h, w = frame.shape[:2] + x = np.clip((uv[:, 0] * (w - 1)).astype(int), 0, w - 1) + y = np.clip((uv[:, 1] * (h - 1)).astype(int), 0, h - 1) + rgb = frame[y, x].astype(np.float32) / 255 + linear = np.where(rgb <= 0.04045, rgb / 12.92, ((rgb + 0.055) / 1.055) ** 2.4) + return (linear @ np.asarray([0.2126, 0.7152, 0.0722], dtype=np.float32)).astype( + np.float32 + ) diff --git a/condor/fly/neural/sources.lock.json b/condor/fly/neural/sources.lock.json new file mode 100644 index 000000000..addd65565 --- /dev/null +++ b/condor/fly/neural/sources.lock.json @@ -0,0 +1,17 @@ +{ + "annotations.feather": { + "url": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/body-annotations-male-cns-v1.0-minconf-0.5.feather", + "bytes": 14483314, + "sha256": "2177e246113e4cfbf1e7772ec37c6da1955ff22e8063d0b1f833101f99a9a3b2" + }, + "neurotransmitters.feather": { + "url": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/body-neurotransmitters-male-cns-v1.0.feather", + "bytes": 43282834, + "sha256": "95c9289220663abeb3409f3ad9e5a7f8a53f8093f5139d15502cd08da8879621" + }, + "edges.feather": { + "url": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/connectome-weights-male-cns-v1.0-minconf-0.5.feather", + "bytes": 1051241946, + "sha256": "e35da783d1c686b2b58b3b87cd6a403ae43bfcfba8bff28e08ef752c1a56afc1" + } +} diff --git a/condor/fly/neural/state.py b/condor/fly/neural/state.py new file mode 100644 index 000000000..5800a6b7b --- /dev/null +++ b/condor/fly/neural/state.py @@ -0,0 +1,82 @@ +"""State allocation for the retained connectome; native integration is separate.""" + +import numpy as np + + +class Brain: + def __init__(self, path, dt=0.1): + if dt != 0.1: + raise ValueError("This audited kernel supports only dt=0.1 ms.") + a = np.load(path) + for k in [ + "ptr", + "post", + "weight", + "ids", + "retina", + "uv", + "lamina", + "sugar", + "superclass", + ]: + setattr(self, k, a[k]) + n = len(self.ids) + for k, dtype in [ + ("ptr", np.int64), + ("post", np.int32), + ("weight", np.float32), + ("ids", np.int64), + ("retina", np.int32), + ("lamina", np.int32), + ("sugar", np.int32), + ]: + x = getattr(self, k) + if x.ndim != 1 or x.dtype != dtype or not x.flags.c_contiguous: + raise ValueError(f"Invalid native graph array: {k}") + if ( + n < 1 + or self.ptr.shape != (n + 1,) + or self.ptr[0] != 0 + or self.ptr[-1] != len(self.post) + or np.any(np.diff(self.ptr) < 0) + or len(self.weight) != len(self.post) + ): + raise ValueError("Invalid CSR graph") + if not np.isfinite(self.weight).all(): + raise ValueError("Nonfinite synaptic weight") + for x in [self.post, self.retina, self.lamina, self.sugar]: + if np.any(x < 0) or np.any(x >= n): + raise ValueError("Graph index out of bounds") + if ( + self.uv.shape != (len(self.retina), 2) + or not np.isfinite(self.uv).all() + or np.any(self.uv < 0) + or np.any(self.uv > 1) + ): + raise ValueError("Invalid receptor UV coordinates") + self.dt = dt + self.n = len(self.ids) + self.cursor = 0 + self.v = np.full(self.n, -52, dtype=np.float32) + self.g = np.zeros(self.n, dtype=np.float32) + self.drive = np.zeros(self.n, dtype=np.float32) + self.refractory = np.zeros(self.n, dtype=np.int16) + self.queue = np.zeros((int(round(1.8 / dt)) + 1, self.n), dtype=np.int32) + self.queue_count = np.zeros(self.queue.shape[0], dtype=np.int32) + self.counts = np.zeros(self.n, dtype=np.int32) + self.luminance = np.zeros(len(self.retina), dtype=np.float32) + self.active = np.zeros(self.n, dtype=np.int32) + self.active_flag = np.zeros(self.n, dtype=np.uint8) + initial = np.unique(np.r_[self.retina, self.lamina, self.sugar]) + self.active[: len(initial)] = initial + self.active_flag[initial] = 1 + self.nactive = np.asarray([len(initial)], dtype=np.int32) + self.total_spikes = 0 + self.sim_ms = 0 + + +class NativeBrain(Brain): + def __init__(self, path, dt=0.1): + super().__init__(path, dt) + self.previous_drive = np.zeros(self.n, dtype=np.float32) + self.last = np.full(self.n, -1, dtype=np.int64) diff --git a/condor/fly/neural/transmitters.py b/condor/fly/neural/transmitters.py new file mode 100644 index 000000000..995d1a800 --- /dev/null +++ b/condor/fly/neural/transmitters.py @@ -0,0 +1,24 @@ +"""Declared neurotransmitter-sign proxy used in DOOMFLY graph preparation.""" + +import numpy as np + + +def transmitter_signs(transmitters, ambiguous_sign=1): + """Declared coarse fast-transmission assumption; never deletes unknown edges. + + ACh +; GABA, glutamate, histamine -. A co-transmitter combination with only + one fast sign uses that sign; conflicting, missing and modulator-only cells + use the explicit sensitivity parameter. This is NOT receptor physiology. + """ + if ambiguous_sign not in (-1, 1): + raise ValueError("Ambiguous edges must remain active with sign +1 or -1.") + signs, uncertain = [], [] + for value in transmitters: + tokens = set(str(value).lower().split(",")) + fast = ({1} if "acetylcholine" in tokens else set()) | ( + {-1} if tokens & {"gaba", "glutamate", "histamine"} else set() + ) + ambiguous = len(fast) != 1 + signs.append(ambiguous_sign if ambiguous else next(iter(fast))) + uncertain.append(ambiguous) + return np.asarray(signs, dtype=np.int8), np.asarray(uncertain, dtype=bool) diff --git a/condor/fly/neural/visual.py b/condor/fly/neural/visual.py new file mode 100644 index 000000000..3dbae6249 --- /dev/null +++ b/condor/fly/neural/visual.py @@ -0,0 +1,155 @@ +"""Candidate R8 display adapter and target-specific cotransmission. + +Anatomy remains exactly MaleCNS. R8->aMe12 net excitation is supported by +Xiao et al. Nature 2023, doi:10.1038/s41586-023-06681-6 (AMA includes aMe12). +The transferred type correspondence, unitary magnitude, LIF photoreceptor +proxy and RGB-to-opsin transfer are explicit modeling assumptions. No UV +channel or sensitivity is invented for R7 or untyped photoreceptors. +""" + +import math + +import numpy as np + +from .brain import MemoryBrain +from .common import annotations, digest + + +def projection(brain, a): + known = np.flatnonzero(a.type.isin(["R8p", "R8y"])) + mapped = [] + hexes = [] + confidence = [] + for i in known: + edges = np.arange(brain.ptr[i], brain.ptr[i + 1]) + targets = brain.post[edges] + valid = ( + a.assignedOlHex1.iloc[targets].notna().to_numpy() + & a.assignedOlHex2.iloc[targets].notna().to_numpy() + ) + votes = {} + for e, j in zip(edges[valid], targets[valid]): + h = (float(a.assignedOlHex1.iloc[j]), float(a.assignedOlHex2.iloc[j])) + votes[h] = votes.get(h, 0) + float(abs(brain.weight[e])) + if not votes: + continue + h = max(votes, key=votes.get) + mapped.append(i) + hexes.append(h) + confidence.append(votes[h] / sum(votes.values())) + mapped = np.asarray(mapped, dtype=np.int32) + hexes = np.asarray(hexes) + xy = np.column_stack( + [hexes[:, 0] - 0.5 * hexes[:, 1], np.sqrt(3) / 2 * hexes[:, 1]] + ) + # Fit the original display projection from the existing R1-R6 anchors so + # R8 does not get a separately stretched eye or arbitrary target position. + from .common import GRAPH + + with np.load(GRAPH) as g: + oldhex = g["hexes"] + oldxy = np.column_stack( + [oldhex[:, 0] - 0.5 * oldhex[:, 1], np.sqrt(3) / 2 * oldhex[:, 1]] + ) + uv = np.empty_like(xy) + for side in ["L", "R"]: + original = a.rootSide.iloc[brain.retina].eq(side).to_numpy() + select = a.rootSide.iloc[mapped].eq(side).to_numpy() + lo = oldxy[original].min(axis=0) + span = np.ptp(oldxy[original], axis=0) + z = (xy[select] - lo) / span + uv[select, 0] = 0.6 * z[:, 0] if side == "L" else 0.4 + 0.6 * (1 - z[:, 0]) + uv[select, 1] = 1 - z[:, 1] + if not a.rootSide.iloc[mapped].isin(["L", "R"]).all(): + raise ValueError("Unresolved eye side") + return ( + mapped, + np.clip(uv, 0, 1).astype(np.float32), + np.asarray(confidence, dtype=np.float32), + ) + + +class VisualMemoryBrain(MemoryBrain): + def __init__(self, **kwargs): + super().__init__(**kwargs) + a = annotations(self.ids) + self.r8, self.r8_uv, self.r8_confidence = projection(self, a) + self.r8_channel = np.where(a.type.iloc[self.r8].eq("R8p"), 2, 1).astype( + np.int32 + ) + self.r8_light = np.zeros(len(self.r8), dtype=np.float32) + corrected = [] + for i in np.flatnonzero(a.type.fillna("").str.startswith("R8")): + edges = np.arange(self.ptr[i], self.ptr[i + 1]) + e = edges[a.type.iloc[self.post[edges]].eq("aMe12").to_numpy()] + corrected.extend(e.tolist()) + self.weight[e] = np.abs(self.weight[e]) + self.corrected_edges = np.asarray(corrected, dtype=np.int64) + self.initial_weight_sha256 = digest(self.weight) + self.fields.append("r8_light") + self.initial["r8_light"] = self.r8_light.copy() + self.visual_report = { + "model": "r8-rgb-ame12-v1", + "mapped_R8p": int((self.r8_channel == 2).sum()), + "mapped_R8y": int((self.r8_channel == 1).sum()), + "known_unmapped": int(a.type.isin(["R8p", "R8y"]).sum() - len(self.r8)), + "projection_confidence_median": float(np.median(self.r8_confidence)), + "projection_below_80_percent": int((self.r8_confidence < 0.8).sum()), + "corrected_existing_edges": len(corrected), + "corrected_edge_sha256": digest(self.corrected_edges), + "coordinate_inference": "Modal column of all outgoing contacts to any column-annotated target; same viewport transform as R1-R6.", + "spectrum": "Linear sRGB B for R8p; G for R8y. Display proxy, not calibrated photon flux or spectral sensitivity. R7, dorsal and untyped R8 receive no invented optical drive.", + "physiology": "R8 to aMe12 net sign positive; original contact magnitudes retained. Other R8 targets retain baseline sign. Photoreceptors still use a LIF rate proxy, not graded in-vivo dynamics.", + "evidence": [ + "https://doi.org/10.1038/s41586-023-06681-6", + "https://doi.org/10.1038/s41467-024-49616-z", + "https://doi.org/10.7554/eLife.71858", + ], + "validated": False, + } + + def rgb_step(self, frame, duration_ms, **kwargs): + if duration_ms > 10: + ticks = round(duration_ms / self.dt) + total = np.zeros(self.n, dtype=np.int32) + wall = 0.0 + while ticks: + n = min(100, ticks) + c, t = self.rgb_step(frame, n * self.dt, **kwargs) + total += c + wall += t + ticks -= n + self.counts[:] = total + return total, wall + from .sensory import retinal_samples + + frame = np.asarray(frame) + if frame.ndim != 3 or frame.shape[2] != 3 or frame.dtype != np.uint8: + raise ValueError("RGB uint8 required") + h, w = frame.shape[:2] + x = np.minimum((self.r8_uv[:, 0] * (w - 1)).astype(int), w - 1) + y = np.minimum((self.r8_uv[:, 1] * (h - 1)).astype(int), h - 1) + values = frame[y, x, self.r8_channel].astype(np.float32) / 255 + values = np.where( + values <= 0.04045, values / 12.92, ((values + 0.055) / 1.055) ** 2.4 + ) + self.r8_light += ( + 1 - math.exp(-round(duration_ms / self.dt) * self.dt / 10) + ) * (values - self.r8_light) + extra = kwargs.pop("stimulation", None) + pulses = ( + [] if extra is None else list(extra) if isinstance(extra, list) else [extra] + ) + pulses.append((self.r8, 30 * self.r8_light / (0.02 + self.r8_light))) + return self.step( + retinal_samples(frame, self.uv), duration_ms, stimulation=pulses, **kwargs + ) + + def configuration_signature(self): + return { + **super().configuration_signature(), + **{ + k: digest(getattr(self, k)) + for k in ["r8", "r8_uv", "r8_channel", "corrected_edges"] + }, + } diff --git a/condor/fly/posture.py b/condor/fly/posture.py new file mode 100644 index 000000000..163cff2bc --- /dev/null +++ b/condor/fly/posture.py @@ -0,0 +1,137 @@ +"""Turn a posture into a full ``pmm_mister`` config. + +The base is the HIP-3 operator's bounded defaults; the posture multiplies the +spreads and leans them. Every money-relevant floor lives here, in code: + +* no spread level below ``min_spread_bps``; +* ``take_profit`` never below ``2.2 ×`` the round-trip maker fee, and never + below 4 bp (Market Making Expert's "TP must exceed round-trip fees" made + mandatory); +* the reference-price lean is capped at half the first-level spread. + +Inventory bands, allocation, leverage cap and the global stop loss are fixed by +the HIP-3 playbook and are not the fly's to move. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass + +from condor.fly.decoder import REGIMES, Posture + +# executor_refresh_time, buy/sell cooldown — Market Making Expert's table. +TIMING: dict[str, tuple[int, int]] = { + "quiet": (20, 30), + "ranging": (30, 60), + "trending_up": (30, 60), + "trending_down": (30, 60), + "volatile": (60, 120), + "pause": (60, 120), +} +assert set(TIMING) == set(REGIMES) + +BPS = 1e-4 + + +@dataclass(frozen=True) +class MarketSpec: + """What the operator settles once per deployment; the fly never changes it.""" + + connector_name: str + trading_pair: str # UPPERCASE issuer prefix, e.g. XYZ:DRAM-USD + total_amount_quote: float + picked_spread_bps: float # the scanner's spread for this market + leverage: int = 3 + maker_fee_bps: float = 1.3 # HIP-3 all-in maker fee per side incl. builder fee + min_spread_bps: float = 3.0 + portfolio_allocation: float = 0.2 + target_base_pct: float = 0.4 + min_base_pct: float = 0.3 + max_base_pct: float = 0.5 + max_active_executors_by_level: int = 2 + global_stop_loss: float = 0.02 + leverage_cap: int = 5 + + def __post_init__(self): + if self.trading_pair != self.trading_pair.upper(): + raise ValueError( + f"HIP-3 trading_pair must be uppercase, got {self.trading_pair!r}" + ) + if not self.trading_pair.endswith("-USD") or ":" not in self.trading_pair: + raise ValueError("HIP-3 pair must look like ISSUER:TOKEN-USD") + for name in ( + "total_amount_quote", + "picked_spread_bps", + "maker_fee_bps", + "min_spread_bps", + ): + value = getattr(self, name) + if not math.isfinite(value) or value <= 0: + raise ValueError(f"{name} must be finite and positive") + if not 1 <= self.leverage <= self.leverage_cap: + raise ValueError(f"leverage must be within 1..{self.leverage_cap}") + if not 0 < self.min_base_pct < self.target_base_pct < self.max_base_pct < 1: + raise ValueError("0 < min_base < target_base < max_base < 1 required") + if not 0 < self.portfolio_allocation <= 1: + raise ValueError("portfolio_allocation must be in (0, 1]") + + +def take_profit_floor(spec: MarketSpec) -> float: + return round(max(4 * BPS, 2.2 * 2 * spec.maker_fee_bps * BPS), 8) + + +def base_levels_bps(spec: MarketSpec) -> tuple[float, float]: + """HIP-3 playbook: level 1 ``max(2, S/2)`` bp, level 2 ``S+1`` bp.""" + s = spec.picked_spread_bps + return max(2.0, s / 2), s + 1 + + +def _fmt(values: list[float]) -> str: + return ",".join(f"{v:.6f}".rstrip("0").rstrip(".") for v in values) + + +def build_config(spec: MarketSpec, posture: Posture) -> dict: + if posture.regime not in TIMING: + raise ValueError(f"Unknown regime {posture.regime!r}") + l1, l2 = base_levels_bps(spec) + levels = [l1 * posture.spread_mult, l2 * posture.spread_mult] + shift = max(-levels[0] / 2, min(levels[0] / 2, posture.shift_bps)) + buy = [max(spec.min_spread_bps, lvl - shift) for lvl in levels] + sell = [max(spec.min_spread_bps, lvl + shift) for lvl in levels] + take_profit = max(take_profit_floor(spec), min(buy[0], sell[0]) * BPS) + refresh, cooldown = TIMING[posture.regime] + return { + "controller_type": "generic", + "controller_name": "pmm_mister", + "connector_name": spec.connector_name, + "trading_pair": spec.trading_pair, + "total_amount_quote": spec.total_amount_quote, + "portfolio_allocation": spec.portfolio_allocation, + "leverage": spec.leverage, + "position_mode": "ONEWAY", + "target_base_pct": spec.target_base_pct, + "min_base_pct": spec.min_base_pct, + "max_base_pct": spec.max_base_pct, + "buy_spreads": _fmt([b * BPS for b in buy]), + "sell_spreads": _fmt([s * BPS for s in sell]), + "buy_amounts_pct": "1,1", + "sell_amounts_pct": "1,1", + "executor_refresh_time": refresh, + "buy_cooldown_time": cooldown, + "sell_cooldown_time": cooldown, + "take_profit": round(take_profit, 8), + "take_profit_order_type": 3, + "open_order_type": 3, + "max_active_executors_by_level": spec.max_active_executors_by_level, + "global_sl_enabled": True, + "global_stop_loss": spec.global_stop_loss, + "manual_kill_switch": posture.regime == "pause", + } + + +def config_diff(old: dict | None, new: dict) -> dict: + """Fields whose value changed; the whole config when there was none.""" + if old is None: + return dict(new) + return {k: v for k, v in new.items() if old.get(k) != v} diff --git a/condor/fly/reinforcement.py b/condor/fly/reinforcement.py new file mode 100644 index 000000000..e501e5a8b --- /dev/null +++ b/condor/fly/reinforcement.py @@ -0,0 +1,56 @@ +"""Turn a change in the fly's P&L into a dopamine pulse kind. + +Stonkfly's rule, with the equity source changed: ``equity`` is the combined net +P&L of the ``pmm_mister`` controllers the fly is quoting for — realized plus +unrealized minus fees — so nothing else on the account can reward or punish it. + +The pulse is binary above a deadband, not proportional, and tiny changes are +not accumulated. It is feedback about value between two observations; it is +not evidence that the last posture caused that change. +""" + +from __future__ import annotations + +import math +from decimal import Decimal +from typing import Literal + +Kind = Literal["reward", "aversive", "none"] + + +def D(value) -> Decimal: + if isinstance(value, bool): + raise ValueError("Boolean is not money") + number = Decimal(str(value)) + if not number.is_finite(): + raise ValueError("Nonfinite quantity") + return number + + +def reinforcement(equity, anchor, deadband) -> tuple[Kind, Decimal]: + """``(kind, delta)`` with ``delta = equity - anchor``.""" + delta = D(equity) - D(anchor) + threshold = D(deadband) + if threshold <= 0: + raise ValueError("Positive reinforcement deadband required") + if delta >= threshold: + return "reward", delta + if delta <= -threshold: + return "aversive", delta + return "none", delta + + +def controller_net(performance: dict) -> float: + """``realized + unrealized`` in quote for one controller's performance + block (the ``performance`` dict inside ``get_active_bots_status``). Fees + are already netted into Hummingbot's realized figure.""" + realized = performance.get("realized_pnl_quote") + unrealized = performance.get("unrealized_pnl_quote") + if realized is None or unrealized is None: + raise ValueError( + "Controller performance lacks realized_pnl_quote/unrealized_pnl_quote" + ) + total = float(realized) + float(unrealized) + if not math.isfinite(total): + raise ValueError("Nonfinite controller P&L") + return total diff --git a/condor/fly/run_state.py b/condor/fly/run_state.py new file mode 100644 index 000000000..fa3456abe --- /dev/null +++ b/condor/fly/run_state.py @@ -0,0 +1,134 @@ +"""One fly run's durable state on disk. + +Stonkfly's pattern: a lock so two workers never share a run directory, two +alternating brain checkpoints so a crash mid-write leaves the previous one +intact, an append-only ``events.jsonl`` audit trail, ``latest.json`` and +``latest-input.png`` for "what did the fly just see and do", and a provenance +signature that refuses to resume a run under a changed protocol. +""" + +from __future__ import annotations + +import fcntl +import hashlib +import json +from pathlib import Path + +import numpy as np +from PIL import Image + +from condor.fsutil import atomic_write_bytes, atomic_write_json + +FLY_PACKAGE = Path(__file__).resolve().parent + + +def source_hashes() -> dict[str, str]: + return { + str(path.relative_to(FLY_PACKAGE)): hashlib.sha256( + path.read_bytes() + ).hexdigest() + for path in sorted(FLY_PACKAGE.rglob("*")) + if path.suffix in (".py", ".cpp") and path.is_file() + } + + +def signature(provenance: dict) -> str: + return hashlib.sha256( + json.dumps(provenance, sort_keys=True, default=str).encode() + ).hexdigest() + + +class RunDir: + def __init__(self, root: Path): + self.root = Path(root) + self.root.mkdir(parents=True, exist_ok=True) + self.state_path = self.root / "state.json" + self.events_path = self.root / "events.jsonl" + self.latest_path = self.root / "latest.json" + self.frame_path = self.root / "latest-input.png" + self.provenance_path = self.root / "provenance.json" + self.lock_path = self.root / "worker.lock" + self.stop_path = self.root / "STOP" + self._lock = None + + # -- exclusivity --------------------------------------------------------- + + def lock(self) -> None: + handle = self.lock_path.open("a") + try: + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + handle.close() + raise RuntimeError(f"A fly worker already owns {self.root}") + self._lock = handle + + def unlock(self) -> None: + if self._lock is not None: + fcntl.flock(self._lock, fcntl.LOCK_UN) + self._lock.close() + self._lock = None + + def stop_requested(self) -> bool: + return self.stop_path.exists() + + # -- state --------------------------------------------------------------- + + def load_state(self) -> dict: + if not self.state_path.exists(): + return {} + return json.loads(self.state_path.read_text()) + + def save_state(self, state: dict) -> None: + atomic_write_json(self.state_path, state, indent=2, default=str) + + def append_event(self, row: dict) -> None: + with self.events_path.open("a") as handle: + handle.write(json.dumps(row, allow_nan=False, default=str) + "\n") + + def write_latest(self, row: dict) -> None: + atomic_write_json(self.latest_path, row, indent=2, default=str) + + def save_frame(self, frame: np.ndarray) -> None: + Image.fromarray(frame).save(self.frame_path) + + def recent_events(self, limit: int) -> list[dict]: + if not self.events_path.exists(): + return [] + lines = self.events_path.read_text().splitlines()[-limit:] + return [json.loads(line) for line in lines if line.strip()] + + # -- checkpoints --------------------------------------------------------- + + def checkpoint_path(self, tick: int) -> Path: + return self.root / f"brain-{tick % 2}.npz" + + def verify_checkpoint(self, info: dict) -> Path: + path = self.root / info["file"] + if not path.exists(): + raise RuntimeError( + f"Checkpoint {path.name} recorded in state.json is missing" + ) + if hashlib.sha256(path.read_bytes()).hexdigest() != info["sha256"]: + raise RuntimeError(f"Checkpoint {path.name} integrity mismatch") + return path + + # -- provenance ---------------------------------------------------------- + + def check_provenance(self, provenance: dict) -> str: + """Write the run's provenance on first start; refuse a changed one.""" + sig = signature(provenance) + if self.provenance_path.exists(): + recorded = json.loads(self.provenance_path.read_text()) + if recorded.get("signature") != sig: + raise RuntimeError( + "Run protocol changed (settings, decoder, dataset or source); " + "use a new run_name or review the migration explicitly" + ) + return sig + atomic_write_json( + self.provenance_path, + {"signature": sig, **provenance}, + indent=2, + default=str, + ) + return sig diff --git a/condor/fly/worker.py b/condor/fly/worker.py new file mode 100644 index 000000000..388f89c40 --- /dev/null +++ b/condor/fly/worker.py @@ -0,0 +1,179 @@ +"""The brain lives here, in its own process. + +``FlyBrain`` wraps stonkfly's ``VisualMemoryBrain`` and measures the three +channels the decoder reads. The module-level ``_init`` / ``_observe`` / +``_checkpoint`` / ``_restore`` functions are the ``ProcessPoolExecutor`` +targets the ``fly_brain`` routine submits, so the C++ integration never runs on +Condor's event loop. One worker holds one brain; the routine creates the pool +with ``max_workers=1`` and the ``spawn`` context. +""" + +from __future__ import annotations + +import hashlib +import math +from pathlib import Path + +import numpy as np + +# MaleCNS v1.0 superclass label of the 1,314 descending neurons in graph.npz. +DESCENDING_SUPERCLASS = "descending_neuron" + + +class FlyBrain: + def __init__( + self, + learning: bool = True, + neural_bin_ms: float = 10.0, + pulse_ms: float = 200.0, + pulse_current: float = 20.0, + ): + from condor.fly.neural.common import annotations + from condor.fly.neural.visual import VisualMemoryBrain + + if neural_bin_ms <= 0 or neural_bin_ms > 10: + raise ValueError("neural_bin_ms must be in (0, 10]") + if pulse_ms <= 0 or pulse_current <= 0: + raise ValueError("pulse_ms and pulse_current must be positive") + self.neural_bin_ms = neural_bin_ms + self.pulse_ms = pulse_ms + self.pulse_current = pulse_current + self.brain = VisualMemoryBrain() + self.brain.weights_frozen = not learning + a = annotations(self.brain.ids) + types = a.type.fillna("") + sides = a.somaSide.fillna("") + self.left = np.flatnonzero(types.eq("DNp20") & sides.eq("L")) + self.right = np.flatnonzero(types.eq("DNp20") & sides.eq("R")) + self.gate = np.flatnonzero(types.eq("DNpe017")) + if not len(self.left) or not len(self.right) or not len(self.gate): + raise RuntimeError("Missing annotated DNp20 / DNpe017 readout cells") + superclass = np.asarray(self.brain.superclass).astype(str) + readouts = np.concatenate([self.left, self.right, self.gate]) + descending = np.flatnonzero(superclass == DESCENDING_SUPERCLASS) + self.descending = np.setdiff1d(descending, readouts) + if not len(self.descending): + raise RuntimeError( + f"No {DESCENDING_SUPERCLASS!r} superclass in graph.npz; " + f"available: {sorted(set(superclass.tolist()))}" + ) + self.cell_ids = { + "left": [str(self.brain.ids[i]) for i in self.left], + "right": [str(self.brain.ids[i]) for i in self.right], + "gate": [str(self.brain.ids[i]) for i in self.gate], + "descending_count": int(len(self.descending)), + } + + def observe( + self, frame: np.ndarray, reinforcement: str, neural_ms: float = 500.0 + ) -> dict: + if reinforcement not in ("none", "reward", "aversive"): + raise ValueError("Unknown reinforcement") + if not math.isfinite(neural_ms) or neural_ms < self.pulse_ms: + raise ValueError("neural_ms must be finite and >= pulse_ms") + frame = np.asarray(frame) + if frame.shape != (180, 320, 3) or frame.dtype != np.uint8: + raise ValueError("frame must be uint8[180, 320, 3]") + b = self.brain + counts = np.zeros(b.n, dtype=np.int32) + wall = 0.0 + remaining = round(neural_ms / b.dt) + pulse = round(self.pulse_ms / b.dt) if reinforcement != "none" else 0 + delivered = 0 + learning = not b.weights_frozen + while remaining: + n = min(remaining, round(self.neural_bin_ms / b.dt)) + if pulse: + n = min(n, pulse) + stimulus = (b.circuit[reinforcement], self.pulse_current) if pulse else None + c, elapsed = b.rgb_step( + frame, n * b.dt, learning=learning, stimulation=stimulus + ) + counts += c + wall += elapsed + remaining -= n + if pulse: + delivered += n + pulse -= n + b.counts[:] = counts + seconds = neural_ms / 1000 + left = float(np.mean(counts[self.left]) / seconds) + right = float(np.mean(counts[self.right]) / seconds) + return { + "trend_hz": right - left, + "left_hz": left, + "right_hz": right, + "arousal_hz": float(np.mean(counts[self.descending]) / seconds), + "gate_spikes": int(counts[self.gate].sum()), + "kc_spikes": int(counts[b.circuit["kc"]].sum()), + "reward_spikes": int(counts[b.circuit["reward"]].sum()), + "aversive_spikes": int(counts[b.circuit["aversive"]].sum()), + "total_spikes": int(counts.sum()), + "stimulus": reinforcement, + "stimulus_ms": delivered * b.dt, + "brain_ms": b.sim_ms, + "compute_seconds": wall, + "spike_sha256": hashlib.sha256(counts.tobytes()).hexdigest(), + "input_sha256": hashlib.sha256(frame.tobytes()).hexdigest(), + "memory": b.memory(), + "cell_ids": self.cell_ids, + } + + def save(self, path: Path) -> None: + self.brain.checkpoint(path) + + def restore(self, path: Path) -> None: + self.brain.restore(path) + + def provenance(self) -> dict: + return { + "circuit": self.brain.circuit["report"], + "vision": self.brain.visual_report, + "readout": { + "trend": "DNp20 mean right minus left rate", + "arousal": f"mean rate of superclass={DESCENDING_SUPERCLASS!r} minus readouts", + "gate": "DNpe017 spike count >= 1", + "cells": self.cell_ids, + "validated": False, + }, + } + + +# ---- ProcessPoolExecutor targets (one brain per worker process) ---------- + +_BRAIN: FlyBrain | None = None + + +def _init( + learning: bool, neural_bin_ms: float, pulse_ms: float, pulse_current: float +) -> None: + global _BRAIN + _BRAIN = FlyBrain( + learning=learning, + neural_bin_ms=neural_bin_ms, + pulse_ms=pulse_ms, + pulse_current=pulse_current, + ) + + +def _brain() -> FlyBrain: + if _BRAIN is None: + raise RuntimeError("worker not initialized") + return _BRAIN + + +def _observe(frame: np.ndarray, reinforcement: str, neural_ms: float) -> dict: + return _brain().observe(frame, reinforcement, neural_ms) + + +def _checkpoint(path: str) -> str: + _brain().save(Path(path)) + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +def _restore(path: str) -> None: + _brain().restore(Path(path)) + + +def _provenance() -> dict: + return _brain().provenance() diff --git a/docs/market_making_fly_design.md b/docs/market_making_fly_design.md new file mode 100644 index 000000000..f6cba31dd --- /dev/null +++ b/docs/market_making_fly_design.md @@ -0,0 +1,571 @@ +# Market Making Fly — implementation design + +Status: **draft for review, nothing implemented yet.** +Date: 2026-09-12 + +## 1. Summary + +Market Making Fly is a new Condor agent modeled on Market Making Expert, but the +discretionary part of market making — *what regime are we in, how wide should +we quote, which way should the reference price lean* — is not decided by an LLM +or by indicator thresholds. It is decoded from the spiking activity of a +simulation of the **MaleCNS v1.0 fly connectome** (166,700 neurons, 25.6 M +directed connections) that is shown a rendered **OHLCV candlestick chart** of the +market, exactly the way [stonkfly](https://github.com/nftechie/stonkfly) shows +a fly a line chart of BTC-USDC and reads buy/sell/hold out of its descending +neurons. + +The strategy P&L is fed back into the fly the same way stonkfly does it: a +positive change pulses the 15 **PAM11** reward dopamine cells, a negative change +pulses the 2 **PPL101** aversive dopamine cells, and the candidate memory rule +adjusts the 7,835 existing **KC → MBON07 / MBON11** synapses. + +Everything that touches money stays deterministic and outside the fly, in the +stonkfly spirit: the connectome **proposes** a quoting posture, a fixed mapping +turns the posture into a `pmm_mister` config, a **guard can veto but never +substitute**, and Hummingbot's controller executes. The Condor LLM agent is the +*operator* — it picks the market, deploys the bot, starts and stops the fly, +reports, and explains — it never overrides the fly's posture with its own view. + +The trading universe is **Hyperliquid HIP-3 perps (xyz issuer)** on +`hyperliquid_perpetual`, reusing Market Making Expert's HIP-3 scanner and its +HIP-3 operating rules. + +FLM (the "Fly Language Model") was the first candidate and is **not** used: +it is a frozen 1.2 B-parameter chat model whose next-token scores get a small +correction from the connectome, and its own paper reports the fly readout does +not beat a matched non-fly control. Stonkfly's direct spiking-simulation route +puts the actual connectome in the decision path, which is what you asked for. + +## 2. What is reused from stonkfly, what is replaced + +| Stonkfly piece | MM Fly | Notes | +|---|---|---| +| `neural/` package: graph import, C++ LIF kernel, `MemoryBrain`, `VisualMemoryBrain`, `circuit` (PAM11/PPL101/KC/MBON), `rule` (anti-Hebbian memory), checksum locks | **Vendored unchanged** into `condor/fly/neural/` | MIT. Only the data-directory constant changes. Keeping it byte-identical keeps its lock/provenance chain valid. | +| `data.py` prepare / verify (1.1 GB MaleCNS download, checksum, compile `graph.npz`) | Vendored, exposed as `condor fly prepare` / `condor fly verify` | Data lives under Condor's local, git-ignored root. | +| `display.market_frame` (320×180 line chart) | **Replaced** by `condor/fly/chart.py`: OHLCV candlesticks + volume | Same canvas size and light palette so the retinal projection is unchanged. | +| `Decoder` (DNp20 R−L → BUY/SELL/HOLD) | **Replaced** by `condor/fly/decoder.py`: DNp20 R−L → trend/skew, a population-rate channel → arousal/spread, gate on DNpe017 | Still a fixed, engineered readout of spike counts. | +| `reinforcement.py` (equity delta vs anchor, deadband) | Same rule, equity = controller `realized + unrealized − fees` | Filtered by the bot/controller, so other bots on the account do not leak in. | +| `risk.Guard` / `Veto` | `condor/fly/guard.py`: fee floor, loss stop, loss-rate breaker, apply cooldown, market-open, collateral, price-move tolerance | Veto keeps the previous config. It never invents a different posture. | +| `broker.py`, `actions.py`, `ledger.py` (Coinbase FOK orders, SQLite intents) | **Dropped** | Execution is Hummingbot `pmm_mister`; the fly never places orders itself. | +| `cli.py` run loop, 2-slot checkpoints, `events.jsonl`, `latest.json`, `latest-input.png`, provenance hash | Re-implemented as the `fly_brain` continuous routine + `condor/fly/run_state.py` | Same durability pattern. | + +## 3. Pipeline + +``` + every interval_sec (default 60 s) + ┌────────────────────────────────────────────────────────────────────────────┐ + │ 1. fetch : last N candles (HIP-3 pair) + live l2Book bid/ask │ + │ 2. render : chart.py → 320×180 RGB uint8 (OHLCV + volume + bid/ask) │ + │ 3. reward : Δ(controller net P&L) vs last observation → reward|aversive|none│ + │ 4. observe: worker process — VisualMemoryBrain.rgb_step for neural_ms │ + │ (default 500 ms neural time), dopamine pulse if reward/aversive │ + │ 5. decode : spike counts → {trend_z, arousal_z, gate} → regime, spread_mult,│ + │ shift_bps (posture) │ + │ 6. map : posture → pmm_mister config (HIP-3 base config × posture) │ + │ 7. guard : vetoes (fee floor, loss stop, cooldown, market closed, …) │ + │ 8. apply : if live mode and posture changed materially → update_config │ + │ 9. persist: checkpoint (2 slots), events.jsonl, latest.json, PNG, LiveReport│ + └────────────────────────────────────────────────────────────────────────────┘ + ▲ │ + Condor LLM agent (operator) ─ deploy / stop / report / explain ┘ +``` + +Ordering follows stonkfly: reinforcement for the *previous* observation is +delivered at the start of the *next* neural window; the checkpoint and +accounting anchor are committed **before** anything is applied to the bot. + +## 4. What the fly sees — chart specification + +`condor/fly/chart.py: market_frame(pair, candles, bid, ask) -> np.uint8[180,320,3]` + +* Canvas 320×180, light background `(235,240,249)`, dark header bar with the + pair name — identical to stonkfly. Stonkfly found the dark chart produced no + Kenyon-cell spikes and switched to light; we keep that. +* **Candles**: last `n_candles = 72` of `interval = 5m` (6 h window) drawn 4 px + wide in the 294 px plot span. Up candle body **blue `(0,101,183)`**, down + body **red `(197,37,78)`**, 1 px wicks in the body color. Blue/red is + stonkfly's palette: the mapped R8p cells read the blue channel and R8y the + green channel, R1–R6 read luminance, so colour is not decoration here — it is + the only chromatic input the fly gets. +* **Volume**: bars in a bottom strip (rows 140–158), grey-blue, scaled to the + window maximum. +* **Bid / ask**: two short horizontal ticks at the right edge at the bid and ask + price, plus stonkfly's `BID x ASK y` footer text. +* Price axis: window `[min low, max high]` padded 12 % top and bottom, with a + floor of 0.2 % of mean price so a flat market does not blow up to full scale + (stonkfly's rule). +* **Not drawn**: the bot's own quotes, inventory, P&L, funding, or anything + from the account. Stonkfly deliberately excludes portfolio state from the + picture; P&L reaches the fly only through dopamine. + +The frame is saved as `latest-input.png` every observation, so you can see what +the fly saw when it made a call. `fly_chart` is also a one-shot routine, so you +can render the chart for any pair from chat and inspect it. + +Candles come from Hummingbot (`client.market_data.get_candles_last_days` on +`hyperliquid_perpetual`), which already serves HIP-3 pairs. The live book comes +from Hyperliquid `l2Book` directly with the `xyz:TOKEN` coin form (the +hummingbot-api `order_book` endpoint 500s on HIP-3 — documented in the HIP-3 +operator playbook). + +## 5. The neural substrate + +* `condor/fly/neural/` = stonkfly's `neural/` package plus `data.py`, vendored + with a `THIRD_PARTY.md` notice (MIT). Files: `brain.py`, `state.py`, + `circuit.py`, `rule.py`, `visual.py`, `sensory.py`, `transmitters.py`, + `connectome.py`, `prepare.py`, `kernel.cpp`, and the four `*.lock.json` + checksum files. The only edit is `common.DATA`, which resolves to Condor's + data dir instead of `./data`. +* Integration parameters stay stonkfly's: 0.1 ms timestep, 500 ms neural time + per observation in 10 ms bins, 200 ms / 20 mV dopamine pulse, KC rest −60 mV + with adaptation, R8 → aMe12 sign correction, memory rule gain 0.001, 1 s + eligibility traces, 1,800 s memory decay, efficacy bounds 0.1–2×. +* **Data**: `condor fly prepare` downloads ~1.1 GB (annotations + edges + feather), verifies SHA-256 against the locks, compiles `graph.npz` + (166,700 × 25,582,938) and builds the C++ kernel with `c++ -O3`. Location: + `/fly/data/` (git-ignored, beside `.condor/agents/`), overridable + with `CONDOR_FLY_DATA`. `condor fly verify` re-checks; `condor doctor` gets a + `fly-data` row. +* **Process model**: the simulation is CPU-bound C++ called through ctypes and + must not run on Condor's event loop. The `fly_brain` routine owns a + `ProcessPoolExecutor(max_workers=1, spawn)` whose initializer loads + `VisualMemoryBrain` once; each tick submits `observe(frame, reinforcement)` + and awaits the future. Frames are 172 KB, results are small dicts. + Checkpoint/restore run in the same worker. One worker per running fly + instance; a per-run lock file prevents two instances on one run directory + (stonkfly's `worker.lock`). +* **Compute budget**: stonkfly does not publish a per-observation timing. + `condor fly bench` will run three observations on a synthetic frame and print + `compute_seconds`; if it exceeds ~40 % of `interval_sec`, lower `neural_ms` + (it is a config field). Memory: the graph is ~300 MB resident, checkpoints + ~100 MB each × 2 slots. + +## 6. Decoder — spikes → quoting posture + +`condor/fly/decoder.py`. Engineered and fixed, like stonkfly's — it reads only +spike counts, and the cell IDs it reads are written to the audit log. + +Raw channels per observation (`seconds = neural_ms / 1000`): + +| Channel | Cells | Measurement | +|---|---|---| +| `trend_hz` | DNp20 left vs right (stonkfly's BUY/SELL cells) | mean right rate − mean left rate | +| `arousal_hz` | descending-neuron population (`superclass == "descending"` in `graph.npz`, minus the DNp20/DNpe017 readouts) | mean firing rate | +| `gate` | DNpe017 | spike count ≥ 1 | + +Baseline-centering (new, addresses a problem stonkfly documented): stonkfly's +six-observation run proposed BUY every time — a persistent turning bias of the +circuit became persistent buying. In market making that would become a +persistent skew. So the decoder keeps a running mean/std of `trend_hz` and +`arousal_hz` over the last `baseline_window = 60` observations (persisted in +`state.json`) and works in z-scores. The first `warmup = 10` observations emit +the neutral posture while the baseline forms. `center_bias` is a config flag +(default on) so a raw, uncentred run is still possible as a control. + +Regime, in precedence order: + +| Regime | Rule | +|---|---| +| `pause` | `arousal_z ≥ 2.5` | +| `volatile` | `arousal_z ≥ 1.0` | +| `trending_up` | `gate` and `trend_z ≥ 1.0` | +| `trending_down` | `gate` and `trend_z ≤ −1.0` | +| `quiet` | `arousal_z ≤ −1.0` | +| `ranging` | otherwise, and during warm-up | + +Continuous outputs: + +* `spread_mult = clip(1 + 0.5 · arousal_z, 0.6, 2.5)` — the fly agitated by a + jumpy chart widens, a calm fly tightens. +* `shift_bps = clip(1.0 · trend_z, −max_shift_bps, +max_shift_bps)` with + `max_shift_bps = 3` and a hard cap of 50 % of the first-level spread — the + reference price leans with the fly's turning direction. + +Hysteresis so the bot is not re-configured on noise: a new posture is only +applied if the regime changed, or `|Δspread_mult| ≥ 0.15`, or +`|Δshift_bps| ≥ 0.5`, and at least `min_apply_interval_sec = 300` have passed +since the last apply. Otherwise the tick is `HOLD`. The fly is still observed and +reinforced every tick; only the application is rate-limited (stonkfly's order +cooldown, transposed). + +Thresholds (1.0 / 2.5 z, 0.5 gain, 3 bp) are declared model choices with no +fitted basis. They are config fields. + +## 7. Posture → `pmm_mister` config + +`condor/fly/posture.py: build_config(base, posture, fees) -> dict`. The base +config is the HIP-3 operator's bounded defaults, the posture multiplies and +shifts it: + +* Base spreads from the scanner's picked spread `S` bp: level 1 + `max(2, S/2)`, level 2 `S+1` (HIP-3 playbook). Then + `buy = base · spread_mult − shift`, `sell = base · spread_mult + shift`, + each floored at `min_spread_bps = 3`. +* `take_profit = max(0.0004, level-1 buy spread)` — and never below + `2.2 × round-trip maker fee` (HIP-3 all-in ~1.3 bp/side → 2.6 bp; floor 5.7 bp + wins). Market Making Expert's rule "TP must exceed round-trip fees" becomes a + hard floor in code rather than advice. +* Timing per regime, from Market Making Expert's table: quiet + `refresh 20 / cooldown 30`, ranging `30 / 60`, trending `30 / 60`, volatile + `60 / 120`. +* Fixed by the HIP-3 playbook, not touched by the fly: `target_base_pct 0.4, + min 0.3, max 0.5`, `portfolio_allocation 0.2`, `max_active_executors_by_level + 2`, `leverage ≤ 5` and ≤ market max, `open/take_profit_order_type 3` + (LIMIT_MAKER), `global_sl_enabled true, global_stop_loss 0.02`, uppercase + `XYZ:TOKEN-USD` pair. +* `pause` regime → `manual_kill_switch = true` (quotes pulled, position kept); + leaving `pause` → `false`. +* Optional (off by default): nudge `target_base_pct` by ±0.05 in a trending + regime. Off because the HIP-3 rules say tight bands are what stopped the SPCX + and DRAM losses. + +Applied through `manage_bots(action="update_config", …, confirm_override=true)` +on the live bot **and** `manage_controllers(action="upsert")` on the saved +config, both layers, per the deploy playbook. + +## 8. Guard — vetoes only + +`condor/fly/guard.py`. Every rule is deterministic; a veto means "keep the +previous config", a halt means "stop the bot and stop the fly until a human +passes `resume_reviewed=true`". None of them chooses a different posture. + +| Check | Effect | +|---|---| +| Market closed (live book missing a side) — HIP-3 equities close off-hours | veto (HOLD); after `closed_ticks_to_stop` consecutive closed ticks, STOP bot, leave fly observing | +| Available USD on the unified Hyperliquid account < required margin | veto | +| Any spread `< min_spread_bps`, TP below fee floor, leverage above cap | veto (should be unreachable after `posture.py` floors; this is the belt to those braces) | +| Apply cooldown (`min_apply_interval_sec`) or daily apply cap (`max_applies_per_day = 48`) | veto | +| Mid moved more than `apply_price_tolerance = 0.5 %` between observation and apply | veto (stonkfly's fresh-book check) | +| `total_net ≤ −max_loss_quote` (default 4 % of `total_amount_quote`) | **halt**: STOP bot, position closed by the controller's `global_stop_loss` / market-close, fly halted | +| HIP-3 loss-rate breaker: `total_net / volume ≤ −5 bp` or no new session high for 25 ticks | **halt** with market-close, exactly the HIP-3 playbook's mandatory rule | +| `update_config` failed 3 consecutive ticks | **halt** for review | +| STOP file in the run directory | halt (stonkfly's `touch runs/live/STOP`) | + +`total_net` always includes unrealized P&L — the HIP-3 playbook's rule. + +## 9. Reinforcement — P&L → dopamine + +`condor/fly/reinforcement.py`, stonkfly's function with a different equity +source: + +* `equity_t = realized_pnl + unrealized_pnl − fees` for **this bot's** + `pmm_mister` controller, read from `manage_bots(action="status")` + performance (the same fields `mm_dashboard` reads), so nothing else on the + account can reward or punish the fly. +* `delta = equity_t − anchor` where `anchor` is the previous observation's + equity. `delta ≥ +deadband` → `reward` (PAM11 ×15), `≤ −deadband` → + `aversive` (PPL101 ×2), else `none`. Deadband default = 1 bp of + `total_amount_quote` (stonkfly used 0.01 on $100, the same 1 bp). +* Binary pulse, 200 ms, 20 mV-equivalent, delivered during the first 200 ms of + the next 500 ms window; not proportional to the amount; tiny changes are not + accumulated. All as stonkfly. +* The memory rule and plastic edge set are stonkfly's, untouched: 7,835 + KC→MBON07/MBON11 edges, baseline-centred anti-Hebbian rule fed by actual + KC and DAN spike rates. +* `learning = true` by default; `frozen = true` freezes efficacies for a + control run. In **shadow** mode with no bot deployed, reinforcement is `none` + every tick (no P&L exists); with a bot deployed under the fly's name it is + the bot's P&L even though the fly's postures are not being applied — stated + plainly in the status report because that signal has nothing to do with the + fly's own proposals. + +What this is and is not: it is feedback about the controller's marked value +between two observations, including fees; it is not evidence that the last +posture caused it. Stonkfly's validation found plastic edges changing *before* +the first external pulse — endogenous dopamine activity drives the rule too. We +report `changed_edges`, `mean_efficacy`, `reward_spikes`, `aversive_spikes` +every tick and make no claim beyond them. + +## 10. Trading universe — HIP-3 + +* Connector `hyperliquid_perpetual`, issuer `xyz`, pairs `XYZ:TOKEN-USD` + (uppercase; lowercase → KeyError → zero orders). +* Market selection is the existing `hip3_market_scanner` routine (copied into + the agent): volume, spread-vs-fee, daily drift, `l2Book` depth filter, + `TOP PICK`. Selection is the operator's job (deterministic routine + LLM + reading it), not the fly's — the fly never chooses which market it is shown, + as in stonkfly's fixed round-robin. +* Venue facts carried over from the HIP-3 operator: unified collateral (read + from `get_portfolio_overview(["hyperliquid_perpetual"])`, not the per-dex + clearinghouse state), isolated margin, trading hours / empty books, ~1.3 + bp/side all-in maker fee including the fixed Hummingbot builder fee, tight + inventory bands, `global_stop_loss 0.02`. +* **Three markets, one brain (decided 2026-09-12).** The fly quotes three + HIP-3 markets in parallel and learning is shared: a single + `VisualMemoryBrain` is shown the three charts in a fixed round-robin + (stonkfly's multi-asset schedule — the network never chooses which market it + sees), one pair per tick, so each pair is observed every `3 × interval_sec`. + Reinforcement is the change in the **combined** net P&L of the three + controllers since the previous observation, delivered while whichever chart + is up — feedback about the fly's whole book, as stonkfly's is about the whole + portfolio. Baselines and postures are **per pair** (keyed by pair in + `state.json`); the KC→MBON weights are shared and persist in one checkpoint + lineage. Rotating a market swaps the pair in the list and resets that pair's + baseline only; the brain and its learned efficacies carry on. +* Rotation: only when flat and the current market is closed, trending against + us, or dominated — re-run the scanner and redeploy that slot. + +## 11. Condor integration — files + +``` +condor/fly/ + __init__.py + neural/ # vendored stonkfly.neural + THIRD_PARTY.md (MIT) + data.py # vendored prepare/verify, Condor data dir + chart.py # OHLCV → 320×180 RGB + decoder.py # counts → channels, z-scores, regime, posture + posture.py # posture → pmm_mister config with floors + guard.py # vetoes / halts + reinforcement.py # equity delta → reward|aversive|none + worker.py # ProcessPoolExecutor target: load brain, observe, checkpoint, restore + run_state.py # run dir: state.json, events.jsonl, latest.json, PNG, 2-slot checkpoints, provenance, lock, STOP + cli.py # condor fly prepare | verify | bench + +agents/market_making_fly/ + AGENT.md # operator brain (LLM) + routines/ + fly_chart.py # one-shot: render + report the frame for a pair + fly_brain.py # CONTINUOUS: the loop in §3; modes shadow|live, learning|frozen, fixture|market + fly_status.py # one-shot: latest posture, channels, z-scores, spikes, memory stats, P&L, vetoes, halt reason + hip3_market_scanner.py # copied from Market Making Expert + mm_dashboard.py # copied + skills/ + fly_mm_deploy/SKILL.md # deploy playbook (adapted from pmm_mister_deploy: scanner → base config → deploy → start fly_brain) + fly_decoder/SKILL.md # how to read fly_status and the decoder + pmm_config_playbook/ # copied + capital_allocation/ # copied + mm_bot_report/ # copied + strategies/fly_hip3_operator/strategy.md # thin loop: keep bot + fly alive, surface halts, rotate when flat + +tests/ + test_fly_chart.py test_fly_decoder.py test_fly_posture.py + test_fly_guard.py test_fly_reinforcement.py + test_fly_full_graph.py # opt-in, CONDOR_FLY_FULL_TEST=1, needs prepared data + +docs/market_making_fly_design.md # this file, kept as the reference +``` + +Run state per fly instance: +`.condor/agents/market_making_fly/fly//` with `state.json` +(baseline stats, anchor, tick, last applied posture/config, halt reason), +`events.jsonl` (one row per tick: quote, frame hash, spike summary, channels, +posture, config diff, guard result), `latest.json`, `latest-input.png`, +`brain-0.npz` / `brain-1.npz`, `provenance.json` (settings, dataset hashes, +circuit report, decoder description, source hashes — a changed protocol refuses +to resume into an old run dir, as stonkfly does), `worker.lock`, optional `STOP`. + +### The agent (LLM) — what it does and does not do + +`AGENT.md` keeps Market Making Expert's domain knowledge (regimes, spread +calibration, inventory, fee rule, `pmm_mister` parameter guide) so it can +*explain* what the fly did in market-making terms, and adds one rule above all +others: + +> While a fly run is live on a pair, the fly quotes and you operate. You deploy, +> start, stop, rotate, and report. You never set spreads, skew, or regime from +> your own analysis, and you never "correct" the fly's posture. If you believe +> the fly is wrong, you stop it and say why; you do not out-vote it. + +Two modes, like Market Making Expert: + +* **Consulted**: "what does the fly see on DRAM right now", "why did it widen", + "is it learning anything" → runs `fly_status` (and `fly_chart`), answers in + key: value lines, quotes the numbers, repeats the caveats in §15. +* **Delegated / loop**: `fly_mm_deploy` skill end-to-end: scanner → TOP PICK → + base config from `pmm_config_playbook` balanced profile adapted with HIP-3 + bounds → deploy with `max_global_drawdown_quote` → start `fly_brain` + (shadow first unless told live) → verify with `mm_bot_report`. The loop + strategy ticks every 5 min: confirms bot and fly instance alive, surfaces + halts/vetoes, rotates when flat and the market is closed. + +Tools list = Market Making Expert's plus `manage_routines` for the fly +routines. `server_name` left empty, `agent_key` copied from Market Making +Expert. + +### Modes + +| Mode | Observes | Applies config | Reinforcement | Use | +|---|---|---|---|---| +| `shadow` (default) | real chart | no — logs what it *would* apply | bot P&L if a bot is deployed, else none | first runs, watch the posture stream against `market_analyzer` | +| `live` | real chart | yes | bot P&L | after shadow looks sane | +| `frozen` flag | any | any | pulses delivered, weights frozen | control run | +| `fixture` flag | synthetic sine candles | never | none | offline plumbing test, `fast=true` skips wall waits | + +## 12. Deploy lifecycle + +1. `condor fly prepare` once per machine (1.1 GB, several minutes, needs `c++`). +2. Operator agent runs the scanner, deploys `pmm_mister` on the TOP PICK with + the neutral (ranging) posture config. +3. `manage_routines(action="start", name="fly_brain", agent="market_making_fly", + config={"trading_pair": "XYZ:DRAM-USD", "bot_name": "dram-mm", + "config_name": "dram_mm_live", "mode": "shadow", "run_name": "dram-2026-09-12"})`. +4. Watch `fly_status` / the LiveReport for an hour; compare the fly's regime + stream with `market_analyzer` for the same window (report only). +5. Restart with `mode: "live"` on the same run directory (baseline and memory + carry over). +6. Halts require `resume_reviewed: true` on restart, and a financial halt cannot + be cleared that way at all (stonkfly's rule) — a new run directory is needed. + +## 13. Observability + +* `fly_brain` maintains one `LiveReport`: latest chart PNG, regime timeline + (last 100 ticks), `trend_z` / `arousal_z` sparklines, spike KPIs, memory + KPIs (`changed_edges`, `mean_efficacy`), P&L and reinforcement history, + applied-config history, vetoes/halts. +* `fly_status` gives the same as text for consults and the Telegram loop. +* The agent journals every apply and halt with the posture and the guard + verdict. `events.jsonl` is the audit trail; each row carries the frame hash, + spike-count hash, and checkpoint hash so a decision can be tied to exactly + what the fly saw and what state it was in. + +## 14. Testing and validation plan + +Unit (no data, run in CI): + +* `chart.py`: deterministic frame for fixed candles (hash), shape/dtype, up + candle pixels are blue and down red, bid/ask ticks land at the right rows, + flat-market scale floor. +* `decoder.py`: synthetic counts → expected regime for each row of the table; + warm-up emits ranging; centring removes a constant bias; hysteresis blocks + small changes and passes regime changes. +* `posture.py`: fee floor beats spread; shift never pushes a side below + `min_spread_bps`; pause sets the kill switch; timing table per regime; pair + is uppercase. +* `guard.py`: every row of the table in §8, halt vs veto, `resume_reviewed` + semantics, financial halt not clearable. +* `reinforcement.py`: stonkfly's four cases. +* `run_state.py`: checkpoint slot alternation, provenance mismatch refuses + resume, lock file. + +Opt-in with prepared data (`CONDOR_FLY_FULL_TEST=1`): stonkfly's full-graph +test transposed — a white frame activates KCs, reward and aversive pulses spike +their DAN cells, eligible edges change, frozen stays identical, checkpoint +round-trips; plus a rendered real chart produces non-zero KC activity (this is +the check stonkfly says is the biggest open modelling question). + +Runs, in order: `fixture + fast + steps=6` offline; `shadow` on a live HIP-3 +pair for at least 60 observations (one baseline window); `live` at +`total_amount_quote = 100` USD; only then normal size. + +Not planned in this pass, and required before any learning claim: held-out +chronological replay, shuffled-reinforcement control, exposure/cash baselines, +retention after reset. The design leaves room for them (`frozen`, fixture +market, run directories) but does not deliver them. + +## 15. What this does and does not claim + +Copied in spirit from stonkfly's `docs/model.md`, because the same limits hold: + +* The connectome supplies anatomy; the LIF cells, transmitter sign proxies, + RGB-to-photoreceptor mapping, compressed clock (500 ms neural per 60 s wall) + and dopamine assignments are engineered choices, not fly physiology. +* The decoder is an interface we chose. "Trend from DNp20, arousal from + descending neurons" is not a discovery of market-making neurons. A persistent + circuit bias becomes a persistent skew; baseline-centring reduces but does + not remove that. +* Candle colours were chosen to hit the R8 channels; that is a display adapter. +* P&L pulses are feedback about the controller's value, not credit assignment. + Weight changes do not show the fly learned to quote. +* No profitable learning is demonstrated by anything in this design. The guard + and the controller's stop-loss are what bound the loss, not the fly. + +## 16. Decisions + +Settled 2026-09-12: 1 vendor, 2 deterministic apply, 3 descending neurons, +4 baseline-centred, 5 72 × 5 m, 6 net P&L incl. unrealized, 7 **three markets +in parallel on one shared brain** (see §10), 8 shadow default, 9 +`.condor/fly/data`, 10 no target-base nudge. The original options are kept +below for the record. + +1. **Vendor stonkfly's neural package into `condor/fly/neural/` (recommended)** + vs `pip install git+…stonkfly`. Installing pulls `coinbase-agentkit` and + `coinbase-advanced-py` into Condor for no use; vendoring adds ~1,400 lines + + the kernel + `pyarrow` as a new dependency. +2. **Deterministic apply (recommended)** — the `fly_brain` routine applies the + config itself and the LLM only operates — vs the LLM loop reading the + posture and applying it each tick. The second puts an LLM back between the + fly and the bot, which is what stonkfly's "no LLM trading policy" rule + avoids, and costs tokens every minute. +3. **Arousal population = descending neurons (recommended)** vs whole-network + mean rate vs Kenyon cells. All three are engineered; descending neurons are + the motor-output side stonkfly already reads from and are labelled in + `graph.npz` without the annotations file. +4. **Baseline-centre the channels (recommended, flag)** vs raw rates. Raw is + stonkfly's choice and is the honest one for an experiment; centred is the + one that will not sit skewed one way for hours because of circuit bias. +5. **Chart window 72 × 5 m (recommended)** vs 100 × 1 m (stonkfly's density) + vs 1 h candles. 5 m matches the regime cadence the config timing table + assumes; 1 m is what the fly could realistically react to at 60 s ticks. +6. **Reinforcement source = controller net P&L incl. unrealized, 1 bp deadband + (recommended)** vs realized-only. The HIP-3 rules insist on unrealized. +7. **Fresh run directory per market (recommended)** vs carrying learned + weights across markets. +8. **Default mode shadow (recommended)**; live is explicit. +9. **Data dir `/fly/data` (recommended)** vs `~/.condor/fly`. +10. Whether the target-base nudge in trending regimes (§7, off by default) + should exist at all. + +## 17. Implementation phases + +| Phase | Deliverable | Verifies | +|---|---|---| +| 1 | `condor/fly/neural` vendored, `data.py`, `condor fly prepare/verify/bench`, `pyarrow` dep, doctor row | `prepare` completes on this Mac, `verify` passes, `bench` prints compute time | +| 2 | `chart.py` + tests, `fly_chart` routine | rendered DRAM chart in a report | +| 3 | `decoder.py`, `posture.py`, `guard.py`, `reinforcement.py` + tests | unit suite green | +| 4 | `worker.py`, `run_state.py`, `fly_brain` routine, fixture mode | `fixture + fast + steps=6` run end-to-end with checkpoint resume | +| 5 | `AGENT.md`, skills, strategy, `fly_status`, copies of scanner/dashboard | consult works; shadow run on a live HIP-3 pair | +| 6 | live mode at small size | your call, after reviewing shadow output | + +Rough effort: phases 1–4 are the bulk; 5 is mostly adaptation of Market Making +Expert text; 6 is operation, not code. + +## 18. Implementation status (2026-09-12) + +Phases 1–5 are implemented; phase 6 (live at small size) is the operator's call. + +| Piece | Where | State | +|---|---|---| +| Vendored neural package, data prepare/verify, kernel | `condor/fly/neural/`, `condor/fly/data.py`, `python -m condor.fly prepare\|verify\|bench` | done; dataset prepared at `.condor/fly/data` (1.6 GB), verified, kernel built | +| Chart, decoder, posture, guard, reinforcement, naming, market, run state, worker | `condor/fly/*.py` | done, 77 unit tests green (`uv run pytest tests/test_fly_*.py`) | +| Loop routine | `agents/market_making_fly/routines/fly_brain.py` | done; 14-observation fixture run end to end (checkpoints, events, live report, halt path) | +| `fly_chart`, `fly_status`, copied scanner/dashboard | `agents/market_making_fly/routines/` | done, load through routine discovery | +| Agent brain, deploy playbook, decoder skill, operator strategy | `agents/market_making_fly/{AGENT.md,skills,strategies}` | done | +| Opt-in full-graph test | `tests/test_fly_full_graph.py` (`CONDOR_FLY_FULL_TEST=1`) | done | +| Doctor row | — | not done; use `python -m condor.fly verify` | + +Deviations from the text above: the descending-neuron superclass label in +`graph.npz` is `descending_neuron` (1,314 cells); the CLI is +`python -m condor.fly …` rather than a `condor fly` subcommand (there is no +`condor` console script in this repo); checkpoints are ~7 MB compressed, not +100 MB. + +### First measurements + +* Bench: brain loads in ~1 s; one 500 ms neural window costs 2–5 s of compute on + this Mac. A 60 s wall interval leaves ample room. +* DNp20 fires 26–46 Hz on a chart with a consistent right-minus-left surplus of + +2 to +12 Hz in every observation — stonkfly's persistent bias, now measured. + Baseline-centring is what keeps it from becoming a permanent upward lean. +* A rendered chart activates 10–17 Kenyon cells per window at first (stonkfly's + chart gave 11–16). An aversive pulse produced 13–30 PPL101 spikes, a reward + pulse ~270 PAM11 spikes, and 5 KC→MBON edges changed from endogenous activity + before any pulse, as stonkfly also reported. +* In the fixture run the network flipped, at the ninth observation, into a + persistent high-activity state (total spikes 388 k → 609 k per window, KC + spikes 12 → 4,000+, descending rate 2.5 → 8 Hz, 2,900 plastic edges + changed). A control run with reinforcement disabled reproduced the flip at + the same tick with the same numbers, so it is driven by the visual input + sequence, not by the dopamine pulses. It is what the arousal channel is meant + to read, but it means "volatile" can persist until the rolling baseline + absorbs the new level (up to one window). Stonkfly's warning that display + sensitivity is the largest open modelling question applies here in full. + +## 19. Housekeeping from this session + +* FLM was cloned, installed, and its downloads started at `~/flm` (4.3 GB) + before the change of direction. The install was stopped. Delete with + `rm -rf ~/flm ~/flm-setup.sh ~/flm-setup.log` when you like; nothing in this + design uses it. +* The scaffold copy `agents/market_making_fly/` from the first attempt was + removed; the tree is clean apart from this document. diff --git a/pyproject.toml b/pyproject.toml index aec2c149b..f478e26bf 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,6 +19,8 @@ dependencies = [ # It pins numba==0.61.2, which caps numpy at <2.3 — that is also the numpy # the API container runs (2.2.6), so the two environments stay aligned. "pandas-ta>=0.4.71b", + # Fly connectome (condor/fly): MaleCNS feather files + "pyarrow", "geckoterminal-py", "mcp", "fastapi", diff --git a/tests/test_fly_chart.py b/tests/test_fly_chart.py new file mode 100644 index 000000000..b9ee06b49 --- /dev/null +++ b/tests/test_fly_chart.py @@ -0,0 +1,119 @@ +"""The frame the fly sees: fixed size, right palette, no silent bad data.""" + +import hashlib + +import numpy as np +import pytest + +from condor.fly import chart + + +def _candles(n=72, start=100.0, step=0.5, up=True): + rows = [] + price = start + for i in range(n): + nxt = price + step if up else price - step + o, c = price, nxt + rows.append( + { + "timestamp": i, + "open": o, + "high": max(o, c) + 0.2, + "low": min(o, c) - 0.2, + "close": c, + "volume": 10 + i, + } + ) + price = nxt + return rows + + +def test_frame_shape_dtype_and_background(): + frame = chart.market_frame("XYZ:DRAM-USD", _candles(), 135.0, 135.2) + assert frame.shape == (chart.HEIGHT, chart.WIDTH, 3) + assert frame.dtype == np.uint8 + assert tuple(frame[90, 2]) == chart.BACKGROUND # left margin + assert tuple(frame[5, 200]) == chart.HEADER + + +def test_up_candles_are_blue_and_down_candles_red(): + up = chart.market_frame("P:A-USD", _candles(up=True), 135.0, 135.2) + down = chart.market_frame("P:A-USD", _candles(up=False), 64.0, 64.2) + plot_up = up[chart.PLOT_TOP : chart.PLOT_BOTTOM, chart.PLOT_LEFT : chart.PLOT_RIGHT] + plot_down = down[ + chart.PLOT_TOP : chart.PLOT_BOTTOM, chart.PLOT_LEFT : chart.PLOT_RIGHT + ] + assert (plot_up == chart.UP).all(axis=2).any() + assert not (plot_up == chart.DOWN).all(axis=2).any() + assert (plot_down == chart.DOWN).all(axis=2).any() + assert not (plot_down == chart.UP).all(axis=2).any() + + +def test_bid_ask_ticks_at_right_edge(): + rows = _candles() + frame = chart.market_frame("P:A-USD", rows, 120.0, 130.0) + lo, span = chart.price_scale(chart.normalize_candles(rows)) + for price in (120.0, 130.0): + y = int(chart._y(price, lo, span)) + strip = frame[y - 1 : y + 2, chart.PLOT_RIGHT + 3 : chart.WIDTH - 2] + assert (strip == chart.TICK).all(axis=2).any() + + +def test_volume_strip_is_drawn(): + frame = chart.market_frame("P:A-USD", _candles(), 135.0, 135.2) + strip = frame[ + chart.VOLUME_TOP : chart.VOLUME_BOTTOM + 1, chart.PLOT_LEFT : chart.PLOT_RIGHT + ] + assert (strip == chart.VOLUME).all(axis=2).any() + + +def test_deterministic(): + a = chart.market_frame("P:A-USD", _candles(), 135.0, 135.2) + b = chart.market_frame("P:A-USD", _candles(), 135.0, 135.2) + assert ( + hashlib.sha256(a.tobytes()).hexdigest() + == hashlib.sha256(b.tobytes()).hexdigest() + ) + + +def test_flat_market_scale_floor(): + rows = [ + {"open": 100, "high": 100, "low": 100, "close": 100, "volume": 1} + for _ in range(10) + ] + lo, span = chart.price_scale(chart.normalize_candles(rows)) + assert span == pytest.approx(100 * 0.002 * 1.24) + assert lo == pytest.approx(100 - 100 * 0.002 * 0.12) + + +def test_keeps_only_last_n_and_right_aligns(): + rows = _candles(n=200) + kept = chart.normalize_candles(rows, 72) + assert len(kept) == 72 and kept[-1]["close"] == rows[-1]["close"] + few = chart.market_frame("P:A-USD", _candles(n=5), 102.0, 102.2) + plot = few[chart.PLOT_TOP : chart.PLOT_BOTTOM] + colored = (plot == chart.UP).all(axis=2).any(axis=0) + assert colored[: chart.PLOT_LEFT + 200].sum() == 0 # nothing on the left + assert colored[chart.PLOT_RIGHT - 30 : chart.PLOT_RIGHT].any() + + +@pytest.mark.parametrize( + "bad", + [ + [], + [{"open": 1, "high": 2, "low": 0.5}], # missing close/volume + [{"open": 1, "high": 0.9, "low": 0.5, "close": 1, "volume": 1}], # high < open + [{"open": 0, "high": 1, "low": 0, "close": 1, "volume": 1}], # zero price + [{"open": float("nan"), "high": 1, "low": 0, "close": 1, "volume": 1}], + ], +) +def test_bad_candles_raise(bad): + with pytest.raises(ValueError): + chart.market_frame("P:A-USD", bad, 1.0, 1.1) + + +def test_bad_quotes_raise(): + with pytest.raises(ValueError): + chart.market_frame("P:A-USD", _candles(), 0, 1) + with pytest.raises(ValueError): + chart.market_frame("P:A-USD", _candles(), 2.0, 1.0) diff --git a/tests/test_fly_decoder.py b/tests/test_fly_decoder.py new file mode 100644 index 000000000..371e25c99 --- /dev/null +++ b/tests/test_fly_decoder.py @@ -0,0 +1,135 @@ +"""Spike channels → posture: the regime table, warm-up, centring, hysteresis.""" + +import pytest + +from condor.fly.decoder import ( + NEUTRAL, + Baseline, + Channels, + DecoderSettings, + Hysteresis, + Posture, + classify, + decode, + should_apply, +) + +S = DecoderSettings(window=20, warmup=5) + + +def _warm(baseline, trend=0.0, arousal=10.0, n=None, jitter=True): + """Feed a calm history with a little spread so std is non-zero.""" + n = n or S.warmup + 5 + for i in range(n): + t = trend + (0.1 if jitter and i % 2 else -0.1) + a = arousal + (0.5 if jitter and i % 2 else -0.5) + decode(Channels(t, a, 0), baseline, S) + + +@pytest.mark.parametrize( + "trend_z,arousal_z,gate,expected", + [ + (0, 3.0, True, "pause"), + (5, 3.0, True, "pause"), # pause beats trending + (0, 1.5, False, "volatile"), + (2, 1.5, True, "volatile"), # volatile beats trending + (2, 0, True, "trending_up"), + (-2, 0, True, "trending_down"), + (2, 0, False, "ranging"), # no gate, no trend call + (0, -1.5, False, "quiet"), + (2, -1.5, True, "trending_up"), # trending beats quiet + (0.5, 0.5, True, "ranging"), + ], +) +def test_regime_table(trend_z, arousal_z, gate, expected): + assert classify(trend_z, arousal_z, gate, S) == expected + + +def test_warmup_emits_neutral(): + b = Baseline() + for _ in range(S.warmup - 1): + assert decode(Channels(50.0, 500.0, 5), b, S) == NEUTRAL + assert decode(Channels(50.0, 500.0, 5), b, S).warm is True + + +def test_centring_removes_constant_bias(): + b = Baseline() + _warm(b, trend=8.0) # the circuit "always turns right" + p = decode(Channels(8.0, 10.0, 3), b, S) + assert p.regime == "ranging" and abs(p.trend_z) < 1 and p.shift_bps == 0 + + +def test_raw_mode_keeps_bias(): + raw = DecoderSettings(window=20, warmup=5, center_bias=False) + b = Baseline() + for _ in range(10): + decode(Channels(8.0, 10.0, 3), b, raw) + p = decode(Channels(8.0, 10.0, 3), b, raw) + assert p.trend_z == pytest.approx(4.0) # 8 Hz / 2 Hz unit + + +def test_trend_up_leans_and_caps(): + b = Baseline() + _warm(b) + p = decode(Channels(50.0, 10.0, 2), b, S) + assert p.regime == "trending_up" and p.shift_bps == S.max_shift_bps and p.gate + + +def test_no_gate_no_lean(): + b = Baseline() + _warm(b) + p = decode(Channels(50.0, 10.0, 0), b, S) + assert p.shift_bps == 0 and p.regime == "ranging" + + +def test_arousal_widens_and_pauses(): + b = Baseline() + _warm(b) + wide = decode(Channels(0.0, 11.5, 0), b, S) + assert wide.spread_mult > 1 + b2 = Baseline() + _warm(b2) + pause = decode(Channels(0.0, 100.0, 0), b2, S) + assert pause.regime == "pause" and pause.spread_mult == S.spread_max + + +def test_baseline_window_and_roundtrip(): + b = Baseline() + _warm(b, n=50) + assert b.count == S.window + again = Baseline.from_dict(b.to_dict()) + assert again.trend == b.trend and again.arousal == b.arousal + + +def test_posture_roundtrip(): + p = Posture("quiet", 0.8, -1.0, -0.2, -1.3, True, True) + assert Posture.from_dict(p.to_dict()) == p + + +def test_channels_validation(): + with pytest.raises(ValueError): + Channels(float("nan"), 1.0, 0) + with pytest.raises(ValueError): + Channels(0.0, -1.0, 0) + + +def test_settings_validation(): + with pytest.raises(ValueError): + DecoderSettings(z_regime=3.0, z_pause=2.5) + with pytest.raises(ValueError): + DecoderSettings(warmup=0) + + +def test_hysteresis(): + h = Hysteresis(min_apply_interval_sec=300) + base = Posture("ranging", 1.0, 0.0, 0, 0, False, True) + assert should_apply(None, base, None, 1000, h)[0] + same = Posture("ranging", 1.05, 0.2, 0, 0, False, True) + assert not should_apply(base, same, 0, 1000, h)[0] + regime = Posture("volatile", 1.0, 0.0, 0, 1.2, False, True) + assert should_apply(base, regime, 0, 1000, h)[0] + assert not should_apply(base, regime, 900, 1000, h)[0] # cooldown + wider = Posture("ranging", 1.2, 0.0, 0, 0, False, True) + assert should_apply(base, wider, 0, 1000, h)[0] + lean = Posture("ranging", 1.0, 0.6, 0, 0, True, True) + assert should_apply(base, lean, 0, 1000, h)[0] diff --git a/tests/test_fly_full_graph.py b/tests/test_fly_full_graph.py new file mode 100644 index 000000000..48d47f937 --- /dev/null +++ b/tests/test_fly_full_graph.py @@ -0,0 +1,69 @@ +"""Opt-in full-connectome checks — stonkfly's integration test transposed. + +Needs the prepared dataset (``python -m condor.fly prepare``) and about a +minute: ``CONDOR_FLY_FULL_TEST=1 uv run pytest tests/test_fly_full_graph.py``. +""" + +import os +from pathlib import Path + +import numpy as np +import pytest + +# The suite isolates CONDOR_RUNTIME_ROOT into a temp dir; the prepared +# connectome lives in the real one unless CONDOR_FLY_DATA says otherwise. +os.environ.setdefault( + "CONDOR_FLY_DATA", + str(Path(__file__).resolve().parent.parent / ".condor" / "fly" / "data"), +) + +pytestmark = pytest.mark.skipif( + os.environ.get("CONDOR_FLY_FULL_TEST") != "1", + reason="Uses the full MaleCNS graph; set CONDOR_FLY_FULL_TEST=1", +) + + +def test_chart_reaches_kenyon_cells_and_pulses_hit_dopamine_cells(tmp_path): + from condor.fly.chart import market_frame + from condor.fly.data import verify + from condor.fly.market import FixtureMarket + from condor.fly.worker import FlyBrain + + assert verify()["neurons"] == 166700 + brain = FlyBrain(learning=True) + assert len(brain.brain.post) == 25582938 + assert len(brain.descending) > 1000 and len(brain.left) and len(brain.right) + + market = FixtureMarket(["XYZ:A-USD"], 72) + import asyncio + + obs = asyncio.run(market.observe("XYZ:A-USD")) + frame = market_frame(obs.pair, obs.candles, obs.bid, obs.ask) + first = brain.observe(frame, "none") + assert first["kc_spikes"] > 0, "the rendered chart must activate Kenyon cells" + assert first["total_spikes"] > 0 and first["stimulus_ms"] == 0 + + brain.save(tmp_path / "before.npz") + before = brain.brain.weight[brain.brain.circuit["edges"]].copy() + + reward = brain.observe(frame, "reward") + assert reward["reward_spikes"] > 0 and reward["stimulus_ms"] == 200 + assert reward["memory"]["changed_edges"] > 0 + rewarded = brain.brain.weight[brain.brain.circuit["edges"]].copy() + + brain.restore(tmp_path / "before.npz") + brain.observe(frame, "none") + assert not np.array_equal( + rewarded, brain.brain.weight[brain.brain.circuit["edges"]] + ) + + brain.restore(tmp_path / "before.npz") + brain.brain.weights_frozen = True + brain.observe(frame, "reward") + assert np.array_equal(before, brain.brain.weight[brain.brain.circuit["edges"]]) + + brain.restore(tmp_path / "before.npz") + brain.brain.weights_frozen = False + loss = brain.observe(frame, "aversive") + assert loss["aversive_spikes"] > 0 and loss["stimulus_ms"] == 200 + assert np.isfinite(brain.brain.weight).all() diff --git a/tests/test_fly_guard.py b/tests/test_fly_guard.py new file mode 100644 index 000000000..47acba10e --- /dev/null +++ b/tests/test_fly_guard.py @@ -0,0 +1,139 @@ +"""Every guard rule: veto vs halt, and a financial halt that review cannot clear.""" + +import pytest + +from condor.fly.decoder import NEUTRAL +from condor.fly.guard import ( + GuardSettings, + GuardState, + Halt, + Veto, + check_apply_window, + check_collateral, + check_config, + check_market_open, + check_not_halted, + check_pnl, + check_price_move, + default_max_loss, + record_apply, + resume, +) +from condor.fly.posture import MarketSpec, build_config + +SPEC = MarketSpec("hyperliquid_perpetual", "XYZ:DRAM-USD", 500, 8.0) +S = GuardSettings() + + +def test_market_open_counts_closed_ticks_then_asks_for_stop(): + st = GuardState() + assert check_market_open("XYZ:DRAM-USD", True, st, S) is False + small = GuardSettings(closed_ticks_to_stop=3) + for _ in range(2): + with pytest.raises(Veto): + check_market_open("XYZ:DRAM-USD", False, st, small) + assert check_market_open("XYZ:DRAM-USD", False, st, small) is True + assert check_market_open("XYZ:DRAM-USD", True, st, small) is False + assert st.closed_ticks["XYZ:DRAM-USD"] == 0 + + +def test_collateral(): + check_collateral(100.0, 50.0) + with pytest.raises(Veto): + check_collateral(40.0, 50.0) + with pytest.raises(Veto): + check_collateral(float("nan"), 50.0) + + +def test_config_floors(): + good = build_config(SPEC, NEUTRAL) + check_config(good, SPEC) + for bad in ( + {"buy_spreads": "0.0001,0.0009"}, + {"take_profit": 0.0001}, + {"leverage": 20}, + {"trading_pair": "XYZ:SPCX-USD"}, + {"global_sl_enabled": False}, + ): + with pytest.raises(Veto): + check_config({**good, **bad}, SPEC) + + +def test_apply_window_resets_per_utc_day(): + st = GuardState() + small = GuardSettings(max_applies_per_day=2) + day1 = 1_700_000_000.0 + check_apply_window(st, day1, small) + record_apply(st, "P", day1, True, small) + record_apply(st, "P", day1 + 60, True, small) + with pytest.raises(Veto): + check_apply_window(st, day1 + 120, small) + check_apply_window(st, day1 + 86_400, small) + assert st.applies_today == 0 + + +def test_price_move(): + check_price_move(100.0, 100.4, S) + with pytest.raises(Veto): + check_price_move(100.0, 101.0, S) + with pytest.raises(Veto): + check_price_move(0, 1, S) + + +def test_apply_failures_halt_transiently_and_resume(): + st = GuardState() + small = GuardSettings(max_apply_failures=2) + record_apply(st, "P", 1.0, False, small) + with pytest.raises(Halt) as info: + record_apply(st, "P", 2.0, False, small) + assert info.value.financial is False + with pytest.raises(Halt): + check_not_halted(st) + with pytest.raises(Halt): + resume(st, reviewed=False) + resume(st, reviewed=True) + check_not_halted(st) + assert st.consecutive_failures == 0 + record_apply(st, "P", 3.0, True, small) + + +def test_loss_stop_is_financial_and_unclearable(): + st = GuardState() + check_pnl(1.0, 1000.0, st, S, max_loss_quote=20.0) + with pytest.raises(Halt) as info: + check_pnl(-20.0, 1000.0, st, S, max_loss_quote=20.0) + assert info.value.financial is True + with pytest.raises(Halt): + resume(st, reviewed=True) + + +def test_loss_rate_breaker_needs_volume(): + st = GuardState() + check_pnl(-0.1, 10.0, st, S, max_loss_quote=100.0) # too little volume to judge + with pytest.raises(Halt): + check_pnl(-1.0, 1000.0, st, S, max_loss_quote=100.0) # -10 bp of volume + + +def test_no_new_high_breaker(): + st = GuardState() + small = GuardSettings(loss_no_new_high_ticks=3) + check_pnl(1.0, 1000.0, st, small, 100.0) + check_pnl(0.9, 1000.0, st, small, 100.0) + check_pnl(0.9, 1000.0, st, small, 100.0) + with pytest.raises(Halt): + check_pnl(0.9, 1000.0, st, small, 100.0) + + +def test_new_high_resets_counter(): + st = GuardState() + small = GuardSettings(loss_no_new_high_ticks=3) + for net in (1.0, 0.9, 0.9, 1.1, 1.0, 1.0): + check_pnl(net, 1000.0, st, small, 100.0) + assert st.ticks_since_high == 2 + + +def test_default_max_loss_and_roundtrip(): + assert default_max_loss([SPEC, SPEC], S) == pytest.approx(40.0) + assert default_max_loss([SPEC], GuardSettings(max_loss_quote=7.0)) == 7.0 + st = GuardState(applies_today=3, closed_ticks={"X:Y-USD": 1}, halted="x") + assert GuardState.from_dict(st.to_dict()) == st diff --git a/tests/test_fly_market.py b/tests/test_fly_market.py new file mode 100644 index 000000000..3f8ae087c --- /dev/null +++ b/tests/test_fly_market.py @@ -0,0 +1,63 @@ +"""Fixture market, book parsing, candle payload shapes, collateral requirement.""" + +import asyncio + +import pytest + +from condor.fly.chart import market_frame +from condor.fly.market import ( + Book, + FixtureMarket, + normalize_candle_payload, + required_collateral, +) +from condor.fly.posture import MarketSpec + +PAIRS = ["XYZ:A-USD", "XYZ:B-USD", "XYZ:C-USD"] + + +def test_fixture_observation_renders_and_advances(): + m = FixtureMarket(PAIRS, 72) + obs = asyncio.run(m.observe("XYZ:B-USD")) + assert obs.open and obs.bid < obs.ask and len(obs.candles) == 72 + frame = market_frame(obs.pair, obs.candles, obs.bid, obs.ask) + assert frame.shape == (180, 320, 3) + again = asyncio.run(m.observe("XYZ:B-USD")) + assert again.candles[-1]["close"] != obs.candles[-1]["close"] + + +def test_fixture_equity_stays_inside_the_breaker(): + m = FixtureMarket(PAIRS, 72) + for tick in range(60): + m.tick = tick + net, volume, per_pair = asyncio.run(m.equity(PAIRS)) + assert abs(net / volume) * 1e4 < 5 + assert per_pair["XYZ:A-USD"]["running"] is False + + +def test_fixture_never_applies(): + m = FixtureMarket(PAIRS, 72) + with pytest.raises(RuntimeError): + asyncio.run(m.apply("XYZ:A-USD", {})) + assert asyncio.run(m.stop_bot("XYZ:A-USD")) is False + + +def test_book_open(): + assert Book(1.0, 1.1).open + assert not Book(None, None).open + + +def test_candle_payload_shapes(): + rows = [{"close": 1}] + assert normalize_candle_payload(rows) == rows + assert normalize_candle_payload({"data": rows}) == rows + assert normalize_candle_payload({"candles": rows}) == rows + with pytest.raises(RuntimeError): + normalize_candle_payload([]) + with pytest.raises(RuntimeError): + normalize_candle_payload({"data": []}) + + +def test_required_collateral(): + spec = MarketSpec("hyperliquid_perpetual", "XYZ:A-USD", 500, 8.0, leverage=2) + assert required_collateral([spec, spec]) == pytest.approx(2 * 500 * 0.5 / 2) diff --git a/tests/test_fly_naming.py b/tests/test_fly_naming.py new file mode 100644 index 000000000..17b6d4597 --- /dev/null +++ b/tests/test_fly_naming.py @@ -0,0 +1,29 @@ +"""Derived names for a HIP-3 pair, and the pairs list rules.""" + +import pytest + +from condor.fly.naming import pair_names, parse_pairs + + +def test_pair_names(): + n = pair_names("XYZ:DRAM-USD") + assert (n.issuer, n.token, n.coin) == ("xyz", "DRAM", "xyz:DRAM") + assert (n.bot_name, n.config_name) == ("dram-fly", "dram_fly_mm") + + +@pytest.mark.parametrize( + "bad", ["xyz:dram-usd", "DRAM-USD", "XYZ:DRAM", "XYZ:-USD", ":DRAM-USD"] +) +def test_bad_pairs(bad): + with pytest.raises(ValueError): + pair_names(bad) + + +def test_parse_pairs(): + assert parse_pairs(" XYZ:A-USD, XYZ:B-USD ") == ["XYZ:A-USD", "XYZ:B-USD"] + with pytest.raises(ValueError): + parse_pairs("") + with pytest.raises(ValueError): + parse_pairs("XYZ:A-USD,XYZ:A-USD") + with pytest.raises(ValueError): + parse_pairs("XYZ:A-USD,XYZ:B-USD,XYZ:C-USD,XYZ:D-USD") diff --git a/tests/test_fly_posture.py b/tests/test_fly_posture.py new file mode 100644 index 000000000..2dba493bc --- /dev/null +++ b/tests/test_fly_posture.py @@ -0,0 +1,107 @@ +"""Posture → pmm_mister config: floors, lean cap, timing table, pause switch.""" + +import pytest + +from condor.fly.decoder import NEUTRAL, Posture +from condor.fly.posture import ( + BPS, + TIMING, + MarketSpec, + base_levels_bps, + build_config, + config_diff, + take_profit_floor, +) + +SPEC = MarketSpec( + connector_name="hyperliquid_perpetual", + trading_pair="XYZ:DRAM-USD", + total_amount_quote=500, + picked_spread_bps=8.0, +) + + +def _spreads(value): + return [float(x) for x in value.split(",")] + + +def test_neutral_config_matches_hip3_base(): + cfg = build_config(SPEC, NEUTRAL) + l1, l2 = base_levels_bps(SPEC) + assert (l1, l2) == (4.0, 9.0) + assert _spreads(cfg["buy_spreads"]) == pytest.approx([4 * BPS, 9 * BPS]) + assert _spreads(cfg["sell_spreads"]) == pytest.approx([4 * BPS, 9 * BPS]) + assert ( + cfg["controller_name"] == "pmm_mister" and cfg["controller_type"] == "generic" + ) + assert cfg["trading_pair"] == "XYZ:DRAM-USD" + assert cfg["global_sl_enabled"] is True and cfg["global_stop_loss"] == 0.02 + assert cfg["manual_kill_switch"] is False + assert (cfg["executor_refresh_time"], cfg["buy_cooldown_time"]) == TIMING["ranging"] + assert cfg["open_order_type"] == 3 and cfg["take_profit_order_type"] == 3 + + +def test_take_profit_floor_beats_fees_and_spread(): + assert take_profit_floor(SPEC) == pytest.approx(2.2 * 2 * 1.3 * BPS) + cfg = build_config(SPEC, NEUTRAL) + assert cfg["take_profit"] >= take_profit_floor(SPEC) + cheap = MarketSpec(**{**SPEC.__dict__, "maker_fee_bps": 0.1}) + assert take_profit_floor(cheap) == pytest.approx(4 * BPS) + + +def test_volatile_widens_quiet_tightens_with_floor(): + wide = build_config(SPEC, Posture("volatile", 2.0, 0.0, 0, 1.5, False, True)) + assert _spreads(wide["buy_spreads"])[0] == pytest.approx(8 * BPS) + assert (wide["executor_refresh_time"], wide["buy_cooldown_time"]) == TIMING[ + "volatile" + ] + tight = build_config(SPEC, Posture("quiet", 0.6, 0.0, 0, -1.5, False, True)) + # 4 bp × 0.6 = 2.4 bp, floored to the 3 bp minimum + assert _spreads(tight["buy_spreads"])[0] == pytest.approx(3 * BPS) + assert (tight["executor_refresh_time"], tight["buy_cooldown_time"]) == TIMING[ + "quiet" + ] + + +def test_lean_is_asymmetric_and_capped(): + up = build_config(SPEC, Posture("trending_up", 1.0, 3.0, 2.0, 0, True, True)) + buy, sell = _spreads(up["buy_spreads"]), _spreads(up["sell_spreads"]) + # lean capped at half of level 1 (4 bp → 2 bp); buy 2 bp floored to 3 bp + assert buy[0] == pytest.approx(3 * BPS) and sell[0] == pytest.approx(6 * BPS) + assert buy[1] == pytest.approx(7 * BPS) and sell[1] == pytest.approx(11 * BPS) + down = build_config(SPEC, Posture("trending_down", 1.0, -3.0, -2.0, 0, True, True)) + assert _spreads(down["sell_spreads"])[0] == pytest.approx(3 * BPS) + assert _spreads(down["buy_spreads"])[0] == pytest.approx(6 * BPS) + + +def test_pause_sets_kill_switch(): + cfg = build_config(SPEC, Posture("pause", 2.5, 0.0, 0, 3.0, False, True)) + assert cfg["manual_kill_switch"] is True + + +def test_every_spread_respects_min(): + for regime in TIMING: + for mult in (0.6, 1.0, 2.5): + for shift in (-3.0, 0.0, 3.0): + cfg = build_config(SPEC, Posture(regime, mult, shift, 0, 0, True, True)) + for key in ("buy_spreads", "sell_spreads"): + assert min(_spreads(cfg[key])) >= SPEC.min_spread_bps * BPS - 1e-12 + + +def test_spec_validation(): + with pytest.raises(ValueError): + MarketSpec("hyperliquid_perpetual", "xyz:dram-usd", 500, 8) + with pytest.raises(ValueError): + MarketSpec("hyperliquid_perpetual", "DRAM-USD", 500, 8) + with pytest.raises(ValueError): + MarketSpec("hyperliquid_perpetual", "XYZ:DRAM-USD", 500, 8, leverage=10) + with pytest.raises(ValueError): + MarketSpec("hyperliquid_perpetual", "XYZ:DRAM-USD", 0, 8) + + +def test_config_diff(): + a = build_config(SPEC, NEUTRAL) + b = build_config(SPEC, Posture("volatile", 2.0, 0.0, 0, 1.5, False, True)) + diff = config_diff(a, b) + assert "buy_spreads" in diff and "trading_pair" not in diff + assert config_diff(None, a) == a diff --git a/tests/test_fly_reinforcement.py b/tests/test_fly_reinforcement.py new file mode 100644 index 000000000..9f3a1a310 --- /dev/null +++ b/tests/test_fly_reinforcement.py @@ -0,0 +1,33 @@ +"""P&L delta → dopamine pulse kind, and the controller net-P&L reader.""" + +from decimal import Decimal + +import pytest + +from condor.fly.reinforcement import controller_net, reinforcement + + +@pytest.mark.parametrize( + "equity,expected", + [("100.03", "reward"), ("99.97", "aversive"), ("100.001", "none"), ("100", "none")], +) +def test_explicit_feedback(equity, expected): + kind, delta = reinforcement(equity, "100", ".01") + assert kind == expected + assert delta == Decimal(equity) - Decimal("100") + + +def test_deadband_must_be_positive(): + with pytest.raises(ValueError): + reinforcement("1", "0", "0") + + +def test_controller_net_includes_unrealized(): + assert ( + controller_net({"realized_pnl_quote": "1.5", "unrealized_pnl_quote": -2}) + == -0.5 + ) + with pytest.raises(ValueError): + controller_net({"realized_pnl_quote": 1}) + with pytest.raises(ValueError): + controller_net({"realized_pnl_quote": float("inf"), "unrealized_pnl_quote": 0}) diff --git a/tests/test_fly_run_state.py b/tests/test_fly_run_state.py new file mode 100644 index 000000000..cbb09cbb3 --- /dev/null +++ b/tests/test_fly_run_state.py @@ -0,0 +1,67 @@ +"""Run directory: lock, state, events, checkpoint slots, provenance refusal.""" + +import json + +import numpy as np +import pytest + +from condor.fly.run_state import RunDir, signature, source_hashes + + +def test_lock_is_exclusive(tmp_path): + a = RunDir(tmp_path / "run") + a.lock() + b = RunDir(tmp_path / "run") + with pytest.raises(RuntimeError): + b.lock() + a.unlock() + b.lock() + b.unlock() + + +def test_state_events_latest_frame(tmp_path): + run = RunDir(tmp_path / "run") + assert run.load_state() == {} + run.save_state({"tick": 3, "anchor": 1.5}) + assert run.load_state() == {"tick": 3, "anchor": 1.5} + for i in range(5): + run.append_event({"tick": i}) + assert [e["tick"] for e in run.recent_events(3)] == [2, 3, 4] + run.write_latest({"tick": 4}) + assert json.loads(run.latest_path.read_text()) == {"tick": 4} + run.save_frame(np.zeros((180, 320, 3), np.uint8)) + assert run.frame_path.exists() + assert not run.stop_requested() + run.stop_path.touch() + assert run.stop_requested() + + +def test_checkpoint_slots_and_integrity(tmp_path): + run = RunDir(tmp_path / "run") + assert run.checkpoint_path(4).name == "brain-0.npz" + assert run.checkpoint_path(5).name == "brain-1.npz" + path = run.checkpoint_path(1) + path.write_bytes(b"brain") + import hashlib + + good = {"file": path.name, "sha256": hashlib.sha256(b"brain").hexdigest()} + assert run.verify_checkpoint(good) == path + with pytest.raises(RuntimeError): + run.verify_checkpoint({"file": path.name, "sha256": "0" * 64}) + with pytest.raises(RuntimeError): + run.verify_checkpoint({"file": "missing.npz", "sha256": "0" * 64}) + + +def test_provenance_refuses_changed_protocol(tmp_path): + run = RunDir(tmp_path / "run") + prov = {"decoder": {"window": 60}, "source_sha256": {"a.py": "1"}} + sig = run.check_provenance(prov) + assert sig == signature(prov) + assert run.check_provenance(dict(prov)) == sig + with pytest.raises(RuntimeError): + run.check_provenance({**prov, "decoder": {"window": 30}}) + + +def test_source_hashes_cover_the_package(): + hashes = source_hashes() + assert "decoder.py" in hashes and "neural/kernel.cpp" in hashes diff --git a/uv.lock b/uv.lock index e41d6e639..4e3882dce 100644 --- a/uv.lock +++ b/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 3 +revision = 2 requires-python = ">=3.12" resolution-markers = [ "python_full_version >= '3.14' and sys_platform == 'win32'", @@ -614,6 +614,7 @@ dependencies = [ { name = "pandas", version = "3.0.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.14'" }, { name = "pandas-ta" }, { name = "plotly" }, + { name = "pyarrow" }, { name = "pydantic-ai" }, { name = "python-dotenv" }, { name = "python-jose", extra = ["cryptography"] }, @@ -651,6 +652,7 @@ requires-dist = [ { name = "pandas" }, { name = "pandas-ta", specifier = ">=0.4.71b0" }, { name = "plotly" }, + { name = "pyarrow" }, { name = "pydantic-ai", extras = ["mcp"] }, { name = "python-dotenv" }, { name = "python-jose", extras = ["cryptography"] }, @@ -3086,6 +3088,42 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7f/15/f9d0171e1ad863ca49e826d5afb6b50566f20dc9b4f76965096d3555ce9e/py_vapid-1.9.4-py2.py3-none-any.whl", hash = "sha256:f165a5bf90dcf966b226114f01f178f137579a09784c7f0628fa2f0a299741b6", size = 23912, upload-time = "2026-01-05T20:42:05.455Z" }, ] +[[package]] +name = "pyarrow" +version = "25.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/3d/e3/27f57f80141379d60defe6703eb50a707325706f07fedfd1312c7a751995/pyarrow-25.0.1.tar.gz", hash = "sha256:9150a83248bfed9813ea3c3af74c3856c1984d444aa28e58bf7733b9750ddf6a", size = 1201653, upload-time = "2026-08-10T12:40:53.904Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a6/e2/9ab15b88cbfac28e16419ce5439ec29234c5172cb8259301b4ba639bdec0/pyarrow-25.0.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:df961f2e7ae9cf496459259d798652c70625f6c080650d6952f8c04053c58ee9", size = 35861559, upload-time = "2026-08-10T12:38:02.567Z" }, + { url = "https://files.pythonhosted.org/packages/58/79/a0036dbe1eabe1f73127427342f1d99982584c4a2cde2651d6c93499c6f6/pyarrow-25.0.1-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:cc4aa407fde9fc660be3939e49ea31f50f3e9fec17c0ec63159f7711edd3efc9", size = 37628383, upload-time = "2026-08-10T12:38:09.083Z" }, + { url = "https://files.pythonhosted.org/packages/13/49/d93a57d375f4bf0cf82913dd6bb54acafde83dd993be2282c81ac5616cad/pyarrow-25.0.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:4340f0ba6c1d2e13f21658de1d7c662ca2545018568d0030a1e9afca159d87e3", size = 46820190, upload-time = "2026-08-10T12:38:15.458Z" }, + { url = "https://files.pythonhosted.org/packages/60/c9/711ca85d79f1ec98f29a5eae2b051e25b4ecec5de3e3c0e2d5c5dcb15664/pyarrow-25.0.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:5389cdf79447ed1515c9e31620e6e1e2302249564d603f2ad727d4f6d313e4c3", size = 50102437, upload-time = "2026-08-10T12:38:22.487Z" }, + { url = "https://files.pythonhosted.org/packages/80/53/8fb8359ff17cfb6263a1cf3ebf7caec9fe197de118719e84fcb1d0618026/pyarrow-25.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:d51592cb7561e87877c506113e7adbf1342ab579e6c21f0ef44b8ba41cb74c80", size = 49942424, upload-time = "2026-08-10T12:38:28.755Z" }, + { url = "https://files.pythonhosted.org/packages/e8/83/4e5ae02a9341571b18a6fca380ac7a58ce6ddae7ab3c060208c0a1e79f02/pyarrow-25.0.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6109c94d8b9f3b17a041daca16cacb2f651ad8f1ef70a4232c2c0f37a23da2a8", size = 53144206, upload-time = "2026-08-10T12:38:34.862Z" }, + { url = "https://files.pythonhosted.org/packages/65/ee/197cbf47e49f83e6ebeb946a5259a48a638dea27ac774db42fe78022179d/pyarrow-25.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:8858d7bfc22e3f51529aeaa4077225029724623e4595dc9eff8c793935c34140", size = 27953934, upload-time = "2026-08-10T12:38:39.808Z" }, + { url = "https://files.pythonhosted.org/packages/cc/8d/8f271a7a034c834910ec925d56fa4b29733b1380f5289419f5aaa3b02777/pyarrow-25.0.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:c7c534ec03c358a76ea3e505e74c1b6aef290af90c444dfd092dbfe23e755b85", size = 35855328, upload-time = "2026-08-10T12:38:45.489Z" }, + { url = "https://files.pythonhosted.org/packages/d2/cd/5bac242f4e841b9971d5eb94fdfe2577e2b70be983e27401e72055786037/pyarrow-25.0.1-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:dda9470024204d7bbf2042b47c6e8a0e47a3eeb8e34405882dfaea6577e0c153", size = 37622415, upload-time = "2026-08-10T12:38:51.107Z" }, + { url = "https://files.pythonhosted.org/packages/63/1f/96d03b4e1506524f7087adb0fd6b2f69f0c9c7aaff1ec36d8030082e15a5/pyarrow-25.0.1-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:44a9120ce5bd81936b8ab9a88076e3fd47c2c6838e0e43630fed83626aca81d9", size = 46813813, upload-time = "2026-08-10T12:38:57.773Z" }, + { url = "https://files.pythonhosted.org/packages/98/d6/33a411115b61dbfc16ad6ad73e71730f6fea654ee3667673bc53ab0e2fe7/pyarrow-25.0.1-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:0befcf816e45a1af33ac775a9970b749e4868a230c7372f0ae5e932bee27039f", size = 50104452, upload-time = "2026-08-10T12:39:04.579Z" }, + { url = "https://files.pythonhosted.org/packages/33/ae/b1b97c9ca87f9f9ddbb5230c798df94eccce61bd79b9b45458c69a478588/pyarrow-25.0.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3f89685964f46e4216103c75483aac0c0692a5f72212d7ca835adba5ede56ce3", size = 49951343, upload-time = "2026-08-10T12:39:11.8Z" }, + { url = "https://files.pythonhosted.org/packages/98/9e/a112df5cfd5a68cb1d9fc31cfe38c28d5aec9f10865ce37ecef2e4450873/pyarrow-25.0.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6943e2fe7954d29d84de45d29d34c8dc36ce96570e67d89aa9976e650a4a9138", size = 53144784, upload-time = "2026-08-10T12:39:20.503Z" }, + { url = "https://files.pythonhosted.org/packages/31/24/97e8bd98f1e3b07e2ba08bcdff690674fbe16d69a7d2712cc3884665e615/pyarrow-25.0.1-cp313-cp313-win_amd64.whl", hash = "sha256:31e49a7888fcdf3a835da33ae777f6bb9a866334e5a789282fc26dcf426f7f15", size = 27870159, upload-time = "2026-08-10T12:39:26.161Z" }, + { url = "https://files.pythonhosted.org/packages/36/4c/b525824ad3094076919273cd97db61fb3d78252dee76fa3b8dc8f76774aa/pyarrow-25.0.1-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:bf0b672390cdcb640d7288f96b826d71ff4e9abb254a86c89890baf51a29cee6", size = 35885255, upload-time = "2026-08-10T12:39:32.366Z" }, + { url = "https://files.pythonhosted.org/packages/08/62/448bb0e940de41aec31d1a956e63ad9c54afdf122a103cc3ab20c2a3ce33/pyarrow-25.0.1-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:38a9a4b4b9613380e200641891495a56c3d5a98a092db4a870af9975e220471d", size = 37644461, upload-time = "2026-08-10T12:39:38.142Z" }, + { url = "https://files.pythonhosted.org/packages/6e/9a/13587e38bd4806fd218f50fd13b8903fab60588a699ff0c406372e5b4043/pyarrow-25.0.1-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:0b726ad7e7b669be982b0c71c07fe4b037d654354130da79a7902a669e93a66b", size = 46877146, upload-time = "2026-08-10T12:39:43.722Z" }, + { url = "https://files.pythonhosted.org/packages/8d/61/1c5d1229fa21da4cff5365e41e57177aaac57c563c727f35419b8513d1c1/pyarrow-25.0.1-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:9171748cdf796972d85a4b60157c279913e242992e350c90c7450182a9838b2a", size = 50131616, upload-time = "2026-08-10T12:39:49.304Z" }, + { url = "https://files.pythonhosted.org/packages/43/20/291e1d65cc0b09aa19f03cf25cf51a2f5fa94b5db315178f2d254ed5cad4/pyarrow-25.0.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:b7a296aac7a71fa0886c08e155ddb6c636a50013f801f6178daafa0f9e726188", size = 50008879, upload-time = "2026-08-10T12:39:56.891Z" }, + { url = "https://files.pythonhosted.org/packages/8b/7c/1b7c9ec28e76576337e4f97b31141c9a181b89b6d1d6221e9d8205621a58/pyarrow-25.0.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:0fe7c8b6c03969b49c8c66182e4a18e3819ab92d07cfab5d8370c531b9369ef0", size = 53170864, upload-time = "2026-08-10T12:40:04.918Z" }, + { url = "https://files.pythonhosted.org/packages/b7/75/f3d789dc06011a765d14d86bda799cf72ac1d715b6a6edecaa0d73d95062/pyarrow-25.0.1-cp314-cp314-win_amd64.whl", hash = "sha256:f729cfdbd36fd99d543b67a914d2de044c84ebe45be8b34902b299b608c15c8f", size = 28620729, upload-time = "2026-08-10T12:40:51.41Z" }, + { url = "https://files.pythonhosted.org/packages/fc/05/647a8ee6f7c2662feb6921315617bc04dcd6034763fb61b1199720bf6162/pyarrow-25.0.1-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:59a2de54c0cbd954da861eee4d1d330f8e909c45b53455baef696380f2c55033", size = 36130288, upload-time = "2026-08-10T12:40:11.014Z" }, + { url = "https://files.pythonhosted.org/packages/93/f8/c9ee997554d7bea94520667dd1933f109ac1da3ee3556d2b49381e023484/pyarrow-25.0.1-cp314-cp314t-macosx_12_0_x86_64.whl", hash = "sha256:35935cd5de130aa5cf4dea052a63e6bf2e17006c35c3a468194242b9b2bf5956", size = 37762187, upload-time = "2026-08-10T12:40:16.592Z" }, + { url = "https://files.pythonhosted.org/packages/a2/08/a28c01c7fe9e96e8233ce2d13df1d402f4f999f848f51d2daacd6bb4c036/pyarrow-25.0.1-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:f3831aaa25c67a99f99dc8b05873cb9d64560390372e2aa197ce9dd4a3f06a44", size = 46888003, upload-time = "2026-08-10T12:40:23.242Z" }, + { url = "https://files.pythonhosted.org/packages/1b/b9/58612e977d28dc58c878448866838369ee8da2f1e7cc8ed2c84b952aafee/pyarrow-25.0.1-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:6a1fdfc6659b6b19022f2e50627fb5cf7156a66c46bf4299379955cbe742382a", size = 50079036, upload-time = "2026-08-10T12:40:29.169Z" }, + { url = "https://files.pythonhosted.org/packages/72/13/66e1402dcc860e1dc2760b1e0292c9a569b62b3bccab69def1b3e907d006/pyarrow-25.0.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:169d3429d5be7c752125890620f75a60776d38b0035eddae939651640822332e", size = 50040226, upload-time = "2026-08-10T12:40:35.186Z" }, + { url = "https://files.pythonhosted.org/packages/78/10/3f1a5497a7ef732ab0f03ecca3e66d89d9c0f57fdc61b4794c456b781f01/pyarrow-25.0.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:119297a6dc197e45d9c6d4415f7814a67ffa36c180d26f68c154c58067ae782d", size = 53149035, upload-time = "2026-08-10T12:40:41.454Z" }, + { url = "https://files.pythonhosted.org/packages/93/c0/37d4a7e8e2f7a6076283673d5298018ca26478b934c6ee369e10505ab32c/pyarrow-25.0.1-cp314-cp314t-win_amd64.whl", hash = "sha256:4288f27577352d608ca08553b0865e4a9b3aa14820c5d95b53337218d609835b", size = 28753071, upload-time = "2026-08-10T12:40:46.623Z" }, +] + [[package]] name = "pyasn1" version = "0.6.3" From 1817ee2af6a577753610856ced8095990480b35e Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sat, 12 Sep 2026 18:13:04 -0700 Subject: [PATCH 02/48] (refactor) the fly lives entirely inside its agent directory Move condor/fly to agents/market_making_fly/flybrain and the tests to agents/market_making_fly/tests so the whole feature ships as one agent directory and no core Condor module changes. Routines put the agent dir on sys.path before importing flybrain; the spawned worker inherits that path. The connectome dataset now lives in the agent's own writable home (.condor/agents/market_making_fly/data) and is prepared from the agent itself by the new fly_setup routine (prepare | verify | bench). fly_brain refuses to start with a clear message until it has been prepared. --- agents/market_making_fly/AGENT.md | 1 + .../market_making_fly/flybrain}/__init__.py | 2 +- .../market_making_fly/flybrain}/__main__.py | 20 +-- .../market_making_fly/flybrain}/chart.py | 0 .../market_making_fly/flybrain}/data.py | 0 .../market_making_fly/flybrain}/decoder.py | 0 .../market_making_fly/flybrain}/guard.py | 2 +- .../market_making_fly/flybrain}/market.py | 7 +- .../market_making_fly/flybrain}/naming.py | 0 .../flybrain}/neural/THIRD_PARTY.md | 2 +- .../flybrain}/neural/__init__.py | 0 .../flybrain}/neural/arrays.lock.json | 0 .../flybrain}/neural/brain.py | 0 .../flybrain}/neural/circuit.py | 0 .../flybrain}/neural/common.py | 10 +- .../flybrain}/neural/connectome.py | 0 .../flybrain}/neural/datasets.json | 0 .../flybrain}/neural/kernel.cpp | 0 .../flybrain}/neural/neurons.lock.json | 0 .../flybrain}/neural/prepare.py | 0 .../flybrain}/neural/rule.py | 0 .../flybrain}/neural/sensory.py | 0 .../flybrain}/neural/sources.lock.json | 0 .../flybrain}/neural/state.py | 0 .../flybrain}/neural/transmitters.py | 0 .../flybrain}/neural/visual.py | 0 .../market_making_fly/flybrain}/posture.py | 2 +- .../flybrain}/reinforcement.py | 0 .../market_making_fly/flybrain}/run_state.py | 0 .../market_making_fly/flybrain}/worker.py | 4 +- .../market_making_fly/routines/fly_brain.py | 40 ++++-- .../market_making_fly/routines/fly_chart.py | 13 +- .../market_making_fly/routines/fly_setup.py | 123 ++++++++++++++++++ .../market_making_fly/routines/fly_status.py | 9 +- .../skills/fly_mm_deploy/SKILL.md | 5 +- agents/market_making_fly/tests/conftest.py | 20 +++ .../tests}/test_fly_chart.py | 3 +- .../tests}/test_fly_decoder.py | 3 +- .../tests}/test_fly_full_graph.py | 18 +-- .../tests}/test_fly_guard.py | 7 +- .../tests}/test_fly_market.py | 7 +- .../tests}/test_fly_naming.py | 3 +- .../tests}/test_fly_posture.py | 5 +- .../tests}/test_fly_reinforcement.py | 3 +- .../tests}/test_fly_run_state.py | 3 +- docs/market_making_fly_design.md | 67 +++++----- 46 files changed, 270 insertions(+), 109 deletions(-) rename {condor/fly => agents/market_making_fly/flybrain}/__init__.py (73%) rename {condor/fly => agents/market_making_fly/flybrain}/__main__.py (79%) rename {condor/fly => agents/market_making_fly/flybrain}/chart.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/data.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/decoder.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/guard.py (99%) rename {condor/fly => agents/market_making_fly/flybrain}/market.py (98%) rename {condor/fly => agents/market_making_fly/flybrain}/naming.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/THIRD_PARTY.md (92%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/__init__.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/arrays.lock.json (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/brain.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/circuit.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/common.py (68%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/connectome.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/datasets.json (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/kernel.cpp (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/neurons.lock.json (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/prepare.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/rule.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/sensory.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/sources.lock.json (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/state.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/transmitters.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/neural/visual.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/posture.py (99%) rename {condor/fly => agents/market_making_fly/flybrain}/reinforcement.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/run_state.py (100%) rename {condor/fly => agents/market_making_fly/flybrain}/worker.py (98%) create mode 100644 agents/market_making_fly/routines/fly_setup.py create mode 100644 agents/market_making_fly/tests/conftest.py rename {tests => agents/market_making_fly/tests}/test_fly_chart.py (99%) rename {tests => agents/market_making_fly/tests}/test_fly_decoder.py (99%) rename {tests => agents/market_making_fly/tests}/test_fly_full_graph.py (78%) rename {tests => agents/market_making_fly/tests}/test_fly_guard.py (97%) rename {tests => agents/market_making_fly/tests}/test_fly_market.py (94%) rename {tests => agents/market_making_fly/tests}/test_fly_naming.py (93%) rename {tests => agents/market_making_fly/tests}/test_fly_posture.py (98%) rename {tests => agents/market_making_fly/tests}/test_fly_reinforcement.py (93%) rename {tests => agents/market_making_fly/tests}/test_fly_run_state.py (96%) diff --git a/agents/market_making_fly/AGENT.md b/agents/market_making_fly/AGENT.md index 5cbdb281e..dd09d9acc 100644 --- a/agents/market_making_fly/AGENT.md +++ b/agents/market_making_fly/AGENT.md @@ -82,6 +82,7 @@ manage_skill(action="read", name="fly_mm_deploy") ## Routines | Routine | Use | |---|---| +| `fly_setup` | `action=prepare` downloads and compiles the connectome into this agent's home (once per install); `verify`; `bench` | | `hip3_market_scanner` | Rank xyz HIP-3 markets; take the top picks and their spreads | | `fly_chart` | Render the exact frame for a pair (what the fly sees) | | `fly_brain` | The loop (continuous). `mode=shadow|live`, `pairs`, `picked_spreads_bps`, `run_name` | diff --git a/condor/fly/__init__.py b/agents/market_making_fly/flybrain/__init__.py similarity index 73% rename from condor/fly/__init__.py rename to agents/market_making_fly/flybrain/__init__.py index f1ab1e8bd..967781230 100644 --- a/condor/fly/__init__.py +++ b/agents/market_making_fly/flybrain/__init__.py @@ -1,6 +1,6 @@ """Market Making Fly — a fly-connectome simulation that proposes a quoting posture. -The neural substrate (``condor.fly.neural``) is vendored from stonkfly; the +The neural substrate (``flybrain.neural``) is vendored from stonkfly; the chart, decoder, posture mapping, guard, reinforcement and worker are Condor's. See ``docs/market_making_fly_design.md``. """ diff --git a/condor/fly/__main__.py b/agents/market_making_fly/flybrain/__main__.py similarity index 79% rename from condor/fly/__main__.py rename to agents/market_making_fly/flybrain/__main__.py index 5c3831c28..f77220866 100644 --- a/condor/fly/__main__.py +++ b/agents/market_making_fly/flybrain/__main__.py @@ -1,4 +1,4 @@ -"""``uv run python -m condor.fly prepare | verify | bench``. +"""``uv run python agents/market_making_fly/flybrain/__main__.py prepare | verify | bench``. ``prepare`` downloads the MaleCNS v1.0 release files (~1.1 GB), verifies their checksums, compiles the retained graph and builds the C++ kernel. ``verify`` @@ -13,10 +13,15 @@ import json import sys import time +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) def main(argv: list[str] | None = None) -> int: - parser = argparse.ArgumentParser(prog="python -m condor.fly", description=__doc__) + parser = argparse.ArgumentParser(prog="python -m flybrain", description=__doc__) sub = parser.add_subparsers(dest="command", required=True) sub.add_parser("prepare", help="download, verify and compile the connectome") sub.add_parser("verify", help="re-check the prepared connectome arrays") @@ -25,26 +30,25 @@ def main(argv: list[str] | None = None) -> int: bench.add_argument("--neural-ms", type=float, default=500.0) args = parser.parse_args(argv) - from condor.fly.neural.common import DATA + from flybrain.neural.common import DATA if args.command == "prepare": - from condor.fly.data import prepare + from flybrain.data import prepare print(f"Data directory: {DATA}", flush=True) prepare() - from condor.fly.neural.brain import build + from flybrain.neural.brain import build print(json.dumps({"kernel": build()["model"], "data": str(DATA)})) return 0 if args.command == "verify": - from condor.fly.data import verify + from flybrain.data import verify print(json.dumps({**verify(), "data": str(DATA)})) return 0 if args.command == "bench": import numpy as np - - from condor.fly.worker import FlyBrain + from flybrain.worker import FlyBrain started = time.perf_counter() brain = FlyBrain(learning=True) diff --git a/condor/fly/chart.py b/agents/market_making_fly/flybrain/chart.py similarity index 100% rename from condor/fly/chart.py rename to agents/market_making_fly/flybrain/chart.py diff --git a/condor/fly/data.py b/agents/market_making_fly/flybrain/data.py similarity index 100% rename from condor/fly/data.py rename to agents/market_making_fly/flybrain/data.py diff --git a/condor/fly/decoder.py b/agents/market_making_fly/flybrain/decoder.py similarity index 100% rename from condor/fly/decoder.py rename to agents/market_making_fly/flybrain/decoder.py diff --git a/condor/fly/guard.py b/agents/market_making_fly/flybrain/guard.py similarity index 99% rename from condor/fly/guard.py rename to agents/market_making_fly/flybrain/guard.py index d7f81c98a..01dd811a6 100644 --- a/condor/fly/guard.py +++ b/agents/market_making_fly/flybrain/guard.py @@ -12,7 +12,7 @@ from dataclasses import asdict, dataclass, field from datetime import datetime, timezone -from condor.fly.posture import MarketSpec, take_profit_floor +from flybrain.posture import MarketSpec, take_profit_floor BPS = 1e-4 diff --git a/condor/fly/market.py b/agents/market_making_fly/flybrain/market.py similarity index 98% rename from condor/fly/market.py rename to agents/market_making_fly/flybrain/market.py index 76aa7579d..27b416342 100644 --- a/condor/fly/market.py +++ b/agents/market_making_fly/flybrain/market.py @@ -13,10 +13,9 @@ from dataclasses import dataclass import aiohttp - -from condor.fly.naming import pair_names -from condor.fly.posture import MarketSpec -from condor.fly.reinforcement import controller_net +from flybrain.naming import pair_names +from flybrain.posture import MarketSpec +from flybrain.reinforcement import controller_net HL_INFO_URL = "https://api.hyperliquid.xyz/info" QUOTE_TOKENS = ("USD", "USDC") diff --git a/condor/fly/naming.py b/agents/market_making_fly/flybrain/naming.py similarity index 100% rename from condor/fly/naming.py rename to agents/market_making_fly/flybrain/naming.py diff --git a/condor/fly/neural/THIRD_PARTY.md b/agents/market_making_fly/flybrain/neural/THIRD_PARTY.md similarity index 92% rename from condor/fly/neural/THIRD_PARTY.md rename to agents/market_making_fly/flybrain/neural/THIRD_PARTY.md index caf7cbc2f..e86bf9716 100644 --- a/condor/fly/neural/THIRD_PARTY.md +++ b/agents/market_making_fly/flybrain/neural/THIRD_PARTY.md @@ -1,6 +1,6 @@ # Third-party notice -`condor/fly/neural/` and `condor/fly/data.py` are vendored from +`agents/market_making_fly/flybrain/neural/` and `agents/market_making_fly/flybrain/data.py` are vendored from https://github.com/nftechie/stonkfly (MIT License, Copyright (c) 2026 nftechie and DOOMFLY contributors). Only `neural/common.py` is modified: the data directory resolves to Condor's runtime root instead of `./data`. diff --git a/condor/fly/neural/__init__.py b/agents/market_making_fly/flybrain/neural/__init__.py similarity index 100% rename from condor/fly/neural/__init__.py rename to agents/market_making_fly/flybrain/neural/__init__.py diff --git a/condor/fly/neural/arrays.lock.json b/agents/market_making_fly/flybrain/neural/arrays.lock.json similarity index 100% rename from condor/fly/neural/arrays.lock.json rename to agents/market_making_fly/flybrain/neural/arrays.lock.json diff --git a/condor/fly/neural/brain.py b/agents/market_making_fly/flybrain/neural/brain.py similarity index 100% rename from condor/fly/neural/brain.py rename to agents/market_making_fly/flybrain/neural/brain.py diff --git a/condor/fly/neural/circuit.py b/agents/market_making_fly/flybrain/neural/circuit.py similarity index 100% rename from condor/fly/neural/circuit.py rename to agents/market_making_fly/flybrain/neural/circuit.py diff --git a/condor/fly/neural/common.py b/agents/market_making_fly/flybrain/neural/common.py similarity index 68% rename from condor/fly/neural/common.py rename to agents/market_making_fly/flybrain/neural/common.py index 1d8cef7a0..07bac74ed 100644 --- a/condor/fly/neural/common.py +++ b/agents/market_making_fly/flybrain/neural/common.py @@ -5,13 +5,13 @@ import os from pathlib import Path -# Condor: the connectome data lives under this install's runtime root -# (``/.condor/fly/data``), overridable with ``CONDOR_FLY_DATA``. This is -# the only edit to the vendored stonkfly code. -from condor.paths import runtime_root +# Condor: the connectome data lives in this agent's writable home +# (``.condor/agents/market_making_fly/data``), overridable with ``CONDOR_FLY_DATA``. +# This is the only edit to the vendored stonkfly code. +from condor.memory.paths import agent_home DATA = Path( - os.environ.get("CONDOR_FLY_DATA") or (runtime_root() / "fly" / "data") + os.environ.get("CONDOR_FLY_DATA") or (agent_home("market_making_fly") / "data") ).resolve() GRAPH = DATA / "graph.npz" OUT = DATA / "cache" diff --git a/condor/fly/neural/connectome.py b/agents/market_making_fly/flybrain/neural/connectome.py similarity index 100% rename from condor/fly/neural/connectome.py rename to agents/market_making_fly/flybrain/neural/connectome.py diff --git a/condor/fly/neural/datasets.json b/agents/market_making_fly/flybrain/neural/datasets.json similarity index 100% rename from condor/fly/neural/datasets.json rename to agents/market_making_fly/flybrain/neural/datasets.json diff --git a/condor/fly/neural/kernel.cpp b/agents/market_making_fly/flybrain/neural/kernel.cpp similarity index 100% rename from condor/fly/neural/kernel.cpp rename to agents/market_making_fly/flybrain/neural/kernel.cpp diff --git a/condor/fly/neural/neurons.lock.json b/agents/market_making_fly/flybrain/neural/neurons.lock.json similarity index 100% rename from condor/fly/neural/neurons.lock.json rename to agents/market_making_fly/flybrain/neural/neurons.lock.json diff --git a/condor/fly/neural/prepare.py b/agents/market_making_fly/flybrain/neural/prepare.py similarity index 100% rename from condor/fly/neural/prepare.py rename to agents/market_making_fly/flybrain/neural/prepare.py diff --git a/condor/fly/neural/rule.py b/agents/market_making_fly/flybrain/neural/rule.py similarity index 100% rename from condor/fly/neural/rule.py rename to agents/market_making_fly/flybrain/neural/rule.py diff --git a/condor/fly/neural/sensory.py b/agents/market_making_fly/flybrain/neural/sensory.py similarity index 100% rename from condor/fly/neural/sensory.py rename to agents/market_making_fly/flybrain/neural/sensory.py diff --git a/condor/fly/neural/sources.lock.json b/agents/market_making_fly/flybrain/neural/sources.lock.json similarity index 100% rename from condor/fly/neural/sources.lock.json rename to agents/market_making_fly/flybrain/neural/sources.lock.json diff --git a/condor/fly/neural/state.py b/agents/market_making_fly/flybrain/neural/state.py similarity index 100% rename from condor/fly/neural/state.py rename to agents/market_making_fly/flybrain/neural/state.py diff --git a/condor/fly/neural/transmitters.py b/agents/market_making_fly/flybrain/neural/transmitters.py similarity index 100% rename from condor/fly/neural/transmitters.py rename to agents/market_making_fly/flybrain/neural/transmitters.py diff --git a/condor/fly/neural/visual.py b/agents/market_making_fly/flybrain/neural/visual.py similarity index 100% rename from condor/fly/neural/visual.py rename to agents/market_making_fly/flybrain/neural/visual.py diff --git a/condor/fly/posture.py b/agents/market_making_fly/flybrain/posture.py similarity index 99% rename from condor/fly/posture.py rename to agents/market_making_fly/flybrain/posture.py index 163cff2bc..73b502680 100644 --- a/condor/fly/posture.py +++ b/agents/market_making_fly/flybrain/posture.py @@ -18,7 +18,7 @@ import math from dataclasses import dataclass -from condor.fly.decoder import REGIMES, Posture +from flybrain.decoder import REGIMES, Posture # executor_refresh_time, buy/sell cooldown — Market Making Expert's table. TIMING: dict[str, tuple[int, int]] = { diff --git a/condor/fly/reinforcement.py b/agents/market_making_fly/flybrain/reinforcement.py similarity index 100% rename from condor/fly/reinforcement.py rename to agents/market_making_fly/flybrain/reinforcement.py diff --git a/condor/fly/run_state.py b/agents/market_making_fly/flybrain/run_state.py similarity index 100% rename from condor/fly/run_state.py rename to agents/market_making_fly/flybrain/run_state.py diff --git a/condor/fly/worker.py b/agents/market_making_fly/flybrain/worker.py similarity index 98% rename from condor/fly/worker.py rename to agents/market_making_fly/flybrain/worker.py index 388f89c40..16da9bd81 100644 --- a/condor/fly/worker.py +++ b/agents/market_making_fly/flybrain/worker.py @@ -28,8 +28,8 @@ def __init__( pulse_ms: float = 200.0, pulse_current: float = 20.0, ): - from condor.fly.neural.common import annotations - from condor.fly.neural.visual import VisualMemoryBrain + from flybrain.neural.common import annotations + from flybrain.neural.visual import VisualMemoryBrain if neural_bin_ms <= 0 or neural_bin_ms > 10: raise ValueError("neural_bin_ms must be in (0, 10]") diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index 87957959c..7e94d8051 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -18,6 +18,13 @@ from __future__ import annotations +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + import asyncio import logging import multiprocessing @@ -28,12 +35,9 @@ import numpy as np import plotly.graph_objects as go -from pydantic import BaseModel, Field -from telegram.ext import ContextTypes - -from condor.fly import worker -from condor.fly.chart import market_frame -from condor.fly.decoder import ( +from flybrain import worker +from flybrain.chart import market_frame +from flybrain.decoder import ( Baseline, Channels, DecoderSettings, @@ -42,7 +46,7 @@ decode, should_apply, ) -from condor.fly.guard import ( +from flybrain.guard import ( GuardSettings, GuardState, Halt, @@ -58,11 +62,14 @@ record_apply, resume, ) -from condor.fly.market import FixtureMarket, LiveMarket, required_collateral -from condor.fly.naming import pair_names, parse_pairs -from condor.fly.posture import MarketSpec, build_config, config_diff -from condor.fly.reinforcement import reinforcement -from condor.fly.run_state import RunDir, source_hashes +from flybrain.market import FixtureMarket, LiveMarket, required_collateral +from flybrain.naming import pair_names, parse_pairs +from flybrain.posture import MarketSpec, build_config, config_diff +from flybrain.reinforcement import reinforcement +from flybrain.run_state import RunDir, source_hashes +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + from condor.memory.paths import agent_home from condor.reports import LiveReport @@ -200,6 +207,13 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: config.reward_deadband_bps * BPS * sum(s.total_amount_quote for s in specs) ) + from flybrain.neural.common import DATA, GRAPH + + if not GRAPH.exists(): + raise RuntimeError( + f"Connectome not prepared at {DATA}; run the fly_setup routine with " + 'action="prepare" first (downloads ~1.1 GB, needs a C++ compiler)' + ) run_dir = RunDir(agent_home(AGENT_SLUG) / "fly" / config.run_name) run_dir.lock() pool = ProcessPoolExecutor( @@ -238,7 +252,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: anchor = state.get("anchor") tick = int(state.get("tick", 0)) - from condor.fly.data import verify + from flybrain.data import verify dataset = await loop.run_in_executor(None, verify) brain_prov = await loop.run_in_executor(pool, worker._provenance) diff --git a/agents/market_making_fly/routines/fly_chart.py b/agents/market_making_fly/routines/fly_chart.py index f0b5fa76a..e5e4f7c0e 100644 --- a/agents/market_making_fly/routines/fly_chart.py +++ b/agents/market_making_fly/routines/fly_chart.py @@ -2,16 +2,23 @@ from __future__ import annotations +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + import logging import aiohttp import plotly.graph_objects as go +from flybrain.chart import market_frame, normalize_candles, price_scale +from flybrain.market import fetch_l2_book, normalize_candle_payload +from flybrain.naming import pair_names from pydantic import BaseModel, Field from telegram.ext import ContextTypes -from condor.fly.chart import market_frame, normalize_candles, price_scale -from condor.fly.market import fetch_l2_book, normalize_candle_payload -from condor.fly.naming import pair_names from condor.reports import ReportBuilder from config_manager import get_client diff --git a/agents/market_making_fly/routines/fly_setup.py b/agents/market_making_fly/routines/fly_setup.py new file mode 100644 index 000000000..198b1f28c --- /dev/null +++ b/agents/market_making_fly/routines/fly_setup.py @@ -0,0 +1,123 @@ +"""Prepare, verify or benchmark the fly's connectome from inside the agent. + +``prepare`` downloads the MaleCNS v1.0 release files (~1.1 GB), verifies their +checksums, compiles the retained graph into this agent's home and builds the +C++ kernel (needs ``c++``). ``verify`` re-checks the prepared arrays. +``bench`` loads the brain and times a few observations so ``fly_brain``'s +``neural_ms`` can be sized against its wall interval. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + +import asyncio +import logging +import time + +import numpy as np +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.reports import ReportBuilder + +logger = logging.getLogger(__name__) + +CATEGORY = "Analysis" + + +class Config(BaseModel): + """Prepare (download + compile), verify, or bench the fly connectome.""" + + action: str = Field(default="verify", description="prepare | verify | bench") + observations: int = Field(default=3, description="bench: observations to time") + neural_ms: float = Field( + default=500.0, description="bench: neural ms per observation" + ) + + +def _prepare() -> dict: + from flybrain.data import prepare + from flybrain.neural.brain import build + + prepare() + return {"kernel": build()["model"]} + + +def _verify() -> dict: + from flybrain.data import verify + + return verify() + + +def _bench(observations: int, neural_ms: float) -> dict: + from flybrain.worker import FlyBrain + + started = time.perf_counter() + brain = FlyBrain(learning=True) + load = time.perf_counter() - started + frame = np.full((180, 320, 3), 235, np.uint8) + rows = [] + for _ in range(observations): + result = brain.observe(frame, "none", neural_ms=neural_ms) + rows.append( + { + "compute_seconds": round(result["compute_seconds"], 3), + "total_spikes": result["total_spikes"], + "kc_spikes": result["kc_spikes"], + "arousal_hz": round(result["arousal_hz"], 3), + } + ) + return {"load_seconds": round(load, 2), "observations": rows} + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + from flybrain.neural.common import DATA + + if config.action not in ("prepare", "verify", "bench"): + raise ValueError("action must be prepare, verify or bench") + loop = asyncio.get_running_loop() + started = time.perf_counter() + if config.action == "prepare": + result = await loop.run_in_executor(None, _prepare) + elif config.action == "verify": + result = await loop.run_in_executor(None, _verify) + else: + result = await loop.run_in_executor( + None, _bench, config.observations, config.neural_ms + ) + elapsed = time.perf_counter() - started + + builder = ReportBuilder(f"Fly setup — {config.action}") + builder.source("routine", "fly_setup") + builder.tags(["fly", "setup"]) + builder.manual_order() + builder.section("01 / RESULT", f"{config.action} in {elapsed:.1f} s") + builder.kpi("Data dir", str(DATA)) + for key, value in result.items(): + if key != "observations": + builder.kpi(key, str(value)) + if config.action == "bench": + builder.table( + result["observations"], + ["compute_seconds", "total_spikes", "kc_spikes", "arousal_hz"], + ) + await builder.save() + + lines = [ + f"action: {config.action}", + f"data_dir: {DATA}", + f"elapsed_s: {elapsed:.1f}", + ] + lines += [f"{k}: {v}" for k, v in result.items() if k != "observations"] + if config.action == "bench": + lines += [ + f"obs{i}: compute={r['compute_seconds']}s kc={r['kc_spikes']} arousal={r['arousal_hz']}Hz" + for i, r in enumerate(result["observations"]) + ] + return "\n".join(lines) diff --git a/agents/market_making_fly/routines/fly_status.py b/agents/market_making_fly/routines/fly_status.py index 597a3f653..fd53ba421 100644 --- a/agents/market_making_fly/routines/fly_status.py +++ b/agents/market_making_fly/routines/fly_status.py @@ -2,13 +2,20 @@ from __future__ import annotations +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + import json import logging +from flybrain.run_state import RunDir from pydantic import BaseModel, Field from telegram.ext import ContextTypes -from condor.fly.run_state import RunDir from condor.memory.paths import agent_home from condor.reports import ReportBuilder diff --git a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md index 228d4ff21..7da5c9b81 100644 --- a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md +++ b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md @@ -40,8 +40,9 @@ For each pair derive `token` (e.g. `DRAM`), `bot_name = {token.lower()}-fly`, ```python run_code(code=""" -from condor.fly.posture import MarketSpec, build_config -from condor.fly.decoder import NEUTRAL +import sys; sys.path.insert(0, "agents/market_making_fly") +from flybrain.posture import MarketSpec, build_config +from flybrain.decoder import NEUTRAL spec = MarketSpec(connector_name="hyperliquid_perpetual", trading_pair="XYZ:DRAM-USD", total_amount_quote=500, picked_spread_bps=8.0, leverage=3) print(build_config(spec, NEUTRAL)) diff --git a/agents/market_making_fly/tests/conftest.py b/agents/market_making_fly/tests/conftest.py new file mode 100644 index 000000000..01690e384 --- /dev/null +++ b/agents/market_making_fly/tests/conftest.py @@ -0,0 +1,20 @@ +"""Make ``flybrain`` importable: it lives in the agent dir, not on the module path.""" + +import os +import sys +from pathlib import Path + +AGENT_DIR = Path(__file__).resolve().parents[1] +REPO_ROOT = AGENT_DIR.parents[1] +# This directory has no __init__.py (a package named ``tests`` would collide +# with the repo's), so pytest does not put the repo root on the path itself. +for entry in (str(REPO_ROOT), str(AGENT_DIR)): + if entry not in sys.path: + sys.path.insert(0, entry) + +# The opt-in full-graph test needs the prepared dataset from this install's +# agent home; every other test uses tmp_path. +os.environ.setdefault( + "CONDOR_FLY_DATA", + str(REPO_ROOT / ".condor" / "agents" / "market_making_fly" / "data"), +) diff --git a/tests/test_fly_chart.py b/agents/market_making_fly/tests/test_fly_chart.py similarity index 99% rename from tests/test_fly_chart.py rename to agents/market_making_fly/tests/test_fly_chart.py index b9ee06b49..45e2b72f6 100644 --- a/tests/test_fly_chart.py +++ b/agents/market_making_fly/tests/test_fly_chart.py @@ -4,8 +4,7 @@ import numpy as np import pytest - -from condor.fly import chart +from flybrain import chart def _candles(n=72, start=100.0, step=0.5, up=True): diff --git a/tests/test_fly_decoder.py b/agents/market_making_fly/tests/test_fly_decoder.py similarity index 99% rename from tests/test_fly_decoder.py rename to agents/market_making_fly/tests/test_fly_decoder.py index 371e25c99..5e80f4e18 100644 --- a/tests/test_fly_decoder.py +++ b/agents/market_making_fly/tests/test_fly_decoder.py @@ -1,8 +1,7 @@ """Spike channels → posture: the regime table, warm-up, centring, hysteresis.""" import pytest - -from condor.fly.decoder import ( +from flybrain.decoder import ( NEUTRAL, Baseline, Channels, diff --git a/tests/test_fly_full_graph.py b/agents/market_making_fly/tests/test_fly_full_graph.py similarity index 78% rename from tests/test_fly_full_graph.py rename to agents/market_making_fly/tests/test_fly_full_graph.py index 48d47f937..4f7c6bdef 100644 --- a/tests/test_fly_full_graph.py +++ b/agents/market_making_fly/tests/test_fly_full_graph.py @@ -1,22 +1,14 @@ """Opt-in full-connectome checks — stonkfly's integration test transposed. -Needs the prepared dataset (``python -m condor.fly prepare``) and about a +Needs the prepared dataset (``python -m flybrain prepare``) and about a minute: ``CONDOR_FLY_FULL_TEST=1 uv run pytest tests/test_fly_full_graph.py``. """ import os -from pathlib import Path import numpy as np import pytest -# The suite isolates CONDOR_RUNTIME_ROOT into a temp dir; the prepared -# connectome lives in the real one unless CONDOR_FLY_DATA says otherwise. -os.environ.setdefault( - "CONDOR_FLY_DATA", - str(Path(__file__).resolve().parent.parent / ".condor" / "fly" / "data"), -) - pytestmark = pytest.mark.skipif( os.environ.get("CONDOR_FLY_FULL_TEST") != "1", reason="Uses the full MaleCNS graph; set CONDOR_FLY_FULL_TEST=1", @@ -24,10 +16,10 @@ def test_chart_reaches_kenyon_cells_and_pulses_hit_dopamine_cells(tmp_path): - from condor.fly.chart import market_frame - from condor.fly.data import verify - from condor.fly.market import FixtureMarket - from condor.fly.worker import FlyBrain + from flybrain.chart import market_frame + from flybrain.data import verify + from flybrain.market import FixtureMarket + from flybrain.worker import FlyBrain assert verify()["neurons"] == 166700 brain = FlyBrain(learning=True) diff --git a/tests/test_fly_guard.py b/agents/market_making_fly/tests/test_fly_guard.py similarity index 97% rename from tests/test_fly_guard.py rename to agents/market_making_fly/tests/test_fly_guard.py index 47acba10e..a06889481 100644 --- a/tests/test_fly_guard.py +++ b/agents/market_making_fly/tests/test_fly_guard.py @@ -1,9 +1,8 @@ """Every guard rule: veto vs halt, and a financial halt that review cannot clear.""" import pytest - -from condor.fly.decoder import NEUTRAL -from condor.fly.guard import ( +from flybrain.decoder import NEUTRAL +from flybrain.guard import ( GuardSettings, GuardState, Halt, @@ -19,7 +18,7 @@ record_apply, resume, ) -from condor.fly.posture import MarketSpec, build_config +from flybrain.posture import MarketSpec, build_config SPEC = MarketSpec("hyperliquid_perpetual", "XYZ:DRAM-USD", 500, 8.0) S = GuardSettings() diff --git a/tests/test_fly_market.py b/agents/market_making_fly/tests/test_fly_market.py similarity index 94% rename from tests/test_fly_market.py rename to agents/market_making_fly/tests/test_fly_market.py index 3f8ae087c..97497e284 100644 --- a/tests/test_fly_market.py +++ b/agents/market_making_fly/tests/test_fly_market.py @@ -3,15 +3,14 @@ import asyncio import pytest - -from condor.fly.chart import market_frame -from condor.fly.market import ( +from flybrain.chart import market_frame +from flybrain.market import ( Book, FixtureMarket, normalize_candle_payload, required_collateral, ) -from condor.fly.posture import MarketSpec +from flybrain.posture import MarketSpec PAIRS = ["XYZ:A-USD", "XYZ:B-USD", "XYZ:C-USD"] diff --git a/tests/test_fly_naming.py b/agents/market_making_fly/tests/test_fly_naming.py similarity index 93% rename from tests/test_fly_naming.py rename to agents/market_making_fly/tests/test_fly_naming.py index 17b6d4597..9a8ea3465 100644 --- a/tests/test_fly_naming.py +++ b/agents/market_making_fly/tests/test_fly_naming.py @@ -1,8 +1,7 @@ """Derived names for a HIP-3 pair, and the pairs list rules.""" import pytest - -from condor.fly.naming import pair_names, parse_pairs +from flybrain.naming import pair_names, parse_pairs def test_pair_names(): diff --git a/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py similarity index 98% rename from tests/test_fly_posture.py rename to agents/market_making_fly/tests/test_fly_posture.py index 2dba493bc..21b3fb6aa 100644 --- a/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -1,9 +1,8 @@ """Posture → pmm_mister config: floors, lean cap, timing table, pause switch.""" import pytest - -from condor.fly.decoder import NEUTRAL, Posture -from condor.fly.posture import ( +from flybrain.decoder import NEUTRAL, Posture +from flybrain.posture import ( BPS, TIMING, MarketSpec, diff --git a/tests/test_fly_reinforcement.py b/agents/market_making_fly/tests/test_fly_reinforcement.py similarity index 93% rename from tests/test_fly_reinforcement.py rename to agents/market_making_fly/tests/test_fly_reinforcement.py index 9f3a1a310..5751497a6 100644 --- a/tests/test_fly_reinforcement.py +++ b/agents/market_making_fly/tests/test_fly_reinforcement.py @@ -3,8 +3,7 @@ from decimal import Decimal import pytest - -from condor.fly.reinforcement import controller_net, reinforcement +from flybrain.reinforcement import controller_net, reinforcement @pytest.mark.parametrize( diff --git a/tests/test_fly_run_state.py b/agents/market_making_fly/tests/test_fly_run_state.py similarity index 96% rename from tests/test_fly_run_state.py rename to agents/market_making_fly/tests/test_fly_run_state.py index cbb09cbb3..981d5b683 100644 --- a/tests/test_fly_run_state.py +++ b/agents/market_making_fly/tests/test_fly_run_state.py @@ -4,8 +4,7 @@ import numpy as np import pytest - -from condor.fly.run_state import RunDir, signature, source_hashes +from flybrain.run_state import RunDir, signature, source_hashes def test_lock_is_exclusive(tmp_path): diff --git a/docs/market_making_fly_design.md b/docs/market_making_fly_design.md index f6cba31dd..e6b5d51be 100644 --- a/docs/market_making_fly_design.md +++ b/docs/market_making_fly_design.md @@ -1,6 +1,6 @@ # Market Making Fly — implementation design -Status: **draft for review, nothing implemented yet.** +Status: **implemented (see §18); this document is the reference.** Date: 2026-09-12 ## 1. Summary @@ -41,14 +41,14 @@ puts the actual connectome in the decision path, which is what you asked for. | Stonkfly piece | MM Fly | Notes | |---|---|---| -| `neural/` package: graph import, C++ LIF kernel, `MemoryBrain`, `VisualMemoryBrain`, `circuit` (PAM11/PPL101/KC/MBON), `rule` (anti-Hebbian memory), checksum locks | **Vendored unchanged** into `condor/fly/neural/` | MIT. Only the data-directory constant changes. Keeping it byte-identical keeps its lock/provenance chain valid. | -| `data.py` prepare / verify (1.1 GB MaleCNS download, checksum, compile `graph.npz`) | Vendored, exposed as `condor fly prepare` / `condor fly verify` | Data lives under Condor's local, git-ignored root. | -| `display.market_frame` (320×180 line chart) | **Replaced** by `condor/fly/chart.py`: OHLCV candlesticks + volume | Same canvas size and light palette so the retinal projection is unchanged. | -| `Decoder` (DNp20 R−L → BUY/SELL/HOLD) | **Replaced** by `condor/fly/decoder.py`: DNp20 R−L → trend/skew, a population-rate channel → arousal/spread, gate on DNpe017 | Still a fixed, engineered readout of spike counts. | +| `neural/` package: graph import, C++ LIF kernel, `MemoryBrain`, `VisualMemoryBrain`, `circuit` (PAM11/PPL101/KC/MBON), `rule` (anti-Hebbian memory), checksum locks | **Vendored unchanged** into `agents/market_making_fly/flybrain/neural/` | MIT. Only the data-directory constant changes. Keeping it byte-identical keeps its lock/provenance chain valid. | +| `data.py` prepare / verify (1.1 GB MaleCNS download, checksum, compile `graph.npz`) | Vendored, exposed as the `fly_setup` routine (`action=prepare|verify|bench`) and `python agents/market_making_fly/flybrain/__main__.py …` | Data lives in the agent's own git-ignored home. | +| `display.market_frame` (320×180 line chart) | **Replaced** by `agents/market_making_fly/flybrain/chart.py`: OHLCV candlesticks + volume | Same canvas size and light palette so the retinal projection is unchanged. | +| `Decoder` (DNp20 R−L → BUY/SELL/HOLD) | **Replaced** by `agents/market_making_fly/flybrain/decoder.py`: DNp20 R−L → trend/skew, a population-rate channel → arousal/spread, gate on DNpe017 | Still a fixed, engineered readout of spike counts. | | `reinforcement.py` (equity delta vs anchor, deadband) | Same rule, equity = controller `realized + unrealized − fees` | Filtered by the bot/controller, so other bots on the account do not leak in. | -| `risk.Guard` / `Veto` | `condor/fly/guard.py`: fee floor, loss stop, loss-rate breaker, apply cooldown, market-open, collateral, price-move tolerance | Veto keeps the previous config. It never invents a different posture. | +| `risk.Guard` / `Veto` | `agents/market_making_fly/flybrain/guard.py`: fee floor, loss stop, loss-rate breaker, apply cooldown, market-open, collateral, price-move tolerance | Veto keeps the previous config. It never invents a different posture. | | `broker.py`, `actions.py`, `ledger.py` (Coinbase FOK orders, SQLite intents) | **Dropped** | Execution is Hummingbot `pmm_mister`; the fly never places orders itself. | -| `cli.py` run loop, 2-slot checkpoints, `events.jsonl`, `latest.json`, `latest-input.png`, provenance hash | Re-implemented as the `fly_brain` continuous routine + `condor/fly/run_state.py` | Same durability pattern. | +| `cli.py` run loop, 2-slot checkpoints, `events.jsonl`, `latest.json`, `latest-input.png`, provenance hash | Re-implemented as the `fly_brain` continuous routine + `agents/market_making_fly/flybrain/run_state.py` | Same durability pattern. | ## 3. Pipeline @@ -77,7 +77,7 @@ accounting anchor are committed **before** anything is applied to the bot. ## 4. What the fly sees — chart specification -`condor/fly/chart.py: market_frame(pair, candles, bid, ask) -> np.uint8[180,320,3]` +`agents/market_making_fly/flybrain/chart.py: market_frame(pair, candles, bid, ask) -> np.uint8[180,320,3]` * Canvas 320×180, light background `(235,240,249)`, dark header bar with the pair name — identical to stonkfly. Stonkfly found the dark chart produced no @@ -111,7 +111,7 @@ operator playbook). ## 5. The neural substrate -* `condor/fly/neural/` = stonkfly's `neural/` package plus `data.py`, vendored +* `agents/market_making_fly/flybrain/neural/` = stonkfly's `neural/` package plus `data.py`, vendored with a `THIRD_PARTY.md` notice (MIT). Files: `brain.py`, `state.py`, `circuit.py`, `rule.py`, `visual.py`, `sensory.py`, `transmitters.py`, `connectome.py`, `prepare.py`, `kernel.cpp`, and the four `*.lock.json` @@ -121,12 +121,11 @@ operator playbook). per observation in 10 ms bins, 200 ms / 20 mV dopamine pulse, KC rest −60 mV with adaptation, R8 → aMe12 sign correction, memory rule gain 0.001, 1 s eligibility traces, 1,800 s memory decay, efficacy bounds 0.1–2×. -* **Data**: `condor fly prepare` downloads ~1.1 GB (annotations + edges +* **Data**: `fly_setup` (`action=prepare`) downloads ~1.1 GB (annotations + edges feather), verifies SHA-256 against the locks, compiles `graph.npz` (166,700 × 25,582,938) and builds the C++ kernel with `c++ -O3`. Location: - `/fly/data/` (git-ignored, beside `.condor/agents/`), overridable - with `CONDOR_FLY_DATA`. `condor fly verify` re-checks; `condor doctor` gets a - `fly-data` row. + `.condor/agents/market_making_fly/data/` (the agent's writable home, git-ignored), + overridable with `CONDOR_FLY_DATA`. `fly_setup` `action=verify` re-checks. * **Process model**: the simulation is CPU-bound C++ called through ctypes and must not run on Condor's event loop. The `fly_brain` routine owns a `ProcessPoolExecutor(max_workers=1, spawn)` whose initializer loads @@ -136,14 +135,14 @@ operator playbook). instance; a per-run lock file prevents two instances on one run directory (stonkfly's `worker.lock`). * **Compute budget**: stonkfly does not publish a per-observation timing. - `condor fly bench` will run three observations on a synthetic frame and print + `fly_setup` `action=bench` will run three observations on a synthetic frame and print `compute_seconds`; if it exceeds ~40 % of `interval_sec`, lower `neural_ms` (it is a config field). Memory: the graph is ~300 MB resident, checkpoints ~100 MB each × 2 slots. ## 6. Decoder — spikes → quoting posture -`condor/fly/decoder.py`. Engineered and fixed, like stonkfly's — it reads only +`agents/market_making_fly/flybrain/decoder.py`. Engineered and fixed, like stonkfly's — it reads only spike counts, and the cell IDs it reads are written to the audit log. Raw channels per observation (`seconds = neural_ms / 1000`): @@ -194,7 +193,7 @@ fitted basis. They are config fields. ## 7. Posture → `pmm_mister` config -`condor/fly/posture.py: build_config(base, posture, fees) -> dict`. The base +`agents/market_making_fly/flybrain/posture.py: build_config(base, posture, fees) -> dict`. The base config is the HIP-3 operator's bounded defaults, the posture multiplies and shifts it: @@ -226,7 +225,7 @@ config, both layers, per the deploy playbook. ## 8. Guard — vetoes only -`condor/fly/guard.py`. Every rule is deterministic; a veto means "keep the +`agents/market_making_fly/flybrain/guard.py`. Every rule is deterministic; a veto means "keep the previous config", a halt means "stop the bot and stop the fly until a human passes `resume_reviewed=true`". None of them chooses a different posture. @@ -246,7 +245,7 @@ passes `resume_reviewed=true`". None of them chooses a different posture. ## 9. Reinforcement — P&L → dopamine -`condor/fly/reinforcement.py`, stonkfly's function with a different equity +`agents/market_making_fly/flybrain/reinforcement.py`, stonkfly's function with a different equity source: * `equity_t = realized_pnl + unrealized_pnl − fees` for **this bot's** @@ -309,7 +308,8 @@ every tick and make no claim beyond them. ## 11. Condor integration — files ``` -condor/fly/ +agents/market_making_fly/ + flybrain/ # everything below lives inside the agent; nothing in condor/ changes __init__.py neural/ # vendored stonkfly.neural + THIRD_PARTY.md (MIT) data.py # vendored prepare/verify, Condor data dir @@ -320,7 +320,7 @@ condor/fly/ reinforcement.py # equity delta → reward|aversive|none worker.py # ProcessPoolExecutor target: load brain, observe, checkpoint, restore run_state.py # run dir: state.json, events.jsonl, latest.json, PNG, 2-slot checkpoints, provenance, lock, STOP - cli.py # condor fly prepare | verify | bench + __main__.py # prepare | verify | bench, runnable by path agents/market_making_fly/ AGENT.md # operator brain (LLM) @@ -338,7 +338,7 @@ agents/market_making_fly/ mm_bot_report/ # copied strategies/fly_hip3_operator/strategy.md # thin loop: keep bot + fly alive, surface halts, rotate when flat -tests/ +agents/market_making_fly/tests/ # conftest puts the agent dir on sys.path test_fly_chart.py test_fly_decoder.py test_fly_posture.py test_fly_guard.py test_fly_reinforcement.py test_fly_full_graph.py # opt-in, CONDOR_FLY_FULL_TEST=1, needs prepared data @@ -394,7 +394,7 @@ Expert. ## 12. Deploy lifecycle -1. `condor fly prepare` once per machine (1.1 GB, several minutes, needs `c++`). +1. `fly_setup` with `action=prepare` once per install (1.1 GB, several minutes, needs `c++`). 2. Operator agent runs the scanner, deploys `pmm_mister` on the TOP PICK with the neutral (ranging) posture config. 3. `manage_routines(action="start", name="fly_brain", agent="market_making_fly", @@ -475,10 +475,10 @@ Copied in spirit from stonkfly's `docs/model.md`, because the same limits hold: Settled 2026-09-12: 1 vendor, 2 deterministic apply, 3 descending neurons, 4 baseline-centred, 5 72 × 5 m, 6 net P&L incl. unrealized, 7 **three markets in parallel on one shared brain** (see §10), 8 shadow default, 9 -`.condor/fly/data`, 10 no target-base nudge. The original options are kept +`.condor/agents/market_making_fly/data`, 10 no target-base nudge. The original options are kept below for the record. -1. **Vendor stonkfly's neural package into `condor/fly/neural/` (recommended)** +1. **Vendor stonkfly's neural package into `agents/market_making_fly/flybrain/neural/` (recommended)** vs `pip install git+…stonkfly`. Installing pulls `coinbase-agentkit` and `coinbase-advanced-py` into Condor for no use; vendoring adds ~1,400 lines + the kernel + `pyarrow` as a new dependency. @@ -510,7 +510,7 @@ below for the record. | Phase | Deliverable | Verifies | |---|---|---| -| 1 | `condor/fly/neural` vendored, `data.py`, `condor fly prepare/verify/bench`, `pyarrow` dep, doctor row | `prepare` completes on this Mac, `verify` passes, `bench` prints compute time | +| 1 | `agents/market_making_fly/flybrain/neural` vendored, `data.py`, `fly_setup` routine, `pyarrow` dep | `prepare` completes on this Mac, `verify` passes, `bench` prints compute time | | 2 | `chart.py` + tests, `fly_chart` routine | rendered DRAM chart in a report | | 3 | `decoder.py`, `posture.py`, `guard.py`, `reinforcement.py` + tests | unit suite green | | 4 | `worker.py`, `run_state.py`, `fly_brain` routine, fixture mode | `fixture + fast + steps=6` run end-to-end with checkpoint resume | @@ -526,19 +526,20 @@ Phases 1–5 are implemented; phase 6 (live at small size) is the operator's cal | Piece | Where | State | |---|---|---| -| Vendored neural package, data prepare/verify, kernel | `condor/fly/neural/`, `condor/fly/data.py`, `python -m condor.fly prepare\|verify\|bench` | done; dataset prepared at `.condor/fly/data` (1.6 GB), verified, kernel built | -| Chart, decoder, posture, guard, reinforcement, naming, market, run state, worker | `condor/fly/*.py` | done, 77 unit tests green (`uv run pytest tests/test_fly_*.py`) | +| Vendored neural package, data prepare/verify, kernel | `agents/market_making_fly/flybrain/neural/`, `flybrain/data.py`, `fly_setup` routine | done; dataset prepared at `.condor/agents/market_making_fly/data` (1.6 GB), verified, kernel built | +| Chart, decoder, posture, guard, reinforcement, naming, market, run state, worker | `agents/market_making_fly/flybrain/*.py` | done, 77 unit tests green (`uv run pytest agents/market_making_fly/tests`) | | Loop routine | `agents/market_making_fly/routines/fly_brain.py` | done; 14-observation fixture run end to end (checkpoints, events, live report, halt path) | | `fly_chart`, `fly_status`, copied scanner/dashboard | `agents/market_making_fly/routines/` | done, load through routine discovery | | Agent brain, deploy playbook, decoder skill, operator strategy | `agents/market_making_fly/{AGENT.md,skills,strategies}` | done | -| Opt-in full-graph test | `tests/test_fly_full_graph.py` (`CONDOR_FLY_FULL_TEST=1`) | done | -| Doctor row | — | not done; use `python -m condor.fly verify` | +| Opt-in full-graph test | `agents/market_making_fly/tests/test_fly_full_graph.py` (`CONDOR_FLY_FULL_TEST=1`) | done | +| Doctor row | — | not done; use `fly_setup` `action=verify` | Deviations from the text above: the descending-neuron superclass label in -`graph.npz` is `descending_neuron` (1,314 cells); the CLI is -`python -m condor.fly …` rather than a `condor fly` subcommand (there is no -`condor` console script in this repo); checkpoints are ~7 MB compressed, not -100 MB. +`graph.npz` is `descending_neuron` (1,314 cells); the whole implementation is +contained in `agents/market_making_fly/` (package `flybrain`, put on `sys.path` +by the routines) so no core Condor module changes — the only repo-level edit is +the `pyarrow` dependency; setup is the `fly_setup` routine rather than a CLI; +checkpoints are ~7 MB compressed, not 100 MB. ### First measurements From 3cdd8bc7488932218966fc82c08389ad86a6cfb7 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sat, 12 Sep 2026 18:18:30 -0700 Subject: [PATCH 03/48] (feat) the operator picks how many markets the fly quotes, one to three --- agents/market_making_fly/AGENT.md | 3 ++- .../market_making_fly/routines/fly_brain.py | 2 +- .../skills/fly_mm_deploy/SKILL.md | 11 +++++++---- .../strategies/fly_hip3_operator/strategy.md | 19 +++++++++++++++++++ 4 files changed, 29 insertions(+), 6 deletions(-) diff --git a/agents/market_making_fly/AGENT.md b/agents/market_making_fly/AGENT.md index dd09d9acc..d62cb82e7 100644 --- a/agents/market_making_fly/AGENT.md +++ b/agents/market_making_fly/AGENT.md @@ -52,7 +52,8 @@ floor, loss stop, loss-rate breaker, apply cooldown, closed books, collateral) c veto or halt, and it never substitutes a posture either. ## What you handle -- Deploying the fly market maker end-to-end on up to three HIP-3 pairs (`fly_mm_deploy`) +- Deploying the fly market maker end-to-end on `n_markets` (1–3) HIP-3 pairs (`fly_mm_deploy`); + the count comes from the strategy config or the task, default 3 - Starting `fly_brain` in shadow, then live; stopping it; rotating a market slot - Reading `fly_status` and explaining a posture in market-making terms - Reporting bot health with `mm_bot_report` / `mm_dashboard` diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index 7e94d8051..fe2fc6b8e 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -87,7 +87,7 @@ class Config(BaseModel): pairs: str = Field( default="XYZ:DRAM-USD,XYZ:SPCX-USD,XYZ:SMSN-USD", - description="Up to 3 uppercase HIP-3 pairs, comma-separated (bot {token}-fly, config {token}_fly_mm)", + description="1 to 3 uppercase HIP-3 pairs, comma-separated — this list IS the market count (bot {token}-fly, config {token}_fly_mm)", ) picked_spreads_bps: str = Field( default="8,8,8", diff --git a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md index 7da5c9b81..f6e68fd65 100644 --- a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md +++ b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md @@ -22,8 +22,8 @@ manage_routines(action="run", name="hip3_market_scanner", config={"issuer": "xyz", "min_spread_bps": 3, "max_daily_drift_pct": 3, "top_n": 5}) ``` -Take up to **three** survivors from the top of the ranking that have an open live -book. Record for each: `pair` (uppercase, e.g. `XYZ:DRAM-USD`) and its **spread in +Take the top **`n_markets`** survivors (1–3; from `[CURRENT CONFIG]` or the task, +default 3) that have an open live book — one brain quotes them all in round-robin. Record for each: `pair` (uppercase, e.g. `XYZ:DRAM-USD`) and its **spread in bp** — this is `picked_spreads_bps`. If fewer than one survivor, stop and report. ## Step 2 — Collateral @@ -72,10 +72,13 @@ controller. ## Step 5 — Start the fly in shadow +`pairs` and `picked_spreads_bps` list exactly the `n_markets` picks, same order. +With `n_markets: 1` that is a single pair and a single spread. + ``` manage_routines(action="start", name="fly_brain", config={ - "pairs": "XYZ:DRAM-USD,XYZ:SPCX-USD,XYZ:SMSN-USD", - "picked_spreads_bps": "8,6,10", + "pairs": "XYZ:DRAM-USD,XYZ:SPCX-USD,XYZ:SMSN-USD", # n_markets entries + "picked_spreads_bps": "8,6,10", # one per pair "total_amount_quote": 500, "leverage": 3, "mode": "shadow", "run_name": "fly-2026-09-12"}) ``` diff --git a/agents/market_making_fly/strategies/fly_hip3_operator/strategy.md b/agents/market_making_fly/strategies/fly_hip3_operator/strategy.md index a77661b40..7d071126d 100644 --- a/agents/market_making_fly/strategies/fly_hip3_operator/strategy.md +++ b/agents/market_making_fly/strategies/fly_hip3_operator/strategy.md @@ -9,6 +9,8 @@ default_config: total_amount_quote: 500 execution_mode: loop run_name: fly + n_markets: 3 + mode: shadow risk_limits: max_position_size_quote: 600 max_open_executors: 12 @@ -22,6 +24,23 @@ created_at: '2026-09-12T00:00:00+00:00' You are Market Making Fly's operator loop. The fly (`fly_brain`) quotes; you keep the plumbing healthy. **You never choose spreads, lean or regime.** +## Configuration at launch + +Read these from `[CURRENT CONFIG]`: +- `n_markets` (1–3, default 3): how many HIP-3 markets the fly quotes at once. The + one shared brain is shown that many charts in round-robin; each pair is observed + every `n_markets × interval_sec`. Fewer markets means each one is seen more often. +- `total_amount_quote`: capital **per market**. +- `run_name`: the fly's run directory (brain lineage). +- `mode`: `shadow` or `live`. +- `trading_context`, if present, may name the pairs explicitly ("MM XYZ:DRAM-USD and + XYZ:SPCX-USD"); then `n_markets` is the count of those pairs. + +If the fly is not yet deployed, run the `fly_mm_deploy` skill with exactly +`n_markets` picks from the scanner. Never start `fly_brain` with more pairs than +`n_markets`, and never fewer unless the scanner has fewer open survivors — say so +in the journal when that happens. + ## Each tick 1. `manage_routines(action="run", name="fly_status", config={"run_name": ""})` From 7c2e6f5c81d94c1e0443d363a752432ab7f2acb4 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sat, 12 Sep 2026 18:19:07 -0700 Subject: [PATCH 04/48] (fix) the deploy summary in the brain counts n_markets picks, not three --- agents/market_making_fly/AGENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/agents/market_making_fly/AGENT.md b/agents/market_making_fly/AGENT.md index d62cb82e7..5f6d0593b 100644 --- a/agents/market_making_fly/AGENT.md +++ b/agents/market_making_fly/AGENT.md @@ -72,7 +72,7 @@ Run `fly_status` (and `fly_chart` for the picture), answer in key: value lines, the numbers (trend_z, arousal_z, gate, regime, spread ×, lean, changed edges), and repeat the caveats. Do not deploy. -**Delegated:** read `fly_mm_deploy` and follow it end-to-end — scanner → three picks → +**Delegated:** read `fly_mm_deploy` and follow it end-to-end — scanner → `n_markets` picks → neutral configs → deploy with a loss cap → start `fly_brain` in shadow — then verify with `mm_bot_report`. Switch to live only when the task says so. From a9d7f665ac0e8f9dbf9b7075d6fec3afa4d12102 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sat, 12 Sep 2026 23:09:17 -0700 Subject: [PATCH 05/48] (fix) first review round: paths, pacing, prices, vanished bots, order size - run_name goes through condor.paths.safe_id in fly_brain and fly_status, so a run can only ever live under the agent's home. - A closed-book tick now waits out the interval like every other tick; the closed-tick and no-new-high counters advance at the configured cadence. - check_price_move and the l2Book reader reject nonfinite or non-positive prices instead of letting NaN slip past the stale-price guard. - equity() carries a vanished bot's last figures forward so a stopped or dropped bot cannot produce a fake equity jump; the carry is persisted. - apply() saves the controller config before touching the live bot; a failed save changes nothing, a failed live update leaves state honest. - Shadow keeps its own per-pair apply clock, so a shadow trial shows the same cooldown live would. - fly_setup bench is bounded (1-20 observations, 200-5000 ms). - Bot names: the deploy tool suffixes instances with -YYYYMMDD-HHMMSS; the fly now finds orcl-fly-20260913-055821 for orcl-fly and refuses ambiguity. - portfolio_allocation is the operator's setting in fly_brain; build_config refuses a spec whose orders would fall under the exchange minimum, and the loop checks it at start (one market at 200 quote needs 0.2 or more). - The provenance signature covers protocol (settings, decoder, guard, dataset, circuit); source hashes are recorded but a bug fix no longer orphans a run. --- agents/market_making_fly/flybrain/guard.py | 5 +- agents/market_making_fly/flybrain/market.py | 86 +++++++-- agents/market_making_fly/flybrain/posture.py | 22 +++ .../market_making_fly/flybrain/run_state.py | 9 +- .../market_making_fly/routines/fly_brain.py | 55 ++++-- .../market_making_fly/routines/fly_setup.py | 9 +- .../market_making_fly/routines/fly_status.py | 3 +- .../skills/fly_mm_deploy/SKILL.md | 11 +- .../tests/test_fly_market.py | 2 +- .../tests/test_fly_posture.py | 13 ++ .../tests/test_fly_review_fixes.py | 164 ++++++++++++++++++ .../tests/test_fly_run_state.py | 2 + 12 files changed, 340 insertions(+), 41 deletions(-) create mode 100644 agents/market_making_fly/tests/test_fly_review_fixes.py diff --git a/agents/market_making_fly/flybrain/guard.py b/agents/market_making_fly/flybrain/guard.py index 01dd811a6..a7b59d045 100644 --- a/agents/market_making_fly/flybrain/guard.py +++ b/agents/market_making_fly/flybrain/guard.py @@ -145,8 +145,9 @@ def check_apply_window(state: GuardState, now: float, s: GuardSettings) -> None: def check_price_move(observed_mid: float, fresh_mid: float, s: GuardSettings) -> None: - if observed_mid <= 0 or fresh_mid <= 0: - raise Veto("non-positive mid") + for name, value in (("observed", observed_mid), ("fresh", fresh_mid)): + if not math.isfinite(value) or value <= 0: + raise Veto(f"{name} mid is not a positive finite price: {value!r}") if abs(fresh_mid - observed_mid) / observed_mid > s.apply_price_tolerance: raise Veto("price moved beyond neural observation tolerance") diff --git a/agents/market_making_fly/flybrain/market.py b/agents/market_making_fly/flybrain/market.py index 27b416342..c9344371e 100644 --- a/agents/market_making_fly/flybrain/market.py +++ b/agents/market_making_fly/flybrain/market.py @@ -9,6 +9,7 @@ from __future__ import annotations import math +import re import time from dataclasses import dataclass @@ -18,6 +19,7 @@ from flybrain.reinforcement import controller_net HL_INFO_URL = "https://api.hyperliquid.xyz/info" +_SUFFIXED = re.compile(r"-\d{8}-\d{6}") QUOTE_TOKENS = ("USD", "USDC") @@ -59,7 +61,12 @@ async def fetch_l2_book(session: aiohttp.ClientSession, coin: str) -> Book: bids, asks = levels if not bids or not asks: return Book(None, None) # closed / empty book, a real state not an error - return Book(float(bids[0]["px"]), float(asks[0]["px"])) + bid, ask = float(bids[0]["px"]), float(asks[0]["px"]) + if not (math.isfinite(bid) and math.isfinite(ask)) or bid <= 0 or ask < bid: + raise RuntimeError( + f"l2Book {coin}: invalid top of book bid={bid!r} ask={ask!r}" + ) + return Book(bid, ask) def normalize_candle_payload(result) -> list[dict]: @@ -113,20 +120,53 @@ async def bots(self) -> dict: data = raw.get("data", raw) return data if isinstance(data, dict) else {} - async def equity(self, pairs: list[str]) -> tuple[float, float, dict]: + @staticmethod + def find_bot(bots: dict, bot_name: str) -> tuple[str | None, dict | None]: + """The deploy tool suffixes instance names with ``-YYYYMMDD-HHMMSS``, + so ``orcl-fly`` runs as ``orcl-fly-20260913-055821``. Match the exact + name or that suffix form; refuse an ambiguous match.""" + matches = [ + name + for name in bots + if name == bot_name + or _SUFFIXED.fullmatch(name[len(bot_name) :]) + and name.startswith(bot_name) + ] + if len(matches) > 1: + raise RuntimeError(f"several bots match {bot_name!r}: {sorted(matches)}") + if not matches: + return None, None + bot = bots[matches[0]] + return matches[0], (bot if isinstance(bot, dict) else None) + + async def equity( + self, pairs: list[str], carry: dict[str, dict] | None = None + ) -> tuple[float, float, dict, dict]: """Combined ``realized + unrealized`` and volume across the fly's bots. - A bot that is not running contributes nothing — there is no P&L to - report. Returns ``(net, volume, per_pair)``.""" + ``carry`` holds the last figures each bot reported. A bot that has never + reported contributes nothing (a shadow run with no bot). A bot that + reported before and is now missing from the active list — stopped, + archived, or dropped from one status response — keeps contributing its + last known figures, frozen, so its result does not vanish from the + combined book and produce a fake equity jump. Returns + ``(net, volume, per_pair, carry)``; persist ``carry`` between ticks.""" bots = await self.bots() + carry = {k: dict(v) for k, v in (carry or {}).items()} net = 0.0 volume = 0.0 per_pair: dict[str, dict] = {} for pair in pairs: names = pair_names(pair) - bot = bots.get(names.bot_name) - if not isinstance(bot, dict): - per_pair[pair] = {"running": False} + _, bot = self.find_bot(bots, names.bot_name) + if bot is None: + previous = carry.get(pair) + if previous: + net += previous["net"] + volume += previous["volume"] + per_pair[pair] = {"running": False, "carried": True, **previous} + else: + per_pair[pair] = {"running": False} continue perf = (bot.get("performance") or {}).get(names.config_name) if not isinstance(perf, dict): @@ -139,9 +179,10 @@ async def equity(self, pairs: list[str]) -> tuple[float, float, dict]: net += pair_net volume += pair_volume per_pair[pair] = {"running": True, "net": pair_net, "volume": pair_volume} + carry[pair] = {"net": pair_net, "volume": pair_volume} if not math.isfinite(net) or not math.isfinite(volume): raise RuntimeError("Nonfinite bot performance") - return net, volume, per_pair + return net, volume, per_pair, carry async def available_usd(self) -> float: state = await self.client.portfolio.get_portfolio_state() @@ -165,23 +206,30 @@ async def available_usd(self) -> float: return total async def apply(self, pair: str, config: dict) -> None: - """Update the live bot's controller and the saved config, both layers.""" + """Update the saved config, then the live bot's controller. + + Durable layer first: if saving fails nothing has changed on the bot and + the tick is a clean error. If the live update then fails, the saved + config is ahead of the bot, the tick is an error, ``applied`` keeps the + old posture, and the next material posture retries both — the bot is + never left running a config the run state does not know about.""" names = pair_names(pair) - bots = await self.bots() - if names.bot_name not in bots: + running, _ = self.find_bot(await self.bots(), names.bot_name) + if running is None: raise RuntimeError(f"bot {names.bot_name} is not running") - await self.client.controllers.update_bot_controller_config( - names.bot_name, names.config_name, config - ) await self.client.controllers.create_or_update_controller_config( names.config_name, config ) + await self.client.controllers.update_bot_controller_config( + running, names.config_name, config + ) async def stop_bot(self, pair: str) -> bool: names = pair_names(pair) - if names.bot_name not in await self.bots(): + running, _ = self.find_bot(await self.bots(), names.bot_name) + if running is None: return False - await self.client.bot_orchestration.stop_and_archive_bot(names.bot_name) + await self.client.bot_orchestration.stop_and_archive_bot(running) return True @@ -222,12 +270,14 @@ async def fresh_mid(self, pair: str) -> float: index = self.pairs.index(pair) return self._price(index, self.tick + self.n_candles) - async def equity(self, pairs: list[str]) -> tuple[float, float, dict]: + async def equity( + self, pairs: list[str], carry: dict[str, dict] | None = None + ) -> tuple[float, float, dict, dict]: # Swings so both pulses fire, drifts up so new highs recur, and stays # within ±2 bp of volume so the loss-rate breaker is not exercised here. net = 2.0 * math.sin(self.tick * 1.3) + 0.05 * self.tick volume = 10_000.0 * (self.tick + 1) - return net, volume, {p: {"running": False, "fixture": True} for p in pairs} + return net, volume, {p: {"running": False, "fixture": True} for p in pairs}, {} async def available_usd(self) -> float: return 1e9 diff --git a/agents/market_making_fly/flybrain/posture.py b/agents/market_making_fly/flybrain/posture.py index 73b502680..1faf79097 100644 --- a/agents/market_making_fly/flybrain/posture.py +++ b/agents/market_making_fly/flybrain/posture.py @@ -52,6 +52,10 @@ class MarketSpec: max_active_executors_by_level: int = 2 global_stop_loss: float = 0.02 leverage_cap: int = 5 + # Exchange minimum per order. pmm_mister sizes one cycle as + # total_amount_quote × portfolio_allocation, split across both sides and + # every level, so a small book needs a larger allocation to clear it. + min_order_notional: float = 10.0 def __post_init__(self): if self.trading_pair != self.trading_pair.upper(): @@ -75,6 +79,23 @@ def __post_init__(self): raise ValueError("0 < min_base < target_base < max_base < 1 required") if not 0 < self.portfolio_allocation <= 1: raise ValueError("portfolio_allocation must be in (0, 1]") + if self.min_order_notional < 0: + raise ValueError("min_order_notional must be >= 0") + + @property + def order_notional(self) -> float: + """Quote size of one order at neutral posture: one cycle's allocation + over two sides and two levels.""" + return self.total_amount_quote * self.portfolio_allocation / 4 + + def check_order_size(self) -> None: + if self.order_notional < self.min_order_notional: + needed = self.min_order_notional * 4 / self.total_amount_quote + raise ValueError( + f"{self.trading_pair}: an order would be {self.order_notional:.2f} quote, " + f"below the {self.min_order_notional:.0f} minimum; raise portfolio_allocation " + f"to at least {min(1.0, needed):.2f} or total_amount_quote" + ) def take_profit_floor(spec: MarketSpec) -> float: @@ -94,6 +115,7 @@ def _fmt(values: list[float]) -> str: def build_config(spec: MarketSpec, posture: Posture) -> dict: if posture.regime not in TIMING: raise ValueError(f"Unknown regime {posture.regime!r}") + spec.check_order_size() l1, l2 = base_levels_bps(spec) levels = [l1 * posture.spread_mult, l2 * posture.spread_mult] shift = max(-levels[0] / 2, min(levels[0] / 2, posture.shift_bps)) diff --git a/agents/market_making_fly/flybrain/run_state.py b/agents/market_making_fly/flybrain/run_state.py index fa3456abe..c56cfeee8 100644 --- a/agents/market_making_fly/flybrain/run_state.py +++ b/agents/market_making_fly/flybrain/run_state.py @@ -32,9 +32,16 @@ def source_hashes() -> dict[str, str]: } +# Recorded in provenance.json for the audit trail, but not part of the +# signature a resume is checked against: a bug fix in the loop must not orphan +# a brain lineage. Settings, decoder, guard, dataset and circuit are. +UNSIGNED_KEYS = ("source_sha256",) + + def signature(provenance: dict) -> str: + signed = {k: v for k, v in provenance.items() if k not in UNSIGNED_KEYS} return hashlib.sha256( - json.dumps(provenance, sort_keys=True, default=str).encode() + json.dumps(signed, sort_keys=True, default=str).encode() ).hexdigest() diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index fe2fc6b8e..3ced88b1e 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -71,6 +71,7 @@ from telegram.ext import ContextTypes from condor.memory.paths import agent_home +from condor.paths import safe_id from condor.reports import LiveReport logger = logging.getLogger(__name__) @@ -100,6 +101,10 @@ class Config(BaseModel): default=500.0, description="Capital per pair (quote)" ) leverage: int = Field(default=3, description="Leverage per pair (cap 5)") + portfolio_allocation: float = Field( + default=0.2, + description="Fraction of total_amount_quote quoted per cycle; each order is total × allocation / 4 and must clear the exchange minimum (10 USD on HIP-3) — one market at 200 quote needs 0.2+", + ) mode: str = Field( default="shadow", description="shadow (record only) or live (apply configs)" ) @@ -108,7 +113,7 @@ class Config(BaseModel): ) run_name: str = Field( default="fly", - description="Run directory under the agent home; new name = new brain lineage", + description="Run directory under the agent home (letters, digits, dot, dash, underscore); new name = new brain lineage", ) resume_reviewed: bool = Field( default=False, description="Clear a transient halt after review" @@ -165,6 +170,7 @@ def _specs(config: Config, pairs: list[str]) -> list[MarketSpec]: total_amount_quote=config.total_amount_quote, picked_spread_bps=spread, leverage=config.leverage, + portfolio_allocation=config.portfolio_allocation, ) for pair, spread in zip(pairs, spreads) ] @@ -194,6 +200,8 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: raise ValueError("interval_sec must be >= 10") pairs = parse_pairs(config.pairs) specs = _specs(config, pairs) + for spec in specs: + spec.check_order_size() # fail at start, not on the first apply by_pair = {s.trading_pair: s for s in specs} decoder_settings = DecoderSettings( window=config.baseline_window, @@ -214,7 +222,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: f"Connectome not prepared at {DATA}; run the fly_setup routine with " 'action="prepare" first (downloads ~1.1 GB, needs a C++ compiler)' ) - run_dir = RunDir(agent_home(AGENT_SLUG) / "fly" / config.run_name) + run_dir = RunDir(agent_home(AGENT_SLUG) / "fly" / safe_id(config.run_name)) run_dir.lock() pool = ProcessPoolExecutor( max_workers=1, @@ -251,6 +259,10 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: } anchor = state.get("anchor") tick = int(state.get("tick", 0)) + pnl_carry: dict = state.get("pnl_carry", {}) + # Shadow keeps its own per-pair apply clock so the trial shows the same + # cooldown live would, without touching the live guard's accounting. + shadow_last: dict = state.get("shadow_last", {}) from flybrain.data import verify @@ -315,10 +327,22 @@ def persist(extra: dict | None = None) -> None: p: (q.to_dict() if q else None) for p, q in postures.items() }, "applied": applied, + "pnl_carry": pnl_carry, + "shadow_last": shadow_last, **(extra or {}), } ) + async def pace() -> None: + """Wait out the rest of the interval; every tick path ends here.""" + if config.fast: + return + until = started + config.interval_sec + while time.monotonic() < until: + if run_dir.stop_requested(): + break + await asyncio.sleep(min(1.0, until - time.monotonic())) + # ---- the loop ------------------------------------------------------- while not config.steps or count < config.steps: started = time.monotonic() @@ -339,7 +363,9 @@ def persist(extra: dict | None = None) -> None: } try: obs = await market.observe(pair) - equity, volume, per_pair = await market.equity(pairs) + equity, volume, per_pair, pnl_carry = await market.equity( + pairs, pnl_carry + ) if anchor is None: kind, delta = "none", 0.0 else: @@ -378,6 +404,10 @@ def persist(extra: dict | None = None) -> None: run_dir.append_event(row) run_dir.write_latest(row) count += 1 + if config.steps and count >= config.steps: + stop_reason = f"{count} steps done" + break + await pace() continue check_market_open(pair, True, guard_state, guard_settings) @@ -407,18 +437,20 @@ def persist(extra: dict | None = None) -> None: row.update({"neural": neural_row, "posture": posture.to_dict()}) now = time.time() + last_apply_ts = ( + guard_state.last_apply.get(pair) + if config.mode == "live" + else shadow_last.get(pair) + ) ok, why = should_apply( - postures[pair], - posture, - guard_state.last_apply.get(pair), - now, - hysteresis, + postures[pair], posture, last_apply_ts, now, hysteresis ) diff = config_diff(applied[pair], proposed) if not ok: row["execution"] = {"status": "HOLD", "reason": why} elif config.mode == "shadow": postures[pair] = posture + shadow_last[pair] = now row["execution"] = { "status": "SHADOW", "reason": why, @@ -621,12 +653,7 @@ def persist(extra: dict | None = None) -> None: if config.steps and count >= config.steps: stop_reason = f"{count} steps done" break - if not config.fast: - until = started + config.interval_sec - while time.monotonic() < until: - if run_dir.stop_requested(): - break - await asyncio.sleep(min(1.0, until - time.monotonic())) + await pace() except asyncio.CancelledError: stop_reason = "cancelled" raise diff --git a/agents/market_making_fly/routines/fly_setup.py b/agents/market_making_fly/routines/fly_setup.py index 198b1f28c..0b5f873ef 100644 --- a/agents/market_making_fly/routines/fly_setup.py +++ b/agents/market_making_fly/routines/fly_setup.py @@ -35,9 +35,14 @@ class Config(BaseModel): """Prepare (download + compile), verify, or bench the fly connectome.""" action: str = Field(default="verify", description="prepare | verify | bench") - observations: int = Field(default=3, description="bench: observations to time") + observations: int = Field( + default=3, ge=1, le=20, description="bench: observations to time (1–20)" + ) neural_ms: float = Field( - default=500.0, description="bench: neural ms per observation" + default=500.0, + ge=200.0, + le=5000.0, + description="bench: neural ms per observation (200–5000)", ) diff --git a/agents/market_making_fly/routines/fly_status.py b/agents/market_making_fly/routines/fly_status.py index fd53ba421..4cd13d10f 100644 --- a/agents/market_making_fly/routines/fly_status.py +++ b/agents/market_making_fly/routines/fly_status.py @@ -17,6 +17,7 @@ from telegram.ext import ContextTypes from condor.memory.paths import agent_home +from condor.paths import safe_id from condor.reports import ReportBuilder logger = logging.getLogger(__name__) @@ -41,7 +42,7 @@ def _fmt(value, digits=3) -> str: async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: - root = agent_home(AGENT_SLUG) / "fly" / config.run_name + root = agent_home(AGENT_SLUG) / "fly" / safe_id(config.run_name) if not root.exists(): raise FileNotFoundError( f"No fly run at {root}; start fly_brain with run_name={config.run_name!r}" diff --git a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md index f6e68fd65..55325ebda 100644 --- a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md +++ b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md @@ -44,11 +44,18 @@ import sys; sys.path.insert(0, "agents/market_making_fly") from flybrain.posture import MarketSpec, build_config from flybrain.decoder import NEUTRAL spec = MarketSpec(connector_name="hyperliquid_perpetual", trading_pair="XYZ:DRAM-USD", - total_amount_quote=500, picked_spread_bps=8.0, leverage=3) + total_amount_quote=500, picked_spread_bps=8.0, leverage=3, + portfolio_allocation=0.2) print(build_config(spec, NEUTRAL)) """) ``` +`build_config` refuses a spec whose orders would fall under the exchange minimum +(10 USD on HIP-3): each order is `total_amount_quote × portfolio_allocation / 4`. +200 quote on one market needs `portfolio_allocation` ≥ 0.2; 100 quote needs ≥ 0.4. +Whatever value you use here, pass the **same** `portfolio_allocation` to `fly_brain` +in Step 5 — it rebuilds every config from it. + Then save it: ``` @@ -79,7 +86,7 @@ With `n_markets: 1` that is a single pair and a single spread. manage_routines(action="start", name="fly_brain", config={ "pairs": "XYZ:DRAM-USD,XYZ:SPCX-USD,XYZ:SMSN-USD", # n_markets entries "picked_spreads_bps": "8,6,10", # one per pair - "total_amount_quote": 500, "leverage": 3, + "total_amount_quote": 500, "leverage": 3, "portfolio_allocation": 0.2, "mode": "shadow", "run_name": "fly-2026-09-12"}) ``` diff --git a/agents/market_making_fly/tests/test_fly_market.py b/agents/market_making_fly/tests/test_fly_market.py index 97497e284..07cb001cb 100644 --- a/agents/market_making_fly/tests/test_fly_market.py +++ b/agents/market_making_fly/tests/test_fly_market.py @@ -29,7 +29,7 @@ def test_fixture_equity_stays_inside_the_breaker(): m = FixtureMarket(PAIRS, 72) for tick in range(60): m.tick = tick - net, volume, per_pair = asyncio.run(m.equity(PAIRS)) + net, volume, per_pair, _ = asyncio.run(m.equity(PAIRS)) assert abs(net / volume) * 1e4 < 5 assert per_pair["XYZ:A-USD"]["running"] is False diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py index 21b3fb6aa..a61c5fa63 100644 --- a/agents/market_making_fly/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -98,6 +98,19 @@ def test_spec_validation(): MarketSpec("hyperliquid_perpetual", "XYZ:DRAM-USD", 0, 8) +def test_order_size_floor(): + small = MarketSpec("hyperliquid_perpetual", "XYZ:A-USD", 200, 5.0) + assert small.order_notional == pytest.approx(10.0) + small.check_order_size() + tiny = MarketSpec("hyperliquid_perpetual", "XYZ:A-USD", 100, 5.0) + with pytest.raises(ValueError, match="at least 0.40"): + build_config(tiny, NEUTRAL) + full = MarketSpec( + "hyperliquid_perpetual", "XYZ:A-USD", 100, 5.0, portfolio_allocation=1.0 + ) + build_config(full, NEUTRAL) + + def test_config_diff(): a = build_config(SPEC, NEUTRAL) b = build_config(SPEC, Posture("volatile", 2.0, 0.0, 0, 1.5, False, True)) diff --git a/agents/market_making_fly/tests/test_fly_review_fixes.py b/agents/market_making_fly/tests/test_fly_review_fixes.py new file mode 100644 index 000000000..33450669e --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_review_fixes.py @@ -0,0 +1,164 @@ +"""Regressions for the first review round: nonfinite prices, vanished bots, +suffixed bot names, durable-first apply, bounded bench, validated run names.""" + +import asyncio +import math + +import pytest +from flybrain.guard import GuardSettings, Veto, check_price_move +from flybrain.market import Book, LiveMarket + + +class _Controllers: + def __init__(self, fail_saved=False, fail_live=False): + self.calls = [] + self.fail_saved = fail_saved + self.fail_live = fail_live + + async def create_or_update_controller_config(self, name, config): + self.calls.append(("saved", name)) + if self.fail_saved: + raise RuntimeError("saved failed") + + async def update_bot_controller_config(self, bot, name, config): + self.calls.append(("live", bot, name)) + if self.fail_live: + raise RuntimeError("live failed") + + +class _Orchestration: + def __init__(self, bots): + self._bots = bots + self.stopped = [] + + async def get_active_bots_status(self): + return {"data": self._bots} + + async def stop_and_archive_bot(self, name): + self.stopped.append(name) + + +class _Client: + def __init__(self, bots, **kw): + self.bot_orchestration = _Orchestration(bots) + self.controllers = _Controllers(**kw) + + +def _perf(net, volume): + return { + "performance": { + "realized_pnl_quote": net, + "unrealized_pnl_quote": 0, + "volume_traded": volume, + } + } + + +def _market(bots, **kw): + return LiveMarket(_Client(bots, **kw), "hyperliquid_perpetual", "5m", 72) + + +@pytest.mark.parametrize("bad", [float("nan"), float("inf"), 0.0, -1.0]) +def test_price_move_rejects_nonfinite(bad): + with pytest.raises(Veto): + check_price_move(100.0, bad, GuardSettings()) + with pytest.raises(Veto): + check_price_move(bad, 100.0, GuardSettings()) + + +def test_find_bot_accepts_deploy_suffix_and_refuses_ambiguity(): + bots = {"orcl-fly-20260913-055821": {}, "dram-fly": {}, "orcl-flyer": {}} + assert LiveMarket.find_bot(bots, "orcl-fly")[0] == "orcl-fly-20260913-055821" + assert LiveMarket.find_bot(bots, "dram-fly")[0] == "dram-fly" + assert LiveMarket.find_bot(bots, "spcx-fly") == (None, None) + with pytest.raises(RuntimeError): + LiveMarket.find_bot({**bots, "orcl-fly": {}}, "orcl-fly") + + +def test_equity_carries_a_vanished_bot(): + running = { + "orcl-fly-20260913-055821": {"performance": {"orcl_fly_mm": _perf(-1.5, 400)}} + } + net, volume, per_pair, carry = asyncio.run( + _market(running).equity(["XYZ:ORCL-USD"]) + ) + assert (net, volume) == (-1.5, 400) and per_pair["XYZ:ORCL-USD"]["running"] + # bot disappears from the status response: its last figures stay in the book + net2, volume2, per_pair2, carry2 = asyncio.run( + _market({}).equity(["XYZ:ORCL-USD"], carry) + ) + assert (net2, volume2) == (-1.5, 400) + assert per_pair2["XYZ:ORCL-USD"] == { + "running": False, + "carried": True, + "net": -1.5, + "volume": 400, + } + assert carry2 == carry + # a bot that never reported contributes nothing + net3, _, per_pair3, _ = asyncio.run(_market({}).equity(["XYZ:DRAM-USD"])) + assert net3 == 0 and per_pair3["XYZ:DRAM-USD"] == {"running": False} + + +def test_apply_saves_before_touching_the_live_bot(): + bots = {"orcl-fly-20260913-055821": {}} + m = _market(bots) + asyncio.run(m.apply("XYZ:ORCL-USD", {"x": 1})) + assert m.client.controllers.calls == [ + ("saved", "orcl_fly_mm"), + ("live", "orcl-fly-20260913-055821", "orcl_fly_mm"), + ] + failing = _market(bots, fail_saved=True) + with pytest.raises(RuntimeError): + asyncio.run(failing.apply("XYZ:ORCL-USD", {"x": 1})) + assert failing.client.controllers.calls == [("saved", "orcl_fly_mm")] + with pytest.raises(RuntimeError): + asyncio.run(_market({}).apply("XYZ:ORCL-USD", {"x": 1})) + + +def test_stop_bot_uses_the_running_name(): + m = _market({"orcl-fly-20260913-055821": {}}) + assert asyncio.run(m.stop_bot("XYZ:ORCL-USD")) is True + assert m.client.bot_orchestration.stopped == ["orcl-fly-20260913-055821"] + assert asyncio.run(_market({}).stop_bot("XYZ:ORCL-USD")) is False + + +def test_book_requires_finite_positive_prices(): + assert Book(1.0, 1.1).open and not Book(None, None).open + assert not math.isfinite(float("nan")) + + +def test_setup_bench_bounds(): + import importlib.util + from pathlib import Path + + path = Path(__file__).resolve().parents[1] / "routines" / "fly_setup.py" + spec = importlib.util.spec_from_file_location("fly_setup", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + assert mod.Config(observations=20, neural_ms=200).observations == 20 + with pytest.raises(ValueError): + mod.Config(observations=0) + with pytest.raises(ValueError): + mod.Config(observations=21) + with pytest.raises(ValueError): + mod.Config(neural_ms=10) + + +def test_status_rejects_escaping_run_name(): + import importlib.util + from pathlib import Path + + from condor.paths import UnsafeIdError + + path = Path(__file__).resolve().parents[1] / "routines" / "fly_status.py" + spec = importlib.util.spec_from_file_location("fly_status", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + + class Ctx: + _chat_id = 1 + + for bad in ("../../etc", "/tmp/x", "a/b"): + with pytest.raises(UnsafeIdError): + asyncio.run(mod.run(mod.Config(run_name=bad), Ctx())) diff --git a/agents/market_making_fly/tests/test_fly_run_state.py b/agents/market_making_fly/tests/test_fly_run_state.py index 981d5b683..1ffc98447 100644 --- a/agents/market_making_fly/tests/test_fly_run_state.py +++ b/agents/market_making_fly/tests/test_fly_run_state.py @@ -59,6 +59,8 @@ def test_provenance_refuses_changed_protocol(tmp_path): assert run.check_provenance(dict(prov)) == sig with pytest.raises(RuntimeError): run.check_provenance({**prov, "decoder": {"window": 30}}) + # source hashes are recorded but a code change does not refuse the resume + assert run.check_provenance({**prov, "source_sha256": {"a.py": "2"}}) == sig def test_source_hashes_cover_the_package(): From 0cc5ae49be37c4dadc0db8f59e3ed9aa79ffafcb Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sat, 12 Sep 2026 23:09:49 -0700 Subject: [PATCH 06/48] (fix) a live config update carries its id --- agents/market_making_fly/flybrain/market.py | 6 ++++-- agents/market_making_fly/tests/test_fly_review_fixes.py | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/agents/market_making_fly/flybrain/market.py b/agents/market_making_fly/flybrain/market.py index c9344371e..139b89fda 100644 --- a/agents/market_making_fly/flybrain/market.py +++ b/agents/market_making_fly/flybrain/market.py @@ -217,11 +217,13 @@ async def apply(self, pair: str, config: dict) -> None: running, _ = self.find_bot(await self.bots(), names.bot_name) if running is None: raise RuntimeError(f"bot {names.bot_name} is not running") + # The live update endpoint requires ``id`` equal to the config name. + payload = {"id": names.config_name, **config} await self.client.controllers.create_or_update_controller_config( - names.config_name, config + names.config_name, payload ) await self.client.controllers.update_bot_controller_config( - running, names.config_name, config + running, names.config_name, payload ) async def stop_bot(self, pair: str) -> bool: diff --git a/agents/market_making_fly/tests/test_fly_review_fixes.py b/agents/market_making_fly/tests/test_fly_review_fixes.py index 33450669e..56ea6d71f 100644 --- a/agents/market_making_fly/tests/test_fly_review_fixes.py +++ b/agents/market_making_fly/tests/test_fly_review_fixes.py @@ -16,6 +16,7 @@ def __init__(self, fail_saved=False, fail_live=False): self.fail_live = fail_live async def create_or_update_controller_config(self, name, config): + assert config["id"] == name self.calls.append(("saved", name)) if self.fail_saved: raise RuntimeError("saved failed") From e423a6667458274104e91899ecafcba760e83a04 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sat, 12 Sep 2026 23:14:18 -0700 Subject: [PATCH 07/48] (fix) a reload sees current flybrain code, an old provenance re-signs, an unreported bot is a state not an error --- agents/market_making_fly/flybrain/market.py | 23 +++++++++++++++---- .../market_making_fly/flybrain/run_state.py | 7 +++++- .../market_making_fly/routines/fly_brain.py | 15 ++++++++++++ .../market_making_fly/routines/fly_chart.py | 4 ++++ .../market_making_fly/routines/fly_setup.py | 4 ++++ .../market_making_fly/routines/fly_status.py | 4 ++++ 6 files changed, 52 insertions(+), 5 deletions(-) diff --git a/agents/market_making_fly/flybrain/market.py b/agents/market_making_fly/flybrain/market.py index 139b89fda..3d64b035d 100644 --- a/agents/market_making_fly/flybrain/market.py +++ b/agents/market_making_fly/flybrain/market.py @@ -170,15 +170,30 @@ async def equity( continue perf = (bot.get("performance") or {}).get(names.config_name) if not isinstance(perf, dict): - raise RuntimeError( - f"bot {names.bot_name} is running without controller {names.config_name}" - ) + # The bot is up but the API has no performance report for its + # controller (no MQTT report yet, or reporting broken). Keep the + # last known figures; the loop decides what that permits. + previous = carry.get(pair) + if previous: + net += previous["net"] + volume += previous["volume"] + per_pair[pair] = { + "running": True, + "reported": False, + **(previous or {}), + } + continue inner = perf.get("performance", perf) pair_net = controller_net(inner) pair_volume = float(inner.get("volume_traded", 0) or 0) net += pair_net volume += pair_volume - per_pair[pair] = {"running": True, "net": pair_net, "volume": pair_volume} + per_pair[pair] = { + "running": True, + "reported": True, + "net": pair_net, + "volume": pair_volume, + } carry[pair] = {"net": pair_net, "volume": pair_volume} if not math.isfinite(net) or not math.isfinite(volume): raise RuntimeError("Nonfinite bot performance") diff --git a/agents/market_making_fly/flybrain/run_state.py b/agents/market_making_fly/flybrain/run_state.py index c56cfeee8..ce7b350f4 100644 --- a/agents/market_making_fly/flybrain/run_state.py +++ b/agents/market_making_fly/flybrain/run_state.py @@ -126,7 +126,12 @@ def check_provenance(self, provenance: dict) -> str: sig = signature(provenance) if self.provenance_path.exists(): recorded = json.loads(self.provenance_path.read_text()) - if recorded.get("signature") != sig: + # Re-sign what was recorded under the current rule, so a change to + # which keys are signed does not itself refuse every existing run. + recorded_sig = signature( + {k: v for k, v in recorded.items() if k != "signature"} + ) + if recorded_sig != sig: raise RuntimeError( "Run protocol changed (settings, decoder, dataset or source); " "use a new run_name or review the migration explicitly" diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index 3ced88b1e..0576c1fb6 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -24,6 +24,10 @@ _AGENT_DIR = str(Path(__file__).resolve().parents[1]) if _AGENT_DIR not in sys.path: sys.path.insert(0, _AGENT_DIR) +# Condor re-executes this file when it changes but keeps imported modules +# cached; drop flybrain's so the reload actually picks up the package. +for _name in [m for m in sys.modules if m == "flybrain" or m.startswith("flybrain.")]: + del sys.modules[_name] import asyncio import logging @@ -459,6 +463,17 @@ async def pace() -> None: else: try: check_not_halted(guard_state) + unreported = [ + p + for p, info in per_pair.items() + if info.get("running") and not info.get("reported") + ] + if unreported: + raise Veto( + "no performance report for " + + ", ".join(unreported) + + "; the guard cannot see P&L" + ) check_apply_window(guard_state, now, guard_settings) check_config(proposed, spec) check_collateral( diff --git a/agents/market_making_fly/routines/fly_chart.py b/agents/market_making_fly/routines/fly_chart.py index e5e4f7c0e..be177976a 100644 --- a/agents/market_making_fly/routines/fly_chart.py +++ b/agents/market_making_fly/routines/fly_chart.py @@ -8,6 +8,10 @@ _AGENT_DIR = str(Path(__file__).resolve().parents[1]) if _AGENT_DIR not in sys.path: sys.path.insert(0, _AGENT_DIR) +# Condor re-executes this file when it changes but keeps imported modules +# cached; drop flybrain's so the reload actually picks up the package. +for _name in [m for m in sys.modules if m == "flybrain" or m.startswith("flybrain.")]: + del sys.modules[_name] import logging diff --git a/agents/market_making_fly/routines/fly_setup.py b/agents/market_making_fly/routines/fly_setup.py index 0b5f873ef..eef945c19 100644 --- a/agents/market_making_fly/routines/fly_setup.py +++ b/agents/market_making_fly/routines/fly_setup.py @@ -15,6 +15,10 @@ _AGENT_DIR = str(Path(__file__).resolve().parents[1]) if _AGENT_DIR not in sys.path: sys.path.insert(0, _AGENT_DIR) +# Condor re-executes this file when it changes but keeps imported modules +# cached; drop flybrain's so the reload actually picks up the package. +for _name in [m for m in sys.modules if m == "flybrain" or m.startswith("flybrain.")]: + del sys.modules[_name] import asyncio import logging diff --git a/agents/market_making_fly/routines/fly_status.py b/agents/market_making_fly/routines/fly_status.py index 4cd13d10f..1eaad53c9 100644 --- a/agents/market_making_fly/routines/fly_status.py +++ b/agents/market_making_fly/routines/fly_status.py @@ -8,6 +8,10 @@ _AGENT_DIR = str(Path(__file__).resolve().parents[1]) if _AGENT_DIR not in sys.path: sys.path.insert(0, _AGENT_DIR) +# Condor re-executes this file when it changes but keeps imported modules +# cached; drop flybrain's so the reload actually picks up the package. +for _name in [m for m in sys.modules if m == "flybrain" or m.startswith("flybrain.")]: + del sys.modules[_name] import json import logging From f79fdbfb6d44908e1f2f1236c9cd086b4b683744 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sat, 12 Sep 2026 23:14:47 -0700 Subject: [PATCH 08/48] (test) unreported bots and re-signed provenance --- .../tests/test_fly_review_fixes.py | 21 ++++++++++++++++++- .../tests/test_fly_run_state.py | 5 +++++ 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/agents/market_making_fly/tests/test_fly_review_fixes.py b/agents/market_making_fly/tests/test_fly_review_fixes.py index 56ea6d71f..7c312a6cc 100644 --- a/agents/market_making_fly/tests/test_fly_review_fixes.py +++ b/agents/market_making_fly/tests/test_fly_review_fixes.py @@ -83,7 +83,7 @@ def test_equity_carries_a_vanished_bot(): net, volume, per_pair, carry = asyncio.run( _market(running).equity(["XYZ:ORCL-USD"]) ) - assert (net, volume) == (-1.5, 400) and per_pair["XYZ:ORCL-USD"]["running"] + assert (net, volume) == (-1.5, 400) and per_pair["XYZ:ORCL-USD"]["reported"] # bot disappears from the status response: its last figures stay in the book net2, volume2, per_pair2, carry2 = asyncio.run( _market({}).equity(["XYZ:ORCL-USD"], carry) @@ -101,6 +101,25 @@ def test_equity_carries_a_vanished_bot(): assert net3 == 0 and per_pair3["XYZ:DRAM-USD"] == {"running": False} +def test_equity_marks_a_running_bot_without_a_report(): + unreported = {"orcl-fly-20260913-055821": {"performance": {}}} + net, volume, per_pair, carry = asyncio.run( + _market(unreported).equity( + ["XYZ:ORCL-USD"], {"XYZ:ORCL-USD": {"net": 2.0, "volume": 50}} + ) + ) + assert (net, volume) == (2.0, 50) + assert per_pair["XYZ:ORCL-USD"] == { + "running": True, + "reported": False, + "net": 2.0, + "volume": 50, + } + assert carry == {"XYZ:ORCL-USD": {"net": 2.0, "volume": 50}} + _, _, fresh, _ = asyncio.run(_market(unreported).equity(["XYZ:ORCL-USD"])) + assert fresh["XYZ:ORCL-USD"] == {"running": True, "reported": False} + + def test_apply_saves_before_touching_the_live_bot(): bots = {"orcl-fly-20260913-055821": {}} m = _market(bots) diff --git a/agents/market_making_fly/tests/test_fly_run_state.py b/agents/market_making_fly/tests/test_fly_run_state.py index 1ffc98447..6d9f97179 100644 --- a/agents/market_making_fly/tests/test_fly_run_state.py +++ b/agents/market_making_fly/tests/test_fly_run_state.py @@ -61,6 +61,11 @@ def test_provenance_refuses_changed_protocol(tmp_path): run.check_provenance({**prov, "decoder": {"window": 30}}) # source hashes are recorded but a code change does not refuse the resume assert run.check_provenance({**prov, "source_sha256": {"a.py": "2"}}) == sig + # a run recorded under an older signing rule is re-signed, not refused + stale = json.loads(run.provenance_path.read_text()) + stale["signature"] = "0" * 64 + run.provenance_path.write_text(json.dumps(stale)) + assert run.check_provenance(prov) == sig def test_source_hashes_cover_the_package(): From 2fefa930e4713e788102035aa3bc3d8f29c6f278 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sat, 12 Sep 2026 23:18:30 -0700 Subject: [PATCH 09/48] (fix) routines keep the cached flybrain package; restart Condor after editing it --- agents/market_making_fly/routines/fly_brain.py | 6 +++--- agents/market_making_fly/routines/fly_chart.py | 4 ---- agents/market_making_fly/routines/fly_setup.py | 4 ---- agents/market_making_fly/routines/fly_status.py | 4 ---- docs/market_making_fly_design.md | 4 +++- 5 files changed, 6 insertions(+), 16 deletions(-) diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index 0576c1fb6..ea4c145bc 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -25,9 +25,9 @@ if _AGENT_DIR not in sys.path: sys.path.insert(0, _AGENT_DIR) # Condor re-executes this file when it changes but keeps imported modules -# cached; drop flybrain's so the reload actually picks up the package. -for _name in [m for m in sys.modules if m == "flybrain" or m.startswith("flybrain.")]: - del sys.modules[_name] +# cached: after editing anything under flybrain/, restart Condor. Purging the +# cache here is not an option — each routine would re-import its own copy and +# the spawned brain worker could no longer pickle flybrain.worker._init. import asyncio import logging diff --git a/agents/market_making_fly/routines/fly_chart.py b/agents/market_making_fly/routines/fly_chart.py index be177976a..e5e4f7c0e 100644 --- a/agents/market_making_fly/routines/fly_chart.py +++ b/agents/market_making_fly/routines/fly_chart.py @@ -8,10 +8,6 @@ _AGENT_DIR = str(Path(__file__).resolve().parents[1]) if _AGENT_DIR not in sys.path: sys.path.insert(0, _AGENT_DIR) -# Condor re-executes this file when it changes but keeps imported modules -# cached; drop flybrain's so the reload actually picks up the package. -for _name in [m for m in sys.modules if m == "flybrain" or m.startswith("flybrain.")]: - del sys.modules[_name] import logging diff --git a/agents/market_making_fly/routines/fly_setup.py b/agents/market_making_fly/routines/fly_setup.py index eef945c19..0b5f873ef 100644 --- a/agents/market_making_fly/routines/fly_setup.py +++ b/agents/market_making_fly/routines/fly_setup.py @@ -15,10 +15,6 @@ _AGENT_DIR = str(Path(__file__).resolve().parents[1]) if _AGENT_DIR not in sys.path: sys.path.insert(0, _AGENT_DIR) -# Condor re-executes this file when it changes but keeps imported modules -# cached; drop flybrain's so the reload actually picks up the package. -for _name in [m for m in sys.modules if m == "flybrain" or m.startswith("flybrain.")]: - del sys.modules[_name] import asyncio import logging diff --git a/agents/market_making_fly/routines/fly_status.py b/agents/market_making_fly/routines/fly_status.py index 1eaad53c9..4cd13d10f 100644 --- a/agents/market_making_fly/routines/fly_status.py +++ b/agents/market_making_fly/routines/fly_status.py @@ -8,10 +8,6 @@ _AGENT_DIR = str(Path(__file__).resolve().parents[1]) if _AGENT_DIR not in sys.path: sys.path.insert(0, _AGENT_DIR) -# Condor re-executes this file when it changes but keeps imported modules -# cached; drop flybrain's so the reload actually picks up the package. -for _name in [m for m in sys.modules if m == "flybrain" or m.startswith("flybrain.")]: - del sys.modules[_name] import json import logging diff --git a/docs/market_making_fly_design.md b/docs/market_making_fly_design.md index e6b5d51be..f49d85fb8 100644 --- a/docs/market_making_fly_design.md +++ b/docs/market_making_fly_design.md @@ -539,7 +539,9 @@ Deviations from the text above: the descending-neuron superclass label in contained in `agents/market_making_fly/` (package `flybrain`, put on `sys.path` by the routines) so no core Condor module changes — the only repo-level edit is the `pyarrow` dependency; setup is the `fly_setup` routine rather than a CLI; -checkpoints are ~7 MB compressed, not 100 MB. +checkpoints are ~7 MB compressed, not 100 MB. Condor re-executes a routine +file when it changes but keeps imported modules cached, so after editing +anything under `flybrain/` restart Condor before starting `fly_brain`. ### First measurements From 4a16cc08747fadb0f8a6700e5f0915b168ffd4c7 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sat, 12 Sep 2026 23:23:24 -0700 Subject: [PATCH 10/48] (fix) sizing and cadence are recorded in provenance, not signed --- .../market_making_fly/flybrain/run_state.py | 14 ++++++++++++++ .../tests/test_fly_run_state.py | 19 +++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/agents/market_making_fly/flybrain/run_state.py b/agents/market_making_fly/flybrain/run_state.py index ce7b350f4..7a347f985 100644 --- a/agents/market_making_fly/flybrain/run_state.py +++ b/agents/market_making_fly/flybrain/run_state.py @@ -36,10 +36,24 @@ def source_hashes() -> dict[str, str]: # signature a resume is checked against: a bug fix in the loop must not orphan # a brain lineage. Settings, decoder, guard, dataset and circuit are. UNSIGNED_KEYS = ("source_sha256",) +# Sizing and cadence are the operator's per-deployment choices, not the +# protocol the brain lineage was formed under; they are recorded, not signed. +UNSIGNED_SETTINGS = ( + "total_amount_quote", + "leverage", + "portfolio_allocation", + "interval_sec", + "connector_name", +) def signature(provenance: dict) -> str: signed = {k: v for k, v in provenance.items() if k not in UNSIGNED_KEYS} + settings = signed.get("settings") + if isinstance(settings, dict): + signed["settings"] = { + k: v for k, v in settings.items() if k not in UNSIGNED_SETTINGS + } return hashlib.sha256( json.dumps(signed, sort_keys=True, default=str).encode() ).hexdigest() diff --git a/agents/market_making_fly/tests/test_fly_run_state.py b/agents/market_making_fly/tests/test_fly_run_state.py index 6d9f97179..4480b9f58 100644 --- a/agents/market_making_fly/tests/test_fly_run_state.py +++ b/agents/market_making_fly/tests/test_fly_run_state.py @@ -68,6 +68,25 @@ def test_provenance_refuses_changed_protocol(tmp_path): assert run.check_provenance(prov) == sig +def test_sizing_settings_are_recorded_but_not_signed(tmp_path): + run = RunDir(tmp_path / "run") + prov = {"settings": {"neural_ms": 500, "total_amount_quote": 200, "leverage": 3}} + sig = run.check_provenance(prov) + resized = { + "settings": { + "neural_ms": 500, + "total_amount_quote": 500, + "leverage": 5, + "portfolio_allocation": 1.0, + } + } + assert run.check_provenance(resized) == sig + with pytest.raises(RuntimeError): + run.check_provenance( + {"settings": {"neural_ms": 700, "total_amount_quote": 200}} + ) + + def test_source_hashes_cover_the_package(): hashes = source_hashes() assert "decoder.py" in hashes and "neural/kernel.cpp" in hashes From 13130b819eb6b5dd32f8c03ed614c4fdf231efa2 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 00:15:47 -0700 Subject: [PATCH 11/48] (fix) the P&L breakers count reported ticks, never silence; the first report sets the high --- agents/market_making_fly/flybrain/guard.py | 8 ++++++-- agents/market_making_fly/routines/fly_brain.py | 15 ++++++++++++--- agents/market_making_fly/tests/test_fly_guard.py | 10 ++++++++++ 3 files changed, 28 insertions(+), 5 deletions(-) diff --git a/agents/market_making_fly/flybrain/guard.py b/agents/market_making_fly/flybrain/guard.py index a7b59d045..219fad624 100644 --- a/agents/market_making_fly/flybrain/guard.py +++ b/agents/market_making_fly/flybrain/guard.py @@ -56,7 +56,9 @@ class GuardState: applies_today: int = 0 last_apply: dict[str, float] = field(default_factory=dict) # per pair consecutive_failures: int = 0 - session_high_net: float = 0.0 + # None until the first tick with reported P&L: an unreported book has no + # high to fall from, and a first negative figure is not a drawdown. + session_high_net: float | None = None ticks_since_high: int = 0 closed_ticks: dict[str, int] = field(default_factory=dict) halted: str | None = None @@ -180,7 +182,9 @@ def check_pnl( unrealized P&L — realized alone can look fine while the open position bleeds.""" if not math.isfinite(total_net) or not math.isfinite(volume): raise Veto("P&L figures are not finite") - if total_net > state.session_high_net: + # Call only on ticks whose P&L is reported: breakers count reported ticks, + # never silence, and the first report sets the high. + if state.session_high_net is None or total_net > state.session_high_net: state.session_high_net = total_net state.ticks_since_high = 0 else: diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index ea4c145bc..f00f62cf0 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -370,7 +370,11 @@ async def pace() -> None: equity, volume, per_pair, pnl_carry = await market.equity( pairs, pnl_carry ) - if anchor is None: + running_bots = [p for p, i in per_pair.items() if i.get("running")] + pnl_known = bool(running_bots) and all( + per_pair[p].get("reported") for p in running_bots + ) + if anchor is None or not pnl_known: kind, delta = "none", 0.0 else: kind, delta = reinforcement(equity, anchor, deadband) @@ -382,10 +386,14 @@ async def pace() -> None: "volume": volume, "pnl_delta": delta, "stimulus": kind, + "pnl_known": pnl_known, "bots": per_pair, } ) - check_pnl(equity, volume, guard_state, guard_settings, max_loss) + if pnl_known: + # Breakers judge reported P&L only; an unreported book is + # neither a loss nor a high. + check_pnl(equity, volume, guard_state, guard_settings, max_loss) if not obs.open: # Nothing new for the fly to see; keep the anchor so the next @@ -433,7 +441,8 @@ async def pace() -> None: # Checkpoint and anchor are committed before any apply. ck = run_dir.checkpoint_path(tick) sha = await loop.run_in_executor(pool, worker._checkpoint, str(ck)) - anchor = equity + if pnl_known: + anchor = equity tick += 1 persist({"checkpoint": {"file": ck.name, "sha256": sha}}) diff --git a/agents/market_making_fly/tests/test_fly_guard.py b/agents/market_making_fly/tests/test_fly_guard.py index a06889481..7d5087453 100644 --- a/agents/market_making_fly/tests/test_fly_guard.py +++ b/agents/market_making_fly/tests/test_fly_guard.py @@ -123,6 +123,16 @@ def test_no_new_high_breaker(): check_pnl(0.9, 1000.0, st, small, 100.0) +def test_first_reported_figure_is_the_high_not_a_drawdown(): + st = GuardState() + small = GuardSettings(loss_no_new_high_ticks=2) + check_pnl(-0.5, 1000.0, st, small, 100.0) # first report, negative + assert st.session_high_net == -0.5 and st.ticks_since_high == 0 + check_pnl(-0.6, 1000.0, st, small, 100.0) + with pytest.raises(Halt): + check_pnl(-0.6, 1000.0, st, small, 100.0) + + def test_new_high_resets_counter(): st = GuardState() small = GuardSettings(loss_no_new_high_ticks=3) From 68f070f81cf09f0c65fe46b1e9c3f4e897def967 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 00:16:23 -0700 Subject: [PATCH 12/48] (test) the first-report rule is tested clear of the loss-rate breaker --- agents/market_making_fly/tests/test_fly_guard.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/agents/market_making_fly/tests/test_fly_guard.py b/agents/market_making_fly/tests/test_fly_guard.py index 7d5087453..dd4855366 100644 --- a/agents/market_making_fly/tests/test_fly_guard.py +++ b/agents/market_making_fly/tests/test_fly_guard.py @@ -126,11 +126,11 @@ def test_no_new_high_breaker(): def test_first_reported_figure_is_the_high_not_a_drawdown(): st = GuardState() small = GuardSettings(loss_no_new_high_ticks=2) - check_pnl(-0.5, 1000.0, st, small, 100.0) # first report, negative + check_pnl(-0.5, 100_000.0, st, small, 100.0) # first report, negative assert st.session_high_net == -0.5 and st.ticks_since_high == 0 - check_pnl(-0.6, 1000.0, st, small, 100.0) + check_pnl(-0.6, 100_000.0, st, small, 100.0) with pytest.raises(Halt): - check_pnl(-0.6, 1000.0, st, small, 100.0) + check_pnl(-0.6, 100_000.0, st, small, 100.0) def test_new_high_resets_counter(): From 0df553fb985f296e6467371d12e34e561b085e2c Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 05:17:55 -0700 Subject: [PATCH 13/48] (fix) a fixture book reports, so the offline run still exercises the pulses MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pnl_known gate requires at least one running book that reported, but FixtureMarket marked every pair not running. An offline run therefore never advanced the equity anchor and sent the worker a `none` stimulus on every tick, so `fixture=true` stopped exercising the reward/aversive dopamine path it exists to validate. A fixture stands in for a full book that reports: every pair is now marked running and reported, and the aggregate is split across them so the per-pair figures sum to it. Nothing can be applied from there regardless — `apply` refuses. The gate itself moves out of the loop into `market.pnl_is_known`, so the rule is named, documented and tested rather than re-derived inline: no running book is silence, a running book that has not reported is silence, and one silent book among several makes the combined figure unusable. Offline: 10 observations now produce reward, aversive and none stimuli, with PAM11 and PPL101 spiking on their pulses. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XoYbpk9wCJBNcSjz4TMJWG --- agents/market_making_fly/flybrain/market.py | 40 ++++++++++++++++-- .../market_making_fly/routines/fly_brain.py | 12 +++--- .../tests/test_fly_market.py | 42 ++++++++++++++++++- 3 files changed, 84 insertions(+), 10 deletions(-) diff --git a/agents/market_making_fly/flybrain/market.py b/agents/market_making_fly/flybrain/market.py index 3d64b035d..3ff8325ea 100644 --- a/agents/market_making_fly/flybrain/market.py +++ b/agents/market_making_fly/flybrain/market.py @@ -290,11 +290,32 @@ async def fresh_mid(self, pair: str) -> float: async def equity( self, pairs: list[str], carry: dict[str, dict] | None = None ) -> tuple[float, float, dict, dict]: - # Swings so both pulses fire, drifts up so new highs recur, and stays - # within ±2 bp of volume so the loss-rate breaker is not exercised here. + """A fixture stands in for a full book that reports. + + Every pair is marked running and reported on purpose: an offline run + exists to exercise the reward/aversive path end to end, and a book that + never reports would pin the stimulus to ``none`` (see ``pnl_is_known``). + Nothing is applied from here either way — ``apply`` refuses. + + The aggregate swings so both pulses fire, drifts up so new highs recur, + and stays within ±2 bp of volume so the loss-rate breaker is not + exercised here; each pair carries an equal share of it. + """ net = 2.0 * math.sin(self.tick * 1.3) + 0.05 * self.tick volume = 10_000.0 * (self.tick + 1) - return net, volume, {p: {"running": False, "fixture": True} for p in pairs}, {} + share_net, share_volume = net / len(pairs), volume / len(pairs) + per_pair = { + p: { + "running": True, + "reported": True, + "fixture": True, + "net": share_net, + "volume": share_volume, + } + for p in pairs + } + carry = {p: {"net": share_net, "volume": share_volume} for p in pairs} + return net, volume, per_pair, carry async def available_usd(self) -> float: return 1e9 @@ -306,6 +327,19 @@ async def stop_bot(self, pair: str) -> bool: return False +def pnl_is_known(per_pair: dict) -> bool: + """True when every running book reported its P&L this tick. + + The reinforcement pulse, the equity anchor and the financial breakers all + hang off this: an unreported book is silence, not a result. No running book + at all is also silence — a shadow run with no bot deployed has nothing to + learn from. A market that stands in for a full book (the fixture) must + therefore report, or it exercises none of that path. + """ + running = [info for info in per_pair.values() if info.get("running")] + return bool(running) and all(info.get("reported") for info in running) + + def required_collateral(specs: list[MarketSpec]) -> float: """Margin the three books could need at their inventory cap.""" return sum(s.total_amount_quote * s.max_base_pct / s.leverage for s in specs) diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index f00f62cf0..81fac8234 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -66,7 +66,12 @@ record_apply, resume, ) -from flybrain.market import FixtureMarket, LiveMarket, required_collateral +from flybrain.market import ( + FixtureMarket, + LiveMarket, + pnl_is_known, + required_collateral, +) from flybrain.naming import pair_names, parse_pairs from flybrain.posture import MarketSpec, build_config, config_diff from flybrain.reinforcement import reinforcement @@ -370,10 +375,7 @@ async def pace() -> None: equity, volume, per_pair, pnl_carry = await market.equity( pairs, pnl_carry ) - running_bots = [p for p, i in per_pair.items() if i.get("running")] - pnl_known = bool(running_bots) and all( - per_pair[p].get("reported") for p in running_bots - ) + pnl_known = pnl_is_known(per_pair) if anchor is None or not pnl_known: kind, delta = "none", 0.0 else: diff --git a/agents/market_making_fly/tests/test_fly_market.py b/agents/market_making_fly/tests/test_fly_market.py index 07cb001cb..1053b8299 100644 --- a/agents/market_making_fly/tests/test_fly_market.py +++ b/agents/market_making_fly/tests/test_fly_market.py @@ -8,6 +8,7 @@ Book, FixtureMarket, normalize_candle_payload, + pnl_is_known, required_collateral, ) from flybrain.posture import MarketSpec @@ -29,9 +30,46 @@ def test_fixture_equity_stays_inside_the_breaker(): m = FixtureMarket(PAIRS, 72) for tick in range(60): m.tick = tick - net, volume, per_pair, _ = asyncio.run(m.equity(PAIRS)) + net, volume, per_pair, carry = asyncio.run(m.equity(PAIRS)) assert abs(net / volume) * 1e4 < 5 - assert per_pair["XYZ:A-USD"]["running"] is False + assert sum(i["net"] for i in per_pair.values()) == pytest.approx(net) + assert sum(i["volume"] for i in per_pair.values()) == pytest.approx(volume) + assert set(carry) == set(PAIRS) + + +def test_fixture_reports_so_an_offline_run_exercises_the_pulses(): + """A fixture book that did not report would pin the stimulus to ``none`` + and never advance the anchor, which is the whole point of the offline run.""" + m = FixtureMarket(PAIRS, 72) + _, _, per_pair, _ = asyncio.run(m.equity(PAIRS)) + assert all(i["running"] and i["reported"] for i in per_pair.values()) + assert pnl_is_known(per_pair) + # and the swing really does cross the deadband both ways + nets = [] + for tick in range(12): + m.tick = tick + net, _, _, _ = asyncio.run(m.equity(PAIRS)) + nets.append(net) + deltas = [b - a for a, b in zip(nets, nets[1:])] + assert max(deltas) > 0.02 and min(deltas) < -0.02 + + +@pytest.mark.parametrize( + "per_pair,expected", + [ + ({}, False), # no book at all is silence, not a result + ({"a": {"running": False}}, False), + ({"a": {"running": True}}, False), # running but no report yet + ({"a": {"running": True, "reported": True}}, True), + ( + {"a": {"running": True, "reported": True}, "b": {"running": True}}, + False, # one silent book makes the combined figure unusable + ), + ({"a": {"running": True, "reported": True}, "b": {"running": False}}, True), + ], +) +def test_pnl_is_known(per_pair, expected): + assert pnl_is_known(per_pair) is expected def test_fixture_never_applies(): From 5c2ba6d12fd16e383969606f5005d9e331b1bc57 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 05:31:06 -0700 Subject: [PATCH 14/48] (fix) second review round: stacked suffixes, resume compatibility, redeploys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four findings from review plus one of my own, all reproduced before fixing. Bot names: a redeploy that hands the running instance name back stacks another -YYYYMMDD-HHMMSS suffix, and the matcher accepted only one, so the safety path could not find the bot it may need to stop. It now accepts any number of suffixes and still refuses a near miss or an ambiguous match. Resume compatibility: dropping every source hash from the signature let a run resume onto code that reads its persisted state differently. That is not theoretical — session_high_net went from 0.0 to None inside this PR, and a run resumed across that change read its first reported figure as a drawdown from zero and halted. The persisted shape now carries STATE_VERSION, compared directly rather than through the signature, since re-signing the recorded provenance would rewrite it and make the check inert. Resume accounting: sizing, leverage, allocation and venue all denominate the anchor, the session high and the carried P&L that a resume restores, so changing one while restoring the other half of the books judged one deployment's numbers by another's thresholds. They are signed again; only cadence stays free to change mid-lineage. Stale P&L reaching the pulses was already closed by the pnl_known gate; a test now pins that an unreported book is silence rather than a result. Mine: a redeployed controller reports from zero, which read as the whole previous deployment's P&L evaporating — a 5 USD false aversive pulse in the case reproduced, and a stale high the fresh book would then be halted against. Volume only accumulates within a controller instance, so a drop detects the restart: no pulse that tick, re-anchor, and the breakers forget their high-water marks. A real loss on growing volume still pulses. 102 agent tests, 4840 repo tests, the full-connectome test, and an offline run that exercises both dopamine pulses. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XoYbpk9wCJBNcSjz4TMJWG --- agents/market_making_fly/flybrain/guard.py | 10 ++++ agents/market_making_fly/flybrain/market.py | 31 +++++++++-- .../market_making_fly/flybrain/run_state.py | 45 +++++++++++---- .../market_making_fly/routines/fly_brain.py | 12 +++- .../tests/test_fly_review_fixes.py | 55 ++++++++++++++++++- .../tests/test_fly_run_state.py | 55 +++++++++++++++---- 6 files changed, 177 insertions(+), 31 deletions(-) diff --git a/agents/market_making_fly/flybrain/guard.py b/agents/market_making_fly/flybrain/guard.py index 219fad624..f2e19e392 100644 --- a/agents/market_making_fly/flybrain/guard.py +++ b/agents/market_making_fly/flybrain/guard.py @@ -171,6 +171,16 @@ def record_apply( raise Halt(state.halted, False) +def rebase(state: GuardState) -> None: + """Forget the session's high-water marks after a book restarted. + + A redeployed controller reports from zero. Measured against the previous + deployment's high, it would be halted for a drawdown that never happened. + """ + state.session_high_net = None + state.ticks_since_high = 0 + + def check_pnl( total_net: float, volume: float, diff --git a/agents/market_making_fly/flybrain/market.py b/agents/market_making_fly/flybrain/market.py index 3ff8325ea..d327f3cd3 100644 --- a/agents/market_making_fly/flybrain/market.py +++ b/agents/market_making_fly/flybrain/market.py @@ -19,7 +19,9 @@ from flybrain.reinforcement import controller_net HL_INFO_URL = "https://api.hyperliquid.xyz/info" -_SUFFIXED = re.compile(r"-\d{8}-\d{6}") +# The deploy tool appends -YYYYMMDD-HHMMSS, and a redeploy that hands the +# running instance name back in stacks another one. +_SUFFIXED = re.compile(r"(?:-\d{8}-\d{6})+") QUOTE_TOKENS = ("USD", "USDC") @@ -123,14 +125,17 @@ async def bots(self) -> dict: @staticmethod def find_bot(bots: dict, bot_name: str) -> tuple[str | None, dict | None]: """The deploy tool suffixes instance names with ``-YYYYMMDD-HHMMSS``, - so ``orcl-fly`` runs as ``orcl-fly-20260913-055821``. Match the exact - name or that suffix form; refuse an ambiguous match.""" + so ``orcl-fly`` runs as ``orcl-fly-20260913-055821``, and a redeploy + that hands the instance name back stacks another suffix. Match the + exact name or any number of those suffixes, and refuse an ambiguous + match: stopping the wrong bot is worse than stopping none.""" matches = [ name for name in bots if name == bot_name - or _SUFFIXED.fullmatch(name[len(bot_name) :]) - and name.startswith(bot_name) + or ( + name.startswith(bot_name) and _SUFFIXED.fullmatch(name[len(bot_name) :]) + ) ] if len(matches) > 1: raise RuntimeError(f"several bots match {bot_name!r}: {sorted(matches)}") @@ -188,12 +193,17 @@ async def equity( pair_volume = float(inner.get("volume_traded", 0) or 0) net += pair_net volume += pair_volume + previous = carry.get(pair) per_pair[pair] = { "running": True, "reported": True, "net": pair_net, "volume": pair_volume, } + if previous and pair_volume < previous["volume"]: + # Volume only accumulates within a controller instance, so this + # is a fresh one: a redeploy, not a collapse in P&L. + per_pair[pair]["restarted"] = True carry[pair] = {"net": pair_net, "volume": pair_volume} if not math.isfinite(net) or not math.isfinite(volume): raise RuntimeError("Nonfinite bot performance") @@ -327,6 +337,17 @@ async def stop_bot(self, pair: str) -> bool: return False +def book_restarted(per_pair: dict) -> list[str]: + """Books whose reported volume went backwards since the last tick. + + Volume only accumulates within one controller instance, so a drop means a + fresh instance reporting from zero — a redeploy. Its P&L is not a loss of + the difference, and the previous deployment's high-water mark is not a + height it has fallen from. + """ + return sorted(p for p, info in per_pair.items() if info.get("restarted")) + + def pnl_is_known(per_pair: dict) -> bool: """True when every running book reported its P&L this tick. diff --git a/agents/market_making_fly/flybrain/run_state.py b/agents/market_making_fly/flybrain/run_state.py index 7a347f985..85051f8c8 100644 --- a/agents/market_making_fly/flybrain/run_state.py +++ b/agents/market_making_fly/flybrain/run_state.py @@ -35,16 +35,29 @@ def source_hashes() -> dict[str, str]: # Recorded in provenance.json for the audit trail, but not part of the # signature a resume is checked against: a bug fix in the loop must not orphan # a brain lineage. Settings, decoder, guard, dataset and circuit are. -UNSIGNED_KEYS = ("source_sha256",) -# Sizing and cadence are the operator's per-deployment choices, not the -# protocol the brain lineage was formed under; they are recorded, not signed. -UNSIGNED_SETTINGS = ( - "total_amount_quote", - "leverage", - "portfolio_allocation", - "interval_sec", - "connector_name", -) +# The shape of what ``state.json`` holds, and how the code reads it back. +# BUMP THIS whenever a persisted field is added, removed or reinterpreted: +# resuming an old run on code that reads its state differently is how a fixed +# bug comes back. ``session_high_net`` going from ``0.0`` to ``None`` did +# exactly that — a resumed run read its first reported figure as a drawdown +# from zero and halted. +STATE_VERSION = 1 + +# Recorded in provenance.json for the audit trail, but not signed: a bug fix +# in the loop must not orphan a brain lineage. What the code *means* by the +# persisted state is pinned by STATE_VERSION instead. +# ``state_version`` is compared directly in check_provenance, so it is kept +# out of the signature — inside it, re-signing would overwrite it and the +# comparison would never fire. +UNSIGNED_KEYS = ("source_sha256", "state_version") + +# Cadence is the operator's to change mid-lineage: it moves no money and +# reinterprets no accounting. Everything else about the deployment is signed. +# Sizing, leverage, allocation and venue all denominate the anchor, the session +# high and the carried P&L that a resume restores, so changing one while +# restoring the other half of the books is how a run halts on a mix of two +# deployments' numbers. Changing them needs a new run_name. +UNSIGNED_SETTINGS = ("interval_sec",) def signature(provenance: dict) -> str: @@ -140,6 +153,16 @@ def check_provenance(self, provenance: dict) -> str: sig = signature(provenance) if self.provenance_path.exists(): recorded = json.loads(self.provenance_path.read_text()) + # Compared directly, never through the signature: re-signing the + # recorded provenance below rewrites whatever the current rule + # controls, which would make this check silently inert. + recorded_version = recorded.get("state_version") + if recorded_version != STATE_VERSION: + raise RuntimeError( + f"Run state is version {recorded_version}, this code reads " + f"version {STATE_VERSION}; its persisted state would be " + "reinterpreted. Use a new run_name." + ) # Re-sign what was recorded under the current rule, so a change to # which keys are signed does not itself refuse every existing run. recorded_sig = signature( @@ -153,7 +176,7 @@ def check_provenance(self, provenance: dict) -> str: return sig atomic_write_json( self.provenance_path, - {"signature": sig, **provenance}, + {"signature": sig, "state_version": STATE_VERSION, **provenance}, indent=2, default=str, ) diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index 81fac8234..fb0dda31c 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -63,12 +63,14 @@ check_pnl, check_price_move, default_max_loss, + rebase, record_apply, resume, ) from flybrain.market import ( FixtureMarket, LiveMarket, + book_restarted, pnl_is_known, required_collateral, ) @@ -362,7 +364,6 @@ async def pace() -> None: stop_reason = f"halted: {guard_state.halted}" break pair = pairs[tick % len(pairs)] - names = pair_names(pair) spec = by_pair[pair] row: dict = { "tick": tick, @@ -376,7 +377,13 @@ async def pace() -> None: pairs, pnl_carry ) pnl_known = pnl_is_known(per_pair) - if anchor is None or not pnl_known: + restarted = book_restarted(per_pair) + if restarted: + # A redeployed controller reports from zero: that is not a + # loss to punish, and its predecessor's high is not a height + # to measure it against. + rebase(guard_state) + if anchor is None or not pnl_known or restarted: kind, delta = "none", 0.0 else: kind, delta = reinforcement(equity, anchor, deadband) @@ -389,6 +396,7 @@ async def pace() -> None: "pnl_delta": delta, "stimulus": kind, "pnl_known": pnl_known, + "restarted": restarted, "bots": per_pair, } ) diff --git a/agents/market_making_fly/tests/test_fly_review_fixes.py b/agents/market_making_fly/tests/test_fly_review_fixes.py index 7c312a6cc..e1b128bc9 100644 --- a/agents/market_making_fly/tests/test_fly_review_fixes.py +++ b/agents/market_making_fly/tests/test_fly_review_fixes.py @@ -6,7 +6,7 @@ import pytest from flybrain.guard import GuardSettings, Veto, check_price_move -from flybrain.market import Book, LiveMarket +from flybrain.market import Book, LiveMarket, book_restarted, pnl_is_known class _Controllers: @@ -182,3 +182,56 @@ class Ctx: for bad in ("../../etc", "/tmp/x", "a/b"): with pytest.raises(UnsafeIdError): asyncio.run(mod.run(mod.Config(run_name=bad), Ctx())) + + +def test_a_redeployed_book_is_not_a_five_dollar_loss(): + """A fresh controller instance reports from zero. Without the restart + check that reads as the whole previous deployment's P&L evaporating: a + large false aversive pulse, and a stale high the new book is measured + against.""" + from flybrain.guard import GuardSettings, GuardState, Halt, check_pnl, rebase + from flybrain.market import book_restarted + + traded = { + "orcl-fly-20260913-055821": {"performance": {"orcl_fly_mm": _perf(5.0, 7000.0)}} + } + net, _, per_pair, carry = asyncio.run(_market(traded).equity(["XYZ:ORCL-USD"])) + assert net == 5.0 and not book_restarted(per_pair) + + guard, settings = GuardState(), GuardSettings(loss_no_new_high_ticks=3) + check_pnl(net, 7000.0, guard, settings, 100.0) + assert guard.session_high_net == 5.0 + + # the operator redeploys; the new instance reports from zero + fresh = { + "orcl-fly-20260913-071220": {"performance": {"orcl_fly_mm": _perf(0.0, 0.0)}} + } + net2, _, per_pair2, _ = asyncio.run(_market(fresh).equity(["XYZ:ORCL-USD"], carry)) + assert book_restarted(per_pair2) == ["XYZ:ORCL-USD"] + + # rebased, the fresh book sets its own high instead of counting down to a halt + rebase(guard) + for _ in range(5): + check_pnl(net2, 0.0, guard, settings, 100.0) + assert guard.session_high_net == 0.0 + + # without the rebase it would have halted against the old deployment's high + stale = GuardState(session_high_net=5.0) + with pytest.raises(Halt): + for _ in range(4): + check_pnl(0.0, 0.0, stale, settings, 100.0) + + +def test_growing_volume_is_not_a_restart(): + traded = { + "orcl-fly-20260913-055821": {"performance": {"orcl_fly_mm": _perf(5.0, 7000.0)}} + } + _, _, _, carry = asyncio.run(_market(traded).equity(["XYZ:ORCL-USD"])) + more = { + "orcl-fly-20260913-055821": { + "performance": {"orcl_fly_mm": _perf(-2.0, 9000.0)} + } + } + _, _, per_pair, _ = asyncio.run(_market(more).equity(["XYZ:ORCL-USD"], carry)) + # P&L fell hard, but volume grew: a real loss, and it must still pulse + assert not book_restarted(per_pair) and pnl_is_known(per_pair) diff --git a/agents/market_making_fly/tests/test_fly_run_state.py b/agents/market_making_fly/tests/test_fly_run_state.py index 4480b9f58..c92b04534 100644 --- a/agents/market_making_fly/tests/test_fly_run_state.py +++ b/agents/market_making_fly/tests/test_fly_run_state.py @@ -68,23 +68,54 @@ def test_provenance_refuses_changed_protocol(tmp_path): assert run.check_provenance(prov) == sig -def test_sizing_settings_are_recorded_but_not_signed(tmp_path): +def test_cadence_may_change_but_the_books_own_terms_may_not(tmp_path): + """A resume restores the anchor, the session high and the carried P&L. + Anything that denominates those must be signed, or a run judges one + deployment's numbers by another's thresholds.""" run = RunDir(tmp_path / "run") - prov = {"settings": {"neural_ms": 500, "total_amount_quote": 200, "leverage": 3}} - sig = run.check_provenance(prov) - resized = { + base = { "settings": { "neural_ms": 500, - "total_amount_quote": 500, - "leverage": 5, - "portfolio_allocation": 1.0, + "interval_sec": 60, + "total_amount_quote": 200, + "leverage": 3, + "portfolio_allocation": 0.2, + "connector_name": "hyperliquid_perpetual", } } - assert run.check_provenance(resized) == sig - with pytest.raises(RuntimeError): - run.check_provenance( - {"settings": {"neural_ms": 700, "total_amount_quote": 200}} - ) + sig = run.check_provenance(base) + # cadence moves no money and reinterprets no accounting + slower = {"settings": {**base["settings"], "interval_sec": 300}} + assert run.check_provenance(slower) == sig + for changed in ( + {"total_amount_quote": 1000}, + {"leverage": 5}, + {"portfolio_allocation": 1.0}, + {"connector_name": "binance_perpetual"}, + {"neural_ms": 700}, + ): + with pytest.raises(RuntimeError): + run.check_provenance({"settings": {**base["settings"], **changed}}) + + +def test_state_version_is_signed(tmp_path): + """A persisted-state change must orphan the run rather than let old state + be reinterpreted: session_high_net 0.0 -> None already proved why.""" + import flybrain.run_state as run_state + + run = RunDir(tmp_path / "run") + prov = {"settings": {"neural_ms": 500}} + sig = run.check_provenance(prov) + recorded = json.loads(run.provenance_path.read_text()) + assert recorded["state_version"] == run_state.STATE_VERSION + original = run_state.STATE_VERSION + try: + run_state.STATE_VERSION = original + 1 + with pytest.raises(RuntimeError): + run.check_provenance(prov) + finally: + run_state.STATE_VERSION = original + assert run.check_provenance(prov) == sig def test_source_hashes_cover_the_package(): From 2128d0e7483680ff40fd6f5dd72bccbd9a5e9829 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 06:27:23 -0700 Subject: [PATCH 15/48] (feat) the fly quotes any CLOB market, spot or perp MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The decoder, the chart and the dopamine feedback never cared which venue they were looking at — a candle chart is a candle chart. Only three things did, and all three bear on money, so they now live in flybrain/venue.py instead of being assumed: Spot or perp, decided by the connector's `_perpetual` suffix. Leverage applies only to a perp; on spot it must be 1 and position_mode is not sent at all (pmm_mister already skips that check on spot). A connector declared as the type it is not is refused rather than silently reinterpreted. The maker fee, from which the take-profit floor follows. Spot fees run three to five times perp fees on the same exchange, so a take-profit that earns comfortably on a perp loses money on spot — and loses it silently, because the bot fills happily and bleeds the difference. Binance perp gives an 8.8 bp floor, Binance spot 33 bp. An unknown venue defaults deliberately wide, since a floor set too low loses money without saying so while one set too high only costs fills. maker_fee_bps overrides it. The top of book, which now comes from hummingbot-api for every connector it serves. HIP-3 pairs keep the Hyperliquid l2Book fallback because that endpoint 500s on them, and a test pins that the generic path is not even attempted for them. Pair grammar is BASE-QUOTE, or ISSUER:TOKEN-QUOTE on HIP-3. Names derive from the whole pair (sol-usdt-fly, xyz-orcl-usd-fly) rather than the base token: BTC-USDT and BTC-USDC were otherwise one bot, and the fly would have read one book's P&L while updating the other's config. parse_pairs refuses a collision. Collateral asks for the quote assets the books are actually denominated in rather than assuming a venue's collateral token. 123 agent tests, 4863 repo tests, and an offline run on two Binance spot pairs that exercises both dopamine pulses at leverage 1 with a 33 bp take-profit. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XoYbpk9wCJBNcSjz4TMJWG --- agents/market_making_fly/AGENT.md | 36 +++++-- agents/market_making_fly/flybrain/market.py | 64 +++++++++--- agents/market_making_fly/flybrain/naming.py | 79 +++++++++++---- agents/market_making_fly/flybrain/posture.py | 59 ++++++++--- agents/market_making_fly/flybrain/venue.py | 98 +++++++++++++++++++ .../market_making_fly/routines/fly_brain.py | 29 ++++-- .../skills/fly_mm_deploy/SKILL.md | 46 ++++++--- .../strategies/fly_hip3_operator/strategy.md | 13 ++- .../tests/test_fly_market.py | 64 ++++++++++++ .../tests/test_fly_naming.py | 41 ++++++-- .../tests/test_fly_posture.py | 55 ++++++++--- .../tests/test_fly_review_fixes.py | 51 ++++++---- .../market_making_fly/tests/test_fly_venue.py | 43 ++++++++ docs/market_making_fly_design.md | 40 +++++++- 14 files changed, 590 insertions(+), 128 deletions(-) create mode 100644 agents/market_making_fly/flybrain/venue.py create mode 100644 agents/market_making_fly/tests/test_fly_venue.py diff --git a/agents/market_making_fly/AGENT.md b/agents/market_making_fly/AGENT.md index 5f6d0593b..8ac7d2c46 100644 --- a/agents/market_making_fly/AGENT.md +++ b/agents/market_making_fly/AGENT.md @@ -2,7 +2,7 @@ name: Market Making Fly description: Market maker whose regime, spread width and reference-price lean are decoded from a simulated fly connectome watching the chart, with P&L fed back as dopamine. - Operates pmm_mister on Hyperliquid HIP-3 perps. + Operates pmm_mister on any CLOB spot or perp market, including Hyperliquid HIP-3. agent_key: claude-acp:sonnet tools: - get_prices @@ -22,7 +22,7 @@ tools: - manage_memory - manage_skill - run_code -when_to_consult: When the user asks what the fly sees or thinks about a HIP-3 market, +when_to_consult: When the user asks what the fly sees or thinks about a market, why it widened or leaned, whether it is learning, or wants the fly market maker deployed, started, stopped or rotated — use consult for questions and delegate for a deployment. @@ -52,8 +52,9 @@ floor, loss stop, loss-rate breaker, apply cooldown, closed books, collateral) c veto or halt, and it never substitutes a posture either. ## What you handle -- Deploying the fly market maker end-to-end on `n_markets` (1–3) HIP-3 pairs (`fly_mm_deploy`); - the count comes from the strategy config or the task, default 3 +- Deploying the fly market maker end-to-end on `n_markets` (1–3) pairs on any CLOB + venue, spot or perp (`fly_mm_deploy`); the count comes from the strategy config or + the task, default 3 - Starting `fly_brain` in shadow, then live; stopping it; rotating a market slot - Reading `fly_status` and explaining a posture in market-making terms - Reporting bot health with `mm_bot_report` / `mm_dashboard` @@ -61,7 +62,9 @@ veto or halt, and it never substitutes a posture either. ## What you do not handle - Choosing spreads, skew or regime yourself while the fly runs -- Non-HIP-3 venues (use Market Making Expert) +- Venues without a central limit order book — the fly reads a book and quotes + two sides, so an AMM or a swap route is Market Making Expert's or the LP + agent's job, not yours - Claims that the fly "understands" the market or has learned to trade — it has not been shown to; see the caveats below @@ -90,8 +93,25 @@ manage_skill(action="read", name="fly_mm_deploy") | `fly_status` | Latest posture per pair, last observation, guard state, memory stats | | `mm_dashboard`, `mm_bot_report` | Inventory, positions, P&L, errors | -Naming is derived from the pair: `XYZ:DRAM-USD` → bot `dram-fly`, config `dram_fly_mm`. -`fly_brain` reads P&L from exactly those names, so deploy with them. +Naming is derived from the **whole** pair, so two markets on one token never collide: +`SOL-USDT` → bot `sol-usdt-fly`, config `sol_usdt_fly_mm`; `XYZ:ORCL-USD` → +`xyz-orcl-usd-fly` / `xyz_orcl_usd_fly_mm`. `fly_brain` reads P&L from exactly those +names, so deploy with them. + +## Spot or perp — settled by the connector, and it matters + +A `_perpetual` suffix means perp; anything else is spot. Two things follow, both +enforced in code rather than left to judgment: + +- **Leverage** applies only to a perp. On spot it must be 1, and `position_mode` + is not sent at all. +- **The fee floor** is derived from the venue's maker fee, and spot fees run three + to five times perp fees on the same exchange. Binance perp is 2 bp a side, so the + take-profit floor is 8.8 bp; Binance spot is 7.5 bp a side, so the floor is 33 bp. + A take-profit that earns comfortably on a perp loses money on spot, and it loses + it silently — the bot fills happily and bleeds the difference. An unknown venue + defaults to a deliberately wide 10 bp spot / 2.5 bp perp. **Pass the exchange's + real maker fee as `maker_fee_bps` whenever you know it.** ## How to read a posture - `regime`: pause > volatile > trending_up/down > quiet > ranging, from z-scored channels @@ -105,7 +125,7 @@ Naming is derived from the pair: `XYZ:DRAM-USD` → bot `dram-fly`, config `dram - Applies happen only on a regime change, a spread-× move ≥ 0.15 or a lean move ≥ 0.5 bp, and at most once per 5 minutes per pair -## HIP-3 facts you must keep (from the HIP-3 operator playbook) +## HIP-3 facts you must keep when the venue is Hyperliquid HIP-3 - Uppercase pair with issuer prefix (`XYZ:DRAM-USD`); lowercase → zero orders - Unified collateral: available USD from `get_portfolio_overview(["hyperliquid_perpetual"])` - Many markets close off-hours; the fly holds while a book is closed and stops the bot diff --git a/agents/market_making_fly/flybrain/market.py b/agents/market_making_fly/flybrain/market.py index d327f3cd3..4e26e6d3a 100644 --- a/agents/market_making_fly/flybrain/market.py +++ b/agents/market_making_fly/flybrain/market.py @@ -1,9 +1,11 @@ """What the fly loop reads from the world, and the one thing it writes. ``LiveMarket`` talks to Hummingbot (candles, bot performance, portfolio, config -updates, stop) and to Hyperliquid's public ``l2Book`` for the live book — the -hummingbot-api order-book endpoint 500s on HIP-3 pairs. ``FixtureMarket`` is a -deterministic offline stand-in for plumbing tests; it never applies anything. +updates, stop) and reads the top of book from hummingbot-api, which serves any +CLOB connector it supports. HIP-3 pairs are the one exception: that endpoint +500s on them, so those fall back to Hyperliquid's public ``l2Book``. +``FixtureMarket`` is a deterministic offline stand-in for plumbing tests; it +never applies anything. """ from __future__ import annotations @@ -22,7 +24,6 @@ # The deploy tool appends -YYYYMMDD-HHMMSS, and a redeploy that hands the # running instance name back in stacks another one. _SUFFIXED = re.compile(r"(?:-\d{8}-\d{6})+") -QUOTE_TOKENS = ("USD", "USDC") @dataclass(frozen=True) @@ -91,8 +92,31 @@ def __init__( self.candle_interval = candle_interval self.n_candles = n_candles - async def observe(self, pair: str) -> Observation: + async def book(self, pair: str) -> Book: + """Top of book, from whichever source serves this market. + + hummingbot-api covers every CLOB connector it supports. HIP-3 pairs are + the exception: its order-book endpoint 500s on them, so those go + straight to Hyperliquid's public one. + """ names = pair_names(pair) + if names.hl_coin and "hyperliquid" in self.connector_name.lower(): + async with aiohttp.ClientSession() as session: + return await fetch_l2_book(session, names.hl_coin) + raw = await self.client.market_data.get_order_book( + self.connector_name, pair, depth=1 + ) + if not isinstance(raw, dict): + raise RuntimeError(f"{pair}: unexpected order book payload") + bids, asks = raw.get("bids") or [], raw.get("asks") or [] + if not bids or not asks: + return Book(None, None) # closed / empty book, a real state + bid, ask = float(bids[0][0]), float(asks[0][0]) + if not (math.isfinite(bid) and math.isfinite(ask)) or bid <= 0 or ask < bid: + raise RuntimeError(f"{pair}: invalid top of book bid={bid!r} ask={ask!r}") + return Book(bid, ask) + + async def observe(self, pair: str) -> Observation: candles = normalize_candle_payload( await self.client.market_data.get_candles( self.connector_name, @@ -101,8 +125,7 @@ async def observe(self, pair: str) -> Observation: max_records=self.n_candles, ) ) - async with aiohttp.ClientSession() as session: - book = await fetch_l2_book(session, names.coin) + book = await self.book(pair) if not book.open: # The chart still exists; the loop decides what a closed book means. last = float(candles[-1]["close"]) @@ -110,8 +133,7 @@ async def observe(self, pair: str) -> Observation: return Observation(pair, candles, book.bid, book.ask, True) async def fresh_mid(self, pair: str) -> float: - async with aiohttp.ClientSession() as session: - book = await fetch_l2_book(session, pair_names(pair).coin) + book = await self.book(pair) if not book.open: raise RuntimeError(f"{pair}: book closed at apply time") return (book.bid + book.ask) / 2 @@ -209,7 +231,14 @@ async def equity( raise RuntimeError("Nonfinite bot performance") return net, volume, per_pair, carry - async def available_usd(self) -> float: + async def available_quote(self, quote_tokens: set[str]) -> float: + """Available balance in the quote assets these books trade against. + + A perp draws margin from its collateral asset; a spot book spends the + quote outright. Either way the figure that matters is what is free in + the asset the pair is denominated in, so the caller names it rather + than this assuming a venue's collateral token. + """ state = await self.client.portfolio.get_portfolio_state() if not isinstance(state, dict): raise RuntimeError("Portfolio state unavailable") @@ -223,7 +252,7 @@ async def available_usd(self) -> float: continue seen = True for token in tokens: - if isinstance(token, dict) and token.get("token") in QUOTE_TOKENS: + if isinstance(token, dict) and token.get("token") in quote_tokens: units = float(token.get("units", 0) or 0) total += float(token.get("available_units", units) or 0) if not seen: @@ -327,7 +356,7 @@ async def equity( carry = {p: {"net": share_net, "volume": share_volume} for p in pairs} return net, volume, per_pair, carry - async def available_usd(self) -> float: + async def available_quote(self, quote_tokens: set[str]) -> float: return 1e9 async def apply(self, pair: str, config: dict) -> None: @@ -362,9 +391,18 @@ def pnl_is_known(per_pair: dict) -> bool: def required_collateral(specs: list[MarketSpec]) -> float: - """Margin the three books could need at their inventory cap.""" + """Quote the books could need at their inventory cap. + + On a perp that is margin, so leverage divides it. On spot leverage is 1 by + construction, and the same expression is the quote actually spent. + """ return sum(s.total_amount_quote * s.max_base_pct / s.leverage for s in specs) +def quote_tokens(specs: list[MarketSpec]) -> set[str]: + """The quote assets these books are denominated in.""" + return {pair_names(s.trading_pair).quote for s in specs} + + def now() -> float: return time.time() diff --git a/agents/market_making_fly/flybrain/naming.py b/agents/market_making_fly/flybrain/naming.py index 479fee4eb..52b1f546c 100644 --- a/agents/market_making_fly/flybrain/naming.py +++ b/agents/market_making_fly/flybrain/naming.py @@ -1,37 +1,73 @@ -"""Derived names for a HIP-3 pair: the bot, its controller config, the l2Book coin.""" +"""Derived names for a trading pair: the bot, its controller config, its book key. + +Two pair shapes are accepted, both uppercase as Hummingbot writes them: + +* ``BASE-QUOTE`` — any CLOB spot or perp market, e.g. ``SOL-USDT`` +* ``ISSUER:TOKEN-QUOTE`` — a Hyperliquid HIP-3 market, e.g. ``XYZ:ORCL-USD`` + +The derived slug carries the **whole** pair, not just its base token. Two +markets on the same token are otherwise the same bot: ``BTC-USDT`` and +``BTC-USDC`` both quoting under one name would have the fly read one book's +P&L for the other and update the wrong controller. +""" from __future__ import annotations +import re from dataclasses import dataclass +# Docker container names must start alphanumeric and contain only +# [a-zA-Z0-9_.-]; the slug feeds one, so it is restricted to that. +_SAFE_SLUG = re.compile(r"[a-z0-9][a-z0-9.-]*") + @dataclass(frozen=True) class Names: - pair: str # XYZ:DRAM-USD (Hummingbot trading_pair, uppercase) - issuer: str # xyz - token: str # DRAM - coin: str # xyz:DRAM (Hyperliquid l2Book coin: lowercase issuer, uppercase token) - bot_name: str # dram-fly - config_name: str # dram_fly_mm + pair: str # SOL-USDT | XYZ:ORCL-USD + base: str # SOL | ORCL + quote: str # USDT | USD + issuer: str # "" | xyz + slug: str # sol-usdt | xyz-orcl-usd + bot_name: str # sol-usdt-fly | xyz-orcl-usd-fly + config_name: str # sol_usdt_fly_mm | xyz_orcl_usd_fly_mm + hl_coin: str # "" | xyz:ORCL (Hyperliquid l2Book key) def pair_names(pair: str) -> Names: - if ":" not in pair or not pair.endswith("-USD"): - raise ValueError(f"HIP-3 pair must look like ISSUER:TOKEN-USD, got {pair!r}") + """Parse a pair into everything derived from it, or refuse.""" + if not isinstance(pair, str) or not pair: + raise ValueError("trading pair is required") if pair != pair.upper(): - raise ValueError(f"HIP-3 pair must be uppercase, got {pair!r}") - issuer, rest = pair.split(":", 1) - token = rest[: -len("-USD")] - if not issuer or not token: - raise ValueError(f"HIP-3 pair must look like ISSUER:TOKEN-USD, got {pair!r}") - base = token.lower() + raise ValueError(f"trading pair must be uppercase, got {pair!r}") + if "-" not in pair: + raise ValueError( + f"trading pair must look like BASE-QUOTE or ISSUER:TOKEN-QUOTE, got {pair!r}" + ) + head, quote = pair.rsplit("-", 1) + issuer = "" + base = head + if ":" in head: + issuer, base = head.split(":", 1) + if ":" in base: + raise ValueError(f"trading pair has more than one issuer prefix: {pair!r}") + if not base or not quote or (":" in head and not issuer): + raise ValueError( + f"trading pair must look like BASE-QUOTE or ISSUER:TOKEN-QUOTE, got {pair!r}" + ) + slug = pair.lower().replace(":", "-").replace("-", "-") + if not _SAFE_SLUG.fullmatch(slug): + raise ValueError(f"trading pair {pair!r} does not make a usable bot name") return Names( pair=pair, + base=base, + quote=quote, issuer=issuer.lower(), - token=token, - coin=f"{issuer.lower()}:{token}", - bot_name=f"{base}-fly", - config_name=f"{base}_fly_mm", + slug=slug, + bot_name=f"{slug}-fly", + config_name=f"{slug.replace('-', '_').replace('.', '_')}_fly_mm", + # Hyperliquid keys a HIP-3 book by lowercase issuer + uppercase token, + # with no quote suffix. A non-HIP-3 pair has no such key. + hl_coin=f"{issuer.lower()}:{base}" if issuer else "", ) @@ -43,6 +79,7 @@ def parse_pairs(value: str, limit: int = 3) -> list[str]: raise ValueError(f"At most {limit} pairs, got {len(pairs)}") if len(set(pairs)) != len(pairs): raise ValueError("Duplicate pair") - for pair in pairs: - pair_names(pair) + slugs = [pair_names(p).slug for p in pairs] + if len(set(slugs)) != len(slugs): + raise ValueError(f"Pairs collide on their derived bot name: {sorted(slugs)}") return pairs diff --git a/agents/market_making_fly/flybrain/posture.py b/agents/market_making_fly/flybrain/posture.py index 1faf79097..d4f528214 100644 --- a/agents/market_making_fly/flybrain/posture.py +++ b/agents/market_making_fly/flybrain/posture.py @@ -1,7 +1,7 @@ """Turn a posture into a full ``pmm_mister`` config. -The base is the HIP-3 operator's bounded defaults; the posture multiplies the -spreads and leans them. Every money-relevant floor lives here, in code: +The base is a bounded default set; the posture multiplies the spreads and leans +them. Every money-relevant floor lives here, in code: * no spread level below ``min_spread_bps``; * ``take_profit`` never below ``2.2 ×`` the round-trip maker fee, and never @@ -9,8 +9,12 @@ mandatory); * the reference-price lean is capped at half the first-level spread. -Inventory bands, allocation, leverage cap and the global stop loss are fixed by -the HIP-3 playbook and are not the fly's to move. +Inventory bands, allocation, leverage cap and the global stop loss are the +operator's, not the fly's to move. + +The spec works on any CLOB market, spot or perp. What the venue decides — +whether leverage applies at all, and what a round trip costs — comes from +``venue.py``; what the fly decides is only ever the posture. """ from __future__ import annotations @@ -18,7 +22,9 @@ import math from dataclasses import dataclass +from flybrain import venue from flybrain.decoder import REGIMES, Posture +from flybrain.naming import pair_names # executor_refresh_time, buy/sell cooldown — Market Making Expert's table. TIMING: dict[str, tuple[int, int]] = { @@ -39,11 +45,16 @@ class MarketSpec: """What the operator settles once per deployment; the fly never changes it.""" connector_name: str - trading_pair: str # UPPERCASE issuer prefix, e.g. XYZ:DRAM-USD + trading_pair: str # BASE-QUOTE, or ISSUER:TOKEN-QUOTE on HIP-3 total_amount_quote: float - picked_spread_bps: float # the scanner's spread for this market - leverage: int = 3 - maker_fee_bps: float = 1.3 # HIP-3 all-in maker fee per side incl. builder fee + picked_spread_bps: float # the observed spread for this market + # Blank means "derive from the connector name"; see venue.resolve. + market_type: str = "" + # 1 on spot, where there is nothing to lever. + leverage: int = 1 + # 0 means "use the venue default"; pass the exchange's real figure when + # known, since the take-profit floor is derived from it. + maker_fee_bps: float = 0.0 min_spread_bps: float = 3.0 portfolio_allocation: float = 0.2 target_base_pct: float = 0.4 @@ -58,12 +69,15 @@ class MarketSpec: min_order_notional: float = 10.0 def __post_init__(self): - if self.trading_pair != self.trading_pair.upper(): - raise ValueError( - f"HIP-3 trading_pair must be uppercase, got {self.trading_pair!r}" + pair_names(self.trading_pair) # refuses anything unparseable + resolved = venue.resolve(self.connector_name, self.market_type) + object.__setattr__(self, "market_type", resolved) + if not self.maker_fee_bps: + object.__setattr__( + self, + "maker_fee_bps", + venue.default_maker_fee_bps(self.connector_name, resolved), ) - if not self.trading_pair.endswith("-USD") or ":" not in self.trading_pair: - raise ValueError("HIP-3 pair must look like ISSUER:TOKEN-USD") for name in ( "total_amount_quote", "picked_spread_bps", @@ -73,7 +87,13 @@ def __post_init__(self): value = getattr(self, name) if not math.isfinite(value) or value <= 0: raise ValueError(f"{name} must be finite and positive") - if not 1 <= self.leverage <= self.leverage_cap: + if self.is_spot: + if self.leverage != 1: + raise ValueError( + f"{self.connector_name} is a spot market: leverage must be 1, " + f"got {self.leverage}" + ) + elif not 1 <= self.leverage <= self.leverage_cap: raise ValueError(f"leverage must be within 1..{self.leverage_cap}") if not 0 < self.min_base_pct < self.target_base_pct < self.max_base_pct < 1: raise ValueError("0 < min_base < target_base < max_base < 1 required") @@ -82,6 +102,10 @@ def __post_init__(self): if self.min_order_notional < 0: raise ValueError("min_order_notional must be >= 0") + @property + def is_spot(self) -> bool: + return self.market_type == venue.SPOT + @property def order_notional(self) -> float: """Quote size of one order at neutral posture: one cycle's allocation @@ -123,7 +147,7 @@ def build_config(spec: MarketSpec, posture: Posture) -> dict: sell = [max(spec.min_spread_bps, lvl + shift) for lvl in levels] take_profit = max(take_profit_floor(spec), min(buy[0], sell[0]) * BPS) refresh, cooldown = TIMING[posture.regime] - return { + config = { "controller_type": "generic", "controller_name": "pmm_mister", "connector_name": spec.connector_name, @@ -131,7 +155,6 @@ def build_config(spec: MarketSpec, posture: Posture) -> dict: "total_amount_quote": spec.total_amount_quote, "portfolio_allocation": spec.portfolio_allocation, "leverage": spec.leverage, - "position_mode": "ONEWAY", "target_base_pct": spec.target_base_pct, "min_base_pct": spec.min_base_pct, "max_base_pct": spec.max_base_pct, @@ -150,6 +173,10 @@ def build_config(spec: MarketSpec, posture: Posture) -> dict: "global_stop_loss": spec.global_stop_loss, "manual_kill_switch": posture.regime == "pause", } + if not spec.is_spot: + # Only a perpetual has one; pmm_mister skips the check on spot. + config["position_mode"] = "ONEWAY" + return config def config_diff(old: dict | None, new: dict) -> dict: diff --git a/agents/market_making_fly/flybrain/venue.py b/agents/market_making_fly/flybrain/venue.py new file mode 100644 index 000000000..99106ab43 --- /dev/null +++ b/agents/market_making_fly/flybrain/venue.py @@ -0,0 +1,98 @@ +"""What differs between one CLOB venue and the next. + +The fly's decoder, its chart and its dopamine feedback are venue-agnostic: a +candle chart is a candle chart. Three things are not, and they all bear on +money, so they live here rather than being assumed: + +* **spot or perp** — a perp quotes on margin and carries leverage and a + position mode; a spot book quotes the inventory you actually hold. +* **the maker fee** — the take-profit floor is derived from it. Spot fees run + three to five times perp fees on the same exchange, so a take-profit that is + comfortably profitable on a perp loses money on spot. Getting this wrong is + silent: the bot fills happily and bleeds the difference. +* **where the top of book comes from** — see ``market.py``. + +The fee table below is a floor-setting default, not a quote. Defaults are +deliberately **conservative** (too high costs fills, too low loses money +without saying so), and every one of them is overridable per deployment. +""" + +from __future__ import annotations + +PERP_MARKERS = ("_perpetual", "_perp", "_futures") + +SPOT = "spot" +PERP = "perp" +MARKET_TYPES = (SPOT, PERP) + +# Maker fee per side, in basis points. Keys are matched as a prefix of the +# connector name, longest first, so `binance_perpetual` beats `binance`. +_MAKER_FEE_BPS: dict[tuple[str, str], float] = { + ("hyperliquid_perpetual", PERP): 1.3, # ~0.29 bp exchange + ~1.0 bp builder + ("hyperliquid", SPOT): 4.0, + ("binance_perpetual", PERP): 2.0, # 0.02 % + ("binance", SPOT): 7.5, # 0.075 % + ("gate_io_perpetual", PERP): 2.0, + ("gate_io", SPOT): 9.0, + ("okx_perpetual", PERP): 2.0, + ("okx", SPOT): 8.0, + ("kucoin_perpetual", PERP): 2.0, + ("kucoin", SPOT): 10.0, + ("backpack", SPOT): 8.0, +} + +# Used when the connector is not in the table at all. A market maker on an +# unknown venue should quote too wide rather than too tight. +_FALLBACK_FEE_BPS = {PERP: 2.5, SPOT: 10.0} + + +def market_type_for(connector_name: str) -> str: + """``perp`` when the connector names itself one, else ``spot``. + + This is the same rule Market Making Expert uses, and the same one + hummingbot follows: the `_perpetual` suffix is the contract type. + """ + if not connector_name: + raise ValueError("connector_name is required") + lowered = connector_name.lower() + return PERP if any(m in lowered for m in PERP_MARKERS) else SPOT + + +def default_maker_fee_bps(connector_name: str, market_type: str) -> float: + """A conservative per-side maker fee for this venue, in bp. + + Always overridable: pass the real figure from the exchange's fee schedule + when it is known, because everything about the take-profit floor follows + from it. + """ + if market_type not in MARKET_TYPES: + raise ValueError(f"market_type must be one of {MARKET_TYPES}") + lowered = (connector_name or "").lower() + matches = [ + (key, fee) + for (key, kind), fee in _MAKER_FEE_BPS.items() + if kind == market_type and lowered.startswith(key) + ] + if not matches: + return _FALLBACK_FEE_BPS[market_type] + # Longest prefix wins: binance_perpetual is not binance. + return max(matches, key=lambda m: len(m[0]))[1] + + +def resolve(connector_name: str, market_type: str = "") -> str: + """The market type to use: an explicit one, else derived from the connector.""" + if not market_type: + return market_type_for(connector_name) + if market_type not in MARKET_TYPES: + raise ValueError( + f"market_type must be one of {MARKET_TYPES}, got {market_type!r}" + ) + derived = market_type_for(connector_name) + if market_type != derived: + # Naming a perp connector spot (or the reverse) silently changes the + # leverage rule and the fee floor, so say so rather than proceed. + raise ValueError( + f"{connector_name!r} looks like a {derived} connector but was declared " + f"{market_type!r}; check the connector name" + ) + return market_type diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index fb0dda31c..a7009d075 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -1,6 +1,7 @@ """The fly loop: chart → connectome → posture → pmm_mister config, with P&L dopamine. -One shared brain is shown up to three HIP-3 markets in round-robin. Each tick: +One shared brain is shown up to three markets in round-robin — any CLOB spot or +perp market hummingbot-api serves. Each tick: 1. fetch candles + live book for this tick's pair; 2. read the combined net P&L of the fly's bots, turn its change since the last @@ -72,6 +73,7 @@ LiveMarket, book_restarted, pnl_is_known, + quote_tokens, required_collateral, ) from flybrain.naming import pair_names, parse_pairs @@ -98,20 +100,32 @@ class Config(BaseModel): """Fly-connectome market maker: one brain, up to three HIP-3 markets, P&L dopamine.""" pairs: str = Field( - default="XYZ:DRAM-USD,XYZ:SPCX-USD,XYZ:SMSN-USD", - description="1 to 3 uppercase HIP-3 pairs, comma-separated — this list IS the market count (bot {token}-fly, config {token}_fly_mm)", + default="XYZ:ORCL-USD", + description="1 to 3 uppercase pairs, comma-separated — BASE-QUOTE on any CLOB venue, or ISSUER:TOKEN-QUOTE on HIP-3. This list IS the market count; each pair names its own bot ({slug}-fly) and config ({slug}_fly_mm)", ) picked_spreads_bps: str = Field( default="8,8,8", description="Scanner spread per pair in bp, same order as pairs", ) connector_name: str = Field( - default="hyperliquid_perpetual", description="Connector" + default="hyperliquid_perpetual", + description="Any CLOB connector hummingbot-api serves, spot or perp (a _perpetual suffix means perp)", + ) + market_type: str = Field( + default="", + description="spot | perp; blank derives it from the connector name", + ) + maker_fee_bps: float = Field( + default=0.0, + description="Maker fee per side in bp; 0 uses a conservative venue default. The take-profit floor is derived from it, so pass the exchange's real figure when you know it", ) total_amount_quote: float = Field( default=500.0, description="Capital per pair (quote)" ) - leverage: int = Field(default=3, description="Leverage per pair (cap 5)") + leverage: int = Field( + default=1, + description="Leverage per pair (perp only, cap 5); must be 1 on spot", + ) portfolio_allocation: float = Field( default=0.2, description="Fraction of total_amount_quote quoted per cycle; each order is total × allocation / 4 and must clear the exchange minimum (10 USD on HIP-3) — one market at 200 quote needs 0.2+", @@ -180,7 +194,9 @@ def _specs(config: Config, pairs: list[str]) -> list[MarketSpec]: trading_pair=pair, total_amount_quote=config.total_amount_quote, picked_spread_bps=spread, + market_type=config.market_type, leverage=config.leverage, + maker_fee_bps=config.maker_fee_bps, portfolio_allocation=config.portfolio_allocation, ) for pair, spread in zip(pairs, spreads) @@ -496,7 +512,8 @@ async def pace() -> None: check_apply_window(guard_state, now, guard_settings) check_config(proposed, spec) check_collateral( - await market.available_usd(), required_collateral(specs) + await market.available_quote(quote_tokens(specs)), + required_collateral(specs), ) check_price_move( obs.mid, await market.fresh_mid(pair), guard_settings diff --git a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md index 55325ebda..1b656d050 100644 --- a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md +++ b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md @@ -1,8 +1,8 @@ --- name: fly_mm_deploy -description: End-to-end deployment of the fly market maker on up to three HIP-3 pairs — - scan, deploy neutral pmm_mister bots with the fly's naming, start fly_brain in shadow, - verify, and (only when told) switch to live. +description: End-to-end deployment of the fly market maker on up to three markets on + any CLOB venue, spot or perp — pick the markets, deploy neutral pmm_mister bots with + the fly's naming, start fly_brain in shadow, verify, and (only when told) go live. when_to_use: When asked to set up, deploy, launch, or restart the fly market maker, or to rotate one of its market slots. Follow it as a delegate task with no mid-flow confirmation. @@ -12,19 +12,37 @@ source: agent:market_making_fly # Fly MM Deploy -You are deploying **Market Making Fly**: one shared fly brain, up to three HIP-3 -markets, `pmm_mister` controllers. The fly decides posture; you set up the plumbing. +You are deploying **Market Making Fly**: one shared fly brain, up to three markets +on any CLOB venue, `pmm_mister` controllers. The fly decides posture; you set up +the plumbing. + +## Step 0 — Settle the venue + +- `connector_name`: any CLOB connector hummingbot-api serves. A `_perpetual` + suffix means perp, anything else is spot. +- **Spot**: `leverage` must be 1, and the fee floor is far wider — check + `maker_fee_bps` before anything else. +- **`maker_fee_bps`**: pass the exchange's real per-side maker fee when you know + it. Leaving it at 0 uses a conservative default, which quotes wider than + necessary rather than tighter than is profitable. ## Step 1 — Pick the markets +On **Hyperliquid HIP-3**, rank them: + ``` manage_routines(action="run", name="hip3_market_scanner", config={"issuer": "xyz", "min_spread_bps": 3, "max_daily_drift_pct": 3, "top_n": 5}) ``` -Take the top **`n_markets`** survivors (1–3; from `[CURRENT CONFIG]` or the task, -default 3) that have an open live book — one brain quotes them all in round-robin. Record for each: `pair` (uppercase, e.g. `XYZ:DRAM-USD`) and its **spread in -bp** — this is `picked_spreads_bps`. If fewer than one survivor, stop and report. +On **any other venue**, either take the pairs the operator named, or rank +candidates with the global `market_scanner` routine and read the spread from +`get_prices` plus the order book. + +Take the top **`n_markets`** (1-3; from `[CURRENT CONFIG]` or the task, default 3) +that have an open live book — one brain quotes them all in round-robin. Record for +each: `pair` (uppercase `BASE-QUOTE`, or `ISSUER:TOKEN-QUOTE` on HIP-3) and its +**spread in bp** — this is `picked_spreads_bps`. If none survive, stop and report. ## Step 2 — Collateral @@ -34,8 +52,10 @@ bp** — this is `picked_spreads_bps`. If fewer than one survivor, stop and repo ## Step 3 — Neutral configs (the fly's starting point) -For each pair derive `token` (e.g. `DRAM`), `bot_name = {token.lower()}-fly`, -`config_name = {token.lower()}_fly_mm`. The neutral config is exactly what +For each pair derive the slug from the **whole** pair — `SOL-USDT` → `sol-usdt`, +`XYZ:ORCL-USD` → `xyz-orcl-usd` — then `bot_name = {slug}-fly` and +`config_name = {slug}_fly_mm` (underscores). Never name a bot after the base token +alone: `BTC-USDT` and `BTC-USDC` would collide. The neutral config is exactly what `fly_brain` would apply for the `ranging` regime; build it with: ```python @@ -43,15 +63,15 @@ run_code(code=""" import sys; sys.path.insert(0, "agents/market_making_fly") from flybrain.posture import MarketSpec, build_config from flybrain.decoder import NEUTRAL -spec = MarketSpec(connector_name="hyperliquid_perpetual", trading_pair="XYZ:DRAM-USD", +spec = MarketSpec(connector_name="binance_perpetual", trading_pair="SOL-USDT", total_amount_quote=500, picked_spread_bps=8.0, leverage=3, - portfolio_allocation=0.2) + portfolio_allocation=0.2) # spot: leverage=1, and check maker_fee_bps print(build_config(spec, NEUTRAL)) """) ``` `build_config` refuses a spec whose orders would fall under the exchange minimum -(10 USD on HIP-3): each order is `total_amount_quote × portfolio_allocation / 4`. +(10 USD on HIP-3, 5-10 USD on most spot venues): each order is `total_amount_quote × portfolio_allocation / 4`. 200 quote on one market needs `portfolio_allocation` ≥ 0.2; 100 quote needs ≥ 0.4. Whatever value you use here, pass the **same** `portfolio_allocation` to `fly_brain` in Step 5 — it rebuilds every config from it. diff --git a/agents/market_making_fly/strategies/fly_hip3_operator/strategy.md b/agents/market_making_fly/strategies/fly_hip3_operator/strategy.md index 7d071126d..989e6dbb1 100644 --- a/agents/market_making_fly/strategies/fly_hip3_operator/strategy.md +++ b/agents/market_making_fly/strategies/fly_hip3_operator/strategy.md @@ -1,7 +1,8 @@ --- -name: Fly HIP-3 Operator -description: Keeps the fly market maker alive on its HIP-3 slots — bots up, fly_brain - running, halts surfaced, closed markets rotated. Never sets a posture itself. +name: Fly Operator +description: Keeps the fly market maker alive on its market slots, on any CLOB venue — + bots up, fly_brain running, halts surfaced, closed markets rotated. Never sets a + posture itself. agent_key: null skills: [] default_config: @@ -19,7 +20,7 @@ created_by: 456181693 created_at: '2026-09-12T00:00:00+00:00' --- -# Fly HIP-3 Operator +# Fly Operator You are Market Making Fly's operator loop. The fly (`fly_brain`) quotes; you keep the plumbing healthy. **You never choose spreads, lean or regime.** @@ -27,10 +28,12 @@ the plumbing healthy. **You never choose spreads, lean or regime.** ## Configuration at launch Read these from `[CURRENT CONFIG]`: -- `n_markets` (1–3, default 3): how many HIP-3 markets the fly quotes at once. The +- `n_markets` (1–3, default 3): how many markets the fly quotes at once. The one shared brain is shown that many charts in round-robin; each pair is observed every `n_markets × interval_sec`. Fewer markets means each one is seen more often. - `total_amount_quote`: capital **per market**. +- `connector_name` and `market_type`: any CLOB venue, spot or perp. On spot, + `leverage` must be 1 and the take-profit floor is several times wider. - `run_name`: the fly's run directory (brain lineage). - `mode`: `shadow` or `live`. - `trading_context`, if present, may name the pairs explicitly ("MM XYZ:DRAM-USD and diff --git a/agents/market_making_fly/tests/test_fly_market.py b/agents/market_making_fly/tests/test_fly_market.py index 1053b8299..cb932dd81 100644 --- a/agents/market_making_fly/tests/test_fly_market.py +++ b/agents/market_making_fly/tests/test_fly_market.py @@ -98,3 +98,67 @@ def test_candle_payload_shapes(): def test_required_collateral(): spec = MarketSpec("hyperliquid_perpetual", "XYZ:A-USD", 500, 8.0, leverage=2) assert required_collateral([spec, spec]) == pytest.approx(2 * 500 * 0.5 / 2) + + +class _MarketData: + """Records what the generic order-book endpoint was asked for.""" + + def __init__(self, book=None): + self.calls = [] + self.book = ( + book if book is not None else {"bids": [[99.0, 5]], "asks": [[101.0, 5]]} + ) + + async def get_order_book(self, connector, pair, depth=1): + self.calls.append((connector, pair, depth)) + return self.book + + +class _Client: + def __init__(self, book=None): + self.market_data = _MarketData(book) + + +def _live(connector, book=None): + from flybrain.market import LiveMarket + + return LiveMarket(_Client(book), connector, "5m", 72) + + +def test_a_plain_pair_reads_the_generic_order_book(): + m = _live("binance_perpetual") + book = asyncio.run(m.book("SOL-USDT")) + assert (book.bid, book.ask) == (99.0, 101.0) and book.open + assert m.client.market_data.calls == [("binance_perpetual", "SOL-USDT", 1)] + assert asyncio.run(m.fresh_mid("SOL-USDT")) == 100.0 + + +def test_an_empty_generic_book_is_closed_not_an_error(): + m = _live("binance", {"bids": [], "asks": []}) + assert not asyncio.run(m.book("SOL-USDT")).open + + +@pytest.mark.parametrize( + "book", + [ + {"bids": [[float("nan"), 1]], "asks": [[101.0, 1]]}, + {"bids": [[0.0, 1]], "asks": [[101.0, 1]]}, + {"bids": [[101.0, 1]], "asks": [[99.0, 1]]}, # crossed + ], +) +def test_a_nonsense_generic_book_raises(book): + with pytest.raises(RuntimeError): + asyncio.run(_live("binance", book).book("SOL-USDT")) + + +def test_a_hip3_pair_does_not_use_the_generic_endpoint(): + """hummingbot-api's order-book endpoint 500s on HIP-3 pairs, so those must + go to Hyperliquid's own. Proven by the generic one never being called.""" + import aiohttp + + m = _live("hyperliquid_perpetual") + try: + asyncio.run(m.book("XYZ:ORCL-USD")) + except (aiohttp.ClientError, RuntimeError, OSError): + pass # offline in CI; the point is which path was taken + assert m.client.market_data.calls == [] diff --git a/agents/market_making_fly/tests/test_fly_naming.py b/agents/market_making_fly/tests/test_fly_naming.py index 9a8ea3465..2e91df3a6 100644 --- a/agents/market_making_fly/tests/test_fly_naming.py +++ b/agents/market_making_fly/tests/test_fly_naming.py @@ -1,17 +1,40 @@ -"""Derived names for a HIP-3 pair, and the pairs list rules.""" +"""Derived names for any CLOB pair, and the pairs-list rules.""" import pytest from flybrain.naming import pair_names, parse_pairs -def test_pair_names(): - n = pair_names("XYZ:DRAM-USD") - assert (n.issuer, n.token, n.coin) == ("xyz", "DRAM", "xyz:DRAM") - assert (n.bot_name, n.config_name) == ("dram-fly", "dram_fly_mm") +def test_hip3_pair(): + n = pair_names("XYZ:ORCL-USD") + assert (n.base, n.quote, n.issuer) == ("ORCL", "USD", "xyz") + assert n.hl_coin == "xyz:ORCL" + assert (n.slug, n.bot_name, n.config_name) == ( + "xyz-orcl-usd", + "xyz-orcl-usd-fly", + "xyz_orcl_usd_fly_mm", + ) + + +def test_plain_pair(): + n = pair_names("SOL-USDT") + assert (n.base, n.quote, n.issuer) == ("SOL", "USDT", "") + assert n.hl_coin == "" # nothing to ask Hyperliquid for + assert (n.slug, n.bot_name, n.config_name) == ( + "sol-usdt", + "sol-usdt-fly", + "sol_usdt_fly_mm", + ) + + +def test_the_whole_pair_names_the_bot(): + """One token on two quotes is two markets. Naming both after the base + would point the fly at one book\'s P&L while updating the other\'s config.""" + assert pair_names("BTC-USDT").bot_name != pair_names("BTC-USDC").bot_name + assert pair_names("XYZ:ORCL-USD").bot_name != pair_names("ABC:ORCL-USD").bot_name @pytest.mark.parametrize( - "bad", ["xyz:dram-usd", "DRAM-USD", "XYZ:DRAM", "XYZ:-USD", ":DRAM-USD"] + "bad", ["sol-usdt", "SOLUSDT", "XYZ:-USD", ":SOL-USDT", "SOL-", "-USDT", ""] ) def test_bad_pairs(bad): with pytest.raises(ValueError): @@ -19,10 +42,10 @@ def test_bad_pairs(bad): def test_parse_pairs(): - assert parse_pairs(" XYZ:A-USD, XYZ:B-USD ") == ["XYZ:A-USD", "XYZ:B-USD"] + assert parse_pairs(" SOL-USDT, BTC-USDT ") == ["SOL-USDT", "BTC-USDT"] with pytest.raises(ValueError): parse_pairs("") with pytest.raises(ValueError): - parse_pairs("XYZ:A-USD,XYZ:A-USD") + parse_pairs("SOL-USDT,SOL-USDT") with pytest.raises(ValueError): - parse_pairs("XYZ:A-USD,XYZ:B-USD,XYZ:C-USD,XYZ:D-USD") + parse_pairs("A-USDT,B-USDT,C-USDT,D-USDT") diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py index a61c5fa63..31458aa11 100644 --- a/agents/market_making_fly/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -88,27 +88,52 @@ def test_every_spread_respects_min(): def test_spec_validation(): - with pytest.raises(ValueError): + with pytest.raises(ValueError): # lowercase pair MarketSpec("hyperliquid_perpetual", "xyz:dram-usd", 500, 8) - with pytest.raises(ValueError): - MarketSpec("hyperliquid_perpetual", "DRAM-USD", 500, 8) - with pytest.raises(ValueError): + with pytest.raises(ValueError): # no quote + MarketSpec("hyperliquid_perpetual", "DRAMUSD", 500, 8) + with pytest.raises(ValueError): # leverage above the cap MarketSpec("hyperliquid_perpetual", "XYZ:DRAM-USD", 500, 8, leverage=10) - with pytest.raises(ValueError): + with pytest.raises(ValueError): # no capital MarketSpec("hyperliquid_perpetual", "XYZ:DRAM-USD", 0, 8) -def test_order_size_floor(): - small = MarketSpec("hyperliquid_perpetual", "XYZ:A-USD", 200, 5.0) - assert small.order_notional == pytest.approx(10.0) - small.check_order_size() - tiny = MarketSpec("hyperliquid_perpetual", "XYZ:A-USD", 100, 5.0) - with pytest.raises(ValueError, match="at least 0.40"): - build_config(tiny, NEUTRAL) - full = MarketSpec( - "hyperliquid_perpetual", "XYZ:A-USD", 100, 5.0, portfolio_allocation=1.0 +def test_spot_and_perp_are_settled_by_the_connector(): + perp = MarketSpec("binance_perpetual", "SOL-USDT", 1000, 6.0, leverage=3) + spot = MarketSpec("binance", "SOL-USDT", 1000, 6.0) + assert (perp.market_type, spot.market_type) == ("perp", "spot") + assert not perp.is_spot and spot.is_spot + # a spot book has nothing to lever + with pytest.raises(ValueError, match="leverage must be 1"): + MarketSpec("binance", "SOL-USDT", 1000, 6.0, leverage=3) + # and a mislabelled connector is refused rather than silently reinterpreted + with pytest.raises(ValueError, match="looks like a perp"): + MarketSpec("binance_perpetual", "SOL-USDT", 1000, 6.0, market_type="spot") + + +def test_position_mode_is_a_perp_field(): + perp = build_config(MarketSpec("binance_perpetual", "SOL-USDT", 1000, 6.0), NEUTRAL) + spot = build_config(MarketSpec("binance", "SOL-USDT", 1000, 6.0), NEUTRAL) + assert perp["position_mode"] == "ONEWAY" and perp["leverage"] == 1 + assert "position_mode" not in spot + + +def test_the_fee_floor_follows_the_venue(): + """Spot fees run several times perp fees, and a take-profit that is + comfortably profitable on a perp loses money on spot.""" + perp = MarketSpec("binance_perpetual", "SOL-USDT", 1000, 6.0) + spot = MarketSpec("binance", "SOL-USDT", 1000, 6.0) + assert perp.maker_fee_bps == 2.0 and spot.maker_fee_bps == 7.5 + assert take_profit_floor(spot) > 3 * take_profit_floor(perp) + assert build_config(spot, NEUTRAL)["take_profit"] >= take_profit_floor(spot) + # an unknown venue is quoted wide, not tight + unknown = MarketSpec("some_new_dex", "SOL-USDT", 1000, 6.0) + assert unknown.maker_fee_bps == 10.0 + # and the real figure always wins + assert ( + MarketSpec("binance", "SOL-USDT", 1000, 6.0, maker_fee_bps=1.0).maker_fee_bps + == 1.0 ) - build_config(full, NEUTRAL) def test_config_diff(): diff --git a/agents/market_making_fly/tests/test_fly_review_fixes.py b/agents/market_making_fly/tests/test_fly_review_fixes.py index e1b128bc9..2f7f313bb 100644 --- a/agents/market_making_fly/tests/test_fly_review_fixes.py +++ b/agents/market_making_fly/tests/test_fly_review_fixes.py @@ -68,17 +68,26 @@ def test_price_move_rejects_nonfinite(bad): def test_find_bot_accepts_deploy_suffix_and_refuses_ambiguity(): - bots = {"orcl-fly-20260913-055821": {}, "dram-fly": {}, "orcl-flyer": {}} - assert LiveMarket.find_bot(bots, "orcl-fly")[0] == "orcl-fly-20260913-055821" - assert LiveMarket.find_bot(bots, "dram-fly")[0] == "dram-fly" - assert LiveMarket.find_bot(bots, "spcx-fly") == (None, None) + bots = { + "xyz-orcl-usd-fly-20260913-055821": {}, + "xyz-dram-usd-fly": {}, + "xyz-orcl-usd-flyer": {}, + } + assert ( + LiveMarket.find_bot(bots, "xyz-orcl-usd-fly")[0] + == "xyz-orcl-usd-fly-20260913-055821" + ) + assert LiveMarket.find_bot(bots, "xyz-dram-usd-fly")[0] == "xyz-dram-usd-fly" + assert LiveMarket.find_bot(bots, "xyz-spcx-usd-fly") == (None, None) with pytest.raises(RuntimeError): - LiveMarket.find_bot({**bots, "orcl-fly": {}}, "orcl-fly") + LiveMarket.find_bot({**bots, "xyz-orcl-usd-fly": {}}, "xyz-orcl-usd-fly") def test_equity_carries_a_vanished_bot(): running = { - "orcl-fly-20260913-055821": {"performance": {"orcl_fly_mm": _perf(-1.5, 400)}} + "xyz-orcl-usd-fly-20260913-055821": { + "performance": {"xyz_orcl_usd_fly_mm": _perf(-1.5, 400)} + } } net, volume, per_pair, carry = asyncio.run( _market(running).equity(["XYZ:ORCL-USD"]) @@ -102,7 +111,7 @@ def test_equity_carries_a_vanished_bot(): def test_equity_marks_a_running_bot_without_a_report(): - unreported = {"orcl-fly-20260913-055821": {"performance": {}}} + unreported = {"xyz-orcl-usd-fly-20260913-055821": {"performance": {}}} net, volume, per_pair, carry = asyncio.run( _market(unreported).equity( ["XYZ:ORCL-USD"], {"XYZ:ORCL-USD": {"net": 2.0, "volume": 50}} @@ -121,25 +130,25 @@ def test_equity_marks_a_running_bot_without_a_report(): def test_apply_saves_before_touching_the_live_bot(): - bots = {"orcl-fly-20260913-055821": {}} + bots = {"xyz-orcl-usd-fly-20260913-055821": {}} m = _market(bots) asyncio.run(m.apply("XYZ:ORCL-USD", {"x": 1})) assert m.client.controllers.calls == [ - ("saved", "orcl_fly_mm"), - ("live", "orcl-fly-20260913-055821", "orcl_fly_mm"), + ("saved", "xyz_orcl_usd_fly_mm"), + ("live", "xyz-orcl-usd-fly-20260913-055821", "xyz_orcl_usd_fly_mm"), ] failing = _market(bots, fail_saved=True) with pytest.raises(RuntimeError): asyncio.run(failing.apply("XYZ:ORCL-USD", {"x": 1})) - assert failing.client.controllers.calls == [("saved", "orcl_fly_mm")] + assert failing.client.controllers.calls == [("saved", "xyz_orcl_usd_fly_mm")] with pytest.raises(RuntimeError): asyncio.run(_market({}).apply("XYZ:ORCL-USD", {"x": 1})) def test_stop_bot_uses_the_running_name(): - m = _market({"orcl-fly-20260913-055821": {}}) + m = _market({"xyz-orcl-usd-fly-20260913-055821": {}}) assert asyncio.run(m.stop_bot("XYZ:ORCL-USD")) is True - assert m.client.bot_orchestration.stopped == ["orcl-fly-20260913-055821"] + assert m.client.bot_orchestration.stopped == ["xyz-orcl-usd-fly-20260913-055821"] assert asyncio.run(_market({}).stop_bot("XYZ:ORCL-USD")) is False @@ -193,7 +202,9 @@ def test_a_redeployed_book_is_not_a_five_dollar_loss(): from flybrain.market import book_restarted traded = { - "orcl-fly-20260913-055821": {"performance": {"orcl_fly_mm": _perf(5.0, 7000.0)}} + "xyz-orcl-usd-fly-20260913-055821": { + "performance": {"xyz_orcl_usd_fly_mm": _perf(5.0, 7000.0)} + } } net, _, per_pair, carry = asyncio.run(_market(traded).equity(["XYZ:ORCL-USD"])) assert net == 5.0 and not book_restarted(per_pair) @@ -204,7 +215,9 @@ def test_a_redeployed_book_is_not_a_five_dollar_loss(): # the operator redeploys; the new instance reports from zero fresh = { - "orcl-fly-20260913-071220": {"performance": {"orcl_fly_mm": _perf(0.0, 0.0)}} + "xyz-orcl-usd-fly-20260913-071220": { + "performance": {"xyz_orcl_usd_fly_mm": _perf(0.0, 0.0)} + } } net2, _, per_pair2, _ = asyncio.run(_market(fresh).equity(["XYZ:ORCL-USD"], carry)) assert book_restarted(per_pair2) == ["XYZ:ORCL-USD"] @@ -224,12 +237,14 @@ def test_a_redeployed_book_is_not_a_five_dollar_loss(): def test_growing_volume_is_not_a_restart(): traded = { - "orcl-fly-20260913-055821": {"performance": {"orcl_fly_mm": _perf(5.0, 7000.0)}} + "xyz-orcl-usd-fly-20260913-055821": { + "performance": {"xyz_orcl_usd_fly_mm": _perf(5.0, 7000.0)} + } } _, _, _, carry = asyncio.run(_market(traded).equity(["XYZ:ORCL-USD"])) more = { - "orcl-fly-20260913-055821": { - "performance": {"orcl_fly_mm": _perf(-2.0, 9000.0)} + "xyz-orcl-usd-fly-20260913-055821": { + "performance": {"xyz_orcl_usd_fly_mm": _perf(-2.0, 9000.0)} } } _, _, per_pair, _ = asyncio.run(_market(more).equity(["XYZ:ORCL-USD"], carry)) diff --git a/agents/market_making_fly/tests/test_fly_venue.py b/agents/market_making_fly/tests/test_fly_venue.py new file mode 100644 index 000000000..61be7f044 --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_venue.py @@ -0,0 +1,43 @@ +"""Spot or perp, and what a round trip costs there.""" + +import pytest +from flybrain import venue + + +@pytest.mark.parametrize( + "connector,expected", + [ + ("binance_perpetual", "perp"), + ("hyperliquid_perpetual", "perp"), + ("gate_io_perpetual", "perp"), + ("binance", "spot"), + ("kucoin", "spot"), + ("backpack", "spot"), + ], +) +def test_market_type_comes_from_the_connector(connector, expected): + assert venue.market_type_for(connector) == expected + + +def test_longest_prefix_wins(): + """binance_perpetual is not binance, and their fees differ by 3.75x.""" + assert venue.default_maker_fee_bps("binance_perpetual", "perp") == 2.0 + assert venue.default_maker_fee_bps("binance", "spot") == 7.5 + + +def test_an_unknown_venue_is_quoted_wide_not_tight(): + """A fee floor set too low loses money silently; too high only costs fills.""" + assert venue.default_maker_fee_bps("brand_new_dex", "spot") == 10.0 + assert venue.default_maker_fee_bps("brand_new_dex_perpetual", "perp") == 2.5 + + +def test_resolve_refuses_a_contradiction(): + assert venue.resolve("binance") == "spot" + assert venue.resolve("binance_perpetual") == "perp" + assert venue.resolve("binance", "spot") == "spot" + with pytest.raises(ValueError): + venue.resolve("binance", "perp") + with pytest.raises(ValueError): + venue.resolve("binance", "margin") + with pytest.raises(ValueError): + venue.market_type_for("") diff --git a/docs/market_making_fly_design.md b/docs/market_making_fly_design.md index f49d85fb8..71a3eaa2f 100644 --- a/docs/market_making_fly_design.md +++ b/docs/market_making_fly_design.md @@ -27,9 +27,13 @@ substitute**, and Hummingbot's controller executes. The Condor LLM agent is the *operator* — it picks the market, deploys the bot, starts and stops the fly, reports, and explains — it never overrides the fly's posture with its own view. -The trading universe is **Hyperliquid HIP-3 perps (xyz issuer)** on -`hyperliquid_perpetual`, reusing Market Making Expert's HIP-3 scanner and its -HIP-3 operating rules. +The trading universe is **any CLOB market hummingbot-api serves, spot or perp**. +Hyperliquid HIP-3 was the first venue and keeps two special cases: its order +book comes from Hyperliquid's own endpoint (hummingbot-api's 500s on HIP-3 +pairs), and its scanner ranks the xyz issuer's markets. Everything venue- +dependent — spot versus perp, whether leverage applies, and what a round trip +costs — lives in `flybrain/venue.py`; the decoder, the chart and the dopamine +feedback are venue-agnostic, because a candle chart is a candle chart. FLM (the "Fly Language Model") was the first candidate and is **not** used: it is a frozen 1.2 B-parameter chat model whose next-token scores get a small @@ -276,7 +280,35 @@ the first external pulse — endogenous dopamine activity drives the rule too. W report `changed_edges`, `mean_efficacy`, `reward_spikes`, `aversive_spikes` every tick and make no claim beyond them. -## 10. Trading universe — HIP-3 +## 10. Trading universe — any CLOB market + +**Venue-independent by default (2026-09-13).** `flybrain/venue.py` holds the +three things that differ between venues, all of which bear on money: + +* **spot or perp**, decided by the connector name (`_perpetual` means perp). + Leverage applies only to a perp; on spot it must be 1 and `position_mode` is + not sent. A connector declared as the type it is not is refused rather than + silently reinterpreted. +* **the maker fee**, from which the take-profit floor follows. Spot fees run + three to five times perp fees on the same exchange, so the floor moves with + it: Binance perp 2 bp a side gives an 8.8 bp floor, Binance spot 7.5 bp gives + 33 bp. An unknown venue defaults deliberately wide (10 bp spot, 2.5 bp perp), + because a floor set too low loses money silently while one set too high only + costs fills. `maker_fee_bps` overrides it per deployment. +* **the top of book**, which comes from hummingbot-api for every connector it + serves. HIP-3 pairs are the one exception: that endpoint 500s on them, so + they fall back to Hyperliquid's public `l2Book`. + +Pair grammar is `BASE-QUOTE` or, on HIP-3, `ISSUER:TOKEN-QUOTE`. Names derive +from the **whole** pair (`SOL-USDT` → `sol-usdt-fly`, `XYZ:ORCL-USD` → +`xyz-orcl-usd-fly`), because naming a bot after its base token alone makes +`BTC-USDT` and `BTC-USDC` the same bot — the fly would read one book's P&L +while updating the other's config. + +The decoder, the chart and the dopamine feedback are unchanged and venue- +agnostic: a candle chart is a candle chart. + +### Hyperliquid HIP-3, the first venue * Connector `hyperliquid_perpetual`, issuer `xyz`, pairs `XYZ:TOKEN-USD` (uppercase; lowercase → KeyError → zero orders). From c07fed5b856c96f1acdcfc95b505e2a645ae045b Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 06:41:38 -0700 Subject: [PATCH 16/48] (feat) a dashboard for a running fly, with the fly in it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Laid out after stonkfly's own dashboard: the fly, the bag, the neuron strip, the latest neural order, the decision log, and the frame the fly last looked at. Stonkfly draws its fly with Three.js — checked in the browser rather than guessed: no window.THREE and no WebGL context on the visible canvas, because scene.js is a 586 KB bundled ES module whose WebGLRenderer draws to an OffscreenCanvas and blits to a 2D canvas with smoothing off, which is where the pixel look comes from. A Condor report cannot carry that. ReportBuilder has no raw-HTML or script escape hatch and its markdown is sanitized, so the fly here is Mesh3d geometry: abdomen, thorax, head, compound eyes, six legs, antennae and two wings that beat across ten animation frames. Plotly serializes frames through pio.to_html intact, and a Plotly 3D scene is natively drag-to-orbit, which is the affordance that mattered. Everything else is read back rather than intended: holdings and P&L from the live bots, the neuron strip and the decision log from the run directory, the sensory panel from the last frame on disk. The neuron panel falls back to the last tick that actually ran the brain, since a halt, a closed book or a failed tick records no observation and would otherwise show dashes over a run with thirty ticks behind it. The P&L curve plots only ticks whose P&L was reported, because charting a stale figure draws a flat line that looks like a result. Verified by rendering the report in a browser against the live halted run. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XoYbpk9wCJBNcSjz4TMJWG --- agents/market_making_fly/AGENT.md | 1 + agents/market_making_fly/flybrain/fly3d.py | 272 ++++++++++++ .../market_making_fly/routines/fly_report.py | 406 ++++++++++++++++++ .../skills/fly_decoder/SKILL.md | 5 +- .../tests/test_fly_report.py | 90 ++++ docs/market_making_fly_design.md | 1 + 6 files changed, 774 insertions(+), 1 deletion(-) create mode 100644 agents/market_making_fly/flybrain/fly3d.py create mode 100644 agents/market_making_fly/routines/fly_report.py create mode 100644 agents/market_making_fly/tests/test_fly_report.py diff --git a/agents/market_making_fly/AGENT.md b/agents/market_making_fly/AGENT.md index 8ac7d2c46..4a1d817cd 100644 --- a/agents/market_making_fly/AGENT.md +++ b/agents/market_making_fly/AGENT.md @@ -91,6 +91,7 @@ manage_skill(action="read", name="fly_mm_deploy") | `fly_chart` | Render the exact frame for a pair (what the fly sees) | | `fly_brain` | The loop (continuous). `mode=shadow|live`, `pairs`, `picked_spreads_bps`, `run_name` | | `fly_status` | Latest posture per pair, last observation, guard state, memory stats | +| `fly_report` | The run dashboard: the orbitable fly, the book, the neuron strip, the decision log, and the frame the fly last saw | | `mm_dashboard`, `mm_bot_report` | Inventory, positions, P&L, errors | Naming is derived from the **whole** pair, so two markets on one token never collide: diff --git a/agents/market_making_fly/flybrain/fly3d.py b/agents/market_making_fly/flybrain/fly3d.py new file mode 100644 index 000000000..28a57ceb6 --- /dev/null +++ b/agents/market_making_fly/flybrain/fly3d.py @@ -0,0 +1,272 @@ +"""A low-poly fly, built as meshes, for the run report. + +Stonkfly's dashboard renders its fly with Three.js — WebGL into an offscreen +canvas, blitted to a 2D canvas with smoothing off for the pixel look. A Condor +report cannot carry a 586 KB ES module: ``ReportBuilder`` has no raw-HTML or +script escape hatch, and its markdown is sanitized. What it does carry is +Plotly, serialized through ``pio.to_html`` with frames intact — and a Plotly 3D +scene is natively drag-to-orbit, which is the affordance that matters here. + +So the fly is ``Mesh3d`` geometry with a wing-flap animation, orbitable by +dragging, in the same palette as the chart the fly is shown. It is decoration +with one honest job: making it obvious at a glance which run you are looking at +and whether it is alive. +""" + +from __future__ import annotations + +import numpy as np + +# The chart's palette, so the report and the fly's own input agree. +BODY = "#c8d4e8" +THORAX = "#8fa3c4" +EYE = "#c5254e" +WING = "#5ce0d8" +LIMB = "#4a5a78" +GROUND = "#0a0e16" +ACCENT = "#c9f24d" + +FLAP_FRAMES = 10 +FLAP_DEGREES = 34.0 + + +def _ellipsoid( + center: tuple[float, float, float], + radii: tuple[float, float, float], + n_u: int = 18, + n_v: int = 11, +) -> tuple[np.ndarray, ...]: + """A closed ellipsoid as (x, y, z, i, j, k) for ``Mesh3d``.""" + u = np.linspace(0, 2 * np.pi, n_u, endpoint=False) + v = np.linspace(0, np.pi, n_v) + uu, vv = np.meshgrid(u, v, indexing="ij") + x = radii[0] * np.sin(vv) * np.cos(uu) + center[0] + y = radii[1] * np.sin(vv) * np.sin(uu) + center[1] + z = radii[2] * np.cos(vv) + center[2] + faces_i, faces_j, faces_k = [], [], [] + for a in range(n_u): + b = (a + 1) % n_u # wrap around the waist + for c in range(n_v - 1): + p0, p1 = a * n_v + c, b * n_v + c + p2, p3 = b * n_v + c + 1, a * n_v + c + 1 + faces_i += [p0, p0] + faces_j += [p1, p2] + faces_k += [p2, p3] + return ( + x.ravel(), + y.ravel(), + z.ravel(), + np.array(faces_i), + np.array(faces_j), + np.array(faces_k), + ) + + +def _rotate_x(y: np.ndarray, z: np.ndarray, pivot: tuple[float, float], degrees: float): + """Rotate (y, z) about a pivot — the wing hinge on the thorax.""" + rad = np.radians(degrees) + cy, cz = pivot + dy, dz = y - cy, z - cz + return ( + cy + dy * np.cos(rad) - dz * np.sin(rad), + cz + dy * np.sin(rad) + dz * np.cos(rad), + ) + + +def _mesh(xyzijk, color, opacity=1.0, name="", lighting=True): + import plotly.graph_objects as go + + x, y, z, i, j, k = xyzijk + return go.Mesh3d( + x=x, + y=y, + z=z, + i=i, + j=j, + k=k, + color=color, + opacity=opacity, + name=name, + flatshading=True, # low-poly: no smoothing between faces + hoverinfo="skip", + showscale=False, + lighting=( + dict(ambient=0.45, diffuse=0.85, specular=0.22, roughness=0.75) + if lighting + else dict(ambient=1.0, diffuse=0.0, specular=0.0) + ), + lightposition=dict(x=120, y=80, z=160), + ) + + +def _wing(side: int, degrees: float): + """One wing, hinged on the thorax and rotated by the flap angle.""" + # Swept back along the body and narrow, the way a resting fly holds them, + # so the wings read as wings rather than as two discs over the thorax. + x, y, z, i, j, k = _ellipsoid( + center=(-0.46, side * 0.40, 0.34), radii=(0.60, 0.21, 0.025), n_u=16, n_v=9 + ) + y, z = _rotate_x(y, z, pivot=(side * 0.12, 0.24), degrees=side * degrees) + return x, y, z, i, j, k + + +def _static_parts(): + """Everything that does not move: abdomen, thorax, head, eyes, legs.""" + import plotly.graph_objects as go + + parts = [ + _mesh(_ellipsoid((-0.72, 0, 0.02), (0.62, 0.30, 0.29)), BODY, name="abdomen"), + _mesh(_ellipsoid((0.02, 0, 0.06), (0.40, 0.33, 0.32)), THORAX, name="thorax"), + _mesh(_ellipsoid((0.56, 0, 0.10), (0.27, 0.27, 0.26)), BODY, name="head"), + _mesh( + _ellipsoid((0.66, 0.19, 0.17), (0.17, 0.13, 0.17), 12, 8), EYE, name="eye" + ), + _mesh( + _ellipsoid((0.66, -0.19, 0.17), (0.17, 0.13, 0.17), 12, 8), EYE, name="eye" + ), + ] + legs_x, legs_y, legs_z = [], [], [] + for hip, reach in ((0.26, 0.30), (-0.02, 0.02), (-0.30, -0.26)): + for side in (1, -1): + legs_x += [hip, hip + reach * 0.6, hip + reach, None] + legs_y += [side * 0.24, side * 0.52, side * 0.66, None] + legs_z += [-0.12, -0.48, -0.74, None] + parts.append( + go.Scatter3d( + x=legs_x, + y=legs_y, + z=legs_z, + mode="lines", + line=dict(color=LIMB, width=5), + hoverinfo="skip", + showlegend=False, + name="legs", + ) + ) + # Antennae + parts.append( + go.Scatter3d( + x=[0.70, 0.86, None, 0.70, 0.86], + y=[0.10, 0.16, None, -0.10, -0.16], + z=[0.30, 0.46, None, 0.30, 0.46], + mode="lines", + line=dict(color=LIMB, width=4), + hoverinfo="skip", + showlegend=False, + name="antennae", + ) + ) + return parts + + +def fly_figure(title: str = "", subtitle: str = "", height: int = 420): + """An orbitable low-poly fly whose wings beat when you press play.""" + import plotly.graph_objects as go + + static = _static_parts() + angles = [ + FLAP_DEGREES * np.sin(2 * np.pi * n / FLAP_FRAMES) for n in range(FLAP_FRAMES) + ] + wing_traces = [ + _mesh(_wing(1, angles[0]), WING, opacity=0.34, name="wing"), + _mesh(_wing(-1, angles[0]), WING, opacity=0.34, name="wing"), + ] + fig = go.Figure(data=static + wing_traces) + + wing_index = [len(static), len(static) + 1] + fig.frames = [ + go.Frame( + name=f"f{n}", + traces=wing_index, + data=[ + _mesh(_wing(1, angle), WING, opacity=0.34), + _mesh(_wing(-1, angle), WING, opacity=0.34), + ], + ) + for n, angle in enumerate(angles) + ] + + hidden = dict( + showbackground=False, + showgrid=False, + zeroline=False, + showticklabels=False, + title="", + visible=False, + ) + fig.update_layout( + height=height, + margin=dict(l=0, r=0, t=40 if title else 8, b=8), + paper_bgcolor=GROUND, + plot_bgcolor=GROUND, + font=dict(color=BODY, family="monospace", size=11), + title=( + dict(text=title, x=0.02, xanchor="left", font=dict(color=ACCENT, size=13)) + if title + else None + ), + showlegend=False, + scene=dict( + xaxis=hidden, + yaxis=hidden, + zaxis=hidden, + bgcolor=GROUND, + aspectmode="data", + camera=dict(eye=dict(x=1.30, y=-1.50, z=0.80)), + annotations=( + [ + dict( + showarrow=False, + x=0, + y=0, + z=-1.15, + text=subtitle, + font=dict(color=LIMB, size=11, family="monospace"), + ) + ] + if subtitle + else [] + ), + ), + updatemenus=[ + dict( + type="buttons", + showactive=False, + x=0.01, + y=0.03, + xanchor="left", + yanchor="bottom", + bgcolor="#141c28", + bordercolor=LIMB, + font=dict(color=ACCENT, size=10, family="monospace"), + buttons=[ + dict( + label="▶ BEAT WINGS", + method="animate", + args=[ + None, + dict( + frame=dict(duration=55, redraw=True), + fromcurrent=True, + transition=dict(duration=0), + mode="immediate", + ), + ], + ), + dict( + label="❚❚ PAUSE", + method="animate", + args=[ + [None], + dict( + frame=dict(duration=0, redraw=False), + mode="immediate", + ), + ], + ), + ], + ) + ], + legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), + ) + return fig diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py new file mode 100644 index 000000000..08560a9f3 --- /dev/null +++ b/agents/market_making_fly/routines/fly_report.py @@ -0,0 +1,406 @@ +"""A dashboard for a running fly: the fly itself, its book, its neurons, its calls. + +Laid out after stonkfly's dashboard — the fly, the bag, the neuron strip, the +latest neural order, the decision log, and what the fly is actually looking at. +Stonkfly draws its fly with Three.js; a Condor report carries Plotly, so the fly +here is Mesh3d geometry that orbits by dragging and beats its wings on play. + +Everything except the fly is read back from the run directory and the live bot, +so the report says what happened rather than what was intended. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + +import json +import logging +import time + +import numpy as np +import plotly.graph_objects as go +from flybrain.fly3d import ACCENT, BODY, EYE, GROUND, LIMB, fly_figure +from flybrain.market import LiveMarket +from flybrain.naming import pair_names +from flybrain.run_state import RunDir +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.memory.paths import agent_home +from condor.paths import safe_id +from condor.reports import ReportBuilder + +logger = logging.getLogger(__name__) + +CATEGORY = "Bot Analysis" +AGENT_SLUG = "market_making_fly" + +# How an execution status reads in the decision log. +RESULT_WORDS = { + "APPLIED": "APPLIED", + "SHADOW": "SHADOW", + "HOLD": "HOLD", + "VETO": "VETO", + "CLOSED": "BOOK CLOSED", + "STOP_BOT": "BOT STOPPED", + "ERROR": "UPDATE FAILED", + "HALT": "HALT", + "TICK_ERROR": "TICK FAILED", +} + + +class Config(BaseModel): + """Dashboard for one fly run: the fly, its book, its neurons and its calls.""" + + run_name: str = Field( + default="fly", description="Run directory under the agent home" + ) + recent: int = Field(default=12, ge=1, le=100, description="Decisions to list") + connector_name: str = Field( + default="hyperliquid_perpetual", description="Connector the bots run on" + ) + + +def _fmt(value, digits=2, plus=False) -> str: + if value is None or isinstance(value, bool): + return "—" + try: + number = float(value) + except (TypeError, ValueError): + return str(value) + return f"{number:+,.{digits}f}" if plus else f"{number:,.{digits}f}" + + +def _clock(wall_time) -> str: + if not wall_time: + return "—" + return time.strftime("%H:%M:%S", time.localtime(float(wall_time))) + + +def _frame_figure(path: Path) -> go.Figure | None: + """The last chart the fly was shown, as the report's sensory panel.""" + if not path.exists(): + return None + from PIL import Image + + frame = np.asarray(Image.open(path).convert("RGB"), dtype=np.uint8) + fig = go.Figure(go.Image(z=frame)) + fig.update_layout( + height=340, + margin=dict(l=0, r=0, t=6, b=6), + paper_bgcolor=GROUND, + plot_bgcolor=GROUND, + xaxis=dict(visible=False), + yaxis=dict(visible=False), + legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), + ) + return fig + + +def _pnl_figure(events: list[dict]) -> go.Figure | None: + """Equity across the ticks whose P&L was actually reported.""" + points = [ + (e["tick"], float(e["equity"])) + for e in events + if e.get("pnl_known") and e.get("equity") is not None + ] + if len(points) < 2: + return None + ticks, equity = zip(*points) + fig = go.Figure( + go.Scatter( + x=list(ticks), + y=list(equity), + mode="lines", + line=dict(color=ACCENT, width=2), + fill="tozeroy", + fillcolor="rgba(201,242,77,0.10)", + name="net P&L", + ) + ) + fig.update_layout( + height=260, + margin=dict(l=48, r=16, t=10, b=36), + paper_bgcolor=GROUND, + plot_bgcolor=GROUND, + font=dict(color=BODY, family="monospace", size=11), + xaxis=dict(title="tick", gridcolor="#18202e", zerolinecolor="#18202e"), + yaxis=dict( + title="net P&L (quote)", gridcolor="#18202e", zerolinecolor="#243044" + ), + showlegend=False, + legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), + ) + fig.add_hline(y=0, line=dict(color=LIMB, width=1, dash="dot")) + return fig + + +async def _holdings(client, connector_name: str, pairs: list[str]) -> list[dict]: + """What each of the fly's bots is holding right now, from the live bot.""" + market = LiveMarket(client, connector_name, "5m", 72) + bots = await market.bots() + rows = [] + for pair in pairs: + names = pair_names(pair) + running, bot = LiveMarket.find_bot(bots, names.bot_name) + if bot is None: + rows.append({"Market": pair, "Bot": "not running", "Side": "—"}) + continue + perf = (bot.get("performance") or {}).get(names.config_name) or {} + inner = perf.get("performance", perf) if isinstance(perf, dict) else {} + positions = inner.get("positions_summary") or [] + amount = sum( + float(p.get("amount", 0) or 0) for p in positions if isinstance(p, dict) + ) + rows.append( + { + "Market": pair, + "Bot": running, + "Side": ("LONG" if amount > 0 else "SHORT" if amount < 0 else "FLAT"), + "Position": _fmt(abs(amount), 4), + "Realized": _fmt(inner.get("realized_pnl_quote"), 4, plus=True), + "Unrealized": _fmt(inner.get("unrealized_pnl_quote"), 4, plus=True), + "Volume": _fmt(inner.get("volume_traded")), + } + ) + return rows + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + from config_manager import get_client + + root = agent_home(AGENT_SLUG) / "fly" / safe_id(config.run_name) + if not root.exists(): + raise FileNotFoundError( + f"No fly run at {root}; start fly_brain with run_name={config.run_name!r}" + ) + run_dir = RunDir(root) + state = run_dir.load_state() + latest = ( + json.loads(run_dir.latest_path.read_text()) + if run_dir.latest_path.exists() + else {} + ) + events = run_dir.recent_events(max(config.recent, 60)) + provenance = ( + json.loads(run_dir.provenance_path.read_text()) + if run_dir.provenance_path.exists() + else {} + ) + + guard = state.get("guard", {}) + postures = state.get("postures", {}) + pairs = list(postures) + # A halt, a closed book or a failed tick records no observation, so the + # neuron panel reads back the last tick that actually ran the brain rather + # than showing dashes over a run with thirty ticks of history behind it. + observed = next( + (e for e in reversed(events) if (e.get("neural") or {}).get("total_spikes")), + latest, + ) + neural = observed.get("neural", {}) + memory = neural.get("memory", {}) + neurons = (provenance.get("circuit") or {}).get("neurons") + last_posture = observed.get("posture") or latest.get("posture") or {} + execution = latest.get("execution") or {} + stale = observed is not latest + + client = await get_client(context._chat_id, context=context) + holdings = await _holdings(client, config.connector_name, pairs) if client else [] + book_net = sum( + float(info.get("net", 0) or 0) + for info in (latest.get("bots") or {}).values() + if isinstance(info, dict) + ) + book_volume = latest.get("volume") + + halted = guard.get("halted") + alive = "HALTED" if halted else "RUNNING" + + builder = ReportBuilder(f"Fly — {config.run_name}") + builder.source("routine", "fly_report") + builder.tags(["fly", "dashboard", config.run_name]) + builder.manual_order() + + # ── FLY.EXE ────────────────────────────────────────────────────────────── + builder.section( + "FLY.EXE", + f"{alive} · run {config.run_name} · tick {state.get('tick', 0)} · " + f"{len(pairs)} market{'s' if len(pairs) != 1 else ''} · drag to orbit", + ) + builder.plotly( + fly_figure( + title=f"FLY.EXE — {alive}", + subtitle=f"{', '.join(pairs) or 'no market'}", + ) + ) + + # ── THE BAG ────────────────────────────────────────────────────────────── + builder.section("THE BAG", "What the fly's own bots are holding right now") + builder.kpi("Net P&L", _fmt(book_net, 4, plus=True)) + builder.kpi("Volume", _fmt(book_volume)) + builder.kpi("Markets", str(len(pairs))) + builder.kpi("Mode", str(latest.get("mode", "—")).upper()) + if holdings: + builder.table( + holdings, + ["Market", "Bot", "Side", "Position", "Realized", "Unrealized", "Volume"], + ) + else: + builder.markdown( + "_No live bot data — either no server is bound to this chat, or the fly is " + "running in shadow with nothing deployed._" + ) + + # ── NEURONS ────────────────────────────────────────────────────────────── + builder.section( + "NEURONS", + "The connectome's own numbers, from the last observation" + + ( + f" (tick {observed.get('tick')}; the newest tick ran no brain)" + if stale + else "" + ), + ) + builder.kpi("Neurons", f"{neurons:,}" if neurons else "—") + builder.kpi( + "Latest spikes", + f"{neural.get('total_spikes'):,}" if neural.get("total_spikes") else "—", + ) + builder.kpi( + "Memory changed", + ( + f"{memory.get('changed_edges'):,}" + if memory.get("changed_edges") is not None + else "—" + ), + ) + builder.kpi( + "Brain time", + f"{float(neural['brain_ms']) / 1000:,.1f} s" if neural.get("brain_ms") else "—", + ) + builder.kpi( + "Kenyon cells", + f"{neural.get('kc_spikes'):,}" if neural.get("kc_spikes") is not None else "—", + ) + builder.kpi("Gate (DNpe017)", str(neural.get("gate_spikes", "—"))) + builder.kpi("Reward (PAM11)", str(neural.get("reward_spikes", "—"))) + builder.kpi("Aversive (PPL101)", str(neural.get("aversive_spikes", "—"))) + builder.kpi("Mean efficacy", _fmt(memory.get("mean_efficacy"), 5)) + + # ── NEURAL ORDER ───────────────────────────────────────────────────────── + builder.section("NEURAL ORDER", "The posture decoded from the last observation") + if last_posture: + builder.kpi("Regime", str(last_posture.get("regime", "—")).upper()) + builder.kpi("Spread ×", _fmt(last_posture.get("spread_mult"))) + builder.kpi("Lean", f"{_fmt(last_posture.get('shift_bps'), 2, plus=True)} bp") + builder.kpi("Trend z", _fmt(last_posture.get("trend_z"), 2, plus=True)) + builder.kpi("Arousal z", _fmt(last_posture.get("arousal_z"), 2, plus=True)) + builder.kpi( + "Result", + RESULT_WORDS.get(execution.get("status"), execution.get("status", "—")), + ) + builder.markdown( + f"**{str(last_posture.get('regime', 'no posture yet')).upper()}** on " + f"`{observed.get('pair', '—')}` — {execution.get('reason', 'nothing recorded')}. " + f"Stimulus `{observed.get('stimulus', 'none')}`, P&L delta " + f"{_fmt(latest.get('pnl_delta'), 4, plus=True)}." + + ("" if last_posture.get("warm", True) else " _Baseline still forming._") + ) + + # ── DECISIONS ──────────────────────────────────────────────────────────── + builder.section( + "DECISIONS", f"Last {min(config.recent, len(events))} observations, newest last" + ) + builder.table( + [ + { + "Time": _clock(e.get("wall_time")), + "Tick": e.get("tick"), + "Market": e.get("pair"), + "Neural proposal": ( + f"{str((e.get('posture') or {}).get('regime', '—')).upper()}" + f" ×{_fmt((e.get('posture') or {}).get('spread_mult'))}" + f" {_fmt((e.get('posture') or {}).get('shift_bps'), 2, plus=True)}bp" + ), + "R−L": f"{_fmt((e.get('neural') or {}).get('trend_hz'), 1, plus=True)} Hz", + "Stimulus": e.get("stimulus", "—"), + "Result": RESULT_WORDS.get( + (e.get("execution") or {}).get("status"), + (e.get("execution") or {}).get("status", "—"), + ), + "Reason": str((e.get("execution") or {}).get("reason", ""))[:54], + } + for e in events[-config.recent :] + ], + [ + "Time", + "Tick", + "Market", + "Neural proposal", + "R−L", + "Stimulus", + "Result", + "Reason", + ], + ) + + pnl = _pnl_figure(events) + if pnl is not None: + builder.plotly(pnl) + + # ── WHAT THE FLY SEES ──────────────────────────────────────────────────── + frame = _frame_figure(run_dir.frame_path) + if frame is not None: + builder.section( + "WHAT THE FLY SEES", + "The last 320×180 frame fed to the retina — candles, volume, bid/ask. " + "No quotes, inventory or P&L are drawn; those reach the fly only as dopamine.", + ) + builder.plotly(frame) + + # ── PERFORMANCE & LIMITS ───────────────────────────────────────────────── + builder.section("PERFORMANCE & LIMITS", "What the guard is watching") + builder.kpi("Halted", halted or "no") + builder.kpi("Session high", _fmt(guard.get("session_high_net"), 4, plus=True)) + builder.kpi("Ticks since high", str(guard.get("ticks_since_high", "—"))) + builder.kpi("Applies today", str(guard.get("applies_today", 0))) + builder.kpi("Anchor", _fmt(state.get("anchor"), 4, plus=True)) + builder.markdown( + "_Regime, spread multiplier and lean are an engineered readout of spike counts, " + "not a discovered market-making circuit. Dopamine pulses report the change in " + "P&L between two observations, not credit for the last posture, and synapses " + "also change from endogenous activity. No profitable learning has been " + "demonstrated: the guard and the controller's stop loss bound the loss, not the " + "fly._" + ) + await builder.save() + + lines = [ + f"run: {config.run_name}", + f"state: {alive}" + (f" ({halted})" if halted else ""), + f"tick: {state.get('tick', 0)}", + f"markets: {', '.join(pairs) or '—'}", + f"net_pnl: {_fmt(book_net, 4, plus=True)}", + f"volume: {_fmt(book_volume)}", + f"last_regime: {last_posture.get('regime', '—')}" + + (f" (observed tick {observed.get('tick')})" if stale else ""), + f"last_result: {execution.get('status', '—')}", + f"latest_spikes: {neural.get('total_spikes', '—')}", + f"changed_edges: {memory.get('changed_edges', '—')}", + ] + for row in holdings: + lines.append( + f"{row['Market']}: {row.get('Side', '—')} {row.get('Position', '')} " + f"realized {row.get('Realized', '—')} volume {row.get('Volume', '—')}" + ) + lines.append( + "caveat: the readout is engineered, the pulses are value feedback rather than " + "credit, and no profitable learning is demonstrated" + ) + return "\n".join(lines) diff --git a/agents/market_making_fly/skills/fly_decoder/SKILL.md b/agents/market_making_fly/skills/fly_decoder/SKILL.md index 40927b42d..af1e264be 100644 --- a/agents/market_making_fly/skills/fly_decoder/SKILL.md +++ b/agents/market_making_fly/skills/fly_decoder/SKILL.md @@ -11,7 +11,10 @@ references_routine: fly_status # Reading the fly -Run `manage_routines(action="run", name="fly_status", config={"run_name": ""})`. +Run `manage_routines(action="run", name="fly_status", config={"run_name": ""})` +for the numbers as text, or `fly_report` for the dashboard — the same data plus the +fly itself, the decision log and the last frame it saw. Use `fly_report` when the +user wants to *look* at the run, `fly_status` when you need to quote figures. ## Channels (per observation, 500 ms of neural time) diff --git a/agents/market_making_fly/tests/test_fly_report.py b/agents/market_making_fly/tests/test_fly_report.py new file mode 100644 index 000000000..70fc5abe9 --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_report.py @@ -0,0 +1,90 @@ +"""The run dashboard: the fly mesh, and the panels' own formatting.""" + +import importlib.util +from pathlib import Path + +import numpy as np +import pytest +from flybrain import fly3d + + +def _module(name): + path = Path(__file__).resolve().parents[1] / "routines" / f"{name}.py" + spec = importlib.util.spec_from_file_location(name, path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def test_every_mesh_face_indexes_a_real_vertex(): + """A Mesh3d with an out-of-range face renders nothing and says nothing.""" + x, y, z, i, j, k = fly3d._ellipsoid((0, 0, 0), (1, 0.5, 0.5), n_u=12, n_v=7) + assert len(x) == len(y) == len(z) == 12 * 7 + assert len(i) == len(j) == len(k) == 12 * (7 - 1) * 2 + for faces in (i, j, k): + assert faces.min() >= 0 and faces.max() < len(x) + assert np.isfinite(np.concatenate([x, y, z])).all() + + +def test_the_wings_actually_move(): + """The flap is the animation; identical frames would be a still image.""" + up = fly3d._wing(1, fly3d.FLAP_DEGREES) + down = fly3d._wing(1, -fly3d.FLAP_DEGREES) + assert not np.allclose(up[2], down[2]) # z differs + # and the two wings are mirrored, not stacked on one side + left, right = fly3d._wing(1, 0.0), fly3d._wing(-1, 0.0) + assert left[1].mean() > 0 > right[1].mean() + + +def test_the_figure_animates_and_orbits(): + fig = fly3d.fly_figure(title="FLY.EXE", subtitle="XYZ:ORCL-USD") + assert len(fig.frames) == fly3d.FLAP_FRAMES + assert len(fig.data) >= 7 # abdomen, thorax, head, 2 eyes, legs, antennae, wings + # every frame retargets exactly the two wing traces + wing_traces = list(fig.frames[0].traces) + assert len(wing_traces) == 2 + assert all(list(f.traces) == wing_traces for f in fig.frames) + # a play button exists, and the scene is a 3D one (drag-to-orbit is native) + assert fig.layout.updatemenus and fig.layout.updatemenus[0].buttons + assert fig.layout.scene.camera.eye.x is not None + + +def test_numbers_are_formatted_or_visibly_absent(): + report = _module("fly_report") + assert report._fmt(None) == "—" + assert report._fmt(float("nan")) == "nan" + assert report._fmt(1234.5678, 2) == "1,234.57" + assert report._fmt(-0.5, 2, plus=True) == "-0.50" + assert report._fmt(0.5, 2, plus=True) == "+0.50" + assert report._clock(None) == "—" + assert ":" in report._clock(1789279456.0) + + +def test_the_pnl_curve_only_plots_reported_ticks(): + """An unreported tick has no P&L to plot; charting its stale figure would + draw a flat line that looks like a result.""" + report = _module("fly_report") + events = [ + {"tick": 1, "equity": 1.0, "pnl_known": True}, + {"tick": 2, "equity": 9.9, "pnl_known": False}, + {"tick": 3, "equity": 2.0, "pnl_known": True}, + ] + fig = report._pnl_figure(events) + assert list(fig.data[0].x) == [1, 3] and list(fig.data[0].y) == [1.0, 2.0] + assert report._pnl_figure(events[:1]) is None + + +def test_every_execution_status_has_a_word(): + report = _module("fly_report") + for status in ( + "APPLIED", + "SHADOW", + "HOLD", + "VETO", + "CLOSED", + "STOP_BOT", + "ERROR", + "HALT", + "TICK_ERROR", + ): + assert report.RESULT_WORDS[status] diff --git a/docs/market_making_fly_design.md b/docs/market_making_fly_design.md index 71a3eaa2f..2ea1d3c13 100644 --- a/docs/market_making_fly_design.md +++ b/docs/market_making_fly_design.md @@ -360,6 +360,7 @@ agents/market_making_fly/ fly_chart.py # one-shot: render + report the frame for a pair fly_brain.py # CONTINUOUS: the loop in §3; modes shadow|live, learning|frozen, fixture|market fly_status.py # one-shot: latest posture, channels, z-scores, spikes, memory stats, P&L, vetoes, halt reason + fly_report.py # one-shot: the run dashboard (fly, book, neurons, decisions, sensory frame) hip3_market_scanner.py # copied from Market Making Expert mm_dashboard.py # copied skills/ From 1a6f52f009e75f88b83c76cca51ee77176a16784 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 07:50:51 -0700 Subject: [PATCH 17/48] (feat) the fly works at a desk, beside the frame it was reading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes to the run dashboard. The sensory panel moves up beside FLY.EXE, where it belongs: the posture and the picture it was decoded from now read together, captioned with the tick and the candle count and interval the run actually used. And the fly gets the reference's scene — desk, monitor, keyboard, mug, speakers and a skyline behind — with the fly in profile facing the screen. The monitor is not decoration: it shows the fly's own input frame. The chart is drawn from a fixed palette, so six colours cover ~99 % of it and the rest is text antialiasing, which collapses to an exact stepped colorscale. Three things that had to be right for that to work, each found by rendering rather than by reading: Nearest-colour search squared channel differences in int16, where 216**2 overflows, so pixels took the wrong palette entry and the chart rendered with its background and header swapped. A Surface interpolates surfacecolor between grid points, and a midpoint between the chart's light background and its grid lines lands in the header's slot, drawing dark bars through the chart. Mesh3d with intensitymode="cell" colours each face outright. Three quarters of the chart is its background, so painting that once behind the mesh and keeping only the faces that differ drops it from ~12,000 triangles to ~3,000 with no pixel lost. The ellipsoids placed n_u coincident vertices at each pole, whose quads collapse to zero-area slivers. A static export hid them; WebGL drew them as a white sawtooth along the body. Poles are single vertices with a fan now, and a test fails on any degenerate face. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XoYbpk9wCJBNcSjz4TMJWG --- agents/market_making_fly/flybrain/fly3d.py | 334 ++++++++++++++++-- .../market_making_fly/routines/fly_report.py | 34 +- .../tests/test_fly_report.py | 16 +- 3 files changed, 346 insertions(+), 38 deletions(-) diff --git a/agents/market_making_fly/flybrain/fly3d.py b/agents/market_making_fly/flybrain/fly3d.py index 28a57ceb6..edf82d7cd 100644 --- a/agents/market_making_fly/flybrain/fly3d.py +++ b/agents/market_making_fly/flybrain/fly3d.py @@ -17,6 +17,13 @@ import numpy as np +DESK = "#1b2432" +DESK_EDGE = "#2b3950" +BEZEL = "#151d29" +KEYBOARD = "#222c3d" +MUG = "#c5254e" +CITY = "#121a27" + # The chart's palette, so the report and the fly's own input agree. BODY = "#c8d4e8" THORAX = "#8fa3c4" @@ -36,26 +43,47 @@ def _ellipsoid( n_u: int = 18, n_v: int = 11, ) -> tuple[np.ndarray, ...]: - """A closed ellipsoid as (x, y, z, i, j, k) for ``Mesh3d``.""" + """A closed ellipsoid as (x, y, z, i, j, k) for ``Mesh3d``. + + Each pole is ONE vertex with a triangle fan around it. Sweeping v through + 0 and pi on a full u grid instead would place n_u coincident vertices at + each pole, and the quads there collapse to zero-area slivers — which WebGL + renders as a white sawtooth along the body, even though a static export + hides it. + """ + rings = np.linspace(0, np.pi, n_v)[1:-1] + n_rings = len(rings) u = np.linspace(0, 2 * np.pi, n_u, endpoint=False) - v = np.linspace(0, np.pi, n_v) - uu, vv = np.meshgrid(u, v, indexing="ij") - x = radii[0] * np.sin(vv) * np.cos(uu) + center[0] - y = radii[1] * np.sin(vv) * np.sin(uu) + center[1] - z = radii[2] * np.cos(vv) + center[2] + uu, vv = np.meshgrid(u, rings, indexing="ij") + x = np.append( + (radii[0] * np.sin(vv) * np.cos(uu) + center[0]).ravel(), + [center[0], center[0]], + ) + y = np.append( + (radii[1] * np.sin(vv) * np.sin(uu) + center[1]).ravel(), + [center[1], center[1]], + ) + z = np.append( + (radii[2] * np.cos(vv) + center[2]).ravel(), + [center[2] + radii[2], center[2] - radii[2]], + ) + north, south = n_u * n_rings, n_u * n_rings + 1 faces_i, faces_j, faces_k = [], [], [] for a in range(n_u): b = (a + 1) % n_u # wrap around the waist - for c in range(n_v - 1): - p0, p1 = a * n_v + c, b * n_v + c - p2, p3 = b * n_v + c + 1, a * n_v + c + 1 + faces_i += [north, south] + faces_j += [b * n_rings, a * n_rings + n_rings - 1] + faces_k += [a * n_rings, b * n_rings + n_rings - 1] + for c in range(n_rings - 1): + p0, p1 = a * n_rings + c, b * n_rings + c + p2, p3 = b * n_rings + c + 1, a * n_rings + c + 1 faces_i += [p0, p0] faces_j += [p1, p2] faces_k += [p2, p3] return ( - x.ravel(), - y.ravel(), - z.ravel(), + x, + y, + z, np.array(faces_i), np.array(faces_j), np.array(faces_k), @@ -91,11 +119,11 @@ def _mesh(xyzijk, color, opacity=1.0, name="", lighting=True): hoverinfo="skip", showscale=False, lighting=( - dict(ambient=0.45, diffuse=0.85, specular=0.22, roughness=0.75) + dict(ambient=0.66, diffuse=0.42, specular=0.10, roughness=0.90) if lighting else dict(ambient=1.0, diffuse=0.0, specular=0.0) ), - lightposition=dict(x=120, y=80, z=160), + lightposition=dict(x=-60, y=-140, z=180), ) @@ -115,9 +143,21 @@ def _static_parts(): import plotly.graph_objects as go parts = [ - _mesh(_ellipsoid((-0.72, 0, 0.02), (0.62, 0.30, 0.29)), BODY, name="abdomen"), - _mesh(_ellipsoid((0.02, 0, 0.06), (0.40, 0.33, 0.32)), THORAX, name="thorax"), - _mesh(_ellipsoid((0.56, 0, 0.10), (0.27, 0.27, 0.26)), BODY, name="head"), + _mesh( + _ellipsoid((-0.72, 0, 0.02), (0.62, 0.30, 0.29), n_u=26, n_v=15), + BODY, + name="abdomen", + ), + _mesh( + _ellipsoid((0.02, 0, 0.06), (0.40, 0.33, 0.32), n_u=24, n_v=14), + THORAX, + name="thorax", + ), + _mesh( + _ellipsoid((0.56, 0, 0.10), (0.27, 0.27, 0.26), n_u=22, n_v=13), + BODY, + name="head", + ), _mesh( _ellipsoid((0.66, 0.19, 0.17), (0.17, 0.13, 0.17), 12, 8), EYE, name="eye" ), @@ -159,11 +199,253 @@ def _static_parts(): return parts -def fly_figure(title: str = "", subtitle: str = "", height: int = 420): - """An orbitable low-poly fly whose wings beat when you press play.""" +def _box(x0, x1, y0, y1, z0, z1): + """An axis-aligned box as (x, y, z, i, j, k).""" + x = np.array([x0, x1, x1, x0, x0, x1, x1, x0], dtype=float) + y = np.array([y0, y0, y1, y1, y0, y0, y1, y1], dtype=float) + z = np.array([z0, z0, z0, z0, z1, z1, z1, z1], dtype=float) + i = np.array([0, 0, 4, 4, 0, 0, 1, 1, 2, 2, 3, 3]) + j = np.array([1, 2, 5, 6, 1, 5, 2, 6, 3, 7, 0, 4]) + k = np.array([2, 3, 6, 7, 5, 4, 6, 5, 7, 6, 4, 7]) + return x, y, z, i, j, k + + +def _cylinder(cx, cy, z0, z1, radius, sides=14): + """A capped cylinder — the mug.""" + a = np.linspace(0, 2 * np.pi, sides, endpoint=False) + rim_x, rim_y = cx + radius * np.cos(a), cy + radius * np.sin(a) + x = np.concatenate([rim_x, rim_x, [cx], [cx]]) + y = np.concatenate([rim_y, rim_y, [cy], [cy]]) + z = np.concatenate([np.full(sides, z0), np.full(sides, z1), [z0], [z1]]) + low_c, high_c = 2 * sides, 2 * sides + 1 + i, j, k = [], [], [] + for a0 in range(sides): + a1 = (a0 + 1) % sides + i += [a0, a0, low_c, high_c] + j += [a1, a1 + sides, a1, a0 + sides] + k += [a1 + sides, a0 + sides, a0, a1 + sides] + return x, y, z, np.array(i), np.array(j), np.array(k) + + +def _quantize(rgb: np.ndarray, colors: int = 10): + """Map the frame onto a small palette and return (index grid, palette). + + The chart is drawn from a fixed palette — background, header, grid, up, + down, volume, text — so six colours already cover ~99 % of it and the rest + is text antialiasing. That collapses to an exact stepped colorscale, which + is how a Plotly surface can show the real chart rather than a stand-in. + """ + # int32, not int16: a squared channel difference reaches 255**2 = 65025, + # which wraps in int16 and silently assigns pixels the wrong colour — + # rendering the chart with its background and header swapped. + flat = rgb.reshape(-1, 3).astype(np.int32) + unique, counts = np.unique(flat, axis=0, return_counts=True) + palette = unique[np.argsort(-counts)[:colors]] + # Nearest palette colour per pixel, in chunks so a full frame stays cheap. + index = np.empty(len(flat), dtype=np.int32) + for start in range(0, len(flat), 8192): + chunk = flat[start : start + 8192] + d = ((chunk[:, None, :] - palette[None, :, :]) ** 2).sum(-1) + index[start : start + 8192] = d.argmin(1) + return index.reshape(rgb.shape[:2]), palette + + +def _screen( + rgb: np.ndarray, + x: float, + y0: float, + y1: float, + z0: float, + z1: float, + width: int = 104, +): + """The monitor's glass: the fly's own input frame, as flat-shaded cells. + + Not a ``Surface``: that interpolates ``surfacecolor`` between grid points, + and a midpoint between the chart's light background and its grid lines + lands in the header's slot, drawing dark bars through the chart. ``Mesh3d`` + with ``intensitymode="cell"`` colours each face outright, so every pixel is + the colour it actually is. + """ + import plotly.graph_objects as go + from PIL import Image + + height = max(2, round(width * rgb.shape[0] / rgb.shape[1])) + small = np.asarray( + Image.fromarray(rgb).resize((width, height), Image.NEAREST), dtype=np.uint8 + ) + index, palette = _quantize(small) + n = len(palette) + scale = [] + for slot, colour in enumerate(palette): + css = f"rgb({colour[0]},{colour[1]},{colour[2]})" + scale += [[slot / n, css], [(slot + 1) / n, css]] + + # Seen from the fly's side (-x looking toward +x), image column 0 is at +y + # and row 0 at the top, so both axes run backwards from the array order. + ys = np.linspace(y1, y0, width + 1) + zs = np.linspace(z1, z0, height + 1) + grid_y, grid_z = np.meshgrid(ys, zs, indexing="xy") + vy, vz = grid_y.ravel(), grid_z.ravel() + stride = width + 1 + rows, cols = np.meshgrid(np.arange(height), np.arange(width), indexing="ij") + v00 = (rows * stride + cols).ravel() + v01, v10 = v00 + 1, v00 + stride + v11 = v10 + 1 + cell = index.ravel() + faces_i = np.concatenate([v00, v00]) + faces_j = np.concatenate([v01, v11]) + faces_k = np.concatenate([v11, v10]) + intensity = np.concatenate([cell, cell]).astype(float) + # Three quarters of the chart is its background. Painting that once as a + # single quad behind the mesh, and keeping only the faces that differ from + # it, drops this from ~12,000 triangles to ~3,000 without losing a pixel. + keep = intensity != 0.0 + faces_i, faces_j, faces_k, intensity = ( + faces_i[keep], + faces_j[keep], + faces_k[keep], + intensity[keep], + ) + backdrop = _mesh( + _box(x + 0.004, x + 0.006, min(y0, y1), max(y0, y1), min(z0, z1), max(z0, z1)), + f"rgb({palette[0][0]},{palette[0][1]},{palette[0][2]})", + lighting=False, + name="screen backdrop", + ) + + detail = go.Mesh3d( + x=np.full_like(vy, x), + y=vy, + z=vz, + i=faces_i, + j=faces_j, + k=faces_k, + intensity=(intensity + 0.5) / n, + intensitymode="cell", + cmin=0, + cmax=1, + colorscale=scale, + showscale=False, + flatshading=True, + hoverinfo="skip", + lighting=dict(ambient=1.0, diffuse=0.0, specular=0.0), + name="screen", + ) + return [backdrop, detail] + + +def desk_parts(chart: np.ndarray | None = None, scale: float = 0.52): + """The desk the fly works at: surface, monitor, keyboard, mug, speakers. + + Modelled on stonkfly's scene, including its proportions — the fly is the + size of the monitor, not of a person. ``scale`` shrinks the furniture about + the desk top, which stays where the fly's legs land. The monitor is the + point of it: the fly is looking at the same frame the report shows beside + it. + """ + import plotly.graph_objects as go + + top = -0.74 # where the fly's legs land + + def box(x0, x1, y0, y1, z0, z1): + return _box( + x0 * scale, + x1 * scale, + y0 * scale, + y1 * scale, + top + (z0 - top) * scale, + top + (z1 - top) * scale, + ) + + def up(z): + return top + (z - top) * scale + + parts = [ + _mesh(box(-3.6, 2.6, -3.3, 3.3, top - 0.18, top), DESK, name="desk"), + _mesh(box(2.52, 2.6, -3.3, 3.3, top, top + 0.06), DESK_EDGE, name="desk edge"), + _mesh(box(2.16, 2.30, -0.16, 0.16, top, top + 0.46), BEZEL, name="stand"), + _mesh(box(2.00, 2.44, -0.70, 0.70, top, top + 0.07), BEZEL, name="foot"), + _mesh( + box(2.20, 2.32, -2.10, 2.10, top + 0.40, top + 2.40), BEZEL, name="bezel" + ), + _mesh(box(0.95, 1.80, -1.15, 1.15, top, top + 0.08), KEYBOARD, name="keyboard"), + _mesh( + _cylinder(1.60 * scale, -1.85 * scale, top, up(top + 0.38), 0.22 * scale), + MUG, + name="mug", + ), + _mesh(box(2.00, 2.38, -2.95, -2.40, top, top + 0.80), BEZEL, name="speaker"), + _mesh(box(2.00, 2.38, 2.40, 2.95, top, top + 0.80), BEZEL, name="speaker"), + ] + # Silhouettes behind the desk, for the room the reference has. + for n, (yy, w, h) in enumerate( + ( + (-3.6, 0.8, 3.0), + (-2.2, 0.6, 4.2), + (-0.9, 0.7, 2.6), + (0.5, 0.6, 3.8), + (1.8, 0.9, 2.8), + (3.0, 0.7, 4.0), + ) + ): + parts.append( + _mesh( + box(3.5 + 0.10 * n, 3.8 + 0.10 * n, yy, yy + w, top, top + h), + CITY, + name="city", + ) + ) + if chart is not None: + parts.extend( + _screen( + chart, + 2.19 * scale, + -2.02 * scale, + 2.02 * scale, + up(top + 0.48), + up(top + 2.32), + ) + ) + else: + parts.append( + _mesh( + box(2.17, 2.20, -2.02, 2.02, top + 0.48, top + 2.32), + "#0d1622", + name="screen off", + ) + ) + parts.append( + go.Scatter3d( + x=[0.95 * scale, 1.80 * scale, None, -3.6 * scale, -3.6 * scale], + y=[-1.15 * scale, -1.15 * scale, None, -3.3 * scale, 3.3 * scale], + z=[up(top + 0.085), up(top + 0.085), None, up(top + 0.02), up(top + 0.02)], + mode="lines", + line=dict(color=WING, width=3), + opacity=0.7, + hoverinfo="skip", + showlegend=False, + name="glow", + ) + ) + return parts + + +def fly_figure( + title: str = "", + subtitle: str = "", + height: int = 460, + chart: np.ndarray | None = None, + desk: bool = True, +): + """An orbitable low-poly fly at its desk, wings beating when you press play. + + ``chart`` is the fly's own input frame; pass it and the monitor shows the + same picture the decoder was reading. + """ import plotly.graph_objects as go - static = _static_parts() + static = (desk_parts(chart) if desk else []) + _static_parts() angles = [ FLAP_DEGREES * np.sin(2 * np.pi * n / FLAP_FRAMES) for n in range(FLAP_FRAMES) ] @@ -212,14 +494,20 @@ def fly_figure(title: str = "", subtitle: str = "", height: int = 420): zaxis=hidden, bgcolor=GROUND, aspectmode="data", - camera=dict(eye=dict(x=1.30, y=-1.50, z=0.80)), + # Over the fly's shoulder, so both it and what it is looking at + # are in frame — the reference's own framing. + camera=dict( + eye=dict(x=-0.06, y=-1.10, z=0.38), + center=dict(x=0.22, y=0, z=-0.16), + up=dict(x=0, y=0, z=1), + ), annotations=( [ dict( showarrow=False, - x=0, + x=-0.4, y=0, - z=-1.15, + z=-1.05, text=subtitle, font=dict(color=LIMB, size=11, family="monospace"), ) diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index 08560a9f3..7bca1dce3 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -82,13 +82,17 @@ def _clock(wall_time) -> str: return time.strftime("%H:%M:%S", time.localtime(float(wall_time))) -def _frame_figure(path: Path) -> go.Figure | None: - """The last chart the fly was shown, as the report's sensory panel.""" +def _read_frame(path: Path) -> np.ndarray | None: + """The exact pixels the retina last received, as saved by the loop.""" if not path.exists(): return None from PIL import Image - frame = np.asarray(Image.open(path).convert("RGB"), dtype=np.uint8) + return np.asarray(Image.open(path).convert("RGB"), dtype=np.uint8) + + +def _frame_figure(frame: np.ndarray) -> go.Figure: + """That frame, full size, as the report's sensory panel.""" fig = go.Figure(go.Image(z=frame)) fig.update_layout( height=340, @@ -193,6 +197,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: else {} ) + settings = provenance.get("settings") or {} guard = state.get("guard", {}) postures = state.get("postures", {}) pairs = list(postures) @@ -233,12 +238,25 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: f"{alive} · run {config.run_name} · tick {state.get('tick', 0)} · " f"{len(pairs)} market{'s' if len(pairs) != 1 else ''} · drag to orbit", ) + sensory = _read_frame(run_dir.frame_path) builder.plotly( fly_figure( title=f"FLY.EXE — {alive}", subtitle=f"{', '.join(pairs) or 'no market'}", + chart=sensory, ) ) + # The same frame, full size, beside the fly that was looking at it. + if sensory is not None: + builder.section( + "WHAT THE FLY SEES", + f"The 320×180 frame fed to the retina on tick {observed.get('tick')} — " + f"{settings.get('n_candles', '?')} × {settings.get('candle_interval', '?')} " + "candles, volume and the live bid/ask. This is the picture the posture " + "above was decoded from; no quotes, inventory or P&L are drawn, because " + "those reach the fly only as dopamine.", + ) + builder.plotly(_frame_figure(sensory)) # ── THE BAG ────────────────────────────────────────────────────────────── builder.section("THE BAG", "What the fly's own bots are holding right now") @@ -354,16 +372,6 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: if pnl is not None: builder.plotly(pnl) - # ── WHAT THE FLY SEES ──────────────────────────────────────────────────── - frame = _frame_figure(run_dir.frame_path) - if frame is not None: - builder.section( - "WHAT THE FLY SEES", - "The last 320×180 frame fed to the retina — candles, volume, bid/ask. " - "No quotes, inventory or P&L are drawn; those reach the fly only as dopamine.", - ) - builder.plotly(frame) - # ── PERFORMANCE & LIMITS ───────────────────────────────────────────────── builder.section("PERFORMANCE & LIMITS", "What the guard is watching") builder.kpi("Halted", halted or "no") diff --git a/agents/market_making_fly/tests/test_fly_report.py b/agents/market_making_fly/tests/test_fly_report.py index 70fc5abe9..9dd4a626d 100644 --- a/agents/market_making_fly/tests/test_fly_report.py +++ b/agents/market_making_fly/tests/test_fly_report.py @@ -19,13 +19,25 @@ def _module(name): def test_every_mesh_face_indexes_a_real_vertex(): """A Mesh3d with an out-of-range face renders nothing and says nothing.""" x, y, z, i, j, k = fly3d._ellipsoid((0, 0, 0), (1, 0.5, 0.5), n_u=12, n_v=7) - assert len(x) == len(y) == len(z) == 12 * 7 - assert len(i) == len(j) == len(k) == 12 * (7 - 1) * 2 + assert len(x) == len(y) == len(z) == 12 * 5 + 2 # 5 rings plus two poles + assert len(i) == len(j) == len(k) for faces in (i, j, k): assert faces.min() >= 0 and faces.max() < len(x) assert np.isfinite(np.concatenate([x, y, z])).all() +def test_no_face_collapses_to_a_sliver(): + """Coincident pole vertices make zero-area triangles, which WebGL draws as + a white sawtooth across the body even though a static export hides them.""" + x, y, z, i, j, k = fly3d._ellipsoid((0, 0, 0), (1, 0.5, 0.5), n_u=12, n_v=7) + p = np.stack([x, y, z], axis=1) + a, b, c = p[i], p[j], p[k] + area = 0.5 * np.linalg.norm(np.cross(b - a, c - a), axis=1) + assert area.min() > 1e-9, "degenerate face in the mesh" + # and each vertex is actually used, so nothing is left stranded + assert set(np.concatenate([i, j, k]).tolist()) == set(range(len(x))) + + def test_the_wings_actually_move(): """The flap is the animation; identical frames would be a still image.""" up = fly3d._wing(1, fly3d.FLAP_DEGREES) From f6baf2e256f26275854e8689a339f956499a8ce3 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 09:38:02 -0700 Subject: [PATCH 18/48] (feat) the fly and the frame it read, side by side, in four panels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scene and the sensory frame move into one figure, split left and right, so the posture and the picture it came from read together. They have to share a figure rather than two grid cells: the report runtime does lay blocks on a 12-column grid that collapses to full width under 800px, but that width is only exposed on data-bound components, not on a plain Plotly block — and this agent does not reach into core Condor to change that. The split favours the scene, which survives shrinking better than a chart does. The sections regroup into four panels: the fly with its frame; the neuron numbers with the posture decoded from them, which is only readable against them; the decision log with the positions those decisions left behind, renamed from THE BAG; and the guard. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XoYbpk9wCJBNcSjz4TMJWG --- agents/market_making_fly/flybrain/fly3d.py | 50 +++++++++++++ .../market_making_fly/routines/fly_report.py | 74 +++++++++---------- 2 files changed, 86 insertions(+), 38 deletions(-) diff --git a/agents/market_making_fly/flybrain/fly3d.py b/agents/market_making_fly/flybrain/fly3d.py index edf82d7cd..e9cc110c5 100644 --- a/agents/market_making_fly/flybrain/fly3d.py +++ b/agents/market_making_fly/flybrain/fly3d.py @@ -558,3 +558,53 @@ def fly_figure( legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), ) return fig + + +def desk_and_chart_figure( + chart: np.ndarray | None, + title: str = "", + subtitle: str = "", + height: int = 460, + scene_share: float = 0.62, +): + """The fly at its desk and the frame it was reading, side by side. + + One figure rather than two blocks: a report component spans the runtime's + 12-column grid, but that width is only exposed on data-bound components, + not on a plain Plotly block — and this agent does not reach into core + Condor to change that. Splitting inside the figure gets the same reading + (posture beside the picture it came from) without touching the runtime. + + On a narrow screen the whole figure scales rather than stacking, so the + split favours the scene, which survives shrinking better than a chart does. + """ + import plotly.graph_objects as go + from plotly.subplots import make_subplots + + if chart is None: + return fly_figure(title=title, subtitle=subtitle, height=height) + + fig = make_subplots( + rows=1, + cols=2, + specs=[[{"type": "scene"}, {"type": "xy"}]], + column_widths=[scene_share, 1 - scene_share], + horizontal_spacing=0.015, + ) + scene = fly_figure(title=title, subtitle=subtitle, height=height, chart=chart) + for trace in scene.data: + fig.add_trace(trace, row=1, col=1) + # Added last, so the wing indices the frames retarget are unchanged. + fig.add_trace(go.Image(z=chart, hoverinfo="skip"), row=1, col=2) + fig.frames = scene.frames + + fig.update_layout(scene.layout) + fig.update_layout( + height=height, + margin=dict(l=0, r=0, t=40 if title else 8, b=8), + xaxis=dict(visible=False), + yaxis=dict(visible=False, scaleanchor="x"), + ) + # The 3D panel keeps the left of the figure; the frame sits at its right. + fig.update_layout(scene=dict(domain=dict(x=[0.0, scene_share], y=[0, 1]))) + return fig diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index 7bca1dce3..f52da85bd 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -24,7 +24,7 @@ import numpy as np import plotly.graph_objects as go -from flybrain.fly3d import ACCENT, BODY, EYE, GROUND, LIMB, fly_figure +from flybrain.fly3d import ACCENT, BODY, GROUND, LIMB, desk_and_chart_figure from flybrain.market import LiveMarket from flybrain.naming import pair_names from flybrain.run_state import RunDir @@ -236,49 +236,34 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: builder.section( "FLY.EXE", f"{alive} · run {config.run_name} · tick {state.get('tick', 0)} · " - f"{len(pairs)} market{'s' if len(pairs) != 1 else ''} · drag to orbit", + f"{len(pairs)} market{'s' if len(pairs) != 1 else ''} · drag to orbit · " + "the monitor and the panel beside it show the fly's own input frame", ) sensory = _read_frame(run_dir.frame_path) builder.plotly( - fly_figure( + desk_and_chart_figure( + sensory, title=f"FLY.EXE — {alive}", subtitle=f"{', '.join(pairs) or 'no market'}", - chart=sensory, ) ) - # The same frame, full size, beside the fly that was looking at it. if sensory is not None: - builder.section( - "WHAT THE FLY SEES", - f"The 320×180 frame fed to the retina on tick {observed.get('tick')} — " - f"{settings.get('n_candles', '?')} × {settings.get('candle_interval', '?')} " - "candles, volume and the live bid/ask. This is the picture the posture " - "above was decoded from; no quotes, inventory or P&L are drawn, because " - "those reach the fly only as dopamine.", - ) - builder.plotly(_frame_figure(sensory)) - - # ── THE BAG ────────────────────────────────────────────────────────────── - builder.section("THE BAG", "What the fly's own bots are holding right now") - builder.kpi("Net P&L", _fmt(book_net, 4, plus=True)) - builder.kpi("Volume", _fmt(book_volume)) - builder.kpi("Markets", str(len(pairs))) - builder.kpi("Mode", str(latest.get("mode", "—")).upper()) - if holdings: - builder.table( - holdings, - ["Market", "Bot", "Side", "Position", "Realized", "Unrealized", "Volume"], - ) - else: builder.markdown( - "_No live bot data — either no server is bound to this chat, or the fly is " - "running in shadow with nothing deployed._" + f"**What the fly sees** — the 320×180 frame fed to the retina on tick " + f"{observed.get('tick')}: {settings.get('n_candles', '?')} × " + f"{settings.get('candle_interval', '?')} candles, volume and the live " + "bid/ask, shown on its monitor and again at full size beside it. This is " + "the picture the posture below was decoded from. No quotes, inventory or " + "P&L are drawn, because those reach the fly only as dopamine." ) - # ── NEURONS ────────────────────────────────────────────────────────────── + # ── NEURONS & NEURAL ORDER ─────────────────────────────────────────────── + # One panel: the connectome's numbers and the posture they were decoded + # into belong together — the second is only readable against the first. builder.section( - "NEURONS", - "The connectome's own numbers, from the last observation" + "NEURONS & NEURAL ORDER", + "The connectome's own numbers from the last observation, and the posture " + "decoded from them" + ( f" (tick {observed.get('tick')}; the newest tick ran no brain)" if stale @@ -310,9 +295,6 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: builder.kpi("Reward (PAM11)", str(neural.get("reward_spikes", "—"))) builder.kpi("Aversive (PPL101)", str(neural.get("aversive_spikes", "—"))) builder.kpi("Mean efficacy", _fmt(memory.get("mean_efficacy"), 5)) - - # ── NEURAL ORDER ───────────────────────────────────────────────────────── - builder.section("NEURAL ORDER", "The posture decoded from the last observation") if last_posture: builder.kpi("Regime", str(last_posture.get("regime", "—")).upper()) builder.kpi("Spread ×", _fmt(last_posture.get("spread_mult"))) @@ -331,9 +313,12 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + ("" if last_posture.get("warm", True) else " _Baseline still forming._") ) - # ── DECISIONS ──────────────────────────────────────────────────────────── + # ── DECISIONS & POSITIONS ──────────────────────────────────────────────── + # One panel: what the fly called, and what those calls left it holding. builder.section( - "DECISIONS", f"Last {min(config.recent, len(events))} observations, newest last" + "DECISIONS & POSITIONS", + f"Last {min(config.recent, len(events))} observations, newest last — and " + "what the fly's own bots are holding right now", ) builder.table( [ @@ -367,10 +352,23 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: "Reason", ], ) - pnl = _pnl_figure(events) if pnl is not None: builder.plotly(pnl) + builder.kpi("Net P&L", _fmt(book_net, 4, plus=True)) + builder.kpi("Volume", _fmt(book_volume)) + builder.kpi("Markets", str(len(pairs))) + builder.kpi("Mode", str(latest.get("mode", "—")).upper()) + if holdings: + builder.table( + holdings, + ["Market", "Bot", "Side", "Position", "Realized", "Unrealized", "Volume"], + ) + else: + builder.markdown( + "_No live bot data — either no server is bound to this chat, or the fly is " + "running in shadow with nothing deployed._" + ) # ── PERFORMANCE & LIMITS ───────────────────────────────────────────────── builder.section("PERFORMANCE & LIMITS", "What the guard is watching") From 27ca592b8a0c7ec20e00451c8a62b2a43512cec0 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 09:49:52 -0700 Subject: [PATCH 19/48] (feat) a plotly figure can span part of the report grid MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ReportBuilder.plotly gains the width every data-bound component already takes: its span of the 12-column grid, defaulting to the full row so every existing report renders byte-identically. A narrower figure leaves the plain `.section` markup behind and becomes a grid item, because `.report-grid > .section` forces `grid-column: 1 / -1` and would otherwise beat the span silently. Below the layout's 800px breakpoint the runtime already collapses grid items to full width, so two figures side by side now genuinely stack on a phone — which a single figure split internally could never do. The fly report uses it: the scene takes seven columns and the frame it was reading takes five, replacing the combined subplot figure, which is deleted. Also removed from the agent: mm_dashboard, a Market Making Expert copy that fly_report supersedes, and the mm_bot_report skill, which instructed running a routine this agent does not have — the agent hit exactly that during the live deployment and had to improvise. The deploy playbook now verifies with fly_report. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XoYbpk9wCJBNcSjz4TMJWG --- agents/market_making_fly/AGENT.md | 5 +- agents/market_making_fly/flybrain/fly3d.py | 50 --- .../market_making_fly/routines/fly_report.py | 13 +- .../routines/mm_dashboard.py | 388 ------------------ .../skills/fly_mm_deploy/SKILL.md | 2 +- .../skills/mm_bot_report/SKILL.md | 40 -- condor/reports/builder.py | 34 +- docs/market_making_fly_design.md | 6 +- tests/test_report_builder.py | 27 ++ 9 files changed, 70 insertions(+), 495 deletions(-) delete mode 100644 agents/market_making_fly/routines/mm_dashboard.py delete mode 100644 agents/market_making_fly/skills/mm_bot_report/SKILL.md diff --git a/agents/market_making_fly/AGENT.md b/agents/market_making_fly/AGENT.md index 4a1d817cd..3fdb45394 100644 --- a/agents/market_making_fly/AGENT.md +++ b/agents/market_making_fly/AGENT.md @@ -57,7 +57,7 @@ veto or halt, and it never substitutes a posture either. the task, default 3 - Starting `fly_brain` in shadow, then live; stopping it; rotating a market slot - Reading `fly_status` and explaining a posture in market-making terms -- Reporting bot health with `mm_bot_report` / `mm_dashboard` +- Reporting the run with `fly_report`, and the numbers with `fly_status` - Saying plainly what the fly has and has not demonstrated ## What you do not handle @@ -77,7 +77,7 @@ repeat the caveats. Do not deploy. **Delegated:** read `fly_mm_deploy` and follow it end-to-end — scanner → `n_markets` picks → neutral configs → deploy with a loss cap → start `fly_brain` in shadow — then verify -with `mm_bot_report`. Switch to live only when the task says so. +with `fly_report`. Switch to live only when the task says so. ``` manage_skill(action="read", name="fly_mm_deploy") @@ -92,7 +92,6 @@ manage_skill(action="read", name="fly_mm_deploy") | `fly_brain` | The loop (continuous). `mode=shadow|live`, `pairs`, `picked_spreads_bps`, `run_name` | | `fly_status` | Latest posture per pair, last observation, guard state, memory stats | | `fly_report` | The run dashboard: the orbitable fly, the book, the neuron strip, the decision log, and the frame the fly last saw | -| `mm_dashboard`, `mm_bot_report` | Inventory, positions, P&L, errors | Naming is derived from the **whole** pair, so two markets on one token never collide: `SOL-USDT` → bot `sol-usdt-fly`, config `sol_usdt_fly_mm`; `XYZ:ORCL-USD` → diff --git a/agents/market_making_fly/flybrain/fly3d.py b/agents/market_making_fly/flybrain/fly3d.py index e9cc110c5..edf82d7cd 100644 --- a/agents/market_making_fly/flybrain/fly3d.py +++ b/agents/market_making_fly/flybrain/fly3d.py @@ -558,53 +558,3 @@ def fly_figure( legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), ) return fig - - -def desk_and_chart_figure( - chart: np.ndarray | None, - title: str = "", - subtitle: str = "", - height: int = 460, - scene_share: float = 0.62, -): - """The fly at its desk and the frame it was reading, side by side. - - One figure rather than two blocks: a report component spans the runtime's - 12-column grid, but that width is only exposed on data-bound components, - not on a plain Plotly block — and this agent does not reach into core - Condor to change that. Splitting inside the figure gets the same reading - (posture beside the picture it came from) without touching the runtime. - - On a narrow screen the whole figure scales rather than stacking, so the - split favours the scene, which survives shrinking better than a chart does. - """ - import plotly.graph_objects as go - from plotly.subplots import make_subplots - - if chart is None: - return fly_figure(title=title, subtitle=subtitle, height=height) - - fig = make_subplots( - rows=1, - cols=2, - specs=[[{"type": "scene"}, {"type": "xy"}]], - column_widths=[scene_share, 1 - scene_share], - horizontal_spacing=0.015, - ) - scene = fly_figure(title=title, subtitle=subtitle, height=height, chart=chart) - for trace in scene.data: - fig.add_trace(trace, row=1, col=1) - # Added last, so the wing indices the frames retarget are unchanged. - fig.add_trace(go.Image(z=chart, hoverinfo="skip"), row=1, col=2) - fig.frames = scene.frames - - fig.update_layout(scene.layout) - fig.update_layout( - height=height, - margin=dict(l=0, r=0, t=40 if title else 8, b=8), - xaxis=dict(visible=False), - yaxis=dict(visible=False, scaleanchor="x"), - ) - # The 3D panel keeps the left of the figure; the frame sits at its right. - fig.update_layout(scene=dict(domain=dict(x=[0.0, scene_share], y=[0, 1]))) - return fig diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index f52da85bd..d9eca9d24 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -24,7 +24,7 @@ import numpy as np import plotly.graph_objects as go -from flybrain.fly3d import ACCENT, BODY, GROUND, LIMB, desk_and_chart_figure +from flybrain.fly3d import ACCENT, BODY, GROUND, LIMB, fly_figure from flybrain.market import LiveMarket from flybrain.naming import pair_names from flybrain.run_state import RunDir @@ -240,13 +240,18 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: "the monitor and the panel beside it show the fly's own input frame", ) sensory = _read_frame(run_dir.frame_path) + # Two halves of the grid, not one figure split internally: below the + # layout's 800px breakpoint these stack on their own. builder.plotly( - desk_and_chart_figure( - sensory, + fly_figure( title=f"FLY.EXE — {alive}", subtitle=f"{', '.join(pairs) or 'no market'}", - ) + chart=sensory, + ), + width=7, ) + if sensory is not None: + builder.plotly(_frame_figure(sensory), width=5) if sensory is not None: builder.markdown( f"**What the fly sees** — the 320×180 frame fed to the retina on tick " diff --git a/agents/market_making_fly/routines/mm_dashboard.py b/agents/market_making_fly/routines/mm_dashboard.py deleted file mode 100644 index 8f53a950b..000000000 --- a/agents/market_making_fly/routines/mm_dashboard.py +++ /dev/null @@ -1,388 +0,0 @@ -"""MM Dashboard: unified portfolio inventory + bot positions + controller performance. - -Consolidates three former routines into one: - • portfolio_scanner → Portfolio / inventory section (get_state + get_total_value) - • mm_bot_report → Bots / positions / PnL section (bot_orchestration.get_active_bots_status) - • bot_position_tracker → superseded (its executor-search grouping is replaced by the - strictly-richer bot_orchestration path used here) - -Bot data comes from client.bot_orchestration.get_active_bots_status(): - bots_data[bot_name]["performance"][ctrl_name]["performance"] - → realized_pnl_quote, unrealized_pnl_quote, volume_traded - → positions_summary (list of open positions with pair/side/amount/breakeven) - → close_type_counts (CloseType.XXX → count) - bots_data[bot_name]["error_logs"] → errors -""" - -import logging -from collections import Counter -from datetime import datetime, timezone - -from pydantic import BaseModel, Field -from telegram.ext import ContextTypes - -from config_manager import get_client - -logger = logging.getLogger(__name__) - -CATEGORY = "Bot Analysis" - -_CLOSE_LABELS = { - "TAKE_PROFIT": "TP", - "STOP_LOSS": "SL", - "TRAILING_STOP": "Trail", - "EARLY_STOP": "Early", - "POSITION_HOLD": "Hold", - "HOLD": "Hold", - "EXPIRED": "Expired", - "INSUFFICIENT_BALANCE": "Insuf$", - "FAILED": "Failed", - "UNKNOWN": "?", -} - -_ERROR_LEVELS = {"ERROR", "CRITICAL", "FATAL"} - - -def _close_label(raw: str) -> str: - """Convert 'CloseType.EARLY_STOP' → 'Early'.""" - code = raw.split(".")[-1] if "." in raw else raw - return _CLOSE_LABELS.get(code.upper(), code) - - -def _side_label(raw: str) -> str: - """Convert 'TradeType.BUY' → 'BUY'.""" - return raw.split(".")[-1] if "." in raw else raw - - -class Config(BaseModel): - """Unified MM dashboard: portfolio inventory, bot positions, PnL, and errors.""" - - connector_name: str = Field( - default="binance_perpetual", - description="Focus connector for portfolio (empty=all)", - ) - trading_pair: str = Field( - default="", description="Filter bot positions by pair (empty = all)" - ) - min_value_usd: float = Field( - default=1.0, description="Hide tokens below this USD value" - ) - include_errors: bool = Field(default=True, description="Include error log summary") - - -async def _fetch_portfolio(client, connector_name: str, min_value_usd: float): - """Scan portfolio inventory. Returns (inv_rows, total_value, error_msg, summary_lines).""" - state = None - errors = [] - try: - state = await client.portfolio.get_state() - except Exception as e: - errors.append(f"get_state: {e}") - - if not state or not isinstance(state, dict): - try: - state = await client.portfolio.get_portfolio_state() - except Exception as e: - errors.append(f"get_portfolio_state: {e}") - - if not state or not isinstance(state, dict): - return [], None, " | ".join(errors) or "No portfolio state available", [] - - inv_rows = [] - summary_lines = [] - for acct_name, acct_data in state.items(): - if not isinstance(acct_data, dict): - continue - for conn_name, tokens in acct_data.items(): - if connector_name and connector_name not in conn_name: - continue - if not isinstance(tokens, list): - continue - - significant = [ - t - for t in tokens - if isinstance(t, dict) and float(t.get("value", 0)) >= min_value_usd - ] - if not significant: - continue - - significant.sort(key=lambda t: float(t.get("value", 0)), reverse=True) - total_value = sum(float(t.get("value", 0)) for t in significant) - - summary_lines.append(f"**{acct_name} / {conn_name}**") - for t in significant: - token = t.get("token", "?") - units = float(t.get("units", 0)) - value = float(t.get("value", 0)) - available = float(t.get("available_units", units)) - pct = (value / total_value * 100) if total_value > 0 else 0 - in_use = units - available - use_flag = f" (in_use: {in_use:.4f})" if in_use > 0.001 else "" - summary_lines.append( - f" {token}: {units:,.4f} = ${value:,.2f} ({pct:.1f}%){use_flag}" - ) - inv_rows.append( - { - "Account": acct_name, - "Connector": conn_name, - "Token": token, - "Units": round(units, 4), - "Value (USD)": f"${value:,.2f}", - "Weight": f"{pct:.1f}%", - "In Use": f"{in_use:.4f}" if in_use > 0.001 else "-", - } - ) - summary_lines.append(f" Subtotal: ${total_value:,.2f}") - summary_lines.append("") - - total_val = None - try: - tv = await client.portfolio.get_total_value() - if tv: - total_val = float(tv) - except Exception: - pass - if total_val is None and inv_rows: - total_val = sum( - float(r["Value (USD)"].replace("$", "").replace(",", "")) for r in inv_rows - ) - - return inv_rows, total_val, None, summary_lines - - -async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: - client = await get_client(context._chat_id, context=context) - if not client: - return "No server available" - - # ── 1. Portfolio / Inventory ───────────────────────────────────────────── - inv_rows, total_value, portfolio_error, portfolio_summary = await _fetch_portfolio( - client, config.connector_name, config.min_value_usd - ) - - # ── 2. Bots / Positions / PnL (bot_orchestration — richer source) ──────── - bots_data: dict = {} - bots_error = None - try: - resp = await client.bot_orchestration.get_active_bots_status() - raw = resp if isinstance(resp, dict) else {} - bots_data = raw.get("data", raw) if isinstance(raw, dict) else {} - if not isinstance(bots_data, dict): - bots_data = {} - except Exception as e: - bots_error = f"Failed to fetch bot status: {e}" - - ctrl_rows = [] # per-controller perf table - pos_rows = [] # open positions table - close_totals: Counter = Counter() - total_realized = total_unrealized = total_volume = 0.0 - total_errors = 0 - error_lines = [] - - for bot_name, bot_data in bots_data.items(): - if not isinstance(bot_data, dict): - continue - - # Error logs - if config.include_errors: - errs = [ - e - for e in (bot_data.get("error_logs") or []) - if isinstance(e, dict) - and str(e.get("level_name", "")).upper() in _ERROR_LEVELS - ] - if errs: - total_errors += len(errs) - error_lines.append(f" • {bot_name}: {len(errs)} error(s)") - - perf_dict = bot_data.get("performance", {}) - if not isinstance(perf_dict, dict): - continue - - for ctrl_name, ctrl_data in perf_dict.items(): - if not isinstance(ctrl_data, dict): - continue - inner = ctrl_data.get("performance", ctrl_data) - if not isinstance(inner, dict): - continue - - # Parse open positions to get pair/connector for filtering - positions = inner.get("positions_summary") or [] - pair = "" - connector = "" - if ( - positions - and isinstance(positions, list) - and isinstance(positions[0], dict) - ): - pair = positions[0].get("trading_pair", "") - connector = positions[0].get("connector_name", "") - - # Filter by trading pair (empty = all controllers) - if config.trading_pair and config.trading_pair not in pair: - continue - - realized = float(inner.get("realized_pnl_quote", 0) or 0) - unrealized = float(inner.get("unrealized_pnl_quote", 0) or 0) - volume = float(inner.get("volume_traded", 0) or 0) - total_realized += realized - total_unrealized += unrealized - total_volume += volume - - # Close type breakdown - close_counts: dict = inner.get("close_type_counts", {}) or {} - ctrl_closes: Counter = Counter() - for raw_ct, cnt in close_counts.items(): - label = _close_label(str(raw_ct)) - ctrl_closes[label] += int(cnt) - close_totals.update(ctrl_closes) - close_str = ( - " | ".join(f"{k}:{v}" for k, v in ctrl_closes.most_common()) or "—" - ) - - ctrl_rows.append( - { - "Controller": ctrl_name, - "Pair": pair or "—", - "Realized PnL": f"${realized:,.4f}", - "Unrealized PnL": f"${unrealized:,.4f}", - "Volume": f"${volume:,.2f}", - "Closes": close_str, - } - ) - - # Open positions - for pos in positions: - if not isinstance(pos, dict): - continue - pos_pair = pos.get("trading_pair", pair) - pos_conn = pos.get("connector_name", connector) - side = _side_label(str(pos.get("side", ""))) - amount = float(pos.get("amount", 0) or 0) - bp = float(pos.get("breakeven_price", 0) or 0) - pos_unreal = float(pos.get("unrealized_pnl_quote", 0) or 0) - pos_real = float(pos.get("realized_pnl_quote", 0) or 0) - pos_rows.append( - { - "Controller": ctrl_name, - "Pair": pos_pair, - "Connector": pos_conn, - "Side": side, - "Amount": f"{amount:,.2f}", - "Breakeven": f"${bp:,.6f}", - "Unrealized PnL": f"${pos_unreal:,.4f}", - "Realized PnL": f"${pos_real:,.4f}", - } - ) - - net_pnl = total_realized + total_unrealized - - # ── Summary text ───────────────────────────────────────────────────────── - now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC") - close_summary = ( - " | ".join(f"{k}:{v}" for k, v in close_totals.most_common()) or "none" - ) - - lines = [f"**MM Dashboard** — {now}", ""] - - # Portfolio block - if portfolio_error and not inv_rows: - lines.append(f"Portfolio: {portfolio_error}") - else: - total_str = f"${total_value:,.2f}" if total_value else "unknown" - lines.append( - f"Portfolio total: {total_str} | {len(inv_rows)} asset(s) on {config.connector_name or 'all'}" - ) - if portfolio_summary: - lines.append("") - lines.extend(portfolio_summary) - - # Bots block - if bots_error: - lines.append(bots_error) - else: - lines.append(f"Controllers: {len(ctrl_rows)} | Open positions: {len(pos_rows)}") - lines.append( - f"Realized PnL: ${total_realized:,.4f} | Unrealized: ${total_unrealized:,.4f} | Net: ${net_pnl:,.4f}" - ) - lines.append(f"Volume: ${total_volume:,.2f}") - lines.append(f"Closes: {close_summary}") - - if config.include_errors and not bots_error: - if total_errors: - lines.append("") - lines.append(f"⚠️ {total_errors} error(s) across active bots:") - lines.extend(error_lines[:8]) - else: - lines.append("") - lines.append("✅ No errors in active bot logs") - - summary = "\n".join(lines) - - # ── Persistent report ───────────────────────────────────────────────────── - from condor.reports import ReportBuilder - - builder = ReportBuilder("MM Dashboard") - builder.source("routine", "mm_dashboard").tags( - ["portfolio", "bots", "positions", "market-making"] - ) - - if total_value: - builder.kpi("Portfolio Value", f"${total_value:,.2f}") - builder.kpi("Active Controllers", str(len(ctrl_rows))) - builder.kpi("Open Positions", str(len(pos_rows))) - builder.kpi("Realized PnL", f"${total_realized:,.4f}") - builder.kpi("Unrealized PnL", f"${total_unrealized:,.4f}") - builder.kpi("Net PnL", f"${net_pnl:,.4f}") - builder.kpi("Volume", f"${total_volume:,.2f}") - if config.include_errors: - builder.kpi("Errors", str(total_errors)) - - if pos_rows: - builder.table( - pos_rows, - [ - "Controller", - "Pair", - "Connector", - "Side", - "Amount", - "Breakeven", - "Unrealized PnL", - "Realized PnL", - ], - ) - if ctrl_rows: - builder.table( - ctrl_rows, - [ - "Controller", - "Pair", - "Realized PnL", - "Unrealized PnL", - "Volume", - "Closes", - ], - ) - if inv_rows: - builder.table( - inv_rows, - [ - "Account", - "Connector", - "Token", - "Units", - "Value (USD)", - "Weight", - "In Use", - ], - ) - elif portfolio_error: - builder.markdown(f"**Portfolio unavailable**: {portfolio_error}") - - builder.markdown(summary) - builder.manual_order() - await builder.save() - - return summary diff --git a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md index 1b656d050..f2c4b98bd 100644 --- a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md +++ b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md @@ -118,7 +118,7 @@ non-neutral postures. ``` manage_routines(action="run", name="fly_status", config={"run_name": "fly-2026-09-12"}) -manage_routines(action="run", name="mm_bot_report", config={}) +manage_routines(action="run", name="fly_report", config={"run_name": "fly-2026-09-12"}) ``` Report: pairs, spreads, bots running, fly tick count, first postures, any vetoes or diff --git a/agents/market_making_fly/skills/mm_bot_report/SKILL.md b/agents/market_making_fly/skills/mm_bot_report/SKILL.md deleted file mode 100644 index 6b5c760e6..000000000 --- a/agents/market_making_fly/skills/mm_bot_report/SKILL.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -name: mm_bot_report -description: 'Run the MM bot status report: running bots, open/hold-mode positions, - closed position breakdown (TP/SL/Early/Hold), PnL, volume, and error summary.' -when_to_use: When the user asks for a bot status report, "how is the bot doing", "show - me the report", "what's the PnL", "any errors", "how are positions", "closed positions - breakdown", or any general health/status check on the running MM bots. Also use - after deploying a new bot to verify it's running correctly. -created: '2026-07-02T15:46:08Z' -source: agent:market_making_fly -references_routine: mm_bot_report ---- - -## MM Bot Report - -Run the `mm_bot_report` routine — it fetches everything in one shot: - -``` -manage_routines(action="run", name="mm_bot_report", config={}) -``` - -**What it returns:** -- **Controllers** — active controller count -- **Open positions** — active executors currently placing quotes (`is_trading=True`) -- **Hold-mode** — active executors paused/holding inventory (`is_trading=False`) -- **Recent closes breakdown** — by close type: TP | SL | Early | Hold | Trail | Expired -- **PnL & Volume** — realized + unrealized PnL per controller, total volume -- **Error summary** — error count per active bot from live logs - -**Config overrides** (pass as `config={}` keys): -- `trading_pair` — filter to one pair (default: all) -- `connector_name` — filter to one connector (default: all) -- `recent_closes` — how many closed executors to analyze (default: 100) -- `include_errors` — set `false` to skip error log fetch (default: true) - -**After reading the output:** -1. Surface the KPIs (open positions, hold-mode count, top close type). -2. Flag any errors — if errors are present, note the bot name and count. For deeper log analysis run `manage_routines(action="run", name="logs_summary")` (global routine). -3. If hold-mode > 0 and user hasn't set it intentionally, flag it — positions holding inventory aren't earning spread. -4. Summarize PnL vs volume to comment on fee efficiency. diff --git a/condor/reports/builder.py b/condor/reports/builder.py index f4bc2388c..2109f5ad1 100644 --- a/condor/reports/builder.py +++ b/condor/reports/builder.py @@ -118,9 +118,15 @@ def section(self, title: str, description: str | None = None) -> ReportBuilder: ) return self - def plotly(self, fig: Any, optimize: bool = True) -> ReportBuilder: + def plotly(self, fig: Any, optimize: bool = True, width: int = 12) -> ReportBuilder: """Attach a Plotly figure. + ``width`` is the figure's span of the report's 12-column grid, the same + knob every data-bound component already takes. The default spans the + row, exactly as before. A narrower one lets two figures sit side by + side and, below the layout's 800px breakpoint, stack on their own — + which a single figure split internally cannot do. + Large figures are re-encoded for display (see :mod:`condor.reports.figure_opt`): evenly spaced x arrays collapse to ``x0``/``dx`` and dense traces move to the WebGL renderer, which keeps a @@ -139,7 +145,13 @@ def plotly(self, fig: Any, optimize: bool = True) -> ReportBuilder: ) else: content = fig.to_html(full_html=False, include_plotlyjs=False) - self._sections.append({"type": "plotly", "content": content}) + self._sections.append( + { + "type": "plotly", + "content": content, + "width": max(1, min(12, int(width))), + } + ) return self def table( @@ -573,9 +585,21 @@ def _render_sections(self) -> str: ) index += 1 elif section["type"] == "plotly": - parts.append( - f'
{section["content"]}
' - ) + # A full-width figure keeps the plain section markup it has + # always had. A narrower one becomes a grid item instead, so it + # spans its columns and collapses to full width on a narrow + # screen; `.report-grid > .section` would otherwise force it + # back to the whole row. + span = section.get("width", 12) + if span < 12: + parts.append( + f'
{section["content"]}
' + ) + else: + parts.append( + f'
{section["content"]}
' + ) index += 1 elif section["type"] == "table": parts.append(self._render_table(section["columns"], section["rows"])) diff --git a/docs/market_making_fly_design.md b/docs/market_making_fly_design.md index 2ea1d3c13..7c2f3737a 100644 --- a/docs/market_making_fly_design.md +++ b/docs/market_making_fly_design.md @@ -254,7 +254,7 @@ source: * `equity_t = realized_pnl + unrealized_pnl − fees` for **this bot's** `pmm_mister` controller, read from `manage_bots(action="status")` - performance (the same fields `mm_dashboard` reads), so nothing else on the + performance (`realized_pnl_quote` + `unrealized_pnl_quote`), so nothing else on the account can reward or punish the fly. * `delta = equity_t − anchor` where `anchor` is the previous observation's equity. `delta ≥ +deadband` → `reward` (PAM11 ×15), `≤ −deadband` → @@ -362,13 +362,11 @@ agents/market_making_fly/ fly_status.py # one-shot: latest posture, channels, z-scores, spikes, memory stats, P&L, vetoes, halt reason fly_report.py # one-shot: the run dashboard (fly, book, neurons, decisions, sensory frame) hip3_market_scanner.py # copied from Market Making Expert - mm_dashboard.py # copied skills/ fly_mm_deploy/SKILL.md # deploy playbook (adapted from pmm_mister_deploy: scanner → base config → deploy → start fly_brain) fly_decoder/SKILL.md # how to read fly_status and the decoder pmm_config_playbook/ # copied capital_allocation/ # copied - mm_bot_report/ # copied strategies/fly_hip3_operator/strategy.md # thin loop: keep bot + fly alive, surface halts, rotate when flat agents/market_making_fly/tests/ # conftest puts the agent dir on sys.path @@ -408,7 +406,7 @@ Two modes, like Market Making Expert: * **Delegated / loop**: `fly_mm_deploy` skill end-to-end: scanner → TOP PICK → base config from `pmm_config_playbook` balanced profile adapted with HIP-3 bounds → deploy with `max_global_drawdown_quote` → start `fly_brain` - (shadow first unless told live) → verify with `mm_bot_report`. The loop + (shadow first unless told live) → verify with `fly_report`. The loop strategy ticks every 5 min: confirms bot and fly instance alive, surfaces halts/vetoes, rotates when flat and the market is closed. diff --git a/tests/test_report_builder.py b/tests/test_report_builder.py index d82379ed7..e707ff7b3 100644 --- a/tests/test_report_builder.py +++ b/tests/test_report_builder.py @@ -409,3 +409,30 @@ def test_hydrate_is_a_no_op_for_a_pre_existing_inlined_report(reports_dir): """The 100 reports already on disk need no migration to keep working.""" legacy = "" assert rendering.hydrate(legacy) == legacy + + +def test_plotly_width_spans_the_grid_and_defaults_to_the_full_row(): + """A narrow figure has to leave the plain `.section` markup behind: + `.report-grid > .section` forces `grid-column: 1 / -1`, which would beat + `.report-component`'s span and silently keep the figure full width.""" + import plotly.graph_objects as go + + from condor.reports import ReportBuilder + + def html_for(**kwargs): + builder = ReportBuilder("t") + builder.plotly(go.Figure(go.Scatter(x=[1, 2], y=[1, 2])), **kwargs) + return builder._render_sections() + + full = html_for() + assert 'class="section plotly-chart report-panel"' in full + assert "--component-span" not in full + + half = html_for(width=6) + assert "--component-span:6" in half + assert 'class="section plotly-chart' not in half # or the grid rule wins + assert "report-component" in half + + # out-of-range widths are clamped rather than emitted as broken CSS + assert "--component-span" not in html_for(width=99) # clamps to the full row + assert "--component-span:1" in html_for(width=0) From dae1768f23af60946ad86ef61c286879beb32155 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 10:12:49 -0700 Subject: [PATCH 20/48] (feat) one market scanner for every venue, ranked against its own fee MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The HIP-3 scanner's ranking was never Hyperliquid-specific — volume, spread, drift and depth are what decides whether a book is worth quoting anywhere. Only its data path was. Two things made generalizing it worth doing rather than duplicating. Condor's own fetch_tickers turns out to enumerate HIP-3 markets: 285 issuer-prefixed pairs come back for hyperliquid_perpetual alongside its native ones, with volume. So one call replaces the Hyperliquid-only metaAndAssetCtxs for enumeration, on every venue. Book depth goes through LiveMarket.levels, which already knew which source serves which market, so that rule now lives in exactly one place and book() delegates to it. And the threshold gets an actual basis. The old routine required 3 bp, a number unrelated to what trading costs. A market now has to clear a multiple of that venue's own round-trip maker fee: 3.9 bp on a HIP-3 perp, 22.5 bp on Binance spot. Quoting inside the fee loses money on every fill, and the old constant could not tell those venues apart. Nothing else in the library did this job — market_scanner has no spread, depth or fee term at all, market_analyzer reads one pair's regime, arb_check compares one pair across venues — so hip3_market_scanner leaves this agent (Market Making Expert keeps its own copy). One honest limit, documented and given a knob: reading a book costs a call, so only the busiest markets get read, and the widest markets are rarely the busiest. Scanning HIP-3 live, the eight heaviest markets all quote under 1.4 bp; the first that cleared the fee sat thirtieth by volume, at 8.18 bp and 3.15x the round trip. The report says how deep it looked and why every rejected market failed. Verified live on two venues: HIP-3 (120 listed, one survivor) and Binance perpetual (713 listed, none — its top books quote 0.01-1.2 bp against a 6 bp floor, which is the true answer, not a broken path). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XoYbpk9wCJBNcSjz4TMJWG --- agents/market_making_fly/AGENT.md | 2 +- agents/market_making_fly/flybrain/market.py | 86 ++++- .../routines/hip3_market_scanner.py | 325 ---------------- .../routines/mm_market_scanner.py | 348 ++++++++++++++++++ .../skills/fly_mm_deploy/SKILL.md | 31 +- .../tests/test_fly_market.py | 49 +++ docs/market_making_fly_design.md | 7 +- 7 files changed, 500 insertions(+), 348 deletions(-) delete mode 100644 agents/market_making_fly/routines/hip3_market_scanner.py create mode 100644 agents/market_making_fly/routines/mm_market_scanner.py diff --git a/agents/market_making_fly/AGENT.md b/agents/market_making_fly/AGENT.md index 3fdb45394..440f0531b 100644 --- a/agents/market_making_fly/AGENT.md +++ b/agents/market_making_fly/AGENT.md @@ -87,7 +87,7 @@ manage_skill(action="read", name="fly_mm_deploy") | Routine | Use | |---|---| | `fly_setup` | `action=prepare` downloads and compiles the connectome into this agent's home (once per install); `verify`; `bench` | -| `hip3_market_scanner` | Rank xyz HIP-3 markets; take the top picks and their spreads | +| `mm_market_scanner` | Rank any venue's markets for market making: spread against that venue's round-trip fee, book depth, drift. Works on HIP-3 too | | `fly_chart` | Render the exact frame for a pair (what the fly sees) | | `fly_brain` | The loop (continuous). `mode=shadow|live`, `pairs`, `picked_spreads_bps`, `run_name` | | `fly_status` | Latest posture per pair, last observation, guard state, memory stats | diff --git a/agents/market_making_fly/flybrain/market.py b/agents/market_making_fly/flybrain/market.py index 4e26e6d3a..abfeed580 100644 --- a/agents/market_making_fly/flybrain/market.py +++ b/agents/market_making_fly/flybrain/market.py @@ -49,6 +49,68 @@ def open(self) -> bool: return self.bid is not None and self.ask is not None +def level(entry) -> tuple[float, float]: + """One book level as ``(price, size)``. Venues disagree on the shape: + Hyperliquid sends ``{"px", "sz"}``, hummingbot-api sends ``[price, qty]``, + and some connectors send ``{"price", "quantity"}``.""" + if isinstance(entry, dict): + price = entry.get("px", entry.get("price", entry.get("Price"))) + size = entry.get( + "sz", entry.get("quantity", entry.get("size", entry.get("amount"))) + ) + return float(price), float(size) + return float(entry[0]), float(entry[1]) + + +def depth_within( + bids: list[tuple[float, float]], + asks: list[tuple[float, float]], + within_bps: float, +) -> tuple[float, float, float]: + """``(bid_notional, ask_notional, spread_bps)`` inside ``within_bps`` of mid. + + What a market maker actually needs to know about a book: how wide the touch + is, and how much is resting close enough to trade against. Levels further + out than the band are not liquidity this strategy will ever see. + """ + if not bids or not asks: + return 0.0, 0.0, 0.0 + best_bid, best_ask = bids[0][0], asks[0][0] + mid = (best_bid + best_ask) / 2 + if mid <= 0 or not math.isfinite(mid): + return 0.0, 0.0, 0.0 + spread_bps = (best_ask - best_bid) / mid * 1e4 + + def side(levels, is_bid): + total = 0.0 + for price, size in levels: + offset = (mid - price) / mid * 1e4 if is_bid else (price - mid) / mid * 1e4 + if offset > within_bps: + break # the ladder is sorted; nothing beyond is closer + total += price * size + return total + + return side(bids, True), side(asks, False), spread_bps + + +async def fetch_l2_levels( + session: aiohttp.ClientSession, coin: str +) -> tuple[list[tuple[float, float]], list[tuple[float, float]]]: + """Full bid/ask ladders from Hyperliquid's public book.""" + async with session.post( + HL_INFO_URL, + json={"type": "l2Book", "coin": coin}, + timeout=aiohttp.ClientTimeout(total=10), + ) as resp: + if resp.status != 200: + raise RuntimeError(f"l2Book {coin}: HTTP {resp.status}") + book = await resp.json() + levels = book.get("levels") if isinstance(book, dict) else None + if not levels or len(levels) != 2: + raise RuntimeError(f"l2Book {coin}: unexpected payload {str(book)[:120]}") + return [level(e) for e in levels[0]], [level(e) for e in levels[1]] + + async def fetch_l2_book(session: aiohttp.ClientSession, coin: str) -> Book: async with session.post( HL_INFO_URL, @@ -92,26 +154,36 @@ def __init__( self.candle_interval = candle_interval self.n_candles = n_candles - async def book(self, pair: str) -> Book: - """Top of book, from whichever source serves this market. + async def levels( + self, pair: str, depth: int = 50 + ) -> tuple[list[tuple[float, float]], list[tuple[float, float]]]: + """Bid/ask ladders, from whichever source serves this market. hummingbot-api covers every CLOB connector it supports. HIP-3 pairs are the exception: its order-book endpoint 500s on them, so those go - straight to Hyperliquid's public one. + straight to Hyperliquid's public one. That rule lives here alone, so + anything reading a book — the loop, the scanner — inherits it. """ names = pair_names(pair) if names.hl_coin and "hyperliquid" in self.connector_name.lower(): async with aiohttp.ClientSession() as session: - return await fetch_l2_book(session, names.hl_coin) + return await fetch_l2_levels(session, names.hl_coin) raw = await self.client.market_data.get_order_book( - self.connector_name, pair, depth=1 + self.connector_name, pair, depth=depth ) if not isinstance(raw, dict): raise RuntimeError(f"{pair}: unexpected order book payload") - bids, asks = raw.get("bids") or [], raw.get("asks") or [] + return ( + [level(e) for e in (raw.get("bids") or [])], + [level(e) for e in (raw.get("asks") or [])], + ) + + async def book(self, pair: str) -> Book: + """Top of book. Source selection lives in :meth:`levels`.""" + bids, asks = await self.levels(pair, depth=1) if not bids or not asks: return Book(None, None) # closed / empty book, a real state - bid, ask = float(bids[0][0]), float(asks[0][0]) + bid, ask = bids[0][0], asks[0][0] if not (math.isfinite(bid) and math.isfinite(ask)) or bid <= 0 or ask < bid: raise RuntimeError(f"{pair}: invalid top of book bid={bid!r} ask={ask!r}") return Book(bid, ask) diff --git a/agents/market_making_fly/routines/hip3_market_scanner.py b/agents/market_making_fly/routines/hip3_market_scanner.py deleted file mode 100644 index a41c85442..000000000 --- a/agents/market_making_fly/routines/hip3_market_scanner.py +++ /dev/null @@ -1,325 +0,0 @@ -"""HIP-3 market scanner — ranks xyz-issuer perps for volume-farming market-making.""" - -import asyncio -import logging -import math - -import aiohttp -from pydantic import BaseModel, Field -from telegram.ext import ContextTypes - -from config_manager import get_client - -logger = logging.getLogger(__name__) - -CATEGORY = "Market Data" - -HL_URL = "https://api.hyperliquid.xyz/info" - - -class Config(BaseModel): - """Scan all markets of a HIP-3 builder issuer and return a shortlist for volume-farming MM.""" - - issuer: str = Field( - default="xyz", description="HIP-3 builder issuer slug (e.g. 'xyz')" - ) - min_spread_bps: float = Field( - default=3.0, description="Minimum impact spread in bps" - ) - max_daily_drift_pct: float = Field( - default=3.0, description="Maximum daily price drift %" - ) - min_oi_notional: float = Field( - default=1_000_000.0, description="Minimum open interest in USD" - ) - min_book_depth_usd: float = Field( - default=10_000.0, - description="Min resting book notional within depth_within_bps, per side (liquidity filter)", - ) - depth_within_bps: float = Field( - default=10.0, - description="Band (bps from mid) over which book depth is measured", - ) - depth_check_top_k: int = Field( - default=12, - description="How many top-scored survivors to depth-check via l2Book (bounds API calls)", - ) - top_n: int = Field(default=5, description="Number of top markets to return") - all_in_fee_bps_roundtrip: float = Field( - default=2.6, - description="Informational: total roundtrip fee in bps (~1.3bps/side)", - ) - - -async def _fetch_book_depth(session, coin, ctx_mid, within_bps): - """Return (bid_depth_usd, ask_depth_usd) resting within `within_bps` of mid. (0,0) on failure/empty.""" - try: - async with session.post( - HL_URL, - json={"type": "l2Book", "coin": coin}, - timeout=aiohttp.ClientTimeout(total=10), - ) as resp: - if resp.status != 200: - return 0.0, 0.0 - book = await resp.json() - levels = book.get("levels") if isinstance(book, dict) else None - if not levels or len(levels) != 2 or not levels[0] or not levels[1]: - return 0.0, 0.0 # empty book = closed / illiquid - bids, asks = levels[0], levels[1] - best_bid = float(bids[0]["px"]) - best_ask = float(asks[0]["px"]) - mid = (best_bid + best_ask) / 2 or ctx_mid - if mid <= 0: - return 0.0, 0.0 - - def _side(side_levels, is_bid): - tot = 0.0 - for lvl in side_levels: - px = float(lvl["px"]) - sz = float(lvl["sz"]) - off = (mid - px) / mid * 1e4 if is_bid else (px - mid) / mid * 1e4 - if off > within_bps: - break - tot += px * sz - return tot - - return _side(bids, True), _side(asks, False) - except Exception as e: - logger.warning(f"l2Book depth fetch failed for {coin}: {e}") - return 0.0, 0.0 - - -async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: - # Client is optional — only used for report persistence, not for the scan itself. - client = await get_client(context._chat_id, context=context) - - issuer = config.issuer.lower() - issuer_upper = issuer.upper() - - # ── 1. Fetch universe + contexts from Hyperliquid public API ────────────── - payload = {"type": "metaAndAssetCtxs", "dex": issuer} - try: - async with aiohttp.ClientSession() as session: - async with session.post( - HL_URL, json=payload, timeout=aiohttp.ClientTimeout(total=10) - ) as resp: - if resp.status != 200: - return f"Hyperliquid API error: HTTP {resp.status}" - data = await resp.json() - except Exception as e: - return f"Failed to fetch Hyperliquid data: {e}" - - if not isinstance(data, list) or len(data) < 2: - return f"Unexpected API response shape: {type(data)}" - - meta, ctxs = data[0], data[1] - universe = meta.get("universe", []) - if not universe: - return f"No markets found for issuer '{issuer}'" - - # ── 2. Compute per-market metrics ───────────────────────────────────────── - markets = [] - for asset, ctx in zip(universe, ctxs): - try: - name = asset.get("name", "") # e.g. "xyz:SMSN" (l2Book coin) - pair = name.upper() + "-USD" # e.g. "XYZ:SMSN-USD" (trading_pair) - - mid_raw = ctx.get("midPx") - mark_raw = ctx.get("markPx") - prev_raw = ctx.get("prevDayPx") - impact_pxs = ctx.get("impactPxs") - - volume = float(ctx.get("dayNtlVlm", 0) or 0) - mid = float(mid_raw) if mid_raw else 0.0 - mark = float(mark_raw) if mark_raw else 0.0 - prev = float(prev_raw) if prev_raw else 0.0 - - open_market = ( - mid > 0 and isinstance(impact_pxs, list) and len(impact_pxs) == 2 - ) - - spread_bps = None - if open_market: - try: - bid_impact = float(impact_pxs[0]) - ask_impact = float(impact_pxs[1]) - spread_bps = (ask_impact - bid_impact) / mid * 1e4 - except (ValueError, TypeError, ZeroDivisionError): - open_market = False - - daily_drift_pct = abs(mark / prev - 1) * 100 if prev else 999.0 - oi_notional = float(ctx.get("openInterest", 0) or 0) * mark - - markets.append( - { - "pair": pair, - "coin": name, - "volume": volume, - "mid": mid, - "mark": mark, - "spread_bps": spread_bps, - "daily_drift_pct": daily_drift_pct, - "oi_notional": oi_notional, - "book_depth_usd": None, - "open_market": open_market, - "maxLeverage": int(asset.get("maxLeverage", 0)), - "funding": ctx.get("funding", "N/A"), - } - ) - except Exception as e: - logger.warning(f"Error processing market {asset.get('name', '?')}: {e}") - - total_markets = len(markets) - - # ── 3. Pre-filters (open / spread / drift / OI) ─────────────────────────── - prelim = [ - m - for m in markets - if ( - m["open_market"] - and m["spread_bps"] is not None - and m["spread_bps"] >= config.min_spread_bps - and m["daily_drift_pct"] <= config.max_daily_drift_pct - and m["oi_notional"] >= config.min_oi_notional - ) - ] - - # ── 4. Score and rank (before depth check) ──────────────────────────────── - for m in prelim: - m["score"] = ( - math.log(max(m["volume"], 1)) - + 0.3 * min(m["spread_bps"], 8.0) - - 0.4 * m["daily_drift_pct"] - ) - prelim.sort(key=lambda m: m["score"], reverse=True) - - # ── 5. LIQUIDITY FILTER — real book depth on the top-scored candidates ──── - # Only depth-check the top_k (bounds l2Book calls; the rest can't outrank them anyway). - candidates = prelim[: config.depth_check_top_k] - if candidates: - try: - async with aiohttp.ClientSession() as session: - depths = await asyncio.gather( - *[ - _fetch_book_depth( - session, m["coin"], m["mid"], config.depth_within_bps - ) - for m in candidates - ] - ) - for m, (bid_d, ask_d) in zip(candidates, depths): - # Require BOTH sides liquid for two-sided MM → use the weaker side. - m["book_depth_usd"] = min(bid_d, ask_d) - except Exception as e: - logger.warning(f"Depth-check batch failed: {e}") - - survivors = [ - m - for m in candidates - if m["book_depth_usd"] is not None - and m["book_depth_usd"] >= config.min_book_depth_usd - ] - survivors.sort(key=lambda m: m["score"], reverse=True) - shortlist = survivors[: config.top_n] - - # ── 6. Fallback if zero survivors ───────────────────────────────────────── - no_survivors = len(survivors) == 0 - fallback = [] - if no_survivors: - fallback = sorted(markets, key=lambda m: m["volume"], reverse=True)[:5] - - # ── 7. Build summary ────────────────────────────────────────────────────── - top_pick = shortlist[0]["pair"] if shortlist else "NONE" - - lines = [ - f"**HIP-3 Market Scanner — issuer: {issuer_upper}**", - f"Scanned: {total_markets} markets | Pre-filter pass: {len(prelim)} | Depth-checked: {len(candidates)} | Survivors: {len(survivors)} | Top-{config.top_n} shown", - f"Filters: spread >= {config.min_spread_bps}bps | drift <= {config.max_daily_drift_pct}% | OI >= ${config.min_oi_notional:,.0f} | depth >= ${config.min_book_depth_usd:,.0f}/side within {config.depth_within_bps}bps", - f"Fee context: {config.all_in_fee_bps_roundtrip}bps round-trip (~{config.all_in_fee_bps_roundtrip / 2:.2f}bps/side)", - "", - ] - - def _mrow(rank, m, note=""): - spd = f"{m['spread_bps']:.2f}" if m["spread_bps"] is not None else "N/A" - dep = ( - f"${m['book_depth_usd']:,.0f}" - if m.get("book_depth_usd") is not None - else "n/a" - ) - flag = f" [{note}]" if note else "" - score_str = f" | Score={m['score']:.3f}" if "score" in m else "" - return ( - f" {rank}. {m['pair']}: Vol=${m['volume']:,.0f} | Spread={spd}bps" - f" | Drift={m['daily_drift_pct']:.2f}% | Depth={dep}/side | OI=${m['oi_notional']:,.0f}" - f" | Lev={m['maxLeverage']}x{score_str}{flag}" - ) - - if no_survivors: - lines.append("WARNING: NONE PASSED FILTERS — top 5 by volume (informational):") - for rank, m in enumerate(fallback, 1): - lines.append(_mrow(rank, m, "NO FILTER PASS")) - else: - lines.append(f"TOP PICK: {top_pick}") - lines.append("") - for rank, m in enumerate(shortlist, 1): - lines.append(_mrow(rank, m)) - - summary = "\n".join(lines) - - # ── 8. Persistent report ────────────────────────────────────────────────── - from condor.reports import ReportBuilder - - builder = ReportBuilder(f"HIP-3 Scanner: {issuer_upper}") - builder.source("routine", "hip3_market_scanner").tags( - ["market-making", "hip3", issuer, "scanner"] - ) - builder.kpi("Markets Scanned", str(total_markets)) - builder.kpi("Survivors", str(len(survivors))) - builder.kpi("Top Pick", top_pick) - builder.kpi("Fee RT", f"{config.all_in_fee_bps_roundtrip}bps") - - display_list = shortlist if not no_survivors else fallback - note_col = "Score" if not no_survivors else "Note" - table_rows = [] - for rank, m in enumerate(display_list, 1): - table_rows.append( - { - "Rank": rank, - "Pair": m["pair"], - "24h Vol ($)": f"${m['volume']:,.0f}", - "Spread (bps)": ( - f"{m['spread_bps']:.2f}" if m["spread_bps"] is not None else "N/A" - ), - "Drift %": f"{m['daily_drift_pct']:.2f}%", - "Depth/side ($)": ( - f"${m['book_depth_usd']:,.0f}" - if m.get("book_depth_usd") is not None - else "n/a" - ), - "OI ($)": f"${m['oi_notional']:,.0f}", - "MaxLev": m["maxLeverage"], - note_col: f"{m['score']:.3f}" if "score" in m else "NO FILTER PASS", - } - ) - - if table_rows: - builder.table( - table_rows, - [ - "Rank", - "Pair", - "24h Vol ($)", - "Spread (bps)", - "Drift %", - "Depth/side ($)", - "OI ($)", - "MaxLev", - note_col, - ], - ) - - builder.markdown(summary) - builder.manual_order() - await builder.save() - - return summary diff --git a/agents/market_making_fly/routines/mm_market_scanner.py b/agents/market_making_fly/routines/mm_market_scanner.py new file mode 100644 index 000000000..1f234e0b6 --- /dev/null +++ b/agents/market_making_fly/routines/mm_market_scanner.py @@ -0,0 +1,348 @@ +"""Rank the markets of any CLOB venue for market making, spot or perp. + +The question a market maker asks before quoting is not "is this liquid" but +"is the spread wide enough to clear what a round trip costs here, and is there +depth to quote into". Nothing else in the library answers it: the global +``market_scanner`` profiles volume and volatility with no spread, depth or fee +term at all, ``market_analyzer`` reads one pair's regime, and ``arb_check`` +compares one pair across venues. + +This is the HIP-3 scanner's ranking generalized. Two things change: + +* **Where the numbers come from.** That routine reads one Hyperliquid call + which returns every market in an issuer's dex. Condor's own ticker fetcher + turns out to enumerate the same markets — 285 issuer-prefixed pairs come + back for ``hyperliquid_perpetual`` alongside its native ones — so one code + path now covers HIP-3 and every other venue. Book depth comes from + ``LiveMarket.levels``, which already knows which source serves which market. +* **What "wide enough" means.** That routine hardcodes 3 bp, which is unrelated + to what trading costs. Here the threshold is a multiple of the venue's own + round-trip maker fee, and that fee differs by a factor of five between a + HIP-3 perp and a spot book. + +One honest limit. Reading a book costs a call, so only the top markets by +volume get read — and the widest markets are rarely the busiest. Scanning the +120 HIP-3 markets, the eight heaviest all quote under 1.4 bp, far too tight to +clear a 2.6 bp round trip; the first market that cleared it sat thirtieth by +volume. So ``prescreen`` is the knob that matters when nothing survives, and +the report says how deep it looked. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + +import asyncio +import logging +import math + +from flybrain import venue +from flybrain.market import LiveMarket, depth_within +from flybrain.naming import pair_names +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.fetchers.market_data import fetch_tickers +from condor.reports import ReportBuilder + +logger = logging.getLogger(__name__) + +CATEGORY = "Market Data" + +# The spread term saturates: past this, more spread says more about how thin +# the book is than about how much a maker earns. +SPREAD_SCORE_CAP_BPS = 8.0 + + +class Config(BaseModel): + """Rank a venue's markets for market making: spread vs fee, depth, drift.""" + + connector_name: str = Field( + default="hyperliquid_perpetual", description="Any CLOB connector, spot or perp" + ) + pairs: str = Field( + default="", + description="Comma-separated pairs to rank; blank ranks everything the venue lists", + ) + quote: str = Field( + default="", + description="Only pairs quoted in this asset, e.g. USDT (blank = any)", + ) + issuer: str = Field( + default="", + description="Only pairs with this issuer prefix, e.g. xyz for HIP-3 (blank = any)", + ) + maker_fee_bps: float = Field( + default=0.0, description="Maker fee per side in bp; 0 uses the venue default" + ) + min_spread_over_fee: float = Field( + default=1.5, + description="Require the spread to be this multiple of the round-trip fee", + ) + min_volume_usd: float = Field(default=250_000.0, description="Minimum 24h volume") + max_daily_drift_pct: float = Field( + default=3.0, description="Maximum 24h price drift %" + ) + min_book_depth_usd: float = Field( + default=10_000.0, + description="Minimum resting notional per side, within the band", + ) + depth_within_bps: float = Field( + default=10.0, description="Band around mid for depth" + ) + prescreen: int = Field( + default=30, + ge=1, + le=120, + description="How many of the highest-volume markets to read the book of. " + "Raise it when nothing survives: the widest markets are rarely the " + "busiest, so a low prescreen sees only tight books", + ) + top_n: int = Field(default=5, ge=1, le=25, description="Markets to report") + + +async def _measure(market: LiveMarket, pair: str, config: Config, sem) -> dict: + """Spread, depth and drift for one market. Never raises: a market whose + book or candles cannot be read is reported as unreadable, not dropped + silently and not allowed to kill the scan.""" + row: dict = {"pair": pair} + async with sem: + try: + bids, asks = await market.levels(pair, depth=50) + bid_usd, ask_usd, spread_bps = depth_within( + bids, asks, config.depth_within_bps + ) + row.update( + { + "spread_bps": spread_bps, + "bid_depth_usd": bid_usd, + "ask_depth_usd": ask_usd, + "open": bool(bids and asks), + } + ) + except Exception as failure: # external feed, one market + row["error"] = repr(failure)[:80] + return row + try: + raw = await market.client.market_data.get_candles( + config.connector_name, pair, interval="1h", max_records=25 + ) + rows = raw if isinstance(raw, list) else raw.get("data", raw.get("candles")) + closes = [float(c["close"]) for c in (rows or []) if c.get("close")] + row["drift_pct"] = ( + abs(closes[-1] / closes[0] - 1) * 100 if len(closes) >= 2 else None + ) + except Exception as failure: + row["drift_pct"] = None + row["drift_error"] = repr(failure)[:60] + return row + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + from config_manager import get_client + + client = await get_client(context._chat_id, context=context) + if not client: + return "No server available" + + market_type = venue.market_type_for(config.connector_name) + fee_bps = config.maker_fee_bps or venue.default_maker_fee_bps( + config.connector_name, market_type + ) + round_trip_bps = 2 * fee_bps + min_spread_bps = round_trip_bps * config.min_spread_over_fee + + # ── 1. Enumerate and screen on volume, which is one call ────────────────── + if config.pairs.strip(): + candidates = {p.strip(): None for p in config.pairs.split(",") if p.strip()} + tickers = (await fetch_tickers(client, config.connector_name)).get( + "tickers" + ) or {} + volumes = { + p: float((tickers.get(p) or {}).get("usd_volume", 0) or 0) + for p in candidates + } + else: + tickers = (await fetch_tickers(client, config.connector_name)).get( + "tickers" + ) or {} + if not tickers: + return ( + f"No tickers for {config.connector_name}; is the connector supported?" + ) + volumes = {} + for pair, row in tickers.items(): + try: + names = pair_names(pair) + except ValueError: + continue # not a shape this agent can quote + if config.quote and names.quote != config.quote.upper(): + continue + if config.issuer and names.issuer != config.issuer.lower(): + continue + volumes[pair] = float((row or {}).get("usd_volume", 0) or 0) + + listed = len(volumes) + screened = sorted( + ((p, v) for p, v in volumes.items() if v >= config.min_volume_usd), + key=lambda kv: -kv[1], + )[: config.prescreen] + if not screened: + return ( + f"{config.connector_name}: {listed} markets listed, none above " + f"${config.min_volume_usd:,.0f} of 24h volume" + ) + + # ── 2. Book and drift, only for what survived — this is the expensive part ─ + market = LiveMarket(client, config.connector_name, "1h", 25) + sem = asyncio.Semaphore(6) + measured = await asyncio.gather( + *(_measure(market, pair, config, sem) for pair, _ in screened) + ) + by_pair = {m["pair"]: m for m in measured} + + rows = [] + for pair, volume in screened: + m = by_pair[pair] + drift = m.get("drift_pct") + depth = min(m.get("bid_depth_usd", 0.0), m.get("ask_depth_usd", 0.0)) + spread = m.get("spread_bps", 0.0) + reasons = [] + if m.get("error"): + reasons.append("book unreadable") + elif not m.get("open"): + reasons.append("book closed") + if spread < min_spread_bps: + reasons.append(f"spread {spread:.1f} < {min_spread_bps:.1f} bp") + if depth < config.min_book_depth_usd: + reasons.append(f"depth ${depth:,.0f}") + if drift is None: + reasons.append("no candles") + elif drift > config.max_daily_drift_pct: + reasons.append(f"drift {drift:.1f}%") + rows.append( + { + "pair": pair, + "volume": volume, + "spread_bps": spread, + "spread_over_fee": spread / round_trip_bps if round_trip_bps else 0.0, + "depth_usd": depth, + "drift_pct": drift, + "survives": not reasons, + "why_not": ", ".join(reasons), + "score": ( + math.log(max(volume, 1)) + + 0.3 * min(spread, SPREAD_SCORE_CAP_BPS) + - 0.4 * (drift if drift is not None else 99.0) + ), + } + ) + survivors = sorted((r for r in rows if r["survives"]), key=lambda r: -r["score"])[ + : config.top_n + ] + rejected = sorted( + (r for r in rows if not r["survives"]), key=lambda r: -r["volume"] + ) + + builder = ReportBuilder(f"MM markets — {config.connector_name}") + builder.source("routine", "mm_market_scanner") + builder.tags(["market-making", "scanner", config.connector_name]) + builder.manual_order() + builder.section( + "WHAT A ROUND TRIP COSTS HERE", + f"{market_type} venue · maker {fee_bps:.2f} bp a side · round trip " + f"{round_trip_bps:.2f} bp · a market must quote at least " + f"{min_spread_bps:.2f} bp to clear it by {config.min_spread_over_fee}×", + ) + builder.kpi("Venue", config.connector_name) + builder.kpi("Type", market_type) + builder.kpi("Maker fee", f"{fee_bps:.2f} bp") + builder.kpi("Round trip", f"{round_trip_bps:.2f} bp") + builder.kpi("Spread floor", f"{min_spread_bps:.2f} bp") + builder.kpi("Listed", f"{listed:,}") + builder.kpi("Book-checked", str(len(screened))) + builder.kpi("Survivors", str(len([r for r in rows if r["survives"]]))) + + builder.section( + "RANKED", f"Top {len(survivors)} by score — volume, spread over fee, and drift" + ) + builder.table( + [ + { + "Pair": r["pair"], + "24h volume": f"${r['volume']:,.0f}", + "Spread": f"{r['spread_bps']:.2f} bp", + "× round trip": f"{r['spread_over_fee']:.2f}×", + "Depth/side": f"${r['depth_usd']:,.0f}", + "Drift": ( + f"{r['drift_pct']:.2f}%" if r["drift_pct"] is not None else "—" + ), + "Score": f"{r['score']:.2f}", + } + for r in survivors + ] + or [{"Pair": "— none survived —"}], + [ + "Pair", + "24h volume", + "Spread", + "× round trip", + "Depth/side", + "Drift", + "Score", + ], + ) + builder.section("REJECTED", "Why each screened market did not make it") + builder.table( + [ + { + "Pair": r["pair"], + "24h volume": f"${r['volume']:,.0f}", + "Spread": f"{r['spread_bps']:.2f} bp", + "Depth/side": f"${r['depth_usd']:,.0f}", + "Reason": r["why_not"], + } + for r in rejected + ] + or [{"Pair": "— none rejected —"}], + ["Pair", "24h volume", "Spread", "Depth/side", "Reason"], + ) + builder.markdown( + f"_Books were read for the {len(screened)} highest-volume markets of " + f"{listed} listed. The widest markets are rarely the busiest, so raise " + "`prescreen` when nothing survives — it is the only thing standing " + "between this scan and the rest of the venue._" + ) + builder.markdown( + "_A wide spread is necessary, not sufficient: it is often wide because the " + "book is thin, which is why depth is filtered separately. Drift is a proxy " + "for how much inventory a maker would accumulate against the trend, not a " + "forecast. Nothing here says a market is profitable._" + ) + await builder.save() + + lines = [ + f"venue: {config.connector_name} ({market_type})", + f"maker_fee_bps: {fee_bps:.2f} (round trip {round_trip_bps:.2f})", + f"spread_floor_bps: {min_spread_bps:.2f}", + f"listed: {listed}, book_checked: {len(screened)}, survivors: {len(survivors)}", + ] + for n, r in enumerate(survivors, 1): + lines.append( + f"{n}. {r['pair']}: spread {r['spread_bps']:.2f} bp " + f"({r['spread_over_fee']:.2f}× round trip), depth ${r['depth_usd']:,.0f}/side, " + f"vol ${r['volume']:,.0f}, drift " + + (f"{r['drift_pct']:.2f}%" if r["drift_pct"] is not None else "—") + ) + if survivors: + lines.append( + f"TOP PICK: {survivors[0]['pair']} at {survivors[0]['spread_bps']:.2f} bp" + ) + else: + lines.append("TOP PICK: none — no market cleared the fee with depth behind it") + return "\n".join(lines) diff --git a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md index f2c4b98bd..76ecd3e3c 100644 --- a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md +++ b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md @@ -28,21 +28,28 @@ the plumbing. ## Step 1 — Pick the markets -On **Hyperliquid HIP-3**, rank them: - ``` -manage_routines(action="run", name="hip3_market_scanner", - config={"issuer": "xyz", "min_spread_bps": 3, "max_daily_drift_pct": 3, "top_n": 5}) +manage_routines(action="run", name="mm_market_scanner", config={ + "connector_name": "", "issuer": "xyz", # issuer only on HIP-3 + "quote": "USDT", # optional, e.g. spot venues + "prescreen": 30, "top_n": 5}) ``` -On **any other venue**, either take the pairs the operator named, or rank -candidates with the global `market_scanner` routine and read the spread from -`get_prices` plus the order book. - -Take the top **`n_markets`** (1-3; from `[CURRENT CONFIG]` or the task, default 3) -that have an open live book — one brain quotes them all in round-robin. Record for -each: `pair` (uppercase `BASE-QUOTE`, or `ISSUER:TOKEN-QUOTE` on HIP-3) and its -**spread in bp** — this is `picked_spreads_bps`. If none survive, stop and report. +It ranks by volume, by how far the spread clears **that venue's** round-trip +maker fee, and by book depth, then reports why every rejected market failed. +Take the top **`n_markets`** (1-3; from `[CURRENT CONFIG]` or the task, default +3) with an open book — one brain quotes them all in round-robin. Record each +`pair` and its **spread in bp**; that is `picked_spreads_bps`. + +**If nothing survives, raise `prescreen` before anything else.** Reading a book +costs a call, so only the busiest markets are read — and the widest markets are +rarely the busiest. On HIP-3 the eight heaviest markets all quote under 1.4 bp, +nowhere near the 3.9 bp needed to clear a 2.6 bp round trip; the first market +that cleared it sat thirtieth by volume. A `TOP PICK: none` line means the scan +did not look far enough, or this venue is genuinely too tight to quote. + +If it still finds nothing, stop and report that rather than lowering the +spread floor: quoting inside the fee loses money on every fill. ## Step 2 — Collateral diff --git a/agents/market_making_fly/tests/test_fly_market.py b/agents/market_making_fly/tests/test_fly_market.py index cb932dd81..47603dabe 100644 --- a/agents/market_making_fly/tests/test_fly_market.py +++ b/agents/market_making_fly/tests/test_fly_market.py @@ -162,3 +162,52 @@ def test_a_hip3_pair_does_not_use_the_generic_endpoint(): except (aiohttp.ClientError, RuntimeError, OSError): pass # offline in CI; the point is which path was taken assert m.client.market_data.calls == [] + + +def test_a_book_level_is_read_whatever_shape_the_venue_sends(): + from flybrain.market import level + + assert level({"px": "100.5", "sz": "2"}) == (100.5, 2.0) # Hyperliquid + assert level([100.5, 2]) == (100.5, 2.0) # hummingbot-api + assert level({"price": 100.5, "quantity": 2}) == (100.5, 2.0) + assert level({"price": 100.5, "amount": 2}) == (100.5, 2.0) + + +def test_depth_only_counts_what_is_close_enough_to_trade_against(): + """Liquidity resting far from mid is not liquidity this strategy sees.""" + from flybrain.market import depth_within + + bids = [(99.9, 10), (99.0, 100), (90.0, 1000)] # ~10bp, ~100bp, ~1000bp out + asks = [(100.1, 10), (101.0, 100), (110.0, 1000)] + bid_usd, ask_usd, spread = depth_within(bids, asks, within_bps=20) + assert spread == pytest.approx(20.0, abs=0.1) + assert bid_usd == pytest.approx(99.9 * 10) # the 99.0 level is ~100bp out + assert ask_usd == pytest.approx(100.1 * 10) + # a wider band reaches further down the ladder + wide_bid, wide_ask, _ = depth_within(bids, asks, within_bps=150) + assert wide_bid > bid_usd and wide_ask > ask_usd + # an empty side is no depth and no spread, not a crash + assert depth_within([], asks, 20) == (0.0, 0.0, 0.0) + + +def test_the_scanner_requires_the_spread_to_clear_the_venues_own_fee(): + """The HIP-3 scanner hardcoded 3bp, unrelated to what a round trip costs. + A spot book's fee is several times a perp's, so the floor has to move.""" + import importlib.util + from pathlib import Path + + from flybrain import venue + + path = Path(__file__).resolve().parents[1] / "routines" / "mm_market_scanner.py" + spec = importlib.util.spec_from_file_location("mm_market_scanner", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + + def floor_for(connector): + cfg = mod.Config(connector_name=connector) + fee = venue.default_maker_fee_bps(connector, venue.market_type_for(connector)) + return 2 * fee * cfg.min_spread_over_fee + + assert floor_for("hyperliquid_perpetual") == pytest.approx(3.9) + assert floor_for("binance") == pytest.approx(22.5) # spot costs far more + assert floor_for("binance") > 5 * floor_for("hyperliquid_perpetual") / 2 diff --git a/docs/market_making_fly_design.md b/docs/market_making_fly_design.md index 7c2f3737a..0bd0f1063 100644 --- a/docs/market_making_fly_design.md +++ b/docs/market_making_fly_design.md @@ -312,8 +312,9 @@ agnostic: a candle chart is a candle chart. * Connector `hyperliquid_perpetual`, issuer `xyz`, pairs `XYZ:TOKEN-USD` (uppercase; lowercase → KeyError → zero orders). -* Market selection is the existing `hip3_market_scanner` routine (copied into - the agent): volume, spread-vs-fee, daily drift, `l2Book` depth filter, +* Market selection is the `mm_market_scanner` routine, which ranks any venue's + markets on the same criteria (superseding the HIP-3-only scanner this agent + used to carry): volume, spread-vs-fee, daily drift, `l2Book` depth filter, `TOP PICK`. Selection is the operator's job (deterministic routine + LLM reading it), not the fly's — the fly never chooses which market it is shown, as in stonkfly's fixed round-robin. @@ -361,7 +362,7 @@ agents/market_making_fly/ fly_brain.py # CONTINUOUS: the loop in §3; modes shadow|live, learning|frozen, fixture|market fly_status.py # one-shot: latest posture, channels, z-scores, spikes, memory stats, P&L, vetoes, halt reason fly_report.py # one-shot: the run dashboard (fly, book, neurons, decisions, sensory frame) - hip3_market_scanner.py # copied from Market Making Expert + mm_market_scanner.py # ranks any venue: spread vs its round-trip fee, depth, drift skills/ fly_mm_deploy/SKILL.md # deploy playbook (adapted from pmm_mister_deploy: scanner → base config → deploy → start fly_brain) fly_decoder/SKILL.md # how to read fly_status and the decoder From e4d0950530f5d069da15e4a8fb3da67b7ab0f370 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 10:35:03 -0700 Subject: [PATCH 21/48] (feat) the connectome, drawn, and coloured by what just fired MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The panel carried fifteen cards and no picture. It now carries six cards and the animal. The graph the fly runs on is pure topology — no coordinates. The MaleCNS annotations have them: 139,662 of the 166,700 retained neurons have a soma position, so this draws the fly's actual anatomy rather than a layout invented to look plausible. flybrain/cloud.py caches a deterministic 7,573-point sample, stratified rather than uniform: uniform would be all optic lobe, which is half the animal, and populations of four and six would vanish, so every cell the decoder and the memory rule read is kept whole and the rest is sampled in proportion with floors. Colour is what actually fired. The worker now carries spike counts at those neurons back from each observation, and the loop writes them to one overwritten file rather than burying events.jsonl. Active-neuron count and mean rate are measured over the whole network, not extrapolated from the sample, because a fraction of a stratified sample is not a fraction of the animal. Silent and firing cells are drawn as separate traces. Nineteen in twenty are silent in a 500ms window, and running them through the same colour scale turned the anatomy into a bright haze that buried the cells that had done something. Layout follows the fly-and-frame panel above it: cards take five of the runtime's twelve columns and wrap two per row, the brain takes seven, and both collapse to full width under its 800px breakpoint. That needed the same width knob on ReportBuilder.kpi that plotly just got — wrapped rather than applied directly, since `.report-grid > .kpi-bar` would beat the span and the inner bar still needs its own card grid. Default is unchanged, so every existing report renders as before. A READOUT panel below shows the three channels the posture is decoded from: DNp20 left against right, whose difference is the entire lean, and the descending mean that sets spread width. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XoYbpk9wCJBNcSjz4TMJWG --- agents/market_making_fly/flybrain/brainviz.py | 256 ++++++++++++++++++ agents/market_making_fly/flybrain/cloud.py | 156 +++++++++++ .../market_making_fly/flybrain/run_state.py | 15 + agents/market_making_fly/flybrain/worker.py | 12 + .../market_making_fly/routines/fly_brain.py | 5 +- .../market_making_fly/routines/fly_report.py | 80 +++--- .../tests/test_fly_report.py | 55 ++++ condor/reports/builder.py | 30 +- tests/test_report_builder.py | 21 ++ 9 files changed, 594 insertions(+), 36 deletions(-) create mode 100644 agents/market_making_fly/flybrain/brainviz.py create mode 100644 agents/market_making_fly/flybrain/cloud.py diff --git a/agents/market_making_fly/flybrain/brainviz.py b/agents/market_making_fly/flybrain/brainviz.py new file mode 100644 index 000000000..024274cc2 --- /dev/null +++ b/agents/market_making_fly/flybrain/brainviz.py @@ -0,0 +1,256 @@ +"""The connectome, drawn, and coloured by what just fired. + +The anatomy comes from :mod:`flybrain.cloud` — real soma coordinates from the +MaleCNS release, not a layout invented to look plausible. The colour comes from +the spike counts the last observation actually produced, so the mushroom body +lighting up is the mushroom body having fired, not an illustration of the idea. +""" + +from __future__ import annotations + +import numpy as np +from flybrain.cloud import GROUPS, load, orient +from flybrain.fly3d import GROUND, LIMB + +# Silent slate through to a hot filament: a neuron that did nothing should +# recede into the anatomy, and one that fired hard should be the brightest +# thing on the panel. +ACTIVITY_SCALE = [ + [0.0, "#1b2536"], + [0.12, "#2f4a6d"], + [0.35, "#4f8fd0"], + [0.62, "#ff9a54"], + [0.85, "#ffd166"], + [1.0, "#fff6e0"], +] + +# Used only when a run has no activity snapshot: colour by what a cell is +# rather than by what it did. +GROUP_COLORS = { + "mushroom body": "#ff9a54", + "dopamine": "#ff4d6d", + "memory output": "#ffd166", + "readout": "#c9f24d", + "descending": "#7ee0ff", + "visual": "#5aa0ff", + "other": "#55637d", +} + + +def brain_figure( + activity: list[int] | None = None, + title: str = "", + height: int = 460, +): + """The somata of the retained graph, orbitable, coloured by firing.""" + import plotly.graph_objects as go + + cloud = load() + points = orient(cloud["xyz"]) + group = cloud["group"] + hidden = dict( + showbackground=False, + showgrid=False, + zeroline=False, + showticklabels=False, + visible=False, + ) + layout = dict( + height=height, + margin=dict(l=0, r=0, t=36 if title else 6, b=6), + paper_bgcolor=GROUND, + plot_bgcolor=GROUND, + font=dict(color="#c8d4e8", family="monospace", size=11), + title=( + dict( + text=title, x=0.02, xanchor="left", font=dict(color="#c9f24d", size=13) + ) + if title + else None + ), + scene=dict( + xaxis=hidden, + yaxis=hidden, + zaxis=hidden, + bgcolor=GROUND, + aspectmode="data", + camera=dict(eye=dict(x=0, y=-1.15, z=0.22), up=dict(x=0, y=0, z=1)), + ), + legend=dict( + orientation="h", + yanchor="top", + y=-0.02, + xanchor="center", + x=0.5, + font=dict(size=10), + ), + ) + + if activity is not None and len(activity) == len(group): + rate = np.asarray(activity, dtype=float) + firing = rate > 0 + ceiling = float(np.percentile(rate[firing], 97)) if firing.any() else 1.0 + shade = np.clip(rate / (ceiling or 1.0), 0, 1) + # Silent and firing are drawn separately on purpose. Nineteen cells in + # twenty are silent in a given window, and running them through the + # same colour scale turns the anatomy into a bright haze that buries + # the few cells that actually did something. + quiet = go.Scatter3d( + x=points[~firing, 0], + y=points[~firing, 1], + z=points[~firing, 2], + mode="markers", + marker=dict(size=1.1, color="#1d2738", opacity=0.5), + hoverinfo="skip", + name="silent", + showlegend=False, + ) + order = np.argsort(shade[firing]) # brightest drawn last + live = go.Scatter3d( + x=points[firing][order, 0], + y=points[firing][order, 1], + z=points[firing][order, 2], + mode="markers", + marker=dict( + size=2.4 + 4.2 * shade[firing][order], + color=shade[firing][order], + colorscale=ACTIVITY_SCALE, + cmin=0, + cmax=1, + opacity=0.95, + showscale=False, + ), + text=[ + f"{GROUPS[g]} · {int(c)} spikes" + for g, c in zip(group[firing][order], rate[firing][order]) + ], + hoverinfo="text", + name="firing", + showlegend=False, + ) + fig = go.Figure([quiet, live]) + fig.update_layout(showlegend=False, **layout) + return fig + + # No activity recorded for this run: show what each cell is instead. + traces = [] + for slot, name in enumerate(GROUPS): + picked = group == slot + if not picked.any(): + continue + traces.append( + go.Scatter3d( + x=points[picked, 0], + y=points[picked, 1], + z=points[picked, 2], + mode="markers", + marker=dict( + size=1.6 if name in ("visual", "other") else 3.0, + color=GROUP_COLORS[name], + opacity=0.55 if name in ("visual", "other") else 0.95, + ), + name=name, + hoverinfo="name", + ) + ) + fig = go.Figure(traces) + fig.update_layout(showlegend=True, **layout) + return fig + + +def coverage() -> tuple[int, int, int]: + """``(drawn, mapped somata, retained neurons)`` — what the picture covers.""" + cloud = load() + return len(cloud["index"]), cloud["mapped_total"], cloud["neurons_total"] + + +def readout_figure(neural: dict, posture: dict | None = None, height: int = 460): + """The three channels the posture is decoded from, as they were measured. + + DNp20 left against DNp20 right is the whole of the lean: the posture reads + their difference, not either one. The descending population mean is the + arousal that sets spread width. The gate is a count, not a rate, so it is + stated rather than drawn — a bar of 4 next to a bar of 38 Hz would invite + a comparison that means nothing. + """ + import plotly.graph_objects as go + + left = float(neural.get("left_hz") or 0.0) + right = float(neural.get("right_hz") or 0.0) + arousal = float(neural.get("arousal_hz") or 0.0) + posture = posture or {} + labels = ["DNp20 left", "DNp20 right", "descending mean"] + values = [left, right, arousal] + colors = ["#5aa0ff", "#7ee0ff", "#ff9a54"] + + fig = go.Figure( + go.Bar( + x=values, + y=labels, + orientation="h", + marker=dict(color=colors), + text=[f"{v:.1f} Hz" for v in values], + textposition="outside", + textfont=dict(color="#c8d4e8", size=11), + hoverinfo="skip", + ) + ) + gate = neural.get("gate_spikes") + trend = right - left + fig.update_layout( + height=height, + margin=dict(l=96, r=20, t=36, b=64), + paper_bgcolor=GROUND, + plot_bgcolor=GROUND, + font=dict(color="#c8d4e8", family="monospace", size=11), + title=dict( + text="READOUT", x=0.02, xanchor="left", font=dict(color="#c9f24d", size=13) + ), + xaxis=dict( + title="firing rate (Hz)", + gridcolor="#18202e", + zerolinecolor="#243044", + range=[0, max(values + [1]) * 1.28], + ), + yaxis=dict(gridcolor="#18202e", automargin=True), + showlegend=False, + legend=dict(orientation="h", yanchor="top", y=-0.15, xanchor="center", x=0.5), + annotations=[ + dict( + x=0, + y=-0.30, + xref="paper", + yref="paper", + showarrow=False, + align="left", + xanchor="left", + font=dict(color=LIMB, size=11, family="monospace"), + text=( + f"right − left = {trend:+.1f} Hz" + + ( + f" (z {posture['trend_z']:+.2f})" + if "trend_z" in posture + else "" + ) + + f"
arousal z {posture.get('arousal_z', float('nan')):+.2f}" + if "arousal_z" in posture + else f"right − left = {trend:+.1f} Hz" + ), + ), + dict( + x=1, + y=-0.30, + xref="paper", + yref="paper", + showarrow=False, + align="right", + xanchor="right", + font=dict( + color="#c9f24d" if gate else LIMB, size=11, family="monospace" + ), + text=f"DNpe017 gate: {gate if gate is not None else '—'} spike(s)" + + ("" if gate else " — no lean without it"), + ), + ], + ) + return fig diff --git a/agents/market_making_fly/flybrain/cloud.py b/agents/market_making_fly/flybrain/cloud.py new file mode 100644 index 000000000..0a7005151 --- /dev/null +++ b/agents/market_making_fly/flybrain/cloud.py @@ -0,0 +1,156 @@ +"""The connectome's somata, as a point cloud the report can draw. + +The graph the fly runs on carries no coordinates — it is pure topology. The +MaleCNS annotations do: 139,662 of the 166,700 retained neurons have a soma +position, which is enough to draw the animal's actual anatomy rather than a +made-up layout. + +166,700 points would not survive a report, so this builds a deterministic +subsample once and caches it beside the graph. The sample is stratified rather +than uniform: every cell the decoder and the memory rule actually read — Kenyon +cells, the two dopamine populations, their MBONs, the DNp20/DNpe017 readouts — +is kept in full, because those are the ones worth looking at. The rest is a +seeded draw, so the same install always draws the same brain. +""" + +from __future__ import annotations + +import numpy as np +from flybrain.neural.common import DATA, GRAPH + +CLOUD = DATA / "soma_cloud.npz" +SEED = 7301 # the interface seed, so the whole model shares one +DEFAULT_POINTS = 7000 + +# Groups are coarse on purpose: the point is where activity sits in the +# animal, not a taxonomy. Order is the colour order in the report. +GROUPS = ( + "mushroom body", # Kenyon cells — where the memory rule acts + "dopamine", # PAM11 / PPL101 — the reward and aversive populations + "memory output", # MBON07 / MBON11 + "readout", # DNp20 / DNpe017 — what the posture is decoded from + "descending", # the arousal channel + "visual", # optic lobe and visual projection — what the chart drives + "other", +) + + +def _positions(ids: np.ndarray) -> np.ndarray: + """Soma coordinates per graph index; NaN where the release has none.""" + import pyarrow.feather as feather + + table = feather.read_table( + DATA / "annotations.feather", columns=["bodyId", "somaLocation"] + ) + frame = table.to_pandas().set_index("bodyId").reindex(ids) + out = np.full((len(ids), 3), np.nan, dtype=np.float32) + for row, value in enumerate(frame.somaLocation.to_numpy()): + if value is None: + continue + try: + out[row] = np.asarray(value, dtype=np.float32)[:3] + except (TypeError, ValueError): + continue + return out + + +def _group_of(ids: np.ndarray, superclass: np.ndarray) -> np.ndarray: + """One coarse group per neuron, identified cells taking precedence.""" + import pyarrow.feather as feather + from flybrain.worker import DESCENDING_SUPERCLASS + + types = ( + feather.read_table(DATA / "annotations.feather", columns=["bodyId", "type"]) + .to_pandas() + .set_index("bodyId") + .reindex(ids) + .type.fillna("") + .to_numpy() + .astype(str) + ) + group = np.full(len(ids), GROUPS.index("other"), dtype=np.int8) + visual = np.isin( + superclass, + ["ol_intrinsic", "ol_sensory", "visual_projection", "visual_centrifugal"], + ) + group[visual] = GROUPS.index("visual") + group[superclass == DESCENDING_SUPERCLASS] = GROUPS.index("descending") + group[np.char.startswith(types, "KC")] = GROUPS.index("mushroom body") + group[np.isin(types, ["MBON07", "MBON11"])] = GROUPS.index("memory output") + group[np.isin(types, ["PAM11", "PPL101"])] = GROUPS.index("dopamine") + group[np.isin(types, ["DNp20", "DNpe017"])] = GROUPS.index("readout") + return group + + +def build(max_points: int = DEFAULT_POINTS) -> dict: + """Subsample the somata and cache the result. Idempotent.""" + with np.load(GRAPH) as graph: + ids = graph["ids"] + superclass = graph["superclass"].astype(str) + xyz = _positions(ids) + group = _group_of(ids, superclass) + mapped = np.flatnonzero(np.isfinite(xyz).all(axis=1)) + + # Sample per group, not uniformly. Uniform would be all optic lobe — it is + # half the animal — and the circuit that actually decides anything would + # vanish. Proportional keeps the silhouette recognisable; the floors keep + # the mushroom body and the descending channel dense enough to watch; the + # handful of identified cells are always kept whole. + rng = np.random.default_rng(SEED) + floors = { + GROUPS.index("mushroom body"): 420, + GROUPS.index("descending"): 420, + } + always = [GROUPS.index(name) for name in ("dopamine", "memory output", "readout")] + picked = [mapped[np.isin(group[mapped], always)]] + budget = max_points - len(picked[0]) + pool = { + g: mapped[group[mapped] == g] for g in range(len(GROUPS)) if g not in always + } + total = sum(len(v) for v in pool.values()) or 1 + for g, members in pool.items(): + want = min( + len(members), max(floors.get(g, 0), round(budget * len(members) / total)) + ) + picked.append( + members + if want >= len(members) + else rng.choice(members, size=want, replace=False) + ) + index = np.sort(np.concatenate(picked)) + + np.savez_compressed( + CLOUD, + index=index.astype(np.int32), + xyz=xyz[index], + group=group[index], + mapped_total=np.int64(len(mapped)), + neurons_total=np.int64(len(ids)), + ) + return load() + + +def load(max_points: int = DEFAULT_POINTS) -> dict: + """The cached cloud, building it on first use.""" + if not CLOUD.exists(): + return build(max_points) + with np.load(CLOUD) as data: + return { + "index": data["index"], + "xyz": data["xyz"], + "group": data["group"], + "mapped_total": int(data["mapped_total"]), + "neurons_total": int(data["neurons_total"]), + } + + +def orient(xyz: np.ndarray) -> np.ndarray: + """Centre the cloud and put the animal the way an atlas figure does. + + MaleCNS voxel axes are (x right, y down, z front-to-back). Negating y puts + dorsal up, so the brain reads as a brain rather than as its own reflection. + """ + out = np.stack([xyz[:, 0], xyz[:, 2], -xyz[:, 1]], axis=1) + centre = np.nanmean(out, axis=0) + scale = np.nanmax(np.abs(out - centre)) + return (out - centre) / (scale or 1.0) diff --git a/agents/market_making_fly/flybrain/run_state.py b/agents/market_making_fly/flybrain/run_state.py index 85051f8c8..59e013b6d 100644 --- a/agents/market_making_fly/flybrain/run_state.py +++ b/agents/market_making_fly/flybrain/run_state.py @@ -80,6 +80,7 @@ def __init__(self, root: Path): self.events_path = self.root / "events.jsonl" self.latest_path = self.root / "latest.json" self.frame_path = self.root / "latest-input.png" + self.activity_path = self.root / "latest-activity.json" self.provenance_path = self.root / "provenance.json" self.lock_path = self.root / "worker.lock" self.stop_path = self.root / "STOP" @@ -122,6 +123,20 @@ def append_event(self, row: dict) -> None: def write_latest(self, row: dict) -> None: atomic_write_json(self.latest_path, row, indent=2, default=str) + def save_activity(self, counts: list[int]) -> None: + """Spike counts at the cloud's neurons, for the report to colour by. + + Overwritten each observation rather than appended: it is a few thousand + numbers, which would bury `events.jsonl` within an hour, and only the + latest is ever drawn. + """ + atomic_write_json(self.activity_path, counts) + + def load_activity(self) -> list[int] | None: + if not self.activity_path.exists(): + return None + return json.loads(self.activity_path.read_text()) + def save_frame(self, frame: np.ndarray) -> None: Image.fromarray(frame).save(self.frame_path) diff --git a/agents/market_making_fly/flybrain/worker.py b/agents/market_making_fly/flybrain/worker.py index 16da9bd81..eae599d17 100644 --- a/agents/market_making_fly/flybrain/worker.py +++ b/agents/market_making_fly/flybrain/worker.py @@ -57,6 +57,12 @@ def __init__( f"No {DESCENDING_SUPERCLASS!r} superclass in graph.npz; " f"available: {sorted(set(superclass.tolist()))}" ) + # The report draws the connectome's somata; carrying their spike counts + # back is what lets it colour the animal by what actually fired, rather + # than by cell class alone. A few thousand ints per observation. + from flybrain.cloud import load as load_cloud + + self.cloud_index = load_cloud()["index"].astype(np.int64) self.cell_ids = { "left": [str(self.brain.ids[i]) for i in self.left], "right": [str(self.brain.ids[i]) for i in self.right], @@ -105,6 +111,11 @@ def observe( "right_hz": right, "arousal_hz": float(np.mean(counts[self.descending]) / seconds), "gate_spikes": int(counts[self.gate].sum()), + # Over the whole network, not extrapolated from the drawn sample: + # the report's cloud is stratified, so a fraction of it would not be + # a fraction of the animal. + "active_neurons": int((counts > 0).sum()), + "mean_rate_hz": float(counts.mean() / seconds), "kc_spikes": int(counts[b.circuit["kc"]].sum()), "reward_spikes": int(counts[b.circuit["reward"]].sum()), "aversive_spikes": int(counts[b.circuit["aversive"]].sum()), @@ -117,6 +128,7 @@ def observe( "input_sha256": hashlib.sha256(frame.tobytes()).hexdigest(), "memory": b.memory(), "cell_ids": self.cell_ids, + "activity": counts[self.cloud_index].astype(int).tolist(), } def save(self, path: Path) -> None: diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index a7009d075..05472d1eb 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -472,7 +472,10 @@ async def pace() -> None: tick += 1 persist({"checkpoint": {"file": ck.name, "sha256": sha}}) - neural_row = {k: v for k, v in neural.items() if k != "cell_ids"} + run_dir.save_activity(neural["activity"]) + neural_row = { + k: v for k, v in neural.items() if k not in ("cell_ids", "activity") + } row.update({"neural": neural_row, "posture": posture.to_dict()}) now = time.time() diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index d9eca9d24..85b5fe95d 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -24,6 +24,7 @@ import numpy as np import plotly.graph_objects as go +from flybrain.brainviz import brain_figure, coverage, readout_figure from flybrain.fly3d import ACCENT, BODY, GROUND, LIMB, fly_figure from flybrain.market import LiveMarket from flybrain.naming import pair_names @@ -38,6 +39,11 @@ logger = logging.getLogger(__name__) CATEGORY = "Bot Analysis" + +# Columns of the report's 12-wide grid. The cards take the narrower half so +# they wrap to two per row beside the brain, and both collapse to full width +# under the runtime's 800px breakpoint. +CARDS, BRAIN = 5, 7 AGENT_SLUG = "market_making_fly" # How an execution status reads in the decision log. @@ -263,59 +269,67 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: ) # ── NEURONS & NEURAL ORDER ─────────────────────────────────────────────── - # One panel: the connectome's numbers and the posture they were decoded - # into belong together — the second is only readable against the first. + # Six cards, not fifteen. The rest of the numbers are narrative, and the + # picture says more about where the activity sat than a card ever could. + drawn, mapped, neurons_total = coverage() + active = neural.get("active_neurons") builder.section( "NEURONS & NEURAL ORDER", - "The connectome's own numbers from the last observation, and the posture " - "decoded from them" + f"{neurons_total:,} neurons · {mapped:,} mapped somata · {drawn:,} drawn" + ( - f" (tick {observed.get('tick')}; the newest tick ran no brain)" + f" · observation {observed.get('tick')}; the newest tick ran no brain" if stale else "" ), ) - builder.kpi("Neurons", f"{neurons:,}" if neurons else "—") - builder.kpi( - "Latest spikes", - f"{neural.get('total_spikes'):,}" if neural.get("total_spikes") else "—", - ) + # Cards on the left, the brain on the right, the same way the fly and its + # frame sit above — five columns and seven of the runtime's twelve, both + # collapsing to full width below its 800px breakpoint. builder.kpi( - "Memory changed", + "Active neurons", ( - f"{memory.get('changed_edges'):,}" - if memory.get("changed_edges") is not None + f"{active:,} · {100 * active / neurons_total:.1f}%" + if active is not None else "—" ), + width=CARDS, ) builder.kpi( - "Brain time", - f"{float(neural['brain_ms']) / 1000:,.1f} s" if neural.get("brain_ms") else "—", + "Mean firing rate", + f"{neural['mean_rate_hz']:.1f} Hz" if neural.get("mean_rate_hz") else "—", + width=CARDS, ) + builder.kpi("Regime", str(last_posture.get("regime", "—")).upper(), width=CARDS) + builder.kpi("Spread ×", _fmt(last_posture.get("spread_mult")), width=CARDS) builder.kpi( - "Kenyon cells", - f"{neural.get('kc_spikes'):,}" if neural.get("kc_spikes") is not None else "—", + "Lean", + f"{_fmt(last_posture.get('shift_bps'), 2, plus=True)} bp", + width=CARDS, ) - builder.kpi("Gate (DNpe017)", str(neural.get("gate_spikes", "—"))) - builder.kpi("Reward (PAM11)", str(neural.get("reward_spikes", "—"))) - builder.kpi("Aversive (PPL101)", str(neural.get("aversive_spikes", "—"))) - builder.kpi("Mean efficacy", _fmt(memory.get("mean_efficacy"), 5)) - if last_posture: - builder.kpi("Regime", str(last_posture.get("regime", "—")).upper()) - builder.kpi("Spread ×", _fmt(last_posture.get("spread_mult"))) - builder.kpi("Lean", f"{_fmt(last_posture.get('shift_bps'), 2, plus=True)} bp") - builder.kpi("Trend z", _fmt(last_posture.get("trend_z"), 2, plus=True)) - builder.kpi("Arousal z", _fmt(last_posture.get("arousal_z"), 2, plus=True)) - builder.kpi( - "Result", - RESULT_WORDS.get(execution.get("status"), execution.get("status", "—")), - ) + builder.kpi( + "Result", + RESULT_WORDS.get(execution.get("status"), execution.get("status", "—")), + width=CARDS, + ) + builder.plotly( + brain_figure(run_dir.load_activity(), title="NEURAL ACTIVITY"), width=BRAIN + ) + builder.plotly(readout_figure(neural, last_posture, height=260)) builder.markdown( f"**{str(last_posture.get('regime', 'no posture yet')).upper()}** on " f"`{observed.get('pair', '—')}` — {execution.get('reason', 'nothing recorded')}. " - f"Stimulus `{observed.get('stimulus', 'none')}`, P&L delta " - f"{_fmt(latest.get('pnl_delta'), 4, plus=True)}." + f"Decoded from trend z {_fmt(last_posture.get('trend_z'), 2, plus=True)} and " + f"arousal z {_fmt(last_posture.get('arousal_z'), 2, plus=True)}, gated on " + f"{neural.get('gate_spikes', '—')} DNpe017 spike(s)." + ("" if last_posture.get("warm", True) else " _Baseline still forming._") + + f"\n\nThe observation ran {_fmt(float(neural['brain_ms']) / 1000, 1) if neural.get('brain_ms') else '—'} s " + f"of neural time and produced {neural.get('total_spikes', 0):,} spikes, " + f"{neural.get('kc_spikes', 0):,} of them in Kenyon cells. Stimulus " + f"`{observed.get('stimulus', 'none')}` — {neural.get('reward_spikes', 0)} PAM11 " + f"and {neural.get('aversive_spikes', 0)} PPL101 spikes — left " + f"{(memory.get('changed_edges') or 0):,} of {(memory.get('plastic_edges') or 0):,} " + f"plastic edges away from baseline, mean efficacy " + f"{_fmt(memory.get('mean_efficacy'), 5)}." ) # ── DECISIONS & POSITIONS ──────────────────────────────────────────────── diff --git a/agents/market_making_fly/tests/test_fly_report.py b/agents/market_making_fly/tests/test_fly_report.py index 9dd4a626d..72dcc33fc 100644 --- a/agents/market_making_fly/tests/test_fly_report.py +++ b/agents/market_making_fly/tests/test_fly_report.py @@ -100,3 +100,58 @@ def test_every_execution_status_has_a_word(): "TICK_ERROR", ): assert report.RESULT_WORDS[status] + + +def test_the_cloud_is_real_anatomy_and_always_holds_the_identified_circuit(): + """The graph carries no coordinates; these come from the release's soma + positions. The cells the decoder and the memory rule read are kept whole, + because a proportional sample would drop populations of four and six.""" + from flybrain.cloud import GROUPS, load + + cloud = load() + index, xyz, group = cloud["index"], cloud["xyz"], cloud["group"] + assert len(index) == len(xyz) == len(group) + assert np.isfinite(xyz).all() # every drawn point has a real position + assert len(np.unique(index)) == len(index) + assert index.max() < cloud["neurons_total"] + assert cloud["mapped_total"] < cloud["neurons_total"] # not every cell is mapped + # the tiny identified populations survive sampling + for name, expected in (("dopamine", 17), ("memory output", 6), ("readout", 4)): + assert int((group == GROUPS.index(name)).sum()) == expected + # and the silhouette is not swamped by one group + assert int((group == GROUPS.index("visual")).sum()) > 1000 + + +def test_orient_centres_and_normalizes_without_distorting(): + from flybrain.cloud import orient + + raw = np.array([[0, 0, 0], [100, 200, 300], [-100, -200, -300]], dtype=np.float32) + out = orient(raw) + assert np.abs(out).max() == pytest.approx(1.0) + assert np.allclose(out.mean(axis=0), 0, atol=1e-6) + # one scale for all axes: a per-axis scale would stretch the animal + spans = (raw.max(axis=0) - raw.min(axis=0))[[0, 2, 1]] + got = out.max(axis=0) - out.min(axis=0) + assert np.allclose(got / got[0], spans / spans[0], atol=1e-5) + + +def test_silent_neurons_are_drawn_apart_from_firing_ones(): + """Nineteen cells in twenty are silent; running them through the same + colour scale turns the anatomy into a haze that buries the live ones.""" + from flybrain.brainviz import brain_figure + from flybrain.cloud import load + + n = len(load()["index"]) + activity = [0] * n + activity[0] = 40 + activity[1] = 5 + fig = brain_figure(activity) + assert len(fig.data) == 2 + quiet, live = fig.data + assert len(quiet.x) == n - 2 and len(live.x) == 2 + assert quiet.marker.size < live.marker.size.min() # silent recede + assert live.marker.color.max() == pytest.approx(1.0) # hottest is the top colour + + # with no snapshot at all it falls back to colouring by cell class + classed = brain_figure(None) + assert len(classed.data) > 2 and classed.layout.showlegend diff --git a/condor/reports/builder.py b/condor/reports/builder.py index 2109f5ad1..6346559da 100644 --- a/condor/reports/builder.py +++ b/condor/reports/builder.py @@ -88,8 +88,22 @@ def auto_refresh(self, seconds: int | None) -> ReportBuilder: return self def kpi( - self, label: str, value: str, delta: str | None = None, trend: str = "neutral" + self, + label: str, + value: str, + delta: str | None = None, + trend: str = "neutral", + width: int = 12, ) -> ReportBuilder: + """Add a KPI card. + + ``width`` is the span of the report's 12-column grid taken by the whole + run of consecutive cards, the first card's value winning. The default + spans the row as before; a narrower one lets the cards sit beside a + figure and, below the layout's 800px breakpoint, stack under it. The + cards keep their own auto-fitting grid inside that span, so a narrow + run simply wraps to fewer per row. + """ self._sections.append( { "type": "kpi", @@ -97,6 +111,7 @@ def kpi( "value": value, "delta": delta, "trend": trend, + "width": max(1, min(12, int(width))), } ) return self @@ -577,7 +592,18 @@ def _render_sections(self) -> str: f'
{html.escape(str(kpi["value"]))}
' f"{delta_html}" ) - parts.append(f'
{"".join(cards)}
') + # A full-width run keeps the markup it has always had. A + # narrower one is wrapped in a grid item rather than given the + # span itself: `.report-grid > .kpi-bar` would beat it, and the + # inner bar still needs its own card grid. + span = kpis[0].get("width", 12) + bar = f'
{"".join(cards)}
' + parts.append( + bar + if span >= 12 + else f'
{bar}
' + ) elif section["type"] == "markdown": parts.append( '
' diff --git a/tests/test_report_builder.py b/tests/test_report_builder.py index e707ff7b3..d2a96a3a2 100644 --- a/tests/test_report_builder.py +++ b/tests/test_report_builder.py @@ -436,3 +436,24 @@ def html_for(**kwargs): # out-of-range widths are clamped rather than emitted as broken CSS assert "--component-span" not in html_for(width=99) # clamps to the full row assert "--component-span:1" in html_for(width=0) + + +def test_kpi_width_wraps_the_bar_so_the_grid_rule_cannot_override_it(): + """`.report-grid > .kpi-bar` forces `grid-column: 1 / -1`, so a narrow run + has to be wrapped rather than given the span itself — and the inner bar + still needs its own card grid.""" + from condor.reports import ReportBuilder + + def html_for(**kwargs): + builder = ReportBuilder("t") + builder.kpi("A", "1", **kwargs) + builder.kpi("B", "2", **kwargs) + return builder._render_sections() + + full = html_for() + assert '
' in full + assert "--component-span" not in full + + half = html_for(width=5) + assert '--component-span:5">
' in half + assert half.count('class="kpi-bar"') == 1 # one bar, wrapped once From a4b29863df3c3692a1157151463992b25fa0da5b Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 10:47:47 -0700 Subject: [PATCH 22/48] (fix) each row's two halves finish level MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both split panels were ragged: the fly stood 494px beside a 400px frame, and the brain 494px beside a 326px card stack. A panel is its figure plus 34px of padding and border. That part was easy to measure and match. The card stack is not ours to size at all — six cards at two per row is three rows of 98px plus two gaps, and the only other arrangements are one or three per row, which overshoot in both directions. So the brain has to come down to the cards rather than the cards stretch to the brain. It could not: `.plotly-chart` carries a 400px floor. That floor is there so a full-width chart is never squashed, which is right, but a panel given an explicit width was placed deliberately beside something else and has to be free to match it. Narrow panels now release it inline, which reaches only panels that opt into a width — behaviour introduced two commits ago, so nothing existing renders differently. Measured in the browser, not guessed: both rows now report identical heights, 430 and 326, and the brain fills its panel at 699x292 instead of floating in the middle of a 460px box. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XoYbpk9wCJBNcSjz4TMJWG --- .../market_making_fly/routines/fly_report.py | 36 ++++++++++++++----- condor/reports/builder.py | 7 +++- tests/test_report_builder.py | 3 ++ 3 files changed, 37 insertions(+), 9 deletions(-) diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index 85b5fe95d..acbb00013 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -40,10 +40,22 @@ CATEGORY = "Bot Analysis" -# Columns of the report's 12-wide grid. The cards take the narrower half so -# they wrap to two per row beside the brain, and both collapse to full width -# under the runtime's 800px breakpoint. +# Columns of the report's 12-wide grid, and the panel heights that make each +# row's two halves finish level on a wide screen. Everything collapses to full +# width under the runtime's 800px breakpoint, where heights stop mattering. +# +# A panel is its figure plus 34px of padding and border, measured rather than +# guessed. Narrow panels are also released from the stylesheet's 400px floor, +# which exists to stop a full-width chart being squashed and would otherwise +# stop the brain matching the card stack beside it. +FLY, FRAME = 6, 6 CARDS, BRAIN = 5, 7 +PANEL_CHROME = 34 # the panel's own padding and border, measured +# Row one is sized so the frame fills its half: 16:9 at six columns is ~335px. +ROW_ONE = 430 +# Row two matches the card stack, which is what it is — three rows of 98px +# plus two 16px gaps. Nothing about a KPI card's height is ours to set. +ROW_TWO = 3 * 98 + 2 * 16 AGENT_SLUG = "market_making_fly" # How an execution status reads in the decision log. @@ -97,11 +109,11 @@ def _read_frame(path: Path) -> np.ndarray | None: return np.asarray(Image.open(path).convert("RGB"), dtype=np.uint8) -def _frame_figure(frame: np.ndarray) -> go.Figure: +def _frame_figure(frame: np.ndarray, height: int = 340) -> go.Figure: """That frame, full size, as the report's sensory panel.""" fig = go.Figure(go.Image(z=frame)) fig.update_layout( - height=340, + height=height, margin=dict(l=0, r=0, t=6, b=6), paper_bgcolor=GROUND, plot_bgcolor=GROUND, @@ -253,11 +265,14 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: title=f"FLY.EXE — {alive}", subtitle=f"{', '.join(pairs) or 'no market'}", chart=sensory, + height=ROW_ONE - PANEL_CHROME, ), - width=7, + width=FLY, ) if sensory is not None: - builder.plotly(_frame_figure(sensory), width=5) + builder.plotly( + _frame_figure(sensory, height=ROW_ONE - PANEL_CHROME), width=FRAME + ) if sensory is not None: builder.markdown( f"**What the fly sees** — the 320×180 frame fed to the retina on tick " @@ -312,7 +327,12 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: width=CARDS, ) builder.plotly( - brain_figure(run_dir.load_activity(), title="NEURAL ACTIVITY"), width=BRAIN + brain_figure( + run_dir.load_activity(), + title="NEURAL ACTIVITY", + height=ROW_TWO - PANEL_CHROME, + ), + width=BRAIN, ) builder.plotly(readout_figure(neural, last_posture, height=260)) builder.markdown( diff --git a/condor/reports/builder.py b/condor/reports/builder.py index 6346559da..3580ce444 100644 --- a/condor/reports/builder.py +++ b/condor/reports/builder.py @@ -618,9 +618,14 @@ def _render_sections(self) -> str: # back to the whole row. span = section.get("width", 12) if span < 12: + # `min-height: 400px` keeps a full-width chart from being + # squashed, but a narrow panel was placed deliberately + # beside something else and has to be free to match it. + # Inline, so it overrides the stylesheet for this panel only. parts.append( f'
{section["content"]}
' + f'style="--component-span:{span};min-height:0">' + f'{section["content"]}
' ) else: parts.append( diff --git a/tests/test_report_builder.py b/tests/test_report_builder.py index d2a96a3a2..76b0344e0 100644 --- a/tests/test_report_builder.py +++ b/tests/test_report_builder.py @@ -430,6 +430,9 @@ def html_for(**kwargs): half = html_for(width=6) assert "--component-span:6" in half + # the stylesheet's 400px floor would stop a narrow panel matching whatever + # it sits beside, so it is released inline for that panel only + assert "min-height:0" in half and "min-height:0" not in full assert 'class="section plotly-chart' not in half # or the grid rule wins assert "report-component" in half From e5f2de30f95d1f3645ab96527d7405e317eae140 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 10:58:00 -0700 Subject: [PATCH 23/48] (fix) the prose sits under its heading, not in a box beneath the figures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both panels explained themselves twice: a terse status line under the heading, and a bordered paragraph below the figures it described. The paragraph now is the heading's description, which is where a reader looks before deciding whether to look at the picture. `section` escapes its description, so the prose is plain text — the bold and backticks it used to carry would have rendered as literal characters. The card spans move from five and seven to six and six. Cards wrap on their own 230px minimum, so a five-column stack needs a ~1150px container before two fit on a row; until then it is a single tall column towering over the brain beside it. Six needs ~970, which covers every screen meaningfully wider than the 800px breakpoint. Row one is exact and row two lands within about five pixels, since a card's height drifts slightly with the container and is not ours to set. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XoYbpk9wCJBNcSjz4TMJWG --- .../market_making_fly/routines/fly_report.py | 83 +++++++++++-------- 1 file changed, 49 insertions(+), 34 deletions(-) diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index acbb00013..4a6ba9eb6 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -49,12 +49,18 @@ # which exists to stop a full-width chart being squashed and would otherwise # stop the brain matching the card stack beside it. FLY, FRAME = 6, 6 -CARDS, BRAIN = 5, 7 +# Six and six, not five and seven: the cards wrap by their own 230px minimum, +# so a five-column stack needs a ~1150px container before it fits two per row +# and stops towering over the brain. Six needs ~970, which covers every screen +# that is meaningfully wider than the 800px breakpoint. +CARDS, BRAIN = 6, 6 PANEL_CHROME = 34 # the panel's own padding and border, measured # Row one is sized so the frame fills its half: 16:9 at six columns is ~335px. ROW_ONE = 430 -# Row two matches the card stack, which is what it is — three rows of 98px -# plus two 16px gaps. Nothing about a KPI card's height is ours to set. +# Row two matches the card stack, which is what it is: six cards at two per +# row is three rows of roughly 98px plus two 16px gaps. A card's exact height +# drifts a few pixels with the container, so this lands within ~5px rather +# than exactly — nothing about a KPI card's height is ours to set. ROW_TWO = 3 * 98 + 2 * 16 AGENT_SLUG = "market_making_fly" @@ -251,13 +257,26 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: builder.manual_order() # ── FLY.EXE ────────────────────────────────────────────────────────────── + sensory = _read_frame(run_dir.frame_path) + # The prose belongs under the heading, not in a box below the figures it + # describes. `section` escapes its description, so this is plain text — + # no markdown syntax, which would show as literal asterisks. builder.section( "FLY.EXE", f"{alive} · run {config.run_name} · tick {state.get('tick', 0)} · " - f"{len(pairs)} market{'s' if len(pairs) != 1 else ''} · drag to orbit · " - "the monitor and the panel beside it show the fly's own input frame", + f"{len(pairs)} market{'s' if len(pairs) != 1 else ''} · drag to orbit." + + ( + f" What the fly sees: the 320×180 frame fed to the retina on tick " + f"{observed.get('tick')} — {settings.get('n_candles', '?')} × " + f"{settings.get('candle_interval', '?')} candles, volume and the live " + "bid/ask — shown on its monitor and again at full size beside it. This " + "is the picture the posture below was decoded from. No quotes, " + "inventory or P&L are drawn, because those reach the fly only as " + "dopamine." + if sensory is not None + else " No input frame has been recorded for this run yet." + ), ) - sensory = _read_frame(run_dir.frame_path) # Two halves of the grid, not one figure split internally: below the # layout's 800px breakpoint these stack on their own. builder.plotly( @@ -273,29 +292,42 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: builder.plotly( _frame_figure(sensory, height=ROW_ONE - PANEL_CHROME), width=FRAME ) - if sensory is not None: - builder.markdown( - f"**What the fly sees** — the 320×180 frame fed to the retina on tick " - f"{observed.get('tick')}: {settings.get('n_candles', '?')} × " - f"{settings.get('candle_interval', '?')} candles, volume and the live " - "bid/ask, shown on its monitor and again at full size beside it. This is " - "the picture the posture below was decoded from. No quotes, inventory or " - "P&L are drawn, because those reach the fly only as dopamine." - ) # ── NEURONS & NEURAL ORDER ─────────────────────────────────────────────── # Six cards, not fifteen. The rest of the numbers are narrative, and the # picture says more about where the activity sat than a card ever could. drawn, mapped, neurons_total = coverage() active = neural.get("active_neurons") + posture_line = ( + f"{str(last_posture.get('regime', 'no posture yet')).upper()} on " + f"{observed.get('pair', '—')} — {execution.get('reason', 'nothing recorded')}. " + f"Decoded from trend z {_fmt(last_posture.get('trend_z'), 2, plus=True)} and " + f"arousal z {_fmt(last_posture.get('arousal_z'), 2, plus=True)}, gated on " + f"{neural.get('gate_spikes', '—')} DNpe017 spike(s)." + + ("" if last_posture.get("warm", True) else " Baseline still forming.") + ) + observation_line = ( + f" The observation ran " + f"{_fmt(float(neural['brain_ms']) / 1000, 1) if neural.get('brain_ms') else '—'} s " + f"of neural time and produced {neural.get('total_spikes', 0):,} spikes, " + f"{neural.get('kc_spikes', 0):,} of them in Kenyon cells. Stimulus " + f"{observed.get('stimulus', 'none')} — {neural.get('reward_spikes', 0)} PAM11 " + f"and {neural.get('aversive_spikes', 0)} PPL101 spikes — left " + f"{(memory.get('changed_edges') or 0):,} of " + f"{(memory.get('plastic_edges') or 0):,} plastic edges away from baseline, " + f"mean efficacy {_fmt(memory.get('mean_efficacy'), 5)}." + ) builder.section( "NEURONS & NEURAL ORDER", f"{neurons_total:,} neurons · {mapped:,} mapped somata · {drawn:,} drawn" + ( - f" · observation {observed.get('tick')}; the newest tick ran no brain" + f" · observation {observed.get('tick')}, the newest tick ran no brain" if stale else "" - ), + ) + + ". " + + posture_line + + observation_line, ) # Cards on the left, the brain on the right, the same way the fly and its # frame sit above — five columns and seven of the runtime's twelve, both @@ -335,23 +367,6 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: width=BRAIN, ) builder.plotly(readout_figure(neural, last_posture, height=260)) - builder.markdown( - f"**{str(last_posture.get('regime', 'no posture yet')).upper()}** on " - f"`{observed.get('pair', '—')}` — {execution.get('reason', 'nothing recorded')}. " - f"Decoded from trend z {_fmt(last_posture.get('trend_z'), 2, plus=True)} and " - f"arousal z {_fmt(last_posture.get('arousal_z'), 2, plus=True)}, gated on " - f"{neural.get('gate_spikes', '—')} DNpe017 spike(s)." - + ("" if last_posture.get("warm", True) else " _Baseline still forming._") - + f"\n\nThe observation ran {_fmt(float(neural['brain_ms']) / 1000, 1) if neural.get('brain_ms') else '—'} s " - f"of neural time and produced {neural.get('total_spikes', 0):,} spikes, " - f"{neural.get('kc_spikes', 0):,} of them in Kenyon cells. Stimulus " - f"`{observed.get('stimulus', 'none')}` — {neural.get('reward_spikes', 0)} PAM11 " - f"and {neural.get('aversive_spikes', 0)} PPL101 spikes — left " - f"{(memory.get('changed_edges') or 0):,} of {(memory.get('plastic_edges') or 0):,} " - f"plastic edges away from baseline, mean efficacy " - f"{_fmt(memory.get('mean_efficacy'), 5)}." - ) - # ── DECISIONS & POSITIONS ──────────────────────────────────────────────── # One panel: what the fly called, and what those calls left it holding. builder.section( From 81c4bd27fc0ea02d59164fb4119143ccc7ecfa96 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 11:09:59 -0700 Subject: [PATCH 24/48] (feat) panels named for what they show, and one place for the totals MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FLY.EXE becomes WHAT THE FLY SEES and NEURONS & NEURAL ORDER becomes THE FLY BRAIN. Both descriptions lose the census that repeated what the panel already showed — the run status line, the neuron counts, and the regime and pair that are already a card and a column. What is left is the part a reader cannot get from the figures: how the posture was decoded, and what the observation cost. The run state has not gone anywhere; it is the scene's own title, where it sits next to the thing it describes. DECISIONS & POSITIONS gives up its four summary cards. Net P&L and volume were never decisions, and they now open a PERFORMANCE section alongside the tick count and a trade count — the sum of each controller's close types, so round trips completed rather than orders placed. It reads as unknown, not zero, when no bot has reported. The guard's state moves into that section's description rather than five more cards. Its loss stop is deliberately not restated: the threshold is the loop's to derive, and a report that recomputes a guard rule is a second source of truth waiting to disagree. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XoYbpk9wCJBNcSjz4TMJWG --- .../market_making_fly/routines/fly_report.py | 77 ++++++++++--------- 1 file changed, 40 insertions(+), 37 deletions(-) diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index 4a6ba9eb6..ef08b2f84 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -168,11 +168,19 @@ def _pnl_figure(events: list[dict]) -> go.Figure | None: return fig -async def _holdings(client, connector_name: str, pairs: list[str]) -> list[dict]: - """What each of the fly's bots is holding right now, from the live bot.""" +async def _holdings( + client, connector_name: str, pairs: list[str] +) -> tuple[list[dict], int | None]: + """What the fly's bots hold now, and how many positions they have closed. + + The trade count is the sum of each controller's close-type counts — round + trips actually completed, not orders placed. ``None`` when no bot reported, + so the report can say "unknown" rather than "zero". + """ market = LiveMarket(client, connector_name, "5m", 72) bots = await market.bots() - rows = [] + rows: list[dict] = [] + trades: int | None = None for pair in pairs: names = pair_names(pair) running, bot = LiveMarket.find_bot(bots, names.bot_name) @@ -181,6 +189,9 @@ async def _holdings(client, connector_name: str, pairs: list[str]) -> list[dict] continue perf = (bot.get("performance") or {}).get(names.config_name) or {} inner = perf.get("performance", perf) if isinstance(perf, dict) else {} + closes = inner.get("close_type_counts") or {} + if isinstance(closes, dict): + trades = (trades or 0) + sum(int(v or 0) for v in closes.values()) positions = inner.get("positions_summary") or [] amount = sum( float(p.get("amount", 0) or 0) for p in positions if isinstance(p, dict) @@ -196,7 +207,7 @@ async def _holdings(client, connector_name: str, pairs: list[str]) -> list[dict] "Volume": _fmt(inner.get("volume_traded")), } ) - return rows + return rows, trades async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: @@ -240,7 +251,9 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: stale = observed is not latest client = await get_client(context._chat_id, context=context) - holdings = await _holdings(client, config.connector_name, pairs) if client else [] + holdings, trades = ( + await _holdings(client, config.connector_name, pairs) if client else ([], None) + ) book_net = sum( float(info.get("net", 0) or 0) for info in (latest.get("bots") or {}).values() @@ -262,19 +275,17 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: # describes. `section` escapes its description, so this is plain text — # no markdown syntax, which would show as literal asterisks. builder.section( - "FLY.EXE", - f"{alive} · run {config.run_name} · tick {state.get('tick', 0)} · " - f"{len(pairs)} market{'s' if len(pairs) != 1 else ''} · drag to orbit." - + ( - f" What the fly sees: the 320×180 frame fed to the retina on tick " - f"{observed.get('tick')} — {settings.get('n_candles', '?')} × " + "WHAT THE FLY SEES", + ( + f"The 320×180 frame fed to the retina on tick {observed.get('tick')} — " + f"{settings.get('n_candles', '?')} × " f"{settings.get('candle_interval', '?')} candles, volume and the live " "bid/ask — shown on its monitor and again at full size beside it. This " "is the picture the posture below was decoded from. No quotes, " "inventory or P&L are drawn, because those reach the fly only as " - "dopamine." + "dopamine. Drag the scene to orbit it." if sensory is not None - else " No input frame has been recorded for this run yet." + else "No input frame has been recorded for this run yet." ), ) # Two halves of the grid, not one figure split internally: below the @@ -299,8 +310,6 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: drawn, mapped, neurons_total = coverage() active = neural.get("active_neurons") posture_line = ( - f"{str(last_posture.get('regime', 'no posture yet')).upper()} on " - f"{observed.get('pair', '—')} — {execution.get('reason', 'nothing recorded')}. " f"Decoded from trend z {_fmt(last_posture.get('trend_z'), 2, plus=True)} and " f"arousal z {_fmt(last_posture.get('arousal_z'), 2, plus=True)}, gated on " f"{neural.get('gate_spikes', '—')} DNpe017 spike(s)." @@ -317,18 +326,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: f"{(memory.get('plastic_edges') or 0):,} plastic edges away from baseline, " f"mean efficacy {_fmt(memory.get('mean_efficacy'), 5)}." ) - builder.section( - "NEURONS & NEURAL ORDER", - f"{neurons_total:,} neurons · {mapped:,} mapped somata · {drawn:,} drawn" - + ( - f" · observation {observed.get('tick')}, the newest tick ran no brain" - if stale - else "" - ) - + ". " - + posture_line - + observation_line, - ) + builder.section("THE FLY BRAIN", posture_line + observation_line) # Cards on the left, the brain on the right, the same way the fly and its # frame sit above — five columns and seven of the runtime's twelve, both # collapsing to full width below its 800px breakpoint. @@ -409,10 +407,6 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: pnl = _pnl_figure(events) if pnl is not None: builder.plotly(pnl) - builder.kpi("Net P&L", _fmt(book_net, 4, plus=True)) - builder.kpi("Volume", _fmt(book_volume)) - builder.kpi("Markets", str(len(pairs))) - builder.kpi("Mode", str(latest.get("mode", "—")).upper()) if holdings: builder.table( holdings, @@ -425,12 +419,21 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: ) # ── PERFORMANCE & LIMITS ───────────────────────────────────────────────── - builder.section("PERFORMANCE & LIMITS", "What the guard is watching") - builder.kpi("Halted", halted or "no") - builder.kpi("Session high", _fmt(guard.get("session_high_net"), 4, plus=True)) - builder.kpi("Ticks since high", str(guard.get("ticks_since_high", "—"))) - builder.kpi("Applies today", str(guard.get("applies_today", 0))) - builder.kpi("Anchor", _fmt(state.get("anchor"), 4, plus=True)) + builder.section( + "PERFORMANCE", + ( + (f"Halted: {halted}. " if halted else "") + + f"Session high {_fmt(guard.get('session_high_net'), 4, plus=True)}, " + f"{guard.get('ticks_since_high', 0)} observation(s) since, " + f"{guard.get('applies_today', 0)} config change(s) applied today." + if guard.get("session_high_net") is not None + else "No P&L has been reported yet, so the breakers have nothing to judge." + ), + ) + builder.kpi("Ticks", f"{state.get('tick', 0):,}") + builder.kpi("Net P&L", _fmt(book_net, 4, plus=True)) + builder.kpi("Trades", f"{trades:,}" if trades is not None else "—") + builder.kpi("Volume", _fmt(book_volume)) builder.markdown( "_Regime, spread multiplier and lean are an engineered readout of spike counts, " "not a discovered market-making circuit. Dopamine pulses report the change in " From 05505e00d9572982e28fd7b732577cd394fba6af Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 11:22:12 -0700 Subject: [PATCH 25/48] (feat) the P&L curve sits with the numbers it summarises The equity curve was the last thing in the decisions panel, below the holdings table; the totals it plots live one section down. Move it under the performance cards so the section reads as figure-follows-numbers, and draw it to the stylesheet's 400px floor for a full-width chart instead of 260, which left a band of empty ground beneath the line. Drop the caveats paragraph from the report body. It says nothing the run itself reports, and the same sentence already ends the routine's text return and the fly_decoder skill, which is where a reader asking "is it learning" actually looks. --- .../market_making_fly/routines/fly_report.py | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index ef08b2f84..a296bb193 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -55,6 +55,7 @@ # that is meaningfully wider than the 800px breakpoint. CARDS, BRAIN = 6, 6 PANEL_CHROME = 34 # the panel's own padding and border, measured +FULL_ROW = 400 # the stylesheet's floor for a full-width chart panel # Row one is sized so the frame fills its half: 16:9 at six columns is ~335px. ROW_ONE = 430 # Row two matches the card stack, which is what it is: six cards at two per @@ -152,7 +153,7 @@ def _pnl_figure(events: list[dict]) -> go.Figure | None: ) ) fig.update_layout( - height=260, + height=FULL_ROW - PANEL_CHROME, margin=dict(l=48, r=16, t=10, b=36), paper_bgcolor=GROUND, plot_bgcolor=GROUND, @@ -404,9 +405,6 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: "Reason", ], ) - pnl = _pnl_figure(events) - if pnl is not None: - builder.plotly(pnl) if holdings: builder.table( holdings, @@ -418,7 +416,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: "running in shadow with nothing deployed._" ) - # ── PERFORMANCE & LIMITS ───────────────────────────────────────────────── + # ── PERFORMANCE ────────────────────────────────────────────────────────── builder.section( "PERFORMANCE", ( @@ -434,14 +432,9 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: builder.kpi("Net P&L", _fmt(book_net, 4, plus=True)) builder.kpi("Trades", f"{trades:,}" if trades is not None else "—") builder.kpi("Volume", _fmt(book_volume)) - builder.markdown( - "_Regime, spread multiplier and lean are an engineered readout of spike counts, " - "not a discovered market-making circuit. Dopamine pulses report the change in " - "P&L between two observations, not credit for the last posture, and synapses " - "also change from endogenous activity. No profitable learning has been " - "demonstrated: the guard and the controller's stop loss bound the loss, not the " - "fly._" - ) + pnl = _pnl_figure(events) + if pnl is not None: + builder.plotly(pnl) await builder.save() lines = [ From e4b47504f26e462bf12818fa55cd69752160a705 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 12:16:31 -0700 Subject: [PATCH 26/48] (fix) what a round trip costs is read per market, not assumed per venue MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One Hyperliquid connector serves two families of market and they do not cost the same. A core perp pays the venue's published maker rate; a HIP-3 market pays that rate scaled by its deployer's setting and again by a tenth where growth mode is on. The agent charged 1.3 bp to both, which is right for XYZ and half of what a core perp actually costs — the take-profit floor built on it would have quoted inside the fee and lost money on every fill. The fee is now read: the published schedule and the dex's own deployer scale come from the venue's info endpoint, and only Hummingbot's builder fee is named rather than fetched, because it is a constant of the client and not of the market. Checked against fills: 173 maker fills on XYZ:ORCL-USD paid 1.29 bp all-in against 1.30 computed, and Hyperliquid reports fee inclusive of the builder fee, which is why one number covers both. The scanner now ranks each market against its own round trip and prints the fee it used. Two things it was hiding: - every market was rejected for "no candles" whatever the cause. The cause was a cold candle feed: the first request for a market hummingbot-api has not seen subscribes one and times out at 30s, and the same call answers on the second attempt. It asks twice and reports the real error. - the loop crashed on its own live report before any P&L was known, because a session high of None has no format. It reads "—" until a bot reports. Also: portfolio.get_portfolio_state is gone from the api client; the rest of Condor calls get_state, and now so does the fly. --- agents/market_making_fly/flybrain/market.py | 2 +- agents/market_making_fly/flybrain/venue.py | 111 +++++++++++++++++- .../market_making_fly/routines/fly_brain.py | 29 ++++- .../routines/mm_market_scanner.py | 96 ++++++++++----- .../tests/test_fly_market.py | 4 +- .../tests/test_fly_posture.py | 4 + .../market_making_fly/tests/test_fly_venue.py | 51 ++++++++ 7 files changed, 258 insertions(+), 39 deletions(-) diff --git a/agents/market_making_fly/flybrain/market.py b/agents/market_making_fly/flybrain/market.py index abfeed580..3c71812c2 100644 --- a/agents/market_making_fly/flybrain/market.py +++ b/agents/market_making_fly/flybrain/market.py @@ -311,7 +311,7 @@ async def available_quote(self, quote_tokens: set[str]) -> float: the asset the pair is denominated in, so the caller names it rather than this assuming a venue's collateral token. """ - state = await self.client.portfolio.get_portfolio_state() + state = await self.client.portfolio.get_state() if not isinstance(state, dict): raise RuntimeError("Portfolio state unavailable") total = 0.0 diff --git a/agents/market_making_fly/flybrain/venue.py b/agents/market_making_fly/flybrain/venue.py index 99106ab43..16623df88 100644 --- a/agents/market_making_fly/flybrain/venue.py +++ b/agents/market_making_fly/flybrain/venue.py @@ -9,7 +9,9 @@ * **the maker fee** — the take-profit floor is derived from it. Spot fees run three to five times perp fees on the same exchange, so a take-profit that is comfortably profitable on a perp loses money on spot. Getting this wrong is - silent: the bot fills happily and bleeds the difference. + silent: the bot fills happily and bleeds the difference. On Hyperliquid it + also differs *within* one connector by a factor of two, so that venue's fee + is fetched per market rather than assumed — see ``hyperliquid_maker_fee_bps``. * **where the top of book comes from** — see ``market.py``. The fee table below is a floor-setting default, not a quote. Defaults are @@ -19,6 +21,8 @@ from __future__ import annotations +import asyncio + PERP_MARKERS = ("_perpetual", "_perp", "_futures") SPOT = "spot" @@ -28,8 +32,11 @@ # Maker fee per side, in basis points. Keys are matched as a prefix of the # connector name, longest first, so `binance_perpetual` beats `binance`. _MAKER_FEE_BPS: dict[tuple[str, str], float] = { - ("hyperliquid_perpetual", PERP): 1.3, # ~0.29 bp exchange + ~1.0 bp builder - ("hyperliquid", SPOT): 4.0, + # Hyperliquid is fetched, not tabled — see hyperliquid_maker_fee_bps. These + # two are what a caller gets if that fetch is not used: the dearer of the + # venue's two families, so an unfetched fee never quotes too tight. + ("hyperliquid_perpetual", PERP): 2.5, + ("hyperliquid", SPOT): 5.0, ("binance_perpetual", PERP): 2.0, # 0.02 % ("binance", SPOT): 7.5, # 0.075 % ("gate_io_perpetual", PERP): 2.0, @@ -96,3 +103,101 @@ def resolve(connector_name: str, market_type: str = "") -> str: f"{market_type!r}; check the connector name" ) return market_type + + +# ── Hyperliquid: the fee is published, so it is read rather than assumed ────── +# +# One connector serves two families of market, and they do not cost the same: +# +# * a core perp pays the venue's own schedule — 1.5 bp a side to a maker at the +# base tier; +# * a HIP-3 market (``ISSUER:TOKEN-QUOTE``) pays that schedule scaled by its +# deployer's own setting, and again by a tenth where the deployer has turned +# growth mode on. The XYZ dex is in growth mode, so it costs 0.3 bp a side. +# +# Everything above is fetched. One number is not: Hummingbot signs every +# Hyperliquid order with the foundation builder code, and the venue charges +# that on top of its own fee. It is a constant of the client, not of the +# market — ``FOUNDATION_BUILDER_FEE_TENTHS_BPS = 10`` in +# hummingbot/connector/derivative/hyperliquid_perpetual/hyperliquid_perpetual_constants.py +# — and it is not importable here, so it is named rather than fetched. +# +# Checked against fills: 173 maker fills on XYZ:ORCL-USD paid 1.29 bp all-in, +# against 1.5 × 2 × 0.1 + 1.0 = 1.30 bp computed. Hyperliquid reports ``fee`` +# inclusive of the builder fee, which is why one number covers both. +HL_INFO_URL = "https://api.hyperliquid.xyz/info" +HUMMINGBOT_BUILDER_FEE_BPS = 1.0 +GROWTH_MODE_FACTOR = 0.1 +# The published schedule, which is what an account with no discounts pays. +# Reading it for a real address would return that account's own rate; a +# market maker sizing a floor wants the undiscounted one. +_SCHEDULE_PROBE_ADDRESS = "0x0000000000000000000000000000000000000001" + +_hl_cache: dict[str, dict] = {} +_hl_lock = asyncio.Lock() + + +def is_hyperliquid(connector_name: str) -> bool: + return (connector_name or "").lower().startswith("hyperliquid") + + +async def _hl_info(payload: dict) -> dict: + """One Hyperliquid info call, cached for the life of the process. + + The fee schedule and a dex's deployer settings change on the order of + months (the XYZ dex last changed its scale in November 2025), so a scan of + 120 markets should not ask 120 times. + """ + import aiohttp + + key = repr(sorted(payload.items())) + async with _hl_lock: + if key not in _hl_cache: + async with aiohttp.ClientSession() as session: + async with session.post(HL_INFO_URL, json=payload) as response: + response.raise_for_status() + _hl_cache[key] = await response.json() + return _hl_cache[key] + + +async def hyperliquid_maker_fee_bps(trading_pair: str, market_type: str) -> float: + """What one maker side of ``trading_pair`` actually costs, in bp.""" + from flybrain.naming import pair_names + + if market_type not in MARKET_TYPES: + raise ValueError(f"market_type must be one of {MARKET_TYPES}") + names = pair_names(trading_pair) + schedule = (await _hl_info({"type": "userFees", "user": _SCHEDULE_PROBE_ADDRESS}))[ + "feeSchedule" + ] + base = float(schedule["spotAdd" if market_type == SPOT else "add"]) * 1e4 + if not names.issuer: + return base + HUMMINGBOT_BUILDER_FEE_BPS + + meta = await _hl_info({"type": "meta", "dex": names.issuer}) + asset = next( + (a for a in meta["universe"] if a["name"] == names.hl_coin), + None, + ) + if asset is None: + raise ValueError( + f"{names.hl_coin!r} is not listed on the {names.issuer!r} dex, so its " + "fee cannot be read" + ) + scale = float(asset["deployerFeeScale"]) + # The deployer's own multiplier, as Hyperliquid documents it: below 1 it + # adds to the venue's fee, at or above 1 it doubles the scale. + multiplier = scale * 2 if scale >= 1 else scale + 1 + if asset.get("growthMode") == "enabled": + multiplier *= GROWTH_MODE_FACTOR + return base * multiplier + HUMMINGBOT_BUILDER_FEE_BPS + + +async def maker_fee_bps( + connector_name: str, market_type: str, trading_pair: str +) -> float: + """The per-side maker fee for one market: fetched where the venue publishes + it per market, the venue default otherwise.""" + if is_hyperliquid(connector_name): + return await hyperliquid_maker_fee_bps(trading_pair, market_type) + return default_maker_fee_bps(connector_name, market_type) diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index 05472d1eb..04766c67a 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -40,7 +40,7 @@ import numpy as np import plotly.graph_objects as go -from flybrain import worker +from flybrain import venue, worker from flybrain.chart import market_frame from flybrain.decoder import ( Baseline, @@ -182,12 +182,20 @@ class Config(BaseModel): ) -def _specs(config: Config, pairs: list[str]) -> list[MarketSpec]: +async def _specs(config: Config, pairs: list[str]) -> list[MarketSpec]: + """One spec per pair, each carrying the fee its own market charges. + + The fee is not a property of the connector: one Hyperliquid account pays + 2.5 bp a side on a core perp and 1.3 bp on a HIP-3 market in growth mode. + The take-profit floor is derived from it, so it is read per pair unless the + caller passed a figure of their own. + """ spreads = [float(x) for x in config.picked_spreads_bps.split(",") if x.strip()] if len(spreads) != len(pairs): raise ValueError( f"picked_spreads_bps has {len(spreads)} entries for {len(pairs)} pairs" ) + market_type = venue.resolve(config.connector_name, config.market_type) return [ MarketSpec( connector_name=config.connector_name, @@ -196,7 +204,10 @@ def _specs(config: Config, pairs: list[str]) -> list[MarketSpec]: picked_spread_bps=spread, market_type=config.market_type, leverage=config.leverage, - maker_fee_bps=config.maker_fee_bps, + maker_fee_bps=( + config.maker_fee_bps + or await venue.maker_fee_bps(config.connector_name, market_type, pair) + ), portfolio_allocation=config.portfolio_allocation, ) for pair, spread in zip(pairs, spreads) @@ -226,7 +237,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: if config.interval_sec < 10: raise ValueError("interval_sec must be >= 10") pairs = parse_pairs(config.pairs) - specs = _specs(config, pairs) + specs = await _specs(config, pairs) for spec in specs: spec.check_order_size() # fail at start, not on the first apply by_pair = {s.trading_pair: s for s in specs} @@ -642,7 +653,15 @@ async def pace() -> None: b.section("05 / GUARD", "Halts, breakers, apply budget") b.kpi("Halted", guard_state.halted or "no") b.kpi("Applies today", str(guard_state.applies_today)) - b.kpi("Session high", f"{guard_state.session_high_net:+.4f}") + # None until a bot has reported P&L: the first report sets the + # high, and a zero here would read as a real high the run had + # already fallen from. + b.kpi( + "Session high", + "—" + if guard_state.session_high_net is None + else f"{guard_state.session_high_net:+.4f}", + ) b.kpi("Ticks since high", str(guard_state.ticks_since_high)) b.kpi("Loss stop", f"-{max_loss:.2f}") b.markdown( diff --git a/agents/market_making_fly/routines/mm_market_scanner.py b/agents/market_making_fly/routines/mm_market_scanner.py index 1f234e0b6..326f3302c 100644 --- a/agents/market_making_fly/routines/mm_market_scanner.py +++ b/agents/market_making_fly/routines/mm_market_scanner.py @@ -128,18 +128,32 @@ async def _measure(market: LiveMarket, pair: str, config: Config, sem) -> dict: except Exception as failure: # external feed, one market row["error"] = repr(failure)[:80] return row - try: - raw = await market.client.market_data.get_candles( - config.connector_name, pair, interval="1h", max_records=25 - ) - rows = raw if isinstance(raw, list) else raw.get("data", raw.get("candles")) - closes = [float(c["close"]) for c in (rows or []) if c.get("close")] - row["drift_pct"] = ( - abs(closes[-1] / closes[0] - 1) * 100 if len(closes) >= 2 else None - ) - except Exception as failure: - row["drift_pct"] = None - row["drift_error"] = repr(failure)[:60] + # The first ask for a market hummingbot-api has not seen subscribes a + # candle feed and returns 504 if it is not ready within 30 s. That is a + # cold feed, not a missing market: the same call answers on the second + # attempt. Asking twice is the difference between a scan that ranks the + # venue and one that rejects all of it for "no candles". + for attempt in (1, 2): + try: + raw = await market.client.market_data.get_candles( + config.connector_name, pair, interval="1h", max_records=25 + ) + rows = ( + raw + if isinstance(raw, list) + else raw.get("data", raw.get("candles")) + ) + closes = [float(c["close"]) for c in (rows or []) if c.get("close")] + row["drift_pct"] = ( + abs(closes[-1] / closes[0] - 1) * 100 if len(closes) >= 2 else None + ) + break + except Exception as failure: # external feed, one market + row["drift_pct"] = None + row["drift_error"] = ( + str(getattr(failure, "message", "") or failure)[:70] + or repr(failure)[:70] + ) return row @@ -151,11 +165,6 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: return "No server available" market_type = venue.market_type_for(config.connector_name) - fee_bps = config.maker_fee_bps or venue.default_maker_fee_bps( - config.connector_name, market_type - ) - round_trip_bps = 2 * fee_bps - min_spread_bps = round_trip_bps * config.min_spread_over_fee # ── 1. Enumerate and screen on volume, which is one call ────────────────── if config.pairs.strip(): @@ -206,9 +215,21 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: ) by_pair = {m["pair"]: m for m in measured} + # One connector can serve two fee families — a Hyperliquid core perp costs + # twice what one of its HIP-3 markets does — so each market is ranked + # against its own round trip, not the venue's average. + fees = { + pair: config.maker_fee_bps + or await venue.maker_fee_bps(config.connector_name, market_type, pair) + for pair, _ in screened + } + rows = [] for pair, volume in screened: m = by_pair[pair] + fee_bps = fees[pair] + round_trip_bps = 2 * fee_bps + min_spread_bps = round_trip_bps * config.min_spread_over_fee drift = m.get("drift_pct") depth = min(m.get("bid_depth_usd", 0.0), m.get("ask_depth_usd", 0.0)) spread = m.get("spread_bps", 0.0) @@ -222,13 +243,15 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: if depth < config.min_book_depth_usd: reasons.append(f"depth ${depth:,.0f}") if drift is None: - reasons.append("no candles") + reasons.append(f"drift unreadable: {m.get('drift_error', 'no candles')}") elif drift > config.max_daily_drift_pct: reasons.append(f"drift {drift:.1f}%") rows.append( { "pair": pair, "volume": volume, + "fee_bps": fee_bps, + "floor_bps": min_spread_bps, "spread_bps": spread, "spread_over_fee": spread / round_trip_bps if round_trip_bps else 0.0, "depth_usd": depth, @@ -253,17 +276,32 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: builder.source("routine", "mm_market_scanner") builder.tags(["market-making", "scanner", config.connector_name]) builder.manual_order() + cheapest, dearest = min(fees.values()), max(fees.values()) + fee_text = ( + f"{cheapest:.2f} bp" + if cheapest == dearest + else f"{cheapest:.2f}–{dearest:.2f} bp" + ) builder.section( "WHAT A ROUND TRIP COSTS HERE", - f"{market_type} venue · maker {fee_bps:.2f} bp a side · round trip " - f"{round_trip_bps:.2f} bp · a market must quote at least " - f"{min_spread_bps:.2f} bp to clear it by {config.min_spread_over_fee}×", + f"{market_type} venue · maker {fee_text} a side · each market must quote " + f"{config.min_spread_over_fee}× its own round trip to clear it" + + ( + "" + if cheapest == dearest + else ". This venue charges different markets differently, so the floor " + "below is per market rather than venue-wide" + ), ) builder.kpi("Venue", config.connector_name) builder.kpi("Type", market_type) - builder.kpi("Maker fee", f"{fee_bps:.2f} bp") - builder.kpi("Round trip", f"{round_trip_bps:.2f} bp") - builder.kpi("Spread floor", f"{min_spread_bps:.2f} bp") + builder.kpi("Maker fee", fee_text) + builder.kpi("Round trip", f"{2 * cheapest:.2f}–{2 * dearest:.2f} bp") + builder.kpi( + "Spread floor", + f"{2 * cheapest * config.min_spread_over_fee:.2f}–" + f"{2 * dearest * config.min_spread_over_fee:.2f} bp", + ) builder.kpi("Listed", f"{listed:,}") builder.kpi("Book-checked", str(len(screened))) builder.kpi("Survivors", str(len([r for r in rows if r["survives"]]))) @@ -277,6 +315,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: "Pair": r["pair"], "24h volume": f"${r['volume']:,.0f}", "Spread": f"{r['spread_bps']:.2f} bp", + "Maker fee": f"{r['fee_bps']:.2f} bp", "× round trip": f"{r['spread_over_fee']:.2f}×", "Depth/side": f"${r['depth_usd']:,.0f}", "Drift": ( @@ -291,6 +330,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: "Pair", "24h volume", "Spread", + "Maker fee", "× round trip", "Depth/side", "Drift", @@ -328,14 +368,14 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: lines = [ f"venue: {config.connector_name} ({market_type})", - f"maker_fee_bps: {fee_bps:.2f} (round trip {round_trip_bps:.2f})", - f"spread_floor_bps: {min_spread_bps:.2f}", + f"maker_fee_bps: {fee_text} a side", f"listed: {listed}, book_checked: {len(screened)}, survivors: {len(survivors)}", ] for n, r in enumerate(survivors, 1): lines.append( - f"{n}. {r['pair']}: spread {r['spread_bps']:.2f} bp " - f"({r['spread_over_fee']:.2f}× round trip), depth ${r['depth_usd']:,.0f}/side, " + f"{n}. {r['pair']}: spread {r['spread_bps']:.2f} bp vs a " + f"{r['floor_bps']:.2f} bp floor ({r['spread_over_fee']:.2f}× round trip), " + f"fee {r['fee_bps']:.2f} bp, depth ${r['depth_usd']:,.0f}/side, " f"vol ${r['volume']:,.0f}, drift " + (f"{r['drift_pct']:.2f}%" if r["drift_pct"] is not None else "—") ) diff --git a/agents/market_making_fly/tests/test_fly_market.py b/agents/market_making_fly/tests/test_fly_market.py index 47603dabe..d5bccdeba 100644 --- a/agents/market_making_fly/tests/test_fly_market.py +++ b/agents/market_making_fly/tests/test_fly_market.py @@ -208,6 +208,6 @@ def floor_for(connector): fee = venue.default_maker_fee_bps(connector, venue.market_type_for(connector)) return 2 * fee * cfg.min_spread_over_fee - assert floor_for("hyperliquid_perpetual") == pytest.approx(3.9) assert floor_for("binance") == pytest.approx(22.5) # spot costs far more - assert floor_for("binance") > 5 * floor_for("hyperliquid_perpetual") / 2 + assert floor_for("binance_perpetual") == pytest.approx(6.0) + assert floor_for("binance") > 3 * floor_for("binance_perpetual") diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py index 31458aa11..c21893572 100644 --- a/agents/market_making_fly/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -17,6 +17,10 @@ trading_pair="XYZ:DRAM-USD", total_amount_quote=500, picked_spread_bps=8.0, + # What a HIP-3 market in growth mode costs all-in: 0.3 bp to the venue plus + # 1.0 bp of builder fee. The loop fetches this per market; a core + # Hyperliquid perp is 2.5 bp, which is why it is not assumed here. + maker_fee_bps=1.3, ) diff --git a/agents/market_making_fly/tests/test_fly_venue.py b/agents/market_making_fly/tests/test_fly_venue.py index 61be7f044..acf454538 100644 --- a/agents/market_making_fly/tests/test_fly_venue.py +++ b/agents/market_making_fly/tests/test_fly_venue.py @@ -1,5 +1,7 @@ """Spot or perp, and what a round trip costs there.""" +import asyncio + import pytest from flybrain import venue @@ -41,3 +43,52 @@ def test_resolve_refuses_a_contradiction(): venue.resolve("binance", "margin") with pytest.raises(ValueError): venue.market_type_for("") + + +def _seed_hyperliquid_fees(): + """The two Hyperliquid payloads the fee comes from, as the venue returns + them. Seeding the cache keeps the arithmetic under test and the network + out of it.""" + venue._hl_cache.clear() + venue._hl_cache[ + repr( + sorted({"type": "userFees", "user": venue._SCHEDULE_PROBE_ADDRESS}.items()) + ) + ] = {"feeSchedule": {"add": "0.00015", "cross": "0.00045", "spotAdd": "0.0004"}} + venue._hl_cache[repr(sorted({"type": "meta", "dex": "xyz"}.items()))] = { + "universe": [ + {"name": "xyz:ORCL", "deployerFeeScale": "1.0", "growthMode": "enabled"}, + {"name": "xyz:NOGROWTH", "deployerFeeScale": "1.0"}, + {"name": "xyz:HALFSCALE", "deployerFeeScale": "0.5"}, + ] + } + + +def test_one_hyperliquid_connector_charges_two_different_fees(): + """A core perp and a HIP-3 market differ by about 2x, and the floor built + on the wrong one either loses money or never fills. Measured against the + run of 2026-09-13: 173 maker fills on XYZ:ORCL-USD paid 1.29 bp all-in.""" + _seed_hyperliquid_fees() + core = asyncio.run(venue.hyperliquid_maker_fee_bps("BTC-USD", "perp")) + hip3 = asyncio.run(venue.hyperliquid_maker_fee_bps("XYZ:ORCL-USD", "perp")) + assert core == pytest.approx(1.5 + venue.HUMMINGBOT_BUILDER_FEE_BPS) + # scale 1.0 doubles the venue rate, growth mode takes a tenth of that + assert hip3 == pytest.approx(1.5 * 2 * 0.1 + venue.HUMMINGBOT_BUILDER_FEE_BPS) + assert abs(hip3 - 1.29) < 0.02, "computed fee has drifted from what was paid" + + +def test_the_deployers_own_setting_moves_the_fee_both_ways(): + """Hyperliquid's rule is not a discount: without growth mode a HIP-3 + market costs more than the core venue, not less.""" + _seed_hyperliquid_fees() + dear = asyncio.run(venue.hyperliquid_maker_fee_bps("XYZ:NOGROWTH-USD", "perp")) + half = asyncio.run(venue.hyperliquid_maker_fee_bps("XYZ:HALFSCALE-USD", "perp")) + assert dear == pytest.approx(1.5 * 2 + venue.HUMMINGBOT_BUILDER_FEE_BPS) + assert half == pytest.approx(1.5 * 1.5 + venue.HUMMINGBOT_BUILDER_FEE_BPS) + assert dear > asyncio.run(venue.hyperliquid_maker_fee_bps("BTC-USD", "perp")) + + +def test_an_unlisted_hip3_market_is_refused_not_guessed(): + _seed_hyperliquid_fees() + with pytest.raises(ValueError, match="not listed"): + asyncio.run(venue.hyperliquid_maker_fee_bps("XYZ:NOTREAL-USD", "perp")) From d3ac543c497094699858fec9361fe734dbde74a5 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 12:19:05 -0700 Subject: [PATCH 27/48] (fix) an order sized to the minimum is an order the exchange refuses Live on XYZ:ORCL-USD every quote came back "Order notional 9.9361 is lower than minimum notional size 10". 200 quote at 0.2 allocation sizes each order to exactly the 10 USD minimum, and exactly is not enough: the controller turns the quote size into a base amount and rounds it down to the market's step, which on a three-decimal market near 148 costs up to seven cents. check_order_size now wants 20 % of headroom over the venue minimum and says what allocation would clear it, so the refusal happens at startup rather than as a bot that quotes nothing and reports no error of its own. --- agents/market_making_fly/flybrain/posture.py | 25 ++++++++++++++++--- .../tests/test_fly_posture.py | 16 ++++++++++++ 2 files changed, 37 insertions(+), 4 deletions(-) diff --git a/agents/market_making_fly/flybrain/posture.py b/agents/market_making_fly/flybrain/posture.py index d4f528214..be81d29af 100644 --- a/agents/market_making_fly/flybrain/posture.py +++ b/agents/market_making_fly/flybrain/posture.py @@ -38,6 +38,10 @@ assert set(TIMING) == set(REGIMES) BPS = 1e-4 +# How far above an exchange minimum an order must be sized to survive the +# venue's own rounding. 20 % covers a step rounded down on a three-decimal +# market near $150 and leaves room for the lean. +ORDER_SIZE_MARGIN = 1.2 @dataclass(frozen=True) @@ -113,12 +117,25 @@ def order_notional(self) -> float: return self.total_amount_quote * self.portfolio_allocation / 4 def check_order_size(self) -> None: - if self.order_notional < self.min_order_notional: - needed = self.min_order_notional * 4 / self.total_amount_quote + """Refuse a size the exchange will reject once it has been rounded. + + An order sized to exactly the minimum does not survive the round trip + through the exchange's own precision: the controller converts the quote + size to a base amount, rounds it down to the market's step, and the + resulting notional lands *under* the minimum. XYZ:ORCL-USD quotes to + three decimals near 148, so $10.00 became $9.94 and Hyperliquid + rejected every order with "lower than minimum notional size". The + margin is what a rounded-down step can cost plus the widest lean. + """ + floor = self.min_order_notional * ORDER_SIZE_MARGIN + if self.order_notional < floor: + needed = floor * 4 / self.total_amount_quote raise ValueError( f"{self.trading_pair}: an order would be {self.order_notional:.2f} quote, " - f"below the {self.min_order_notional:.0f} minimum; raise portfolio_allocation " - f"to at least {min(1.0, needed):.2f} or total_amount_quote" + f"under the {floor:.2f} needed to clear a " + f"{self.min_order_notional:.0f} minimum after rounding; raise " + f"portfolio_allocation to at least {min(1.0, needed):.2f} or " + "total_amount_quote" ) diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py index c21893572..fd5c612ea 100644 --- a/agents/market_making_fly/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -146,3 +146,19 @@ def test_config_diff(): diff = config_diff(a, b) assert "buy_spreads" in diff and "trading_pair" not in diff assert config_diff(None, a) == a + + +def test_an_order_sized_to_the_bare_minimum_is_refused(): + """The live failure of 2026-09-13: 200 quote at 0.2 allocation sizes each + order to exactly the 10 USD minimum, the controller rounds the base amount + down to the market's step, and Hyperliquid rejected all of them at 9.94.""" + bare = MarketSpec( + **{**SPEC.__dict__, "total_amount_quote": 200, "portfolio_allocation": 0.2} + ) + assert bare.order_notional == pytest.approx(10.0) + with pytest.raises(ValueError, match="after rounding"): + bare.check_order_size() + # and the message names an allocation that actually clears it + roomy = MarketSpec(**{**bare.__dict__, "portfolio_allocation": 0.3}) + roomy.check_order_size() + assert roomy.order_notional == pytest.approx(15.0) From 54687f59eb32cfffc736d26618510ecfe9a350da Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 12:29:02 -0700 Subject: [PATCH 28/48] (feat) the spread floor is the market's own fee, not a number someone picked MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit min_spread_bps was a parameter defaulting to 3 bp, and 3 bp is wrong in both directions: too wide for a HIP-3 perp that costs 1.3 bp all-in, far too tight for a spot book at 7.5. Nobody could set it correctly without already knowing the fee, which the spec now carries and fetches per market. So it is derived rather than passed. A buy at one fee below mid and a sell at one fee above capture exactly the round trip — break-even. Tighter than that loses money on every completed pair whatever the fly decodes, which makes it the one width that is not the strategy's to choose. Everything wider stays the posture's business. On XYZ:ORCL-USD this moves the floor from 3 bp to 1.3 and lets a leaned quote sit where the fly put it instead of being pushed out to a fixed number. --- agents/market_making_fly/flybrain/posture.py | 25 +++++++++++------ .../market_making_fly/routines/fly_brain.py | 8 +++--- .../tests/test_fly_posture.py | 27 +++++++++++++++---- docs/market_making_fly_design.md | 7 ++--- 4 files changed, 48 insertions(+), 19 deletions(-) diff --git a/agents/market_making_fly/flybrain/posture.py b/agents/market_making_fly/flybrain/posture.py index be81d29af..db791b793 100644 --- a/agents/market_making_fly/flybrain/posture.py +++ b/agents/market_making_fly/flybrain/posture.py @@ -3,7 +3,8 @@ The base is a bounded default set; the posture multiplies the spreads and leans them. Every money-relevant floor lives here, in code: -* no spread level below ``min_spread_bps``; +* no spread level closer to mid than the market's own maker fee, so a + two-sided round trip at the floor at least pays for itself; * ``take_profit`` never below ``2.2 ×`` the round-trip maker fee, and never below 4 bp (Market Making Expert's "TP must exceed round-trip fees" made mandatory); @@ -59,7 +60,6 @@ class MarketSpec: # 0 means "use the venue default"; pass the exchange's real figure when # known, since the take-profit floor is derived from it. maker_fee_bps: float = 0.0 - min_spread_bps: float = 3.0 portfolio_allocation: float = 0.2 target_base_pct: float = 0.4 min_base_pct: float = 0.3 @@ -82,12 +82,7 @@ def __post_init__(self): "maker_fee_bps", venue.default_maker_fee_bps(self.connector_name, resolved), ) - for name in ( - "total_amount_quote", - "picked_spread_bps", - "maker_fee_bps", - "min_spread_bps", - ): + for name in ("total_amount_quote", "picked_spread_bps", "maker_fee_bps"): value = getattr(self, name) if not math.isfinite(value) or value <= 0: raise ValueError(f"{name} must be finite and positive") @@ -110,6 +105,20 @@ def __post_init__(self): def is_spot(self) -> bool: return self.market_type == venue.SPOT + @property + def min_spread_bps(self) -> float: + """The closest to mid a quote may sit: one maker fee. + + A buy at −f and a sell at +f capture exactly 2f, which is the round + trip — break-even. Anything tighter loses money on every completed + pair whatever the fly decodes, so it is the one width that is not the + strategy's to choose. It was a fixed 3 bp, which is both too wide for + a HIP-3 market at 1.3 bp and too tight for a spot book at 7.5, and it + was a parameter nobody could set correctly without already knowing the + fee the spec now carries. + """ + return self.maker_fee_bps + @property def order_notional(self) -> float: """Quote size of one order at neutral posture: one cycle's allocation diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index 04766c67a..ffc61c920 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -658,9 +658,11 @@ async def pace() -> None: # already fallen from. b.kpi( "Session high", - "—" - if guard_state.session_high_net is None - else f"{guard_state.session_high_net:+.4f}", + ( + "—" + if guard_state.session_high_net is None + else f"{guard_state.session_high_net:+.4f}" + ), ) b.kpi("Ticks since high", str(guard_state.ticks_since_high)) b.kpi("Loss stop", f"-{max_loss:.2f}") diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py index fd5c612ea..d46b23745 100644 --- a/agents/market_making_fly/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -59,8 +59,8 @@ def test_volatile_widens_quiet_tightens_with_floor(): "volatile" ] tight = build_config(SPEC, Posture("quiet", 0.6, 0.0, 0, -1.5, False, True)) - # 4 bp × 0.6 = 2.4 bp, floored to the 3 bp minimum - assert _spreads(tight["buy_spreads"])[0] == pytest.approx(3 * BPS) + # 4 bp × 0.6 = 2.4 bp, which clears this market's 1.3 bp fee floor + assert _spreads(tight["buy_spreads"])[0] == pytest.approx(2.4 * BPS) assert (tight["executor_refresh_time"], tight["buy_cooldown_time"]) == TIMING[ "quiet" ] @@ -69,14 +69,31 @@ def test_volatile_widens_quiet_tightens_with_floor(): def test_lean_is_asymmetric_and_capped(): up = build_config(SPEC, Posture("trending_up", 1.0, 3.0, 2.0, 0, True, True)) buy, sell = _spreads(up["buy_spreads"]), _spreads(up["sell_spreads"]) - # lean capped at half of level 1 (4 bp → 2 bp); buy 2 bp floored to 3 bp - assert buy[0] == pytest.approx(3 * BPS) and sell[0] == pytest.approx(6 * BPS) + # lean capped at half of level 1 (4 bp → 2 bp), and 2 bp still clears the fee + assert buy[0] == pytest.approx(2 * BPS) and sell[0] == pytest.approx(6 * BPS) assert buy[1] == pytest.approx(7 * BPS) and sell[1] == pytest.approx(11 * BPS) down = build_config(SPEC, Posture("trending_down", 1.0, -3.0, -2.0, 0, True, True)) - assert _spreads(down["sell_spreads"])[0] == pytest.approx(3 * BPS) + assert _spreads(down["sell_spreads"])[0] == pytest.approx(2 * BPS) assert _spreads(down["buy_spreads"])[0] == pytest.approx(6 * BPS) +def test_the_spread_floor_is_the_market_own_fee(): + """A quote at the floor breaks even: buy at −f and sell at +f capture 2f, + exactly the round trip. A fixed 3 bp was too wide for a HIP-3 perp and far + too tight for a spot book.""" + assert SPEC.min_spread_bps == pytest.approx(SPEC.maker_fee_bps) + dear = MarketSpec( + connector_name="binance", + trading_pair="SOL-USDT", + total_amount_quote=500, + picked_spread_bps=8.0, + ) + assert dear.min_spread_bps == pytest.approx(7.5) # binance spot + # a lean that would quote inside the fee is pushed back out to it + leaned = build_config(dear, Posture("trending_up", 1.0, 3.0, 2.0, 0, True, True)) + assert min(_spreads(leaned["buy_spreads"])) == pytest.approx(7.5 * BPS) + + def test_pause_sets_kill_switch(): cfg = build_config(SPEC, Posture("pause", 2.5, 0.0, 0, 3.0, False, True)) assert cfg["manual_kill_switch"] is True diff --git a/docs/market_making_fly_design.md b/docs/market_making_fly_design.md index 0bd0f1063..d23c0900b 100644 --- a/docs/market_making_fly_design.md +++ b/docs/market_making_fly_design.md @@ -204,7 +204,8 @@ shifts it: * Base spreads from the scanner's picked spread `S` bp: level 1 `max(2, S/2)`, level 2 `S+1` (HIP-3 playbook). Then `buy = base · spread_mult − shift`, `sell = base · spread_mult + shift`, - each floored at `min_spread_bps = 3`. + each floored at the market's own maker fee, so a two-sided round trip at + the floor breaks even (1.3 bp on a HIP-3 perp, 7.5 on a spot book). * `take_profit = max(0.0004, level-1 buy spread)` — and never below `2.2 × round-trip maker fee` (HIP-3 all-in ~1.3 bp/side → 2.6 bp; floor 5.7 bp wins). Market Making Expert's rule "TP must exceed round-trip fees" becomes a @@ -237,7 +238,7 @@ passes `resume_reviewed=true`". None of them chooses a different posture. |---|---| | Market closed (live book missing a side) — HIP-3 equities close off-hours | veto (HOLD); after `closed_ticks_to_stop` consecutive closed ticks, STOP bot, leave fly observing | | Available USD on the unified Hyperliquid account < required margin | veto | -| Any spread `< min_spread_bps`, TP below fee floor, leverage above cap | veto (should be unreachable after `posture.py` floors; this is the belt to those braces) | +| Any spread inside the market's maker fee, TP below fee floor, leverage above cap | veto (should be unreachable after `posture.py` floors; this is the belt to those braces) | | Apply cooldown (`min_apply_interval_sec`) or daily apply cap (`max_applies_per_day = 48`) | veto | | Mid moved more than `apply_price_tolerance = 0.5 %` between observation and apply | veto (stonkfly's fresh-book check) | | `total_net ≤ −max_loss_quote` (default 4 % of `total_amount_quote`) | **halt**: STOP bot, position closed by the controller's `global_stop_loss` / market-close, fly halted | @@ -462,7 +463,7 @@ Unit (no data, run in CI): warm-up emits ranging; centring removes a constant bias; hysteresis blocks small changes and passes regime changes. * `posture.py`: fee floor beats spread; shift never pushes a side below - `min_spread_bps`; pause sets the kill switch; timing table per regime; pair + the fee floor; pause sets the kill switch; timing table per regime; pair is uppercase. * `guard.py`: every row of the table in §8, halt vs veto, `resume_reviewed` semantics, financial halt not clearable. From 433234740d879b67272ac60bd7e46e7dce7c8fa6 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 12:38:43 -0700 Subject: [PATCH 29/48] (feat) rank markets by whether they come to the fly, not by their touch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scanner asked whether a market's touch is already wider than a round trip. That is the test for a maker who joins the touch. The fly does not: it rests a quote away from mid and waits, so what decides whether a market is worth quoting is whether the price travels the trip the fly has to make — from mid down to level 1, never inside the fee, then back out through the take-profit floor. Measured as the median candle range at the fly's own interval over that cycle. One candle call now serves both it and the 24 h drift. The old test rejected all 120 HIP-3 markets, including XYZ:ORCL-USD, which I then picked by hand anyway — and ORCL turns out to be the wrong market by the right test: its median 5-minute range is 4 bp against a 7.7 bp cycle, reach 0.52, and in twenty minutes of live quoting it filled once and never reached its take-profit. The same scan now surfaces XYZ:DRAM-USD at 1.28× and, across the venue, ZEC-USD at 2.23× — while rejecting BTC and ETH, whose enormous books barely move 4 and 7 bp in five minutes. Two smaller things follow from it. The geometry rules the fly quotes by are now plain functions of fee and spread, so the scanner asks posture.py what a cycle costs rather than keeping a second copy. And RANKED lists the best of what was measured whether or not anything cleared, with a Clears column: a scan that prints "none survived" and nothing else tells the operator nothing about how close the venue came. --- agents/market_making_fly/flybrain/posture.py | 20 +- .../routines/mm_market_scanner.py | 217 ++++++++++++------ .../tests/test_fly_market.py | 30 +-- 3 files changed, 180 insertions(+), 87 deletions(-) diff --git a/agents/market_making_fly/flybrain/posture.py b/agents/market_making_fly/flybrain/posture.py index db791b793..d0045f3a9 100644 --- a/agents/market_making_fly/flybrain/posture.py +++ b/agents/market_making_fly/flybrain/posture.py @@ -148,14 +148,26 @@ def check_order_size(self) -> None: ) +# The two geometry rules, as plain functions of what they actually depend on: +# a market's fee and its observed spread. The scanner needs them before a spec +# exists — it is deciding whether a market is worth quoting at all — and two +# copies of a money rule is one too many. +def take_profit_floor_bps(maker_fee_bps: float) -> float: + """Where a position must close to have been worth opening, in bp.""" + return max(4.0, 2.2 * 2 * maker_fee_bps) + + +def base_levels_from_spread(picked_spread_bps: float) -> tuple[float, float]: + """HIP-3 playbook: level 1 ``max(2, S/2)`` bp, level 2 ``S+1`` bp.""" + return max(2.0, picked_spread_bps / 2), picked_spread_bps + 1 + + def take_profit_floor(spec: MarketSpec) -> float: - return round(max(4 * BPS, 2.2 * 2 * spec.maker_fee_bps * BPS), 8) + return round(take_profit_floor_bps(spec.maker_fee_bps) * BPS, 8) def base_levels_bps(spec: MarketSpec) -> tuple[float, float]: - """HIP-3 playbook: level 1 ``max(2, S/2)`` bp, level 2 ``S+1`` bp.""" - s = spec.picked_spread_bps - return max(2.0, s / 2), s + 1 + return base_levels_from_spread(spec.picked_spread_bps) def _fmt(values: list[float]) -> str: diff --git a/agents/market_making_fly/routines/mm_market_scanner.py b/agents/market_making_fly/routines/mm_market_scanner.py index 326f3302c..6ccedb7cd 100644 --- a/agents/market_making_fly/routines/mm_market_scanner.py +++ b/agents/market_making_fly/routines/mm_market_scanner.py @@ -7,7 +7,7 @@ term at all, ``market_analyzer`` reads one pair's regime, and ``arb_check`` compares one pair across venues. -This is the HIP-3 scanner's ranking generalized. Two things change: +This is the HIP-3 scanner's ranking generalized. Three things change: * **Where the numbers come from.** That routine reads one Hyperliquid call which returns every market in an issuer's dex. Condor's own ticker fetcher @@ -20,12 +20,19 @@ round-trip maker fee, and that fee differs by a factor of five between a HIP-3 perp and a spot book. +* **What "worth quoting" means.** Both older scanners asked whether the touch + is already wider than a round trip. That is the test for someone joining the + touch. The fly does not join it — it rests a quote away from mid and waits — + so the question is whether the market *comes to it*: does a typical candle + travel the round trip the fly must make, from mid down to its quote and back + out through the take-profit? On Hyperliquid the touch test rejected all 120 + HIP-3 markets including the one the fly was quoting profitably by hand. + One honest limit. Reading a book costs a call, so only the top markets by -volume get read — and the widest markets are rarely the busiest. Scanning the -120 HIP-3 markets, the eight heaviest all quote under 1.4 bp, far too tight to -clear a 2.6 bp round trip; the first market that cleared it sat thirtieth by -volume. So ``prescreen`` is the knob that matters when nothing survives, and -the report says how deep it looked. +volume get read — and the busiest markets are not the most reachable. So +``prescreen`` is the knob that matters when nothing survives, and the report +says how deep it looked, and ranks the best of what it measured whether or not +anything cleared. """ from __future__ import annotations @@ -40,10 +47,12 @@ import asyncio import logging import math +import statistics from flybrain import venue from flybrain.market import LiveMarket, depth_within from flybrain.naming import pair_names +from flybrain.posture import base_levels_from_spread, take_profit_floor_bps from pydantic import BaseModel, Field from telegram.ext import ContextTypes @@ -54,9 +63,14 @@ CATEGORY = "Market Data" -# The spread term saturates: past this, more spread says more about how thin -# the book is than about how much a maker earns. -SPREAD_SCORE_CAP_BPS = 8.0 +# Reach saturates: past three cycles in a median candle the market is moving +# faster than a maker can requote, and the extra movement is adverse selection +# rather than income. +REACH_SCORE_CAP = 3.0 + +# A day of candles at the fly's own interval: enough for a stable median range +# and for the 24 h drift, from one call per market. +CANDLES_PER_DAY = 288 class Config(BaseModel): @@ -80,9 +94,15 @@ class Config(BaseModel): maker_fee_bps: float = Field( default=0.0, description="Maker fee per side in bp; 0 uses the venue default" ) - min_spread_over_fee: float = Field( - default=1.5, - description="Require the spread to be this multiple of the round-trip fee", + min_range_over_cycle: float = Field( + default=1.0, + description="Require a typical candle's range to be this multiple of the " + "round trip the fly would have to travel (quote distance + take-profit). " + "1.0 means a median candle completes one cycle", + ) + candle_interval: str = Field( + default="5m", + description="Candle the range is measured on; use the fly's own interval", ) min_volume_usd: float = Field(default=250_000.0, description="Minimum 24h volume") max_daily_drift_pct: float = Field( @@ -106,6 +126,18 @@ class Config(BaseModel): top_n: int = Field(default=5, ge=1, le=25, description="Markets to report") +def cycle_bps(spread_bps: float, fee_bps: float) -> float: + """The round trip the fly must travel on this market, in bp. + + From mid down to where it would rest level 1 — ``max(2, S/2)``, never + inside the fee — and back out through the take-profit floor. This is the + distance a market has to move for one completed pair, and the thing a + candle's range is compared against. + """ + entry = max(base_levels_from_spread(spread_bps)[0], fee_bps) + return entry + take_profit_floor_bps(fee_bps) + + async def _measure(market: LiveMarket, pair: str, config: Config, sem) -> dict: """Spread, depth and drift for one market. Never raises: a market whose book or candles cannot be read is reported as unreadable, not dropped @@ -128,29 +160,43 @@ async def _measure(market: LiveMarket, pair: str, config: Config, sem) -> dict: except Exception as failure: # external feed, one market row["error"] = repr(failure)[:80] return row - # The first ask for a market hummingbot-api has not seen subscribes a - # candle feed and returns 504 if it is not ready within 30 s. That is a - # cold feed, not a missing market: the same call answers on the second - # attempt. Asking twice is the difference between a scan that ranks the - # venue and one that rejects all of it for "no candles". + # One day of the fly's own candle gives both numbers: the drift across + # it, and how far a typical one of them travels. The first ask for a + # market hummingbot-api has not seen subscribes a candle feed and + # returns 504 if it is not ready within 30 s. That is a cold feed, not + # a missing market: the same call answers on the second attempt. for attempt in (1, 2): try: raw = await market.client.market_data.get_candles( - config.connector_name, pair, interval="1h", max_records=25 + config.connector_name, + pair, + interval=config.candle_interval, + max_records=CANDLES_PER_DAY, ) rows = ( raw if isinstance(raw, list) else raw.get("data", raw.get("candles")) ) - closes = [float(c["close"]) for c in (rows or []) if c.get("close")] - row["drift_pct"] = ( - abs(closes[-1] / closes[0] - 1) * 100 if len(closes) >= 2 else None + bars = [ + (float(c["high"]), float(c["low"]), float(c["close"])) + for c in (rows or []) + if c.get("close") and float(c["close"]) > 0 + ] + if len(bars) < 2: + raise ValueError(f"only {len(bars)} usable candles") + row["drift_pct"] = abs(bars[-1][2] / bars[0][2] - 1) * 100 + # Median, not mean: one news bar should not make a quiet market + # look reachable. + row["range_bps"] = statistics.median( + (high - low) / close * 1e4 for high, low, close in bars ) + row["candles"] = len(bars) break except Exception as failure: # external feed, one market row["drift_pct"] = None - row["drift_error"] = ( + row["range_bps"] = None + row["candle_error"] = ( str(getattr(failure, "message", "") or failure)[:70] or repr(failure)[:70] ) @@ -207,8 +253,10 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: f"${config.min_volume_usd:,.0f} of 24h volume" ) - # ── 2. Book and drift, only for what survived — this is the expensive part ─ - market = LiveMarket(client, config.connector_name, "1h", 25) + # ── 2. Book and candles, only for what survived — the expensive part ───── + market = LiveMarket( + client, config.connector_name, config.candle_interval, CANDLES_PER_DAY + ) sem = asyncio.Semaphore(6) measured = await asyncio.gather( *(_measure(market, pair, config, sem) for pair, _ in screened) @@ -228,39 +276,58 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: for pair, volume in screened: m = by_pair[pair] fee_bps = fees[pair] - round_trip_bps = 2 * fee_bps - min_spread_bps = round_trip_bps * config.min_spread_over_fee drift = m.get("drift_pct") depth = min(m.get("bid_depth_usd", 0.0), m.get("ask_depth_usd", 0.0)) spread = m.get("spread_bps", 0.0) + # What the fly would actually do here: rest level 1 at max(2, S/2) bp, + # never inside the fee, and close at the take-profit floor. The round + # trip it must travel is the sum — down to the quote, then back up + # through the exit. + entry_bps = max(base_levels_from_spread(spread)[0], fee_bps) + exit_bps = take_profit_floor_bps(fee_bps) + cycle = cycle_bps(spread, fee_bps) + range_bps = m.get("range_bps") + # How far a typical candle travels against that cycle. Above 1 the + # median candle completes one; below it, the market does not come to + # the fly often enough to matter, however wide its touch looks. + reach = (range_bps / cycle) if range_bps else 0.0 reasons = [] if m.get("error"): reasons.append("book unreadable") elif not m.get("open"): reasons.append("book closed") - if spread < min_spread_bps: - reasons.append(f"spread {spread:.1f} < {min_spread_bps:.1f} bp") + if range_bps is None: + reasons.append(f"candles unreadable: {m.get('candle_error', 'none')}") + elif reach < config.min_range_over_cycle: + reasons.append( + f"range {range_bps:.1f} < {config.min_range_over_cycle:g}× the " + f"{cycle:.1f} bp cycle" + ) if depth < config.min_book_depth_usd: reasons.append(f"depth ${depth:,.0f}") - if drift is None: - reasons.append(f"drift unreadable: {m.get('drift_error', 'no candles')}") - elif drift > config.max_daily_drift_pct: + if drift is not None and drift > config.max_daily_drift_pct: reasons.append(f"drift {drift:.1f}%") rows.append( { "pair": pair, "volume": volume, "fee_bps": fee_bps, - "floor_bps": min_spread_bps, + "entry_bps": entry_bps, + "exit_bps": exit_bps, + "cycle_bps": cycle, + "range_bps": range_bps, + "reach": reach, "spread_bps": spread, - "spread_over_fee": spread / round_trip_bps if round_trip_bps else 0.0, "depth_usd": depth, "drift_pct": drift, "survives": not reasons, "why_not": ", ".join(reasons), + # Reach is what earns: a market that traverses the fly's cycle + # twice as often is worth twice as much to it, up to the point + # where the movement is adverse selection rather than noise. "score": ( math.log(max(volume, 1)) - + 0.3 * min(spread, SPREAD_SCORE_CAP_BPS) + + 2.0 * min(reach, REACH_SCORE_CAP) - 0.4 * (drift if drift is not None else 99.0) ), } @@ -268,6 +335,9 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: survivors = sorted((r for r in rows if r["survives"]), key=lambda r: -r["score"])[ : config.top_n ] + # Rank the best measured markets whether or not they clear, so a scan that + # finds no survivor still says which markets came closest and by how much. + ranked = sorted(rows, key=lambda r: -r["score"])[: config.top_n] rejected = sorted( (r for r in rows if not r["survives"]), key=lambda r: -r["volume"] ) @@ -282,59 +352,58 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: if cheapest == dearest else f"{cheapest:.2f}–{dearest:.2f} bp" ) + cycles = [r["cycle_bps"] for r in rows] builder.section( - "WHAT A ROUND TRIP COSTS HERE", - f"{market_type} venue · maker {fee_text} a side · each market must quote " - f"{config.min_spread_over_fee}× its own round trip to clear it" - + ( - "" - if cheapest == dearest - else ". This venue charges different markets differently, so the floor " - "below is per market rather than venue-wide" - ), + "WHETHER THE MARKET COMES TO THE FLY", + f"{market_type} venue · maker {fee_text} a side. The fly does not join " + "the touch: it rests a quote and waits. So the test is not whether the " + "spread is already wide, but whether a typical " + f"{config.candle_interval} candle travels the round trip it would have " + f"to — down to its quote and back out through the take-profit, " + f"{min(cycles):.1f}–{max(cycles):.1f} bp here — at least " + f"{config.min_range_over_cycle:g}× over.", ) builder.kpi("Venue", config.connector_name) builder.kpi("Type", market_type) builder.kpi("Maker fee", fee_text) - builder.kpi("Round trip", f"{2 * cheapest:.2f}–{2 * dearest:.2f} bp") - builder.kpi( - "Spread floor", - f"{2 * cheapest * config.min_spread_over_fee:.2f}–" - f"{2 * dearest * config.min_spread_over_fee:.2f} bp", - ) + builder.kpi("Cycle to travel", f"{min(cycles):.1f}–{max(cycles):.1f} bp") builder.kpi("Listed", f"{listed:,}") builder.kpi("Book-checked", str(len(screened))) builder.kpi("Survivors", str(len([r for r in rows if r["survives"]]))) builder.section( - "RANKED", f"Top {len(survivors)} by score — volume, spread over fee, and drift" + "RANKED", + f"Best {len(ranked)} of the {len(screened)} measured, clearing or not — " + "reach is the median candle's range over the cycle the fly must travel", ) builder.table( [ { "Pair": r["pair"], + "Clears": "yes" if r["survives"] else "no", "24h volume": f"${r['volume']:,.0f}", + "Median range": (f"{r['range_bps']:.2f} bp" if r["range_bps"] else "—"), + "Cycle": f"{r['cycle_bps']:.2f} bp", + "Reach": f"{r['reach']:.2f}×", "Spread": f"{r['spread_bps']:.2f} bp", - "Maker fee": f"{r['fee_bps']:.2f} bp", - "× round trip": f"{r['spread_over_fee']:.2f}×", "Depth/side": f"${r['depth_usd']:,.0f}", "Drift": ( f"{r['drift_pct']:.2f}%" if r["drift_pct"] is not None else "—" ), - "Score": f"{r['score']:.2f}", } - for r in survivors + for r in ranked ] - or [{"Pair": "— none survived —"}], + or [{"Pair": "— nothing measured —"}], [ "Pair", + "Clears", "24h volume", + "Median range", + "Cycle", + "Reach", "Spread", - "Maker fee", - "× round trip", "Depth/side", "Drift", - "Score", ], ) builder.section("REJECTED", "Why each screened market did not make it") @@ -359,10 +428,12 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: "between this scan and the rest of the venue._" ) builder.markdown( - "_A wide spread is necessary, not sufficient: it is often wide because the " - "book is thin, which is why depth is filtered separately. Drift is a proxy " - "for how much inventory a maker would accumulate against the trend, not a " - "forecast. Nothing here says a market is profitable._" + "_Reach says the market reaches the fly's quote, not that the fills are " + "good ones: the same movement that fills a maker is what runs him over, " + "and a median range hides the bar that gaps through both levels. Depth " + "is filtered separately because a wide touch is usually a thin one. " + "Drift is a proxy for the inventory a maker accumulates against a trend, " + "not a forecast. Nothing here says a market is profitable._" ) await builder.save() @@ -371,18 +442,24 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: f"maker_fee_bps: {fee_text} a side", f"listed: {listed}, book_checked: {len(screened)}, survivors: {len(survivors)}", ] - for n, r in enumerate(survivors, 1): + for n, r in enumerate(ranked, 1): lines.append( - f"{n}. {r['pair']}: spread {r['spread_bps']:.2f} bp vs a " - f"{r['floor_bps']:.2f} bp floor ({r['spread_over_fee']:.2f}× round trip), " - f"fee {r['fee_bps']:.2f} bp, depth ${r['depth_usd']:,.0f}/side, " - f"vol ${r['volume']:,.0f}, drift " - + (f"{r['drift_pct']:.2f}%" if r["drift_pct"] is not None else "—") + f"{n}. {r['pair']}: reach {r['reach']:.2f}× (median " + + (f"{r['range_bps']:.2f}" if r["range_bps"] else "—") + + f" bp range vs a {r['cycle_bps']:.2f} bp cycle), spread " + f"{r['spread_bps']:.2f} bp, fee {r['fee_bps']:.2f} bp, depth " + f"${r['depth_usd']:,.0f}/side, vol ${r['volume']:,.0f}" + + ("" if r["survives"] else f" — REJECTED: {r['why_not']}") ) if survivors: + best = survivors[0] lines.append( - f"TOP PICK: {survivors[0]['pair']} at {survivors[0]['spread_bps']:.2f} bp" + f"TOP PICK: {best['pair']} at reach {best['reach']:.2f}×, " + f"picked_spread_bps={best['spread_bps']:.2f}" ) else: - lines.append("TOP PICK: none — no market cleared the fee with depth behind it") + lines.append( + "TOP PICK: none — no market travels the fly's own round trip often " + "enough, with depth behind it" + ) return "\n".join(lines) diff --git a/agents/market_making_fly/tests/test_fly_market.py b/agents/market_making_fly/tests/test_fly_market.py index d5bccdeba..1315b0280 100644 --- a/agents/market_making_fly/tests/test_fly_market.py +++ b/agents/market_making_fly/tests/test_fly_market.py @@ -190,24 +190,28 @@ def test_depth_only_counts_what_is_close_enough_to_trade_against(): assert depth_within([], asks, 20) == (0.0, 0.0, 0.0) -def test_the_scanner_requires_the_spread_to_clear_the_venues_own_fee(): - """The HIP-3 scanner hardcoded 3bp, unrelated to what a round trip costs. - A spot book's fee is several times a perp's, so the floor has to move.""" +def test_the_scanner_measures_the_trip_the_fly_would_actually_make(): + """Both older scanners asked whether the touch already clears a round trip, + which is the test for joining the touch. The fly rests a quote away from + mid, so what matters is the distance it must travel: down to level 1, never + inside the fee, then out through the take-profit floor.""" import importlib.util from pathlib import Path - from flybrain import venue - path = Path(__file__).resolve().parents[1] / "routines" / "mm_market_scanner.py" spec = importlib.util.spec_from_file_location("mm_market_scanner", path) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) - def floor_for(connector): - cfg = mod.Config(connector_name=connector) - fee = venue.default_maker_fee_bps(connector, venue.market_type_for(connector)) - return 2 * fee * cfg.min_spread_over_fee - - assert floor_for("binance") == pytest.approx(22.5) # spot costs far more - assert floor_for("binance_perpetual") == pytest.approx(6.0) - assert floor_for("binance") > 3 * floor_for("binance_perpetual") + # XYZ:ORCL-USD as it was on 2026-09-13: 2 bp touch, 1.3 bp all-in fee. + # Level 1 is max(2, 2/2) = 2 bp, the take-profit floor 2.2 × 2.6 = 5.72. + assert mod.cycle_bps(2.0, 1.3) == pytest.approx(2.0 + 5.72) + # A dearer market has to travel further for the same quote, because the + # exit is fee-derived: binance spot at 7.5 bp needs 33 bp of take-profit. + assert mod.cycle_bps(2.0, 7.5) == pytest.approx(7.5 + 33.0) + # and the fee is a floor on the entry too — a market quoting inside it + # cannot be quoted inside it + assert mod.cycle_bps(0.2, 7.5) == pytest.approx(7.5 + 33.0) + # A market whose median candle travels less than the cycle never completes + # one, however wide its touch looks. + assert 4.0 / mod.cycle_bps(2.0, 1.3) < 1.0 From 8ffe29798376288e0fa7b69ce18516fcc811e3ea Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 12:41:59 -0700 Subject: [PATCH 30/48] (fix) the outer quote level never lands inside the inner one The playbook sets level 1 at max(2, S/2) and level 2 at S+1, which assumes a market quoting several bp. XYZ:DRAM-USD quotes 0.35, where S+1 is 1.35 and lands inside the 2 bp first level: the ladder inverts, the level meant to sit further out fills first, and the inventory ladder stops meaning anything. Level 2 is now at least a bp beyond level 1. Wide markets are unchanged -- S=8 still gives 4 and 9. --- agents/market_making_fly/flybrain/posture.py | 13 +++++++++++-- .../market_making_fly/tests/test_fly_posture.py | 17 +++++++++++++++++ 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/agents/market_making_fly/flybrain/posture.py b/agents/market_making_fly/flybrain/posture.py index d0045f3a9..ad1467efe 100644 --- a/agents/market_making_fly/flybrain/posture.py +++ b/agents/market_making_fly/flybrain/posture.py @@ -157,9 +157,18 @@ def take_profit_floor_bps(maker_fee_bps: float) -> float: return max(4.0, 2.2 * 2 * maker_fee_bps) +# The outer level must stay outside the inner one. The playbook's S+1 assumed +# a market quoting several bp; on a tight book — XYZ:DRAM-USD quotes 0.35 — +# S+1 lands inside max(2, S/2) and the ladder inverts, so the "outer" level +# fills first and the inventory ladder means nothing. +LEVEL_STEP_BPS = 1.0 + + def base_levels_from_spread(picked_spread_bps: float) -> tuple[float, float]: - """HIP-3 playbook: level 1 ``max(2, S/2)`` bp, level 2 ``S+1`` bp.""" - return max(2.0, picked_spread_bps / 2), picked_spread_bps + 1 + """HIP-3 playbook: level 1 ``max(2, S/2)`` bp, level 2 ``S+1`` bp, with the + second never inside the first.""" + first = max(2.0, picked_spread_bps / 2) + return first, max(picked_spread_bps + 1, first + LEVEL_STEP_BPS) def take_profit_floor(spec: MarketSpec) -> float: diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py index d46b23745..c2cedcdbe 100644 --- a/agents/market_making_fly/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -179,3 +179,20 @@ def test_an_order_sized_to_the_bare_minimum_is_refused(): roomy = MarketSpec(**{**bare.__dict__, "portfolio_allocation": 0.3}) roomy.check_order_size() assert roomy.order_notional == pytest.approx(15.0) + + +def test_the_outer_level_never_lands_inside_the_inner_one(): + """XYZ:DRAM-USD quotes 0.35 bp, where the playbook's S+1 (1.35) falls + inside max(2, S/2) (2.0) and the ladder inverts.""" + from flybrain.posture import base_levels_from_spread + + for spread in (0.1, 0.35, 1.75, 2.0, 8.0, 20.0): + first, second = base_levels_from_spread(spread) + assert second > first, f"levels inverted at S={spread}" + assert base_levels_from_spread(0.35) == (2.0, 3.0) + assert base_levels_from_spread(8.0) == (4.0, 9.0) # wide markets unchanged + tight = build_config( + MarketSpec(**{**SPEC.__dict__, "picked_spread_bps": 0.35}), NEUTRAL + ) + buys = _spreads(tight["buy_spreads"]) + assert buys == sorted(buys) and len(set(buys)) == 2 From 8345c2bd82f12b27b6218c4a80ed9ef5867cb9de Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 13:01:08 -0700 Subject: [PATCH 31/48] (feat) an aroused fly leans in, and the rest of the config is the expert's Two changes to what the fly decides, and one to everything it does not. Arousal now tightens the spread and raises the share of the book quoted, where before it only widened. The gains are signed (-0.5 on spread, +0.5 on size) rather than positive-only, so the old direction is one config away and the two are finally testable against each other on the same market. Neither sign is derived from anything: arousal is the descending population's rate against its own recent average, and nothing in the pipeline ties it to volatility. The size multiplier is clamped so a scaled-down cycle never sizes an order under the venue minimum and a scaled-up one never quotes more than the whole book. Everything that is not spread, floor or feedback now matches Market Making Expert's balanced profile: inventory band 0.5/0.35/0.65, three executors a level, min_skew 1.5 -- the expert's third domain, which the fly had no answer to at all -- effectivization at 120s, the tolerance triple, and the global SL layer spelled out. Several of these the fly had been leaving to the controller's defaults, which means a hummingbot release could have moved its risk without anyone deciding to. One of them is a loosening and worth saying out loud: the per-position stop goes from 0.02 to the expert's 0.05. The loop's own 4%-of-capital loss stop is unchanged. --- agents/market_making_fly/flybrain/decoder.py | 36 ++++++++-- agents/market_making_fly/flybrain/posture.py | 55 ++++++++++++++-- .../market_making_fly/routines/fly_brain.py | 4 +- .../market_making_fly/routines/fly_report.py | 9 ++- .../market_making_fly/routines/fly_status.py | 2 + .../skills/fly_decoder/SKILL.md | 13 ++-- .../tests/test_fly_decoder.py | 33 ++++++---- .../tests/test_fly_market.py | 2 +- .../tests/test_fly_posture.py | 65 ++++++++++++++++--- 9 files changed, 180 insertions(+), 39 deletions(-) diff --git a/agents/market_making_fly/flybrain/decoder.py b/agents/market_making_fly/flybrain/decoder.py index e6f49224e..c1ec7ddf9 100644 --- a/agents/market_making_fly/flybrain/decoder.py +++ b/agents/market_making_fly/flybrain/decoder.py @@ -7,7 +7,7 @@ * ``trend_hz`` — mean DNp20 right rate minus mean left rate (stonkfly's BUY/SELL cells). Sign → which way the reference price leans. * ``arousal_hz`` — mean rate of the descending-neuron population. Higher → - wider spreads. + tighter spreads and a larger share of the book quoted. * ``gate`` — DNpe017 spikes ≥ 1, required for a trending call. Channels are z-scored against a rolling per-pair baseline. Stonkfly's own run @@ -47,9 +47,21 @@ class DecoderSettings: warmup: int = 10 z_regime: float = 1.0 z_pause: float = 2.5 - spread_gain: float = 0.5 + # Negative: an aroused fly quotes *tighter*, not wider. Arousal is the + # descending population's rate against its own recent average — nothing + # ties it to volatility, so neither sign is derived from anything. This one + # says an active market is one to lean into; +0.5 says it is one to back + # away from, which is the textbook answer. Both are testable against each + # other on the same market and neither has been. + spread_gain: float = -0.5 spread_min: float = 0.6 spread_max: float = 2.5 + # And it quotes more of the book while it is active. Bounded the same way, + # then clamped in build_config so an order never falls under the venue + # minimum or the allocation over 1. + size_gain: float = 0.5 + size_min: float = 0.6 + size_max: float = 2.5 shift_gain_bps: float = 1.0 max_shift_bps: float = 3.0 center_bias: bool = True @@ -59,12 +71,22 @@ def __post_init__(self): raise ValueError("window >= 2 and 1 <= warmup <= window required") if not 0 < self.z_regime < self.z_pause: raise ValueError("0 < z_regime < z_pause required") - if not 0 < self.spread_min <= 1 <= self.spread_max: - raise ValueError("spread_min <= 1 <= spread_max required") - for name in ("spread_gain", "shift_gain_bps", "max_shift_bps"): + for lo, hi, what in ( + (self.spread_min, self.spread_max, "spread"), + (self.size_min, self.size_max, "size"), + ): + if not 0 < lo <= 1 <= hi: + raise ValueError(f"{what}_min <= 1 <= {what}_max required") + for name in ("shift_gain_bps", "max_shift_bps"): value = getattr(self, name) if not math.isfinite(value) or value <= 0: raise ValueError(f"{name} must be finite and positive") + # The two arousal gains carry a direction, so they may be negative — + # but not zero, which would mean the channel is read and discarded. + for name in ("spread_gain", "size_gain"): + value = getattr(self, name) + if not math.isfinite(value) or value == 0: + raise ValueError(f"{name} must be finite and non-zero") @dataclass(frozen=True) @@ -121,6 +143,7 @@ def _z(value: float, history: list[float], center: bool) -> float: class Posture: regime: str spread_mult: float + size_mult: float shift_bps: float trend_z: float arousal_z: float @@ -138,6 +161,7 @@ def from_dict(cls, data: dict) -> "Posture": NEUTRAL = Posture( regime="ranging", spread_mult=1.0, + size_mult=1.0, shift_bps=0.0, trend_z=0.0, arousal_z=0.0, @@ -176,12 +200,14 @@ def decode(channels: Channels, baseline: Baseline, s: DecoderSettings) -> Postur gate = channels.gate_spikes >= 1 regime = classify(trend_z, arousal_z, gate, s) spread_mult = min(s.spread_max, max(s.spread_min, 1 + s.spread_gain * arousal_z)) + size_mult = min(s.size_max, max(s.size_min, 1 + s.size_gain * arousal_z)) shift = max(-s.max_shift_bps, min(s.max_shift_bps, s.shift_gain_bps * trend_z)) if not gate: shift = 0.0 # no descending gate spike, no directional lean return Posture( regime=regime, spread_mult=round(spread_mult, 4), + size_mult=round(size_mult, 4), shift_bps=round(shift, 3), trend_z=round(trend_z, 4), arousal_z=round(arousal_z, 4), diff --git a/agents/market_making_fly/flybrain/posture.py b/agents/market_making_fly/flybrain/posture.py index ad1467efe..6666e2f8e 100644 --- a/agents/market_making_fly/flybrain/posture.py +++ b/agents/market_making_fly/flybrain/posture.py @@ -60,12 +60,23 @@ class MarketSpec: # 0 means "use the venue default"; pass the exchange's real figure when # known, since the take-profit floor is derived from it. maker_fee_bps: float = 0.0 - portfolio_allocation: float = 0.2 - target_base_pct: float = 0.4 - min_base_pct: float = 0.3 - max_base_pct: float = 0.5 - max_active_executors_by_level: int = 2 - global_stop_loss: float = 0.02 + # Everything below is Market Making Expert's balanced profile, which is the + # vetted steady state for this controller. The fly's own contribution is + # the spread geometry, the fee-derived floor and the dopamine loop; there + # was no reason for the rest of the config to differ from what the expert + # deploys, and several of these were quietly the controller's defaults + # rather than a choice. + portfolio_allocation: float = 0.15 + target_base_pct: float = 0.5 + min_base_pct: float = 0.35 + max_base_pct: float = 0.65 + max_active_executors_by_level: int = 3 + min_skew: float = 1.5 + effectivization_time: int = 120 + price_distance_tolerance: float = 0.0005 + refresh_tolerance: float = 0.0005 + tolerance_scaling: float = 1.2 + global_stop_loss: float = 0.05 leverage_cap: int = 5 # Exchange minimum per order. pmm_mister sizes one cycle as # total_amount_quote × portfolio_allocation, split across both sides and @@ -105,6 +116,11 @@ def __post_init__(self): def is_spot(self) -> bool: return self.market_type == venue.SPOT + @property + def min_portfolio_allocation(self) -> float: + """The smallest allocation whose orders still clear the venue minimum.""" + return self.min_order_notional * ORDER_SIZE_MARGIN * 4 / self.total_amount_quote + @property def min_spread_bps(self) -> float: """The closest to mid a quote may sit: one maker fee. @@ -187,6 +203,17 @@ def build_config(spec: MarketSpec, posture: Posture) -> dict: if posture.regime not in TIMING: raise ValueError(f"Unknown regime {posture.regime!r}") spec.check_order_size() + # Arousal moves how much of the book is quoted as well as how tight: an + # active market gets leaned into on both. Clamped so a scaled-down cycle + # never sizes an order under the venue minimum, nor a scaled-up one over + # the whole book. + allocation = min( + 1.0, + max( + spec.min_portfolio_allocation, + spec.portfolio_allocation * posture.size_mult, + ), + ) l1, l2 = base_levels_bps(spec) levels = [l1 * posture.spread_mult, l2 * posture.spread_mult] shift = max(-levels[0] / 2, min(levels[0] / 2, posture.shift_bps)) @@ -200,7 +227,7 @@ def build_config(spec: MarketSpec, posture: Posture) -> dict: "connector_name": spec.connector_name, "trading_pair": spec.trading_pair, "total_amount_quote": spec.total_amount_quote, - "portfolio_allocation": spec.portfolio_allocation, + "portfolio_allocation": round(allocation, 6), "leverage": spec.leverage, "target_base_pct": spec.target_base_pct, "min_base_pct": spec.min_base_pct, @@ -216,8 +243,22 @@ def build_config(spec: MarketSpec, posture: Posture) -> dict: "take_profit_order_type": 3, "open_order_type": 3, "max_active_executors_by_level": spec.max_active_executors_by_level, + # Stated rather than left to the controller's defaults, so a change in + # hummingbot cannot move the fly's risk without anyone deciding to. + "buy_position_effectivization_time": spec.effectivization_time, + "sell_position_effectivization_time": spec.effectivization_time, + "price_distance_tolerance": spec.price_distance_tolerance, + "refresh_tolerance": spec.refresh_tolerance, + "tolerance_scaling": spec.tolerance_scaling, + # The expert's third domain, which the fly had no answer to at all: + # quote the accumulating side wider until the inventory comes back. + "min_skew": spec.min_skew, + "tick_mode": False, + "global_tp_enabled": False, "global_sl_enabled": True, "global_stop_loss": spec.global_stop_loss, + "global_sl_activation_from": "target_base", + "global_pnl_reference": "position", "manual_kill_switch": posture.regime == "pause", } if not spec.is_spot: diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index ffc61c920..1ffc06463 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -571,6 +571,7 @@ async def pace() -> None: ) b.kpi("Regime", posture.regime) b.kpi("Spread ×", f"{posture.spread_mult:.2f}") + b.kpi("Size ×", f"{posture.size_mult:.2f}") b.kpi("Lean", f"{posture.shift_bps:+.2f} bp") b.kpi("Trend z", f"{posture.trend_z:+.2f}") b.kpi("Arousal z", f"{posture.arousal_z:+.2f}") @@ -679,7 +680,8 @@ async def pace() -> None: chat_id=chat_id, text=( f"🪰 {pair} tick {tick - 1}: {row['execution']['status']} — " - f"{posture.regime} ×{posture.spread_mult:.2f} {posture.shift_bps:+.1f}bp " + f"{posture.regime} ×{posture.spread_mult:.2f} " + f"size ×{posture.size_mult:.2f} {posture.shift_bps:+.1f}bp " f"({row['execution']['reason']})" )[:900], ) diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index a296bb193..218664d98 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -346,7 +346,14 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: width=CARDS, ) builder.kpi("Regime", str(last_posture.get("regime", "—")).upper(), width=CARDS) - builder.kpi("Spread ×", _fmt(last_posture.get("spread_mult")), width=CARDS) + # One card, not two: both multipliers are read off the same channel, and + # the stack's height is matched to the brain beside it at six cards. + builder.kpi( + "Spread / size ×", + f"{_fmt(last_posture.get('spread_mult'))} / " + f"{_fmt(last_posture.get('size_mult'))}", + width=CARDS, + ) builder.kpi( "Lean", f"{_fmt(last_posture.get('shift_bps'), 2, plus=True)} bp", diff --git a/agents/market_making_fly/routines/fly_status.py b/agents/market_making_fly/routines/fly_status.py index 4cd13d10f..15c046a81 100644 --- a/agents/market_making_fly/routines/fly_status.py +++ b/agents/market_making_fly/routines/fly_status.py @@ -86,6 +86,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: if posture: lines.append( f"{pair}: {posture['regime']} spread_x={posture['spread_mult']:.2f} " + f"size_x={posture.get('size_mult', 1.0):.2f} " f"lean_bp={posture['shift_bps']:+.2f} trend_z={posture['trend_z']:+.2f} " f"arousal_z={posture['arousal_z']:+.2f} baseline_n={len((baselines.get(pair) or {}).get('trend', []))}" ) @@ -115,6 +116,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: "Pair": pair, "Regime": (p or {}).get("regime", "—"), "Spread ×": (p or {}).get("spread_mult", "—"), + "Size ×": (p or {}).get("size_mult", "—"), "Lean bp": (p or {}).get("shift_bps", "—"), "Trend z": (p or {}).get("trend_z", "—"), "Arousal z": (p or {}).get("arousal_z", "—"), diff --git a/agents/market_making_fly/skills/fly_decoder/SKILL.md b/agents/market_making_fly/skills/fly_decoder/SKILL.md index af1e264be..ed7e679b8 100644 --- a/agents/market_making_fly/skills/fly_decoder/SKILL.md +++ b/agents/market_making_fly/skills/fly_decoder/SKILL.md @@ -21,7 +21,7 @@ user wants to *look* at the run, `fly_status` when you need to quote figures. | Field | Cells | Meaning | |---|---|---| | `trend_hz` | DNp20 right mean rate − left mean rate | lean direction; stonkfly's BUY/SELL cells | -| `arousal_hz` | mean rate of the 1,314 descending neurons (minus the readouts) | spread width | +| `arousal_hz` | mean rate of the 1,314 descending neurons (minus the readouts) | spread width **and** how much of the book is quoted | | `gate_spikes` | DNpe017 | ≥ 1 required for a trending call and for any lean | | `kc_spikes` | Kenyon cells | did the chart reach the mushroom body at all (0 = the fly saw nothing useful) | | `reward_spikes` / `aversive_spikes` | PAM11 / PPL101 | did the pulse arrive | @@ -32,8 +32,12 @@ user wants to *look* at the run, `fly_status` when you need to quote figures. window 60 observations. `warm=False` for the first 10 — neutral posture. * Regime precedence: `pause` (arousal_z ≥ 2.5) > `volatile` (≥ 1) > `trending_up/down` (gate and |trend_z| ≥ 1) > `quiet` (arousal_z ≤ −1) > `ranging`. -* `spread ×` = clip(1 + 0.5·arousal_z, 0.6, 2.5). `lean` = clip(trend_z, ±3 bp), 0 without - a gate spike, and capped at half the first spread level when mapped. +* `spread ×` = clip(1 − 0.5·arousal_z, 0.6, 2.5) — an aroused fly quotes **tighter**. + `size ×` = clip(1 + 0.5·arousal_z, 0.6, 2.5) — and quotes **more** of the book, + clamped so an order never falls under the venue minimum nor the allocation over 1. + The sign on each is a choice, not a finding: arousal is a population rate against + its own average and nothing ties it to volatility. `lean` = clip(trend_z, ±3 bp), 0 + without a gate spike, and capped at half the first spread level when mapped. * Mapping: level 1 = max(2, S/2) bp, level 2 = S+1 bp (S = scanner spread), times `spread ×`, buy −lean / sell +lean, floor 3 bp; TP = max(4 bp, 2.2 × round-trip fee, first level); timing per regime; `pause` sets `manual_kill_switch`. @@ -47,7 +51,8 @@ update failed · `HALT` loop stopped · `TICK_ERROR` data fetch failed, loop con ## What you may say -* "The fly's arousal channel is 1.8 σ above its baseline on DRAM, so it widened to 1.9×." +* "The fly's arousal channel is 1.8 σ above its baseline on DRAM, so it tightened to + 0.6× and quoted 1.9× the usual share of the book." * "No gate spike this observation, so no lean regardless of trend." * "Changed edges rose from 0 to 312 after the first aversive pulse." diff --git a/agents/market_making_fly/tests/test_fly_decoder.py b/agents/market_making_fly/tests/test_fly_decoder.py index 5e80f4e18..f21e1da85 100644 --- a/agents/market_making_fly/tests/test_fly_decoder.py +++ b/agents/market_making_fly/tests/test_fly_decoder.py @@ -81,15 +81,26 @@ def test_no_gate_no_lean(): assert p.shift_bps == 0 and p.regime == "ranging" -def test_arousal_widens_and_pauses(): +def test_arousal_tightens_sizes_up_and_pauses(): + """An aroused fly leans in: tighter quotes and more of the book. The sign + is a choice, not a finding — arousal is a population rate against its own + average and nothing ties it to volatility.""" b = Baseline() _warm(b) - wide = decode(Channels(0.0, 11.5, 0), b, S) - assert wide.spread_mult > 1 + hot = decode(Channels(0.0, 11.5, 0), b, S) + assert hot.arousal_z > 1 + assert hot.spread_mult < 1 and hot.size_mult > 1 b2 = Baseline() _warm(b2) - pause = decode(Channels(0.0, 100.0, 0), b2, S) - assert pause.regime == "pause" and pause.spread_mult == S.spread_max + calm = decode(Channels(0.0, 8.0, 0), b2, S) + assert calm.arousal_z < 0 + assert calm.spread_mult > 1 and calm.size_mult < 1 + # the breaker still fires on the same channel, and both knobs stay clipped + b3 = Baseline() + _warm(b3) + pause = decode(Channels(0.0, 100.0, 0), b3, S) + assert pause.regime == "pause" + assert pause.spread_mult == S.spread_min and pause.size_mult == S.size_max def test_baseline_window_and_roundtrip(): @@ -101,7 +112,7 @@ def test_baseline_window_and_roundtrip(): def test_posture_roundtrip(): - p = Posture("quiet", 0.8, -1.0, -0.2, -1.3, True, True) + p = Posture("quiet", 0.8, 1.0, -1.0, -0.2, -1.3, True, True) assert Posture.from_dict(p.to_dict()) == p @@ -121,14 +132,14 @@ def test_settings_validation(): def test_hysteresis(): h = Hysteresis(min_apply_interval_sec=300) - base = Posture("ranging", 1.0, 0.0, 0, 0, False, True) + base = Posture("ranging", 1.0, 1.0, 0.0, 0, 0, False, True) assert should_apply(None, base, None, 1000, h)[0] - same = Posture("ranging", 1.05, 0.2, 0, 0, False, True) + same = Posture("ranging", 1.05, 1.0, 0.2, 0, 0, False, True) assert not should_apply(base, same, 0, 1000, h)[0] - regime = Posture("volatile", 1.0, 0.0, 0, 1.2, False, True) + regime = Posture("volatile", 1.0, 1.0, 0.0, 0, 1.2, False, True) assert should_apply(base, regime, 0, 1000, h)[0] assert not should_apply(base, regime, 900, 1000, h)[0] # cooldown - wider = Posture("ranging", 1.2, 0.0, 0, 0, False, True) + wider = Posture("ranging", 1.2, 1.0, 0.0, 0, 0, False, True) assert should_apply(base, wider, 0, 1000, h)[0] - lean = Posture("ranging", 1.0, 0.6, 0, 0, True, True) + lean = Posture("ranging", 1.0, 1.0, 0.6, 0, 0, True, True) assert should_apply(base, lean, 0, 1000, h)[0] diff --git a/agents/market_making_fly/tests/test_fly_market.py b/agents/market_making_fly/tests/test_fly_market.py index 1315b0280..5456f1c1b 100644 --- a/agents/market_making_fly/tests/test_fly_market.py +++ b/agents/market_making_fly/tests/test_fly_market.py @@ -97,7 +97,7 @@ def test_candle_payload_shapes(): def test_required_collateral(): spec = MarketSpec("hyperliquid_perpetual", "XYZ:A-USD", 500, 8.0, leverage=2) - assert required_collateral([spec, spec]) == pytest.approx(2 * 500 * 0.5 / 2) + assert required_collateral([spec, spec]) == pytest.approx(2 * 500 * 0.65 / 2) class _MarketData: diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py index c2cedcdbe..8f9964df8 100644 --- a/agents/market_making_fly/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -38,7 +38,7 @@ def test_neutral_config_matches_hip3_base(): cfg["controller_name"] == "pmm_mister" and cfg["controller_type"] == "generic" ) assert cfg["trading_pair"] == "XYZ:DRAM-USD" - assert cfg["global_sl_enabled"] is True and cfg["global_stop_loss"] == 0.02 + assert cfg["global_sl_enabled"] is True and cfg["global_stop_loss"] == 0.05 assert cfg["manual_kill_switch"] is False assert (cfg["executor_refresh_time"], cfg["buy_cooldown_time"]) == TIMING["ranging"] assert cfg["open_order_type"] == 3 and cfg["take_profit_order_type"] == 3 @@ -53,12 +53,12 @@ def test_take_profit_floor_beats_fees_and_spread(): def test_volatile_widens_quiet_tightens_with_floor(): - wide = build_config(SPEC, Posture("volatile", 2.0, 0.0, 0, 1.5, False, True)) + wide = build_config(SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, False, True)) assert _spreads(wide["buy_spreads"])[0] == pytest.approx(8 * BPS) assert (wide["executor_refresh_time"], wide["buy_cooldown_time"]) == TIMING[ "volatile" ] - tight = build_config(SPEC, Posture("quiet", 0.6, 0.0, 0, -1.5, False, True)) + tight = build_config(SPEC, Posture("quiet", 0.6, 1.0, 0.0, 0, -1.5, False, True)) # 4 bp × 0.6 = 2.4 bp, which clears this market's 1.3 bp fee floor assert _spreads(tight["buy_spreads"])[0] == pytest.approx(2.4 * BPS) assert (tight["executor_refresh_time"], tight["buy_cooldown_time"]) == TIMING[ @@ -67,12 +67,14 @@ def test_volatile_widens_quiet_tightens_with_floor(): def test_lean_is_asymmetric_and_capped(): - up = build_config(SPEC, Posture("trending_up", 1.0, 3.0, 2.0, 0, True, True)) + up = build_config(SPEC, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, True, True)) buy, sell = _spreads(up["buy_spreads"]), _spreads(up["sell_spreads"]) # lean capped at half of level 1 (4 bp → 2 bp), and 2 bp still clears the fee assert buy[0] == pytest.approx(2 * BPS) and sell[0] == pytest.approx(6 * BPS) assert buy[1] == pytest.approx(7 * BPS) and sell[1] == pytest.approx(11 * BPS) - down = build_config(SPEC, Posture("trending_down", 1.0, -3.0, -2.0, 0, True, True)) + down = build_config( + SPEC, Posture("trending_down", 1.0, 1.0, -3.0, -2.0, 0, True, True) + ) assert _spreads(down["sell_spreads"])[0] == pytest.approx(2 * BPS) assert _spreads(down["buy_spreads"])[0] == pytest.approx(6 * BPS) @@ -90,12 +92,14 @@ def test_the_spread_floor_is_the_market_own_fee(): ) assert dear.min_spread_bps == pytest.approx(7.5) # binance spot # a lean that would quote inside the fee is pushed back out to it - leaned = build_config(dear, Posture("trending_up", 1.0, 3.0, 2.0, 0, True, True)) + leaned = build_config( + dear, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, True, True) + ) assert min(_spreads(leaned["buy_spreads"])) == pytest.approx(7.5 * BPS) def test_pause_sets_kill_switch(): - cfg = build_config(SPEC, Posture("pause", 2.5, 0.0, 0, 3.0, False, True)) + cfg = build_config(SPEC, Posture("pause", 2.5, 1.0, 0.0, 0, 3.0, False, True)) assert cfg["manual_kill_switch"] is True @@ -103,7 +107,9 @@ def test_every_spread_respects_min(): for regime in TIMING: for mult in (0.6, 1.0, 2.5): for shift in (-3.0, 0.0, 3.0): - cfg = build_config(SPEC, Posture(regime, mult, shift, 0, 0, True, True)) + cfg = build_config( + SPEC, Posture(regime, mult, 1.0, shift, 0, 0, True, True) + ) for key in ("buy_spreads", "sell_spreads"): assert min(_spreads(cfg[key])) >= SPEC.min_spread_bps * BPS - 1e-12 @@ -159,7 +165,7 @@ def test_the_fee_floor_follows_the_venue(): def test_config_diff(): a = build_config(SPEC, NEUTRAL) - b = build_config(SPEC, Posture("volatile", 2.0, 0.0, 0, 1.5, False, True)) + b = build_config(SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, False, True)) diff = config_diff(a, b) assert "buy_spreads" in diff and "trading_pair" not in diff assert config_diff(None, a) == a @@ -196,3 +202,44 @@ def test_the_outer_level_never_lands_inside_the_inner_one(): ) buys = _spreads(tight["buy_spreads"]) assert buys == sorted(buys) and len(set(buys)) == 2 + + +def test_size_follows_arousal_and_stays_inside_both_limits(): + """The fly quotes more of the book when aroused, but never so little that + an order falls under the venue minimum, nor more than the whole book.""" + # 200 quote at 0.3 is a 15 order; scaled down by 0.6 it would be 9, under + # the 12 the venue needs once the base amount is rounded, so the floor + # binds before the multiplier does. + spec = MarketSpec( + **{**SPEC.__dict__, "total_amount_quote": 200, "portfolio_allocation": 0.3} + ) + hot = build_config(spec, Posture("ranging", 1.0, 2.5, 0.0, 0, 0, False, True)) + calm = build_config(spec, Posture("ranging", 1.0, 0.6, 0.0, 0, 0, False, True)) + assert hot["portfolio_allocation"] > spec.portfolio_allocation + assert calm["portfolio_allocation"] < spec.portfolio_allocation + assert calm["portfolio_allocation"] == pytest.approx(0.24) + assert calm["portfolio_allocation"] == pytest.approx(spec.min_portfolio_allocation) + # and a big book scaled up still cannot quote more than all of itself + big = MarketSpec(**{**SPEC.__dict__, "portfolio_allocation": 0.5}) + assert build_config(big, Posture("ranging", 1.0, 2.5, 0.0, 0, 0, False, True))[ + "portfolio_allocation" + ] == pytest.approx(1.0) + + +def test_the_config_matches_the_experts_balanced_profile(): + """Everything that is not spread, floor or feedback is the expert's vetted + profile, and is stated rather than left to the controller's defaults.""" + cfg = build_config(SPEC, NEUTRAL) + assert cfg["target_base_pct"] == 0.5 + assert cfg["min_base_pct"] == 0.35 and cfg["max_base_pct"] == 0.65 + assert cfg["max_active_executors_by_level"] == 3 + assert cfg["min_skew"] == 1.5 + assert cfg["buy_position_effectivization_time"] == 120 + assert cfg["sell_position_effectivization_time"] == 120 + assert cfg["price_distance_tolerance"] == 0.0005 + assert cfg["refresh_tolerance"] == 0.0005 + assert cfg["tolerance_scaling"] == 1.2 + assert cfg["global_tp_enabled"] is False and cfg["global_sl_enabled"] is True + assert cfg["global_sl_activation_from"] == "target_base" + assert cfg["global_pnl_reference"] == "position" + assert cfg["tick_mode"] is False From 8ad6ebf35ec35289c578fa57b6ec36a64c2f6940 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 13:07:43 -0700 Subject: [PATCH 32/48] (fix) the guard stops vetoing the floor the posture builder just applied Live on DRAM, tick 18: a trending posture leaned the buy side down to the fee floor, build_config clamped it there, the config serialized it as 0.00013 -- and check_config vetoed it, because 0.00013 parses to a float one ulp under 1.3 * 1e-4. The guard was refusing its own arithmetic, and every leaned posture that reaches the floor would have been refused the same way. A real apply was lost to it. Both floor comparisons now carry a relative tolerance far below any width that could matter, and the test reproduces the exact config that failed. Two reporting fixes from reading the same run: - "the observation ran 10.0 s of neural time" was the brain's cumulative time, not the observation's: twenty 500 ms observations. The worker now records its own observation_ms and the sentence says both. - the readout notes were positioned as a fraction of the plot, so drawing the panel to its full height pushed them outside the margin meant to hold them. They offset in pixels now, clear of the tick labels, and the panel fills its 400 px floor like the P&L curve does. --- agents/market_making_fly/flybrain/brainviz.py | 15 ++++++++--- agents/market_making_fly/flybrain/guard.py | 13 +++++++-- agents/market_making_fly/flybrain/worker.py | 4 +++ .../market_making_fly/routines/fly_report.py | 27 +++++++++++++++---- .../market_making_fly/tests/test_fly_guard.py | 25 +++++++++++++++++ 5 files changed, 74 insertions(+), 10 deletions(-) diff --git a/agents/market_making_fly/flybrain/brainviz.py b/agents/market_making_fly/flybrain/brainviz.py index 024274cc2..416fc897c 100644 --- a/agents/market_making_fly/flybrain/brainviz.py +++ b/agents/market_making_fly/flybrain/brainviz.py @@ -164,6 +164,15 @@ def coverage() -> tuple[int, int, int]: return len(cloud["index"]), cloud["mapped_total"], cloud["neurons_total"] +READOUT_TOP, READOUT_BOTTOM = 36, 64 + + +def _note_y(height: int) -> float: + """Paper-coordinate y that puts a note clear of the tick labels at any + height. 56 px: the axis numbers sit about 20 below the axis.""" + return -56 / max(1, height - READOUT_TOP - READOUT_BOTTOM) + + def readout_figure(neural: dict, posture: dict | None = None, height: int = 460): """The three channels the posture is decoded from, as they were measured. @@ -199,7 +208,7 @@ def readout_figure(neural: dict, posture: dict | None = None, height: int = 460) trend = right - left fig.update_layout( height=height, - margin=dict(l=96, r=20, t=36, b=64), + margin=dict(l=96, r=20, t=READOUT_TOP, b=READOUT_BOTTOM), paper_bgcolor=GROUND, plot_bgcolor=GROUND, font=dict(color="#c8d4e8", family="monospace", size=11), @@ -218,7 +227,7 @@ def readout_figure(neural: dict, posture: dict | None = None, height: int = 460) annotations=[ dict( x=0, - y=-0.30, + y=_note_y(height), xref="paper", yref="paper", showarrow=False, @@ -239,7 +248,7 @@ def readout_figure(neural: dict, posture: dict | None = None, height: int = 460) ), dict( x=1, - y=-0.30, + y=_note_y(height), xref="paper", yref="paper", showarrow=False, diff --git a/agents/market_making_fly/flybrain/guard.py b/agents/market_making_fly/flybrain/guard.py index f2e19e392..78d62612f 100644 --- a/agents/market_making_fly/flybrain/guard.py +++ b/agents/market_making_fly/flybrain/guard.py @@ -122,13 +122,22 @@ def _spreads(value: str) -> list[float]: return [float(x) for x in str(value).split(",") if x.strip()] +# A config that sits exactly on a floor reaches this check as a serialized +# decimal, and the float it parses back to can be one ulp under the float the +# floor computes to: 1.3 bp writes as 0.00013, which is less than 1.3 * 1e-4. +# Vetoing that is the guard refusing the posture builder's own arithmetic — it +# cost a real apply on 2026-09-13, tick 18. The tolerance is relative and far +# below any width that could matter. +FLOOR_TOLERANCE = 1e-9 + + def check_config(config: dict, spec: MarketSpec) -> None: """Belt to ``posture.build_config``'s braces: refuse anything below the floors.""" for key in ("buy_spreads", "sell_spreads"): for level in _spreads(config[key]): - if level < spec.min_spread_bps * BPS: + if level < spec.min_spread_bps * BPS * (1 - FLOOR_TOLERANCE): raise Veto(f"{key} level {level} below {spec.min_spread_bps} bp") - if float(config["take_profit"]) < take_profit_floor(spec): + if float(config["take_profit"]) < take_profit_floor(spec) * (1 - FLOOR_TOLERANCE): raise Veto("take_profit below fee floor") if int(config["leverage"]) > spec.leverage_cap: raise Veto(f"leverage {config['leverage']} above cap {spec.leverage_cap}") diff --git a/agents/market_making_fly/flybrain/worker.py b/agents/market_making_fly/flybrain/worker.py index eae599d17..a3439fb9b 100644 --- a/agents/market_making_fly/flybrain/worker.py +++ b/agents/market_making_fly/flybrain/worker.py @@ -122,6 +122,10 @@ def observe( "total_spikes": int(counts.sum()), "stimulus": reinforcement, "stimulus_ms": delivered * b.dt, + # This observation's neural time, and the brain's total since it + # was seeded. Reporting the second as the first says a 500 ms + # observation ran for ten seconds once twenty of them have gone by. + "observation_ms": float(neural_ms), "brain_ms": b.sim_ms, "compute_seconds": wall, "spike_sha256": hashlib.sha256(counts.tobytes()).hexdigest(), diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index 218664d98..6a8c603f6 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -316,10 +316,21 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: f"{neural.get('gate_spikes', '—')} DNpe017 spike(s)." + ("" if last_posture.get("warm", True) else " Baseline still forming.") ) + # A run recorded before the loop knew its own observation length has only + # the brain's lifetime; printing "— ms" for the missing half reads worse + # than not claiming it. + ran = ( + f" The observation ran {_fmt(neural['observation_ms'], 0)} ms of neural time" + if neural.get("observation_ms") + else " The observation" + ) + if neural.get("brain_ms"): + ran += ( + f" ({_fmt(float(neural['brain_ms']) / 1000, 1)} s on this brain since it " + "was seeded)" + ) observation_line = ( - f" The observation ran " - f"{_fmt(float(neural['brain_ms']) / 1000, 1) if neural.get('brain_ms') else '—'} s " - f"of neural time and produced {neural.get('total_spikes', 0):,} spikes, " + f"{ran} produced {neural.get('total_spikes', 0):,} spikes, " f"{neural.get('kc_spikes', 0):,} of them in Kenyon cells. Stimulus " f"{observed.get('stimulus', 'none')} — {neural.get('reward_spikes', 0)} PAM11 " f"and {neural.get('aversive_spikes', 0)} PPL101 spikes — left " @@ -327,7 +338,13 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: f"{(memory.get('plastic_edges') or 0):,} plastic edges away from baseline, " f"mean efficacy {_fmt(memory.get('mean_efficacy'), 5)}." ) - builder.section("THE FLY BRAIN", posture_line + observation_line) + builder.section( + "THE FLY BRAIN", + posture_line + + observation_line + + " In the scene, brighter and larger is more spikes in the window and the " + "dim haze is every cell that stayed silent; hover one for its group.", + ) # Cards on the left, the brain on the right, the same way the fly and its # frame sit above — five columns and seven of the runtime's twelve, both # collapsing to full width below its 800px breakpoint. @@ -372,7 +389,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: ), width=BRAIN, ) - builder.plotly(readout_figure(neural, last_posture, height=260)) + builder.plotly(readout_figure(neural, last_posture, height=FULL_ROW - PANEL_CHROME)) # ── DECISIONS & POSITIONS ──────────────────────────────────────────────── # One panel: what the fly called, and what those calls left it holding. builder.section( diff --git a/agents/market_making_fly/tests/test_fly_guard.py b/agents/market_making_fly/tests/test_fly_guard.py index dd4855366..afbabea94 100644 --- a/agents/market_making_fly/tests/test_fly_guard.py +++ b/agents/market_making_fly/tests/test_fly_guard.py @@ -146,3 +146,28 @@ def test_default_max_loss_and_roundtrip(): assert default_max_loss([SPEC], GuardSettings(max_loss_quote=7.0)) == 7.0 st = GuardState(applies_today=3, closed_ticks={"X:Y-USD": 1}, halted="x") assert GuardState.from_dict(st.to_dict()) == st + + +def test_a_config_sitting_exactly_on_the_floor_is_not_vetoed(): + """Live on 2026-09-13, tick 18: a trending posture leaned the buy side down + to the fee floor, the config serialized it as 0.00013, and the guard vetoed + it because 0.00013 < 1.3 * 1e-4 by one ulp. The guard was refusing the + posture builder's own arithmetic, and a real apply was lost.""" + from flybrain.decoder import Posture + from flybrain.posture import MarketSpec, build_config + + spec = MarketSpec( + connector_name="hyperliquid_perpetual", + trading_pair="XYZ:DRAM-USD", + total_amount_quote=200, + picked_spread_bps=0.35, + leverage=1, + portfolio_allocation=0.3, + maker_fee_bps=1.3, + ) + leaned = Posture("trending_up", 1.14, 1.0, 1.27, 2.0, 0.3, True, True) + config = build_config(spec, leaned) + assert min(float(x) for x in config["buy_spreads"].split(",")) == pytest.approx( + spec.min_spread_bps * 1e-4 + ) + check_config(config, spec) # must not raise From 0217ae3595db0d8c0cc6efc5231863479a8a67f0 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 13:09:49 -0700 Subject: [PATCH 33/48] (fix) a cancelled quote is not a trade The report said 70 trades on a run with five fills and no completed pair. It was summing every close type, and 71 of them were EARLY_STOP -- the controller replacing its own unfilled orders every 30 seconds. A number that overstates activity fourteen-fold is worse than no number. The card now counts round trips: closes that ended a position and realized its P&L. On this run that is 0, and the section says the five positions were opened and are still held, which is the whole story of the run. --- .../market_making_fly/routines/fly_report.py | 52 ++++++++++++++----- 1 file changed, 40 insertions(+), 12 deletions(-) diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index 6a8c603f6..0fb98e936 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -169,19 +169,31 @@ def _pnl_figure(events: list[dict]) -> go.Figure | None: return fig +# A quote that was cancelled on refresh never traded. Counting every close +# type as a trade said 70 on a run with five fills and no completed pair: 71 +# of those were EARLY_STOP, which is the controller replacing its own unfilled +# orders. These are the closes that end a position and realize its P&L. +ROUND_TRIP_CLOSES = frozenset( + {"TAKE_PROFIT", "STOP_LOSS", "TRAILING_STOP", "TIME_LIMIT", "COMPLETED"} +) + + async def _holdings( client, connector_name: str, pairs: list[str] -) -> tuple[list[dict], int | None]: - """What the fly's bots hold now, and how many positions they have closed. - - The trade count is the sum of each controller's close-type counts — round - trips actually completed, not orders placed. ``None`` when no bot reported, - so the report can say "unknown" rather than "zero". +) -> tuple[list[dict], int | None, int]: + """What the fly's bots hold now, how many positions they closed, and how + many they are still holding. + + The trade count is round trips actually completed — closes that ended a + position and realized its P&L — not quotes placed, and not the quotes the + controller cancelled in order to replace them. ``None`` when no bot + reported, so the report can say "unknown" rather than "zero". """ market = LiveMarket(client, connector_name, "5m", 72) bots = await market.bots() rows: list[dict] = [] trades: int | None = None + held = 0 for pair in pairs: names = pair_names(pair) running, bot = LiveMarket.find_bot(bots, names.bot_name) @@ -192,7 +204,16 @@ async def _holdings( inner = perf.get("performance", perf) if isinstance(perf, dict) else {} closes = inner.get("close_type_counts") or {} if isinstance(closes, dict): - trades = (trades or 0) + sum(int(v or 0) for v in closes.values()) + trades = (trades or 0) + sum( + int(count or 0) + for name, count in closes.items() + if str(name).split(".")[-1] in ROUND_TRIP_CLOSES + ) + held = held + sum( + int(count or 0) + for name, count in closes.items() + if str(name).split(".")[-1] == "POSITION_HOLD" + ) positions = inner.get("positions_summary") or [] amount = sum( float(p.get("amount", 0) or 0) for p in positions if isinstance(p, dict) @@ -208,7 +229,7 @@ async def _holdings( "Volume": _fmt(inner.get("volume_traded")), } ) - return rows, trades + return rows, trades, held async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: @@ -252,8 +273,10 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: stale = observed is not latest client = await get_client(context._chat_id, context=context) - holdings, trades = ( - await _holdings(client, config.connector_name, pairs) if client else ([], None) + holdings, trades, held = ( + await _holdings(client, config.connector_name, pairs) + if client + else ([], None, 0) ) book_net = sum( float(info.get("net", 0) or 0) @@ -447,14 +470,19 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: (f"Halted: {halted}. " if halted else "") + f"Session high {_fmt(guard.get('session_high_net'), 4, plus=True)}, " f"{guard.get('ticks_since_high', 0)} observation(s) since, " - f"{guard.get('applies_today', 0)} config change(s) applied today." + f"{guard.get('applies_today', 0)} config change(s) applied today. " + + ( + f"{held} position(s) opened and still held." + if held + else "No position is open." + ) if guard.get("session_high_net") is not None else "No P&L has been reported yet, so the breakers have nothing to judge." ), ) builder.kpi("Ticks", f"{state.get('tick', 0):,}") builder.kpi("Net P&L", _fmt(book_net, 4, plus=True)) - builder.kpi("Trades", f"{trades:,}" if trades is not None else "—") + builder.kpi("Round trips", f"{trades:,}" if trades is not None else "—") builder.kpi("Volume", _fmt(book_volume)) pnl = _pnl_figure(events) if pnl is not None: From 7bc49646d5820df47241c9715e0d49af1a18d828 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 13:21:12 -0700 Subject: [PATCH 34/48] (feat) the memory rule finally reaches a decision The dopamine loop moved 3,353 KC->MBON synapses on the DRAM run and changed nothing the fly did, because the decoder never read MBONs. Those synapses are exactly what dopamine writes to, so MBON07 minus MBON11 -- approach minus avoidance -- is the only channel a P&L pulse can reach. It is now the fourth channel, z-scored against its own per-pair baseline like the others, and it drives the size the fly commits: +0.5 * valence_z added to the arousal term. Kenyon drive becomes the confidence test it was always described as. A scene that produces no sparse code, or far less than this pair usually produces, leaves every other channel reading the network's own noise -- so the posture is marked unconfident and the loop holds the config it already has, with the reason recorded, rather than applying a reading of nothing. Verified end to end on a fixture run: valence_z swings -2.0 to -0.6 and moves size_mult between 0.60 and 1.23 while spread tracks arousal separately, and confidence follows kc_spikes. This makes the loop a loop. It does not make it right: the pulse still reports the P&L change between two observations rather than credit for the posture that caused it, and the synapses still drift from endogenous activity. What it buys is that the question is now answerable -- valence_gain 0 is the control. --- agents/market_making_fly/flybrain/decoder.py | 74 ++++++++++++++++--- agents/market_making_fly/flybrain/worker.py | 19 +++++ .../market_making_fly/routines/fly_brain.py | 7 +- .../market_making_fly/routines/fly_report.py | 3 +- .../market_making_fly/routines/fly_status.py | 1 + .../skills/fly_decoder/SKILL.md | 12 ++- .../tests/test_fly_decoder.py | 48 ++++++++++-- .../market_making_fly/tests/test_fly_guard.py | 2 +- .../tests/test_fly_posture.py | 28 ++++--- 9 files changed, 161 insertions(+), 33 deletions(-) diff --git a/agents/market_making_fly/flybrain/decoder.py b/agents/market_making_fly/flybrain/decoder.py index c1ec7ddf9..7ffc69c4e 100644 --- a/agents/market_making_fly/flybrain/decoder.py +++ b/agents/market_making_fly/flybrain/decoder.py @@ -9,6 +9,15 @@ * ``arousal_hz`` — mean rate of the descending-neuron population. Higher → tighter spreads and a larger share of the book quoted. * ``gate`` — DNpe017 spikes ≥ 1, required for a trending call. +* ``valence_hz`` — mean MBON07 rate minus mean MBON11 rate: approach minus + avoidance. These are the cells the KC→MBON memory rule + writes to, so this is the only channel a P&L pulse can + reach. Higher → more of the book quoted. Without it the + dopamine loop changes synapses that change nothing. +* ``kc_spikes`` — Kenyon cell drive. Not a posture: a confidence test. A scene + that barely reaches the mushroom body leaves the other + channels reading noise, so the posture is marked unconfident + and the loop holds rather than applying it. Channels are z-scored against a rolling per-pair baseline. Stonkfly's own run proposed BUY six times out of six — a persistent turning bias of the circuit @@ -62,6 +71,13 @@ class DecoderSettings: size_gain: float = 0.5 size_min: float = 0.6 size_max: float = 2.5 + # What the fly has learned about scenes like this one, on the same scale as + # arousal and added to it, so the size the fly commits carries both "the + # market is active" and "this looked good last time". + valence_gain: float = 0.5 + # A scene this far below the mushroom body's own recent drive is one the + # fly effectively did not see; its z-scores are noise. + z_kc_quiet: float = -1.5 shift_gain_bps: float = 1.0 max_shift_bps: float = 3.0 center_bias: bool = True @@ -83,7 +99,9 @@ def __post_init__(self): raise ValueError(f"{name} must be finite and positive") # The two arousal gains carry a direction, so they may be negative — # but not zero, which would mean the channel is read and discarded. - for name in ("spread_gain", "size_gain"): + if self.z_kc_quiet >= 0: + raise ValueError("z_kc_quiet must be negative") + for name in ("spread_gain", "size_gain", "valence_gain"): value = getattr(self, name) if not math.isfinite(value) or value == 0: raise ValueError(f"{name} must be finite and non-zero") @@ -94,29 +112,44 @@ class Channels: trend_hz: float arousal_hz: float gate_spikes: int + valence_hz: float = 0.0 + kc_spikes: int = 0 def __post_init__(self): - if not math.isfinite(self.trend_hz) or not math.isfinite(self.arousal_hz): - raise ValueError("Nonfinite channel") - if self.arousal_hz < 0 or self.gate_spikes < 0: + for name in ("trend_hz", "arousal_hz", "valence_hz"): + if not math.isfinite(getattr(self, name)): + raise ValueError(f"Nonfinite channel {name}") + if self.arousal_hz < 0 or self.gate_spikes < 0 or self.kc_spikes < 0: raise ValueError("Negative rate or spike count") @dataclass class Baseline: - """Rolling per-pair history of the two channels; persisted in state.json.""" + """Rolling per-pair history of each channel; persisted in state.json.""" trend: list[float] = field(default_factory=list) arousal: list[float] = field(default_factory=list) + valence: list[float] = field(default_factory=list) + kc: list[float] = field(default_factory=list) @classmethod def from_dict(cls, data: dict | None) -> "Baseline": if not data: return cls() - return cls(trend=list(data["trend"]), arousal=list(data["arousal"])) + return cls( + trend=list(data["trend"]), + arousal=list(data["arousal"]), + valence=list(data["valence"]), + kc=list(data["kc"]), + ) def to_dict(self) -> dict: - return {"trend": list(self.trend), "arousal": list(self.arousal)} + return { + "trend": list(self.trend), + "arousal": list(self.arousal), + "valence": list(self.valence), + "kc": list(self.kc), + } @property def count(self) -> int: @@ -125,8 +158,10 @@ def count(self) -> int: def push(self, channels: Channels, window: int) -> None: self.trend.append(channels.trend_hz) self.arousal.append(channels.arousal_hz) - del self.trend[:-window] - del self.arousal[:-window] + self.valence.append(channels.valence_hz) + self.kc.append(float(channels.kc_spikes)) + for history in (self.trend, self.arousal, self.valence, self.kc): + del history[:-window] def _z(value: float, history: list[float], center: bool) -> float: @@ -147,8 +182,10 @@ class Posture: shift_bps: float trend_z: float arousal_z: float + valence_z: float gate: bool warm: bool # False while the baseline is still forming + confident: bool = True # False when the scene never reached the mushroom body def to_dict(self) -> dict: return asdict(self) @@ -165,6 +202,7 @@ def from_dict(cls, data: dict) -> "Posture": shift_bps=0.0, trend_z=0.0, arousal_z=0.0, + valence_z=0.0, gate=False, warm=False, ) @@ -197,10 +235,19 @@ def decode(channels: Channels, baseline: Baseline, s: DecoderSettings) -> Postur else: trend_z = channels.trend_hz / RAW_TREND_UNIT_HZ arousal_z = _z(channels.arousal_hz, baseline.arousal, center=True) + valence_z = _z(channels.valence_hz, baseline.valence, center=True) + kc_z = _z(float(channels.kc_spikes), baseline.kc, center=True) gate = channels.gate_spikes >= 1 + # No Kenyon drive at all, or far below what this pair usually produces: the + # chart did not reach the mushroom body, so every z above is measuring the + # network's own noise rather than the picture. + confident = channels.kc_spikes > 0 and kc_z > s.z_kc_quiet regime = classify(trend_z, arousal_z, gate, s) spread_mult = min(s.spread_max, max(s.spread_min, 1 + s.spread_gain * arousal_z)) - size_mult = min(s.size_max, max(s.size_min, 1 + s.size_gain * arousal_z)) + size_mult = min( + s.size_max, + max(s.size_min, 1 + s.size_gain * arousal_z + s.valence_gain * valence_z), + ) shift = max(-s.max_shift_bps, min(s.max_shift_bps, s.shift_gain_bps * trend_z)) if not gate: shift = 0.0 # no descending gate spike, no directional lean @@ -211,8 +258,10 @@ def decode(channels: Channels, baseline: Baseline, s: DecoderSettings) -> Postur shift_bps=round(shift, 3), trend_z=round(trend_z, 4), arousal_z=round(arousal_z, 4), + valence_z=round(valence_z, 4), gate=gate, warm=True, + confident=confident, ) @@ -231,6 +280,11 @@ def should_apply( h: Hysteresis, ) -> tuple[bool, str]: """Rate-limit configuration changes to material posture moves.""" + if not new.confident: + # Holding is the conservative act: the last applied config stays, and + # the reason is recorded rather than a neutral posture being written + # as though the fly had decided on one. + return False, "kenyon drive below baseline — scene not seen" if previous is None: return True, "first posture" if last_apply_ts is not None and now - last_apply_ts < h.min_apply_interval_sec: diff --git a/agents/market_making_fly/flybrain/worker.py b/agents/market_making_fly/flybrain/worker.py index a3439fb9b..461d9020b 100644 --- a/agents/market_making_fly/flybrain/worker.py +++ b/agents/market_making_fly/flybrain/worker.py @@ -48,6 +48,15 @@ def __init__( self.gate = np.flatnonzero(types.eq("DNpe017")) if not len(self.left) or not len(self.right) or not len(self.gate): raise RuntimeError("Missing annotated DNp20 / DNpe017 readout cells") + # The memory rule's own output. KC→MBON07/11 are the synapses dopamine + # moves, so their firing is the only place a P&L pulse can reach a + # decision. Read here rather than through the vendored circuit map, + # which bundles both into one `mb` array, and for the same reason the + # other readouts are read here: this is what the decoder consumes. + self.mbon_approach = np.flatnonzero(types.eq("MBON07")) + self.mbon_avoid = np.flatnonzero(types.eq("MBON11")) + if not len(self.mbon_approach) or not len(self.mbon_avoid): + raise RuntimeError("Missing annotated MBON07 / MBON11 memory outputs") superclass = np.asarray(self.brain.superclass).astype(str) readouts = np.concatenate([self.left, self.right, self.gate]) descending = np.flatnonzero(superclass == DESCENDING_SUPERCLASS) @@ -67,6 +76,8 @@ def __init__( "left": [str(self.brain.ids[i]) for i in self.left], "right": [str(self.brain.ids[i]) for i in self.right], "gate": [str(self.brain.ids[i]) for i in self.gate], + "mbon_approach": [str(self.brain.ids[i]) for i in self.mbon_approach], + "mbon_avoid": [str(self.brain.ids[i]) for i in self.mbon_avoid], "descending_count": int(len(self.descending)), } @@ -105,10 +116,17 @@ def observe( seconds = neural_ms / 1000 left = float(np.mean(counts[self.left]) / seconds) right = float(np.mean(counts[self.right]) / seconds) + approach = float(np.mean(counts[self.mbon_approach]) / seconds) + avoid = float(np.mean(counts[self.mbon_avoid]) / seconds) return { "trend_hz": right - left, "left_hz": left, "right_hz": right, + # What the memory rule has made of this scene: MBON07 approach + # minus MBON11 avoidance. Positive is the learned "go". + "valence_hz": approach - avoid, + "mbon_approach_hz": approach, + "mbon_avoid_hz": avoid, "arousal_hz": float(np.mean(counts[self.descending]) / seconds), "gate_spikes": int(counts[self.gate].sum()), # Over the whole network, not extrapolated from the drawn sample: @@ -148,6 +166,7 @@ def provenance(self) -> dict: "readout": { "trend": "DNp20 mean right minus left rate", "arousal": f"mean rate of superclass={DESCENDING_SUPERCLASS!r} minus readouts", + "valence": "mean MBON07 rate minus mean MBON11 rate", "gate": "DNpe017 spike count >= 1", "cells": self.cell_ids, "validated": False, diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index 1ffc06463..e21385619 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -468,7 +468,11 @@ async def pace() -> None: ) posture = decode( Channels( - neural["trend_hz"], neural["arousal_hz"], neural["gate_spikes"] + neural["trend_hz"], + neural["arousal_hz"], + neural["gate_spikes"], + neural["valence_hz"], + neural["kc_spikes"], ), baselines[pair], decoder_settings, @@ -572,6 +576,7 @@ async def pace() -> None: b.kpi("Regime", posture.regime) b.kpi("Spread ×", f"{posture.spread_mult:.2f}") b.kpi("Size ×", f"{posture.size_mult:.2f}") + b.kpi("Valence z", f"{posture.valence_z:+.2f}") b.kpi("Lean", f"{posture.shift_bps:+.2f} bp") b.kpi("Trend z", f"{posture.trend_z:+.2f}") b.kpi("Arousal z", f"{posture.arousal_z:+.2f}") diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index 0fb98e936..77f0a87f4 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -335,7 +335,8 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: active = neural.get("active_neurons") posture_line = ( f"Decoded from trend z {_fmt(last_posture.get('trend_z'), 2, plus=True)} and " - f"arousal z {_fmt(last_posture.get('arousal_z'), 2, plus=True)}, gated on " + f"arousal z {_fmt(last_posture.get('arousal_z'), 2, plus=True)}, learned " + f"valence z {_fmt(last_posture.get('valence_z'), 2, plus=True)}, gated on " f"{neural.get('gate_spikes', '—')} DNpe017 spike(s)." + ("" if last_posture.get("warm", True) else " Baseline still forming.") ) diff --git a/agents/market_making_fly/routines/fly_status.py b/agents/market_making_fly/routines/fly_status.py index 15c046a81..8e3aba49a 100644 --- a/agents/market_making_fly/routines/fly_status.py +++ b/agents/market_making_fly/routines/fly_status.py @@ -87,6 +87,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: lines.append( f"{pair}: {posture['regime']} spread_x={posture['spread_mult']:.2f} " f"size_x={posture.get('size_mult', 1.0):.2f} " + f"valence_z={posture.get('valence_z', 0.0):+.2f} " f"lean_bp={posture['shift_bps']:+.2f} trend_z={posture['trend_z']:+.2f} " f"arousal_z={posture['arousal_z']:+.2f} baseline_n={len((baselines.get(pair) or {}).get('trend', []))}" ) diff --git a/agents/market_making_fly/skills/fly_decoder/SKILL.md b/agents/market_making_fly/skills/fly_decoder/SKILL.md index ed7e679b8..4cad7a40f 100644 --- a/agents/market_making_fly/skills/fly_decoder/SKILL.md +++ b/agents/market_making_fly/skills/fly_decoder/SKILL.md @@ -23,7 +23,8 @@ user wants to *look* at the run, `fly_status` when you need to quote figures. | `trend_hz` | DNp20 right mean rate − left mean rate | lean direction; stonkfly's BUY/SELL cells | | `arousal_hz` | mean rate of the 1,314 descending neurons (minus the readouts) | spread width **and** how much of the book is quoted | | `gate_spikes` | DNpe017 | ≥ 1 required for a trending call and for any lean | -| `kc_spikes` | Kenyon cells | did the chart reach the mushroom body at all (0 = the fly saw nothing useful) | +| `valence_hz` | mean MBON07 rate − mean MBON11 rate | approach minus avoidance — **the memory rule's own output**, and the only channel a P&L pulse can reach. Drives how much of the book is quoted | +| `kc_spikes` | Kenyon cells | the confidence test: no sparse code of the chart means every other channel is reading noise, so the posture is marked unconfident and the loop holds | | `reward_spikes` / `aversive_spikes` | PAM11 / PPL101 | did the pulse arrive | ## Posture @@ -35,6 +36,8 @@ user wants to *look* at the run, `fly_status` when you need to quote figures. * `spread ×` = clip(1 − 0.5·arousal_z, 0.6, 2.5) — an aroused fly quotes **tighter**. `size ×` = clip(1 + 0.5·arousal_z, 0.6, 2.5) — and quotes **more** of the book, clamped so an order never falls under the venue minimum nor the allocation over 1. + `size ×` also carries `+0.5·valence_z`, so what the fly has learned about scenes + like this one moves the capital it commits. The sign on each is a choice, not a finding: arousal is a population rate against its own average and nothing ties it to volatility. `lean` = clip(trend_z, ±3 bp), 0 without a gate spike, and capped at half the first spread level when mapped. @@ -60,13 +63,16 @@ update failed · `HALT` loop stopped · `TICK_ERROR` data fetch failed, loop con * That the fly detected a regime: it emitted a z-score we labelled. * That a P&L pulse taught it anything: pulses are value feedback, not credit assignment, - and edges change from endogenous dopamine activity too. + and edges change from endogenous dopamine activity too. The valence channel means the + memory rule can now *reach* a decision — it does not mean what it reaches is right. * That a run with positive P&L shows the fly works: a rising market makes any long inventory look skilled. No held-out replay or shuffled-reinforcement control exists. ## Health checks -* `kc_spikes` stays 0 → the chart is not activating the mushroom body; report it. +* `kc_spikes` stays 0 → the chart is not activating the mushroom body. The loop now + holds instead of applying (`HOLD` with "scene not seen"); a run that does this + every tick is quoting a stale config, so report it. * `baseline_n` not growing for a pair → that pair's book is closed or its feed fails. * `halted` set → read the reason; financial halts need a new `run_name`. * `compute_seconds` above ~40 % of the interval → lower `neural_ms`. diff --git a/agents/market_making_fly/tests/test_fly_decoder.py b/agents/market_making_fly/tests/test_fly_decoder.py index f21e1da85..96f983746 100644 --- a/agents/market_making_fly/tests/test_fly_decoder.py +++ b/agents/market_making_fly/tests/test_fly_decoder.py @@ -112,7 +112,7 @@ def test_baseline_window_and_roundtrip(): def test_posture_roundtrip(): - p = Posture("quiet", 0.8, 1.0, -1.0, -0.2, -1.3, True, True) + p = Posture("quiet", 0.8, 1.0, -1.0, -0.2, -1.3, 0.0, True, True) assert Posture.from_dict(p.to_dict()) == p @@ -132,14 +132,50 @@ def test_settings_validation(): def test_hysteresis(): h = Hysteresis(min_apply_interval_sec=300) - base = Posture("ranging", 1.0, 1.0, 0.0, 0, 0, False, True) + base = Posture("ranging", 1.0, 1.0, 0.0, 0, 0, 0.0, False, True) assert should_apply(None, base, None, 1000, h)[0] - same = Posture("ranging", 1.05, 1.0, 0.2, 0, 0, False, True) + same = Posture("ranging", 1.05, 1.0, 0.2, 0, 0, 0.0, False, True) assert not should_apply(base, same, 0, 1000, h)[0] - regime = Posture("volatile", 1.0, 1.0, 0.0, 0, 1.2, False, True) + regime = Posture("volatile", 1.0, 1.0, 0.0, 0, 1.2, 0.0, False, True) assert should_apply(base, regime, 0, 1000, h)[0] assert not should_apply(base, regime, 900, 1000, h)[0] # cooldown - wider = Posture("ranging", 1.2, 1.0, 0.0, 0, 0, False, True) + wider = Posture("ranging", 1.2, 1.0, 0.0, 0, 0, 0.0, False, True) assert should_apply(base, wider, 0, 1000, h)[0] - lean = Posture("ranging", 1.0, 1.0, 0.6, 0, 0, True, True) + lean = Posture("ranging", 1.0, 1.0, 0.6, 0, 0, 0.0, True, True) assert should_apply(base, lean, 0, 1000, h)[0] + + +def test_valence_moves_the_size_the_fly_commits(): + """MBON07 minus MBON11 is what the KC→MBON memory rule writes to, so it is + the only path a P&L pulse has to a decision. Without it the dopamine loop + moved thousands of synapses and changed nothing the fly did.""" + b = Baseline() + for _ in range(S.warmup): + b.push(Channels(0.0, 8.0, 0, 0.0, 400), S.window) + good = decode(Channels(0.0, 8.0, 0, 6.0, 400), b, S) + assert good.valence_z > 1 and good.size_mult > 1 + b2 = Baseline() + for _ in range(S.warmup): + b2.push(Channels(0.0, 8.0, 0, 0.0, 400), S.window) + bad = decode(Channels(0.0, 8.0, 0, -6.0, 400), b2, S) + assert bad.valence_z < -1 and bad.size_mult < 1 + # and it moves size only — the spread is the arousal channel's + assert good.spread_mult == pytest.approx(bad.spread_mult) + + +def test_a_scene_that_never_reached_the_mushroom_body_is_not_acted_on(): + """Kenyon drive is the confidence test: with no sparse code of the chart, + every other channel is reading the network's own noise.""" + b = Baseline() + for _ in range(S.warmup): + b.push(Channels(0.0, 8.0, 0, 0.0, 400), S.window) + seen = decode(Channels(0.0, 8.0, 1, 0.0, 400), b, S) + assert seen.confident + + dark = Baseline() + for _ in range(S.warmup): + dark.push(Channels(0.0, 8.0, 0, 0.0, 400), S.window) + blind = decode(Channels(0.0, 8.0, 1, 0.0, 0), dark, S) + assert not blind.confident + ok, why = should_apply(seen, blind, None, 0.0, Hysteresis()) + assert ok is False and "kenyon" in why.lower() diff --git a/agents/market_making_fly/tests/test_fly_guard.py b/agents/market_making_fly/tests/test_fly_guard.py index afbabea94..526fc7075 100644 --- a/agents/market_making_fly/tests/test_fly_guard.py +++ b/agents/market_making_fly/tests/test_fly_guard.py @@ -165,7 +165,7 @@ def test_a_config_sitting_exactly_on_the_floor_is_not_vetoed(): portfolio_allocation=0.3, maker_fee_bps=1.3, ) - leaned = Posture("trending_up", 1.14, 1.0, 1.27, 2.0, 0.3, True, True) + leaned = Posture("trending_up", 1.14, 1.0, 1.27, 2.0, 0.3, 0.0, True, True) config = build_config(spec, leaned) assert min(float(x) for x in config["buy_spreads"].split(",")) == pytest.approx( spec.min_spread_bps * 1e-4 diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py index 8f9964df8..29e6bb7f8 100644 --- a/agents/market_making_fly/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -53,12 +53,16 @@ def test_take_profit_floor_beats_fees_and_spread(): def test_volatile_widens_quiet_tightens_with_floor(): - wide = build_config(SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, False, True)) + wide = build_config( + SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, 0.0, False, True) + ) assert _spreads(wide["buy_spreads"])[0] == pytest.approx(8 * BPS) assert (wide["executor_refresh_time"], wide["buy_cooldown_time"]) == TIMING[ "volatile" ] - tight = build_config(SPEC, Posture("quiet", 0.6, 1.0, 0.0, 0, -1.5, False, True)) + tight = build_config( + SPEC, Posture("quiet", 0.6, 1.0, 0.0, 0, -1.5, 0.0, False, True) + ) # 4 bp × 0.6 = 2.4 bp, which clears this market's 1.3 bp fee floor assert _spreads(tight["buy_spreads"])[0] == pytest.approx(2.4 * BPS) assert (tight["executor_refresh_time"], tight["buy_cooldown_time"]) == TIMING[ @@ -67,13 +71,15 @@ def test_volatile_widens_quiet_tightens_with_floor(): def test_lean_is_asymmetric_and_capped(): - up = build_config(SPEC, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, True, True)) + up = build_config( + SPEC, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) + ) buy, sell = _spreads(up["buy_spreads"]), _spreads(up["sell_spreads"]) # lean capped at half of level 1 (4 bp → 2 bp), and 2 bp still clears the fee assert buy[0] == pytest.approx(2 * BPS) and sell[0] == pytest.approx(6 * BPS) assert buy[1] == pytest.approx(7 * BPS) and sell[1] == pytest.approx(11 * BPS) down = build_config( - SPEC, Posture("trending_down", 1.0, 1.0, -3.0, -2.0, 0, True, True) + SPEC, Posture("trending_down", 1.0, 1.0, -3.0, -2.0, 0, 0.0, True, True) ) assert _spreads(down["sell_spreads"])[0] == pytest.approx(2 * BPS) assert _spreads(down["buy_spreads"])[0] == pytest.approx(6 * BPS) @@ -93,13 +99,13 @@ def test_the_spread_floor_is_the_market_own_fee(): assert dear.min_spread_bps == pytest.approx(7.5) # binance spot # a lean that would quote inside the fee is pushed back out to it leaned = build_config( - dear, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, True, True) + dear, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) ) assert min(_spreads(leaned["buy_spreads"])) == pytest.approx(7.5 * BPS) def test_pause_sets_kill_switch(): - cfg = build_config(SPEC, Posture("pause", 2.5, 1.0, 0.0, 0, 3.0, False, True)) + cfg = build_config(SPEC, Posture("pause", 2.5, 1.0, 0.0, 0, 3.0, 0.0, False, True)) assert cfg["manual_kill_switch"] is True @@ -108,7 +114,7 @@ def test_every_spread_respects_min(): for mult in (0.6, 1.0, 2.5): for shift in (-3.0, 0.0, 3.0): cfg = build_config( - SPEC, Posture(regime, mult, 1.0, shift, 0, 0, True, True) + SPEC, Posture(regime, mult, 1.0, shift, 0, 0, 0.0, True, True) ) for key in ("buy_spreads", "sell_spreads"): assert min(_spreads(cfg[key])) >= SPEC.min_spread_bps * BPS - 1e-12 @@ -165,7 +171,7 @@ def test_the_fee_floor_follows_the_venue(): def test_config_diff(): a = build_config(SPEC, NEUTRAL) - b = build_config(SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, False, True)) + b = build_config(SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, 0.0, False, True)) diff = config_diff(a, b) assert "buy_spreads" in diff and "trading_pair" not in diff assert config_diff(None, a) == a @@ -213,15 +219,15 @@ def test_size_follows_arousal_and_stays_inside_both_limits(): spec = MarketSpec( **{**SPEC.__dict__, "total_amount_quote": 200, "portfolio_allocation": 0.3} ) - hot = build_config(spec, Posture("ranging", 1.0, 2.5, 0.0, 0, 0, False, True)) - calm = build_config(spec, Posture("ranging", 1.0, 0.6, 0.0, 0, 0, False, True)) + hot = build_config(spec, Posture("ranging", 1.0, 2.5, 0.0, 0, 0, 0.0, False, True)) + calm = build_config(spec, Posture("ranging", 1.0, 0.6, 0.0, 0, 0, 0.0, False, True)) assert hot["portfolio_allocation"] > spec.portfolio_allocation assert calm["portfolio_allocation"] < spec.portfolio_allocation assert calm["portfolio_allocation"] == pytest.approx(0.24) assert calm["portfolio_allocation"] == pytest.approx(spec.min_portfolio_allocation) # and a big book scaled up still cannot quote more than all of itself big = MarketSpec(**{**SPEC.__dict__, "portfolio_allocation": 0.5}) - assert build_config(big, Posture("ranging", 1.0, 2.5, 0.0, 0, 0, False, True))[ + assert build_config(big, Posture("ranging", 1.0, 2.5, 0.0, 0, 0, 0.0, False, True))[ "portfolio_allocation" ] == pytest.approx(1.0) From 3687bc9b18a9d0d880ff8ecfbf00f357736366d4 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 13:48:23 -0700 Subject: [PATCH 35/48] (feat) replay the fly over recorded candles, against its own controls Every claim this agent makes needs a comparison, and every live run is one sample of a market that never repeats. This walks a fixed candle series with the same frame, the same brain, the same decoder and the same geometry, and simulates only the market's answer: whether a resting quote filled. The fill model is optimistic and says so -- a quote the price touched is assumed ours, with no queue position, and it is most generous exactly when the market is moving. Replay P&L is an upper bound. What makes it worth running is that the bias is identical across variants, which is what a comparison needs. Five variants, each on its own freshly seeded brain in its own process, because a network that has already learned from one variant is not a control for the next: live, no-memory (plasticity frozen), no-valence (the rule runs, its output disconnected), shuffled (same pulse frequency and magnitude, sign randomised) and widen (the old arousal direction). They run concurrently -- independent work, 11 cores, ~230 MB a brain -- so five variants cost one variant's wall time. The statistic is on increments, not on the equity curves. A curve is cumulative: once two runs separate, every later tick inherits the gap, so a t on levels reports when they diverged rather than whether they earn differently. Written the wrong way first, it returned |t| of 27 to 63 for variants a few percent apart -- a measure of autocorrelation, nothing else. Degenerate variance is guarded too: the increments of a perfect ramp differ in their last bits, which took the naive expression to 4e15. Curves are persisted beside the report, so a better statistic never costs another hour of brains. --- agents/market_making_fly/flybrain/replay.py | 341 +++++++++++++++ .../market_making_fly/routines/fly_replay.py | 394 ++++++++++++++++++ .../tests/test_fly_replay.py | 111 +++++ 3 files changed, 846 insertions(+) create mode 100644 agents/market_making_fly/flybrain/replay.py create mode 100644 agents/market_making_fly/routines/fly_replay.py create mode 100644 agents/market_making_fly/tests/test_fly_replay.py diff --git a/agents/market_making_fly/flybrain/replay.py b/agents/market_making_fly/flybrain/replay.py new file mode 100644 index 000000000..a8bb9901f --- /dev/null +++ b/agents/market_making_fly/flybrain/replay.py @@ -0,0 +1,341 @@ +"""Replay the fly over recorded candles, with a maker fill model for its P&L. + +Every experiment this agent needs is a comparison: does the memory rule change +anything, does real reinforcement beat shuffled, does a gain matter. Answering +any of them live costs a bot, an hour and real capital, and produces one sample +of a market that never repeats. So the same loop runs here over a fixed candle +series instead, where the only thing that differs between two runs is the +setting under test. + +What is identical to the live loop: the frame the retina sees (``market_frame`` +over the same sliding window), the brain (the same worker, the same seeded +network), the decoder, the posture geometry and the fee. What is simulated is +only the market's answer — whether a resting quote filled — and that is the +part to be honest about: + +* a quote at price ``p`` fills if the next candle trades through it. There is + no queue position, so every fill a price *touches* is assumed ours. Real + fills are a fraction of that, and the fraction is worst exactly when the + market is moving, which is when this model is most generous. +* a fill's take-profit closes when a later candle trades through it. The same + optimism applies, and inventory that never reaches its take-profit is marked + to the close, exactly as the live loop's unrealized P&L would be. +* adverse selection appears only through price: if the market runs, the model + fills the losing side and holds the position, which is the real failure mode + it does reproduce. + +So replay P&L is an upper bound, not a forecast. It is useful because the bias +is the *same* for every variant, which is what a comparison needs. +""" + +from __future__ import annotations + +import math +import random +from dataclasses import dataclass, field + +import numpy as np +from flybrain.chart import market_frame +from flybrain.decoder import Baseline, Channels, DecoderSettings, decode, should_apply +from flybrain.decoder import Hysteresis, Posture +from flybrain.posture import BPS, MarketSpec, build_config, take_profit_floor +from flybrain.reinforcement import reinforcement + +# A held position is marked to the candle close, like the live loop's +# unrealized P&L; a closed one has both fees taken out of it. +SIDES = ("buy", "sell") + + +@dataclass +class Lot: + """One filled quote, still open.""" + + side: str + price: float + amount: float # base units + take_profit: float # fraction from entry + + +@dataclass +class Ledger: + """Fills, closes and P&L for one replay, in quote units.""" + + realized: float = 0.0 + fees: float = 0.0 + fills: int = 0 + round_trips: int = 0 + volume: float = 0.0 + open_lots: list[Lot] = field(default_factory=list) + + @property + def inventory(self) -> float: + return sum(l.amount if l.side == "buy" else -l.amount for l in self.open_lots) + + def equity(self, mark: float) -> float: + """Realized less fees, plus the open inventory marked to ``mark``.""" + unrealized = sum( + ( + (mark - l.price) * l.amount + if l.side == "buy" + else (l.price - mark) * l.amount + ) + for l in self.open_lots + ) + return self.realized - self.fees + unrealized + + +def _levels(config: dict, key: str) -> list[float]: + return [float(x) for x in config[key].split(",")] + + +def quote_prices(config: dict, mid: float) -> dict[str, list[float]]: + """Where this config rests its orders around ``mid``.""" + return { + "buy": [mid * (1 - s) for s in _levels(config, "buy_spreads")], + "sell": [mid * (1 + s) for s in _levels(config, "sell_spreads")], + } + + +def step( + ledger: Ledger, + config: dict, + spec: MarketSpec, + candle: dict, + mid: float, + max_lots: int, +) -> None: + """One candle against one config: close what reaches take-profit, then fill. + + Closes are settled before new fills so a lot cannot open and close inside + the same candle, which the data cannot support: a candle says a price was + touched, not in what order. + """ + high, low = float(candle["high"]), float(candle["low"]) + fee = spec.maker_fee_bps * BPS + + still_open: list[Lot] = [] + for lot in ledger.open_lots: + target = ( + lot.price * (1 + lot.take_profit) + if lot.side == "buy" + else lot.price * (1 - lot.take_profit) + ) + reached = high >= target if lot.side == "buy" else low <= target + if not reached: + still_open.append(lot) + continue + gross = ( + (target - lot.price) * lot.amount + if lot.side == "buy" + else (lot.price - target) * lot.amount + ) + ledger.realized += gross + ledger.fees += target * lot.amount * fee # the closing side + ledger.volume += target * lot.amount + ledger.round_trips += 1 + ledger.open_lots = still_open + + notional = float(config["total_amount_quote"]) * float( + config["portfolio_allocation"] + ) + per_order = notional / 4 + prices = quote_prices(config, mid) + for side in SIDES: + for price in prices[side]: + if len(ledger.open_lots) >= max_lots: + break + touched = low <= price if side == "buy" else high >= price + if not touched or price <= 0: + continue + amount = per_order / price + ledger.open_lots.append( + Lot( + side=side, + price=price, + amount=amount, + take_profit=float(config["take_profit"]), + ) + ) + ledger.fills += 1 + ledger.fees += per_order * fee + ledger.volume += per_order + + +@dataclass +class ReplayResult: + variant: str + ticks: int + applies: int + holds: int + unconfident: int + ledger: Ledger + equity_curve: list[float] + postures: list[Posture] + + def summary(self) -> dict: + regimes: dict[str, int] = {} + for p in self.postures: + regimes[p.regime] = regimes.get(p.regime, 0) + 1 + spreads = [p.spread_mult for p in self.postures] or [0.0] + sizes = [p.size_mult for p in self.postures] or [0.0] + return { + "variant": self.variant, + "ticks": self.ticks, + "net": round(self.equity_curve[-1] if self.equity_curve else 0.0, 4), + "realized": round(self.ledger.realized - self.ledger.fees, 4), + "fees": round(self.ledger.fees, 4), + "fills": self.ledger.fills, + "round_trips": self.ledger.round_trips, + "volume": round(self.ledger.volume, 2), + "open_lots": len(self.ledger.open_lots), + "applies": self.applies, + "unconfident": self.unconfident, + "mean_spread_mult": round(sum(spreads) / len(spreads), 3), + "mean_size_mult": round(sum(sizes) / len(sizes), 3), + "regimes": regimes, + } + + +def frames( + pair: str, candles: list[dict], window: int +) -> list[tuple[np.ndarray, dict, float]]: + """``(frame, candle, mid)`` per tick, over a sliding window of candles. + + The bid/ask drawn on the frame are the previous close plus and minus half + the observed spread of that bar — the live loop draws the live book, and a + candle series has no book. The tick's *decisions* are then applied to the + following candle, so nothing is decided on a bar it can already see. + """ + out = [] + for i in range(window, len(candles) - 1): + history = candles[i - window : i] + last = float(history[-1]["close"]) + half = max( + last * 1e-5, (float(history[-1]["high"]) - float(history[-1]["low"])) / 200 + ) + frame = market_frame(pair, history, last - half, last + half, n_candles=window) + out.append((frame, candles[i], last)) + return out + + +def replay( + variant: str, + pair: str, + candles: list[dict], + spec: MarketSpec, + settings: DecoderSettings, + observe, + window: int = 72, + deadband_bps: float = 1.0, + shuffle_seed: int | None = None, + neural_ms: float = 500.0, + max_lots: int = 8, +) -> ReplayResult: + """Walk the candles once, exactly as the live loop walks wall time. + + ``observe(frame, stimulus, neural_ms) -> dict`` is the brain, injected so a + caller can hand in a fresh worker per variant — a brain that has already + learned from one variant is not a control for the next. + + ``shuffle_seed`` replaces each pulse with a randomly signed one of the same + magnitude. The fly still gets reinforced exactly as often and as hard; only + the correspondence to its own P&L is destroyed. If that scores the same, + what the synapses hold is not about this market. + """ + baseline = Baseline() + ledger = Ledger() + hysteresis = Hysteresis(min_apply_interval_sec=0.0) + rng = random.Random(shuffle_seed) if shuffle_seed is not None else None + deadband = deadband_bps * BPS * spec.total_amount_quote + + config = build_config(spec, decode(Channels(0.0, 0.0, 0), Baseline(), settings)) + previous: Posture | None = None + anchor = 0.0 + applies = holds = unconfident = 0 + curve: list[float] = [] + postures: list[Posture] = [] + + for tick, (frame, candle, mid) in enumerate(frames(pair, candles, window)): + equity = ledger.equity(mid) + kind, _ = reinforcement(equity, anchor, deadband) + if rng is not None and kind != "none": + kind = rng.choice(["reward", "aversive"]) + anchor = equity + + neural = observe(frame, kind, neural_ms) + posture = decode( + Channels( + neural["trend_hz"], + neural["arousal_hz"], + neural["gate_spikes"], + neural["valence_hz"], + neural["kc_spikes"], + ), + baseline, + settings, + ) + postures.append(posture) + if not posture.confident: + unconfident += 1 + ok, _ = should_apply(previous, posture, None, float(tick), hysteresis) + if ok: + config = build_config(spec, posture) + previous = posture + applies += 1 + else: + holds += 1 + + step(ledger, config, spec, candle, mid, max_lots) + curve.append(ledger.equity(float(candle["close"]))) + + return ReplayResult( + variant=variant, + ticks=len(curve), + applies=applies, + holds=holds, + unconfident=unconfident, + ledger=ledger, + equity_curve=curve, + postures=postures, + ) + + +def paired_stats(a: list[float], b: list[float]) -> dict: + """Do two runs earn differently per tick? Mean difference, sd, and t. + + On the *increments*, not the levels. An equity curve is cumulative: once + two runs separate, every later tick inherits that gap, so a t-test on + levels measures how long ago they diverged rather than whether they earn + differently. Run on levels this returned |t| of 27 to 63 for variants whose + final P&L differed by a few percent — a number that says nothing except + that a random walk is autocorrelated. + + Even on increments this is a weak instrument, and it is reported as one: + one replay of one market, with fills that assume a touched price was ours. + A |t| under 2 is not evidence of a difference; it is also not evidence of + sameness. + """ + n = min(len(a), len(b)) + if n < 4: + return {"n": 0, "mean_diff": 0.0, "sd": 0.0, "t": 0.0, "final_gap": 0.0} + gains_a = [a[i] - a[i - 1] for i in range(1, n)] + gains_b = [b[i] - b[i - 1] for i in range(1, n)] + deltas = [x - y for x, y in zip(gains_a, gains_b)] + k = len(deltas) + mean = sum(deltas) / k + var = sum((d - mean) ** 2 for d in deltas) / (k - 1) + sd = math.sqrt(var) + # A difference with no spread has no sampling variation, so t is undefined + # rather than enormous — and floating point will not produce a clean zero: + # the increments of a perfect ramp differ in the last bits, which took the + # naive expression to 4e15. Below a hair of the scale it measures, the + # answer is "read the mean, there is nothing here to test". + scale = max(abs(mean), max((abs(d) for d in deltas), default=0.0)) + degenerate = sd <= scale * 1e-9 + return { + "n": k, + "mean_diff": mean, + "sd": sd, + "t": 0.0 if degenerate else mean / (sd / math.sqrt(k)), + "final_gap": a[n - 1] - b[n - 1], + } diff --git a/agents/market_making_fly/routines/fly_replay.py b/agents/market_making_fly/routines/fly_replay.py new file mode 100644 index 000000000..1fb8bdc60 --- /dev/null +++ b/agents/market_making_fly/routines/fly_replay.py @@ -0,0 +1,394 @@ +"""Run the fly over recorded candles, several ways, and compare them. + +The agent's central claim — that P&L feedback shapes what the fly does — has +never had a control, because every live run is one sample of a market that +never repeats. This runs the same candles through the same brain as many times +as there are variants, changing one setting each time: + +* ``live`` — the fly as deployed. +* ``no-memory``— the memory rule frozen. If this scores the same, the plastic + synapses are decoration. +* ``no-valence``— the memory rule still runs, but its output is disconnected + from the posture (``valence_gain`` ~ 0). Separates "the rule + does nothing" from "the rule does something the decoder does + not read". +* ``shuffled`` — reinforcement of the same frequency and magnitude, with the + sign randomised. The control the caveats have always demanded. +* ``widen`` — the old arousal direction, for the A/B that motivated the flip. + +Each variant gets its own brain process: a network that has already learned +from one variant is not a control for the next. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +_AGENT_DIR = str(Path(__file__).resolve().parents[1]) +if _AGENT_DIR not in sys.path: + sys.path.insert(0, _AGENT_DIR) + +import asyncio +import json +import logging +import multiprocessing +from concurrent.futures import ProcessPoolExecutor + +import plotly.graph_objects as go +from flybrain import venue, worker +from flybrain.decoder import DecoderSettings +from flybrain.fly3d import ACCENT, BODY, GROUND, LIMB +from flybrain.posture import MarketSpec +from flybrain.replay import paired_stats, replay +from pydantic import BaseModel, Field +from telegram.ext import ContextTypes + +from condor.memory.paths import agent_home +from condor.paths import safe_id +from condor.reports import ReportBuilder + +logger = logging.getLogger(__name__) + +CATEGORY = "Bot Analysis" +AGENT_SLUG = "market_making_fly" + +# A gain cannot be zero — the decoder refuses a channel it would read and +# discard — so "disconnected" is the smallest gain that rounds out of every +# posture the size multiplier can express. +OFF = 1e-9 + +VARIANTS: dict[str, dict] = { + "live": {}, + "no-memory": {"learning": False}, + "no-valence": {"valence_gain": OFF}, + "shuffled": {"shuffle": True}, + "widen": {"spread_gain": 0.5}, +} + + +class Config(BaseModel): + """Replay the fly over historical candles and compare its variants.""" + + trading_pair: str = Field(default="XYZ:DRAM-USD", description="Market to replay") + connector_name: str = Field(default="hyperliquid_perpetual") + interval: str = Field(default="5m", description="Candle interval, as the fly sees") + max_records: int = Field( + default=1000, ge=200, le=5000, description="Candles to fetch" + ) + variants: str = Field( + default="live,no-memory,no-valence,shuffled,widen", + description=f"Comma-separated, from: {', '.join(VARIANTS)}", + ) + total_amount_quote: float = Field(default=200.0) + portfolio_allocation: float = Field(default=0.3) + picked_spread_bps: float = Field( + default=0.0, description="0 measures it from the candles" + ) + maker_fee_bps: float = Field(default=0.0, description="0 fetches the venue's") + leverage: int = Field(default=1) + n_candles: int = Field(default=72, description="Window the retina sees") + neural_ms: float = Field(default=500.0) + baseline_window: int = Field(default=60) + baseline_warmup: int = Field(default=10) + seed: int = Field(default=7301, description="Seed for the shuffled control") + concurrency: int = Field( + default=4, + ge=1, + le=8, + description="Variants to replay at once. Each holds its own brain (~250 MB) " + "and saturates one core; they are independent, so this is wall time " + "divided rather than work shared", + ) + + +def _settings(config: Config, overrides: dict) -> DecoderSettings: + fields = { + "window": config.baseline_window, + "warmup": config.baseline_warmup, + } + fields.update( + {k: v for k, v in overrides.items() if k not in ("learning", "shuffle")} + ) + return DecoderSettings(**fields) + + +def _curve_figure(results: list) -> go.Figure: + fig = go.Figure() + palette = [ACCENT, "#7fb2ff", "#f2a35c", "#c58cff", "#6fd3c0"] + for n, result in enumerate(results): + fig.add_trace( + go.Scatter( + x=list(range(len(result.equity_curve))), + y=result.equity_curve, + mode="lines", + name=result.variant, + line=dict(color=palette[n % len(palette)], width=2), + ) + ) + fig.update_layout( + height=366, + margin=dict(l=56, r=16, t=10, b=40), + paper_bgcolor=GROUND, + plot_bgcolor=GROUND, + font=dict(color=BODY, family="monospace", size=11), + xaxis=dict(title="tick", gridcolor="#18202e", zerolinecolor="#18202e"), + yaxis=dict( + title="net P&L (quote)", gridcolor="#18202e", zerolinecolor="#243044" + ), + legend=dict(orientation="h", yanchor="top", y=-0.18, xanchor="center", x=0.5), + ) + fig.add_hline(y=0, line=dict(color=LIMB, width=1, dash="dot")) + return fig + + +async def _candles(client, config: Config) -> list[dict]: + """The series every variant replays. One fetch, so they cannot differ.""" + for attempt in (1, 2): # a cold feed answers on the second ask + try: + raw = await client.market_data.get_candles( + config.connector_name, + config.trading_pair, + interval=config.interval, + max_records=config.max_records, + ) + rows = raw if isinstance(raw, list) else raw.get("data", raw.get("candles")) + if rows: + return list(rows) + except Exception: + if attempt == 2: + raise + raise RuntimeError(f"No candles for {config.trading_pair}") + + +def _spread_from_candles(candles: list[dict]) -> float: + """A stand-in for the scanner's measured touch, in bp: the median bar's + range is the only width a candle series knows about. Replay cannot see a + book, and saying so is better than defaulting to a number.""" + import statistics + + ranges = [ + (float(c["high"]) - float(c["low"])) / float(c["close"]) * 1e4 + for c in candles + if float(c.get("close") or 0) > 0 + ] + return round(statistics.median(ranges) / 4, 2) if ranges else 2.0 + + +async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + from config_manager import get_client + + names = [v.strip() for v in config.variants.split(",") if v.strip()] + unknown = [v for v in names if v not in VARIANTS] + if unknown: + raise ValueError(f"Unknown variant(s) {unknown}; choose from {list(VARIANTS)}") + + client = await get_client(context._chat_id, context=context) + if not client: + return "No server available" + candles = await _candles(client, config) + market_type = venue.market_type_for(config.connector_name) + fee = config.maker_fee_bps or await venue.maker_fee_bps( + config.connector_name, market_type, config.trading_pair + ) + spread = config.picked_spread_bps or _spread_from_candles(candles) + spec = MarketSpec( + connector_name=config.connector_name, + trading_pair=config.trading_pair, + total_amount_quote=config.total_amount_quote, + picked_spread_bps=spread, + leverage=config.leverage, + portfolio_allocation=config.portfolio_allocation, + maker_fee_bps=fee, + ) + spec.check_order_size() + + loop = asyncio.get_running_loop() + gate = asyncio.Semaphore(config.concurrency) + + async def one(name: str): + overrides = VARIANTS[name] + async with gate: + # A fresh process per variant: the brain is stateful, and one that + # has already learned is not a control for the next. + pool = ProcessPoolExecutor( + max_workers=1, + mp_context=multiprocessing.get_context("spawn"), + initializer=worker._init, + initargs=(overrides.get("learning", True), 10.0, 200.0, 20.0), + ) + try: + + def observe(frame, stimulus, neural_ms, _pool=pool): + return _pool.submit( + worker._observe, frame, stimulus, neural_ms + ).result() + + result = await loop.run_in_executor( + None, + lambda o=overrides, ob=observe: replay( + variant=name, + pair=config.trading_pair, + candles=candles, + spec=spec, + settings=_settings(config, o), + observe=ob, + window=config.n_candles, + shuffle_seed=config.seed if o.get("shuffle") else None, + neural_ms=config.neural_ms, + ), + ) + finally: + pool.shutdown(wait=True) + await context.bot.send_message( + chat_id=context._chat_id, + text=f"🪰 replay {name}: net {result.equity_curve[-1]:+.4f} " + f"over {result.ticks} ticks, {result.ledger.fills} fills", + ) + return result + + # Order is the caller's, not the order they finished in: the first variant + # is the baseline every control is compared against. + results = list(await asyncio.gather(*(one(name) for name in names))) + summaries = [r.summary() for r in results] + # The brains are the expensive part and the arithmetic over their output is + # not; keeping the curves means a better statistic never costs another run. + record = agent_home(AGENT_SLUG) / "replay" / f"{safe_id(config.trading_pair)}.json" + record.parent.mkdir(parents=True, exist_ok=True) + record.write_text( + json.dumps( + { + "pair": config.trading_pair, + "interval": config.interval, + "candles": len(candles), + "spread_bps": spread, + "fee_bps": fee, + "summaries": summaries, + "curves": {r.variant: r.equity_curve for r in results}, + }, + indent=1, + ) + ) + builder = ReportBuilder(f"Fly replay — {config.trading_pair}") + builder.source("routine", "fly_replay") + builder.tags(["fly", "replay", "control", config.trading_pair]) + builder.manual_order() + builder.section( + "WHAT WAS REPLAYED", + f"{len(candles):,} {config.interval} candles of {config.trading_pair}, " + f"{results[0].ticks if results else 0} ticks after the " + f"{config.n_candles}-candle window. Every variant saw the same series and " + f"the same book geometry — {spread:.2f} bp measured spread, {fee:.2f} bp " + "maker fee — and each ran on its own freshly seeded brain. Fills assume " + "a quote the price touched was ours, so every P&L here is an upper " + "bound; the bias is identical across variants, which is what makes the " + "comparison worth reading and the absolute number not.", + ) + builder.kpi("Market", config.trading_pair) + builder.kpi("Candles", f"{len(candles):,}") + builder.kpi("Ticks", f"{results[0].ticks if results else 0:,}") + builder.kpi("Maker fee", f"{fee:.2f} bp") + + builder.section("VARIANTS", "One row per run, over identical candles") + builder.table( + [ + { + "Variant": s["variant"], + "Net P&L": f"{s['net']:+.4f}", + "Realized": f"{s['realized']:+.4f}", + "Fees": f"{s['fees']:.4f}", + "Fills": f"{s['fills']:,}", + "Round trips": f"{s['round_trips']:,}", + "Applies": f"{s['applies']:,}", + "Unconfident": f"{s['unconfident']:,}", + "Spread ×": f"{s['mean_spread_mult']:.2f}", + "Size ×": f"{s['mean_size_mult']:.2f}", + } + for s in summaries + ], + [ + "Variant", + "Net P&L", + "Realized", + "Fees", + "Fills", + "Round trips", + "Applies", + "Unconfident", + "Spread ×", + "Size ×", + ], + ) + builder.plotly(_curve_figure(results)) + + if len(results) > 1: + base = results[0] + rows = [] + for other in results[1:]: + stats = paired_stats(base.equity_curve, other.equity_curve) + rows.append( + { + "Against": f"{base.variant} − {other.variant}", + "Mean per-tick earnings difference": f"{stats['mean_diff']:+.5f}", + "SD": f"{stats['sd']:.5f}", + "Final gap": f"{stats['final_gap']:+.4f}", + "t": f"{stats['t']:+.2f}", + "Reads as": ( + "distinguishable" + if abs(stats["t"]) >= 2 + else "not distinguishable" + ), + } + ) + builder.section( + "AGAINST THE CONTROLS", + "How differently the deployed fly earns per tick, against each " + "control. On increments, not on the equity curves themselves: a " + "curve is cumulative, so once two runs separate every later tick " + "inherits the gap and a t on levels measures when they diverged " + "rather than whether they earn differently. One replay of one " + "market is still a weak instrument — |t| under 2 is not evidence of " + "a difference, and it is not evidence of sameness either.", + ) + builder.table( + rows, + [ + "Against", + "Mean per-tick earnings difference", + "SD", + "Final gap", + "t", + "Reads as", + ], + ) + + builder.markdown( + "_A variant that scores better here has not been shown to make money: " + "the fill model is optimistic, one market is one sample, and the " + "take-profit that never fills is marked to the close rather than to " + "what it would cost to get out. What replay can establish is the " + "negative — that a variant is **not** distinguishable from its control, " + "which is the claim this agent has never been able to test._" + ) + await builder.save() + + lines = [ + f"pair: {config.trading_pair} ({config.interval})", + f"candles: {len(candles)}, ticks: {results[0].ticks if results else 0}", + f"spread: {spread:.2f} bp, fee: {fee:.2f} bp", + ] + for s in summaries: + lines.append( + f"{s['variant']}: net {s['net']:+.4f}, {s['fills']} fills, " + f"{s['round_trips']} round trips, {s['applies']} applies, " + f"spread ×{s['mean_spread_mult']:.2f}, size ×{s['mean_size_mult']:.2f}" + ) + if len(results) > 1: + for other in results[1:]: + stats = paired_stats(results[0].equity_curve, other.equity_curve) + lines.append( + f"{results[0].variant} vs {other.variant}: mean per-tick earnings " + f"{stats['mean_diff']:+.5f}, final gap {stats['final_gap']:+.4f}, " + f"t {stats['t']:+.2f}" + ) + return "\n".join(lines) diff --git a/agents/market_making_fly/tests/test_fly_replay.py b/agents/market_making_fly/tests/test_fly_replay.py new file mode 100644 index 000000000..81fdc1ef9 --- /dev/null +++ b/agents/market_making_fly/tests/test_fly_replay.py @@ -0,0 +1,111 @@ +"""The replay fill model: what it counts as a fill, a close, and a P&L.""" + +import pytest +from flybrain.decoder import NEUTRAL, DecoderSettings +from flybrain.posture import MarketSpec, build_config +from flybrain.replay import Ledger, Lot, frames, paired_stats, quote_prices, step + +SPEC = MarketSpec( + connector_name="hyperliquid_perpetual", + trading_pair="XYZ:DRAM-USD", + total_amount_quote=200, + picked_spread_bps=2.0, + leverage=1, + portfolio_allocation=0.3, + maker_fee_bps=1.3, +) +CONFIG = build_config(SPEC, NEUTRAL) + + +def _candle(high, low, close=None): + return {"high": high, "low": low, "close": close if close else (high + low) / 2} + + +def test_a_quote_the_price_never_reached_does_not_fill(): + ledger = Ledger() + mid = 100.0 + # a candle that never moves cannot touch a quote resting away from mid + step(ledger, CONFIG, SPEC, _candle(100.0, 100.0), mid, max_lots=8) + assert ledger.fills == 0 and ledger.open_lots == [] + + +def test_a_quote_the_price_traded_through_fills_and_pays_its_fee(): + ledger = Ledger() + mid = 100.0 + prices = quote_prices(CONFIG, mid) + lowest_buy = min(prices["buy"]) + step(ledger, CONFIG, SPEC, _candle(100.0, lowest_buy * 0.999), mid, max_lots=8) + assert ledger.fills == len(prices["buy"]) # both buy levels, no sell + assert all(l.side == "buy" for l in ledger.open_lots) + assert ledger.fees == pytest.approx( + ledger.fills * (200 * 0.3 / 4) * SPEC.maker_fee_bps * 1e-4 + ) + + +def test_a_lot_closes_when_its_take_profit_is_traded_through(): + ledger = Ledger() + tp = float(CONFIG["take_profit"]) + ledger.open_lots = [Lot(side="buy", price=100.0, amount=1.0, take_profit=tp)] + # max_lots=0 admits no new fill, so only the close is under test — a buy + # fills whenever the low is under its price, which no choice of mid avoids + step(ledger, CONFIG, SPEC, _candle(100.0 * (1 + tp) * 1.001, 99.999), 100.0, 0) + assert ledger.round_trips == 1 + assert ledger.realized == pytest.approx(100.0 * tp, rel=1e-3) + assert not ledger.open_lots + + +def test_equity_marks_open_inventory_and_nets_fees(): + ledger = Ledger(realized=1.0, fees=0.25) + ledger.open_lots = [Lot(side="buy", price=100.0, amount=2.0, take_profit=0.001)] + assert ledger.equity(101.0) == pytest.approx(1.0 - 0.25 + 2.0) + assert ledger.equity(99.0) == pytest.approx(1.0 - 0.25 - 2.0) + assert ledger.inventory == pytest.approx(2.0) + + +def test_a_tick_never_decides_on_a_candle_it_can_see(): + """The frame is built from candles strictly before the one it trades.""" + candles = [ + {"open": 1, "high": 1 + i, "low": 1, "close": 1 + i, "volume": 1} + for i in range(1, 12) + ] + out = frames("XYZ:DRAM-USD", candles, window=5) + assert len(out) == len(candles) - 5 - 1 + _, traded, mid = out[0] + assert traded is candles[5] + assert mid == pytest.approx(float(candles[4]["close"])) + + +def test_paired_stats_compares_earnings_not_the_gap_it_already_has(): + """A cumulative curve inherits every past difference, so a t on levels + reports how long ago two runs diverged. Run that way on the 2026-09-13 + replay it returned |t| of 27 to 63 for variants a few percent apart.""" + flat = [0.0] * 50 + assert paired_stats(flat, flat)["t"] == 0.0 + + # one run earns a steady amount more every tick: a real, detectable edge + steady = paired_stats([i * 0.01 for i in range(50)], flat) + assert steady["mean_diff"] == pytest.approx(0.01) + assert steady["t"] == 0.0 # perfectly steady: no variance to test against + + # a run that jumped once and then tracked its control earns the same after + # the jump, and must not read as an edge however wide the gap stays + jumped = [0.0] * 10 + [5.0] * 40 + stats = paired_stats(jumped, flat) + assert stats["final_gap"] == pytest.approx(5.0) + assert abs(stats["t"]) < 2, "a single old jump is not a per-tick edge" + + +def test_the_disconnected_gain_is_a_gain_the_decoder_accepts(): + """`valence_gain: 0` is refused by design — a channel read and discarded — + so the control uses the smallest gain that cannot move a posture.""" + import importlib.util + from pathlib import Path + + path = Path(__file__).resolve().parents[1] / "routines" / "fly_replay.py" + spec = importlib.util.spec_from_file_location("fly_replay", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + with pytest.raises(ValueError): + DecoderSettings(valence_gain=0.0) + settings = DecoderSettings(valence_gain=mod.OFF) + assert round(1 + settings.valence_gain * 3.0, 4) == 1.0 From 4fe3c06a3cc7d4da6e41fc7977e4dd9e57cbf812 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 14:01:56 -0700 Subject: [PATCH 36/48] (fix) resolve the replay output path before spending an hour on brains safe_id refuses the colon in a HIP-3 pair, so writing the record threw -- after all five brains had finished. The persistence that exists to stop a re-run costing another eleven minutes discarded eleven minutes of curves. The path is the pair slug, which is what naming.py is for, and it resolves before the replay loop rather than after it. An output that cannot be written should fail in the first second. --- agents/market_making_fly/flybrain/replay.py | 11 +++++++++-- agents/market_making_fly/routines/fly_replay.py | 10 +++++++--- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/agents/market_making_fly/flybrain/replay.py b/agents/market_making_fly/flybrain/replay.py index a8bb9901f..de323faf6 100644 --- a/agents/market_making_fly/flybrain/replay.py +++ b/agents/market_making_fly/flybrain/replay.py @@ -36,8 +36,15 @@ import numpy as np from flybrain.chart import market_frame -from flybrain.decoder import Baseline, Channels, DecoderSettings, decode, should_apply -from flybrain.decoder import Hysteresis, Posture +from flybrain.decoder import ( + Baseline, + Channels, + DecoderSettings, + Hysteresis, + Posture, + decode, + should_apply, +) from flybrain.posture import BPS, MarketSpec, build_config, take_profit_floor from flybrain.reinforcement import reinforcement diff --git a/agents/market_making_fly/routines/fly_replay.py b/agents/market_making_fly/routines/fly_replay.py index 1fb8bdc60..6123c7e5d 100644 --- a/agents/market_making_fly/routines/fly_replay.py +++ b/agents/market_making_fly/routines/fly_replay.py @@ -39,13 +39,13 @@ from flybrain import venue, worker from flybrain.decoder import DecoderSettings from flybrain.fly3d import ACCENT, BODY, GROUND, LIMB +from flybrain.naming import pair_names from flybrain.posture import MarketSpec from flybrain.replay import paired_stats, replay from pydantic import BaseModel, Field from telegram.ext import ContextTypes from condor.memory.paths import agent_home -from condor.paths import safe_id from condor.reports import ReportBuilder logger = logging.getLogger(__name__) @@ -202,6 +202,12 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: maker_fee_bps=fee, ) spec.check_order_size() + record = ( + agent_home(AGENT_SLUG) + / "replay" + / f"{pair_names(config.trading_pair).slug}.json" + ) + record.parent.mkdir(parents=True, exist_ok=True) loop = asyncio.get_running_loop() gate = asyncio.Semaphore(config.concurrency) @@ -253,8 +259,6 @@ def observe(frame, stimulus, neural_ms, _pool=pool): summaries = [r.summary() for r in results] # The brains are the expensive part and the arithmetic over their output is # not; keeping the curves means a better statistic never costs another run. - record = agent_home(AGENT_SLUG) / "replay" / f"{safe_id(config.trading_pair)}.json" - record.parent.mkdir(parents=True, exist_ok=True) record.write_text( json.dumps( { From bad8c63ea2c053c866ab1c50288bcc64dd7355b6 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 14:07:46 -0700 Subject: [PATCH 37/48] (feat) quotes are placed against how far the market travels Level 1 came from the observed touch -- half of 1.75 bp on DRAM, so 2 bp -- while a typical 5-minute bar there ranges 9.9 bp and so reaches about 5 bp either side of mid. Every level the fly could express, from 0.6x to 2.5x, sat inside what a normal bar covers, so the same candles touched the same orders whatever the posture decided. The replay showed it exactly: five variants, 23 fills and 15 round trips apiece. The base is now half the market's median candle range, with the second rung a quarter-bar beyond it. On DRAM that is 4.95 and 7.43 bp, and the multiplier spans 2.97 to 12.38 -- a typical bar reaches 4.95, so tightening genuinely fills more and widening genuinely fills less. The lever bites without a new channel. picked_spread_bps becomes range_bps throughout, and the scanner recommends the median range it already measures for reach, because a spread nothing reads is worse than no field at all. --- agents/market_making_fly/AGENT.md | 2 +- agents/market_making_fly/flybrain/posture.py | 37 ++++++++---- .../market_making_fly/routines/fly_brain.py | 20 ++++--- .../market_making_fly/routines/fly_replay.py | 25 ++++---- .../routines/mm_market_scanner.py | 19 +++--- .../skills/fly_decoder/SKILL.md | 3 +- .../skills/fly_mm_deploy/SKILL.md | 34 ++++++----- .../market_making_fly/tests/test_fly_guard.py | 2 +- .../tests/test_fly_posture.py | 59 ++++++++++--------- .../tests/test_fly_replay.py | 2 +- 10 files changed, 114 insertions(+), 89 deletions(-) diff --git a/agents/market_making_fly/AGENT.md b/agents/market_making_fly/AGENT.md index 440f0531b..824d57b47 100644 --- a/agents/market_making_fly/AGENT.md +++ b/agents/market_making_fly/AGENT.md @@ -89,7 +89,7 @@ manage_skill(action="read", name="fly_mm_deploy") | `fly_setup` | `action=prepare` downloads and compiles the connectome into this agent's home (once per install); `verify`; `bench` | | `mm_market_scanner` | Rank any venue's markets for market making: spread against that venue's round-trip fee, book depth, drift. Works on HIP-3 too | | `fly_chart` | Render the exact frame for a pair (what the fly sees) | -| `fly_brain` | The loop (continuous). `mode=shadow|live`, `pairs`, `picked_spreads_bps`, `run_name` | +| `fly_brain` | The loop (continuous). `mode=shadow|live`, `pairs`, `picked_ranges_bps`, `run_name` | | `fly_status` | Latest posture per pair, last observation, guard state, memory stats | | `fly_report` | The run dashboard: the orbitable fly, the book, the neuron strip, the decision log, and the frame the fly last saw | diff --git a/agents/market_making_fly/flybrain/posture.py b/agents/market_making_fly/flybrain/posture.py index 6666e2f8e..eb69671fe 100644 --- a/agents/market_making_fly/flybrain/posture.py +++ b/agents/market_making_fly/flybrain/posture.py @@ -52,7 +52,7 @@ class MarketSpec: connector_name: str trading_pair: str # BASE-QUOTE, or ISSUER:TOKEN-QUOTE on HIP-3 total_amount_quote: float - picked_spread_bps: float # the observed spread for this market + range_bps: float # median candle range for this market, at the fly's interval # Blank means "derive from the connector name"; see venue.resolve. market_type: str = "" # 1 on spot, where there is nothing to lever. @@ -93,7 +93,7 @@ def __post_init__(self): "maker_fee_bps", venue.default_maker_fee_bps(self.connector_name, resolved), ) - for name in ("total_amount_quote", "picked_spread_bps", "maker_fee_bps"): + for name in ("total_amount_quote", "range_bps", "maker_fee_bps"): value = getattr(self, name) if not math.isfinite(value) or value <= 0: raise ValueError(f"{name} must be finite and positive") @@ -173,18 +173,31 @@ def take_profit_floor_bps(maker_fee_bps: float) -> float: return max(4.0, 2.2 * 2 * maker_fee_bps) -# The outer level must stay outside the inner one. The playbook's S+1 assumed -# a market quoting several bp; on a tight book — XYZ:DRAM-USD quotes 0.35 — -# S+1 lands inside max(2, S/2) and the ladder inverts, so the "outer" level -# fills first and the inventory ladder means nothing. +# The outer level must stay outside the inner one, whatever the arithmetic +# says: a ladder whose second rung is inside its first fills in the wrong +# order and means nothing. LEVEL_STEP_BPS = 1.0 +# Quotes are placed against how far the market actually travels, not against +# how wide its touch is. The playbook derived level 1 from the observed spread +# — half of 1.75 bp on DRAM, so 2 bp — while a typical 5-minute bar there +# ranges 9.9 bp and therefore reaches about 5 bp either side of mid. Every +# level the fly could express, tight or wide, sat inside what a normal bar +# covers, so the same candles touched the same orders whatever it decided: +# five replay variants returned 23 fills and 15 round trips apiece. +# +# Half the range puts level 1 where a typical bar just reaches it, so the +# spread multiplier straddles the fill boundary instead of living inside it: +# 0.6x fills most bars, 2.5x fills few. +LEVEL_ONE_RANGE_FRACTION = 0.5 +LEVEL_STEP_RANGE_FRACTION = 0.25 -def base_levels_from_spread(picked_spread_bps: float) -> tuple[float, float]: - """HIP-3 playbook: level 1 ``max(2, S/2)`` bp, level 2 ``S+1`` bp, with the - second never inside the first.""" - first = max(2.0, picked_spread_bps / 2) - return first, max(picked_spread_bps + 1, first + LEVEL_STEP_BPS) + +def base_levels_from_range(range_bps: float) -> tuple[float, float]: + """Level 1 at half a typical bar's range, level 2 a quarter-bar beyond.""" + first = max(2.0, LEVEL_ONE_RANGE_FRACTION * range_bps) + step = max(LEVEL_STEP_BPS, LEVEL_STEP_RANGE_FRACTION * range_bps) + return first, first + step def take_profit_floor(spec: MarketSpec) -> float: @@ -192,7 +205,7 @@ def take_profit_floor(spec: MarketSpec) -> float: def base_levels_bps(spec: MarketSpec) -> tuple[float, float]: - return base_levels_from_spread(spec.picked_spread_bps) + return base_levels_from_range(spec.range_bps) def _fmt(values: list[float]) -> str: diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index e21385619..687347667 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -103,9 +103,11 @@ class Config(BaseModel): default="XYZ:ORCL-USD", description="1 to 3 uppercase pairs, comma-separated — BASE-QUOTE on any CLOB venue, or ISSUER:TOKEN-QUOTE on HIP-3. This list IS the market count; each pair names its own bot ({slug}-fly) and config ({slug}_fly_mm)", ) - picked_spreads_bps: str = Field( - default="8,8,8", - description="Scanner spread per pair in bp, same order as pairs", + picked_ranges_bps: str = Field( + default="10,10,10", + description="Scanner median candle range per pair in bp, same order as " + "pairs. Quote levels are placed against how far the market travels, not " + "against how wide its touch is", ) connector_name: str = Field( default="hyperliquid_perpetual", @@ -190,10 +192,10 @@ async def _specs(config: Config, pairs: list[str]) -> list[MarketSpec]: The take-profit floor is derived from it, so it is read per pair unless the caller passed a figure of their own. """ - spreads = [float(x) for x in config.picked_spreads_bps.split(",") if x.strip()] - if len(spreads) != len(pairs): + ranges = [float(x) for x in config.picked_ranges_bps.split(",") if x.strip()] + if len(ranges) != len(pairs): raise ValueError( - f"picked_spreads_bps has {len(spreads)} entries for {len(pairs)} pairs" + f"picked_ranges_bps has {len(ranges)} entries for {len(pairs)} pairs" ) market_type = venue.resolve(config.connector_name, config.market_type) return [ @@ -201,7 +203,7 @@ async def _specs(config: Config, pairs: list[str]) -> list[MarketSpec]: connector_name=config.connector_name, trading_pair=pair, total_amount_quote=config.total_amount_quote, - picked_spread_bps=spread, + range_bps=market_range, market_type=config.market_type, leverage=config.leverage, maker_fee_bps=( @@ -210,7 +212,7 @@ async def _specs(config: Config, pairs: list[str]) -> list[MarketSpec]: ), portfolio_allocation=config.portfolio_allocation, ) - for pair, spread in zip(pairs, spreads) + for pair, market_range in zip(pairs, ranges) ] @@ -313,7 +315,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: if k not in ( "pairs", - "picked_spreads_bps", + "picked_ranges_bps", "mode", "fast", "steps", diff --git a/agents/market_making_fly/routines/fly_replay.py b/agents/market_making_fly/routines/fly_replay.py index 6123c7e5d..7dc9a8072 100644 --- a/agents/market_making_fly/routines/fly_replay.py +++ b/agents/market_making_fly/routines/fly_replay.py @@ -82,8 +82,8 @@ class Config(BaseModel): ) total_amount_quote: float = Field(default=200.0) portfolio_allocation: float = Field(default=0.3) - picked_spread_bps: float = Field( - default=0.0, description="0 measures it from the candles" + range_bps: float = Field( + default=0.0, description="0 measures the median candle range from the data" ) maker_fee_bps: float = Field(default=0.0, description="0 fetches the venue's") leverage: int = Field(default=1) @@ -161,10 +161,9 @@ async def _candles(client, config: Config) -> list[dict]: raise RuntimeError(f"No candles for {config.trading_pair}") -def _spread_from_candles(candles: list[dict]) -> float: - """A stand-in for the scanner's measured touch, in bp: the median bar's - range is the only width a candle series knows about. Replay cannot see a - book, and saying so is better than defaulting to a number.""" +def _range_from_candles(candles: list[dict]) -> float: + """The median bar's range in bp — the quantity the quote levels are built + from, and the same median the scanner ranks reach by.""" import statistics ranges = [ @@ -172,7 +171,9 @@ def _spread_from_candles(candles: list[dict]) -> float: for c in candles if float(c.get("close") or 0) > 0 ] - return round(statistics.median(ranges) / 4, 2) if ranges else 2.0 + if not ranges: + raise ValueError("No usable candles to measure a range from") + return round(statistics.median(ranges), 2) async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: @@ -191,12 +192,12 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: fee = config.maker_fee_bps or await venue.maker_fee_bps( config.connector_name, market_type, config.trading_pair ) - spread = config.picked_spread_bps or _spread_from_candles(candles) + market_range = config.range_bps or _range_from_candles(candles) spec = MarketSpec( connector_name=config.connector_name, trading_pair=config.trading_pair, total_amount_quote=config.total_amount_quote, - picked_spread_bps=spread, + range_bps=market_range, leverage=config.leverage, portfolio_allocation=config.portfolio_allocation, maker_fee_bps=fee, @@ -265,7 +266,7 @@ def observe(frame, stimulus, neural_ms, _pool=pool): "pair": config.trading_pair, "interval": config.interval, "candles": len(candles), - "spread_bps": spread, + "range_bps": market_range, "fee_bps": fee, "summaries": summaries, "curves": {r.variant: r.equity_curve for r in results}, @@ -282,7 +283,7 @@ def observe(frame, stimulus, neural_ms, _pool=pool): f"{len(candles):,} {config.interval} candles of {config.trading_pair}, " f"{results[0].ticks if results else 0} ticks after the " f"{config.n_candles}-candle window. Every variant saw the same series and " - f"the same book geometry — {spread:.2f} bp measured spread, {fee:.2f} bp " + f"the same geometry — {market_range:.2f} bp median bar range, {fee:.2f} bp " "maker fee — and each ran on its own freshly seeded brain. Fills assume " "a quote the price touched was ours, so every P&L here is an upper " "bound; the bias is identical across variants, which is what makes the " @@ -379,7 +380,7 @@ def observe(frame, stimulus, neural_ms, _pool=pool): lines = [ f"pair: {config.trading_pair} ({config.interval})", f"candles: {len(candles)}, ticks: {results[0].ticks if results else 0}", - f"spread: {spread:.2f} bp, fee: {fee:.2f} bp", + f"median range: {market_range:.2f} bp, fee: {fee:.2f} bp", ] for s in summaries: lines.append( diff --git a/agents/market_making_fly/routines/mm_market_scanner.py b/agents/market_making_fly/routines/mm_market_scanner.py index 6ccedb7cd..01ed25ab5 100644 --- a/agents/market_making_fly/routines/mm_market_scanner.py +++ b/agents/market_making_fly/routines/mm_market_scanner.py @@ -52,7 +52,7 @@ from flybrain import venue from flybrain.market import LiveMarket, depth_within from flybrain.naming import pair_names -from flybrain.posture import base_levels_from_spread, take_profit_floor_bps +from flybrain.posture import base_levels_from_range, take_profit_floor_bps from pydantic import BaseModel, Field from telegram.ext import ContextTypes @@ -126,15 +126,14 @@ class Config(BaseModel): top_n: int = Field(default=5, ge=1, le=25, description="Markets to report") -def cycle_bps(spread_bps: float, fee_bps: float) -> float: +def cycle_bps(range_bps: float, fee_bps: float) -> float: """The round trip the fly must travel on this market, in bp. - From mid down to where it would rest level 1 — ``max(2, S/2)``, never - inside the fee — and back out through the take-profit floor. This is the - distance a market has to move for one completed pair, and the thing a - candle's range is compared against. + From mid down to where it would rest level 1 — half a typical bar's range, + never inside the fee — and back out through the take-profit floor. This is + the distance a market has to move for one completed pair. """ - entry = max(base_levels_from_spread(spread_bps)[0], fee_bps) + entry = max(base_levels_from_range(range_bps)[0], fee_bps) return entry + take_profit_floor_bps(fee_bps) @@ -283,9 +282,9 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: # never inside the fee, and close at the take-profit floor. The round # trip it must travel is the sum — down to the quote, then back up # through the exit. - entry_bps = max(base_levels_from_spread(spread)[0], fee_bps) + entry_bps = max(base_levels_from_range(range_bps or 0.0)[0], fee_bps) exit_bps = take_profit_floor_bps(fee_bps) - cycle = cycle_bps(spread, fee_bps) + cycle = cycle_bps(range_bps or 0.0, fee_bps) range_bps = m.get("range_bps") # How far a typical candle travels against that cycle. Above 1 the # median candle completes one; below it, the market does not come to @@ -455,7 +454,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: best = survivors[0] lines.append( f"TOP PICK: {best['pair']} at reach {best['reach']:.2f}×, " - f"picked_spread_bps={best['spread_bps']:.2f}" + f"picked_ranges_bps={best['range_bps']:.2f}" ) else: lines.append( diff --git a/agents/market_making_fly/skills/fly_decoder/SKILL.md b/agents/market_making_fly/skills/fly_decoder/SKILL.md index 4cad7a40f..f07494173 100644 --- a/agents/market_making_fly/skills/fly_decoder/SKILL.md +++ b/agents/market_making_fly/skills/fly_decoder/SKILL.md @@ -41,7 +41,8 @@ user wants to *look* at the run, `fly_status` when you need to quote figures. The sign on each is a choice, not a finding: arousal is a population rate against its own average and nothing ties it to volatility. `lean` = clip(trend_z, ±3 bp), 0 without a gate spike, and capped at half the first spread level when mapped. -* Mapping: level 1 = max(2, S/2) bp, level 2 = S+1 bp (S = scanner spread), times +* Mapping: level 1 = max(2, R/2) bp, level 2 = level 1 + R/4 (R = the market's + median candle range), times `spread ×`, buy −lean / sell +lean, floor 3 bp; TP = max(4 bp, 2.2 × round-trip fee, first level); timing per regime; `pause` sets `manual_kill_switch`. diff --git a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md index 76ecd3e3c..29e640330 100644 --- a/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md +++ b/agents/market_making_fly/skills/fly_mm_deploy/SKILL.md @@ -35,21 +35,25 @@ manage_routines(action="run", name="mm_market_scanner", config={ "prescreen": 30, "top_n": 5}) ``` -It ranks by volume, by how far the spread clears **that venue's** round-trip -maker fee, and by book depth, then reports why every rejected market failed. +It ranks by volume, by whether a typical candle travels the round trip the fly +must make — down to its quote, back out through the take-profit — and by book +depth, then reports why every rejected market failed. Take the top **`n_markets`** (1-3; from `[CURRENT CONFIG]` or the task, default 3) with an open book — one brain quotes them all in round-robin. Record each -`pair` and its **spread in bp**; that is `picked_spreads_bps`. +`pair` and its **median candle range in bp**; that is `picked_ranges_bps`. The +quote levels are built from how far the market travels, not from how wide its +touch is: level 1 sits at half a typical bar, where the spread multiplier can +actually change whether a bar reaches it. **If nothing survives, raise `prescreen` before anything else.** Reading a book -costs a call, so only the busiest markets are read — and the widest markets are -rarely the busiest. On HIP-3 the eight heaviest markets all quote under 1.4 bp, -nowhere near the 3.9 bp needed to clear a 2.6 bp round trip; the first market -that cleared it sat thirtieth by volume. A `TOP PICK: none` line means the scan -did not look far enough, or this venue is genuinely too tight to quote. +costs a call, so only the busiest markets are read — and the busiest markets are +not the most reachable. On Hyperliquid the heaviest HIP-3 markets barely move +2 bp in five minutes against a 7.7 bp cycle; the first market that cleared it +sat fourth by volume. A `TOP PICK: none` line means the scan did not look far +enough, or this venue genuinely does not come to a resting quote. If it still finds nothing, stop and report that rather than lowering the -spread floor: quoting inside the fee loses money on every fill. +floor: quoting inside the fee loses money on every fill. ## Step 2 — Collateral @@ -71,7 +75,7 @@ import sys; sys.path.insert(0, "agents/market_making_fly") from flybrain.posture import MarketSpec, build_config from flybrain.decoder import NEUTRAL spec = MarketSpec(connector_name="binance_perpetual", trading_pair="SOL-USDT", - total_amount_quote=500, picked_spread_bps=8.0, leverage=3, + total_amount_quote=500, range_bps=10.0, leverage=3, portfolio_allocation=0.2) # spot: leverage=1, and check maker_fee_bps print(build_config(spec, NEUTRAL)) """) @@ -106,13 +110,13 @@ controller. ## Step 5 — Start the fly in shadow -`pairs` and `picked_spreads_bps` list exactly the `n_markets` picks, same order. -With `n_markets: 1` that is a single pair and a single spread. +`pairs` and `picked_ranges_bps` list exactly the `n_markets` picks, same order. +With `n_markets: 1` that is a single pair and a single range. ``` manage_routines(action="start", name="fly_brain", config={ "pairs": "XYZ:DRAM-USD,XYZ:SPCX-USD,XYZ:SMSN-USD", # n_markets entries - "picked_spreads_bps": "8,6,10", # one per pair + "picked_ranges_bps": "10,8,14", # one per pair "total_amount_quote": 500, "leverage": 3, "portfolio_allocation": 0.2, "mode": "shadow", "run_name": "fly-2026-09-12"}) ``` @@ -128,7 +132,7 @@ manage_routines(action="run", name="fly_status", config={"run_name": "fly-2026-0 manage_routines(action="run", name="fly_report", config={"run_name": "fly-2026-09-12"}) ``` -Report: pairs, spreads, bots running, fly tick count, first postures, any vetoes or +Report: pairs, ranges, bots running, fly tick count, first postures, any vetoes or halts, and the caveat that the fly's learning is not validated. ## Step 7 — Live (only when the task says so) @@ -143,7 +147,7 @@ period first. When a slot's market is closed, dominated, or the operator asks: stop that bot (`manage_bots(action="stop_bot", bot_name=...)`), re-run the scanner, deploy the new pick with Steps 3–4, stop `fly_brain` and start it again with the updated `pairs` and -`picked_spreads_bps` and the same `run_name`. The brain keeps its memory; only the +`picked_ranges_bps` and the same `run_name`. The brain keeps its memory; only the swapped pair's baseline starts over. ## Halts diff --git a/agents/market_making_fly/tests/test_fly_guard.py b/agents/market_making_fly/tests/test_fly_guard.py index 526fc7075..c07682deb 100644 --- a/agents/market_making_fly/tests/test_fly_guard.py +++ b/agents/market_making_fly/tests/test_fly_guard.py @@ -160,7 +160,7 @@ def test_a_config_sitting_exactly_on_the_floor_is_not_vetoed(): connector_name="hyperliquid_perpetual", trading_pair="XYZ:DRAM-USD", total_amount_quote=200, - picked_spread_bps=0.35, + range_bps=4.0, leverage=1, portfolio_allocation=0.3, maker_fee_bps=1.3, diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py index 29e6bb7f8..3dbff88a9 100644 --- a/agents/market_making_fly/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -16,7 +16,7 @@ connector_name="hyperliquid_perpetual", trading_pair="XYZ:DRAM-USD", total_amount_quote=500, - picked_spread_bps=8.0, + range_bps=10.0, # What a HIP-3 market in growth mode costs all-in: 0.3 bp to the venue plus # 1.0 bp of builder fee. The loop fetches this per market; a core # Hyperliquid perp is 2.5 bp, which is why it is not assumed here. @@ -28,12 +28,12 @@ def _spreads(value): return [float(x) for x in value.split(",")] -def test_neutral_config_matches_hip3_base(): +def test_neutral_config_sits_at_half_a_bar_and_a_quarter_beyond(): cfg = build_config(SPEC, NEUTRAL) l1, l2 = base_levels_bps(SPEC) - assert (l1, l2) == (4.0, 9.0) - assert _spreads(cfg["buy_spreads"]) == pytest.approx([4 * BPS, 9 * BPS]) - assert _spreads(cfg["sell_spreads"]) == pytest.approx([4 * BPS, 9 * BPS]) + assert (l1, l2) == (5.0, 7.5) # a 10 bp bar: half of it, then a quarter more + assert _spreads(cfg["buy_spreads"]) == pytest.approx([5 * BPS, 7.5 * BPS]) + assert _spreads(cfg["sell_spreads"]) == pytest.approx([5 * BPS, 7.5 * BPS]) assert ( cfg["controller_name"] == "pmm_mister" and cfg["controller_type"] == "generic" ) @@ -56,15 +56,15 @@ def test_volatile_widens_quiet_tightens_with_floor(): wide = build_config( SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, 0.0, False, True) ) - assert _spreads(wide["buy_spreads"])[0] == pytest.approx(8 * BPS) + assert _spreads(wide["buy_spreads"])[0] == pytest.approx(10 * BPS) assert (wide["executor_refresh_time"], wide["buy_cooldown_time"]) == TIMING[ "volatile" ] tight = build_config( SPEC, Posture("quiet", 0.6, 1.0, 0.0, 0, -1.5, 0.0, False, True) ) - # 4 bp × 0.6 = 2.4 bp, which clears this market's 1.3 bp fee floor - assert _spreads(tight["buy_spreads"])[0] == pytest.approx(2.4 * BPS) + # 5 bp × 0.6 = 3 bp, well inside a typical bar and clear of the 1.3 bp fee + assert _spreads(tight["buy_spreads"])[0] == pytest.approx(3 * BPS) assert (tight["executor_refresh_time"], tight["buy_cooldown_time"]) == TIMING[ "quiet" ] @@ -75,14 +75,14 @@ def test_lean_is_asymmetric_and_capped(): SPEC, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) ) buy, sell = _spreads(up["buy_spreads"]), _spreads(up["sell_spreads"]) - # lean capped at half of level 1 (4 bp → 2 bp), and 2 bp still clears the fee - assert buy[0] == pytest.approx(2 * BPS) and sell[0] == pytest.approx(6 * BPS) - assert buy[1] == pytest.approx(7 * BPS) and sell[1] == pytest.approx(11 * BPS) + # lean capped at half of level 1 (5 bp → 2.5 bp), clear of the fee floor + assert buy[0] == pytest.approx(2.5 * BPS) and sell[0] == pytest.approx(7.5 * BPS) + assert buy[1] == pytest.approx(5 * BPS) and sell[1] == pytest.approx(10 * BPS) down = build_config( SPEC, Posture("trending_down", 1.0, 1.0, -3.0, -2.0, 0, 0.0, True, True) ) - assert _spreads(down["sell_spreads"])[0] == pytest.approx(2 * BPS) - assert _spreads(down["buy_spreads"])[0] == pytest.approx(6 * BPS) + assert _spreads(down["sell_spreads"])[0] == pytest.approx(2.5 * BPS) + assert _spreads(down["buy_spreads"])[0] == pytest.approx(7.5 * BPS) def test_the_spread_floor_is_the_market_own_fee(): @@ -94,7 +94,7 @@ def test_the_spread_floor_is_the_market_own_fee(): connector_name="binance", trading_pair="SOL-USDT", total_amount_quote=500, - picked_spread_bps=8.0, + range_bps=10.0, ) assert dear.min_spread_bps == pytest.approx(7.5) # binance spot # a lean that would quote inside the fee is pushed back out to it @@ -193,19 +193,24 @@ def test_an_order_sized_to_the_bare_minimum_is_refused(): assert roomy.order_notional == pytest.approx(15.0) -def test_the_outer_level_never_lands_inside_the_inner_one(): - """XYZ:DRAM-USD quotes 0.35 bp, where the playbook's S+1 (1.35) falls - inside max(2, S/2) (2.0) and the ladder inverts.""" - from flybrain.posture import base_levels_from_spread - - for spread in (0.1, 0.35, 1.75, 2.0, 8.0, 20.0): - first, second = base_levels_from_spread(spread) - assert second > first, f"levels inverted at S={spread}" - assert base_levels_from_spread(0.35) == (2.0, 3.0) - assert base_levels_from_spread(8.0) == (4.0, 9.0) # wide markets unchanged - tight = build_config( - MarketSpec(**{**SPEC.__dict__, "picked_spread_bps": 0.35}), NEUTRAL - ) +def test_quotes_are_placed_against_how_far_the_market_travels(): + """Level 1 sat at half the *touch* — 2 bp on a market whose typical bar + ranges 9.9 — so every multiplier the fly could express stayed inside what a + normal bar covers, and five replay variants returned the same 23 fills. + Half the range puts it where a typical bar just reaches.""" + from flybrain.posture import base_levels_from_range + + assert base_levels_from_range(9.9) == (pytest.approx(4.95), pytest.approx(7.425)) + # the multiplier now straddles a typical bar's reach instead of living + # inside it: tight fills most bars, wide fills few + first, _ = base_levels_from_range(9.9) + assert first * 0.6 < 9.9 / 2 < first * 2.5 + # a dead market still gets a floor rather than a quote on top of mid + assert base_levels_from_range(0.4)[0] == 2.0 + for market_range in (0.1, 0.4, 4.0, 9.9, 40.0): + first, second = base_levels_from_range(market_range) + assert second > first, f"levels inverted at range={market_range}" + tight = build_config(MarketSpec(**{**SPEC.__dict__, "range_bps": 0.4}), NEUTRAL) buys = _spreads(tight["buy_spreads"]) assert buys == sorted(buys) and len(set(buys)) == 2 diff --git a/agents/market_making_fly/tests/test_fly_replay.py b/agents/market_making_fly/tests/test_fly_replay.py index 81fdc1ef9..9aca65614 100644 --- a/agents/market_making_fly/tests/test_fly_replay.py +++ b/agents/market_making_fly/tests/test_fly_replay.py @@ -9,7 +9,7 @@ connector_name="hyperliquid_perpetual", trading_pair="XYZ:DRAM-USD", total_amount_quote=200, - picked_spread_bps=2.0, + range_bps=10.0, leverage=1, portfolio_allocation=0.3, maker_fee_bps=1.3, From a112137ce4779f72deb1cfa006c65ae75c62ff3e Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 14:33:15 -0700 Subject: [PATCH 38/48] (fix) pin the replay window, or a lever's before and after are two anecdotes The venue serves only the latest N candles, so every replay fetched a different market. Two runs ninety minutes apart, same code and same seeds, reversed the sign of the fly's difference from its frozen-plasticity control: 0.63 behind in one window, 0.31 ahead in the next. Comparing a lever's before and after across such runs measures the window, not the lever. The first fetch is now pinned to disk per pair and interval and reused until refresh_candles asks for a new one, and the report says which window it scored. --- .../market_making_fly/routines/fly_replay.py | 52 +++++++++++++++---- 1 file changed, 42 insertions(+), 10 deletions(-) diff --git a/agents/market_making_fly/routines/fly_replay.py b/agents/market_making_fly/routines/fly_replay.py index 7dc9a8072..c82eefef5 100644 --- a/agents/market_making_fly/routines/fly_replay.py +++ b/agents/market_making_fly/routines/fly_replay.py @@ -33,6 +33,7 @@ import json import logging import multiprocessing +import time from concurrent.futures import ProcessPoolExecutor import plotly.graph_objects as go @@ -92,6 +93,13 @@ class Config(BaseModel): baseline_window: int = Field(default=60) baseline_warmup: int = Field(default=10) seed: int = Field(default=7301, description="Seed for the shuffled control") + refresh_candles: bool = Field( + default=False, + description="Fetch a new window and pin it. Off by default: the venue " + "only serves the latest N candles, so two runs an hour apart replay " + "different markets and are not comparable — the sign of a control's " + "difference flipped between two such windows on 2026-09-13", + ) concurrency: int = Field( default=4, ge=1, @@ -142,8 +150,20 @@ def _curve_figure(results: list) -> go.Figure: return fig -async def _candles(client, config: Config) -> list[dict]: - """The series every variant replays. One fetch, so they cannot differ.""" +async def _candles(client, config: Config, pinned: Path) -> list[dict]: + """The series every variant replays — and every *later* run replays too. + + The venue serves only the latest N candles, so fetching each time means two + runs an hour apart are scored on different markets. That is not a detail: + between two such windows the sign of the fly's difference from its frozen + control reversed. So the first fetch is pinned to disk and reused until + someone asks for a new one, which is what makes a lever's before and after + a comparison rather than two anecdotes. + """ + if pinned.exists() and not config.refresh_candles: + saved = json.loads(pinned.read_text()) + if saved.get("interval") == config.interval and saved.get("candles"): + return saved["candles"] for attempt in (1, 2): # a cold feed answers on the second ask try: raw = await client.market_data.get_candles( @@ -154,6 +174,17 @@ async def _candles(client, config: Config) -> list[dict]: ) rows = raw if isinstance(raw, list) else raw.get("data", raw.get("candles")) if rows: + pinned.parent.mkdir(parents=True, exist_ok=True) + pinned.write_text( + json.dumps( + { + "pair": config.trading_pair, + "interval": config.interval, + "fetched_at": time.time(), + "candles": list(rows), + } + ) + ) return list(rows) except Exception: if attempt == 2: @@ -187,7 +218,11 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: client = await get_client(context._chat_id, context=context) if not client: return "No server available" - candles = await _candles(client, config) + slug = pair_names(config.trading_pair).slug + home = agent_home(AGENT_SLUG) / "replay" + home.mkdir(parents=True, exist_ok=True) + pinned = home / f"{slug}-{config.interval}-candles.json" + candles = await _candles(client, config, pinned) market_type = venue.market_type_for(config.connector_name) fee = config.maker_fee_bps or await venue.maker_fee_bps( config.connector_name, market_type, config.trading_pair @@ -203,12 +238,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: maker_fee_bps=fee, ) spec.check_order_size() - record = ( - agent_home(AGENT_SLUG) - / "replay" - / f"{pair_names(config.trading_pair).slug}.json" - ) - record.parent.mkdir(parents=True, exist_ok=True) + record = home / f"{slug}.json" loop = asyncio.get_running_loop() gate = asyncio.Semaphore(config.concurrency) @@ -280,7 +310,9 @@ def observe(frame, stimulus, neural_ms, _pool=pool): builder.manual_order() builder.section( "WHAT WAS REPLAYED", - f"{len(candles):,} {config.interval} candles of {config.trading_pair}, " + f"{len(candles):,} {config.interval} candles of {config.trading_pair}" + + (" (freshly fetched)" if config.refresh_candles else " (the pinned window)") + + ", " f"{results[0].ticks if results else 0} ticks after the " f"{config.n_candles}-candle window. Every variant saw the same series and " f"the same geometry — {market_range:.2f} bp median bar range, {fee:.2f} bp " From ebc7df6e52c52a57449c3af95e7b626f46f02a33 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 14:39:28 -0700 Subject: [PATCH 39/48] (feat) the fly decides how long to hold, not just where to quote The take-profit was one number for every market and every posture -- the fee floor, 5.72 bp -- so five replay variants closed the same eleven round trips however differently they quoted. It is the only thing that decides when a position ends, and nothing was deciding it. The base is now three quarters of a typical bar, which scales with the market instead of with the fee alone, and arousal moves it: the same channel that tightens the entry widens the exit, because an active market both fills a quote sooner and travels further afterwards. The fee floor and the fly's own first level stay underneath it -- an exit inside the spread it quotes would close for nothing. A flat-tp variant joins the replay, with the exit where the range puts it and the fly unable to move it, so the value of deciding how long to hold can be read separately from the value of deciding where to quote. --- agents/market_making_fly/flybrain/decoder.py | 16 ++++- agents/market_making_fly/flybrain/posture.py | 25 +++++++- agents/market_making_fly/flybrain/replay.py | 2 + .../market_making_fly/routines/fly_brain.py | 1 + .../market_making_fly/routines/fly_replay.py | 11 +++- .../market_making_fly/routines/fly_report.py | 5 +- .../skills/fly_decoder/SKILL.md | 4 +- .../tests/test_fly_decoder.py | 12 ++-- .../market_making_fly/tests/test_fly_guard.py | 2 +- .../tests/test_fly_posture.py | 62 +++++++++++++++---- 10 files changed, 112 insertions(+), 28 deletions(-) diff --git a/agents/market_making_fly/flybrain/decoder.py b/agents/market_making_fly/flybrain/decoder.py index 7ffc69c4e..9ebdd5a5a 100644 --- a/agents/market_making_fly/flybrain/decoder.py +++ b/agents/market_making_fly/flybrain/decoder.py @@ -7,7 +7,8 @@ * ``trend_hz`` — mean DNp20 right rate minus mean left rate (stonkfly's BUY/SELL cells). Sign → which way the reference price leans. * ``arousal_hz`` — mean rate of the descending-neuron population. Higher → - tighter spreads and a larger share of the book quoted. + tighter spreads, a larger share of the book quoted, and a + wider take-profit to hold for. * ``gate`` — DNpe017 spikes ≥ 1, required for a trending call. * ``valence_hz`` — mean MBON07 rate minus mean MBON11 rate: approach minus avoidance. These are the cells the KC→MBON memory rule @@ -75,6 +76,12 @@ class DecoderSettings: # arousal and added to it, so the size the fly commits carries both "the # market is active" and "this looked good last time". valence_gain: float = 0.5 + # How long to hold for. An active market both fills a quote sooner and + # travels further afterwards, so the same arousal that tightens the entry + # widens the exit: the fly asks for more of a range it can see moving. + tp_gain: float = 0.5 + tp_min: float = 0.6 + tp_max: float = 2.5 # A scene this far below the mushroom body's own recent drive is one the # fly effectively did not see; its z-scores are noise. z_kc_quiet: float = -1.5 @@ -90,6 +97,7 @@ def __post_init__(self): for lo, hi, what in ( (self.spread_min, self.spread_max, "spread"), (self.size_min, self.size_max, "size"), + (self.tp_min, self.tp_max, "tp"), ): if not 0 < lo <= 1 <= hi: raise ValueError(f"{what}_min <= 1 <= {what}_max required") @@ -101,7 +109,7 @@ def __post_init__(self): # but not zero, which would mean the channel is read and discarded. if self.z_kc_quiet >= 0: raise ValueError("z_kc_quiet must be negative") - for name in ("spread_gain", "size_gain", "valence_gain"): + for name in ("spread_gain", "size_gain", "valence_gain", "tp_gain"): value = getattr(self, name) if not math.isfinite(value) or value == 0: raise ValueError(f"{name} must be finite and non-zero") @@ -179,6 +187,7 @@ class Posture: regime: str spread_mult: float size_mult: float + tp_mult: float shift_bps: float trend_z: float arousal_z: float @@ -199,6 +208,7 @@ def from_dict(cls, data: dict) -> "Posture": regime="ranging", spread_mult=1.0, size_mult=1.0, + tp_mult=1.0, shift_bps=0.0, trend_z=0.0, arousal_z=0.0, @@ -248,6 +258,7 @@ def decode(channels: Channels, baseline: Baseline, s: DecoderSettings) -> Postur s.size_max, max(s.size_min, 1 + s.size_gain * arousal_z + s.valence_gain * valence_z), ) + tp_mult = min(s.tp_max, max(s.tp_min, 1 + s.tp_gain * arousal_z)) shift = max(-s.max_shift_bps, min(s.max_shift_bps, s.shift_gain_bps * trend_z)) if not gate: shift = 0.0 # no descending gate spike, no directional lean @@ -255,6 +266,7 @@ def decode(channels: Channels, baseline: Baseline, s: DecoderSettings) -> Postur regime=regime, spread_mult=round(spread_mult, 4), size_mult=round(size_mult, 4), + tp_mult=round(tp_mult, 4), shift_bps=round(shift, 3), trend_z=round(trend_z, 4), arousal_z=round(arousal_z, 4), diff --git a/agents/market_making_fly/flybrain/posture.py b/agents/market_making_fly/flybrain/posture.py index eb69671fe..a5065dbee 100644 --- a/agents/market_making_fly/flybrain/posture.py +++ b/agents/market_making_fly/flybrain/posture.py @@ -173,6 +173,20 @@ def take_profit_floor_bps(maker_fee_bps: float) -> float: return max(4.0, 2.2 * 2 * maker_fee_bps) +# The exit was one number for every market and every posture, so a replay of +# five variants closed the same eleven round trips apiece however they quoted. +# Three quarters of a typical bar is a target the market reaches often without +# giving the whole excursion away, and it scales with the market rather than +# with the fee alone — the fee only ever sets the floor. +TP_RANGE_FRACTION = 0.75 + + +def take_profit_base_bps(maker_fee_bps: float, range_bps: float) -> float: + """The exit before the fly moves it: a fraction of the range, never under + what a round trip costs.""" + return max(take_profit_floor_bps(maker_fee_bps), TP_RANGE_FRACTION * range_bps) + + # The outer level must stay outside the inner one, whatever the arithmetic # says: a ladder whose second rung is inside its first fills in the wrong # order and means nothing. @@ -232,7 +246,16 @@ def build_config(spec: MarketSpec, posture: Posture) -> dict: shift = max(-levels[0] / 2, min(levels[0] / 2, posture.shift_bps)) buy = [max(spec.min_spread_bps, lvl - shift) for lvl in levels] sell = [max(spec.min_spread_bps, lvl + shift) for lvl in levels] - take_profit = max(take_profit_floor(spec), min(buy[0], sell[0]) * BPS) + # The fly asks for more of the range when it is aroused, but never less + # than a round trip costs, and never less than its own first level — a + # take-profit inside the spread it quotes would close for nothing. + take_profit = max( + take_profit_floor(spec), + min(buy[0], sell[0]) * BPS, + take_profit_base_bps(spec.maker_fee_bps, spec.range_bps) + * posture.tp_mult + * BPS, + ) refresh, cooldown = TIMING[posture.regime] config = { "controller_type": "generic", diff --git a/agents/market_making_fly/flybrain/replay.py b/agents/market_making_fly/flybrain/replay.py index de323faf6..5f6d27745 100644 --- a/agents/market_making_fly/flybrain/replay.py +++ b/agents/market_making_fly/flybrain/replay.py @@ -185,6 +185,7 @@ def summary(self) -> dict: regimes[p.regime] = regimes.get(p.regime, 0) + 1 spreads = [p.spread_mult for p in self.postures] or [0.0] sizes = [p.size_mult for p in self.postures] or [0.0] + tps = [p.tp_mult for p in self.postures] or [0.0] return { "variant": self.variant, "ticks": self.ticks, @@ -199,6 +200,7 @@ def summary(self) -> dict: "unconfident": self.unconfident, "mean_spread_mult": round(sum(spreads) / len(spreads), 3), "mean_size_mult": round(sum(sizes) / len(sizes), 3), + "mean_tp_mult": round(sum(tps) / len(tps), 3), "regimes": regimes, } diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index 687347667..a7d44930c 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -579,6 +579,7 @@ async def pace() -> None: b.kpi("Spread ×", f"{posture.spread_mult:.2f}") b.kpi("Size ×", f"{posture.size_mult:.2f}") b.kpi("Valence z", f"{posture.valence_z:+.2f}") + b.kpi("TP ×", f"{posture.tp_mult:.2f}") b.kpi("Lean", f"{posture.shift_bps:+.2f} bp") b.kpi("Trend z", f"{posture.trend_z:+.2f}") b.kpi("Arousal z", f"{posture.arousal_z:+.2f}") diff --git a/agents/market_making_fly/routines/fly_replay.py b/agents/market_making_fly/routines/fly_replay.py index c82eefef5..df872d18b 100644 --- a/agents/market_making_fly/routines/fly_replay.py +++ b/agents/market_making_fly/routines/fly_replay.py @@ -15,6 +15,9 @@ * ``shuffled`` — reinforcement of the same frequency and magnitude, with the sign randomised. The control the caveats have always demanded. * ``widen`` — the old arousal direction, for the A/B that motivated the flip. +* ``flat-tp`` — the exit fixed where the range puts it, with the fly unable to + move it. Says whether deciding how long to hold is worth + anything, separately from deciding where to quote. Each variant gets its own brain process: a network that has already learned from one variant is not a control for the next. @@ -65,6 +68,7 @@ "no-valence": {"valence_gain": OFF}, "shuffled": {"shuffle": True}, "widen": {"spread_gain": 0.5}, + "flat-tp": {"tp_gain": OFF}, } @@ -78,7 +82,7 @@ class Config(BaseModel): default=1000, ge=200, le=5000, description="Candles to fetch" ) variants: str = Field( - default="live,no-memory,no-valence,shuffled,widen", + default="live,no-memory,no-valence,shuffled,widen,flat-tp", description=f"Comma-separated, from: {', '.join(VARIANTS)}", ) total_amount_quote: float = Field(default=200.0) @@ -340,6 +344,7 @@ def observe(frame, stimulus, neural_ms, _pool=pool): "Unconfident": f"{s['unconfident']:,}", "Spread ×": f"{s['mean_spread_mult']:.2f}", "Size ×": f"{s['mean_size_mult']:.2f}", + "TP ×": f"{s['mean_tp_mult']:.2f}", } for s in summaries ], @@ -354,6 +359,7 @@ def observe(frame, stimulus, neural_ms, _pool=pool): "Unconfident", "Spread ×", "Size ×", + "TP ×", ], ) builder.plotly(_curve_figure(results)) @@ -418,7 +424,8 @@ def observe(frame, stimulus, neural_ms, _pool=pool): lines.append( f"{s['variant']}: net {s['net']:+.4f}, {s['fills']} fills, " f"{s['round_trips']} round trips, {s['applies']} applies, " - f"spread ×{s['mean_spread_mult']:.2f}, size ×{s['mean_size_mult']:.2f}" + f"spread ×{s['mean_spread_mult']:.2f}, size ×{s['mean_size_mult']:.2f}, " + f"tp ×{s['mean_tp_mult']:.2f}" ) if len(results) > 1: for other in results[1:]: diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index 77f0a87f4..c6095b8cd 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -390,9 +390,10 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: # One card, not two: both multipliers are read off the same channel, and # the stack's height is matched to the brain beside it at six cards. builder.kpi( - "Spread / size ×", + "Spread / size / TP ×", f"{_fmt(last_posture.get('spread_mult'))} / " - f"{_fmt(last_posture.get('size_mult'))}", + f"{_fmt(last_posture.get('size_mult'))} / " + f"{_fmt(last_posture.get('tp_mult'))}", width=CARDS, ) builder.kpi( diff --git a/agents/market_making_fly/skills/fly_decoder/SKILL.md b/agents/market_making_fly/skills/fly_decoder/SKILL.md index f07494173..41f0402a0 100644 --- a/agents/market_making_fly/skills/fly_decoder/SKILL.md +++ b/agents/market_making_fly/skills/fly_decoder/SKILL.md @@ -37,7 +37,9 @@ user wants to *look* at the run, `fly_status` when you need to quote figures. `size ×` = clip(1 + 0.5·arousal_z, 0.6, 2.5) — and quotes **more** of the book, clamped so an order never falls under the venue minimum nor the allocation over 1. `size ×` also carries `+0.5·valence_z`, so what the fly has learned about scenes - like this one moves the capital it commits. + like this one moves the capital it commits. `tp ×` = clip(1 + 0.5·arousal_z, 0.6, + 2.5) on a base of 0.75 × the market's median candle range — an active market both + fills sooner and travels further, so the fly holds for more of it. The sign on each is a choice, not a finding: arousal is a population rate against its own average and nothing ties it to volatility. `lean` = clip(trend_z, ±3 bp), 0 without a gate spike, and capped at half the first spread level when mapped. diff --git a/agents/market_making_fly/tests/test_fly_decoder.py b/agents/market_making_fly/tests/test_fly_decoder.py index 96f983746..52f8c417e 100644 --- a/agents/market_making_fly/tests/test_fly_decoder.py +++ b/agents/market_making_fly/tests/test_fly_decoder.py @@ -112,7 +112,7 @@ def test_baseline_window_and_roundtrip(): def test_posture_roundtrip(): - p = Posture("quiet", 0.8, 1.0, -1.0, -0.2, -1.3, 0.0, True, True) + p = Posture("quiet", 0.8, 1.0, 1.0, -1.0, -0.2, -1.3, 0.0, True, True) assert Posture.from_dict(p.to_dict()) == p @@ -132,16 +132,16 @@ def test_settings_validation(): def test_hysteresis(): h = Hysteresis(min_apply_interval_sec=300) - base = Posture("ranging", 1.0, 1.0, 0.0, 0, 0, 0.0, False, True) + base = Posture("ranging", 1.0, 1.0, 1.0, 0.0, 0, 0, 0.0, False, True) assert should_apply(None, base, None, 1000, h)[0] - same = Posture("ranging", 1.05, 1.0, 0.2, 0, 0, 0.0, False, True) + same = Posture("ranging", 1.05, 1.0, 1.0, 0.2, 0, 0, 0.0, False, True) assert not should_apply(base, same, 0, 1000, h)[0] - regime = Posture("volatile", 1.0, 1.0, 0.0, 0, 1.2, 0.0, False, True) + regime = Posture("volatile", 1.0, 1.0, 1.0, 0.0, 0, 1.2, 0.0, False, True) assert should_apply(base, regime, 0, 1000, h)[0] assert not should_apply(base, regime, 900, 1000, h)[0] # cooldown - wider = Posture("ranging", 1.2, 1.0, 0.0, 0, 0, 0.0, False, True) + wider = Posture("ranging", 1.2, 1.0, 1.0, 0.0, 0, 0, 0.0, False, True) assert should_apply(base, wider, 0, 1000, h)[0] - lean = Posture("ranging", 1.0, 1.0, 0.6, 0, 0, 0.0, True, True) + lean = Posture("ranging", 1.0, 1.0, 1.0, 0.6, 0, 0, 0.0, True, True) assert should_apply(base, lean, 0, 1000, h)[0] diff --git a/agents/market_making_fly/tests/test_fly_guard.py b/agents/market_making_fly/tests/test_fly_guard.py index c07682deb..7c43ae143 100644 --- a/agents/market_making_fly/tests/test_fly_guard.py +++ b/agents/market_making_fly/tests/test_fly_guard.py @@ -165,7 +165,7 @@ def test_a_config_sitting_exactly_on_the_floor_is_not_vetoed(): portfolio_allocation=0.3, maker_fee_bps=1.3, ) - leaned = Posture("trending_up", 1.14, 1.0, 1.27, 2.0, 0.3, 0.0, True, True) + leaned = Posture("trending_up", 1.14, 1.0, 1.0, 1.27, 2.0, 0.3, 0.0, True, True) config = build_config(spec, leaned) assert min(float(x) for x in config["buy_spreads"].split(",")) == pytest.approx( spec.min_spread_bps * 1e-4 diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py index 3dbff88a9..540095d3d 100644 --- a/agents/market_making_fly/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -54,14 +54,14 @@ def test_take_profit_floor_beats_fees_and_spread(): def test_volatile_widens_quiet_tightens_with_floor(): wide = build_config( - SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, 0.0, False, True) + SPEC, Posture("volatile", 2.0, 1.0, 1.0, 0.0, 0, 1.5, 0.0, False, True) ) assert _spreads(wide["buy_spreads"])[0] == pytest.approx(10 * BPS) assert (wide["executor_refresh_time"], wide["buy_cooldown_time"]) == TIMING[ "volatile" ] tight = build_config( - SPEC, Posture("quiet", 0.6, 1.0, 0.0, 0, -1.5, 0.0, False, True) + SPEC, Posture("quiet", 0.6, 1.0, 1.0, 0.0, 0, -1.5, 0.0, False, True) ) # 5 bp × 0.6 = 3 bp, well inside a typical bar and clear of the 1.3 bp fee assert _spreads(tight["buy_spreads"])[0] == pytest.approx(3 * BPS) @@ -72,14 +72,14 @@ def test_volatile_widens_quiet_tightens_with_floor(): def test_lean_is_asymmetric_and_capped(): up = build_config( - SPEC, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) + SPEC, Posture("trending_up", 1.0, 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) ) buy, sell = _spreads(up["buy_spreads"]), _spreads(up["sell_spreads"]) # lean capped at half of level 1 (5 bp → 2.5 bp), clear of the fee floor assert buy[0] == pytest.approx(2.5 * BPS) and sell[0] == pytest.approx(7.5 * BPS) assert buy[1] == pytest.approx(5 * BPS) and sell[1] == pytest.approx(10 * BPS) down = build_config( - SPEC, Posture("trending_down", 1.0, 1.0, -3.0, -2.0, 0, 0.0, True, True) + SPEC, Posture("trending_down", 1.0, 1.0, 1.0, -3.0, -2.0, 0, 0.0, True, True) ) assert _spreads(down["sell_spreads"])[0] == pytest.approx(2.5 * BPS) assert _spreads(down["buy_spreads"])[0] == pytest.approx(7.5 * BPS) @@ -99,13 +99,15 @@ def test_the_spread_floor_is_the_market_own_fee(): assert dear.min_spread_bps == pytest.approx(7.5) # binance spot # a lean that would quote inside the fee is pushed back out to it leaned = build_config( - dear, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) + dear, Posture("trending_up", 1.0, 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) ) assert min(_spreads(leaned["buy_spreads"])) == pytest.approx(7.5 * BPS) def test_pause_sets_kill_switch(): - cfg = build_config(SPEC, Posture("pause", 2.5, 1.0, 0.0, 0, 3.0, 0.0, False, True)) + cfg = build_config( + SPEC, Posture("pause", 2.5, 1.0, 1.0, 0.0, 0, 3.0, 0.0, False, True) + ) assert cfg["manual_kill_switch"] is True @@ -114,7 +116,7 @@ def test_every_spread_respects_min(): for mult in (0.6, 1.0, 2.5): for shift in (-3.0, 0.0, 3.0): cfg = build_config( - SPEC, Posture(regime, mult, 1.0, shift, 0, 0, 0.0, True, True) + SPEC, Posture(regime, mult, 1.0, 1.0, shift, 0, 0, 0.0, True, True) ) for key in ("buy_spreads", "sell_spreads"): assert min(_spreads(cfg[key])) >= SPEC.min_spread_bps * BPS - 1e-12 @@ -171,7 +173,9 @@ def test_the_fee_floor_follows_the_venue(): def test_config_diff(): a = build_config(SPEC, NEUTRAL) - b = build_config(SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, 0.0, False, True)) + b = build_config( + SPEC, Posture("volatile", 2.0, 1.0, 1.0, 0.0, 0, 1.5, 0.0, False, True) + ) diff = config_diff(a, b) assert "buy_spreads" in diff and "trading_pair" not in diff assert config_diff(None, a) == a @@ -224,17 +228,21 @@ def test_size_follows_arousal_and_stays_inside_both_limits(): spec = MarketSpec( **{**SPEC.__dict__, "total_amount_quote": 200, "portfolio_allocation": 0.3} ) - hot = build_config(spec, Posture("ranging", 1.0, 2.5, 0.0, 0, 0, 0.0, False, True)) - calm = build_config(spec, Posture("ranging", 1.0, 0.6, 0.0, 0, 0, 0.0, False, True)) + hot = build_config( + spec, Posture("ranging", 1.0, 2.5, 1.0, 0.0, 0, 0, 0.0, False, True) + ) + calm = build_config( + spec, Posture("ranging", 1.0, 0.6, 1.0, 0.0, 0, 0, 0.0, False, True) + ) assert hot["portfolio_allocation"] > spec.portfolio_allocation assert calm["portfolio_allocation"] < spec.portfolio_allocation assert calm["portfolio_allocation"] == pytest.approx(0.24) assert calm["portfolio_allocation"] == pytest.approx(spec.min_portfolio_allocation) # and a big book scaled up still cannot quote more than all of itself big = MarketSpec(**{**SPEC.__dict__, "portfolio_allocation": 0.5}) - assert build_config(big, Posture("ranging", 1.0, 2.5, 0.0, 0, 0, 0.0, False, True))[ - "portfolio_allocation" - ] == pytest.approx(1.0) + assert build_config( + big, Posture("ranging", 1.0, 2.5, 1.0, 0.0, 0, 0, 0.0, False, True) + )["portfolio_allocation"] == pytest.approx(1.0) def test_the_config_matches_the_experts_balanced_profile(): @@ -254,3 +262,31 @@ def test_the_config_matches_the_experts_balanced_profile(): assert cfg["global_sl_activation_from"] == "target_base" assert cfg["global_pnl_reference"] == "position" assert cfg["tick_mode"] is False + + +def test_the_exit_scales_with_the_market_and_never_goes_under_the_fee(): + """The take-profit was one number whatever the market and whatever the + posture, so five replay variants closed the same eleven round trips. It is + now three quarters of a typical bar, moved by arousal, with the fee floor + and the fly's own first level underneath it.""" + from flybrain.posture import take_profit_base_bps, take_profit_floor_bps + + # a market that moves: the range sets the exit, not the fee + assert take_profit_base_bps(1.3, 20.0) == pytest.approx(15.0) + # a market that barely moves: the fee floor holds it up + assert take_profit_base_bps(1.3, 2.0) == pytest.approx(take_profit_floor_bps(1.3)) + + wide = build_config( + SPEC, Posture("volatile", 1.0, 1.0, 2.5, 0.0, 0, 1.5, 0.0, False, True) + ) + flat = build_config(SPEC, NEUTRAL) + assert float(wide["take_profit"]) > float(flat["take_profit"]) + # SPEC's 10 bp range puts the base at 7.5 bp; 2.5x is 18.75 + assert float(wide["take_profit"]) == pytest.approx(18.75 * BPS) + + # and a fly that wants a tight exit still cannot quote one inside the fee + tight = MarketSpec(**{**SPEC.__dict__, "range_bps": 2.0}) + cfg = build_config( + tight, Posture("quiet", 0.6, 1.0, 0.6, 0.0, 0, -1.5, 0.0, False, True) + ) + assert float(cfg["take_profit"]) >= take_profit_floor(tight) From b29dc89c1c89f5ad4a49b433fb2d24e3d6416eef Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 14:57:44 -0700 Subject: [PATCH 40/48] (fix) the replay's position cap comes from the controller, not from me max_lots was a fixed 8 chosen here, and it quietly became the thing under test: widening the take-profit leaves lots open, open lots hit the cap, and the cap blocks new fills -- so lever 2's exit was measured through a limit that has nothing to do with the strategy. live filled 19 against flat-tp's 32 on the same candles, and most of that gap is the cap, not the decision. It is now what pmm_mister would actually tolerate: max_active_executors_by_ level, two levels a side, both sides -- 12 with the expert's profile. --- agents/market_making_fly/flybrain/replay.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/agents/market_making_fly/flybrain/replay.py b/agents/market_making_fly/flybrain/replay.py index 5f6d27745..a929108b7 100644 --- a/agents/market_making_fly/flybrain/replay.py +++ b/agents/market_making_fly/flybrain/replay.py @@ -238,7 +238,6 @@ def replay( deadband_bps: float = 1.0, shuffle_seed: int | None = None, neural_ms: float = 500.0, - max_lots: int = 8, ) -> ReplayResult: """Walk the candles once, exactly as the live loop walks wall time. @@ -251,6 +250,12 @@ def replay( the correspondence to its own P&L is destroyed. If that scores the same, what the synapses hold is not about this market. """ + # How many open positions the controller would tolerate, rather than a + # number picked here: two levels a side, each allowed its own concurrent + # executors. A cap set independently of the strategy quietly becomes the + # thing under test — with a fixed 8, widening the take-profit blocked new + # fills by leaving lots open, so the exit was measured through the cap. + max_lots = spec.max_active_executors_by_level * 2 * len(SIDES) baseline = Baseline() ledger = Ledger() hysteresis = Hysteresis(min_apply_interval_sec=0.0) From 464d48d1eda399bca471d55fa17e9ff46007a2a1 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 15:07:24 -0700 Subject: [PATCH 41/48] (feat) a gated trend takes a side off the book Leaning moved the quote 2.5 bp on a market whose bars run 8, which is a rounding error against the thing it was meant to prevent: the fly filled five buys and no sells into a fall and held every one of them. Twice. Past |trend_z| >= 1.5 with a gate spike, the losing side is quoted at zero size -- a path pmm_mister supports by name, skipping the level and logging it -- and the surviving side's orders double, because the controller normalizes amounts across both sides. The replay mirrors that normalization, or it would have scored one-sided quoting as half a book rather than the same book through half as many orders. A two-sided variant joins the controls with a threshold no trend reaches, so what taking a side away is worth can be read on its own. The test for this first asserted that 2.0 Hz was a mild move and 40 a strong one, against a flat baseline the decoder scores identically at 2.2 sigma whatever the magnitude. The baseline now has variance in it, which is the only condition under which any of these thresholds mean anything. --- agents/market_making_fly/flybrain/decoder.py | 18 +++++++++++- agents/market_making_fly/flybrain/posture.py | 9 ++++-- agents/market_making_fly/flybrain/replay.py | 13 +++++++-- .../market_making_fly/routines/fly_replay.py | 9 ++++-- .../skills/fly_decoder/SKILL.md | 4 +++ .../tests/test_fly_decoder.py | 28 +++++++++++++++++++ .../tests/test_fly_posture.py | 18 ++++++++++++ 7 files changed, 92 insertions(+), 7 deletions(-) diff --git a/agents/market_making_fly/flybrain/decoder.py b/agents/market_making_fly/flybrain/decoder.py index 9ebdd5a5a..72315d02c 100644 --- a/agents/market_making_fly/flybrain/decoder.py +++ b/agents/market_making_fly/flybrain/decoder.py @@ -9,7 +9,8 @@ * ``arousal_hz`` — mean rate of the descending-neuron population. Higher → tighter spreads, a larger share of the book quoted, and a wider take-profit to hold for. -* ``gate`` — DNpe017 spikes ≥ 1, required for a trending call. +* ``gate`` — DNpe017 spikes ≥ 1, required for a trending call, and for + taking a side off the book. * ``valence_hz`` — mean MBON07 rate minus mean MBON11 rate: approach minus avoidance. These are the cells the KC→MBON memory rule writes to, so this is the only channel a P&L pulse can @@ -85,6 +86,11 @@ class DecoderSettings: # A scene this far below the mushroom body's own recent drive is one the # fly effectively did not see; its z-scores are noise. z_kc_quiet: float = -1.5 + # Past this, a gated trend stops nudging the reference price and takes a + # side away. Leaning moved the quote 2.5 bp on a market whose bars run 8; + # the failure it was meant to prevent — filling one side over and over + # into a move — needs the other side gone, not discounted. + z_side: float = 1.5 shift_gain_bps: float = 1.0 max_shift_bps: float = 3.0 center_bias: bool = True @@ -109,6 +115,8 @@ def __post_init__(self): # but not zero, which would mean the channel is read and discarded. if self.z_kc_quiet >= 0: raise ValueError("z_kc_quiet must be negative") + if self.z_side < self.z_regime: + raise ValueError("z_side must be at least z_regime") for name in ("spread_gain", "size_gain", "valence_gain", "tp_gain"): value = getattr(self, name) if not math.isfinite(value) or value == 0: @@ -194,6 +202,7 @@ class Posture: valence_z: float gate: bool warm: bool # False while the baseline is still forming + side: str = "both" # "buy" or "sell" when a gated trend takes one away confident: bool = True # False when the scene never reached the mushroom body def to_dict(self) -> dict: @@ -262,6 +271,12 @@ def decode(channels: Channels, baseline: Baseline, s: DecoderSettings) -> Postur shift = max(-s.max_shift_bps, min(s.max_shift_bps, s.shift_gain_bps * trend_z)) if not gate: shift = 0.0 # no descending gate spike, no directional lean + # A trend strong enough to act on takes the other side off the book rather + # than pricing it a little worse. Quoting with the trend is the same + # direction the lean already expressed, carried to its conclusion. + side = "both" + if gate and abs(trend_z) >= s.z_side: + side = "buy" if trend_z > 0 else "sell" return Posture( regime=regime, spread_mult=round(spread_mult, 4), @@ -274,6 +289,7 @@ def decode(channels: Channels, baseline: Baseline, s: DecoderSettings) -> Postur gate=gate, warm=True, confident=confident, + side=side, ) diff --git a/agents/market_making_fly/flybrain/posture.py b/agents/market_making_fly/flybrain/posture.py index a5065dbee..9a88df0b1 100644 --- a/agents/market_making_fly/flybrain/posture.py +++ b/agents/market_making_fly/flybrain/posture.py @@ -270,8 +270,13 @@ def build_config(spec: MarketSpec, posture: Posture) -> dict: "max_base_pct": spec.max_base_pct, "buy_spreads": _fmt([b * BPS for b in buy]), "sell_spreads": _fmt([s * BPS for s in sell]), - "buy_amounts_pct": "1,1", - "sell_amounts_pct": "1,1", + # A suppressed side is quoted at zero size, which pmm_mister skips by + # name ("The amount of the level is 0. Skipping."). It also normalizes + # amounts across both sides, so the surviving side's orders double — + # the same capital through half as many quotes, which is what taking a + # side off the book should mean. + "buy_amounts_pct": "0,0" if posture.side == "sell" else "1,1", + "sell_amounts_pct": "0,0" if posture.side == "buy" else "1,1", "executor_refresh_time": refresh, "buy_cooldown_time": cooldown, "sell_cooldown_time": cooldown, diff --git a/agents/market_making_fly/flybrain/replay.py b/agents/market_making_fly/flybrain/replay.py index a929108b7..83640705c 100644 --- a/agents/market_making_fly/flybrain/replay.py +++ b/agents/market_making_fly/flybrain/replay.py @@ -95,6 +95,11 @@ def _levels(config: dict, key: str) -> list[float]: return [float(x) for x in config[key].split(",")] +def _amounts(config: dict, side: str) -> list[float]: + """The per-level amount fractions for one side; all zero when suppressed.""" + return [float(x) for x in config[f"{side}_amounts_pct"].split(",") if float(x)] + + def quote_prices(config: dict, mid: float) -> dict[str, list[float]]: """Where this config rests its orders around ``mid``.""" return { @@ -145,9 +150,12 @@ def step( notional = float(config["total_amount_quote"]) * float( config["portfolio_allocation"] ) - per_order = notional / 4 + # pmm_mister normalizes amounts across both sides, so a side quoted at zero + # size does not shrink the book — it doubles the other side's orders. + active = [s for s in SIDES if any(_amounts(config, s))] + per_order = notional / max(1, sum(len(_amounts(config, s)) for s in active)) prices = quote_prices(config, mid) - for side in SIDES: + for side in active: for price in prices[side]: if len(ledger.open_lots) >= max_lots: break @@ -201,6 +209,7 @@ def summary(self) -> dict: "mean_spread_mult": round(sum(spreads) / len(spreads), 3), "mean_size_mult": round(sum(sizes) / len(sizes), 3), "mean_tp_mult": round(sum(tps) / len(tps), 3), + "one_sided": sum(1 for p in self.postures if p.side != "both"), "regimes": regimes, } diff --git a/agents/market_making_fly/routines/fly_replay.py b/agents/market_making_fly/routines/fly_replay.py index df872d18b..45b931b7a 100644 --- a/agents/market_making_fly/routines/fly_replay.py +++ b/agents/market_making_fly/routines/fly_replay.py @@ -18,6 +18,8 @@ * ``flat-tp`` — the exit fixed where the range puts it, with the fly unable to move it. Says whether deciding how long to hold is worth anything, separately from deciding where to quote. +* ``two-sided``— a side threshold no trend reaches, so both sides stay on the + book. Says what taking a side away is worth. Each variant gets its own brain process: a network that has already learned from one variant is not a control for the next. @@ -69,6 +71,7 @@ "shuffled": {"shuffle": True}, "widen": {"spread_gain": 0.5}, "flat-tp": {"tp_gain": OFF}, + "two-sided": {"z_side": 99.0}, } @@ -82,7 +85,7 @@ class Config(BaseModel): default=1000, ge=200, le=5000, description="Candles to fetch" ) variants: str = Field( - default="live,no-memory,no-valence,shuffled,widen,flat-tp", + default="live,no-memory,no-valence,shuffled,widen,flat-tp,two-sided", description=f"Comma-separated, from: {', '.join(VARIANTS)}", ) total_amount_quote: float = Field(default=200.0) @@ -345,6 +348,7 @@ def observe(frame, stimulus, neural_ms, _pool=pool): "Spread ×": f"{s['mean_spread_mult']:.2f}", "Size ×": f"{s['mean_size_mult']:.2f}", "TP ×": f"{s['mean_tp_mult']:.2f}", + "One-sided": f"{s['one_sided']:,}", } for s in summaries ], @@ -360,6 +364,7 @@ def observe(frame, stimulus, neural_ms, _pool=pool): "Spread ×", "Size ×", "TP ×", + "One-sided", ], ) builder.plotly(_curve_figure(results)) @@ -425,7 +430,7 @@ def observe(frame, stimulus, neural_ms, _pool=pool): f"{s['variant']}: net {s['net']:+.4f}, {s['fills']} fills, " f"{s['round_trips']} round trips, {s['applies']} applies, " f"spread ×{s['mean_spread_mult']:.2f}, size ×{s['mean_size_mult']:.2f}, " - f"tp ×{s['mean_tp_mult']:.2f}" + f"tp ×{s['mean_tp_mult']:.2f}, {s['one_sided']} one-sided" ) if len(results) > 1: for other in results[1:]: diff --git a/agents/market_making_fly/skills/fly_decoder/SKILL.md b/agents/market_making_fly/skills/fly_decoder/SKILL.md index 41f0402a0..ff3bbd018 100644 --- a/agents/market_making_fly/skills/fly_decoder/SKILL.md +++ b/agents/market_making_fly/skills/fly_decoder/SKILL.md @@ -33,6 +33,10 @@ user wants to *look* at the run, `fly_status` when you need to quote figures. window 60 observations. `warm=False` for the first 10 — neutral posture. * Regime precedence: `pause` (arousal_z ≥ 2.5) > `volatile` (≥ 1) > `trending_up/down` (gate and |trend_z| ≥ 1) > `quiet` (arousal_z ≤ −1) > `ranging`. +* **Side**: a gated trend past |trend_z| ≥ 1.5 takes the other side off the book — + `buy` on a positive trend, `sell` on a negative one — by quoting it at zero size, + which pmm_mister skips. The surviving side's orders double, because the controller + normalizes amounts across both. * `spread ×` = clip(1 − 0.5·arousal_z, 0.6, 2.5) — an aroused fly quotes **tighter**. `size ×` = clip(1 + 0.5·arousal_z, 0.6, 2.5) — and quotes **more** of the book, clamped so an order never falls under the venue minimum nor the allocation over 1. diff --git a/agents/market_making_fly/tests/test_fly_decoder.py b/agents/market_making_fly/tests/test_fly_decoder.py index 52f8c417e..db0b8dc8d 100644 --- a/agents/market_making_fly/tests/test_fly_decoder.py +++ b/agents/market_making_fly/tests/test_fly_decoder.py @@ -179,3 +179,31 @@ def test_a_scene_that_never_reached_the_mushroom_body_is_not_acted_on(): assert not blind.confident ok, why = should_apply(seen, blind, None, 0.0, Hysteresis()) assert ok is False and "kenyon" in why.lower() + + +def _varied(gate: int) -> Baseline: + """A baseline with real spread in it. A flat history has zero variance, so + the decoder scores *any* departure from it at the same 2.2 sigma however + small — which made the first version of this test assert that 2.0 Hz was a + mild move and 40 a strong one when the decoder could not tell them apart.""" + b = Baseline() + for value in (-4.0, -2.0, 0.0, 2.0, 4.0): + b.push(Channels(value, 8.0, gate, 0.0, 400), S.window) + return b + + +def test_a_side_is_only_taken_away_on_a_gated_trend(): + # a strong trend with no gate spike is still both sides: the gate is what + # separates a reading from a decision + ungated = decode(Channels(20.0, 8.0, 0, 0.0, 400), _varied(0), S) + assert ungated.trend_z > S.z_side and ungated.side == "both" + + up = decode(Channels(20.0, 8.0, 1, 0.0, 400), _varied(1), S) + assert up.trend_z > S.z_side and up.side == "buy" + + down = decode(Channels(-20.0, 8.0, 1, 0.0, 400), _varied(1), S) + assert down.trend_z < -S.z_side and down.side == "sell" + + # and a trend too weak to act on leaves both sides up + mild = decode(Channels(2.0, 8.0, 1, 0.0, 400), _varied(1), S) + assert abs(mild.trend_z) < S.z_side and mild.side == "both" diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py index 540095d3d..8366e19ed 100644 --- a/agents/market_making_fly/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -290,3 +290,21 @@ def test_the_exit_scales_with_the_market_and_never_goes_under_the_fee(): tight, Posture("quiet", 0.6, 1.0, 0.6, 0.0, 0, -1.5, 0.0, False, True) ) assert float(cfg["take_profit"]) >= take_profit_floor(tight) + + +def test_a_gated_trend_takes_a_side_off_the_book(): + """Leaning moved the quote 2.5 bp on a market whose bars run 8, which is + why the fly filled five buys and no sells into a fall and held them. A + trend worth acting on removes the other side instead of discounting it.""" + up = build_config( + SPEC, + Posture("trending_up", 1.0, 1.0, 1.0, 2.0, 2.0, 0, 0.0, True, True, "buy"), + ) + assert up["sell_amounts_pct"] == "0,0" and up["buy_amounts_pct"] == "1,1" + down = build_config( + SPEC, + Posture("trending_down", 1.0, 1.0, 1.0, -2.0, -2.0, 0, 0.0, True, True, "sell"), + ) + assert down["buy_amounts_pct"] == "0,0" and down["sell_amounts_pct"] == "1,1" + # both sides is still the default, and the guard sees a valid config either way + assert build_config(SPEC, NEUTRAL)["buy_amounts_pct"] == "1,1" From 1b0e1209dbe890b7adac53235801940613523f9f Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 15:17:07 -0700 Subject: [PATCH 42/48] (fix) the fly stops deciding how long to hold, because it was not worth it Lever 2 shipped two things: an exit that scales with the market, and the fly moving it. The replay says keep the first and drop the second. On the pinned window, a fixed range-based exit closed 68 round trips to the decoded one's 35 and finished ahead -- by 0.12 on a 6.2 loss, which is nothing, but it is nothing twice, in both windows, for half the activity. That is not evidence the decode is harmful. It is the absence of any evidence it helps, at the cost of halving the round trips, which is enough not to ship it. The range-based base stays: the exit was the fee floor whether a market moved 2 bp a bar or 20, and that was simply wrong. --- agents/market_making_fly/flybrain/decoder.py | 18 +---- agents/market_making_fly/flybrain/posture.py | 11 ++- agents/market_making_fly/flybrain/replay.py | 2 - .../market_making_fly/routines/fly_brain.py | 1 - .../market_making_fly/routines/fly_replay.py | 10 +-- .../market_making_fly/routines/fly_report.py | 5 +- .../skills/fly_decoder/SKILL.md | 6 +- .../tests/test_fly_decoder.py | 12 +-- .../market_making_fly/tests/test_fly_guard.py | 2 +- .../tests/test_fly_posture.py | 74 +++++++++---------- 10 files changed, 58 insertions(+), 83 deletions(-) diff --git a/agents/market_making_fly/flybrain/decoder.py b/agents/market_making_fly/flybrain/decoder.py index 72315d02c..c4f6897b5 100644 --- a/agents/market_making_fly/flybrain/decoder.py +++ b/agents/market_making_fly/flybrain/decoder.py @@ -7,8 +7,9 @@ * ``trend_hz`` — mean DNp20 right rate minus mean left rate (stonkfly's BUY/SELL cells). Sign → which way the reference price leans. * ``arousal_hz`` — mean rate of the descending-neuron population. Higher → - tighter spreads, a larger share of the book quoted, and a - wider take-profit to hold for. + tighter spreads and a larger share of the book quoted. It + moved the take-profit too until a replay showed that cost + half the round trips and earned nothing for them. * ``gate`` — DNpe017 spikes ≥ 1, required for a trending call, and for taking a side off the book. * ``valence_hz`` — mean MBON07 rate minus mean MBON11 rate: approach minus @@ -77,12 +78,6 @@ class DecoderSettings: # arousal and added to it, so the size the fly commits carries both "the # market is active" and "this looked good last time". valence_gain: float = 0.5 - # How long to hold for. An active market both fills a quote sooner and - # travels further afterwards, so the same arousal that tightens the entry - # widens the exit: the fly asks for more of a range it can see moving. - tp_gain: float = 0.5 - tp_min: float = 0.6 - tp_max: float = 2.5 # A scene this far below the mushroom body's own recent drive is one the # fly effectively did not see; its z-scores are noise. z_kc_quiet: float = -1.5 @@ -103,7 +98,6 @@ def __post_init__(self): for lo, hi, what in ( (self.spread_min, self.spread_max, "spread"), (self.size_min, self.size_max, "size"), - (self.tp_min, self.tp_max, "tp"), ): if not 0 < lo <= 1 <= hi: raise ValueError(f"{what}_min <= 1 <= {what}_max required") @@ -117,7 +111,7 @@ def __post_init__(self): raise ValueError("z_kc_quiet must be negative") if self.z_side < self.z_regime: raise ValueError("z_side must be at least z_regime") - for name in ("spread_gain", "size_gain", "valence_gain", "tp_gain"): + for name in ("spread_gain", "size_gain", "valence_gain"): value = getattr(self, name) if not math.isfinite(value) or value == 0: raise ValueError(f"{name} must be finite and non-zero") @@ -195,7 +189,6 @@ class Posture: regime: str spread_mult: float size_mult: float - tp_mult: float shift_bps: float trend_z: float arousal_z: float @@ -217,7 +210,6 @@ def from_dict(cls, data: dict) -> "Posture": regime="ranging", spread_mult=1.0, size_mult=1.0, - tp_mult=1.0, shift_bps=0.0, trend_z=0.0, arousal_z=0.0, @@ -267,7 +259,6 @@ def decode(channels: Channels, baseline: Baseline, s: DecoderSettings) -> Postur s.size_max, max(s.size_min, 1 + s.size_gain * arousal_z + s.valence_gain * valence_z), ) - tp_mult = min(s.tp_max, max(s.tp_min, 1 + s.tp_gain * arousal_z)) shift = max(-s.max_shift_bps, min(s.max_shift_bps, s.shift_gain_bps * trend_z)) if not gate: shift = 0.0 # no descending gate spike, no directional lean @@ -281,7 +272,6 @@ def decode(channels: Channels, baseline: Baseline, s: DecoderSettings) -> Postur regime=regime, spread_mult=round(spread_mult, 4), size_mult=round(size_mult, 4), - tp_mult=round(tp_mult, 4), shift_bps=round(shift, 3), trend_z=round(trend_z, 4), arousal_z=round(arousal_z, 4), diff --git a/agents/market_making_fly/flybrain/posture.py b/agents/market_making_fly/flybrain/posture.py index 9a88df0b1..4fd4a4635 100644 --- a/agents/market_making_fly/flybrain/posture.py +++ b/agents/market_making_fly/flybrain/posture.py @@ -246,15 +246,14 @@ def build_config(spec: MarketSpec, posture: Posture) -> dict: shift = max(-levels[0] / 2, min(levels[0] / 2, posture.shift_bps)) buy = [max(spec.min_spread_bps, lvl - shift) for lvl in levels] sell = [max(spec.min_spread_bps, lvl + shift) for lvl in levels] - # The fly asks for more of the range when it is aroused, but never less - # than a round trip costs, and never less than its own first level — a - # take-profit inside the spread it quotes would close for nothing. + # The exit scales with the market, never below what a round trip costs and + # never below the fly's own first level — a take-profit inside the spread + # it quotes would close for nothing. The fly does not move it: letting + # arousal widen it halved the round trips and earned nothing for them. take_profit = max( take_profit_floor(spec), min(buy[0], sell[0]) * BPS, - take_profit_base_bps(spec.maker_fee_bps, spec.range_bps) - * posture.tp_mult - * BPS, + take_profit_base_bps(spec.maker_fee_bps, spec.range_bps) * BPS, ) refresh, cooldown = TIMING[posture.regime] config = { diff --git a/agents/market_making_fly/flybrain/replay.py b/agents/market_making_fly/flybrain/replay.py index 83640705c..884ecf772 100644 --- a/agents/market_making_fly/flybrain/replay.py +++ b/agents/market_making_fly/flybrain/replay.py @@ -193,7 +193,6 @@ def summary(self) -> dict: regimes[p.regime] = regimes.get(p.regime, 0) + 1 spreads = [p.spread_mult for p in self.postures] or [0.0] sizes = [p.size_mult for p in self.postures] or [0.0] - tps = [p.tp_mult for p in self.postures] or [0.0] return { "variant": self.variant, "ticks": self.ticks, @@ -208,7 +207,6 @@ def summary(self) -> dict: "unconfident": self.unconfident, "mean_spread_mult": round(sum(spreads) / len(spreads), 3), "mean_size_mult": round(sum(sizes) / len(sizes), 3), - "mean_tp_mult": round(sum(tps) / len(tps), 3), "one_sided": sum(1 for p in self.postures if p.side != "both"), "regimes": regimes, } diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index a7d44930c..687347667 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -579,7 +579,6 @@ async def pace() -> None: b.kpi("Spread ×", f"{posture.spread_mult:.2f}") b.kpi("Size ×", f"{posture.size_mult:.2f}") b.kpi("Valence z", f"{posture.valence_z:+.2f}") - b.kpi("TP ×", f"{posture.tp_mult:.2f}") b.kpi("Lean", f"{posture.shift_bps:+.2f} bp") b.kpi("Trend z", f"{posture.trend_z:+.2f}") b.kpi("Arousal z", f"{posture.arousal_z:+.2f}") diff --git a/agents/market_making_fly/routines/fly_replay.py b/agents/market_making_fly/routines/fly_replay.py index 45b931b7a..34f65f4a3 100644 --- a/agents/market_making_fly/routines/fly_replay.py +++ b/agents/market_making_fly/routines/fly_replay.py @@ -15,9 +15,6 @@ * ``shuffled`` — reinforcement of the same frequency and magnitude, with the sign randomised. The control the caveats have always demanded. * ``widen`` — the old arousal direction, for the A/B that motivated the flip. -* ``flat-tp`` — the exit fixed where the range puts it, with the fly unable to - move it. Says whether deciding how long to hold is worth - anything, separately from deciding where to quote. * ``two-sided``— a side threshold no trend reaches, so both sides stay on the book. Says what taking a side away is worth. @@ -70,7 +67,6 @@ "no-valence": {"valence_gain": OFF}, "shuffled": {"shuffle": True}, "widen": {"spread_gain": 0.5}, - "flat-tp": {"tp_gain": OFF}, "two-sided": {"z_side": 99.0}, } @@ -85,7 +81,7 @@ class Config(BaseModel): default=1000, ge=200, le=5000, description="Candles to fetch" ) variants: str = Field( - default="live,no-memory,no-valence,shuffled,widen,flat-tp,two-sided", + default="live,no-memory,no-valence,shuffled,widen,two-sided", description=f"Comma-separated, from: {', '.join(VARIANTS)}", ) total_amount_quote: float = Field(default=200.0) @@ -347,7 +343,6 @@ def observe(frame, stimulus, neural_ms, _pool=pool): "Unconfident": f"{s['unconfident']:,}", "Spread ×": f"{s['mean_spread_mult']:.2f}", "Size ×": f"{s['mean_size_mult']:.2f}", - "TP ×": f"{s['mean_tp_mult']:.2f}", "One-sided": f"{s['one_sided']:,}", } for s in summaries @@ -363,7 +358,6 @@ def observe(frame, stimulus, neural_ms, _pool=pool): "Unconfident", "Spread ×", "Size ×", - "TP ×", "One-sided", ], ) @@ -430,7 +424,7 @@ def observe(frame, stimulus, neural_ms, _pool=pool): f"{s['variant']}: net {s['net']:+.4f}, {s['fills']} fills, " f"{s['round_trips']} round trips, {s['applies']} applies, " f"spread ×{s['mean_spread_mult']:.2f}, size ×{s['mean_size_mult']:.2f}, " - f"tp ×{s['mean_tp_mult']:.2f}, {s['one_sided']} one-sided" + f"{s['one_sided']} one-sided" ) if len(results) > 1: for other in results[1:]: diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index c6095b8cd..77f0a87f4 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -390,10 +390,9 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: # One card, not two: both multipliers are read off the same channel, and # the stack's height is matched to the brain beside it at six cards. builder.kpi( - "Spread / size / TP ×", + "Spread / size ×", f"{_fmt(last_posture.get('spread_mult'))} / " - f"{_fmt(last_posture.get('size_mult'))} / " - f"{_fmt(last_posture.get('tp_mult'))}", + f"{_fmt(last_posture.get('size_mult'))}", width=CARDS, ) builder.kpi( diff --git a/agents/market_making_fly/skills/fly_decoder/SKILL.md b/agents/market_making_fly/skills/fly_decoder/SKILL.md index ff3bbd018..d00761195 100644 --- a/agents/market_making_fly/skills/fly_decoder/SKILL.md +++ b/agents/market_making_fly/skills/fly_decoder/SKILL.md @@ -41,9 +41,9 @@ user wants to *look* at the run, `fly_status` when you need to quote figures. `size ×` = clip(1 + 0.5·arousal_z, 0.6, 2.5) — and quotes **more** of the book, clamped so an order never falls under the venue minimum nor the allocation over 1. `size ×` also carries `+0.5·valence_z`, so what the fly has learned about scenes - like this one moves the capital it commits. `tp ×` = clip(1 + 0.5·arousal_z, 0.6, - 2.5) on a base of 0.75 × the market's median candle range — an active market both - fills sooner and travels further, so the fly holds for more of it. + like this one moves the capital it commits. The exit is 0.75 × the market's median + candle range, floored by the round-trip fee and by the first level; the fly does + not move it — letting arousal widen it halved the round trips and earned nothing. The sign on each is a choice, not a finding: arousal is a population rate against its own average and nothing ties it to volatility. `lean` = clip(trend_z, ±3 bp), 0 without a gate spike, and capped at half the first spread level when mapped. diff --git a/agents/market_making_fly/tests/test_fly_decoder.py b/agents/market_making_fly/tests/test_fly_decoder.py index db0b8dc8d..ef040e6f9 100644 --- a/agents/market_making_fly/tests/test_fly_decoder.py +++ b/agents/market_making_fly/tests/test_fly_decoder.py @@ -112,7 +112,7 @@ def test_baseline_window_and_roundtrip(): def test_posture_roundtrip(): - p = Posture("quiet", 0.8, 1.0, 1.0, -1.0, -0.2, -1.3, 0.0, True, True) + p = Posture("quiet", 0.8, 1.0, -1.0, -0.2, -1.3, 0.0, True, True) assert Posture.from_dict(p.to_dict()) == p @@ -132,16 +132,16 @@ def test_settings_validation(): def test_hysteresis(): h = Hysteresis(min_apply_interval_sec=300) - base = Posture("ranging", 1.0, 1.0, 1.0, 0.0, 0, 0, 0.0, False, True) + base = Posture("ranging", 1.0, 1.0, 0.0, 0, 0, 0.0, False, True) assert should_apply(None, base, None, 1000, h)[0] - same = Posture("ranging", 1.05, 1.0, 1.0, 0.2, 0, 0, 0.0, False, True) + same = Posture("ranging", 1.05, 1.0, 0.2, 0, 0, 0.0, False, True) assert not should_apply(base, same, 0, 1000, h)[0] - regime = Posture("volatile", 1.0, 1.0, 1.0, 0.0, 0, 1.2, 0.0, False, True) + regime = Posture("volatile", 1.0, 1.0, 0.0, 0, 1.2, 0.0, False, True) assert should_apply(base, regime, 0, 1000, h)[0] assert not should_apply(base, regime, 900, 1000, h)[0] # cooldown - wider = Posture("ranging", 1.2, 1.0, 1.0, 0.0, 0, 0, 0.0, False, True) + wider = Posture("ranging", 1.2, 1.0, 0.0, 0, 0, 0.0, False, True) assert should_apply(base, wider, 0, 1000, h)[0] - lean = Posture("ranging", 1.0, 1.0, 1.0, 0.6, 0, 0, 0.0, True, True) + lean = Posture("ranging", 1.0, 1.0, 0.6, 0, 0, 0.0, True, True) assert should_apply(base, lean, 0, 1000, h)[0] diff --git a/agents/market_making_fly/tests/test_fly_guard.py b/agents/market_making_fly/tests/test_fly_guard.py index 7c43ae143..c07682deb 100644 --- a/agents/market_making_fly/tests/test_fly_guard.py +++ b/agents/market_making_fly/tests/test_fly_guard.py @@ -165,7 +165,7 @@ def test_a_config_sitting_exactly_on_the_floor_is_not_vetoed(): portfolio_allocation=0.3, maker_fee_bps=1.3, ) - leaned = Posture("trending_up", 1.14, 1.0, 1.0, 1.27, 2.0, 0.3, 0.0, True, True) + leaned = Posture("trending_up", 1.14, 1.0, 1.27, 2.0, 0.3, 0.0, True, True) config = build_config(spec, leaned) assert min(float(x) for x in config["buy_spreads"].split(",")) == pytest.approx( spec.min_spread_bps * 1e-4 diff --git a/agents/market_making_fly/tests/test_fly_posture.py b/agents/market_making_fly/tests/test_fly_posture.py index 8366e19ed..a5ee4f812 100644 --- a/agents/market_making_fly/tests/test_fly_posture.py +++ b/agents/market_making_fly/tests/test_fly_posture.py @@ -54,14 +54,14 @@ def test_take_profit_floor_beats_fees_and_spread(): def test_volatile_widens_quiet_tightens_with_floor(): wide = build_config( - SPEC, Posture("volatile", 2.0, 1.0, 1.0, 0.0, 0, 1.5, 0.0, False, True) + SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, 0.0, False, True) ) assert _spreads(wide["buy_spreads"])[0] == pytest.approx(10 * BPS) assert (wide["executor_refresh_time"], wide["buy_cooldown_time"]) == TIMING[ "volatile" ] tight = build_config( - SPEC, Posture("quiet", 0.6, 1.0, 1.0, 0.0, 0, -1.5, 0.0, False, True) + SPEC, Posture("quiet", 0.6, 1.0, 0.0, 0, -1.5, 0.0, False, True) ) # 5 bp × 0.6 = 3 bp, well inside a typical bar and clear of the 1.3 bp fee assert _spreads(tight["buy_spreads"])[0] == pytest.approx(3 * BPS) @@ -72,14 +72,14 @@ def test_volatile_widens_quiet_tightens_with_floor(): def test_lean_is_asymmetric_and_capped(): up = build_config( - SPEC, Posture("trending_up", 1.0, 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) + SPEC, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) ) buy, sell = _spreads(up["buy_spreads"]), _spreads(up["sell_spreads"]) # lean capped at half of level 1 (5 bp → 2.5 bp), clear of the fee floor assert buy[0] == pytest.approx(2.5 * BPS) and sell[0] == pytest.approx(7.5 * BPS) assert buy[1] == pytest.approx(5 * BPS) and sell[1] == pytest.approx(10 * BPS) down = build_config( - SPEC, Posture("trending_down", 1.0, 1.0, 1.0, -3.0, -2.0, 0, 0.0, True, True) + SPEC, Posture("trending_down", 1.0, 1.0, -3.0, -2.0, 0, 0.0, True, True) ) assert _spreads(down["sell_spreads"])[0] == pytest.approx(2.5 * BPS) assert _spreads(down["buy_spreads"])[0] == pytest.approx(7.5 * BPS) @@ -99,15 +99,13 @@ def test_the_spread_floor_is_the_market_own_fee(): assert dear.min_spread_bps == pytest.approx(7.5) # binance spot # a lean that would quote inside the fee is pushed back out to it leaned = build_config( - dear, Posture("trending_up", 1.0, 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) + dear, Posture("trending_up", 1.0, 1.0, 3.0, 2.0, 0, 0.0, True, True) ) assert min(_spreads(leaned["buy_spreads"])) == pytest.approx(7.5 * BPS) def test_pause_sets_kill_switch(): - cfg = build_config( - SPEC, Posture("pause", 2.5, 1.0, 1.0, 0.0, 0, 3.0, 0.0, False, True) - ) + cfg = build_config(SPEC, Posture("pause", 2.5, 1.0, 0.0, 0, 3.0, 0.0, False, True)) assert cfg["manual_kill_switch"] is True @@ -116,7 +114,7 @@ def test_every_spread_respects_min(): for mult in (0.6, 1.0, 2.5): for shift in (-3.0, 0.0, 3.0): cfg = build_config( - SPEC, Posture(regime, mult, 1.0, 1.0, shift, 0, 0, 0.0, True, True) + SPEC, Posture(regime, mult, 1.0, shift, 0, 0, 0.0, True, True) ) for key in ("buy_spreads", "sell_spreads"): assert min(_spreads(cfg[key])) >= SPEC.min_spread_bps * BPS - 1e-12 @@ -173,9 +171,7 @@ def test_the_fee_floor_follows_the_venue(): def test_config_diff(): a = build_config(SPEC, NEUTRAL) - b = build_config( - SPEC, Posture("volatile", 2.0, 1.0, 1.0, 0.0, 0, 1.5, 0.0, False, True) - ) + b = build_config(SPEC, Posture("volatile", 2.0, 1.0, 0.0, 0, 1.5, 0.0, False, True)) diff = config_diff(a, b) assert "buy_spreads" in diff and "trading_pair" not in diff assert config_diff(None, a) == a @@ -228,21 +224,17 @@ def test_size_follows_arousal_and_stays_inside_both_limits(): spec = MarketSpec( **{**SPEC.__dict__, "total_amount_quote": 200, "portfolio_allocation": 0.3} ) - hot = build_config( - spec, Posture("ranging", 1.0, 2.5, 1.0, 0.0, 0, 0, 0.0, False, True) - ) - calm = build_config( - spec, Posture("ranging", 1.0, 0.6, 1.0, 0.0, 0, 0, 0.0, False, True) - ) + hot = build_config(spec, Posture("ranging", 1.0, 2.5, 0.0, 0, 0, 0.0, False, True)) + calm = build_config(spec, Posture("ranging", 1.0, 0.6, 0.0, 0, 0, 0.0, False, True)) assert hot["portfolio_allocation"] > spec.portfolio_allocation assert calm["portfolio_allocation"] < spec.portfolio_allocation assert calm["portfolio_allocation"] == pytest.approx(0.24) assert calm["portfolio_allocation"] == pytest.approx(spec.min_portfolio_allocation) # and a big book scaled up still cannot quote more than all of itself big = MarketSpec(**{**SPEC.__dict__, "portfolio_allocation": 0.5}) - assert build_config( - big, Posture("ranging", 1.0, 2.5, 1.0, 0.0, 0, 0, 0.0, False, True) - )["portfolio_allocation"] == pytest.approx(1.0) + assert build_config(big, Posture("ranging", 1.0, 2.5, 0.0, 0, 0, 0.0, False, True))[ + "portfolio_allocation" + ] == pytest.approx(1.0) def test_the_config_matches_the_experts_balanced_profile(): @@ -265,10 +257,14 @@ def test_the_config_matches_the_experts_balanced_profile(): def test_the_exit_scales_with_the_market_and_never_goes_under_the_fee(): - """The take-profit was one number whatever the market and whatever the - posture, so five replay variants closed the same eleven round trips. It is - now three quarters of a typical bar, moved by arousal, with the fee floor - and the fly's own first level underneath it.""" + """The take-profit was the fee floor whether a market moved 2 bp a bar or + 20, so five replay variants closed the same eleven round trips. It is now + three quarters of a typical bar, with the fee floor and the fly's own first + level underneath it. + + The fly does not move it. Letting arousal widen it halved the round trips — + 35 against a fixed exit's 68 on the same candles — and earned nothing for + them, so that half of the change was reverted and this is the half kept.""" from flybrain.posture import take_profit_base_bps, take_profit_floor_bps # a market that moves: the range sets the exit, not the fee @@ -276,19 +272,19 @@ def test_the_exit_scales_with_the_market_and_never_goes_under_the_fee(): # a market that barely moves: the fee floor holds it up assert take_profit_base_bps(1.3, 2.0) == pytest.approx(take_profit_floor_bps(1.3)) - wide = build_config( - SPEC, Posture("volatile", 1.0, 1.0, 2.5, 0.0, 0, 1.5, 0.0, False, True) - ) - flat = build_config(SPEC, NEUTRAL) - assert float(wide["take_profit"]) > float(flat["take_profit"]) - # SPEC's 10 bp range puts the base at 7.5 bp; 2.5x is 18.75 - assert float(wide["take_profit"]) == pytest.approx(18.75 * BPS) - - # and a fly that wants a tight exit still cannot quote one inside the fee + # SPEC's 10 bp range puts the exit at 7.5 bp, whatever the posture says + for posture in ( + NEUTRAL, + Posture("volatile", 1.0, 1.0, 0.0, 0, 1.5, 0.0, False, True), + Posture("quiet", 0.6, 1.0, 0.0, 0, -1.5, 0.0, False, True), + ): + assert float(build_config(SPEC, posture)["take_profit"]) == pytest.approx( + 7.5 * BPS + ) + + # and on a market that barely moves, the fee floor holds the exit up tight = MarketSpec(**{**SPEC.__dict__, "range_bps": 2.0}) - cfg = build_config( - tight, Posture("quiet", 0.6, 1.0, 0.6, 0.0, 0, -1.5, 0.0, False, True) - ) + cfg = build_config(tight, NEUTRAL) assert float(cfg["take_profit"]) >= take_profit_floor(tight) @@ -298,12 +294,12 @@ def test_a_gated_trend_takes_a_side_off_the_book(): trend worth acting on removes the other side instead of discounting it.""" up = build_config( SPEC, - Posture("trending_up", 1.0, 1.0, 1.0, 2.0, 2.0, 0, 0.0, True, True, "buy"), + Posture("trending_up", 1.0, 1.0, 2.0, 2.0, 0, 0.0, True, True, "buy"), ) assert up["sell_amounts_pct"] == "0,0" and up["buy_amounts_pct"] == "1,1" down = build_config( SPEC, - Posture("trending_down", 1.0, 1.0, 1.0, -2.0, -2.0, 0, 0.0, True, True, "sell"), + Posture("trending_down", 1.0, 1.0, -2.0, -2.0, 0, 0.0, True, True, "sell"), ) assert down["buy_amounts_pct"] == "0,0" and down["sell_amounts_pct"] == "1,1" # both sides is still the default, and the guard sees a valid config either way From 7613a110ddb522b1a961aa1f0877042fa5ee09aa Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 15:32:36 -0700 Subject: [PATCH 43/48] (docs) what the replay controls actually established The design has said since day one that it had no control for its learning claim. It has one now, and the answer is that real reinforcement and randomly-signed reinforcement produce strategies this instrument cannot tell apart: t = +0.24 over 327 ticks. Freezing the memory rule outright does not separate them either. Recorded with the two measurement errors found on the way, because both inflated earlier results: a t-test run on cumulative curves instead of their increments, and a position cap chosen in the harness that bound before the strategy did. --- docs/market_making_fly_design.md | 44 ++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/docs/market_making_fly_design.md b/docs/market_making_fly_design.md index d23c0900b..41fd2a0a9 100644 --- a/docs/market_making_fly_design.md +++ b/docs/market_making_fly_design.md @@ -503,6 +503,50 @@ Copied in spirit from stonkfly's `docs/model.md`, because the same limits hold: * No profitable learning is demonstrated by anything in this design. The guard and the controller's stop-loss are what bound the loss, not the fly. +### What the replay controls established, 2026-09-13 + +`fly_replay` walks a pinned candle series with the same brain, decoder and +geometry, changing one setting per variant. On 327 ticks of XYZ:DRAM-USD at +5 m, with the position cap taken from the controller rather than chosen: + +| against `live` | mean per-tick | final gap | t | +|---|---|---|---| +| shuffled reinforcement | +0.00013 | +0.04 | **+0.24** | +| frozen plasticity | +0.00165 | +0.54 | +1.06 | +| valence disconnected | +0.00307 | +1.00 | +1.20 | + +**The fly with real P&L reinforcement is not distinguishable from the fly whose +reinforcement sign is random.** That is the control this design has always said +it lacked, and it now exists and does not separate them. Nor does freezing the +memory rule entirely. One market, one window, and a fill model that assumes a +touched price was ours — so this is not proof of a null, but it is the first +evidence on the question and it points at one. + +Two measurement lessons are worth more than the numbers: + +* An earlier statistic ran on the equity curves rather than their increments. + A curve is cumulative, so it returned |t| of 27 to 63 for variants a few + percent apart — a measure of when two runs diverged, not of whether they earn + differently. +* The replay's open-position cap was a number chosen in the harness. It bound + before the strategy did, and manufactured |t| of 1.5 to 2.0 across four runs + that collapsed to 0.1–1.0 once the cap came from `pmm_mister`'s own limit. + Every "suggestive" result before that fix should be read as withdrawn. + +### What the levers were worth + +* **Quote levels off the bar range, not the touch** (kept). Level 1 sat at half + the observed spread — 2 bp where a typical bar ranged 9.9 — so every + multiplier the fly could express stayed inside what a normal bar covers, and + five variants returned identical fills. At half the range, `widen` fills 18 + where `live` fills 38. +* **An exit that scales with the market** (kept) — it was the fee floor whether + a market moved 2 bp a bar or 20. +* **The fly moving that exit** (reverted). Halved the round trips, 35 against + 68, and earned nothing for them in either window. +* **A gated trend taking a side off the book** (kept, unproven). Fired on 13 of + 327 ticks and finished +0.22 ahead of the two-sided control at t = +0.67. + ## 16. Decisions Settled 2026-09-12: 1 vendor, 2 deterministic apply, 3 descending neurons, From b76ab35f165811dea070752cd27cc7b77f9607b7 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 19:38:09 -0700 Subject: [PATCH 44/48] (refactor) the fly's design doc is the agent's README, and pyarrow is an extra Three bits of tidying before review. The design doc moves from docs/ into the agent as README.md. Everything else about this agent lives in agents/market_making_fly; the document describing it should not be the one file somewhere else, and a reader who opens the directory should find the reference in it. Retitled to introduce the agent rather than a dated design, with the install as section 0. pyarrow leaves Condor's core dependencies for an optional `fly` extra. It is 122 MB, nothing else in Condor reads Arrow, and every import of it in the agent is already inside a function -- so a Condor installed without it is a normal Condor with this one agent unavailable. The two entry points that reach a feather file now check first and raise with `uv sync --extra fly`, because an ImportError from three frames inside a vendored connectome loader tells an operator nothing about what to do next. The strategy is renamed fly_hip3_operator -> mm_operator. It stopped being HIP-3-specific when the agent generalized to any CLOB venue, and its own text never mentioned HIP-3 -- only the directory name did. --- agents/market_making_fly/AGENT.md | 4 +-- .../market_making_fly/README.md | 30 +++++++++++++++---- agents/market_making_fly/flybrain/__init__.py | 2 +- agents/market_making_fly/flybrain/cloud.py | 2 ++ agents/market_making_fly/flybrain/deps.py | 27 +++++++++++++++++ .../market_making_fly/routines/fly_brain.py | 2 +- .../market_making_fly/routines/fly_setup.py | 5 ++++ .../strategy.md | 0 .../market_making_fly/tests/test_fly_venue.py | 25 ++++++++++++++++ pyproject.toml | 11 +++++-- uv.lock | 9 ++++-- 11 files changed, 103 insertions(+), 14 deletions(-) rename docs/market_making_fly_design.md => agents/market_making_fly/README.md (96%) create mode 100644 agents/market_making_fly/flybrain/deps.py rename agents/market_making_fly/strategies/{fly_hip3_operator => mm_operator}/strategy.md (100%) diff --git a/agents/market_making_fly/AGENT.md b/agents/market_making_fly/AGENT.md index 824d57b47..a27f14c2d 100644 --- a/agents/market_making_fly/AGENT.md +++ b/agents/market_making_fly/AGENT.md @@ -86,7 +86,7 @@ manage_skill(action="read", name="fly_mm_deploy") ## Routines | Routine | Use | |---|---| -| `fly_setup` | `action=prepare` downloads and compiles the connectome into this agent's home (once per install); `verify`; `bench` | +| `fly_setup` | `action=prepare` downloads and compiles the connectome into this agent's home (once per install, needs `uv sync --extra fly`); `verify`; `bench` | | `mm_market_scanner` | Rank any venue's markets for market making: spread against that venue's round-trip fee, book depth, drift. Works on HIP-3 too | | `fly_chart` | Render the exact frame for a pair (what the fly sees) | | `fly_brain` | The loop (continuous). `mode=shadow|live`, `pairs`, `picked_ranges_bps`, `run_name` | @@ -141,7 +141,7 @@ enforced in code rather than left to judgment: posture; synapses also change from endogenous activity - No profitable learning has been demonstrated; the guard and the controller's stop loss bound the loss, not the fly -- Full detail: `docs/market_making_fly_design.md` +- Full detail: `agents/market_making_fly/README.md` ## Memory & Skills Check `manage_memory` and `manage_skill` before answering; update them when the user diff --git a/docs/market_making_fly_design.md b/agents/market_making_fly/README.md similarity index 96% rename from docs/market_making_fly_design.md rename to agents/market_making_fly/README.md index 41fd2a0a9..1e69e0cc8 100644 --- a/docs/market_making_fly_design.md +++ b/agents/market_making_fly/README.md @@ -1,7 +1,25 @@ -# Market Making Fly — implementation design +# Market Making Fly -Status: **implemented (see §18); this document is the reference.** -Date: 2026-09-12 +A Condor agent whose market-making discretion — regime, spread width, how much +to commit, which way to lean — is decoded from a simulation of the MaleCNS v1.0 +fly connectome watching a chart, with the bots' P&L fed back as dopamine. + +Status: **implemented**. This file is the agent's reference: the design, the +decisions behind it, and §15 for what has and has not been demonstrated. +Written 2026-09-12, evidence added 2026-09-13. + +## 0. Install + +``` +uv sync --extra fly # pyarrow, for the MaleCNS feather files +manage_routines(action="run", name="fly_setup", config={"action": "prepare"}) +``` + +`pyarrow` is 122 MB and nothing else in Condor reads Arrow, so it is an extra +rather than a core dependency — a Condor installed without it is a normal +Condor with this one agent unavailable, and the two entry points that need it +say so with the command rather than an ImportError. `prepare` downloads ~1.1 GB +of connectome and compiles the kernel; it is a once-per-install step. ## 1. Summary @@ -369,14 +387,14 @@ agents/market_making_fly/ fly_decoder/SKILL.md # how to read fly_status and the decoder pmm_config_playbook/ # copied capital_allocation/ # copied - strategies/fly_hip3_operator/strategy.md # thin loop: keep bot + fly alive, surface halts, rotate when flat + strategies/mm_operator/strategy.md # thin loop: keep bot + fly alive, surface halts, rotate when flat agents/market_making_fly/tests/ # conftest puts the agent dir on sys.path test_fly_chart.py test_fly_decoder.py test_fly_posture.py test_fly_guard.py test_fly_reinforcement.py test_fly_full_graph.py # opt-in, CONDOR_FLY_FULL_TEST=1, needs prepared data -docs/market_making_fly_design.md # this file, kept as the reference +README.md # this file: design, decisions, and what is claimed ``` Run state per fly instance: @@ -558,7 +576,7 @@ below for the record. 1. **Vendor stonkfly's neural package into `agents/market_making_fly/flybrain/neural/` (recommended)** vs `pip install git+…stonkfly`. Installing pulls `coinbase-agentkit` and `coinbase-advanced-py` into Condor for no use; vendoring adds ~1,400 lines + - the kernel + `pyarrow` as a new dependency. + the kernel + `pyarrow` as a new optional dependency (extra `fly`). 2. **Deterministic apply (recommended)** — the `fly_brain` routine applies the config itself and the LLM only operates — vs the LLM loop reading the posture and applying it each tick. The second puts an LLM back between the diff --git a/agents/market_making_fly/flybrain/__init__.py b/agents/market_making_fly/flybrain/__init__.py index 967781230..b08bd6771 100644 --- a/agents/market_making_fly/flybrain/__init__.py +++ b/agents/market_making_fly/flybrain/__init__.py @@ -2,5 +2,5 @@ The neural substrate (``flybrain.neural``) is vendored from stonkfly; the chart, decoder, posture mapping, guard, reinforcement and worker are Condor's. -See ``docs/market_making_fly_design.md``. +See ``agents/market_making_fly/README.md``. """ diff --git a/agents/market_making_fly/flybrain/cloud.py b/agents/market_making_fly/flybrain/cloud.py index 0a7005151..d320494a3 100644 --- a/agents/market_making_fly/flybrain/cloud.py +++ b/agents/market_making_fly/flybrain/cloud.py @@ -16,6 +16,7 @@ from __future__ import annotations import numpy as np +from flybrain.deps import require_pyarrow from flybrain.neural.common import DATA, GRAPH CLOUD = DATA / "soma_cloud.npz" @@ -84,6 +85,7 @@ def _group_of(ids: np.ndarray, superclass: np.ndarray) -> np.ndarray: def build(max_points: int = DEFAULT_POINTS) -> dict: """Subsample the somata and cache the result. Idempotent.""" + require_pyarrow() with np.load(GRAPH) as graph: ids = graph["ids"] superclass = graph["superclass"].astype(str) diff --git a/agents/market_making_fly/flybrain/deps.py b/agents/market_making_fly/flybrain/deps.py new file mode 100644 index 000000000..6bf45ffe1 --- /dev/null +++ b/agents/market_making_fly/flybrain/deps.py @@ -0,0 +1,27 @@ +"""What the fly needs that Condor core does not. + +The MaleCNS release ships as Arrow feather files, so reading it needs pyarrow — +122 MB that no other part of Condor uses. It is an optional extra rather than a +core dependency, which means a Condor installed without it is a normal Condor +and only the fly is unavailable. Saying so plainly is the whole point of this +module: an ImportError from inside a vendored connectome loader tells an +operator nothing about what to do next. +""" + +from __future__ import annotations + +INSTALL = "uv sync --extra fly" + + +def require_pyarrow() -> None: + """Raise with the command to run, rather than an ImportError three frames + deep in a feather reader.""" + try: + import pyarrow # noqa: F401 + except ImportError as missing: # pragma: no cover - depends on the install + raise RuntimeError( + "Market Making Fly reads the MaleCNS connectome from Arrow feather " + f"files, which needs pyarrow. Install the extra with `{INSTALL}` " + "and run this again. Condor's other agents do not need it, which is " + "why it is not a core dependency." + ) from missing diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index 687347667..29d627a09 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -678,7 +678,7 @@ async def pace() -> None: "_The regime, spread multiplier and lean are decoded from spike counts " "of identified cells; the mapping is engineered and unvalidated. Dopamine " "pulses report P&L change, not credit for the last posture. See " - "docs/market_making_fly_design.md §15._" + "agents/market_making_fly/README.md §15._" ) await report.update() diff --git a/agents/market_making_fly/routines/fly_setup.py b/agents/market_making_fly/routines/fly_setup.py index 0b5f873ef..565499477 100644 --- a/agents/market_making_fly/routines/fly_setup.py +++ b/agents/market_making_fly/routines/fly_setup.py @@ -82,8 +82,13 @@ def _bench(observations: int, neural_ms: float) -> dict: async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: + from flybrain.deps import require_pyarrow from flybrain.neural.common import DATA + # Every path from here reads a feather file. Check once, at the front, so + # the answer is a command to run rather than an ImportError from inside the + # vendored loader after a 1.1 GB download. + require_pyarrow() if config.action not in ("prepare", "verify", "bench"): raise ValueError("action must be prepare, verify or bench") loop = asyncio.get_running_loop() diff --git a/agents/market_making_fly/strategies/fly_hip3_operator/strategy.md b/agents/market_making_fly/strategies/mm_operator/strategy.md similarity index 100% rename from agents/market_making_fly/strategies/fly_hip3_operator/strategy.md rename to agents/market_making_fly/strategies/mm_operator/strategy.md diff --git a/agents/market_making_fly/tests/test_fly_venue.py b/agents/market_making_fly/tests/test_fly_venue.py index acf454538..474e74f60 100644 --- a/agents/market_making_fly/tests/test_fly_venue.py +++ b/agents/market_making_fly/tests/test_fly_venue.py @@ -92,3 +92,28 @@ def test_an_unlisted_hip3_market_is_refused_not_guessed(): _seed_hyperliquid_fees() with pytest.raises(ValueError, match="not listed"): asyncio.run(venue.hyperliquid_maker_fee_bps("XYZ:NOTREAL-USD", "perp")) + + +def test_the_fly_says_what_to_install_rather_than_failing_deep(): + """pyarrow is 122 MB that only this agent uses, so it is an extra. An + operator who skipped it should get the command, not an ImportError three + frames inside a vendored connectome loader.""" + import builtins + + from flybrain.deps import INSTALL, require_pyarrow + + real_import = builtins.__import__ + + def without_pyarrow(name, *args, **kwargs): + if name == "pyarrow" or name.startswith("pyarrow."): + raise ImportError("No module named 'pyarrow'") + return real_import(name, *args, **kwargs) + + builtins.__import__ = without_pyarrow + try: + with pytest.raises(RuntimeError, match=INSTALL): + require_pyarrow() + finally: + builtins.__import__ = real_import + + require_pyarrow() # installed here, so it must stay silent diff --git a/pyproject.toml b/pyproject.toml index f478e26bf..b56e6c5f4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,8 +19,6 @@ dependencies = [ # It pins numba==0.61.2, which caps numpy at <2.3 — that is also the numpy # the API container runs (2.2.6), so the two environments stay aligned. "pandas-ta>=0.4.71b", - # Fly connectome (condor/fly): MaleCNS feather files - "pyarrow", "geckoterminal-py", "mcp", "fastapi", @@ -41,6 +39,15 @@ dependencies = [ "pywebpush>=2.5.0", ] +[project.optional-dependencies] +# Market Making Fly reads the MaleCNS release from Arrow feather files. pyarrow +# is 122 MB installed and nothing else in Condor touches it, so it is an extra +# rather than a core dependency: `uv sync --extra fly`. Every import of it in +# the agent is inside a function, and the two entry points that reach one check +# first and say this — so a Condor without the extra runs normally and the fly +# refuses to start with an instruction rather than an ImportError. +fly = ["pyarrow"] + [dependency-groups] # PEP 735 group: this is what ``uv sync --dev`` installs. Declared as an # optional-dependencies extra it was invisible to ``--dev``, so CI ran diff --git a/uv.lock b/uv.lock index 4e3882dce..69e8a6d75 100644 --- a/uv.lock +++ b/uv.lock @@ -614,7 +614,6 @@ dependencies = [ { name = "pandas", version = "3.0.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.14'" }, { name = "pandas-ta" }, { name = "plotly" }, - { name = "pyarrow" }, { name = "pydantic-ai" }, { name = "python-dotenv" }, { name = "python-jose", extra = ["cryptography"] }, @@ -627,6 +626,11 @@ dependencies = [ { name = "yfinance" }, ] +[package.optional-dependencies] +fly = [ + { name = "pyarrow" }, +] + [package.dev-dependencies] dev = [ { name = "black" }, @@ -652,7 +656,7 @@ requires-dist = [ { name = "pandas" }, { name = "pandas-ta", specifier = ">=0.4.71b0" }, { name = "plotly" }, - { name = "pyarrow" }, + { name = "pyarrow", marker = "extra == 'fly'" }, { name = "pydantic-ai", extras = ["mcp"] }, { name = "python-dotenv" }, { name = "python-jose", extras = ["cryptography"] }, @@ -664,6 +668,7 @@ requires-dist = [ { name = "watchfiles" }, { name = "yfinance", specifier = ">=1.7.0" }, ] +provides-extras = ["fly"] [package.metadata.requires-dev] dev = [ From 81d91d65db29161e6d95824a4647e352ea018e7f Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 19:42:17 -0700 Subject: [PATCH 45/48] (fix) four from review: trade count, activity pairing, bounds, book order **Trade count.** The report counted round trips with its own copy of the close types. Condor already has the authority -- count_trade_closes, shared by the live snapshot and the cumulative history -- and a report that answered the same question a third way was one answer too many. It calls the helper now. (The set it had derived independently was identical, so the numbers do not move.) **Activity pairing.** The neuron snapshot was written when the brain ran and carried no tick or pair, but a tick can still fail afterwards in the guard or the apply -- and the report then falls back to the last observation that completed, which is a different one. In a round-robin run that drew one market's neurons beside another market's posture. The snapshot is stamped and the report asks for the observation it is showing; a mismatch draws the anatomy instead of somebody else's activity. **Bounds.** Every scanner threshold accepted negatives. A negative fee or multiple does not loosen the spread floor, it puts it below zero, after which every market that is not crossed "clears the fee" -- the one mistake the routine exists to prevent. All of them are bounded now, with a test per field. **Book order.** depth_within took the ladder as sorted, and the walk stops at the first level outside the band -- so one out-of-order rung from a connector would hide every closer level behind it, understating depth and rejecting an eligible market. It sorts first, which costs nothing next to the call that fetched the book. --- agents/market_making_fly/flybrain/market.py | 9 +++- .../market_making_fly/flybrain/run_state.py | 28 +++++++++-- .../market_making_fly/routines/fly_brain.py | 2 +- .../market_making_fly/routines/fly_report.py | 22 +++------ .../routines/mm_market_scanner.py | 18 ++++++-- .../tests/test_fly_market.py | 46 +++++++++++++++++++ 6 files changed, 100 insertions(+), 25 deletions(-) diff --git a/agents/market_making_fly/flybrain/market.py b/agents/market_making_fly/flybrain/market.py index 3c71812c2..cde758e1a 100644 --- a/agents/market_making_fly/flybrain/market.py +++ b/agents/market_making_fly/flybrain/market.py @@ -75,6 +75,13 @@ def depth_within( """ if not bids or not asks: return 0.0, 0.0, 0.0 + # Sorted here rather than assumed. The touch is the best price and the walk + # stops at the first level outside the band, so one out-of-order rung from + # a connector would hide every closer level behind it — understating depth + # and rejecting a market that was eligible. Sorting a book this size costs + # nothing next to the call that fetched it. + bids = sorted(bids, key=lambda level: -level[0]) + asks = sorted(asks, key=lambda level: level[0]) best_bid, best_ask = bids[0][0], asks[0][0] mid = (best_bid + best_ask) / 2 if mid <= 0 or not math.isfinite(mid): @@ -86,7 +93,7 @@ def side(levels, is_bid): for price, size in levels: offset = (mid - price) / mid * 1e4 if is_bid else (price - mid) / mid * 1e4 if offset > within_bps: - break # the ladder is sorted; nothing beyond is closer + break # now genuinely sorted; nothing beyond is closer total += price * size return total diff --git a/agents/market_making_fly/flybrain/run_state.py b/agents/market_making_fly/flybrain/run_state.py index 59e013b6d..5286b2cd1 100644 --- a/agents/market_making_fly/flybrain/run_state.py +++ b/agents/market_making_fly/flybrain/run_state.py @@ -123,19 +123,39 @@ def append_event(self, row: dict) -> None: def write_latest(self, row: dict) -> None: atomic_write_json(self.latest_path, row, indent=2, default=str) - def save_activity(self, counts: list[int]) -> None: + def save_activity(self, counts: list[int], tick: int, pair: str) -> None: """Spike counts at the cloud's neurons, for the report to colour by. Overwritten each observation rather than appended: it is a few thousand numbers, which would bury `events.jsonl` within an hour, and only the latest is ever drawn. + + Stamped with the observation it came from. The file is written when the + brain runs, but the tick can still fail afterwards in the guard or the + apply — and the report then falls back to the last observation that + completed, which is a *different* one. Unstamped, a round-robin run + would draw one market's neurons beside another market's posture. """ - atomic_write_json(self.activity_path, counts) + atomic_write_json( + self.activity_path, {"tick": tick, "pair": pair, "counts": counts} + ) - def load_activity(self) -> list[int] | None: + def load_activity( + self, tick: int | None = None, pair: str = "" + ) -> list[int] | None: + """The spike counts, but only if they belong to the observation asked + for. A mismatch returns None, which draws the anatomy instead of + somebody else's activity.""" if not self.activity_path.exists(): return None - return json.loads(self.activity_path.read_text()) + saved = json.loads(self.activity_path.read_text()) + if not isinstance(saved, dict): + return None # a snapshot from before they were stamped + if tick is not None and ( + saved.get("tick") != tick or saved.get("pair") != pair + ): + return None + return saved.get("counts") def save_frame(self, frame: np.ndarray) -> None: Image.fromarray(frame).save(self.frame_path) diff --git a/agents/market_making_fly/routines/fly_brain.py b/agents/market_making_fly/routines/fly_brain.py index 29d627a09..9c531c90f 100644 --- a/agents/market_making_fly/routines/fly_brain.py +++ b/agents/market_making_fly/routines/fly_brain.py @@ -489,7 +489,7 @@ async def pace() -> None: tick += 1 persist({"checkpoint": {"file": ck.name, "sha256": sha}}) - run_dir.save_activity(neural["activity"]) + run_dir.save_activity(neural["activity"], tick - 1, pair) neural_row = { k: v for k, v in neural.items() if k not in ("cell_ids", "activity") } diff --git a/agents/market_making_fly/routines/fly_report.py b/agents/market_making_fly/routines/fly_report.py index 77f0a87f4..adb7e2cb3 100644 --- a/agents/market_making_fly/routines/fly_report.py +++ b/agents/market_making_fly/routines/fly_report.py @@ -32,6 +32,7 @@ from pydantic import BaseModel, Field from telegram.ext import ContextTypes +from condor.fetchers.bot_performance import count_trade_closes from condor.memory.paths import agent_home from condor.paths import safe_id from condor.reports import ReportBuilder @@ -169,15 +170,6 @@ def _pnl_figure(events: list[dict]) -> go.Figure | None: return fig -# A quote that was cancelled on refresh never traded. Counting every close -# type as a trade said 70 on a run with five fills and no completed pair: 71 -# of those were EARLY_STOP, which is the controller replacing its own unfilled -# orders. These are the closes that end a position and realize its P&L. -ROUND_TRIP_CLOSES = frozenset( - {"TAKE_PROFIT", "STOP_LOSS", "TRAILING_STOP", "TIME_LIMIT", "COMPLETED"} -) - - async def _holdings( client, connector_name: str, pairs: list[str] ) -> tuple[list[dict], int | None, int]: @@ -204,11 +196,11 @@ async def _holdings( inner = perf.get("performance", perf) if isinstance(perf, dict) else {} closes = inner.get("close_type_counts") or {} if isinstance(closes, dict): - trades = (trades or 0) + sum( - int(count or 0) - for name, count in closes.items() - if str(name).split(".")[-1] in ROUND_TRIP_CLOSES - ) + # Condor's own definition of a trade, not a second copy of it: the + # live snapshot and the cumulative history already count round + # trips this way, and a report that disagreed with them would be a + # third answer to the same question. + trades = (trades or 0) + count_trade_closes(inner) held = held + sum( int(count or 0) for name, count in closes.items() @@ -407,7 +399,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: ) builder.plotly( brain_figure( - run_dir.load_activity(), + run_dir.load_activity(observed.get("tick"), observed.get("pair", "")), title="NEURAL ACTIVITY", height=ROW_TWO - PANEL_CHROME, ), diff --git a/agents/market_making_fly/routines/mm_market_scanner.py b/agents/market_making_fly/routines/mm_market_scanner.py index 01ed25ab5..18cb4dc30 100644 --- a/agents/market_making_fly/routines/mm_market_scanner.py +++ b/agents/market_making_fly/routines/mm_market_scanner.py @@ -91,11 +91,18 @@ class Config(BaseModel): default="", description="Only pairs with this issuer prefix, e.g. xyz for HIP-3 (blank = any)", ) + # Every threshold below is bounded. A negative fee or a negative multiple + # does not loosen a filter, it inverts it: the floor goes below zero and + # every market that is not crossed "clears the fee", which is the one + # mistake this routine exists to prevent. maker_fee_bps: float = Field( - default=0.0, description="Maker fee per side in bp; 0 uses the venue default" + default=0.0, + ge=0.0, + description="Maker fee per side in bp; 0 uses the venue default", ) min_range_over_cycle: float = Field( default=1.0, + gt=0.0, description="Require a typical candle's range to be this multiple of the " "round trip the fly would have to travel (quote distance + take-profit). " "1.0 means a median candle completes one cycle", @@ -104,16 +111,19 @@ class Config(BaseModel): default="5m", description="Candle the range is measured on; use the fly's own interval", ) - min_volume_usd: float = Field(default=250_000.0, description="Minimum 24h volume") + min_volume_usd: float = Field( + default=250_000.0, ge=0.0, description="Minimum 24h volume" + ) max_daily_drift_pct: float = Field( - default=3.0, description="Maximum 24h price drift %" + default=3.0, ge=0.0, description="Maximum 24h price drift %" ) min_book_depth_usd: float = Field( default=10_000.0, + ge=0.0, description="Minimum resting notional per side, within the band", ) depth_within_bps: float = Field( - default=10.0, description="Band around mid for depth" + default=10.0, gt=0.0, description="Band around mid for depth" ) prescreen: int = Field( default=30, diff --git a/agents/market_making_fly/tests/test_fly_market.py b/agents/market_making_fly/tests/test_fly_market.py index 5456f1c1b..e622a9930 100644 --- a/agents/market_making_fly/tests/test_fly_market.py +++ b/agents/market_making_fly/tests/test_fly_market.py @@ -7,6 +7,7 @@ from flybrain.market import ( Book, FixtureMarket, + depth_within, normalize_candle_payload, pnl_is_known, required_collateral, @@ -215,3 +216,48 @@ def test_the_scanner_measures_the_trip_the_fly_would_actually_make(): # A market whose median candle travels less than the cycle never completes # one, however wide its touch looks. assert 4.0 / mod.cycle_bps(2.0, 1.3) < 1.0 + + +def test_depth_survives_a_ladder_that_arrives_out_of_order(): + """The walk stops at the first level outside the band, so one out-of-order + rung would hide every closer level behind it — understating depth and + rejecting a market that was eligible.""" + tidy_bids = [(100.0, 5.0), (99.9, 5.0), (99.0, 5.0)] + tidy_asks = [(100.1, 5.0), (100.2, 5.0), (101.0, 5.0)] + expected = depth_within(tidy_bids, tidy_asks, within_bps=30) + + shuffled_bids = [(99.0, 5.0), (100.0, 5.0), (99.9, 5.0)] + shuffled_asks = [(101.0, 5.0), (100.1, 5.0), (100.2, 5.0)] + assert depth_within(shuffled_bids, shuffled_asks, within_bps=30) == expected + # and the touch itself is the best price, not whatever arrived first + assert depth_within(shuffled_bids, shuffled_asks, 30)[2] == pytest.approx( + (100.1 - 100.0) / 100.05 * 1e4 + ) + + +def test_the_scanner_refuses_a_threshold_that_would_invert_a_filter(): + """A negative fee or multiple does not loosen the spread floor, it puts it + below zero — after which every market that is not crossed 'clears the + fee', which is the one mistake this routine exists to prevent.""" + import importlib.util + from pathlib import Path + + import pydantic + + path = Path(__file__).resolve().parents[1] / "routines" / "mm_market_scanner.py" + spec = importlib.util.spec_from_file_location("mm_market_scanner", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + + for field, bad in ( + ("maker_fee_bps", -1.3), + ("min_range_over_cycle", -1.0), + ("min_range_over_cycle", 0.0), + ("min_volume_usd", -1.0), + ("max_daily_drift_pct", -3.0), + ("min_book_depth_usd", -10_000.0), + ("depth_within_bps", 0.0), + ): + with pytest.raises(pydantic.ValidationError): + mod.Config(**{field: bad}) + mod.Config() # the defaults are all inside their own bounds From 1bdff1018c070b7d950de9ee6bf25c7e62cf0b5e Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 19:42:52 -0700 Subject: [PATCH 46/48] (refactor) drop two skills the fly copied and never referenced pmm_config_playbook and capital_allocation were copied from Market Making Expert at scaffold time and are byte-identical to it apart from the `source:` line in their frontmatter. Nothing in this agent reads either: the deploy playbook builds its config from posture.build_config, and sizing is the operator's `total_amount_quote`. Two copies of a playbook is two things to keep in step, and the copy nobody reads is the one that silently goes stale. The README's reference to the playbook's balanced profile is now a pointer to build_config, which since today's parity change *is* that profile everywhere the fly does not decide. --- agents/market_making_fly/README.md | 7 +- .../skills/capital_allocation/SKILL.md | 162 ------------------ .../skills/pmm_config_playbook/SKILL.md | 106 ------------ .../pmm_config_playbook/config_aggressive.md | 76 -------- .../pmm_config_playbook/config_balanced.md | 70 -------- .../config_conservative.md | 92 ---------- 6 files changed, 3 insertions(+), 510 deletions(-) delete mode 100644 agents/market_making_fly/skills/capital_allocation/SKILL.md delete mode 100644 agents/market_making_fly/skills/pmm_config_playbook/SKILL.md delete mode 100644 agents/market_making_fly/skills/pmm_config_playbook/config_aggressive.md delete mode 100644 agents/market_making_fly/skills/pmm_config_playbook/config_balanced.md delete mode 100644 agents/market_making_fly/skills/pmm_config_playbook/config_conservative.md diff --git a/agents/market_making_fly/README.md b/agents/market_making_fly/README.md index 1e69e0cc8..0393b5915 100644 --- a/agents/market_making_fly/README.md +++ b/agents/market_making_fly/README.md @@ -385,8 +385,6 @@ agents/market_making_fly/ skills/ fly_mm_deploy/SKILL.md # deploy playbook (adapted from pmm_mister_deploy: scanner → base config → deploy → start fly_brain) fly_decoder/SKILL.md # how to read fly_status and the decoder - pmm_config_playbook/ # copied - capital_allocation/ # copied strategies/mm_operator/strategy.md # thin loop: keep bot + fly alive, surface halts, rotate when flat agents/market_making_fly/tests/ # conftest puts the agent dir on sys.path @@ -424,8 +422,9 @@ Two modes, like Market Making Expert: "is it learning anything" → runs `fly_status` (and `fly_chart`), answers in key: value lines, quotes the numbers, repeats the caveats in §15. * **Delegated / loop**: `fly_mm_deploy` skill end-to-end: scanner → TOP PICK → - base config from `pmm_config_playbook` balanced profile adapted with HIP-3 - bounds → deploy with `max_global_drawdown_quote` → start `fly_brain` + base config built by `posture.build_config`, which *is* Market Making + Expert's balanced profile everywhere the fly does not decide (see §15) → + deploy with `max_global_drawdown_quote` → start `fly_brain` (shadow first unless told live) → verify with `fly_report`. The loop strategy ticks every 5 min: confirms bot and fly instance alive, surfaces halts/vetoes, rotates when flat and the market is closed. diff --git a/agents/market_making_fly/skills/capital_allocation/SKILL.md b/agents/market_making_fly/skills/capital_allocation/SKILL.md deleted file mode 100644 index 44c45c0bf..000000000 --- a/agents/market_making_fly/skills/capital_allocation/SKILL.md +++ /dev/null @@ -1,162 +0,0 @@ ---- -name: capital_allocation -description: How total_amount_quote and initial_positions define a controller's isolated - capital — the budget a single pmm_mister controller trades with, and how to seed it - with base assets you already hold so each controller is independent from the wider portfolio. -when_to_use: When sizing a controller, deciding total_amount_quote, splitting one market - into several controllers, or when the user already holds the base asset (spot) and wants - to fund the strategy with existing inventory instead of buying fresh. Also read this - before answering "how much capital does this bot use" or "how do I use the BTC I already have". -source: agent:market_making_fly ---- - -# Capital Allocation: total_amount_quote & initial_positions - -This skill explains the two knobs that define **how much capital a single -controller trades with** and **which of that capital comes from assets you -already hold**. Together they make each controller's book independent from the -rest of your portfolio. - ---- - -## 1. `total_amount_quote` — the controller's budget - -`total_amount_quote` is the **total capital assigned to that one controller**, -denominated in the quote asset (e.g. USDC for a BTC-USDC market, BRL for a -BTC-BRL market). - -This is the reference amount everything else is measured against: - -- `portfolio_allocation` — fraction of `total_amount_quote` actively deployed - per iteration. -- `target_base_pct` / `min_base_pct` / `max_base_pct` — the inventory band. These - percentages are **percentages of `total_amount_quote`**, expressed in quote - value. If `total_amount_quote = 2000` and `target_base_pct = 50`, the target - base inventory is worth **$1,000** of the base asset. - -So `total_amount_quote` is the denominator. Change it and every absolute -position size, order size, and inventory band scales with it. It is the single -number that says "this controller is allowed to work with this much money." - ---- - -## 2. The default: starting fresh from quote - -By default a controller assumes it starts with **`total_amount_quote` worth of -quote asset and zero base**. It then builds up base inventory toward -`target_base_pct` by getting its buy orders filled — buying the base with quote -as the market comes to it. - -This is fine when you have plenty of quote and don't mind the controller -acquiring the base itself. Nothing extra is needed. - ---- - -## 3. `initial_positions` — seeding with assets you already hold - -If you are trading **spot and you already own the base asset**, you don't have to -make the controller buy it from scratch. You can hand existing inventory to the -controller at startup via `initial_positions`: - -```yaml -initial_positions: - - amount: 0.08328 - connector_name: binance - side: BUY - trading_pair: BTC-BRL -``` - -When you deploy through the normal flow, configs are upserted as a `config_data` -**dict** (`manage_controllers(action="upsert", target="config", ...)`), so -`initial_positions` is a **list of dicts** — the copy-paste-ready form is: - -```json -"initial_positions": [ - { - "amount": 0.08328, - "connector_name": "binance", - "side": "BUY", - "trading_pair": "BTC-BRL" - } -] -``` - -Both forms are equivalent — YAML for config files, JSON/dict for the -`manage_controllers` upsert path. - -What this means: - -- You declare **how many units of the base asset from your portfolio you want to - assign to this controller at the start** of the strategy. -- That amount goes **directly into position hold** — the controller starts - already holding this inventory as an open BUY position, instead of holding pure - quote. -- `side: BUY` marks it as a long base position that the strategy now manages - (its TP/SL and inventory logic apply to it just like a position it opened - itself). - -### You choose whether to use existing assets or not - -- **Don't assign them** → the controller starts fresh from quote. Use this when - you have enough quote to fund `total_amount_quote` on its own and you'd rather - leave your existing base untouched. You can hold assets and simply not use - them. -- **Assign them via `initial_positions`** → the controller starts with that base - inventory already in hand, counting toward its `target_base_pct` band. Use this - when you want your existing holdings to be the working inventory rather than - buying more. - -The base you assign should be consistent with the controller's budget: the quote -value of the assigned base is part of the `total_amount_quote` this controller -manages, so it counts toward the inventory band (`target/min/max_base_pct`). - ---- - -## 4. Why this matters: portfolio independence - -This is the mechanism that makes **each controller's book independent from the -overall portfolio**. Instead of one giant strategy over your whole balance, you -carve the portfolio into slices and hand each slice to its own controller with -its own `total_amount_quote` and its own seeded inventory. - -### Worked example — splitting a market into 10 controllers - -Say you hold **$10k USDC and $10k of BTC** (dollar value) and want to market-make -BTC-USDC. You can deploy **10 controllers**, each with: - -- `total_amount_quote = 2000` (2,000 USDC of budget per controller → 10 × 2,000 = - $20k total, matching your combined capital). -- A proportional slice of your existing BTC assigned via `initial_positions` — - i.e. split the BTC you hold across the 10 controllers so each starts with - ~1/10 of it as seeded base inventory. - -Now each controller runs its own isolated book on a $2,000 budget, half funded by -quote and half by the BTC you already had. The controllers don't fight over one -shared balance — each has a fixed, known slice, so their PnL, inventory bands, -and risk are measured independently. **This is how we make the strategy's capital -independent from the general portfolio.** - ---- - -## Quick reference - -| Concept | Meaning | -|---------|---------| -| `total_amount_quote` | Total capital assigned to **this one controller**, in quote units. The denominator for allocation and all base-pct bands. | -| `target/min/max_base_pct` | Inventory band as a % of `total_amount_quote`. | -| Default start | Controller assumes `total_amount_quote` in quote, 0 base — buys base itself. | -| `initial_positions` | Seed the controller with base you already hold; goes straight to position hold as a managed BUY. Optional. | -| Splitting a market | Deploy N controllers, each with `total_amount_quote = budget/N` and a proportional slice of existing base via `initial_positions` → N independent books. | - -### `initial_positions` fields - -- `amount` — units of the **base** asset to assign (e.g. `0.08328` BTC). -- `connector_name` — the connector holding the asset (e.g. `binance`). -- `side` — `BUY` for a long base position the strategy will manage. -- `trading_pair` — the controller's pair (e.g. `BTC-BRL`). - -Only applies to **spot** with existing base inventory. It's optional — assign -existing assets when you want them to be the working inventory; omit it to start -fresh from quote. - - diff --git a/agents/market_making_fly/skills/pmm_config_playbook/SKILL.md b/agents/market_making_fly/skills/pmm_config_playbook/SKILL.md deleted file mode 100644 index b3dd2b8d2..000000000 --- a/agents/market_making_fly/skills/pmm_config_playbook/SKILL.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -name: pmm_config_playbook -description: Ready-to-deploy pmm_mister config profiles (aggressive / balanced / conservative) - — full parameter coverage including spreads, effectivization times, tolerance, order - types, skew, and global TP/SL. -when_to_use: When you need a starting pmm_mister controller config and want a vetted - template instead of hand-tuning every parameter — pick a profile by regime, fetch - its template, then adapt the connector/pair/amount. -source: builtin ---- - -# pmm_mister Config Playbook - -Three vetted `pmm_mister` profiles, one per risk posture. Each profile lives in a -**companion file** — fetch only the one you need so the others never load into -context: - -``` -manage_skill(action="read_file", name="pmm_config_playbook", file="config_aggressive.md") -``` - -## Pick a profile by regime - -| Regime | Profile | File | -|-----------------------------------------|------------------|-----------------------------|\n| Quiet / low-vol ranging (ADX < 18) | **Aggressive** | `config_aggressive.md` | -| Ranging / normal (ADX < 25) | **Balanced** | `config_balanced.md` | -| Volatile / trending / uncertain | **Conservative** | `config_conservative.md` | - -- **Aggressive** — tight spreads, fast refresh, short effectivization (60s), - wide inventory bands, high allocation. Maximizes fill rate in calm markets. - Most inventory/PnL risk. -- **Balanced** — the default. Moderate spreads, 120s effectivization, standard - tolerances. Good steady-state when regime is unclear. -- **Conservative** — wide spreads, slow refresh, long effectivization (300s), - tight inventory bands, strong skew enforcement (min_skew=2.0), low - allocation/leverage, both global TP and SL active. Capital preservation in - chop/vol. - -## How to use a template - -1. Read the chosen companion file. It contains a full `config_data` block for - `manage_controllers(action="upsert", target="config")`. -2. **Always adapt** these to the actual operation before deploying: - - `connector_name`, `trading_pair` - - `total_amount_quote` (respect the strategy's risk limit) - - `leverage` (never above the strategy's cap; templates default low) -3. Deploy via the normal flow (`manage_controllers` upsert → `manage_bots` deploy). - Live retunes go through `manage_bots(action="update_config", confirm_override=true)`. - -## Key parameter reference - -Parameters most commonly tuned in real operations: - -**Inventory & allocation** -- `portfolio_allocation` — fraction of `total_amount_quote` actively deployed per iteration -- `target_base_pct` / `min_base_pct` / `max_base_pct` — inventory band; skew - kicks in when base drifts outside min/max -- `min_skew` — minimum spread multiplier applied to the heavy side when inventory - drifts; 1.0 = no minimum, 2.0 = at least 2× wider on the accumulating side -- `max_active_executors_by_level` — max concurrent open executors per level; - controls total directional exposure (fills × allocation per level) - -**Order timing** -- `executor_refresh_time` — how often (seconds) the controller checks and potentially - replaces open orders; lower = tighter to mid price but more rate limit usage -- `buy/sell_cooldown_time` — after a fill, how long to wait before placing a new order - on that side; lower = faster re-entry, more risk of accumulating at similar prices -- `buy/sell_position_effectivization_time` — how long (seconds) the per-fill - LIMIT_MAKER TP order stays on the book after a fill; when this expires, the TP - order is **removed** and the position transitions to "hold" mode managed only by - the global SL/TP layer. Lower = TP has less time to fill → positions accumulate - into hold faster. Higher = TP order stays on book longer → more chances of hitting TP. - -**Spread & refresh tolerance** -- `price_distance_tolerance` — minimum price gap required between open orders at - the same level; prevents stacking orders too close together -- `refresh_tolerance` — minimum mid-price move required to trigger a quote - refresh/replacement; lower = more responsive, higher cancel/replace churn -- `tolerance_scaling` — multiplier applied to tolerance values as the number of - active executors grows; prevents cancel-loops when multiple orders are open at - the same level - -**Order types** (3 = LIMIT_MAKER / post-only, 2 = LIMIT, 1 = MARKET) -- `open_order_type` — order type for entry orders (always use 3 unless exchange rejects) -- `take_profit_order_type` — order type for TP orders (always 3) - -**Per-fill risk** -- `take_profit` — offset from fill price where the LIMIT_MAKER TP order is placed; - must be > round-trip fees to be profitable - -**Global portfolio guardrails (position hold phase)** -- `global_tp_enabled` / `global_take_profit` — when total held position gains this %, close and restart MM -- `global_sl_enabled` / `global_stop_loss` — when total held position loses this %, close and restart MM -- `global_sl_activation_from` — inventory threshold from which global SL activates - (`"min_base"` = when base is light, `"target_base"` = from neutral) -- `global_tp_activation_from` — inventory threshold from which global TP activates -- `global_pnl_reference` — PnL basis: `"position"` (unrealized open PnL) or - `"portfolio"` (total portfolio value) -- `position_profit_protection` — blocks inventory reduction at unfavorable prices - -**Other** -- `tick_mode` — if true, controller runs only on candle ticks; false = continuous (default) - -The full regime→param decision logic lives in the `pmm_mister_operator` strategy. -These templates are starting points — every deploy still goes through normal -risk/confirmation controls. diff --git a/agents/market_making_fly/skills/pmm_config_playbook/config_aggressive.md b/agents/market_making_fly/skills/pmm_config_playbook/config_aggressive.md deleted file mode 100644 index 3a1fa49dc..000000000 --- a/agents/market_making_fly/skills/pmm_config_playbook/config_aggressive.md +++ /dev/null @@ -1,76 +0,0 @@ -# Aggressive pmm_mister Profile - -**Use when:** quiet / low-volatility ranging market (ADX < 18, BBW < ~3%). You -want maximum fill rate and volume capture. **Highest** inventory + PnL risk — -do NOT use in trending or volatile regimes. - -Tight spreads, fast order refresh, short cooldowns and effectivization times, -wide inventory tolerance, higher allocation. - -```json -{ - "controller_type": "generic", - "controller_name": "pmm_mister", - "connector_name": "binance_perpetual", - "trading_pair": "JTO-USDT", - "total_amount_quote": 500, - "portfolio_allocation": 0.25, - "target_base_pct": 0.5, - "min_base_pct": 0.3, - "max_base_pct": 0.7, - "buy_spreads": "0.0008,0.0015", - "sell_spreads": "0.0008,0.0015", - "buy_amounts_pct": "1,1", - "sell_amounts_pct": "1,1", - "executor_refresh_time": 20, - "buy_cooldown_time": 30, - "sell_cooldown_time": 30, - "buy_position_effectivization_time": 60, - "sell_position_effectivization_time": 60, - "price_distance_tolerance": 0.0005, - "refresh_tolerance": 0.0003, - "tolerance_scaling": 1.1, - "open_order_type": 3, - "take_profit": 0.0008, - "take_profit_order_type": 3, - "leverage": 10, - "position_mode": "ONEWAY", - "position_side": "BUY", - "max_active_executors_by_level": 4, - "tick_mode": false, - "min_skew": 1.0, - "global_tp_enabled": false, - "global_sl_enabled": true, - "global_stop_loss": 0.05, - "global_sl_activation_from": "target_base", - "global_pnl_reference": "position" -} -``` - -**Parameter notes** -- `buy/sell_position_effectivization_time` (60s): The per-fill LIMIT_MAKER TP - order stays on the book for only 60s after each fill. In a quiet, low-drift - market the price barely moves, so the TP is unlikely to fill in that window — - positions quickly transition to hold mode. This is intentional: in calm - conditions we let fills accumulate into a held position rather than chasing - individual TPs. global_tp is disabled here, so held positions grow until the - global SL triggers. -- `price_distance_tolerance` (0.0005): Minimum gap required between stacked - orders at the same level. Keeps orders spread out to avoid clustering. -- `refresh_tolerance` (0.0003): Tighter than default — triggers a quote - refresh/replacement with smaller mid-price moves. More responsive in calm - markets where small moves matter. -- `tolerance_scaling` (1.1): Low multiplier — tolerance widens slowly as - executors accumulate. Stay close to mid. -- `open_order_type` / `take_profit_order_type` (3 = LIMIT_MAKER): Post-only. - Never takes liquidity. Change to 2 (LIMIT) only if the exchange rejects makers. -- `tick_mode` (false): Keep false for continuous market making. -- `min_skew` (1.0): No minimum skew enforced — spreads stay symmetric when - inventory is balanced. - -**Tuning notes** -- If fills are too one-sided, narrow the inventory band (raise `min_base_pct` / - lower `max_base_pct`) so skew kicks in sooner. -- If the market starts trending, switch to **balanced** or **conservative** — - tight two-sided spreads bleed into a trend. -- `global_sl_enabled` stays on even here: 5% hard stop is the floor. diff --git a/agents/market_making_fly/skills/pmm_config_playbook/config_balanced.md b/agents/market_making_fly/skills/pmm_config_playbook/config_balanced.md deleted file mode 100644 index e18f149d0..000000000 --- a/agents/market_making_fly/skills/pmm_config_playbook/config_balanced.md +++ /dev/null @@ -1,70 +0,0 @@ -# Balanced pmm_mister Profile - -**Use when:** normal ranging market (ADX < 25, moderate BBW). This is the -**default** steady-state profile — moderate spreads, allocation and cooldowns. -Good when no regime signal is strong enough to justify aggressive or -conservative. - -```json -{ - "controller_type": "generic", - "controller_name": "pmm_mister", - "connector_name": "binance_perpetual", - "trading_pair": "JTO-USDT", - "total_amount_quote": 500, - "portfolio_allocation": 0.15, - "target_base_pct": 0.5, - "min_base_pct": 0.35, - "max_base_pct": 0.65, - "buy_spreads": "0.0012,0.0025", - "sell_spreads": "0.0012,0.0025", - "buy_amounts_pct": "1,1", - "sell_amounts_pct": "1,1", - "executor_refresh_time": 30, - "buy_cooldown_time": 60, - "sell_cooldown_time": 60, - "buy_position_effectivization_time": 120, - "sell_position_effectivization_time": 120, - "price_distance_tolerance": 0.0005, - "refresh_tolerance": 0.0005, - "tolerance_scaling": 1.2, - "open_order_type": 3, - "take_profit": 0.001, - "take_profit_order_type": 3, - "leverage": 8, - "position_mode": "ONEWAY", - "position_side": "BUY", - "max_active_executors_by_level": 3, - "tick_mode": false, - "min_skew": 1.5, - "global_tp_enabled": false, - "global_sl_enabled": true, - "global_stop_loss": 0.05, - "global_sl_activation_from": "target_base", - "global_pnl_reference": "position" -} -``` - -**Parameter notes** -- `buy/sell_position_effectivization_time` (120s): The per-fill LIMIT_MAKER TP - order stays on the book for 2 minutes after each fill. If the market moves - enough to hit the TP in that window, the position closes with a per-fill profit. - If not, the position transitions to hold mode after 120s and is managed by - the global SL layer. Balanced between giving the TP time to fill and not - leaving stale positions open indefinitely. -- `price_distance_tolerance` / `refresh_tolerance` (0.0005): Controller defaults. - Balanced refresh cadence — not too aggressive, not too slow. -- `tolerance_scaling` (1.2): Default multiplier. Tolerance widens moderately - as executors accumulate, preventing cancel-loops in ranging markets. -- `open_order_type` / `take_profit_order_type` (3 = LIMIT_MAKER): Post-only - orders. Change to 2 (LIMIT) only if the exchange rejects makers. -- `tick_mode` (false): Keep false for continuous market making. -- `min_skew` (1.5): Enforces a minimum 1.5× spread multiplier on the heavy side - when inventory drifts. Mild protection against runaway accumulation. - -**Tuning notes** -- Start here when unsure, then shift toward aggressive (calm) or conservative - (vol/trend) as the regime clarifies. -- For a mild trend, make spreads asymmetric: widen the side you don't want to - trade into (e.g. wider `sell_spreads` in an uptrend). -- Increase `min_skew` to 2.0+ if inventory keeps drifting despite the band. diff --git a/agents/market_making_fly/skills/pmm_config_playbook/config_conservative.md b/agents/market_making_fly/skills/pmm_config_playbook/config_conservative.md deleted file mode 100644 index 559900e48..000000000 --- a/agents/market_making_fly/skills/pmm_config_playbook/config_conservative.md +++ /dev/null @@ -1,92 +0,0 @@ -# Conservative pmm_mister Profile - -**Use when:** volatile, trending, or uncertain market (ATR expanding, BBW > ~6%, -ADX > 25, volume surge). Capital preservation first — wide spreads, slow -refresh, tight inventory bands, long effectivization times, low -allocation/leverage, and **both** global TP and SL protections enabled. - -```json -{ - "controller_type": "generic", - "controller_name": "pmm_mister", - "connector_name": "binance_perpetual", - "trading_pair": "JTO-USDT", - "total_amount_quote": 500, - "portfolio_allocation": 0.1, - "target_base_pct": 0.5, - "min_base_pct": 0.4, - "max_base_pct": 0.6, - "buy_spreads": "0.003,0.006", - "sell_spreads": "0.003,0.006", - "buy_amounts_pct": "1,1", - "sell_amounts_pct": "1,1", - "executor_refresh_time": 60, - "buy_cooldown_time": 120, - "sell_cooldown_time": 120, - "buy_position_effectivization_time": 300, - "sell_position_effectivization_time": 300, - "price_distance_tolerance": 0.001, - "refresh_tolerance": 0.001, - "tolerance_scaling": 1.3, - "open_order_type": 3, - "take_profit": 0.0015, - "take_profit_order_type": 3, - "leverage": 5, - "position_mode": "ONEWAY", - "position_side": "BUY", - "max_active_executors_by_level": 2, - "tick_mode": false, - "min_skew": 2.0, - "position_profit_protection": true, - "global_tp_enabled": true, - "global_take_profit": 0.03, - "global_tp_activation_from": "min_base", - "global_sl_enabled": true, - "global_stop_loss": 0.04, - "global_sl_activation_from": "target_base", - "global_pnl_reference": "position" -} -``` - -**Parameter notes** -- `buy/sell_position_effectivization_time` (300s): The per-fill LIMIT_MAKER TP - order stays on the book for 5 minutes after each fill. In volatile markets - with frequent wicks, this gives the TP more time to be hit — individual fills - get closed profitably before the position transitions to hold. If the TP is - not hit in 300s, the position enters hold mode and both global TP (3%) and - global SL (4%) take over risk management. -- `price_distance_tolerance` / `refresh_tolerance` (0.001): Wider than default. - Avoids over-refreshing when price is moving constantly — reduces - cancel/replace churn and fees in volatile conditions. -- `tolerance_scaling` (1.3): Higher multiplier — tolerance grows faster per - executor so the controller doesn't thrash in choppy conditions. -- `open_order_type` / `take_profit_order_type` (3 = LIMIT_MAKER): Post-only. - Never takes liquidity — critical in volatile markets to avoid adverse fills. -- `tick_mode` (false): Keep false. Tick mode reduces update frequency but adds - complexity not needed here. -- `min_skew` (2.0): Forces at least 2× spread multiplier on the accumulating - side when inventory drifts. Aggressively discourages one-sided fills in - trending conditions. -- `position_profit_protection` (true): Blocks inventory reductions at - unfavorable prices — won't dump positions into a spike. -- `global_tp_enabled` / `global_take_profit` (3%): Portfolio-level TP. When - the held position's PnL crosses +3%, the controller begins winding down. -- `global_tp_activation_from` ("min_base"): TP activates when base inventory - is at or below `min_base_pct` — when the portfolio is light on base and - already showing profit. -- `global_sl_activation_from` ("target_base"): SL activates when base inventory - is at or above `target_base_pct` — protecting against heavy accumulation - losing value. -- `global_pnl_reference` ("position"): PnL is measured against the current - open position value (unrealized). Use "portfolio" to measure against total - portfolio value instead. - -**Tuning notes** -- In extreme volatility, drop `portfolio_allocation` further or pause the bot - entirely rather than widening spreads indefinitely. -- `position_profit_protection` blocks reductions at unfavorable prices — keep - it on so the controller won't dump inventory into a spike. -- Tighter `global_stop_loss` (4%) than the other profiles: cut losers faster - when the regime is hostile. -- If wicks keep triggering the SL, increase `global_stop_loss` slightly or - widen `buy/sell_spreads` so entry prices have more buffer. From 775329bc2c62ef5c133c2dfadfefb5b660fb23ba Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 19:44:54 -0700 Subject: [PATCH 47/48] (feat) replay several windows, and judge a control by how many it lost The null this agent now rests on -- real reinforcement indistinguishable from randomly-signed reinforcement -- came from one window of one market, and one window is exactly what let the sign of a result flip between two runs ninety minutes apart. The weakness applies to the finding as much as to what it displaced. `windows` splits the pinned series into contiguous slices sharing no candle, replays every variant on each with a freshly seeded brain, and reports the increments pooled across all of them alongside how many slices each control actually won. A difference that is real should show in both; one that shows only in the pooled total was carried by a slice, and the table says so rather than calling it distinguishable. A window too thin to mean anything is refused rather than scored: 1000 candles split eight ways leaves four ticks after the retina's own 120-candle window, which is arithmetically valid and statistically worthless. --- agents/market_making_fly/flybrain/replay.py | 59 +++++++++++ .../market_making_fly/routines/fly_replay.py | 99 ++++++++++++++++--- .../tests/test_fly_replay.py | 28 ++++++ 3 files changed, 172 insertions(+), 14 deletions(-) diff --git a/agents/market_making_fly/flybrain/replay.py b/agents/market_making_fly/flybrain/replay.py index 884ecf772..3882c0855 100644 --- a/agents/market_making_fly/flybrain/replay.py +++ b/agents/market_making_fly/flybrain/replay.py @@ -52,6 +52,11 @@ # unrealized P&L; a closed one has both fees taken out of it. SIDES = ("buy", "sell") +# Below this a window is arithmetically valid and statistically worthless: it +# would fit the retina's candle window and leave a handful of ticks to score, +# which is not a sample of anything. Refusing is better than reporting it. +MIN_TICKS_PER_WINDOW = 30 + @dataclass class Lot: @@ -321,6 +326,60 @@ def replay( ) +def windows(candles: list[dict], count: int, span: int) -> list[list[dict]]: + """Split a series into ``count`` contiguous slices of equal length. + + One window is one sample of one market, and a sample of one is how the sign + of a result flips between two runs ninety minutes apart. Contiguous rather + than overlapping, so the slices share no candle and each is an independent + draw of the same market. + """ + if count < 1: + raise ValueError("count must be at least 1") + usable = len(candles) // count + if usable - span - 1 < MIN_TICKS_PER_WINDOW: + raise ValueError( + f"{len(candles)} candles split {count} ways leaves {usable} each, " + f"which is {max(0, usable - span - 1)} ticks after a {span}-candle " + f"window — not enough to score. A window needs at least " + f"{MIN_TICKS_PER_WINDOW}: fetch more candles or ask for fewer windows" + ) + return [candles[i * usable : (i + 1) * usable] for i in range(count)] + + +def pooled_stats(pairs: list[tuple[list[float], list[float]]]) -> dict: + """One verdict over several windows: the increments pooled, and how many + windows the first run actually led in. + + The pooled t answers "does it earn differently"; the count answers "or did + one window carry it". A result that is real should do both, and a result + that leads in two windows of four is noise however large its total. + """ + gains: list[float] = [] + led = 0 + for a, b in pairs: + n = min(len(a), len(b)) + if n < 4: + continue + gains.extend((a[i] - a[i - 1]) - (b[i] - b[i - 1]) for i in range(1, n)) + if a[n - 1] > b[n - 1]: + led += 1 + if len(gains) < 4: + return {"n": 0, "windows": len(pairs), "led": led, "mean_diff": 0.0, "t": 0.0} + mean = sum(gains) / len(gains) + var = sum((g - mean) ** 2 for g in gains) / (len(gains) - 1) + sd = math.sqrt(var) + scale = max(abs(mean), max((abs(g) for g in gains), default=0.0)) + return { + "n": len(gains), + "windows": len(pairs), + "led": led, + "mean_diff": mean, + "sd": sd, + "t": 0.0 if sd <= scale * 1e-9 else mean / (sd / math.sqrt(len(gains))), + } + + def paired_stats(a: list[float], b: list[float]) -> dict: """Do two runs earn differently per tick? Mean difference, sd, and t. diff --git a/agents/market_making_fly/routines/fly_replay.py b/agents/market_making_fly/routines/fly_replay.py index 34f65f4a3..7d1f8b2c8 100644 --- a/agents/market_making_fly/routines/fly_replay.py +++ b/agents/market_making_fly/routines/fly_replay.py @@ -44,7 +44,7 @@ from flybrain.fly3d import ACCENT, BODY, GROUND, LIMB from flybrain.naming import pair_names from flybrain.posture import MarketSpec -from flybrain.replay import paired_stats, replay +from flybrain.replay import paired_stats, pooled_stats, replay, windows from pydantic import BaseModel, Field from telegram.ext import ContextTypes @@ -96,6 +96,15 @@ class Config(BaseModel): baseline_window: int = Field(default=60) baseline_warmup: int = Field(default=10) seed: int = Field(default=7301, description="Seed for the shuffled control") + windows: int = Field( + default=1, + ge=1, + le=8, + description="Replay every variant on this many contiguous slices of the " + "series, and judge a control by how many of them it lost. One window is " + "one sample: the sign of a result flipped between two windows on " + "2026-09-13, so a single one settles nothing either way", + ) refresh_candles: bool = Field( default=False, description="Fetch a new window and pin it. Off by default: the venue " @@ -246,7 +255,7 @@ async def run(config: Config, context: ContextTypes.DEFAULT_TYPE) -> str: loop = asyncio.get_running_loop() gate = asyncio.Semaphore(config.concurrency) - async def one(name: str): + async def one(name: str, series: list[dict], label: str): overrides = VARIANTS[name] async with gate: # A fresh process per variant: the brain is stateful, and one that @@ -269,7 +278,7 @@ def observe(frame, stimulus, neural_ms, _pool=pool): lambda o=overrides, ob=observe: replay( variant=name, pair=config.trading_pair, - candles=candles, + candles=series, spec=spec, settings=_settings(config, o), observe=ob, @@ -282,14 +291,21 @@ def observe(frame, stimulus, neural_ms, _pool=pool): pool.shutdown(wait=True) await context.bot.send_message( chat_id=context._chat_id, - text=f"🪰 replay {name}: net {result.equity_curve[-1]:+.4f} " + text=f"🪰 replay {name}{label}: net {result.equity_curve[-1]:+.4f} " f"over {result.ticks} ticks, {result.ledger.fills} fills", ) return result + slices = windows(candles, config.windows, config.n_candles) # Order is the caller's, not the order they finished in: the first variant # is the baseline every control is compared against. - results = list(await asyncio.gather(*(one(name) for name in names))) + per_window: list[list] = [] + for index, series in enumerate(slices, 1): + label = f" w{index}" if len(slices) > 1 else "" + per_window.append( + list(await asyncio.gather(*(one(name, series, label) for name in names))) + ) + results = per_window[0] summaries = [r.summary() for r in results] # The brains are the expensive part and the arithmetic over their output is # not; keeping the curves means a better statistic never costs another run. @@ -363,6 +379,46 @@ def observe(frame, stimulus, neural_ms, _pool=pool): ) builder.plotly(_curve_figure(results)) + if len(results) > 1 and len(per_window) > 1: + # The verdict that matters when there is more than one window: the + # increments pooled across all of them, and how many windows the fly + # actually lost. A control it beats in two of four is noise whatever + # the total says. + builder.section( + "ACROSS WINDOWS", + f"{len(per_window)} contiguous slices of the same series, each " + "replayed by every variant on its own freshly seeded brain. " + "'Windows lost' counts the slices where the control finished ahead " + "of the deployed fly — a real difference should show in the pooled " + "increments *and* in most windows, and one that shows in only the " + "total was carried by one slice.", + ) + pooled_rows = [] + for position, name in enumerate(names[1:], start=1): + curves = [ + (window[0].equity_curve, window[position].equity_curve) + for window in per_window + ] + stats = pooled_stats(curves) + pooled_rows.append( + { + "Against": f"{names[0]} − {name}", + "Pooled per-tick": f"{stats['mean_diff']:+.5f}", + "Pooled t": f"{stats['t']:+.2f}", + "Windows won": f"{stats['led']}/{stats['windows']}", + "Reads as": ( + "distinguishable" + if abs(stats["t"]) >= 2 + and stats["led"] in (0, stats["windows"]) + else "not distinguishable" + ), + } + ) + builder.table( + pooled_rows, + ["Against", "Pooled per-tick", "Pooled t", "Windows won", "Reads as"], + ) + if len(results) > 1: base = results[0] rows = [] @@ -383,7 +439,7 @@ def observe(frame, stimulus, neural_ms, _pool=pool): } ) builder.section( - "AGAINST THE CONTROLS", + "AGAINST THE CONTROLS" + (" (first window)" if len(per_window) > 1 else ""), "How differently the deployed fly earns per tick, against each " "control. On increments, not on the equity curves themselves: a " "curve is cumulative, so once two runs separate every later tick " @@ -416,7 +472,8 @@ def observe(frame, stimulus, neural_ms, _pool=pool): lines = [ f"pair: {config.trading_pair} ({config.interval})", - f"candles: {len(candles)}, ticks: {results[0].ticks if results else 0}", + f"candles: {len(candles)} in {len(per_window)} window(s), " + f"ticks each: {results[0].ticks if results else 0}", f"median range: {market_range:.2f} bp, fee: {fee:.2f} bp", ] for s in summaries: @@ -427,11 +484,25 @@ def observe(frame, stimulus, neural_ms, _pool=pool): f"{s['one_sided']} one-sided" ) if len(results) > 1: - for other in results[1:]: - stats = paired_stats(results[0].equity_curve, other.equity_curve) - lines.append( - f"{results[0].variant} vs {other.variant}: mean per-tick earnings " - f"{stats['mean_diff']:+.5f}, final gap {stats['final_gap']:+.4f}, " - f"t {stats['t']:+.2f}" - ) + for position, name in enumerate(names[1:], start=1): + if len(per_window) > 1: + stats = pooled_stats( + [ + (window[0].equity_curve, window[position].equity_curve) + for window in per_window + ] + ) + lines.append( + f"{names[0]} vs {name}: pooled per-tick {stats['mean_diff']:+.5f}, " + f"t {stats['t']:+.2f}, won {stats['led']}/{stats['windows']} windows" + ) + else: + stats = paired_stats( + results[0].equity_curve, results[position].equity_curve + ) + lines.append( + f"{names[0]} vs {name}: mean per-tick earnings " + f"{stats['mean_diff']:+.5f}, final gap {stats['final_gap']:+.4f}, " + f"t {stats['t']:+.2f}" + ) return "\n".join(lines) diff --git a/agents/market_making_fly/tests/test_fly_replay.py b/agents/market_making_fly/tests/test_fly_replay.py index 9aca65614..30b95593a 100644 --- a/agents/market_making_fly/tests/test_fly_replay.py +++ b/agents/market_making_fly/tests/test_fly_replay.py @@ -109,3 +109,31 @@ def test_the_disconnected_gain_is_a_gain_the_decoder_accepts(): DecoderSettings(valence_gain=0.0) settings = DecoderSettings(valence_gain=mod.OFF) assert round(1 + settings.valence_gain * 3.0, 4) == 1.0 + + +def test_windows_are_contiguous_and_share_no_candle(): + from flybrain.replay import windows + + series = [{"close": i} for i in range(1000)] + cut = windows(series, 4, 72) + assert [len(w) for w in cut] == [250] * 4 + assert [w[0]["close"] for w in cut] == [0, 250, 500, 750] + # a split too thin to score is refused rather than silently returning stubs + with pytest.raises(ValueError, match="not enough"): + windows(series, 8, 120) + + +def test_a_result_carried_by_one_window_is_not_a_result(): + """The whole point of several windows: a control beaten in one slice and + lost in the rest should not read as an edge because the totals add up.""" + from flybrain.replay import pooled_stats + + flat = [0.0] * 60 + jump = [0.0] * 30 + [9.0] * 30 # one window, one big move, then nothing + carried = pooled_stats([(jump, flat), (flat, jump), (flat, jump), (flat, jump)]) + assert carried["led"] == 1 and carried["windows"] == 4 + + # and a steady per-tick edge in every window shows in both + steady = [i * 0.05 for i in range(60)] + real = pooled_stats([(steady, flat)] * 4) + assert real["led"] == 4 and real["mean_diff"] > 0 From 792756d4432e598d0f3f5368672fd1fc2d0a8b37 Mon Sep 17 00:00:00 2001 From: Michael Feng Date: Sun, 13 Sep 2026 21:19:54 -0700 Subject: [PATCH 48/48] (docs) the null holds on a second market and eight windows The single-window numbers in section 15 were one slice of one market, which is the weakness that flipped a sign earlier the same day -- and it applied to the finding as much as to what it displaced. Four windows each of XYZ:DRAM-USD and ZEC-USD, two fee families, one regime where every variant earned and one where every variant lost: the fly with real reinforcement beats randomly-signed reinforcement in exactly two windows of four, on both markets. A reader can judge that without trusting the statistic. Also recorded: on the volatile stretch every variant made money, so the geometry earns and it is the connectome contribution that is unmeasurable. --- agents/market_making_fly/README.md | 44 ++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/agents/market_making_fly/README.md b/agents/market_making_fly/README.md index 0393b5915..4a132c351 100644 --- a/agents/market_making_fly/README.md +++ b/agents/market_making_fly/README.md @@ -523,21 +523,35 @@ Copied in spirit from stonkfly's `docs/model.md`, because the same limits hold: ### What the replay controls established, 2026-09-13 `fly_replay` walks a pinned candle series with the same brain, decoder and -geometry, changing one setting per variant. On 327 ticks of XYZ:DRAM-USD at -5 m, with the position cap taken from the controller rather than chosen: - -| against `live` | mean per-tick | final gap | t | -|---|---|---|---| -| shuffled reinforcement | +0.00013 | +0.04 | **+0.24** | -| frozen plasticity | +0.00165 | +0.54 | +1.06 | -| valence disconnected | +0.00307 | +1.00 | +1.20 | - -**The fly with real P&L reinforcement is not distinguishable from the fly whose -reinforcement sign is random.** That is the control this design has always said -it lacked, and it now exists and does not separate them. Nor does freezing the -memory rule entirely. One market, one window, and a fill model that assumes a -touched price was ours — so this is not proof of a null, but it is the first -evidence on the question and it points at one. +geometry, changing one setting per variant. Two markets, four contiguous +windows each, 327 ticks per window, every variant on its own freshly seeded +brain: + +| market | against `live` | pooled per-tick | pooled t | windows won | +|---|---|---|---|---| +| XYZ:DRAM-USD (1.3 bp fee, 13.7 bp bars) | shuffled reinforcement | +0.00035 | +0.17 | **2 / 4** | +| | frozen plasticity | +0.00136 | +0.65 | 3 / 4 | +| ZEC-USD (2.5 bp fee, 41.3 bp bars) | shuffled reinforcement | −0.00501 | −0.80 | **2 / 4** | +| | frozen plasticity | −0.00813 | −1.11 | 1 / 4 | + +**The fly with real P&L reinforcement beats the fly whose reinforcement sign is +random in exactly half the windows, on both markets.** That is a coin flip. It +is the control this design has always said it lacked, it exists now, and it +does not separate them — on two fee families, in a regime where the strategy +made money (DRAM, all variants +1.7 to +1.8) and one where it lost (ZEC, all +variants −12.5 to −14.6). Freezing the memory rule outright does not separate +them either, and on ZEC the frozen fly was ahead. + +The fill model still assumes a touched price was ours, and eight windows of two +markets is not the last word. But the earlier single-window figures this +replaces (t = +0.24 against shuffled) were one slice of one market, and the +multi-window result is the one to quote: a reader can judge "2 of 4 windows" +without trusting the statistic at all. + +What this does *not* say is that the market maker does not work. On DRAM's +volatile stretch every variant earned, which is the geometry earning — quote +levels placed against how far the market travels. The connectome's contribution +to that is what is unmeasurable. Two measurement lessons are worth more than the numbers: