One engine, the whole crypto market: rewire onto the merged Crocodile - #5
Merged
Conversation
crypcodile (crypto) and stockodile (US equities) merged into a single `crocodile` package. FlowMap now pins that one dependency instead of two, with the `market` extra, which is what turns the crypto side from a hand-written venue list into the whole market: ccxt. The compat shims the merge ships are top-level `__init__.py` only — they re-export the package root and warn, but have no submodules — so every `crypcodile.exchanges.*` import was a hard break with no grace path. The migration is real; this commit is only its dependency half. The bundle and CI import checks were verifying a strict subset of what booting the server already proves, while `import crocodile` loads none of the 12 submodules the server actually uses. They now cover the LAZY closure instead — above all the equity providers (pandas, pyarrow, yfinance, bs4), reached only on a real `equity:` subscribe. A bundle that lost pyarrow would previously have passed every gate and died on the first equity symbol. `aiohttp`/`certifi` are declared explicitly too: first-party code imports them directly rather than inheriting them. Version 1.4.0: the installers' contents change materially (+ccxt, ~+13 MB compressed), so they must not be published over v1.3.0's artifacts. Known release risk, documented in bundle-python.sh: ccxt pins `cryptography==49.0.0` exactly, and that release publishes no macOS x86_64 wheel, so the Intel leg builds it from sdist. Rust is installed before that step for Tauri's sake, so the toolchain is there — but that path has never run on `macos-15-intel`. Dry-run the workflow before tagging.
…ne has Two halves of one change, because they touch the same seams. THE SCHEMA. The merge unified three record unions into one, so the renames had to be read for MEANING, not matched by name: crypto `exchange` and equity `provider` both became `source`; `exchange_ts` became `source_ts`; equity `Trade.size` became `amount`; `Bar` is gone (use `OHLCV`, and `trade_count` is `num_trades`); `asset_class` is required; and `buy_volume`/`sell_volume` went from `0.0` to `None`, so passing them through `float()` raises and a venue with no taker split now says so instead of drawing a fake all-sell bar. THE VENUES. Three hardcoded crypto markets become 104. Ten connectors are hand-written, five of those are order-flow readers; every other ccxt id falls through to the universal connector. That connector re-reads the whole book each tick instead of streaming diffs, which is a real fidelity difference, so it is a badge (`L2` vs `L2-snapshot`) and a behaviour: a snapshot-driven feed does not fabricate a `gap` marker per poll, and it dedups by trade id because the REST poll has no `since` cursor and would otherwise re-emit its whole window every interval, inflating tape and CVD under a `tape: "tick"` badge. Symbols are enumerated from the venue itself, native-first — the engine's own enumerator routes through ccxt even for native venues, which would hand the picker `ETH/BTC` for a connector that wants `ETHBTC`. Either spelling now works: `resolve_symbol` translates through ccxt's market table, which carries both. Backfill: the engine's REST klines cover Binance alone, so ccxt `fetchOHLCV` serves the other 103. The Binance path gets the certifi hardening the live feed already had (without it every native REST call fails TLS on stock macOS framework Python and looks like "venue has no symbols"), which also fixes the engine wiring usdm/coinm klines to the SPOT base. Routing was copy-pasted in two places; it is one `feeds/router.py` now. MEASURED, NOT ASSUMED — the capability table is only honest if someone asks the venue. Against the live public sockets: binance usdm `btcusdt@forceOrder` -> accepted okx `liq-orders`+instId -> "channel doesn't exist" bybit linear `liquidation.BTCUSDT` -> "handler not found" Bybit sends all topics in one frame, so that one dead topic was making it reject the ENTIRE subscribe: `bybit-linear` and `bybit-inverse` streamed zero events, and `conn.run()` never raised, so the session's degraded path never fired. Two venue segments were silently dead. Both venues are out of the liquidation table now (engine-side spellings — re-measure on the next pin bump), and both stream again. Deribit gains the marker instead: it flags liquidations on the ordinary trades stream we already take, so under-promising was a lie in the other direction. `<exchange>-<segment>` validates BOTH halves now. It used to check only the exchange, so `binance-<anything>` passed — each distinct string a fresh session key, cache entry, recording directory and outbound venue request. Segments are a closed set; a venue that cannot use one rejects it rather than silently dropping it and labelling spot data as a perp. Enumeration is bounded (25 s) — the engine's readers have no wall-clock limit of their own and were free to hold a per-market lock for minutes inside a request handler. A failed listing is remembered for 30 s, not 15 minutes, and a surviving last-good list is re-stamped so an outage stops re-fetching on every keystroke. The clock rename is now pinned by tests that can actually fail: swapping `source_ts` for `local_ts` used to leave all 389 tests green. It fails 5.
With three crypto venues a switch was rare. With 104 it is the dominant interaction, and four kinds of per-session state were surviving it. THE DEDUP CURSOR. `Connection.lastColSeq` and `epochMap` were never cleared. Every new session's grid restarts at epoch 0 with a low col_seq and its whole attach snapshot arrives `final=true`, so the previous symbol's cursor discarded it — the chart showed a right-edge sliver with no history. Cleared in `subscribe()` and ONLY when the stream identity differs: a transport reconnect re-enters through `sendSubscribe()`, where that same cursor is exactly what dedups the re-sent snapshot. THE CLOSED BANNER. Nothing cleared `feedState`. The server broadcasts `Status` only on a TRANSITION and a healthy session starts `live`, so a fresh crypto session sends none at all and a stale `closed` from an after-hours equity symbol survived forever. THE TAPE. Clearing the buffers when we ASK for a new symbol is not enough: frames for the old one are still in flight and refill them. On a liquid name the next prints push them out within a second; on a thin ccxt pair nothing ever does, so the tape sat there showing another instrument's trades at another instrument's prices. The buffers are now also cleared when `sessionId` changes — the moment the stream provably swapped. THE RING. Same race, worse consequence. `onDepthColumn` had no session gate, so a pre-Hello column sized the ring at the PREVIOUS symbol's row count; every later column then failed the `rows !== ring.rows` check and returned, with nothing left to recreate it. Crypto grids are 2048 rows and equity 4096, so a crypto→equity switch blanked the heatmap until the next switch. A gate now drops stale columns, and — so it can never latch shut and never draw — it also opens on any column whose geometry the store can size authoritatively, and a row-count change rebuilds the ring instead of skipping. Two smaller ones from the same family: the session reset key omitted `band`, though the band IS subscription identity and `deep` is a 4096-row grid, so switching bands left a stale-geometry ring while the comment claimed the opposite; and `normSeed` was missing from the reset block, which matters more than its neighbours because the renderer latches it once — a survivor normalised the entire new session against the old symbol's density scale. `protocolVersion` was deliberately left: it describes the server build, not the session, and has no consumer.
A flat symbol list worked for 40 curated pairs. It cannot present 104 venues with thousands of instruments each, so the palette gains a venue scope: `venue ▾` turns the same input into a filter over every market string, ordered native-first, and picking one enumerates that venue live. Opening stays instant — the bundled directory and movers, no blocking fetch. Enumeration is opt-in because a venue's first read takes seconds, so it gets a named loading state rather than an empty list, results are memoised per venue, and switching scope aborts the in-flight request. A failed listing is no longer cached as "empty". It used to be: `getJson` returns null for both a 502 and an offline socket, that became `[]` in a ref-Map that outlives the palette, and the truthy empty array short-circuited every retry — one transient failure blanked a venue until the app restarted, wearing the same words as a genuinely empty one. Now nothing is cached on failure, re-picking retries, and the two states say different things. The capability chips had two classifiers that disagreed. `SYNTH_PROFILE` read as REAL depth in the palette and synthetic in the top bar; `TAPE POLL` and `SIDE INFERRED` disagreed the same way, with the top bar honest each time. Both surfaces now share one function, and absence gets its own tier — finnhub's equity depth is literally `N/A`, and it was inheriting the real-depth accent with no tooltip. Six reachable depth strings, six honest renderings.
One engine, not two. And the numbers are the shipped ones, checked against a running server rather than against a constant: `/api/venues` returns 106 rows — 104 crypto, plus equity and sim. Five crypto venues carry `L2`, not ten: the engine has ten hand-written connectors, but five of them read CoinGecko candles or on-chain pool events and cannot serve an order book at all, so they are not offered.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
The version was hand-maintained in six places and three had already come apart: Cargo.toml said 1.4.0, Cargo.lock 1.3.0, and build-dmg.sh a third 1.3.0 of its own — so the disk image was about to be *named* for one release while containing another. package-lock.json had never been given a version at all (0.0.0). Set every one of them to 1.3.1 and delete the two copies that had no business existing: build-dmg.sh now reads the version out of tauri.conf.json, the same field the bundler stamps into the app. The DMG install window is the one place a version can still rot, because there it is pixels — make-bg.py renders it into the background plate. It now writes a RENDERED_VERSION stamp beside the art, and build-dmg.sh refuses to build a disk image whose window advertises a different version than the app inside it, quoting the regeneration command in the error.
Windows on ARM was the one platform with no installer: Snapdragon X
laptops, ARM Surfaces and Windows 11 under Parallels on Apple Silicon
could only run the x64 build under emulation — an emulated CPython under
an emulated webview, for a renderer whose whole point is frame cost.
`windows-11-arm` is a GA GitHub-hosted image, so the leg is a native
build and needs no `--target`: the bundle lands in the same path as every
other runner and the existing smoke test and upload globs cover it
unchanged. python-build-standalone publishes an
`aarch64-pc-windows-msvc` install_only runtime, and setup-uv already
resolves `uv-aarch64-pc-windows-msvc`.
What is NOT free is the wheel closure. Of the 75 locked distributions,
71 have a win_arm64 wheel; four do not, and each is handled by removing
exactly as much as the ARM64 bundle can live without:
pyarrow no win_arm64 wheel in ANY release. Reachable only
through crocodile's Arrow-IPC export helpers, which
FlowMap never imports — its recording layer is
polars, whose parquet codec is native Rust. Dropped,
and the import gate is told so explicitly rather
than being quietly weakened.
zlib-ng a ccxt decompression speed-up imported inside
aiohttp-fast-zlib `try: … except ImportError: pass`. Absence is a
supported configuration; ARM64 uses stdlib zlib.
cryptography NOT droppable — ccxt imports it at the top of
base/exchange.py. 46.0.3 is the last release that
still ships a win_arm64 wheel (47.0.0 onward are
win_amd64/win32 only) and ccxt only uses long-stable
hazmat primitives, so it is pinned back for this
target alone.
uvloop already excluded by its own marker on Windows.
The exception list is fail-loud. If a name it drops leaves the lock, or
cryptography's locked version moves off 49.0.0, the build stops and says
so instead of shipping a dependency set that no longer matches the
comment describing it.
Also: every leg now writes its produced installers and their sizes into
the job summary, so the README's download table has a source that is not
a guess — and because dry-run builds report it too, the table can be
refreshed before the tag is cut.
The first native windows-11-arm build got all the way through: every one
of the 60 wheels installed on ARM64, cryptography 46.0.3 and all, and the
server's own module imported. Then the import gate stopped it —
MISSING: crocodile.equity.client.collect: No module named 'pyarrow'
— which is exactly what that gate is for. pyarrow was dropped because
nothing here asks for Arrow, but `crocodile.equity.client.__init__`
imported `export`, and `export` imported pyarrow at module scope. So a
format FlowMap never uses decided whether live equity collection was
importable, and the ARM64 bundle would have booted green and died on the
first `equity:AAPL`.
Crocodile 2e22c90 defers it: both Arrow writers now load pyarrow at the
point of use via `crocodile.core.optional.require`, raising ConfigError
naming pyarrow when `fmt="arrow"` is asked for without it. Re-pinned here,
which also picks up the eight commits since 8ff6a75 — including the
crypto venue-registry lookup fix and the py.typed packaging fix.
Verified against a scratch environment built from this lock with the same
exceptions the ARM64 leg applies (no pyarrow, no zlib-ng, cryptography
46.0.3): all 26 modules of the import gate load, and asking for Arrow
raises the named error. 402 server tests green on the new engine.
app/README.md documents the ARM64 leg and its exception table.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FlowMap depended on two market-data libraries —
crypcodile(crypto) andstockodile(US equities). They merged into onecrocodilepackage, which also unified three record unions into one. This rewires FlowMap onto it and, in the same pass, lifts the crypto venue ceiling from 3 to 104.What changed
crocodile[market]@8ff6a75)binance-spot,binance-usdm,okx)feeds/router.pyGate
--workers=1)Why the venue count is 104 and not 109
The engine ships ten hand-written connectors, but five of them read CoinGecko daily candles or on-chain pool events — no book, no tick tape. They are excluded from the venue set itself rather than merely hidden in the picker, so the router, the backfill dispatch, quote routing and discovery cannot disagree about what is subscribable.
Capability honesty
A ccxt venue re-reads the whole book each tick instead of streaming diffs. That is carried as
L2-snapshotvsL2— a distinct badge with its own tooltip, never a silent downgrade — and it changes behaviour: no fabricatedgapmarker per poll, and trade-id dedup because the REST poll has nosincecursor.The liquidation table was measured against the live public sockets, not read off the docs:
Bybit sends all topics in one frame, so that one dead topic made it reject the entire subscribe —
bybit-linearandbybit-inversewere streaming zero events, andconn.run()never raised, so the session's degraded path never fired. Both stream again.Review record
Four adversarial referees reviewed the branch; all returned NOT READY and every blocking finding is fixed in these commits. The most useful one was a mutation proof: swapping the venue clock (
source_ts) for the receive clock (local_ts) left all 389 tests green. That rename is the single most load-bearing line of the migration. It now fails 5 tests.Known, not fixed
columns_from_candlesreturnsNonefor a banded grid, and the client's default isdeep. Measured on kraken 1INCH/EUR: 120 candles fetched →native120 columns,wide/deep0. Pre-existing and affects every symbol equally; fixing it means changing epoch anchoring in the grid.cryptography==49.0.0exactly and that release has no macOS x86_64 wheel, somacos-15-intelbuilds it from sdist. Rust is installed before that step, but the path has never run on that runner — dry-run the workflow before tagging.