diff --git a/.github/workflows/claude.yml b/.github/workflows/claude.yml index d300267f1..6b15fac7a 100644 --- a/.github/workflows/claude.yml +++ b/.github/workflows/claude.yml @@ -46,5 +46,5 @@ jobs: # Optional: Add claude_args to customize behavior and configuration # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md # or https://code.claude.com/docs/en/cli-reference for available options - # claude_args: '--allowed-tools Bash(gh pr:*)' + # claude_args: '--allowed-tools Bash(gh pr *)' diff --git a/planning/MARKET_INTERFACE.md b/planning/MARKET_INTERFACE.md new file mode 100644 index 000000000..f5d9762b6 --- /dev/null +++ b/planning/MARKET_INTERFACE.md @@ -0,0 +1,438 @@ +# Market Data Interface + +The unified Python API for retrieving stock prices in FinAlly. One interface, +two implementations: the **Massive API** when `MASSIVE_API_KEY` is set, the +**GBM simulator** otherwise. + +Companion documents: `MASSIVE_API.md` (the real API), `MARKET_SIMULATOR.md` +(the simulation model). + +--- + +## 1. Design Goal + +Every consumer of price data in FinAlly — the SSE stream, portfolio valuation, +trade execution, the LLM's context builder — must be **completely unaware of +where prices come from**. Swapping a simulated feed for a live one is an +environment-variable change, not a code change. + +This is achieved with two ideas: + +1. **A producer interface** (`MarketDataSource`) that both backends implement. +2. **A shared cache** (`PriceCache`) that decouples producers from consumers. + +``` + ┌──────────────────────────────────────────┐ + │ MarketDataSource (ABC) │ + │ start / stop / add_ticker / │ + │ remove_ticker / get_tickers │ + └──────────────────────────────────────────┘ + ▲ ▲ + │ │ + ┌───────────┴────────┐ ┌─────────┴──────────┐ + │ SimulatorDataSource│ │ MassiveDataSource │ + │ GBM, 500ms ticks │ │ REST poll, 15s │ + └───────────┬────────┘ └─────────┬──────────┘ + │ │ + └──────── writes ──────┘ + │ + ▼ + ┌───────────────────────┐ + │ PriceCache │ + │ thread-safe, in-mem │ + │ + version counter │ + └───────────┬───────────┘ + │ reads + ┌────────────────┼────────────────┐ + ▼ ▼ ▼ + SSE /api/stream Portfolio Trade execution + /prices valuation & LLM context +``` + +**The cache is the only contract consumers depend on.** No consumer ever holds +a reference to a `MarketDataSource`. + +--- + +## 2. Module Layout + +Located at `backend/app/market/`: + +| Module | Responsibility | +|---|---| +| `models.py` | `PriceUpdate` — the immutable unit of price data | +| `interface.py` | `MarketDataSource` — the abstract producer contract | +| `cache.py` | `PriceCache` — thread-safe store, versioned | +| `factory.py` | `create_market_data_source()` — environment-driven selection | +| `simulator.py` | `GBMSimulator` + `SimulatorDataSource` | +| `massive_client.py` | `MassiveDataSource` — REST poller | +| `seed_prices.py` | Seed prices, GBM params, correlation groups | +| `stream.py` | `create_stream_router()` — SSE endpoint factory | + +Public surface, re-exported from `app.market`: + +```python +from app.market import ( + PriceUpdate, + PriceCache, + MarketDataSource, + create_market_data_source, + create_stream_router, +) +``` + +--- + +## 3. `PriceUpdate` — The Data Unit + +An immutable, frozen, slotted dataclass. Immutability matters: instances are +handed to SSE serializers and portfolio math concurrently, and nothing should +be able to mutate a price after the fact. + +```python +@dataclass(frozen=True, slots=True) +class PriceUpdate: + ticker: str + price: float + previous_price: float + timestamp: float = field(default_factory=time.time) # Unix seconds + + @property + def change(self) -> float: ... # price - previous_price + @property + def change_percent(self) -> float: ... # % change, 0.0 if previous == 0 + @property + def direction(self) -> str: ... # "up" | "down" | "flat" + + def to_dict(self) -> dict: ... # JSON/SSE serialization +``` + +Design decisions: + +- **Derived values are properties, not fields.** `change`, `change_percent`, + and `direction` cannot drift out of sync with the prices they derive from. +- **`previous_price` is the previous *tick*, not the previous *close*.** This + drives the frontend's green/red flash animation. Day-over-day change is a + separate concern computed by the portfolio layer. +- **`timestamp` is Unix seconds (float)**, normalized regardless of source. + Massive's nanosecond timestamps are converted at the boundary. +- **Division-by-zero guard** on `change_percent` returns `0.0` rather than + raising — a price feed glitch must not propagate an exception into the SSE + generator. + +`to_dict()` output — this is the wire format the frontend consumes: + +```json +{ + "ticker": "AAPL", + "price": 190.42, + "previous_price": 190.35, + "timestamp": 1785432000.123, + "change": 0.07, + "change_percent": 0.0368, + "direction": "up" +} +``` + +--- + +## 4. `MarketDataSource` — The Producer Contract + +```python +class MarketDataSource(ABC): + @abstractmethod + async def start(self, tickers: list[str]) -> None: ... + @abstractmethod + async def stop(self) -> None: ... + @abstractmethod + async def add_ticker(self, ticker: str) -> None: ... + @abstractmethod + async def remove_ticker(self, ticker: str) -> None: ... + @abstractmethod + def get_tickers(self) -> list[str]: ... +``` + +### Contract semantics + +| Method | Guarantee | +|---|---| +| `start(tickers)` | Begins a background task writing to the cache. Called exactly once. Populates the cache with initial prices **before returning**, so the first SSE client never sees an empty payload. | +| `stop()` | Cancels the background task and releases resources. Idempotent — safe to call multiple times, including when never started. After `stop()`, the source never writes to the cache again. | +| `add_ticker(t)` | Idempotent. Ticker appears in the next update cycle (or immediately, for the simulator). | +| `remove_ticker(t)` | Idempotent. **Also removes the ticker from the cache**, so a de-watchlisted symbol stops appearing in the SSE payload. | +| `get_tickers()` | Synchronous (no I/O). Returns a copy — callers cannot mutate internal state. | + +### Why this shape + +- **Async lifecycle, sync accessor.** `start`/`stop`/`add`/`remove` are async + because the Massive implementation performs network I/O and both manage + asyncio tasks. `get_tickers()` is a pure in-memory read, so forcing `await` + on it would be noise. +- **No `get_price()` on the interface.** Deliberately absent. If sources + exposed price reads, consumers would couple to the source and the cache + would become an implementation detail rather than the contract. Prices are + read from `PriceCache`, always. +- **Push, not pull.** Sources write to the cache on their own schedule. + Consumers never trigger a fetch, so a slow API can never block a request. + +--- + +## 5. `PriceCache` — The Consumer Contract + +```python +class PriceCache: + def update(self, ticker: str, price: float, + timestamp: float | None = None) -> PriceUpdate: ... + def get(self, ticker: str) -> PriceUpdate | None: ... + def get_price(self, ticker: str) -> float | None: ... + def get_all(self) -> dict[str, PriceUpdate]: ... + def remove(self, ticker: str) -> None: ... + + @property + def version(self) -> int: ... + + def __len__(self) -> int: ... + def __contains__(self, ticker: str) -> bool: ... +``` + +### Thread safety + +Guarded by a `threading.Lock`, not an `asyncio.Lock`. This is deliberate: the +Massive client is synchronous and runs inside `asyncio.to_thread(...)`, so +writes genuinely originate from a worker thread while reads happen on the event +loop. An asyncio lock would not protect that boundary. + +Every method holds the lock for the shortest possible span. `get_all()` returns +a **shallow copy** of the dict — safe to iterate outside the lock because +`PriceUpdate` is frozen. + +### The version counter + +`version` is a monotonic integer incremented on every `update()`. It exists so +the SSE generator can answer "has anything changed since I last sent?" in O(1) +without diffing dicts or comparing timestamps. + +```python +last_version = -1 +while True: + if price_cache.version != last_version: + last_version = price_cache.version + yield f"data: {json.dumps(...)}\n\n" + await asyncio.sleep(0.5) +``` + +This keeps idle connections silent — when the simulator is stopped or the +Massive poll fails, no redundant frames are pushed. + +### First-update semantics + +On the first `update()` for a ticker, `previous_price` is set equal to `price`, +so `direction` is `"flat"` and `change` is `0.0`. The frontend therefore never +flashes a spurious green/red on page load. + +Prices are rounded to 2 decimal places on write. Rounding at the cache boundary +means every consumer sees identical values — no display/execution mismatch +where the UI shows $190.42 but a trade fills at $190.41999999. + +--- + +## 6. `create_market_data_source()` — Selection + +```python +def create_market_data_source(price_cache: PriceCache) -> MarketDataSource: + api_key = os.environ.get("MASSIVE_API_KEY", "").strip() + if api_key: + logger.info("Market data source: Massive API (real data)") + return MassiveDataSource(api_key=api_key, price_cache=price_cache) + logger.info("Market data source: GBM Simulator") + return SimulatorDataSource(price_cache=price_cache) +``` + +Selection rules: + +| `MASSIVE_API_KEY` | Result | +|---|---| +| Unset | Simulator | +| Empty string | Simulator | +| Whitespace only | Simulator (`.strip()` handles this) | +| Non-empty | Massive | + +The `.strip()` matters in practice: a `.env` file containing `MASSIVE_API_KEY=` +followed by a trailing space would otherwise select the real API with a garbage +key, and the app would silently show no prices. + +The factory returns an **unstarted** source — the caller owns the lifecycle. +This keeps the factory synchronous and makes it trivial to unit-test selection +logic without spawning background tasks. + +--- + +## 7. Application Wiring + +```python +from contextlib import asynccontextmanager +from fastapi import FastAPI +from app.market import PriceCache, create_market_data_source, create_stream_router + + +@asynccontextmanager +async def lifespan(app: FastAPI): + cache = PriceCache() + source = create_market_data_source(cache) + + tickers = load_watchlist_tickers() # from SQLite + await source.start(tickers) + + app.state.price_cache = cache + app.state.market_source = source + try: + yield + finally: + await source.stop() + + +app = FastAPI(lifespan=lifespan) +app.include_router(create_stream_router(app.state.price_cache)) +``` + +Both the cache and the source live on `app.state` rather than in module-level +globals, which keeps tests isolated — each test app gets its own cache. + +### Keeping the source in sync with the watchlist + +Watchlist mutations must update the database *and* the data source: + +```python +@router.post("/api/watchlist") +async def add_to_watchlist(body: TickerBody, request: Request): + ticker = body.ticker.upper().strip() + db_add_ticker(ticker) + await request.app.state.market_source.add_ticker(ticker) + return {"ok": True} + + +@router.delete("/api/watchlist/{ticker}") +async def remove_from_watchlist(ticker: str, request: Request): + ticker = ticker.upper().strip() + db_remove_ticker(ticker) + await request.app.state.market_source.remove_ticker(ticker) + return {"ok": True} +``` + +Ticker normalization (`.upper().strip()`) happens at the API boundary *and* +defensively inside both sources, since the cache is keyed by exact string. + +--- + +## 8. Reading Prices — Consumer Patterns + +```python +cache: PriceCache = request.app.state.price_cache + +# Single price for trade execution +price = cache.get_price("AAPL") +if price is None: + raise HTTPException(400, "No price available for AAPL") + +# Full update with direction, for display +update = cache.get("AAPL") + +# Portfolio valuation across all positions +prices = cache.get_all() +total = cash + sum( + pos.quantity * prices[pos.ticker].price + for pos in positions + if pos.ticker in prices +) +``` + +**Always handle `None`.** A ticker can be absent from the cache when it was +just added and the first poll has not landed, or when a Massive poll failed +before any successful fetch. Trade execution must reject rather than assume a +price. + +--- + +## 9. SSE Streaming Layer + +`create_stream_router(cache)` returns a FastAPI `APIRouter` exposing +`GET /api/stream/prices`. + +``` +retry: 1000 + +data: {"AAPL": {"ticker":"AAPL","price":190.42,...}, "GOOGL": {...}} + +data: {...} +``` + +Behaviour: + +- Emits a `retry: 1000` directive first, so `EventSource` reconnects after 1s. +- Sends **all** tracked tickers in each frame, keyed by ticker. Simpler for the + client than per-ticker events, and at ~10 tickers the payload is trivial. +- Polls the cache every 500ms and only emits when `version` changed. +- Detects disconnect via `await request.is_disconnected()` and exits the loop. +- Sets `X-Accel-Buffering: no` and `Cache-Control: no-cache` so the stream is + not buffered by an intermediate proxy. + +The 500ms stream cadence is independent of the source cadence. With the +simulator (500ms ticks) the client sees a frame per tick. With Massive (15s +polls) the version is unchanged between polls, so frames are emitted only when +new data actually arrives — no wasted bandwidth. + +--- + +## 10. Implementation Comparison + +| Aspect | `SimulatorDataSource` | `MassiveDataSource` | +|---|---|---| +| Update cadence | 500ms | 15s (configurable) | +| Mechanism | In-process GBM step | REST poll, batched | +| Blocking I/O | None | Yes — wrapped in `asyncio.to_thread` | +| New ticker latency | Immediate (seeded on add) | Next poll cycle | +| Failure mode | Logs and continues | Logs and continues; cache goes stale | +| External dependency | None | Network + API key + paid plan | +| Cost | Free | Snapshot endpoints need Starter+ | + +Both share the same failure philosophy: **the loop never dies**. Exceptions are +caught, logged, and the next cycle proceeds. A market data problem degrades to +stale prices, never to a 500 on the SSE endpoint. + +--- + +## 11. Testing Strategy + +The interface makes each layer independently testable: + +- **`PriceCache`** — pure unit tests: update/get/remove, version increments, + first-update flat semantics, rounding, concurrent writes from threads. +- **`PriceUpdate`** — property math, zero-division guard, `to_dict()` shape. +- **Factory** — `monkeypatch.setenv` across unset/empty/whitespace/valid, + asserting the returned type. No network, no tasks. +- **`SimulatorDataSource`** — start, let a few ticks elapse, assert the cache + populated and prices moved; assert `stop()` halts writes. +- **`MassiveDataSource`** — mock `RESTClient.get_snapshot_all` with realistic + `TickerSnapshot` objects built via `TickerSnapshot.from_dict(raw_json)`. + Building from raw JSON rather than hand-constructing the dataclass is what + catches field-name errors like `sip_timestamp` vs `timestamp`. +- **Consumers** — inject a `PriceCache` pre-populated via `update()`. No data + source needed at all to test portfolio math or trade execution. + +A conformance test parametrized over both implementations asserts they satisfy +the contract identically (idempotent `stop()`, `remove_ticker` clears the +cache, `get_tickers()` reflects mutations). + +--- + +## 12. Extending to a New Provider + +To add, say, an Alpaca or Finnhub feed: + +1. Implement `MarketDataSource` in a new module. +2. Convert that provider's price and timestamp into Unix-seconds floats and + call `cache.update(...)`. +3. Add a branch to `create_market_data_source()`. +4. Add it to the conformance test parametrization. + +No consumer changes. That property is the entire point of the design. diff --git a/planning/MARKET_SIMULATOR.md b/planning/MARKET_SIMULATOR.md new file mode 100644 index 000000000..69d8cad33 --- /dev/null +++ b/planning/MARKET_SIMULATOR.md @@ -0,0 +1,419 @@ +# Market Simulator + +The default market data source for FinAlly. Generates realistic, correlated, +continuously-moving stock prices with no external dependency, no API key, and +no cost. + +Companion documents: `MARKET_INTERFACE.md` (the contract it implements), +`MASSIVE_API.md` (the real-data alternative). + +--- + +## 1. Why Simulate + +The simulator is the **default**, not a fallback. Most people running FinAlly +will never set `MASSIVE_API_KEY`, and the simulator is a better experience for +them: + +| Property | Benefit | +|---|---| +| Always moving | Real markets are closed nights, weekends, and holidays — a demo showing frozen prices looks broken | +| 500ms updates | Matches the UI's flash-animation design; real free-tier data is 15-minute delayed | +| No API key | Zero-setup `docker run` | +| No rate limits | Add 50 tickers, no throttling | +| Deterministic under seed | Reproducible tests | +| Free | Massive's snapshot endpoints require a paid plan | + +The design goal is **plausibility, not prediction**. Prices must look like a +trading terminal: drifting, occasionally jumping, with tech names moving +together. Nobody should mistake this for a forecast. + +--- + +## 2. The Model — Geometric Brownian Motion + +GBM is the standard model for equity price paths (the basis of Black-Scholes). +It has the two properties that matter here: prices stay **strictly positive**, +and **returns** rather than absolute prices are normally distributed, so a $800 +stock and a $15 stock both move in plausible percentage terms. + +The discrete-time update: + +``` +S(t+dt) = S(t) · exp( (μ − σ²/2)·dt + σ·√dt·Z ) + └──────┘ └────────────┘ └───────┘ + current drift diffusion +``` + +| Symbol | Meaning | +|---|---| +| `S(t)` | Current price | +| `μ` | Annualized drift (expected return) | +| `σ` | Annualized volatility | +| `dt` | Time step, as a fraction of a trading year | +| `Z` | Correlated standard normal draw | + +The `−σ²/2` term is the Itô correction. Without it the *median* path drifts +below the intended `μ`, because `E[exp(X)] ≠ exp(E[X])` for a normal `X`. Its +presence is what makes `μ` mean "expected annual return" rather than an +arbitrary tuning knob. + +### Choosing `dt` + +`dt` must be expressed in the same time units as `μ` and `σ`, which are +annualized. A trading year is: + +``` +252 trading days × 6.5 hours/day × 3600 s/hour = 5,896,800 seconds +``` + +So a 500ms tick is: + +```python +TRADING_SECONDS_PER_YEAR = 252 * 6.5 * 3600 # 5,896,800 +DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR # ≈ 8.479e-8 +``` + +Using wall-clock seconds per year (31.5M) instead would understate volatility +by roughly 2.3×, making the terminal look sleepy. Anchoring to *trading* time +means one hour of watching FinAlly produces about as much price action as one +hour of watching a real market. + +### Verified magnitudes + +Per-tick standard deviation is `σ·√dt`. Measured against the shipped seed +values: + +| Ticker | σ | Per-tick move | Per-tick $ | Over 1 min | Over 1 hr | +|---|---|---|---|---|---| +| AAPL | 0.22 | 0.0064% | $0.012 | 0.070% | 0.544% | +| TSLA | 0.50 | 0.0146% | $0.036 | 0.159% | 1.235% | +| NVDA | 0.40 | 0.0116% | $0.093 | 0.128% | 0.988% | +| JPM | 0.18 | 0.0052% | $0.010 | 0.057% | 0.445% | +| V | 0.17 | 0.0050% | $0.014 | 0.054% | 0.420% | + +An hour of TSLA moving ~1.2% and JPM ~0.45% is squarely in realistic +territory, and the sub-cent-to-few-cent per-tick moves produce the continuous +flicker the UI wants without prices visibly running away. + +--- + +## 3. Correlation via Cholesky Decomposition + +Independent random walks look wrong. In a real market, tech names move +together — when the sector sells off, the whole watchlist reddens at once. That +collective motion is most of what makes a terminal feel alive. + +### The technique + +To generate correlated normals with a target correlation matrix `C`: + +1. Compute the Cholesky decomposition `C = L·Lᵀ`, where `L` is lower-triangular. +2. Draw a vector `Z` of independent standard normals. +3. `Z_correlated = L · Z` now has correlation matrix exactly `C`. + +```python +z_independent = np.random.standard_normal(n) +z_correlated = self._cholesky @ z_independent +``` + +Each ticker then applies its *own* `σ` to its correlated draw, so tickers share +directional tendency while keeping individual volatility. + +### The correlation structure + +```python +CORRELATION_GROUPS = { + "tech": {"AAPL", "GOOGL", "MSFT", "AMZN", "META", "NVDA", "NFLX"}, + "finance": {"JPM", "V"}, +} + +INTRA_TECH_CORR = 0.6 # tech names move together +INTRA_FINANCE_CORR = 0.5 # banks/payments move together +CROSS_GROUP_CORR = 0.3 # broad market beta between sectors +TSLA_CORR = 0.3 # TSLA does its own thing +``` + +Pairwise resolution order: + +```python +if t1 == "TSLA" or t2 == "TSLA": return TSLA_CORR # 0.3 +if t1 in tech and t2 in tech: return INTRA_TECH_CORR # 0.6 +if t1 in finance and t2 in finance: return INTRA_FINANCE_CORR # 0.5 +return CROSS_GROUP_CORR # 0.3 +``` + +TSLA is checked **first**, before the tech-group test. It is a member of the +tech set but is deliberately decorrelated — it is the ticker most likely to be +doing something idiosyncratic, and giving it independence adds visual variety. + +Unknown tickers (anything a user adds) fall through to `CROSS_GROUP_CORR`, +which means a newly added symbol still participates in broad market moves. + +### Verified behaviour + +Building the 10-ticker default matrix and reconstructing `L·Lᵀ`: + +``` +AAPL/MSFT → 0.600 JPM/V → 0.500 AAPL/JPM → 0.300 TSLA/AAPL → 0.300 +minimum eigenvalue → 0.400 (positive definite ✓) +``` + +Empirical correlation of log returns over 20,000 simulated steps: + +| Pair | Target | Measured | +|---|---|---| +| AAPL / MSFT | 0.60 | 0.593 | +| JPM / V | 0.50 | 0.497 | +| AAPL / JPM | 0.30 | 0.298 | +| TSLA / AAPL | 0.30 | 0.295 | + +The model reproduces its target structure to within sampling error. + +> **Positive-definiteness constraint.** `np.linalg.cholesky` raises +> `LinAlgError` on a non-positive-definite matrix. The current values are safe +> (min eigenvalue 0.4), but arbitrary hand-tuned correlations can easily be +> invalid — e.g. A/B = 0.9, A/C = 0.9, B/C = 0.0 is not a realizable +> correlation matrix. Any change to these constants must be checked with +> `np.linalg.eigvalsh(corr).min() > 0`, and there is a unit test asserting the +> matrix builds for the default watchlist. + +--- + +## 4. Random Shock Events + +Pure GBM is smooth. Real markets gap on news. Each tick, each ticker has a +small chance of a discrete jump: + +```python +if random.random() < self._event_prob: # default 0.001 + shock_magnitude = random.uniform(0.02, 0.05) # 2-5% + shock_sign = random.choice([-1, 1]) + self._prices[ticker] *= 1 + shock_magnitude * shock_sign +``` + +Expected frequency with the default watchlist: + +``` +10 tickers × 2 ticks/sec × 0.001 = 0.02 events/sec ≈ one event every 50 seconds +``` + +Roughly one dramatic move per minute across the board — frequent enough that a +user watching for a minute sees something happen, rare enough that it reads as +an event rather than noise. The shock is symmetric (equal up/down probability), +so it adds variance without biasing long-run drift. + +--- + +## 5. Code Structure + +Two classes with a clean split of concerns: + +``` +GBMSimulator ← pure math, fully synchronous, no I/O, no asyncio + ├── step() → advance all tickers one tick, return {ticker: price} + ├── add_ticker() → seed price/params, rebuild Cholesky + ├── remove_ticker() → drop state, rebuild Cholesky + ├── get_price() → current price for one ticker + └── get_tickers() → tracked tickers + +SimulatorDataSource ← implements MarketDataSource, owns the asyncio task + ├── start() → build simulator, seed cache, launch _run_loop + ├── stop() → cancel task + ├── add_ticker() → delegate + seed cache immediately + ├── remove_ticker() → delegate + evict from cache + ├── get_tickers() → delegate + └── _run_loop() → step → write cache → sleep, forever +``` + +**`GBMSimulator` is deliberately free of asyncio and of `PriceCache`.** It is a +deterministic function of its state plus the RNG, which makes the math directly +unit-testable: seed numpy, call `step()` a thousand times, assert on the +distribution of returns. No event loop, no mocking. + +`SimulatorDataSource` handles everything stateful and asynchronous. This split +mirrors `MassiveDataSource`, where the REST client is likewise isolated behind +the async lifecycle. + +### The core loop + +```python +async def _run_loop(self) -> None: + while True: + try: + if self._sim: + prices = self._sim.step() + for ticker, price in prices.items(): + self._cache.update(ticker=ticker, price=price) + except Exception: + logger.exception("Simulator step failed") + await asyncio.sleep(self._interval) +``` + +The `try`/`except` sits **inside** the loop, so a transient failure (e.g. a +Cholesky error after a bad ticker add) logs and retries on the next tick rather +than silently killing the background task and freezing every price in the app. + +### Startup seeding + +`start()` writes initial prices to the cache **before** returning: + +```python +async def start(self, tickers: list[str]) -> None: + self._sim = GBMSimulator(tickers=tickers, event_probability=self._event_prob) + for ticker in tickers: + price = self._sim.get_price(ticker) + if price is not None: + self._cache.update(ticker=ticker, price=price) + self._task = asyncio.create_task(self._run_loop(), name="simulator-loop") +``` + +Without this, the first SSE frame after startup would be empty and the +watchlist would render blank for up to 500ms. `add_ticker()` seeds the same way, +so a newly added ticker shows a price instantly rather than after a tick. + +--- + +## 6. Seed Data + +`seed_prices.py` holds all tunable parameters, separate from the logic: + +```python +SEED_PRICES = { + "AAPL": 190.00, "GOOGL": 175.00, "MSFT": 420.00, "AMZN": 185.00, + "TSLA": 250.00, "NVDA": 800.00, "META": 500.00, "JPM": 195.00, + "V": 280.00, "NFLX": 600.00, +} + +TICKER_PARAMS = { + "AAPL": {"sigma": 0.22, "mu": 0.05}, + "GOOGL": {"sigma": 0.25, "mu": 0.05}, + "MSFT": {"sigma": 0.20, "mu": 0.05}, + "AMZN": {"sigma": 0.28, "mu": 0.05}, + "TSLA": {"sigma": 0.50, "mu": 0.03}, # high volatility + "NVDA": {"sigma": 0.40, "mu": 0.08}, # high volatility, strong drift + "META": {"sigma": 0.30, "mu": 0.05}, + "JPM": {"sigma": 0.18, "mu": 0.04}, # low volatility (bank) + "V": {"sigma": 0.17, "mu": 0.04}, # low volatility (payments) + "NFLX": {"sigma": 0.35, "mu": 0.05}, +} + +DEFAULT_PARAMS = {"sigma": 0.25, "mu": 0.05} +``` + +Volatilities are chosen to match each name's real-world character: TSLA the +most volatile at 0.50, NVDA next at 0.40 with the strongest drift, the payment +and banking names calmest at 0.17-0.18. The ordering is what a user notices — +TSLA visibly jumping around while V barely moves is the detail that sells the +simulation. + +### Unknown tickers + +A ticker not in `SEED_PRICES` (anything the user or the LLM adds) gets: + +```python +self._prices[ticker] = SEED_PRICES.get(ticker, random.uniform(50.0, 300.0)) +self._params[ticker] = TICKER_PARAMS.get(ticker, dict(DEFAULT_PARAMS)) +``` + +A random starting price in $50-300 and mid-range parameters. Note +`dict(DEFAULT_PARAMS)` — a **copy**. Sharing the dict would mean per-ticker +parameter tuning silently mutating the defaults for every other unknown ticker. + +--- + +## 7. Known Limitations + +Documented deliberately, since the simulator is the default experience. + +### Low-priced tickers barely move visibly + +Prices are rounded to 2 decimals at the cache boundary. When `σ·√dt·S` is well +under a cent, most ticks round to the same displayed price. Measured fraction +of ticks with **no visible change**: + +| Price level | Flat ticks | Visible move | +|---|---|---| +| $800 (NVDA) | 4.4% | 95.6% | +| $190 (AAPL) | 31.3% | 68.7% | +| $195 (JPM) | 35.3% | 64.7% | +| **$15** | **92.4%** | **7.6%** | + +A user adding a sub-$20 ticker sees a nearly frozen price and almost no flash +animation. The default watchlist is unaffected (cheapest is GOOGL at $175), but +this is a real edge for user-added penny-ish names. + +Mitigation if it matters: scale volatility upward for low-priced tickers when +assigning `DEFAULT_PARAMS`, or set a floor on per-tick movement. Not currently +implemented — the default watchlist doesn't hit it. + +### Other simplifications + +- **No mean reversion.** Prices random-walk; over a long session a ticker can + wander far from its seed. Acceptable for a demo, unrealistic over days. +- **No volume, bid/ask, or order book.** The plan specifies market orders with + instant fill, so none is needed. +- **No market hours.** Prices move 24/7 by design — a frozen weekend terminal + would look broken. +- **No day-open reference.** `PriceUpdate.previous_price` is the previous + *tick*. Day-change percentage (which Massive provides directly as + `todays_change_percent`) has no simulator equivalent; the portfolio layer + computes change against the session's first observed price instead. +- **Correlation is static.** Real correlations spike in a crash. Here they are + fixed constants. + +--- + +## 8. Testing + +The math/async split makes both halves straightforwardly testable. + +**`GBMSimulator` — pure, synchronous:** + +- Prices stay strictly positive across many thousands of steps. +- `step()` returns an entry for every tracked ticker, rounded to 2dp. +- With `np.random.seed(...)` fixed, output is reproducible. +- Empirical volatility of log returns ≈ `σ·√dt` for each ticker. +- Empirical correlation matches the target structure (verified above). +- Cholesky builds and is positive definite for the default watchlist. +- `add_ticker`/`remove_ticker` rebuild the matrix to the right dimension; + adding a duplicate is a no-op; removing an unknown ticker is a no-op. +- With `event_probability=1.0`, every tick applies a 2-5% shock; with `0.0`, + none ever does. +- Unknown tickers get a price in $50-300 and a *copy* of `DEFAULT_PARAMS`. +- Single-ticker and empty-ticker cases don't crash (Cholesky is `None` when + `n <= 1`; `step()` returns `{}` when empty). + +**`SimulatorDataSource` — async integration:** + +- `start()` populates the cache before returning. +- Prices change after letting several ticks elapse. +- `stop()` halts writes — cache version is stable afterward. +- `stop()` is idempotent, including when never started. +- `add_ticker()` seeds the cache immediately; `remove_ticker()` evicts it. +- An exception raised inside `step()` is logged without killing the loop. + +Tests use a short `update_interval` (e.g. 0.01s) so integration cases finish in +milliseconds rather than seconds. + +--- + +## 9. Tuning Guide + +| Goal | Change | +|---|---| +| More dramatic price action | Raise `sigma` in `TICKER_PARAMS` | +| Faster/slower updates | `SimulatorDataSource(update_interval=...)` | +| More/fewer shock events | `event_probability` (default `0.001`) | +| Stronger sector coupling | Raise `INTRA_TECH_CORR` — **re-verify positive definiteness** | +| Different starting prices | `SEED_PRICES` | +| Persistent upward market | Raise `mu` across `TICKER_PARAMS` | + +`update_interval` and `dt` are independent knobs. Changing the interval without +changing `dt` alters how much simulated time passes per real second — halving +the interval to 250ms while leaving `dt` at the 500ms value makes the market +run at half speed. To keep them consistent, derive `dt` from the interval: + +```python +dt = update_interval / TRADING_SECONDS_PER_YEAR +``` diff --git a/planning/MASSIVE_API.md b/planning/MASSIVE_API.md new file mode 100644 index 000000000..86a15b0bd --- /dev/null +++ b/planning/MASSIVE_API.md @@ -0,0 +1,474 @@ +# Massive API Reference (formerly Polygon.io) + +Reference for the Massive REST API as used by FinAlly's market data layer. + +> **Verification note.** Every field name, method signature, and default in this +> document was verified against the installed `massive` Python SDK by +> introspecting the dataclasses and `RESTClient.__init__`, and cross-checked +> against the published REST schemas at . Where the +> raw JSON field names differ from the SDK attribute names, both are given — +> this is the single most common source of bugs when working with this API. + +--- + +## 1. Background + +Polygon.io rebranded to **Massive** on 30 October 2025. + +| Item | Value | +|---|---| +| Base URL | `https://api.massive.com` | +| Legacy base URL | `https://api.polygon.io` (still supported for an extended period) | +| Python package | `massive` (renamed from `polygon-api-client`) | +| Install | `uv add massive` / `pip install -U massive` | +| Minimum Python | 3.9+ | +| Auth | `Authorization: Bearer ` — handled by the SDK | +| API key env var | `MASSIVE_API_KEY` | + +Existing Polygon.io API keys continue to work unchanged. + +--- + +## 2. Rate Limits & Plan Tiers + +| Tier | Price | Rate limit | Data recency | +|---|---|---|---| +| Basic (free) | $0 | **5 requests/minute** | End-of-day / delayed | +| Starter | ~$29/mo | Unlimited | 15-minute delayed | +| Developer | ~$79/mo | Unlimited | Real-time | +| Advanced | ~$199/mo | Unlimited + WebSocket | Real-time | + +"Unlimited" is not literally unbounded — Massive monitors usage and recommends +staying **under 100 requests/second**. Exceeding the free tier returns +**HTTP 429**. + +### What this means for FinAlly + +The free tier's 5 req/min is the binding constraint on our polling design. This +is why we use the **full market snapshot** endpoint: it returns *every requested +ticker in a single HTTP call*, so watchlist size does not affect our request +count. + +| Tier | Recommended poll interval | Requests/min | +|---|---|---| +| Free | 15s | 4 | +| Starter / Developer | 5s | 12 | +| Advanced | 2s | 30 | + +A 15-second poll on the free tier leaves one spare request per minute of +headroom for retries. + +> **Important caveat:** the single-ticker and full-market snapshot endpoints are +> **not included in the free Basic plan** (they require Starter or above). On a +> free key, `get_snapshot_all` will return HTTP 403. FinAlly treats any Massive +> failure as non-fatal and keeps serving the last cached prices, but a free-tier +> key will effectively yield no data. Users without a paid plan should leave +> `MASSIVE_API_KEY` unset and use the simulator. + +--- + +## 3. Client Initialization + +```python +from massive import RESTClient + +# Reads MASSIVE_API_KEY from the environment +client = RESTClient() + +# Or pass the key explicitly (what FinAlly does) +client = RESTClient(api_key="your_key_here") +``` + +Verified constructor defaults: + +```python +RESTClient( + api_key: str | None = None, # falls back to MASSIVE_API_KEY env var + connect_timeout: float = 10.0, + read_timeout: float = 10.0, + num_pools: int = 10, + retries: int = 3, # automatic retry on 5xx + base: str = "https://api.massive.com", + pagination: bool = True, + verbose: bool = False, + trace: bool = False, # trace=True prints request/response +) +``` + +**The client is synchronous.** It uses `urllib3` under the hood and will block +the event loop if called directly from async code. Always wrap calls in +`asyncio.to_thread(...)` — see §7. + +--- + +## 4. Primary Endpoint — Full Market Snapshot + +This is the workhorse endpoint for FinAlly: current prices for many tickers in +one call. + +**REST:** `GET /v2/snapshot/locale/us/markets/stocks/tickers` + +| Query param | Type | Notes | +|---|---|---| +| `tickers` | comma-separated list | Optional. Omit to get the entire market. | +| `include_otc` | boolean | Optional, defaults to `false`. | + +**Python:** + +```python +from massive import RESTClient +from massive.rest.models import SnapshotMarketType + +client = RESTClient(api_key=api_key) + +snapshots = client.get_snapshot_all( + market_type=SnapshotMarketType.STOCKS, + tickers=["AAPL", "GOOGL", "MSFT", "AMZN", "TSLA"], +) + +for snap in snapshots: + print(f"{snap.ticker}: ${snap.last_trade.price}") + print(f" Day change: {snap.todays_change_percent:.2f}%") + print(f" Prev close: ${snap.prev_day.close}") + print(f" Day OHLC: O={snap.day.open} H={snap.day.high} " + f"L={snap.day.low} C={snap.day.close}") +``` + +Verified signature: + +```python +get_snapshot_all( + market_type: str | SnapshotMarketType, + tickers: str | list[str] | None = None, + params: dict | None = None, + raw: bool = False, + include_otc: bool | None = False, + options: RequestOptionBuilder | None = None, +) -> list[TickerSnapshot] | HTTPResponse +``` + +`SnapshotMarketType` members: `STOCKS`, `FOREX`, `CRYPTO`, `INDICES`. + +### 4.1 Raw JSON vs. SDK attribute names + +The API returns terse single-letter JSON keys; the SDK maps them to readable +attribute names. **Use the right-hand column in Python code.** + +Top level (`TickerSnapshot`): + +| Raw JSON | SDK attribute | Meaning | +|---|---|---| +| `ticker` | `.ticker` | Symbol | +| `todaysChange` | `.todays_change` | Absolute change vs. previous close | +| `todaysChangePerc` | `.todays_change_percent` | Percent change vs. previous close | +| `updated` | `.updated` | Last update, Unix **nanoseconds** | +| `day` | `.day` | Today's aggregate bar (`Agg`) | +| `prevDay` | `.prev_day` | Previous day's bar (`Agg`) | +| `min` | `.min` | Most recent minute bar (`MinuteSnapshot`) | +| `lastTrade` | `.last_trade` | Most recent trade (`LastTrade`) | +| `lastQuote` | `.last_quote` | Most recent NBBO quote (`LastQuote`) | +| `fmv` | `.fair_market_value` | Fair market value (Business plans) | + +`Agg` (used by both `.day` and `.prev_day`): + +| Raw | SDK attribute | +|---|---| +| `o` | `.open` | +| `h` | `.high` | +| `l` | `.low` | +| `c` | `.close` | +| `v` | `.volume` | +| `vw` | `.vwap` | +| `t` | `.timestamp` | +| `n` | `.transactions` | + +`LastTrade` (snapshot variant): + +| Raw | SDK attribute | +|---|---| +| `p` | `.price` | +| `s` | `.size` | +| `x` | `.exchange` | +| `t` | **`.sip_timestamp`** | +| `i` | `.id` | +| `c` | `.conditions` | + +`LastQuote`: + +| Raw | SDK attribute | +|---|---| +| `p` | `.bid_price` | +| `s` | `.bid_size` | +| `P` | `.ask_price` | +| `S` | `.ask_size` | +| `t` | `.sip_timestamp` | + +> ### ⚠️ Two field-name traps +> +> **1. `last_trade` has no `.timestamp` attribute.** The raw key `t` maps to +> **`.sip_timestamp`**. Writing `snap.last_trade.timestamp` raises +> `AttributeError`. Full verified field list for the snapshot `LastTrade`: +> `ticker`, `trf_timestamp`, `sequence_number`, `sip_timestamp`, +> `participant_timestamp`, `conditions`, `correction`, `id`, `price`, `trf_id`, +> `size`, `exchange`, `tape`. +> +> **2. There is no `day.previous_close` or `day.change_percent`.** Previous +> close lives at **`snap.prev_day.close`**; percent change at +> **`snap.todays_change_percent`**. + +### 4.2 Timestamp units + +Massive mixes units across fields — this is a frequent source of "prices from +1970" bugs. + +| Field | Unit | Convert to Unix seconds | +|---|---|---| +| `last_trade.sip_timestamp` | **nanoseconds** | `/ 1_000_000_000` | +| `last_quote.sip_timestamp` | **nanoseconds** | `/ 1_000_000_000` | +| `updated` | **nanoseconds** | `/ 1_000_000_000` | +| `day.timestamp`, `prev_day.timestamp` | **milliseconds** | `/ 1_000` | +| Aggregate bar `t` (`list_aggs`, `prev`) | **milliseconds** | `/ 1_000` | + +Rather than hardcode a divisor, prefer a defensive normalizer (see §7). + +--- + +## 5. Other Relevant Endpoints + +### 5.1 Single Ticker Snapshot + +`GET /v2/snapshot/locale/us/markets/stocks/tickers/{stocksTicker}` + +```python +snap = client.get_snapshot_ticker( + market_type=SnapshotMarketType.STOCKS, + ticker="AAPL", +) +print(snap.last_trade.price, snap.todays_change_percent) +``` + +Returns the same `TickerSnapshot` shape as §4. FinAlly does not use this in the +poll loop — batching via `get_snapshot_all` is strictly better for rate limits. + +### 5.2 Previous Day Bar + +`GET /v2/aggs/ticker/{stocksTicker}/prev` + +Useful for establishing seed/reference prices without a snapshot subscription. + +```python +for agg in client.get_previous_close_agg(ticker="AAPL"): + print(f"Prev close: ${agg.close} O={agg.open} H={agg.high} L={agg.low}") +``` + +Query param: `adjusted` (boolean, default `true` — split-adjusted). + +### 5.3 Custom Bars (Aggregates) + +`GET /v2/aggs/ticker/{ticker}/range/{multiplier}/{timespan}/{from}/{to}` + +Historical OHLCV. Not used for live polling, but the natural source if FinAlly +later adds real historical charts (currently sparklines accumulate client-side +from the SSE stream). + +```python +bars = list(client.list_aggs( + ticker="AAPL", + multiplier=1, + timespan="day", # minute | hour | day | week | month | quarter | year + from_="2026-01-01", + to="2026-01-31", + limit=50000, +)) +for b in bars: + print(b.timestamp, b.open, b.high, b.low, b.close, b.volume) +``` + +Pagination is on by default; `limit` controls **page size**, not total results. +Pass `pagination=False` to `RESTClient(...)` for a fixed result count. + +### 5.4 Unified Snapshot (v3) + +`GET /v3/snapshot` — cross-asset-class snapshot with a cleaner schema +(`last_trade.price`, `last_quote.bid`/`.ask`, `session`, `market_status`). + +```python +snaps = list(client.list_universal_snapshots( + market_type="stocks", + ticker_any_of=["AAPL", "GOOGL", "MSFT"], +)) +``` + +Constraint: `ticker.any_of` accepts **at most 250 tickers**; `limit` defaults to +10 and maxes at 250 — remember to raise it or you will silently get 10 results. + +FinAlly uses v2 `get_snapshot_all` because it is the better-documented, more +widely available endpoint for a stocks-only use case, and it has no per-request +ticker cap for our scale. + +### 5.5 Last Trade / Last Quote + +```python +trade = client.get_last_trade(ticker="AAPL") +print(trade.price, trade.size) + +quote = client.get_last_quote(ticker="AAPL") +print(quote.bid_price, quote.ask_price) +``` + +One HTTP call per ticker — avoid in the poll loop. + +--- + +## 6. Error Handling + +| Status | Meaning | FinAlly response | +|---|---|---| +| 401 | Invalid / missing API key | Log error, keep serving cached prices | +| 403 | Plan does not include endpoint | Log error, keep serving cached prices | +| 429 | Rate limit exceeded (free tier) | Back off, retry next interval | +| 5xx | Server error | SDK auto-retries 3× before raising | + +The SDK raises `urllib3`/`requests`-style exceptions and a `BadResponse` on +non-2xx. Because a market data outage must never take down the app, the poller +catches **all** exceptions, logs, and continues — the cache simply holds stale +prices until the next successful poll. + +Individual snapshots may also be partially populated (e.g. `last_trade` is +`None` outside market hours on some plans), so per-ticker parsing is wrapped in +its own try/except and skips rather than aborting the batch. + +--- + +## 7. Reference Implementation — Poll Loop + +This is the pattern FinAlly's `MassiveDataSource` follows. Note the two details +that matter: threading the blocking client off the event loop, and defensive +timestamp normalization. + +```python +import asyncio +import logging +from massive import RESTClient +from massive.rest.models import SnapshotMarketType + +logger = logging.getLogger(__name__) + +# Anything larger than this is not plausibly Unix seconds, so scale it down. +_YEAR_2100_SECONDS = 4_102_444_800 + + +def normalize_timestamp(raw: float | None) -> float | None: + """Coerce a Massive timestamp (s / ms / us / ns) to Unix seconds. + + Massive returns nanoseconds for trade/quote SIP timestamps but + milliseconds for aggregate bars. Rather than depend on the field, scale + by orders of magnitude until the value is a plausible epoch-seconds value. + """ + if raw is None: + return None + ts = float(raw) + while ts > _YEAR_2100_SECONDS: + ts /= 1000.0 + return ts + + +def extract_price(snap) -> float | None: + """Best available current price, in priority order.""" + if snap.last_trade and snap.last_trade.price: + return snap.last_trade.price + if snap.min and snap.min.close: # most recent minute bar + return snap.min.close + if snap.day and snap.day.close: # today's close so far + return snap.day.close + if snap.prev_day and snap.prev_day.close: # market closed / pre-open + return snap.prev_day.close + return None + + +async def poll_massive(api_key, get_tickers, price_cache, interval=15.0): + client = RESTClient(api_key=api_key) + + while True: + tickers = get_tickers() + if tickers: + try: + # RESTClient is synchronous — never call it on the event loop. + snapshots = await asyncio.to_thread( + client.get_snapshot_all, + market_type=SnapshotMarketType.STOCKS, + tickers=tickers, + ) + for snap in snapshots: + try: + price = extract_price(snap) + if price is None: + continue + ts = normalize_timestamp( + snap.last_trade.sip_timestamp if snap.last_trade + else snap.updated + ) + price_cache.update( + ticker=snap.ticker, price=price, timestamp=ts + ) + except (AttributeError, TypeError) as e: + logger.warning("Skipping %s: %s", + getattr(snap, "ticker", "???"), e) + except Exception as e: + # Never let a data outage kill the loop. + logger.error("Massive poll failed: %s", e) + + await asyncio.sleep(interval) +``` + +### Why the price fallback chain matters + +`last_trade` is only populated when the plan includes trade data and there has +been a trade. Outside US market hours, or on a delayed plan, relying solely on +`last_trade.price` yields no data at all. The chain +`last_trade → min → day → prev_day` guarantees a usable price whenever the API +returns anything at all — important because FinAlly's portfolio valuation and +trade execution both read from the cache. + +--- + +## 8. Market Hours Behaviour + +- The `day` aggregate resets at market open; during pre-market its values may + still reflect the previous session. +- `last_trade.price` includes extended-hours trades on plans that carry them. +- On weekends and holidays, snapshots return the last session's values and + `todays_change` is typically `0`. +- FinAlly's UI makes no distinction — the price cache is the only contract, and + a stale-but-valid price renders identically to a live one. + +--- + +## 9. Applicability to FinAlly + +| Requirement | Endpoint | Notes | +|---|---|---| +| Live prices for watchlist | `get_snapshot_all` | One call for all tickers | +| Day change % for watchlist | `.todays_change_percent` | Already computed by API | +| Reference / previous close | `.prev_day.close` | No extra call needed | +| Seed prices (simulator parity) | `get_previous_close_agg` | Optional bootstrap | +| Historical charts (future) | `list_aggs` | Not in current scope | + +Everything FinAlly needs in the live path comes from **one endpoint, one call +per poll cycle**. See `MARKET_INTERFACE.md` for how this is wrapped behind the +provider-agnostic interface. + +--- + +## Sources + +- [Massive API docs](https://massive.com/docs) +- [Stocks REST API overview](https://massive.com/docs/rest/stocks/overview) +- [Full market snapshot](https://massive.com/docs/rest/stocks/snapshots/full-market-snapshot.md) +- [Single ticker snapshot](https://massive.com/docs/rest/stocks/snapshots/single-ticker-snapshot.md) +- [Unified snapshot](https://massive.com/docs/rest/stocks/snapshots/unified-snapshot.md) +- [Previous day bar](https://massive.com/docs/rest/stocks/aggregates/previous-day-bar.md) +- [massive-com/client-python](https://github.com/massive-com/client-python) +- [Polygon.io is now Massive](https://massive.com/blog/polygon-is-now-massive) +- [REST request limits](https://massive.com/knowledge-base/article/what-is-the-request-limit-for-massives-restful-apis) +- [Pricing](https://massive.com/pricing)