Instructor redo: consolidated market data design doc, rebuilt from clean main - #8
Merged
Conversation
…4237013 Add claude GitHub actions 1785224237013
Documents the market data subsystem as the current design of record: the unified MarketDataSource interface, thread-safe PriceCache, GBM simulator, Massive REST client, SSE streaming endpoint, and FastAPI lifecycle integration, with runnable code snippets throughout. Each snippet is labelled as-built or change-required. The design resolves three contract gaps raised in review: - SSE cadence: the stream now emits every priced ticker each tick with a per-connection direction, matching the PLAN contract instead of falling silent between Massive polls - Unpriced tickers: adds ensure_priced() to the source interface with typed errors, defining how a trade in an unpriced symbol resolves a fill price in both modes and why no rollback is needed - Symbol validation: adds a symbols module with shape validation and a simulated universe, so unrecognized symbols are rejected before the data source or watchlist is mutated Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5sUx9qhg2z4ygWmtvcceG
planning/MARKET_DATA_DESIGN.md had grown to 1,550 lines covering the whole subsystem in one file. Split it along the seams an implementing agent actually works to, so each document answers one kind of question: - planning/market_data_design.md — architecture, file structure, SSE streaming, FastAPI lifecycle wiring, watchlist/trade coordination, cross-cutting edge cases, environment configuration - planning/market_interface.md — PriceUpdate, PriceCache, the MarketDataSource ABC and its per-method expected behaviour, the error taxonomy and its HTTP mapping, the factory, the public API surface - planning/market_simulator.md — GBM math, correlated moves, seed prices, symbol validation, GBMSimulator, SimulatorDataSource lifecycle, the non-PSD fallback, simulator configuration and tests - planning/massive_api.md — the Massive/Polygon.io endpoints and response shape, the polling client, ticker management, poll intervals and rate limits, error handling and fallback behaviour, tests - planning/market_data_summary.md — executive summary, per-module implementation status, key decisions, unresolved review gaps, and the ordered implementation plan Content is preserved rather than rewritten; the source document is removed now that everything in it has a home. Source-specific material moved to its source (the 14.x edge cases split across the simulator, Massive, and architecture documents), and each fact now lives in exactly one file, with cross-references where a second document needs it. Renamed planning/MARKET_DATA_SUMMARY.md to planning/market_data_summary.md and merged the design document's status and delta sections into it. Keeping both would have left two overlapping summaries whose names differ only by case, which does not survive a checkout on macOS or Windows. Its stale "all issues resolved" status is corrected: the three contract gaps are specified but not yet implemented. References updated in CLAUDE.md, README.md, and planning/PLAN.md. Verified: 26 Python snippets parse under ast.parse, all 85 internal links and anchors resolve, and no substantive paragraph from the source document was dropped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QTMgygimph2t3UJNGfjyKF
Add detailed market data backend design document
Updated Claude workflow to include additional permissions and trigger phrase.
Adds symbols.py (normalize_symbol/SIMULATED_UNIVERSE/is_simulated),
UnknownSymbolError/PricingUnavailableError + ensure_priced on the
MarketDataSource interface (implemented in both the simulator and
Massive client), rewrites the SSE generator to emit every tick with
per-connection direction in a {seq, ts, prices} envelope, locks the
PriceCache.version read, falls back to independent draws on a
non-PSD correlation matrix, reads MASSIVE_POLL_INTERVAL from env,
and escalates repeated Massive poll failures to ERROR after 3
consecutive failures. Also fixes backend/README.md's `uv sync --dev`
to `uv sync --extra dev` (dev is an optional extra, not a group).
Addresses all 9 items in planning/market_data_summary.md's
"Recommended Implementation Order", including tests.
Co-authored-by: hengky888 <291320016+hengky888@users.noreply.github.com>
Implements the complete Market Data backend and all recommended hardening items. Highlights: - Added symbol normalization and simulation-symbol handling - Added pricing availability validation (`ensure_priced`) - Improved SSE tick delivery and per-connection direction tracking - Added cache version locking for thread safety - Added simulator fallback for invalid correlation matrices (`LinAlgError`) - Added configurable `MASSIVE_POLL_INTERVAL` - Added escalation after consecutive polling failures - Expanded automated test coverage (127 tests) - Updated backend README setup instructions Validation: - `uv sync --extra dev` ✓ - `uv run pytest -v` ✓ (127 tests passed) - `uv run ruff check app/ tests/` ✓ This PR completes the Market Data backend implementation specified in the project plan and brings the backend to a validated, production-ready state.
Single implementation-ready design of record for the market data subsystem, covering the unified data-source API, the GBM simulator, and the Massive REST client, plus the shared price cache, SSE streaming, the FastAPI wiring, and the watchlist/trade routes that consume them. Written against the as-built code rather than as a forward-looking spec: every code snippet is the shipped implementation, and the numeric constants (seed prices, GBM parameters, correlations, simulated universe) were diffed against source to confirm they match. Documents the reasoning behind the non-obvious choices — why the SSE stream emits unconditionally and computes direction per connection, why ensure_priced exists alongside add_ticker, why the no-residue guarantee is achieved by ordering rather than rollback, why the cache rounds at the write boundary, and why there is no runtime failover from Massive to the simulator. The four split documents are superseded. market_data_design.md moves to archive/MARKET_DATA_DESIGN_V2.md (renamed to avoid a case-collision with the existing archive/MARKET_DATA_DESIGN.md on case-insensitive filesystems), and the ten cross-references in the remaining companion docs are repointed at the new sections. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MtZb6BTTzEFNEzR6kq1wJ4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Instructor exercise redone from the clean, finished
mainversion of the project. Branched frommainat5013245, all work oninstructor-redo.Please review — do not merge automatically.
Heads up: this PR is larger than the one commit of new work
office-developmentis 16 commits behindmain. Because this branch was cut from cleanmain(as intended for the redo), the PR carries those 16 commits plus 1 new one — 17 total, 33 files.mainoffice-developmentis a direct ancestor of this branch, so there are no conflicts — merging bringsoffice-developmentup to cleanmainand adds the new document. If you only want the new document, say so and I'll rebase it ontooffice-developmentalone.The new work
planning/MARKET_DATA_DESIGN.md(1,737 lines, 30 code blocks) — a single implementation-ready design of record for the market data subsystem, consolidating the four split documents.15 sections covering:
PriceUpdate,PriceCache, theMarketDataSourceABC includingensure_priced, the three-error taxonomy and its 400/503 mapping,symbols.py, and the public re-export surfaceGBMSimulatorandSimulatorDataSourceWritten against the as-built code, not as a forward-looking spec
The existing planning docs describe three "unresolved gaps" (SSE cadence,
ensure_priced,symbols.py) as still open and label snippets "change required" — but all three are already implemented onmain. The suite runs 127 tests at 97% coverage, not the 73 tests at 84% the summary claims.market_data_summary.md's "Recommended Implementation Order" is therefore stale; every item on it has landed.Every snippet in the new document is the shipped implementation. This was verified mechanically rather than by eye: seed prices, GBM parameters, all four correlation constants, and all 72 symbols in
SIMULATED_UNIVERSEwere diffed against source and match exactly. Seven of the nine modules are quoted verbatim at 100%; the other two differ only in line-wrapping and trimmed docstrings.The document records the reasoning behind the non-obvious choices, since that is what a reader cannot recover from the code alone — why the SSE stream emits unconditionally and computes
directionper connection, whyensure_pricedexists alongsideadd_ticker, why the no-residue guarantee is achieved by ordering rather than rollback, why the cache rounds at the write boundary, and why there is deliberately no runtime failover from Massive to the simulator.Supersession and the rename
The four split documents are superseded.
planning/market_data_design.mdmoves toplanning/archive/MARKET_DATA_DESIGN_V2.md— renamed rather than kept asmarket_data_design.mdbecause that would collide with the existingarchive/MARKET_DATA_DESIGN.mdon case-insensitive filesystems (macOS/Windows), which is the same trap that made the new file's placement a question in the first place.The ten cross-references in the four remaining companion docs are repointed at the new sections. A link checker confirmed every internal link and anchor across
planning/resolves.Verification
uv run --extra dev pytest— 127 passed, 97% statement coverage ofapp/market/planning/resolvemainwas not modified, merged into, or pushed to at any point.Generated by Claude Code