Intraday daemon silently drops entry/exit triggers when surveillance-universe ticker subscriptions exceed IB's market-data-line cap (SPY ENTER missed 2026-07-20)

[cipher813]

Summary

Today (2026-07-20), the morning planner approved a small SPY ENTER (12 shares, portfolio-optimizer target weight 0.67%) and wrote it to the order book. The intraday daemon ran cleanly all session (14:54:15–20:00:28 UTC) but never executed the SPY order — no fill, no error, no Telegram "trade not executed" notice. Root cause: the daemon's market-data subscription silently dropped SPY due to hitting IB Gateway's per-connection ticker-line cap.

Root cause

executor/daemon.py::run() builds a "surveillance universe" via compute_surveillance_universe() (executor/intraday_snapshot.py:138) = signals.signals ∪ buy_candidates ∪ order_book_tickers ∪ current_positions ∪ {SPY}. Today this produced 902 unique tickers, and PriceMonitor.subscribe() (executor/price_monitor.py:32-41) requests reqMktData for every one of them in list order with no prioritization and no cap awareness.

IB Gateway's line returned Error 101: Max number of tickers has been reached starting partway through the (alphabetically-ish ordered) subscription loop. SPY's subscription request hit this error at 2026-07-20T14:56:35 UTC (confirmed via /var/log/daemon.log on i-018eb3307a21329bf, reqId 2144). _subscribe_one() does not check for this async error — it unconditionally adds the symbol to self._contracts (price_monitor.py:61) and logs "Subscribed to N/N tickers," so nothing downstream knows the subscription failed.

Because SPY's live feed (self.prices["SPY"]) was therefore never populated, EntryTriggerEngine.should_enter() (executor/entry_triggers.py:58-60) hit its first guard clause (current_price is None) and returned (False, "") on every tick, all day — including the unconditional 3:55 PM ET time_expiry fallback (entry_triggers.py:107-109), since that check is unreachable once the function returns early. The order sat status: "pending" in the order book and was simply dropped when the daemon shut down at market close; no cancellation, no failure log, no distinct alert.

By contrast, AMD's ENTER (also in today's order book) filled fine via graduated_entry at 19:30:03 UTC — its ticker happened to get subscribed before the cap was hit.

Evidence

• Order book (s3://alpha-engine-research/trades/order_book/2026-07-20.json): SPY ENTER, 12 shares, status: "pending".
• /var/log/daemon.log on i-018eb3307a21329bf, 2026-07-20T14:56:35.328434+00:00: Error 101, reqId 2144: Max number of tickers has been reached, contract: ...symbol='SPY'....
• Same log: 300+ distinct tickers hit the identical Error 101 in the same window (BEN through roughly the W's alphabetically) — this is systemic, not SPY-specific.
• trades_2026-07-20.db (trades table): no row for SPY (or any ticker) dated 2026-07-20 at all except AMD's fill.
• journalctl -u alpha-engine-daemon.service: clean start (14:54:15 UTC) and clean stop (20:00:28 UTC, 53s total CPU) — no crash, no restart. This is a silent logic gap, not an outage.
• A flow-doctor Telegram alert did fire once for the Error-101 burst (report=019f8007d085970a4016b89f4b03, dispatched 14:57:04 UTC, sig 40b24823 deduped ~300 subsequent occurrences) — but it reports the IB error, not "an approved order never got to evaluate its trigger." No GitHub issue was auto-filed for this error class (unlike e.g. UnscoredBuyCandidatesError or the RuntimeError classes that already have precedent issues here).

Impact

Any approved entry (or exit — same PriceMonitor feeds IntradayExitManager) whose ticker sorts late enough in the surveillance-universe subscription order to lose the race against IB's ticker cap silently never gets evaluated for the rest of the session. This is not a one-off — the surveillance universe is defined as effectively the full research-tracked universe (signals ∪ buy_candidates), which is routinely several hundred names, so the cap is likely being hit most trading days. Today it cost a 12-share/~$9k SPY entry; on another day it could just as easily strand a large open position's exit trigger.

Suggested fix (institutional-grade, not a patch)

1. Prioritize subscription order: current positions + order-book tickers (entries, urgent exits, active stops) + SPY must subscribe first, before the broader signals/buy-candidates surveillance set — those are the only tickers that can actually cause an order to fire.
2. Make _subscribe_one cap-aware: treat IB error 101 as a hard subscribion failure (not a silent success) — do not add the symbol to self._contracts when the request errors, and surface which specific symbols failed.
3. Alert distinctly when a ticker with a pending order-book entry or an open position fails to subscribe — this is a different, higher-severity condition than an ordinary surveillance-ticker miss and should not just ride flow-doctor's generic Error-101 dedup.
4. Separately: consider whether IB Gateway's line cap can be raised (paper-account market data subscription tier) now that the tracked universe is ~900 names, vs. permanently shrinking the surveillance set.

Acceptance criteria

• A regression test proves that when the ticker-cap is exceeded, order-book/position tickers are subscribed before generic surveillance tickers.
• A regression test proves PriceMonitor does not mark a symbol as subscribed when IB returns error 101 for it.
• A new alert path (or an existing one, wired up) fires specifically when an order-book-tracked ticker loses its subscription intraday.