Skip to content

t8 scheduler: four audited jobs on the app's event loop, lifespan-owned start/stop, /health/ready - #3

Merged
vlobus merged 2 commits into
mainfrom
t8-scheduler
Jul 14, 2026
Merged

vlobus merged 2 commits into
mainfrom
t8-scheduler

Conversation

@vlobus

@vlobus vlobus commented Jul 9, 2026

Copy link
Copy Markdown
Owner

what

  • ingest/jobs.py — four recurring jobs (ingest_bars, ingest_news, detect_anomalies, explain_anomalies) plus the cold-start backfill as a one-shot, all through run_audited: an ingest_runs row commits before work (in-flight runs are visible in the table), closes ok/error, and exceptions are recorded, never raised — a bad tick can't kill the scheduler. Every job is re-runnable: upserts against natural keys, overlapping lookbacks, reads only stored state. The explain tick caps spend (EXPLAIN_BATCH=5, newest first via a new AnomalyRepository.unexplained), aborts the batch on LLMUnavailable (error run) and skips just the one anomaly on MalformedReply.
  • ingest/scheduler.pybuild_scheduler on AsyncIOScheduler (D5): jobs run as tasks on the app's own loop, max_instances=1 (a slow run means a skipped tick, not overlap), coalesce=True, misfire_grace_time=60; the backfill one-shot gets unlimited grace so a slow startup delays it instead of silently dropping it.
  • serving/app.py — lifespan owns start/stop, in order: scheduler stops first, then the inference executor, then resources (an AsyncExitStack the composition root fills with engine/client cleanup). Shutdown semantics documented honestly: APScheduler 3.x cancels in-flight async ticks; a bounded grace lets a cancelled tick unwind its session, and a tick cut off there leaves a truthful "running" audit row redone at next startup. SchedulerLike Protocol keeps APScheduler off serving's type surface.
  • serving/routes/health.py/health/ready (DB answers + scheduler running → 200, else 503 with reasons), distinct from /health/live: restart vs route-around.
  • main.py — full composition root: settings (fail-fast), engine, one shared httpx client, both connectors, FinBERT, AnthropicExplainer, the single-lane executor, scheduler, exit stack.
  • T7 review carry-overs: the explain job shares the serving app's single-worker inference executor, so all FinBERT calls in the process go through one lane (D18 — tokenizer thread-safety, no CPU oversubscription); AnthropicExplainer now has an explicit 60s timeout (SDK default is 600s — longer than the explain interval).

tests

  • unit: scheduler config introspection (four interval jobs + one-shot, overlap protection settings), lifespan starts scheduler → job fires on the loop → shutdown stops it before resources close (asserted inside the exit-stack callback), ready-503 when nothing is wired, Anthropic adapter with a stubbed client (text-block joining, transport error → LLMUnavailable, bounded timeout).
  • integration (real Postgres): audit rows for ok / dead-source-skip / crashed runs, detect flags a synthetic spike and reruns idempotently, explain job explains then goes quiet (llm.calls == 1 across two ticks), LLM-down marks the run error with nothing stored, and the acceptance: a lifespan-started scheduler fires the backfill into ingest_runs while /health/ready answers 200, then stops clean.
  • LLM mocked everywhere; no live API, no real credentials.
  • subagent code review done; both must-fixes (backfill misfire drop, overclaimed shutdown comments) fixed in the second commit, plus the layering/tiebreaker/style optionals.
  • full gate green: ruff, mypy strict, import-linter, pytest (100 passed).

vlobus added 2 commits July 9, 2026 23:06
…io scheduler, lifespan start/stop, /health/ready, full composition root; explain shares the single-lane inference executor, anthropic timeout bounded
…rop), honest shutdown semantics + bounded unwind grace, session factory alias into store, unexplained tiebreaker
@vlobus
vlobus merged commit 9a853c2 into main Jul 14, 2026
1 check passed
@vlobus
vlobus deleted the t8-scheduler branch July 14, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant