t8 scheduler: four audited jobs on the app's event loop, lifespan-owned start/stop, /health/ready - #3
Merged
Merged
Conversation
…io scheduler, lifespan start/stop, /health/ready, full composition root; explain shares the single-lane inference executor, anthropic timeout bounded
…rop), honest shutdown semantics + bounded unwind grace, session factory alias into store, unexplained tiebreaker
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
what
ingest/jobs.py— four recurring jobs (ingest_bars,ingest_news,detect_anomalies,explain_anomalies) plus the cold-start backfill as a one-shot, all throughrun_audited: aningest_runsrow commits before work (in-flight runs are visible in the table), closes ok/error, and exceptions are recorded, never raised — a bad tick can't kill the scheduler. Every job is re-runnable: upserts against natural keys, overlapping lookbacks, reads only stored state. The explain tick caps spend (EXPLAIN_BATCH=5, newest first via a newAnomalyRepository.unexplained), aborts the batch onLLMUnavailable(error run) and skips just the one anomaly onMalformedReply.ingest/scheduler.py—build_scheduleronAsyncIOScheduler(D5): jobs run as tasks on the app's own loop,max_instances=1(a slow run means a skipped tick, not overlap),coalesce=True,misfire_grace_time=60; the backfill one-shot gets unlimited grace so a slow startup delays it instead of silently dropping it.serving/app.py— lifespan owns start/stop, in order: scheduler stops first, then the inference executor, thenresources(anAsyncExitStackthe composition root fills with engine/client cleanup). Shutdown semantics documented honestly: APScheduler 3.x cancels in-flight async ticks; a bounded grace lets a cancelled tick unwind its session, and a tick cut off there leaves a truthful "running" audit row redone at next startup.SchedulerLikeProtocol keeps APScheduler off serving's type surface.serving/routes/health.py—/health/ready(DB answers + scheduler running → 200, else 503 with reasons), distinct from/health/live: restart vs route-around.main.py— full composition root: settings (fail-fast), engine, one shared httpx client, both connectors, FinBERT,AnthropicExplainer, the single-lane executor, scheduler, exit stack.AnthropicExplainernow has an explicit 60s timeout (SDK default is 600s — longer than the explain interval).tests
LLMUnavailable, bounded timeout).llm.calls == 1across two ticks), LLM-down marks the run error with nothing stored, and the acceptance: a lifespan-started scheduler fires the backfill intoingest_runswhile/health/readyanswers 200, then stops clean.