From bee9411f13860279c9b0df344f93e68e7a45ee58 Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Sun, 9 Aug 2026 09:54:12 -0400 Subject: [PATCH] Describe the finals gate that ships, not the one that broke WINDOWS_SCHEDULING.md still explained --require-final as a wall-clock check: norgatedata.last_database_update_time() for both databases against --final-cutoff, default 20:55. That has not been the mechanism since the gate went data-driven, and the cutoff flag is accepted-but-ignored in the code. The doc was describing a version that was replaced precisely because it failed. It is also load-bearing in a way most stale prose is not. Nobody reads this file casually: they read it while wiring a producer, and then they build what it says. It was read that way last week, by someone specifying an equivalent gate for marketdata, who correctly reproduced the description and would have reproduced the 2026-07-27 outage with it. So the replacement paragraph leads with the rule that actually runs, states the ES/CL/ZC quorum, and gives the two consequences an operator can check against behaviour: no trading calendar is needed because a weekend and an unsettled session are the same answer, and Norgate publishes no in-progress bar, so bar presence is already a settled-session signal. The 2026-07-27 failure moves into a blockquote on --final-cutoff, kept rather than deleted because a reader with that flag in an existing wrapper needs to know it does nothing now and why removing it is safe. Two smaller corrections in the same pass. The restart-on-failure paragraph described each retry as a last_database_update_time check, naming the dead mechanism in a sentence otherwise about scheduling. And the testing section said a daytime run "only proves the wrapper resolves", which stopped being unconditional: under a date comparison a daytime run defers only if last night captured, so a missed or failed night now self-heals at the next trigger instead of waiting for the evening. That is a real operational property the old wording hid. The design doc's Problem section still describes the cutoff in the present tense and is deliberately left alone: it records what was true when the change was argued, and rewriting it would destroy the reasoning it exists to preserve. The example wrapper needed no change, being mechanism-agnostic already. Co-Authored-By: Claude Opus 5 --- docs/WINDOWS_SCHEDULING.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/WINDOWS_SCHEDULING.md b/docs/WINDOWS_SCHEDULING.md index 9484f3e..1fa30db 100644 --- a/docs/WINDOWS_SCHEDULING.md +++ b/docs/WINDOWS_SCHEDULING.md @@ -93,9 +93,19 @@ Register-ScheduledTask -TaskName "cotdata COT (Fri release)" -Action $act -Trigg (Or in the Task Scheduler GUI: New Task → Trigger *Weekly, Friday, 3:25pm* → check *"Repeat task every: 2 minutes for a duration of: 45 minutes."*) -**Event-driven prices with `--require-final`.** cotdata reads two Norgate databases: **Continuous Futures** (the `&ES` / `_CCB` series) and **Futures** (the individual `ES-2026H` contracts used to reconstruct volume). Their **Final** prices land ~8:40pm ET (Futures) and ~8:55pm ET (Continuous Futures), but your Norgate Data Updater still has to *pull* them on its next poll. Rather than guess a fixed time, `--require-final` checks `norgatedata.last_database_update_time()` for both databases and only fetches once each has been refreshed at/after `--final-cutoff` (default `20:55` local — set it to your machine's local equivalent of 8:55pm ET). Until then it **defers with a non-zero exit**, so the restart setting below turns "fire at 8:55pm" into "run the moment NDU has the Finals." +**Event-driven prices with `--require-final`.** Norgate's **Final** futures prices land ~8:40pm ET (the `Futures` database) and ~8:55pm ET (`Continuous Futures`), but your Norgate Data Updater still has to *pull* them on its next poll. `--require-final` waits for that, and it asks the **data** rather than the clock: -**Retry / wait via restart-on-failure.** Give each task a *restart on failure* — it does double duty: it retries transient fetch errors, and (for the price task) waits out the gap between 8:55pm and NDU actually pulling the Finals (each retry is a cheap `last_database_update_time` check that exits immediately until ready). On a genuine no-session day the retries simply exhaust, harmlessly. `schtasks` can't set this, so use PowerShell (applies to all three tasks): +``` +ready := norgate_latest_bar_date > store_latest_bar_date +``` + +checked across a quorum of liquid continuous references (`ES`, `CL`, `ZC`), all of which must have advanced, so one lagging reference cannot green-light a partial capture. Until then it **defers with a non-zero exit**, so the restart setting below turns "fire at 8:55pm" into "run the moment NDU has the Finals." + +Two properties follow from comparing bar dates, and both are why nothing here needs a trading calendar. Weekends and holidays produce no new bar, so "no session today" and "session not settled yet" give the same answer. And Norgate does not publish an in-progress session's bar at all (probed on the producer, 2026-07-28: at 11am Tuesday the newest continuous bar was Monday's final OHLC, with no Tuesday bar in existence), so bar *presence* is already a settled-session signal. + +> **`--final-cutoff` is deprecated and ignored.** It is still accepted so an existing wrapper does not break, and it does nothing. The flag used to be the whole mechanism: `--require-final` checked `norgatedata.last_database_update_time()` for both databases against a fixed local cutoff (default `20:55`). That **broke in production on 2026-07-27**, when Norgate finalized `Futures` at 8:49pm and `Continuous Futures` at 8:55pm. The check wanted both at or after 20:55, so the task deferred on every attempt, the chained sync never ran, and prices sat on the prior Friday's bar. The failure is intrinsic to a clock threshold: it must sit below the earliest evening publish and above any daytime refresh, and Norgate's publish time drifts night to night, so no single value is safe. Delete the flag from your wrapper if it is there. + +**Retry / wait via restart-on-failure.** Give each task a *restart on failure* — it does double duty: it retries transient fetch errors, and (for the price task) waits out the gap between 8:55pm and NDU actually pulling the Finals (each retry is a short trailing-window read and a date compare, exiting immediately until ready). On a genuine no-session day the retries simply exhaust, harmlessly. `schtasks` can't set this, so use PowerShell (applies to all three tasks): ```powershell $s = New-ScheduledTaskSettingsSet -RestartInterval (New-TimeSpan -Minutes 10) -RestartCount 6 @@ -142,7 +152,7 @@ Confirm the relevant `newest data` date advanced (and `last write (UTC)` is rece ### Testing the timing and conditions -- **A daytime prices run only proves the wrapper resolves** — with no Finals yet it just defers. To exercise the actual write path in daylight, run `cotdata-prices --prices --metadata` by hand (no `--require-final`), or fire the task after ~8:55pm ET. +- **A daytime prices run usually only proves the wrapper resolves** — if last night's run captured, the store already holds the newest settled bar, so a daytime run finds nothing newer and defers. The exception is worth knowing: if last night's run *failed or never fired*, a daytime run finds Norgate ahead of the store and captures immediately, so a missed night self-heals at the next trigger rather than waiting for the evening. To exercise the write path on demand regardless, run `cotdata-prices --prices --metadata` by hand (no `--require-final`), or fire the task after ~8:55pm ET. - **Test the trigger itself** by moving it a couple of minutes out, watching it fire, then setting it back: ```bat schtasks /Change /TN "cotdata prices" /ST 14:20