From de062136f0f970d72c7701717e8a816afd75627b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 17:11:17 +0000 Subject: [PATCH] Add the Linux scheduling docs the databento port left behind The provider moved here in #15; its operational half stayed in cotdata and then went with cotdata's own price docs. So the box that actually runs databento has a wrapper calling `cotdata-prices` -- an entry point that no longer resolves -- and nowhere in this repo telling it what to call instead. docs/examples/linux/run-prices.sh is the replacement to copy, and docs/LINUX_SCHEDULING.md is the crontab, the flock setup and the troubleshooting. TWO CHANGES FROM THE SCRIPT IT REPLACES, both about not paying twice. Stage 2 now runs even when Stage 1 fails, and the script drops `set -e` to allow it. Stage 1 is the only step that costs money, it writes incrementally and it is resumable, so a network failure partway through still leaves new raw data on disk. Aborting before the free local rebuild would discard something already paid for, and the next run would have to pay again to reach the same place. Stage 1's exit code is captured and returned, so cron still sees a failure and still retries. The cold-start backfill is documented as a manual step rather than the nightly job, with --windowed-n1-stats on it. That flag restricts the second contract's statistics to windows around roll dates, where the series is the only place it is read -- accuracy-neutral, and the largest avoidable download. It buys nothing on a nightly incremental, so it does not belong in the crontab line. Troubleshooting leads with the failure this move causes -- "command not found" from a wrapper still calling cotdata-prices -- and covers the stuck-symbol case that is worth knowing about: a ledger entry claiming rows whose parquet is missing carries a current last_date, so every run skips it as "already current". No error, no row, a permanent hole in a paid dataset. --reconcile-databento is the repair. Verified rather than asserted: the quoted "no databento-capable symbols" message matches the provider, the eight `databento: null` markets are the eight named, and the rc-propagation shape was exercised with a stub -- both stages run, exit code 7 survives. --- README.md | 4 + docs/LINUX_SCHEDULING.md | 145 ++++++++++++++++++++++++++++++ docs/examples/linux/run-prices.sh | 40 +++++++++ 3 files changed, 189 insertions(+) create mode 100644 docs/LINUX_SCHEDULING.md create mode 100755 docs/examples/linux/run-prices.sh diff --git a/README.md b/README.md index 36f1346..0828403 100644 --- a/README.md +++ b/README.md @@ -113,6 +113,10 @@ Both vendors land in the same store under different paths (`bars/futures/norgate/` beside `bars/futures/databento/`), which is what makes `scripts/validate_databento_vs_norgate.py` a single-store comparison. +**Scheduling it:** [docs/LINUX_SCHEDULING.md](docs/LINUX_SCHEDULING.md) — the wrapper, the +crontab line, why the cold-start backfill is not the nightly job, and what to do when the +resume ledger and the disk disagree. + ### Pinning a store for a study A study that quotes numbers is only reproducible if the data behind them is diff --git a/docs/LINUX_SCHEDULING.md b/docs/LINUX_SCHEDULING.md new file mode 100644 index 0000000..a0f5b61 --- /dev/null +++ b/docs/LINUX_SCHEDULING.md @@ -0,0 +1,145 @@ +# Scheduling marketdata on Linux (cron) + +For the **databento** producer — the cross-platform one, no Norgate and no Windows. The +Norgate producer runs on Windows and is scheduled from Task Scheduler; see +[cotdata's WINDOWS_SCHEDULING.md](https://github.com/mspinola/cotdata/blob/main/docs/WINDOWS_SCHEDULING.md), +which still covers that box because the COT job lives there too. + +> **Moved here by ADR-0007.** This used to be `cotdata/docs/LINUX_SCHEDULING.md`, back when +> databento lived in cotdata. A server upgrading past that split has a `run-prices.sh` +> calling `cotdata-prices`, **an entry point that no longer exists** — the job fails every +> night until the wrapper is replaced with the one beside this file. + +## Goal + +Bars nightly, from a box that cannot run Norgate. Two properties make the schedule simple: + +- **Resumable.** Stage 1 records what it fetched per (symbol, feed, schema) and restarts + from `last_date + 1`, so a re-run pulls only new days and a mid-pull failure costs + nothing but the time. Running before new data lands is a harmless no-op. +- **Fails loudly.** A run exits non-zero on a hard fetch error, not on "nothing new". A + failed or missed run is picked up by the next one, so no explicit retry logic is needed. + +## Wrapper script + +Cron runs with a bare environment, so keep the config and the venv path in a wrapper. + +> **Ready-made template:** copy [`examples/linux/run-prices.sh`](examples/linux/run-prices.sh) +> into your ``, `chmod +x` it, and fill in the markers — keep it outside the repo so a +> `git pull` never clobbers your edited paths. + +Overwrite the plain-text markers: `REPLACE_WITH_STORE_PATH` = your bar store, +`REPLACE_WITH_DATABENTO_KEY` = your Databento key, `REPLACE_WITH_VENV_PATH` = your venv. +They are plain markers rather than `<...>` placeholders because an unedited `<...>` would +be read as a shell redirection and the script would fail on its own comments. + +```bash +#!/usr/bin/env bash +set -uo pipefail # NOT -e: see below + +export MARKETDATA_STORE=REPLACE_WITH_STORE_PATH +export DATABENTO_API_KEY=REPLACE_WITH_DATABENTO_KEY +BIN=REPLACE_WITH_VENV_PATH/bin/marketdata-update + +rc=0 +"$BIN" --ingest-databento || rc=$? # Stage 1 (PAID) +"$BIN" --build-databento || rc=$? # Stage 2 (FREE) +exit "$rc" +``` + +**Why Stage 2 runs even when Stage 1 fails.** Stage 1 is the only step that costs money, +it writes incrementally, and it is resumable — so a network failure partway through still +leaves new raw data on disk. Refusing to build from it would discard something already +paid for, and the next run would have to pay again to reach the same place. The Stage 1 +exit code is preserved and returned, so cron still sees the failure and still retries. +This is why the script does not use `set -e`. + +**`MARKETDATA_STORE` is a different directory from `COTDATA_STORE`**, not an alias. Both +packages keep a `manifest.json` at their own root and each does a read-modify-write on it, +so one shared root means two producers dropping each other's entries. Sharing a *parent* +folder is fine and makes the pair easy to sync. + +## Crontab entries + +Add the job with `crontab -e`. Cron uses the **server's local** timezone, so convert the +ET time below if it is not on Eastern. `flock` stops a slow run from overlapping the next, +and the redirect keeps a log: + +```cron +# Bars — nightly (Mon-Sat). GLBX settlements are disseminated the morning after the +# session, so an early-morning run captures the prior session's finalized settlement. +30 6 * * 1-6 flock -n /tmp/marketdata-bars.lock /run-prices.sh >> /bars.log 2>&1 +``` + +Set `MAILTO=you@example.com` at the top of the crontab to have cron email any run that +writes to stderr or exits non-zero. Check coverage and freshness any time with +`marketdata-update --check`, which reads the manifest and no network. + +## The cold-start backfill is not the nightly job + +Run the first pull by hand, not from cron — it is the expensive one, and it wants a flag +the nightly job does not: + +```bash +marketdata-update --ingest-databento --windowed-n1-stats +marketdata-update --build-databento +``` + +`--windowed-n1-stats` fetches the second contract's statistics only in a window around +each roll date. That series is read *only* at rolls, so restricting it is accuracy-neutral +and drops the largest avoidable download. Leave it off the nightly job, where the +incremental window is a few days anyway. + +If a from-inception pull keeps timing out at the gateway, `--batch` submits a databento +batch job instead of streaming. It is not the better default — a single from-inception +continuous job can 504 — but it is the right tool for a bounded catch-up. + +## Troubleshooting + +### The job fails with "command not found" + +Almost certainly a wrapper still calling `cotdata-prices`. That entry point was removed +with ADR-0007; the command is `marketdata-update` and the store variable is +`MARKETDATA_STORE`. Replace the wrapper with the template above. + +### Cron job runs manually but not on schedule + +Cron's environment is far barer than an interactive shell — no `PATH` beyond +`/usr/bin:/bin`, no `.bashrc`/`.profile`, no venv activation. That is why the wrapper calls +the venv's binary by full path and `export`s every variable rather than relying on a login +shell. If a script works by hand but not under cron, start by asking what your interactive +shell set up implicitly. + +### The run "succeeded" but wrote nothing + +Check the log for `no databento-capable symbols`. Eight registry markets are not on CME +Globex — the ICE softs, lumber and the dollar index — and carry `databento: null`, so a run +scoped to only those has nothing to do and says so. Otherwise `marketdata-update --check` +shows what the store actually holds. + +### A symbol is stuck and never re-fetches + +The resume ledger and the disk have diverged. An entry claiming rows whose parquet is +missing still carries a current `last_date`, so every run skips it as "already current" — +no error, no row, a permanent hole in a paid dataset. Repair it from local files: + +```bash +marketdata-update --reconcile-databento +``` + +It fixes both directions: it records tables an interrupted run left unrecorded (so a +restart does not re-pay for them) and prunes ghosts (so a restart re-fetches them). It +never touches the API. + +### Overlapping runs / stale lock + +`flock -n` fails fast rather than blocking, so a slow run will not stack with the next — +the second invocation no-ops and exits. The lock releases when the holding process exits, +including on a crash, so a lock that blocks forever means a *hung* process rather than +something to delete. Check `ps aux | grep marketdata` before removing anything in `/tmp`. + +### The store is filling up + +`_raw/` is databento's append-only bronze store: producer-internal, the bulk of the bytes, +and useless to a consumer. Point `MARKETDATA_DATABENTO_RAW` at cheaper disk, and exclude +`_raw/` from any sync to replicas. diff --git a/docs/examples/linux/run-prices.sh b/docs/examples/linux/run-prices.sh new file mode 100755 index 0000000..ad84292 --- /dev/null +++ b/docs/examples/linux/run-prices.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# marketdata bar update wrapper for cron / systemd (Linux, databento producer). +# +# Copy this file next to your crontab dir and overwrite the markers below: +# REPLACE_WITH_STORE_PATH = your bar store e.g. /srv/marketdata_store +# REPLACE_WITH_DATABENTO_KEY = your Databento API key e.g. db-... +# REPLACE_WITH_VENV_PATH = your venv e.g. /opt/marketdata/.venv +# (Plain-text markers, not angle-bracket placeholders: an unedited <...> would be +# read as a shell redirection and the script would fail on its own comments.) +# +# REPLACES cotdata's run-prices.sh. ADR-0007 moved every price producer here, so +# the old script's `cotdata-prices --ingest-databento` no longer resolves at all -- +# that entry point is gone. Note MARKETDATA_STORE: a DIFFERENT directory from +# COTDATA_STORE, not an alias for it. COT is still cotdata's job and still has its +# own wrapper (run-cot.sh) in that repo. +# +# See docs/LINUX_SCHEDULING.md for the crontab and flock setup. +set -uo pipefail # NOT -e: see the two-stage note below + +export MARKETDATA_STORE=REPLACE_WITH_STORE_PATH +export DATABENTO_API_KEY=REPLACE_WITH_DATABENTO_KEY +# Optional. Defaults to _raw/databento under the store. Put it on cheap disk if the +# store is on something small or expensive -- the bronze store is the bulk of the +# bytes and never leaves this machine. +# export MARKETDATA_DATABENTO_RAW=/srv/databento_raw + +BIN=REPLACE_WITH_VENV_PATH/bin/marketdata-update + +# Stage 1 is the PAID pull and Stage 2 is a free local rebuild, so Stage 2 runs even +# when Stage 1 fails -- and that is deliberate rather than sloppy. Stage 1 is +# resumable and writes incrementally, so a network failure partway through still +# leaves new raw data on disk; refusing to build from it would discard something +# already paid for, and the next run would have to pay again to get back to the same +# place. The Stage 1 exit code is kept and returned at the end, so cron still sees a +# failure and still retries. +rc=0 +"$BIN" --ingest-databento || rc=$? # Stage 1 (PAID): raw .n.0/.n.1 -> raw store +"$BIN" --build-databento || rc=$? # Stage 2 (FREE): backadj+unadj bars + +exit "$rc"