Skip to content

Add the Linux scheduling docs the databento port left behind - #16

Merged
mspinola merged 1 commit into
mainfrom
claude/linux-scheduling
Aug 21, 2026
Merged

Add the Linux scheduling docs the databento port left behind#16
mspinola merged 1 commit into
mainfrom
claude/linux-scheduling

Conversation

@mspinola

@mspinola mspinola commented Aug 9, 2026

Copy link
Copy Markdown
Owner

#15 moved the databento provider here; its operational half stayed in cotdata and then went with cotdata's own price docs. So the box that actually runs databento has a wrapper calling cotdata-prices — an entry point that no longer resolves — and nothing in this repo telling it what to call instead.

  • docs/examples/linux/run-prices.sh — the replacement to copy onto the box
  • docs/LINUX_SCHEDULING.md — crontab, flock, troubleshooting

Two changes from the script it replaces, both about not paying twice

Stage 2 now runs even when Stage 1 fails, and the script drops set -e to allow it. Stage 1 is the only step that costs money, it writes incrementally, and it is resumable — so a network failure partway through still leaves new raw data on disk. Aborting before the free local rebuild would discard something already paid for, and the next run would have to pay again to reach the same place. Stage 1's exit code is captured and returned, so cron still sees a failure and still retries.

The cold-start backfill is documented as a manual step, not the nightly job, with --windowed-n1-stats on it. That flag restricts the second contract's statistics to windows around roll dates — the only place that series is read, so it is accuracy-neutral and it is the largest avoidable download. It buys nothing on a nightly incremental, so it does not belong in the crontab line.

Troubleshooting leads with the failure this move causes

command not found, from a wrapper still calling cotdata-prices.

It also covers the stuck-symbol case worth knowing about: a ledger entry claiming rows whose parquet is missing still carries a current last_date, so every run skips it as "already current". No error, no row, a permanent hole in a paid dataset. --reconcile-databento is the repair, and it fixes both directions.

Verified rather than asserted

  • the quoted no databento-capable symbols message matches the provider's actual string
  • the eight databento: null markets are the eight named (CC, CT, DX, KC, LBR, OJ, SB, WBS)
  • the rc-propagation shape was exercised with a stub: both stages run, exit code 7 survives

ruff clean; 200 passed, 12 skipped (unchanged — docs only).

Still needs a human

I can't reach the server, so the wrapper on it is unchanged. Copy this file over, fill in the three markers, and update the crontab path.


Generated by Claude Code

The provider moved here in #15; its operational half stayed in cotdata and then
went with cotdata's own price docs. So the box that actually runs databento has a
wrapper calling `cotdata-prices` -- an entry point that no longer resolves -- and
nowhere in this repo telling it what to call instead.

docs/examples/linux/run-prices.sh is the replacement to copy, and
docs/LINUX_SCHEDULING.md is the crontab, the flock setup and the troubleshooting.

TWO CHANGES FROM THE SCRIPT IT REPLACES, both about not paying twice.

Stage 2 now runs even when Stage 1 fails, and the script drops `set -e` to allow
it. Stage 1 is the only step that costs money, it writes incrementally and it is
resumable, so a network failure partway through still leaves new raw data on disk.
Aborting before the free local rebuild would discard something already paid for,
and the next run would have to pay again to reach the same place. Stage 1's exit
code is captured and returned, so cron still sees a failure and still retries.

The cold-start backfill is documented as a manual step rather than the nightly job,
with --windowed-n1-stats on it. That flag restricts the second contract's
statistics to windows around roll dates, where the series is the only place it is
read -- accuracy-neutral, and the largest avoidable download. It buys nothing on a
nightly incremental, so it does not belong in the crontab line.

Troubleshooting leads with the failure this move causes -- "command not found" from
a wrapper still calling cotdata-prices -- and covers the stuck-symbol case that is
worth knowing about: a ledger entry claiming rows whose parquet is missing carries a
current last_date, so every run skips it as "already current". No error, no row, a
permanent hole in a paid dataset. --reconcile-databento is the repair.

Verified rather than asserted: the quoted "no databento-capable symbols" message
matches the provider, the eight `databento: null` markets are the eight named, and
the rc-propagation shape was exercised with a stub -- both stages run, exit code 7
survives.
@mspinola
mspinola marked this pull request as ready for review August 9, 2026 17:27
@mspinola
mspinola merged commit 84bb51f into main Aug 21, 2026
5 checks passed
@mspinola
mspinola deleted the claude/linux-scheduling branch August 24, 2026 23:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants