Add the Linux scheduling docs the databento port left behind - #16
Merged
Conversation
The provider moved here in #15; its operational half stayed in cotdata and then went with cotdata's own price docs. So the box that actually runs databento has a wrapper calling `cotdata-prices` -- an entry point that no longer resolves -- and nowhere in this repo telling it what to call instead. docs/examples/linux/run-prices.sh is the replacement to copy, and docs/LINUX_SCHEDULING.md is the crontab, the flock setup and the troubleshooting. TWO CHANGES FROM THE SCRIPT IT REPLACES, both about not paying twice. Stage 2 now runs even when Stage 1 fails, and the script drops `set -e` to allow it. Stage 1 is the only step that costs money, it writes incrementally and it is resumable, so a network failure partway through still leaves new raw data on disk. Aborting before the free local rebuild would discard something already paid for, and the next run would have to pay again to reach the same place. Stage 1's exit code is captured and returned, so cron still sees a failure and still retries. The cold-start backfill is documented as a manual step rather than the nightly job, with --windowed-n1-stats on it. That flag restricts the second contract's statistics to windows around roll dates, where the series is the only place it is read -- accuracy-neutral, and the largest avoidable download. It buys nothing on a nightly incremental, so it does not belong in the crontab line. Troubleshooting leads with the failure this move causes -- "command not found" from a wrapper still calling cotdata-prices -- and covers the stuck-symbol case that is worth knowing about: a ledger entry claiming rows whose parquet is missing carries a current last_date, so every run skips it as "already current". No error, no row, a permanent hole in a paid dataset. --reconcile-databento is the repair. Verified rather than asserted: the quoted "no databento-capable symbols" message matches the provider, the eight `databento: null` markets are the eight named, and the rc-propagation shape was exercised with a stub -- both stages run, exit code 7 survives.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#15 moved the databento provider here; its operational half stayed in cotdata and then went with cotdata's own price docs. So the box that actually runs databento has a wrapper calling
cotdata-prices— an entry point that no longer resolves — and nothing in this repo telling it what to call instead.docs/examples/linux/run-prices.sh— the replacement to copy onto the boxdocs/LINUX_SCHEDULING.md— crontab,flock, troubleshootingTwo changes from the script it replaces, both about not paying twice
Stage 2 now runs even when Stage 1 fails, and the script drops
set -eto allow it. Stage 1 is the only step that costs money, it writes incrementally, and it is resumable — so a network failure partway through still leaves new raw data on disk. Aborting before the free local rebuild would discard something already paid for, and the next run would have to pay again to reach the same place. Stage 1's exit code is captured and returned, so cron still sees a failure and still retries.The cold-start backfill is documented as a manual step, not the nightly job, with
--windowed-n1-statson it. That flag restricts the second contract's statistics to windows around roll dates — the only place that series is read, so it is accuracy-neutral and it is the largest avoidable download. It buys nothing on a nightly incremental, so it does not belong in the crontab line.Troubleshooting leads with the failure this move causes
command not found, from a wrapper still callingcotdata-prices.It also covers the stuck-symbol case worth knowing about: a ledger entry claiming rows whose parquet is missing still carries a current
last_date, so every run skips it as "already current". No error, no row, a permanent hole in a paid dataset.--reconcile-databentois the repair, and it fixes both directions.Verified rather than asserted
no databento-capable symbolsmessage matches the provider's actual stringdatabento: nullmarkets are the eight named (CC, CT, DX, KC, LBR, OJ, SB, WBS)ruffclean; 200 passed, 12 skipped (unchanged — docs only).Still needs a human
I can't reach the server, so the wrapper on it is unchanged. Copy this file over, fill in the three markers, and update the crontab path.
Generated by Claude Code