Repository navigation
fix: inclusive date bounds, UTC conversions, and Massive aggregate pagination - #74
Open
quantum-like wants to merge 7 commits into
Open
quantum-like wants to merge 7 commits into
quantum-like wants to merge 7 commits into
Conversation
A YYYY-MM end bound used day 28, so daily and weekly observations on the 29th through the 31st were dropped from the requested month. Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
Storage reads treat the end bound as exclusive, but gap detection treats it as inclusive. A bar timestamped on the requested end date was omitted and reported as a one-day gap. Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
yfinance treats the download end as exclusive. The treasury fallback passed the configured end through unchanged, so the last requested session was omitted. Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
Naive datetimes were converted with the local timezone, and a date-only end stopped at midnight, so the requested end day was shifted or truncated. Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
Date strings were turned into unix timestamps in the process timezone, so the requested window moved by the local offset. Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
Chunk windows used naive local timestamps, so the CLOB query did not cover the UTC calendar days the caller requested. Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
OHLCV responses stopped after the first page. Requests larger than the 50,000-row limit silently returned only the earliest bars. Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Owning issue
Closes #73
Outcome
7 correctness fixes, one commit each, each with a focused test:
YYYY-MMend filters used day 28, so daily/weekly rows on the 29th–31st were dropped. They now use the real month end. Test:tests/test_fama_french_provider.py::test_fetch_month_filter_includes_final_calendar_day. (3a942fc)tests/test_update_manager.py::test_detect_gaps_in_storage_keeps_bar_on_inclusive_end. (9825551)endis exclusive. It now requests the next day. Test: updated assertion intests/test_macro_downloader.py. (b062fa5)tests/test_finnhub_provider.py::test_fetch_raw_data_uses_inclusive_utc_day. (9501712)tests/test_kalshi_unit.py::test_fetch_raw_data_uses_utc_calendar_bounds. (d4a4008)tests/test_polymarket_unit.py::test_fetch_price_history_uses_utc_day_bounds. (55f6523)providers/polygon.py) returned only the first 50,000-row page, silently truncating long minute history. They now follownext_url. Test:tests/test_polygon_provider.py::test_fetch_raw_data_follows_next_url. (e09a0e8)Compatibility
Requests now include the configured end date and use UTC day bounds, so results can contain bars that were previously dropped; long Massive histories return all pages.
Verification
mainand passes with its fix)ruff format --checkis clean on the changed files andruff checkreports nothing new in them versusmain, but I did not run the full pre-commit suite)typasses (not run on this branch)pytest tests -q -ra: 3656 passed before, 3662 passed after (283 network, paid-tier and credentialed tests deselected both times). The 7 touched modules also pass underTZ=America/New_York(293 passed). Integration, paid-tier and credentialed tests were not run.Documentation and release
No documentation changes. The behavior changes above may be worth a line in the next patch-release notes.
Amir