Skip to content

fix: inclusive date bounds, UTC conversions, and Massive aggregate pagination - #74

Open
quantum-like wants to merge 7 commits into
ml4t:mainfrom
quantum-like:review/fixes
Open

quantum-like wants to merge 7 commits into
ml4t:mainfrom
quantum-like:review/fixes

Conversation

@quantum-like

Copy link
Copy Markdown

Owning issue

Closes #73

Outcome

7 correctness fixes, one commit each, each with a focused test:

  • Fama-French YYYY-MM end filters used day 28, so daily/weekly rows on the 29th–31st were dropped. They now use the real month end. Test: tests/test_fama_french_provider.py::test_fetch_month_filter_includes_final_calendar_day. (3a942fc)
  • Storage gap detection passed an inclusive end into an exclusive read, so a complete series reported a one-day gap on the end date. The end bar is now included. Test: tests/test_update_manager.py::test_detect_gaps_in_storage_keeps_bar_on_inclusive_end. (9825551)
  • The macro yfinance fallback omitted the configured end date because Yahoo's end is exclusive. It now requests the next day. Test: updated assertion in tests/test_macro_downloader.py. (b062fa5)
  • Finnhub date bounds were converted in local time and a date-only end dropped that day's bars. Bounds are now UTC, with the end at 23:59:59 UTC. Test: tests/test_finnhub_provider.py::test_fetch_raw_data_uses_inclusive_utc_day. (9501712)
  • Kalshi candlestick bounds were converted in local time. They are now UTC. Test: tests/test_kalshi_unit.py::test_fetch_raw_data_uses_utc_calendar_bounds. (d4a4008)
  • Polymarket price-history bounds were converted in local time. They are now UTC. Test: tests/test_polymarket_unit.py::test_fetch_price_history_uses_utc_day_bounds. (55f6523)
  • Massive aggregate fetches (providers/polygon.py) returned only the first 50,000-row page, silently truncating long minute history. They now follow next_url. Test: tests/test_polygon_provider.py::test_fetch_raw_data_follows_next_url. (e09a0e8)

Compatibility

  • No public compatibility impact
  • Compatibility impact is documented in the owning issue and migration guidance

Requests now include the configured end date and use UTC day bounds, so results can contain bars that were previously dropped; long Massive histories return all pages.

Verification

  • Tests reproduce the prior failure or verify the requested behavior (each listed test fails on main and passes with its fix)
  • Ruff lint and format checks pass (ruff format --check is clean on the changed files and ruff check reports nothing new in them versus main, but I did not run the full pre-commit suite)
  • ty passes (not run on this branch)
  • Test suite passes: Offline pytest tests -q -ra: 3656 passed before, 3662 passed after (283 network, paid-tier and credentialed tests deselected both times). The 7 touched modules also pass under TZ=America/New_York (293 passed). Integration, paid-tier and credentialed tests were not run.
  • Package build passes when applicable (not run)
  • Strict MkDocs build passes when documentation is affected (no documentation changes)
  • Ecosystem qualification passes (not run)

Documentation and release

No documentation changes. The behavior changes above may be worth a line in the next patch-release notes.

Amir

cursoragent and others added 7 commits October 7, 2026 07:34
A YYYY-MM end bound used day 28, so daily and weekly observations on the
29th through the 31st were dropped from the requested month.

Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
Storage reads treat the end bound as exclusive, but gap detection treats
it as inclusive. A bar timestamped on the requested end date was omitted
and reported as a one-day gap.

Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
yfinance treats the download end as exclusive. The treasury fallback passed
the configured end through unchanged, so the last requested session was omitted.

Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
Naive datetimes were converted with the local timezone, and a date-only end
stopped at midnight, so the requested end day was shifted or truncated.

Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
Date strings were turned into unix timestamps in the process timezone, so
the requested window moved by the local offset.

Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
Chunk windows used naive local timestamps, so the CLOB query did not cover
the UTC calendar days the caller requested.

Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>
OHLCV responses stopped after the first page. Requests larger than the
50,000-row limit silently returned only the earliest bars.

Co-authored-by: quantum-like <quantum-like@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants