Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/provider-contracts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,11 @@ jobs:
tests/integration/test_binance_public.py::TestBinancePublicProvider::test_fetch_daily_spot_btc
tests/integration/test_coingecko.py::TestCoinGeckoProvider::test_fetch_ohlcv_btc
tests/integration/test_kalshi.py::TestKalshiProvider::test_list_markets
tests/integration/test_kalshi.py::TestKalshiResolvedHistory
tests/integration/test_forecastex.py
tests/integration/test_polymarket_us.py
tests/integration/test_polymarket.py::TestPolymarketProvider::test_get_market_by_slug
tests/integration/test_polymarket.py::TestPolymarketResolvedHistory
tests/integration/test_yahoo.py::TestYahooFinanceProvider::test_fetch_ohlcv_stock_daily
tests/integration/test_release_provider_contracts.py

Expand Down
2 changes: 1 addition & 1 deletion docs/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
- [x] DataBento (institutional, trial)
- [x] Wiki Prices (historical 1962-2018)
- [x] Tiingo, Finnhub, Polygon, Twelve Data, CryptoCompare, Oanda
- [x] FRED, AQR, Fama-French, Kalshi, Polymarket, OKX, Binance public bulk, ITCH sample
- [x] FRED, AQR, Fama-French, Kalshi, Polymarket, Polymarket US, ForecastEx, OKX, Binance public bulk, ITCH sample
- [x] Synthetic and mock providers for testing

### Storage (Complete)
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/provider-selection.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ data = SyntheticProvider(seed=42).fetch_ohlcv(
| Futures and options | Databento, Binance, OKX | Historical or licensed data may be metered |
| [Macroeconomic series](../providers/macro.md) | FRED, FXMacroData | FRED requires an API key for normal use |
| [Research factors](../providers/factors.md) | Fama-French, AQR | Network access and source-specific terms |
| [Prediction markets](../providers/prediction_markets.md) | Kalshi, Polymarket | Network access and changing public endpoints |
| [Prediction markets](../providers/prediction_markets.md) | Kalshi, ForecastEx, Polymarket US, Polymarket | Network access and changing public endpoints |
| Frozen equity history | Wiki Prices | Local historical dataset ending in 2018 |
| CFTC positioning | COT | Install the `cot` extra |

Expand Down
98 changes: 98 additions & 0 deletions docs/providers/forecastex.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# ForecastEx Provider

**Provider**: `ForecastExProvider`
**Website**: [forecastex.com](https://forecastex.com/data)
**API Key**: Not required
**Free Tier**: Free daily files from 2024-08-01

---

## Overview

ForecastEx LLC is a CFTC-registered exchange (DCM) and clearinghouse (DCO) affiliated with
Interactive Brokers. Each Forecast Market lists a YES and a NO contract that settle at $1.00 or
$0.00. ForecastEx publishes daily CSV files in the public S3 bucket `forecastex-public-data`;
the provider reads them directly.

| File | Content | Provider method |
|------|---------|-----------------|
| `prices/daily_prices_YYYYMMDD.csv` | Daily open, high, low, close, settlement price, paired quantity, open interest and VWAP per contract side | `fetch_prices()` |
| `pairs/pairs_YYYYMMDD.csv` | Every matched YES/NO pair (trade); the current day's file is refreshed every 10 minutes | `fetch_trades()` |
| `daily_summary/summary_YYYYMMDD.csv` | Product names and categories (from 2025-01-01) | `fetch_products()` |

All methods take an inclusive range of trading dates (Central Time). Days without a published
file are skipped, and a range with no files returns an empty, typed frame.

---

## Quick Start

```python
from ml4t.data.providers import ForecastExProvider

provider = ForecastExProvider()

markets = provider.fetch_markets("2026-09-15", "2026-09-16", product="FFDEC")
trades = provider.fetch_trades("2026-09-16", "2026-09-16", product="FFDEC")
prices = provider.fetch_prices("2026-09-16", "2026-09-16", product="FFDEC")

provider.close()
```

`event_contract` (the `ticker` column) encodes product, date and threshold, for example
`FFDEC_091626_E25`. The `product` column is the prefix before the first underscore. Suffix
semantics come from each product's terms PDF under `regulatory/` in the same bucket.

---

## Resolution Outcome

The files carry no outcome column. `fetch_markets()` derives it:

1. It also reads the prices file for the day after `end`. A contract absent from the next
published file has expired.
2. If the YES `settlement_price` on the contract's last row is 1.00 or 0.00, that is the payout:
`status="resolved"`, `result` is `"yes"` or `"no"`, and `settlement_value` is the payout.
3. Contracts that expire after the daily file is cut, such as hourly temperature products, drop
out with their last daily mark instead of the payout. They are reported as
`status="expired"` with a null `result`, because the files do not publish their outcome.
4. Contracts still listed in the next file, or trading on `end` when the next file is not yet
published, are `status="open"`.

`settlement_ts` is the contract's expiration time; ForecastEx publishes no separate settlement
timestamp. The clearinghouse settles daily at 13:00 CT for contracts resolved in the previous
24 hours.

| Column | Meaning |
|--------|---------|
| `ticker`, `product`, `expiration` | Event contract, product code, expiration (UTC) |
| `status` | `resolved`, `expired` or `open` |
| `first_date`, `last_date` | First and last trading date seen in the range |
| `result`, `settlement_value`, `settlement_ts` | Normalized outcome, YES payout, expiration time |
| `last_close`, `volume` | Last daily close and paired contracts in the range |

---

## Trades

`fetch_trades()` returns one row per matched pair with the shared prediction-market trade columns
`trade_id`, `ticker`, `timestamp` (UTC), `price` (YES price), `count`, `taker_side` and
`is_block_trade`, plus `no_price`, `product` and `expiration`. ForecastEx pairs have no aggressor
side, so `taker_side` is always null, and `is_block_trade` is always false.

---

## Limits

- No order-book history; the live book is available only through an FCM such as IBKR.
- No documented rate limit; the provider defaults to 10 requests per second.
- High and low are 0.00 on days without trades, while open and close carry the daily marks.

---

## See Also

- [Prediction-market data sources](prediction_markets.md)
- [Kalshi Provider](kalshi.md)
- [Polymarket US Provider](polymarket_us.md)
- [ForecastEx data page](https://forecastex.com/data)
6 changes: 4 additions & 2 deletions docs/providers/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ For the wider vendor landscape, including sources the library does not wrap, sta
| [Wiki Prices](wiki_prices.md) | Historical | Static | Thread | No |
| [Kalshi](kalshi.md) | Prediction Markets | Public data | Thread | No |
| [Polymarket](polymarket.md) | Prediction Markets | Public data | Thread | No |
| [Polymarket US](polymarket_us.md) | Prediction Markets | Public data | Thread | No |
| [ForecastEx](forecastex.md) | Prediction Markets | Daily files | Thread | No |
| [Binance Public](binance_public.md) | Crypto | Bulk downloads | Thread | No |
| [NASDAQ ITCH](nasdaq_itch.md) | Tick Data | Sample data | Thread | No |
| [EODHD](eodhd.md) | Global Stocks | 20 calls/day | Native | Yes |
Expand Down Expand Up @@ -80,15 +82,15 @@ provider's capabilities before passing it to `DataManager` or `async_batch_load(

| Requirement | Providers to Evaluate |
|-------------|-----------------------|
| No credential | Yahoo Finance, CoinGecko, Fama-French, AQR, Kalshi, Polymarket, Binance Public, NASDAQ ITCH |
| No credential | Yahoo Finance, CoinGecko, Fama-French, AQR, Kalshi, Polymarket, Polymarket US, ForecastEx, Binance Public, NASDAQ ITCH |
| US equities | Yahoo Finance, Alpaca, Tiingo, Massive |
| Global equities | EODHD, Finnhub, Twelve Data |
| Futures and options | Databento, Massive |
| Cryptocurrency | Binance, Binance Public, OKX, CoinGecko, CryptoCompare |
| Foreign exchange | Oanda, Twelve Data, FXMacroData |
| Economic series | FRED, FXMacroData; external official sources are compared in [Macroeconomic Data Sources](macro.md) |
| Academic factors | Fama-French, AQR; external libraries are compared in [Research Factor Data Sources](factors.md) |
| Prediction markets | Kalshi, Polymarket; external markets are compared in [Prediction-Market Data Sources](prediction_markets.md) |
| Prediction markets | Kalshi, ForecastEx, Polymarket US, Polymarket; external markets are compared in [Prediction-Market Data Sources](prediction_markets.md) |
| Equity fundamentals | Yahoo Finance, EODHD, Finnhub, Massive; other vendors and the SEC routes are compared in [Fundamental Data Sources](fundamentals.md) |

Provider access, coverage, retention, and redistribution terms can differ by account tier. Confirm
Expand Down
110 changes: 110 additions & 0 deletions docs/providers/kalshi.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,114 @@ provider.close()

---

## Resolved Markets and Outcomes

`fetch_markets()` enumerates every market matching the filters, following Kalshi's cursor
pagination across two tiers: the live `GET /markets` endpoint and the archive
`GET /historical/markets`, which alone holds markets settled before the cutoff reported by
`GET /historical/cutoff` (`get_historical_cutoff()`). The archive accepts only one of
`event_ticker`, `series_ticker` or `mve_filter=exclude`, so the remaining filters (status,
close-time window, the other multivariate selections) are applied on the client. Archived
markets are all settled, so the archive is skipped for other statuses.

`mve_filter` selects multivariate combo (parlay) markets, whose tickers start with `KXMVE`:
None includes them (Kalshi's default), `"exclude"` drops them and `"only"` keeps only them. The
archive rejects `mve_filter=only`, so selecting combos there pages through the whole archive and
filters on the client; use `max_pages` to sample.

```python
from ml4t.data.providers import KalshiProvider

provider = KalshiProvider()

fed = provider.fetch_markets(status="settled", series_ticker="KXFEDDECISION")
print(fed.select("ticker", "result", "settlement_value", "settlement_ts", "source"))

old = provider.fetch_markets(event_ticker="FED-23DEC")
window = provider.fetch_markets(
status="settled", min_close_ts="2026-01-01", max_close_ts="2026-06-30",
mve_filter="exclude",
)
combos = provider.fetch_markets(status="settled", mve_filter="only", max_pages=1)
provider.close()
```

| Column | Meaning |
|--------|---------|
| `ticker`, `event_ticker`, `title`, `status`, `market_type` | Market identity and lifecycle status |
| `series` | The `series_ticker` filter when given, otherwise the event-ticker prefix (legacy events such as `FED-23DEC` belong to a differently named series, `KXFED`) |
| `open_time`, `close_time` | Trading window (UTC) |
| `result` | `yes`, `no`, `other` (scalar settlement) or null while unresolved |
| `result_raw` | Kalshi's own `result` label |
| `settlement_value` | YES payout in dollars |
| `settlement_ts` | Settlement time (UTC) |
| `expiration_value`, `last_price`, `previous_price`, `volume`, `open_interest` | Settlement source value and last market statistics (prices in dollars, volume in contracts) |
| `source` | `live` or `historical` tier |

`iter_market_frames(chunk_size=100_000, ...)` streams the same normalized rows as frames of at
most `chunk_size` rows, for listings too large to hold in memory (the full settled listing runs to
millions of markets). `iter_markets()` yields the raw market dictionaries instead. `list_markets()` keeps returning a
single page of raw dictionaries.

---

## Price History Across Tiers

`fetch_candles(ticker, start_ts, end_ts, period="1h", series_ticker=None, tier=None)` returns
candlesticks for one market from whichever tier holds it. Markets settled before the historical
cutoff are served only by `GET /historical/markets/{ticker}/candlesticks`; the others by
`GET /series/{series}/markets/{ticker}/candlesticks`, which needs the market's series.

- Pass `tier` (the `source` column of `fetch_markets()`) to go straight to the right endpoint.
- Pass `series_ticker` (the `series` column, or the event's series) for live markets. Without
it, the ticker prefix is tried first; if both tiers answer 404, the series is looked up from
the market's event and the live endpoint is retried.
- Without `tier`, the live endpoint is tried first and the archive on a 404.
- Ranges longer than Kalshi's 5000-candle limit are split into several requests.

```python
candles = provider.fetch_candles(
"KXFEDDECISION-26JUL-H0", "2026-07-26", "2026-07-29", period="1h",
series_ticker="KXFEDDECISION", tier="historical",
)
```

| Column | Meaning |
|--------|---------|
| `timestamp` | End of the candle period (UTC) |
| `open`, `high`, `low`, `close`, `mean`, `previous` | Trade prices in dollars; null when nothing traded (except `previous`) |
| `yes_bid_close`, `yes_ask_close` | Closing YES bid and ask in dollars |
| `volume`, `open_interest` | Contracts |
| `source` | `live` or `historical` tier |

Live candles carry `*_dollars`/`*_fp` fields and archive candles unsuffixed dollar strings;
both are returned in dollars (0-1). `fetch_ohlcv()` uses the same tier routing, so it also
works for archived markets such as `FED-23DEC-T5.25`, and it keeps its OHLCV schema.

---

## Trade History

`fetch_trades(ticker, min_ts=None, max_ts=None)` pages through `GET /markets/trades` and
`GET /historical/trades`, skipping a tier the time window cannot reach, and returns one row per
trade sorted by time:

| Column | Meaning |
|--------|---------|
| `trade_id`, `ticker` | Trade and market identifiers |
| `timestamp` | Trade time (UTC) |
| `price` | YES price in dollars (0-1) |
| `count` | Contracts; fractional counts are allowed |
| `taker_side` | `yes` or `no`; falls back to `taker_outcome_side` |
| `is_block_trade` | Block-trade flag |
| `taker_outcome_side`, `taker_book_side` | Outcome side and book side (`bid`/`ask`) of the taker, null when absent |
| `source` | `live` or `historical` tier |

Prices and counts are read from the fixed-point `*_dollars` and `*_fp` fields; legacy unsuffixed
fields are also accepted (integers as cents, strings as dollars).

---

## Regulatory Status

Kalshi is regulated by the CFTC (Commodity Futures Trading Commission) as a Designated Contract Market (DCM).
Expand All @@ -60,4 +168,6 @@ Kalshi is regulated by the CFTC (Commodity Futures Trading Commission) as a Desi
- [Prediction-market data sources](prediction_markets.md)
- [Kalshi Developer Docs](https://kalshi.com/developer)
- [Polymarket Provider](polymarket.md)
- [Polymarket US Provider](polymarket_us.md)
- [ForecastEx Provider](forecastex.md)
- [Provider reference](index.md)
Loading
Loading