Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,17 @@ moves every bar here.
uv venv --python 3.11 && uv pip install -e ".[yahoo,dev]" "setuptools<81"
```

On the **Windows futures producer**, add the `norgate` extra — nothing else pulls
`norgatedata`, and without it `--domain futures` stops before it fetches:

```bash
uv pip install -e ".[yahoo,norgate,dev]" "setuptools<81"
```

Installing it elsewhere does not help. It drives a locally installed Norgate Data
Updater rather than an API, and NDU is Windows-only, so every other machine reads
a synced store instead of producing one.

## Use

```bash
Expand Down Expand Up @@ -151,6 +162,44 @@ silently substituting a vendor is what ADR-0006 forbids.
The store root may share a parent folder with cotdata's, but the two must not
share a `manifest.json`. Both producers do a read-modify-write on it.

### On the Windows futures producer

That box now runs **two producers**, so it needs **both** store variables set at
once, pointing at **different roots**. This is new with the futures domain: until
ADR-0007 moved bars here, `COTDATA_STORE` alone was the whole story.

```cmd
setx COTDATA_STORE C:\Users\YourUsername\cotdata_store
setx MARKETDATA_STORE C:\Users\YourUsername\marketdata_store
```

`setx` persists; plain `set` lasts only for the current Command Prompt, which is
the usual reason a scheduled task cannot find a store an interactive shell could.
Open a NEW prompt afterwards — `setx` does not affect the one you typed it in —
and verify:

```cmd
echo %COTDATA_STORE%
echo %MARKETDATA_STORE%
marketdata-update --check
```

`--check` reads the manifest and no network, so it is the cheap confirmation that
the variable points where you think. An unset variable is refused by name rather
than defaulted, because a silent default would write a second store somewhere
nobody looks.

**Do not point them at one root.** Sharing a parent folder is fine and makes the
pair easy to sync; sharing a root is not, because both packages keep a
`manifest.json` at their root and each does a read-modify-write on it, so the two
producers would eventually drop each other's entries.

Python, virtualenv and Task Scheduler setup are identical to cotdata's and are
not duplicated here — see
[cotdata's Windows setup guide](https://github.com/mspinola/cotdata/blob/main/docs/WINDOWS_SETUP.md).
The only marketdata-specific pieces are the `norgate` extra in **Install** above
and the two variables here.

## Tests

```bash
Expand Down
38 changes: 38 additions & 0 deletions docs/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,44 @@ use — they read a **synced** one. That is the answer, not a temporary state:
failing, and `--domain futures` on such a box explains why instead of raising
`ModuleNotFoundError`.

### Verified against cotdata, 2026-08-09

Run on the Windows producer with `scripts/verify_against_cotdata.py`, against a
cotdata store built by the original producer. **Every compared series identical,
exit 0.**

| symbol | rows per tier | passthrough | reconstruction |
|---|---:|---|---|
| ES | 7,279 | identical | identical |
| CL | 10,887 | identical | identical |
| GC | 12,156 | identical | identical |
| ZS | 12,271 | identical | identical |
| DC | 7,299 | identical | identical |

49,892 rows per tier, both tiers, plus contract specs for all five. Exact
equality, not a tolerance: the two producers drive the same Norgate install
through two code paths, so any difference would have been a port bug rather than
vendor disagreement.

Two things worth recording beyond the verdict.

**The reconstruction columns matched too, which was not expected.** Both
producers reconstruct volume incrementally over their own store's history, so a
fresh marketdata store recomputing 12,000 bars and a cotdata store that
accumulated them over months looked like a legitimate source of drift. They agree
exactly, and the reason holds generally: Norgate's historical individual-contract
volumes are immutable and the algorithm is the same, so the incremental path
converges on what a full recompute produces. `--strict-volume` is therefore
usable rather than theoretical.

**The first two runs found defects offline testing could not.** `--domain
futures` stopped at the import guard because the `norgate` extra was never
declared, and the reconstruction columns turned out to have no consumer-side
`volume=` switch — the producer wrote them and nothing served them. Neither is
visible to a test suite that cannot install the vendor or call the missing
parameter. The comparison harness is what caught the second, by reporting which
columns it had NOT compared instead of staying silent about them.

### What is NOT ported

`MME` and `MFS` (MSCI EM and EAFE). Norgate carries no continuous series for
Expand Down
7 changes: 7 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@ Issues = "https://github.com/mspinola/marketdata/issues"
# history(auto_adjust=...) has changed across yfinance versions — see
# docs/design.md. Verified against 1.5.2.
yahoo = ["yfinance>=1.5.2"]
# The futures producer, WINDOWS ONLY. The package installs anywhere, but it drives
# a locally installed Norgate Data Updater rather than an API, and NDU exists for
# Windows alone — so on a Mac or Linux box it imports fine and then has nothing to
# talk to. That is why this is an extra rather than a dependency, and why the
# provider guards the import and the service separately: a missing package and a
# stopped NDU are different problems with different fixes.
norgate = ["norgatedata"]
dev = ["pytest>=7", "ruff==0.15.22"]

[project.scripts]
Expand Down
Loading
Loading