Declare marketdata, and document the second store the server now needs

.github/workflows/python-test.yml

@@ -30,6 +30,15 @@ jobs:
         repository: mspinola/cotdata
         path: cotdata
 
+    - name: Check out marketdata (daily bars — resolves `-e ../marketdata`)
+      # ADR-0007 moved bars out of cotdata, and marketdata is not on PyPI, so this
+      # sibling checkout is the only thing that resolves the dependency cotmetrics
+      # now imports.
+      uses: actions/checkout@v4
+      with:
+        repository: mspinola/marketdata
+        path: marketdata
+
     - name: Check out cotmetrics (metrics/data layer — resolves `-e ../cotmetrics`)
       uses: actions/checkout@v4
       with:
@@ -48,7 +57,7 @@ jobs:
         python -m pip install -r requirements.txt
 
     - name: Verify internal dependency floors
-      # cotdata / cotmetrics are installed editable from ../ at whatever HEAD was
+      # cotdata / marketdata / cotmetrics are installed editable from ../ at whatever HEAD was
       # checked out, so pip never enforces the version floors in pyproject.toml.
       # This asserts the installed siblings actually satisfy those declared floors.
       working-directory: cot-analyzer
@@ -62,9 +71,12 @@ jobs:
       working-directory: cot-analyzer
       env:
         PYTHONPATH: src
-        # Dummy store so cotdata's COTDATA_STORE guard doesn't error at import.
+        # Dummy stores so each package's store guard doesn't error at import.
+        # Two of them since ADR-0007: positioning and bars are separate roots, and
+        # neither package defaults a missing root to somewhere plausible.
         COTDATA_STORE: /tmp/cotdata_store
+        MARKETDATA_STORE: /tmp/marketdata_store
         COTMETRICS_CACHE: /tmp/cotmetrics_cache
       run: |
-        mkdir -p /tmp/cotdata_store /tmp/cotmetrics_cache
+        mkdir -p /tmp/cotdata_store /tmp/marketdata_store /tmp/cotmetrics_cache
         pytest tests/

pyproject.toml

@@ -33,6 +33,7 @@ dependencies = [
     "pyyaml==6.0.3",
     "requests==2.32.5",
     "cotdata>=0.1.0",
+    "marketdata>=0.1.0",       # bars; ADR-0007 moved them out of cotdata
     "cotmetrics[options]>=0.3.0",
 ]
 

requirements.txt

@@ -18,10 +18,12 @@ ruff==0.15.22           # Linter — pinned so CI installs it and stays reproduc
 tomli==2.0.1; python_version < "3.11"   # for scripts/check_dep_floors.py on <3.11
 
 # data/metrics layer (editable siblings). cotmetrics provides the COT index,
-# signals, indexer + the legacy ETL/options/scheduler; it depends on cotdata.
-# Both are published on PyPI (cotdata, cotmetrics), but we install them EDITABLE
+# signals, indexer + the legacy ETL/options/scheduler; it depends on cotdata for
+# CFTC positioning and, since ADR-0007 moved bars out, on marketdata for prices.
+# All three are published on PyPI, but we install them EDITABLE
 # from sibling checkouts so in-workspace changes are picked up immediately and CI
 # catches integration breaks before those siblings are released. Pin the PyPI
 # releases instead if you want reproducible, checkout-free builds.
 -e ../cotdata
+-e ../marketdata
 -e ../cotmetrics[options]

scripts/check_dep_floors.py

@@ -31,6 +31,7 @@
 # rather than a package index, so only these need the extra check.
 INTERNAL = {
     "cotdata",
+    "marketdata",
     "cotmetrics",
     "crucible",
     "crucible-stack",

server-side/README.md

@@ -140,12 +140,23 @@ chmod 600 /root/trading_workspace/cot-analyzer/.env
 **Required:**
 
 ```bash
-COTDATA_STORE=/root/cotdata_store     # the synced price store, see step 6
+COTDATA_STORE=/root/cotdata_store        # synced CFTC positioning store, see step 6
+MARKETDATA_STORE=/root/marketdata_store  # synced daily-bar store, see step 6
 ```
 
 Nothing works without `COTDATA_STORE`. Every entry point resolves instruments through
 it, so a missing or empty store fails at import rather than degrading.
 
+**`MARKETDATA_STORE` is required too, and it is new.** ADR-0007 makes `cotdata` CFTC
+positioning only and moves every bar to `marketdata`, so the price reads behind the
+indexer, the signal rejection scores and the options max-pain snapshot now resolve
+against a second store. It fails the same way the first one does — by name, at the
+point of read — rather than serving an empty frame that reads as "no data for this
+instrument".
+
+Both are **synced**, not produced here. This box cannot produce bars at all:
+`norgatedata` drives a locally installed Norgate Data Updater and NDU is Windows-only.
+
 **Required for the emailed Signal Matrix report:**
 
 ```bash
@@ -215,8 +226,11 @@ They are cotmetrics runtime state, and PR #12 moved them out of this repo into
 `.local-state/cot-analyzer/`, so the deprecated script could not have shipped them anyway.
 **The server rebuilds the cache itself** — confirmed by the maintainer on 2026-08-04, not
 inferred from the push having been broken. `CotIndexer` writes per-instrument parquet under
-`COTMETRICS_CACHE` on first use and busts it on the upstream `cotdata.schema_version()` plus
-`METRICS_CACHE_VERSION`, so a store push is the only input it needs.
+`COTMETRICS_CACHE` on first use and busts it on BOTH upstream store versions
+(`cotdata.schema_version()` and `marketdata.schema_version()`) plus
+`METRICS_CACHE_VERSION`, so a store push is the only input it needs. Both are watched
+because the case that guard was written for — reconstructed volume being promoted — was
+a *price* schema bump, and prices now live in the second store.
 
 The first sync after a release is therefore slower than steady state, because the cache is
 cold. That is a latency cost on one request, not a missing payload.
@@ -226,8 +240,15 @@ The store push by hand, for reference, is:
 ```bash
 rsync -avz --no-o --no-g --progress --exclude='.DS_Store' \
       /path/to/cotdata_store/ USER@HOST:/root/cotdata_store/
+
+rsync -avz --no-o --no-g --progress --exclude='.DS_Store' \
+      /path/to/marketdata_store/ USER@HOST:/root/marketdata_store/
 ```
 
+Two stores, two pushes, and they must stay **separate roots**. Each package keeps its
+own `manifest.json` at its root and does a read-modify-write on it, so merging the two
+into one directory would have the producers dropping each other's entries.
+
 `data/` is a third gitignored directory, and most of it does **not** need to travel:
 
 - `xls_data` (441M) and `cot_data` (94M) are CFTC archives the ETL downloads from
@@ -406,6 +427,11 @@ slated for removal... Refrain from using this package or pin to Setuptools<81."*
 **Store errors, or every instrument empty.** `COTDATA_STORE` is unset, points somewhere
 wrong, or the store was never synced. It is read at import.
 
+**`MARKETDATA_STORE is not set`, or price reads fail while COT works.** The second store
+is missing. Positioning and prices come from different packages since ADR-0007, so one
+can be healthy while the other is absent — and this is the shape that failure takes.
+Set it and sync `marketdata_store` the same way as the first.
+
 **Charts render but prices are missing or stale.** Expected if the producer has not run.
 The server cannot fetch prices; see the top of this file.