From 413400b5d4f913eb265bf8c9e2a17567ad6c6068 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 17:00:33 +0000 Subject: [PATCH 1/3] Pre-register the offside-capitulation evaluation and its runnable check Defines offside (adverse >=1-sigma 4-week move against a trailing NPF extreme) and capitulation (top-decile one-week unwind of the crowded net) on the 8 Legacy markets of the NPF variance check, and freezes a two-sided event study - the washout and stampede folk claims disagree on the sign, so the sign is the question. Controls and the recentred block-bootstrap p inherit the forced-flow verdict's mean-reversion lesson from the start. This sandbox's network policy blocks cftc.gov and every probed price host, so execution is handed to a trusted-network session per the CLAUDE.md precedent; the script degrades to COT-only descriptives and reports a blocked run as blocked, never as a null. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01RMg9W6WCCNS69zx3cEaCKb --- .gitignore | 2 + ...6-08-23-offside-capitulation-evaluation.md | 158 +++++++ scripts/offside_capitulation_check.py | 431 ++++++++++++++++++ 3 files changed, 591 insertions(+) create mode 100644 docs/handoffs/2026-08-23-offside-capitulation-evaluation.md create mode 100644 scripts/offside_capitulation_check.py diff --git a/.gitignore b/.gitignore index a1c1102..cd3af20 100644 --- a/.gitignore +++ b/.gitignore @@ -27,3 +27,5 @@ dist/ # local scheduler backups (same machine-specific paths as /scheduler/) /scheduler.bak*/ /scheduler_backup*/ +# download cache of the offside-capitulation check +scripts/occ_cache/ diff --git a/docs/handoffs/2026-08-23-offside-capitulation-evaluation.md b/docs/handoffs/2026-08-23-offside-capitulation-evaluation.md new file mode 100644 index 0000000..52563a5 --- /dev/null +++ b/docs/handoffs/2026-08-23-offside-capitulation-evaluation.md @@ -0,0 +1,158 @@ +# Offside capitulation: a pre-registered first-look evaluation + +**Date:** 2026-08-23. **Status: OPEN — design frozen, execution handed off.** +The executable form of §4 is [`scripts/offside_capitulation_check.py`](../../scripts/offside_capitulation_check.py); +the script and this document were written together, before any look at the +statistic they define. §§1–5 are frozen: an executing session runs the script, +reports what comes back, and does not edit them. Results belong in a new dated +`docs/analysis/` file, per the doc lifecycle. + +**Ownership note, stated rather than hidden.** Per ADR-0007, cotdata is CFTC +positioning and a consumer question like this one belongs with the consumer +(`npf` is the natural final home). It is filed here because the task arrived on +a cotdata branch and the deliverable is self-contained: the script downloads +its own data exactly as `npf_check.py` did and touches nothing in `src/`. + +--- + +## 1. The question + +"Offside capitulation": a speculative crowd at a positioning extreme, with +price moving against it (**offside**), rapidly unwinds its net (**capitulates**). +What do returns do next? The folk readings disagree on the *sign*, and the +workspace's own vocabulary carries both: + +- **Claim A, "washout"**: capitulation ends the adverse move — the selling (or + covering) that was still to come has happened, so continuation stops or + reverts. This is the reading behind cotmetrics' `FLAG_BULL_CAPITULATION` + ("the institutional floor") and behind treating spec washouts as bottoms. +- **Claim B, "stampede"**: capitulation removes the last resistance and the + adverse move continues. This is the reading behind cotmetrics' + `CAPITULATION` ("liquidity vacuum... no bottom in sight") and + `COMMS_CAPITULATION` ("the Briese stampede"). + +Both cannot be right at the same horizon on the same event. That built-in +contradiction is what makes this worth measuring, and it is why the test is +two-sided. + +## 2. Vocabulary provenance + +- **Offside** is crowdmon's term: crowding = *lopsided, offside, trapped* + (crowdmon `amendments-2026-08-04 §D9`). There, "offside" is operationalized + as distance-to-forced-flip; here it is operationalized directly as an + adverse move against an extreme, because Legacy COT has no trigger machinery + and needs none for this question. +- **Capitulation** is the npf/cotmetrics term (signal family + `CAPITULATION` / `COMMS_CAPITULATION` / `FLAG_*_CAPITULATION`), where it is + defined on daily candles, OI kinetics and crowding z-scores. Here it is + reduced to the one ingredient Legacy COT carries: an unusually large + one-week unwind of the crowded net. + +## 3. What is already measured, and what is not + +Nothing below is re-tested here; it is the register this evaluation sits in. + +1. **The offside pool does capitulate when its level is hit — marginally.** + The forced-flow mechanism verdict + (`npf/docs/crowdmon/2026-08-06-forced-flow-mechanism-verdict.md`) returned + `supported` on its pre-registered criteria, then showed most of the effect + was positioning mean reversion (the placebo carried 52–74% of the + headline); the crossing-specific residual is ~1–2% of the pool's size, with + the Disaggregated side marginal. Its respecification + (`crowdmon/docs/handoffs/2026-08-06-forced-flow-respecification.md` §3) + adds an advisory crosstab: after a crossing the *agreeing* pool's week is + `long_liquidation`-dominated 3–5x more often than the contradicted pool's. + So "the offside crowd capitulates" has measured support as a *flow* + statement. **What returns do afterward was not the outcome variable of any + of it.** +2. **The capitulation-family booleans were never individually scored.** They + entered the ML feature stack, whose honest point-in-time re-evaluation + returned no edge (`npf/docs/npf/ml_lookahead.md`); no per-signal + pre-registered verdict on a capitulation event exists in the register this + session searched (`npf`, `crowdmon`, `cotdata`, `cotmetrics`). +3. **Mean reversion of positioning is the trap.** The FFM verdict's central + lesson: any contrast whose groups are defined by the sign or size of + `pool_net` gets a "result" from books drifting toward flat. §4's controls + exist because of that lesson. + +## 4. Frozen protocol + +Universe: the 8 Legacy futures-only markets of the NPF variance check (gold, +silver, WTI, nat gas, corn, EUR FX, 10Y note, E-mini S&P), 1990–present, each +market entering when its data does. Weekly non-commercial net; NPF = net/OI. +Report weeks not 6–8 days apart are never differenced (the FFM gap rule). + +Definitions (all trailing, per market; parameters frozen in the script header): + +- **Extreme**: NPF's percentile rank within its trailing 156-week window + (min 104) ≥ 0.80 (crowded-long) or ≤ 0.20 (crowded-short). +- **Offside**: extreme, and the trailing 4-week log return is ≥ 1.0 trailing + sigma (of 4-week returns, 156w window) *against* the crowd's side. +- **Capitulation week**: |ΔNPF| ≥ its own trailing 90th percentile (156w), + signed as an unwind of the previous week's crowd side. +- **Event**: capitulation at `t` with offside at `t−1` or `t−2`; 8-week + per-market-and-side cooldown. +- **Outcome**: forward sigma-scaled weekly log returns (trailing 52w sigma), + summed over h ∈ {1, 4, 13} weeks, **signed so that positive = the adverse + move continues** (claim B direction; negative = claim A). +- **Controls**, both inheriting the crowd side and its sign convention: + (a) offside without capitulation; (b) capitulation without offside. +- **Statistic**: pooled mean (median reported beside it), two-sided `p_null` + from a 13-week calendar-block bootstrap recentred on zero (the FFM p-value + correction, applied from the start rather than discovered again), 5000 + draws, seed 20260823. + +**Declared readings**: 2 sides x 3 horizons on the event group = 6, primary = +the two 4-week event readings; the same 6 on each control as context. Total +declared: 18. Anything else quoted from a run is an addition and must say so. + +**Pre-committed reading of the outcome, at the 4-week primaries:** + +- **Claim A supported**: both sides' event means negative, `p_null` < 0.05, + and each more negative than its offside-without-capitulation control. +- **Claim B supported**: the mirror image, positive. +- **Insufficient data**: pooled event n < 30 on a side — report counts, no + verdict on that side. Events are conjunctions of three rare conditions; + this outcome is likely and is a finding, not a failure. +- **Anything else**: "no separation" — the event carries no forward-return + information beyond its conditioning, which, given §3.1, would itself be + the coherent completion of the FFM story. + +This is a first look, not a strategy test: no trades, no `TradeLog`, no +claim that a reader of the Friday release could have acted (the FFM §3.2 +scope refusal is inherited; report-date alignment ignores the 3-day +publication lag deliberately). + +## 5. Execution constraints, measured + +This session's sandbox **cannot run the evaluation**: the proxy answers 403 to +CONNECT for `www.cftc.gov`, and `stooq.com`, `query1.finance.yahoo.com`, +`fred.stlouisfed.org` and `publicreporting.cftc.gov` are all unreachable +(probed 2026-08-23). This is the same block the NPF variance check hit, and +the same remedy is used: execution handed to a trusted-network session via a +one-shot trigger, per the CLAUDE.md precedent. + +Price caveats the executor must carry into the results file: + +- stooq continuous futures are spliced, not back-adjusted; roll carry + inflates adverse-move and forward-return magnitudes in high-carry markets + (nat gas above all). Acceptable for a first look; flagged, not fixed. +- If a market's price series fails to resolve, the script logs it and the + market drops from the priced panel; the log line belongs in the results. +- If no price source resolves at all, the script reports COT-only + capitulation descriptives and states the run is blocked. A blocked run is + reported as blocked, never as a null. + +## 6. Instructions to the executing session + +1. Check out this branch (`claude/offside-capitulation-evaluation-b55iv9`). +2. `pip install pandas numpy requests xlrd`, then + `python scripts/offside_capitulation_check.py` (optionally `--selftest` + first; it must pass before a live run is attempted). +3. Commit the full stdout as `docs/analysis/2026-08-23-offside-capitulation-run.txt` + and write `docs/analysis/2026-08-23-offside-capitulation-results.md`: + the tables, the §4 pre-committed reading applied literally, per-market + sign counts, price-source log, and any deviation from this document. +4. Push to the same branch. Do not edit §§1–5 of this file, the script's + frozen parameters, or anything in `src/`. If the run fails, commit the + failure output and say what blocked it. diff --git a/scripts/offside_capitulation_check.py b/scripts/offside_capitulation_check.py new file mode 100644 index 0000000..8d6e917 --- /dev/null +++ b/scripts/offside_capitulation_check.py @@ -0,0 +1,431 @@ +"""Offside-capitulation evaluation on CFTC Legacy COT data. + +Pre-registered by docs/handoffs/2026-08-23-offside-capitulation-evaluation.md. +Read that document before running or interpreting this; the definitions below +are frozen there and this script is their executable form. + +The question: when a crowded speculative book is OFFSIDE (price has moved +against a positioning extreme) and then CAPITULATES (an unusually large +one-week unwind of its net, in the adverse direction), what do subsequent +returns do? Two folk claims disagree on the sign: + + Claim A ("washout"): capitulation ends the adverse move; forward returns + stop continuing against the ex-crowd, or revert. + Claim B ("stampede"): capitulation removes the last resistance; the + adverse move continues. + +Everything is measured per market, backward-looking, with trailing windows. +No claim is made that a reader of the Friday release could have acted; this +is an event study on report-date-aligned data (the FFM test's §3.2 scope +refusal is inherited). + +Data: CFTC Legacy futures-only annual zips (same URLs as +src/cotdata/providers/cftc.py) and daily continuous-futures closes from +stooq.com, sampled at each report date. If no price source is reachable the +script degrades to COT-only descriptives and says so; it never fabricates. + +Usage: + python offside_capitulation_check.py # full run (needs network) + python offside_capitulation_check.py --selftest # synthetic smoke test + python offside_capitulation_check.py --cot-only # skip prices deliberately + +Deps: pandas numpy requests xlrd +""" + +import argparse +import io +import sys +import zipfile +import warnings +from pathlib import Path + +import numpy as np +import pandas as pd + +warnings.filterwarnings("ignore") + +CACHE = Path(__file__).resolve().parent / "occ_cache" + +# Legacy futures-only market codes -> (name, stooq symbol candidates, tried in order) +MARKETS = { + "088691": ("Gold", ["gc.f"]), + "084691": ("Silver", ["si.f"]), + "067651": ("Crude WTI", ["cl.f"]), + "023651": ("Nat gas", ["ng.f"]), + "002602": ("Corn", ["c.f", "zc.f"]), + "099741": ("EUR FX", ["6e.f", "eurusd"]), + "043602": ("10Y Note", ["zn.f", "ty.f"]), + "13874A": ("E-mini S&P", ["es.f", "sp.f", "^spx"]), +} + +COLS = ["Market_and_Exchange_Names", "Report_Date_as_MM_DD_YYYY", + "CFTC_Contract_Market_Code", "Open_Interest_All", + "NonComm_Positions_Long_All", "NonComm_Positions_Short_All"] + +# ---- frozen parameters (see handoff §4; do not tune) ----------------------- +RANK_WIN = 156 # trailing weeks for NPF percentile / thresholds +RANK_MIN = 104 # minimum observations before ranks are valid +EXT_HI, EXT_LO = 0.80, 0.20 # crowded-long / crowded-short percentile gates +ADV_LOOKBACK = 4 # weeks of adverse move defining "offside" +ADV_SIGMA = 1.0 # adverse move must be >= this many trailing sigmas +CAP_PCTL = 0.90 # |dNPF| must clear this trailing percentile +OFFSIDE_MEMORY = 2 # capitulation must follow offside within this many weeks +COOLDOWN = 8 # weeks between events per market+side +HORIZONS = (1, 4, 13) # forward horizons, weeks +VOL_WIN = 52 # trailing weeks for weekly-return sigma scaling +BLOCK_WEEKS = 13 # calendar block size for the bootstrap +N_BOOT = 5000 +SEED = 20260823 + +START_YEAR, END_YEAR = 1990, 2026 + + +# ---------------------------------------------------------------- data layer +def cot_year_frame(year, requests): + if year < 2004: + url = f"https://www.cftc.gov/files/dea/history/deafut_xls_{year}.zip" + else: + url = f"https://www.cftc.gov/files/dea/history/dea_fut_xls_{year}.zip" + zp = CACHE / url.rsplit("/", 1)[1] + if not zp.exists(): + r = requests.get(url, timeout=120) + r.raise_for_status() + zp.write_bytes(r.content) + with zipfile.ZipFile(zp) as zf: + data = zf.open(zf.namelist()[0]).read() + df = pd.read_excel(io.BytesIO(data), usecols=lambda c: c in COLS) + df["CFTC_Contract_Market_Code"] = ( + df["CFTC_Contract_Market_Code"].astype(str).str.strip()) + df["Report_Date_as_MM_DD_YYYY"] = pd.to_datetime(df["Report_Date_as_MM_DD_YYYY"]) + return df + + +def load_cot(requests): + frames, failures = [], [] + for y in range(START_YEAR, END_YEAR + 1): + try: + frames.append(cot_year_frame(y, requests)) + except Exception as e: # noqa: BLE001 - report, don't die + failures.append((y, repr(e))) + if failures: + print(f"COT downloads failed for {len(failures)} year(s): " + f"{[y for y, _ in failures]}", file=sys.stderr) + print(f" first error: {failures[0][1]}", file=sys.stderr) + if not frames: + return None + return pd.concat(frames, ignore_index=True) + + +def load_prices_stooq(requests): + """Daily closes per market from stooq; returns dict code -> Series, and a log.""" + out, log = {}, [] + for code, (name, syms) in MARKETS.items(): + got = None + for sym in syms: + url = f"https://stooq.com/q/d/l/?s={sym}&i=d" + try: + r = requests.get(url, timeout=60) + r.raise_for_status() + px = pd.read_csv(io.StringIO(r.text)) + if "Close" not in px.columns or len(px) < 500: + raise ValueError(f"unusable payload ({len(px)} rows)") + px["Date"] = pd.to_datetime(px["Date"]) + got = px.set_index("Date")["Close"].astype(float).sort_index() + log.append(f" {name}: stooq {sym}, {len(got)} daily bars " + f"{got.index[0].date()} .. {got.index[-1].date()}") + break + except Exception as e: # noqa: BLE001 + log.append(f" {name}: stooq {sym} FAILED ({e!r})") + out[code] = got + return out, log + + +def load_prices_local(): + """Optional local fallback: scripts/occ_prices_daily.csv with columns + date,market_code,close - lets an executor supply prices from any store.""" + p = Path(__file__).resolve().parent / "occ_prices_daily.csv" + if not p.exists(): + return None + df = pd.read_csv(p, parse_dates=["date"]) + return {code: g.set_index("date")["close"].astype(float).sort_index() + for code, g in df.groupby("market_code")} + + +# ------------------------------------------------------------- event builder +def rolling_pct_rank(s, win, minp): + """Percentile rank of the last value within its trailing window.""" + def _rank(a): + return (a[:-1] <= a[-1]).mean() + return s.rolling(win, min_periods=minp).apply(_rank, raw=True) + + +def build_market_panel(cot_sub, price): + """Weekly panel for one market: NPF states, event flags, forward outcomes.""" + sub = (cot_sub.sort_values("Report_Date_as_MM_DD_YYYY") + .drop_duplicates("Report_Date_as_MM_DD_YYYY") + .set_index("Report_Date_as_MM_DD_YYYY")) + net = (sub["NonComm_Positions_Long_All"] + - sub["NonComm_Positions_Short_All"]).astype(float) + oi = sub["Open_Interest_All"].astype(float).replace(0, np.nan) + npf = (net / oi).dropna() + idx = npf.index + + # drop report weeks not ~7 days apart (the FFM rule: never difference a gap) + gap = idx.to_series().diff().dt.days + ok_gap = (gap >= 6) & (gap <= 8) + + d = pd.DataFrame(index=idx) + d["npf"] = npf + d["dnpf"] = npf.diff().where(ok_gap) + d["side"] = np.sign(npf) + + d["rank"] = rolling_pct_rank(npf, RANK_WIN, RANK_MIN) + d["ext_long"] = d["rank"] >= EXT_HI + d["ext_short"] = d["rank"] <= EXT_LO + + cap_thresh = d["dnpf"].abs().rolling(RANK_WIN, min_periods=RANK_MIN)\ + .quantile(CAP_PCTL) + big = d["dnpf"].abs() >= cap_thresh + prev_side = d["side"].shift(1) + d["cap_long"] = big & (d["dnpf"] < 0) & (prev_side > 0) + d["cap_short"] = big & (d["dnpf"] > 0) & (prev_side < 0) + + if price is not None: + # sample the last close on or before each report date + p = price.reindex(price.index.union(idx)).ffill().reindex(idx) + lr = np.log(p).diff() + d["p"] = p + d["r4"] = np.log(p / p.shift(ADV_LOOKBACK)) + sig4 = d["r4"].rolling(RANK_WIN, min_periods=RANK_MIN).std() + d["off_long"] = d["ext_long"] & (d["r4"] <= -ADV_SIGMA * sig4) + d["off_short"] = d["ext_short"] & (d["r4"] >= ADV_SIGMA * sig4) + sig1 = lr.rolling(VOL_WIN, min_periods=40).std() + d["u"] = lr / sig1 # sigma-scaled weekly log return + for h in HORIZONS: + d[f"fu{h}"] = d["u"].shift(-1).rolling(h).sum().shift(-(h - 1)) + return d + + +def flag_events(d, side): + """Offside-capitulation events for one side, with memory + cooldown.""" + off, cap = (d["off_long"], d["cap_long"]) if side == "long" \ + else (d["off_short"], d["cap_short"]) + off_recent = off.shift(1).fillna(False) + for k in range(2, OFFSIDE_MEMORY + 1): + off_recent |= off.shift(k).fillna(False) + raw = cap & off_recent + # cooldown + out, last = [], None + for t, v in raw.items(): + if v and (last is None or (t - last).days > COOLDOWN * 7): + out.append(t) + last = t + ev = pd.Series(False, index=d.index) + ev.loc[out] = True + return ev, off_recent, cap + + +# ---------------------------------------------------------------- statistics +def block_key(ts): + return (ts.year, (ts.dayofyear - 1) // (BLOCK_WEEKS * 7)) + + +def boot_p(rows, col, rng): + """Two-sided p for mean(col)=0 via calendar-block bootstrap, recentred + on zero (the FFM p_null correction; a literal fraction-below test is + uninformative on a bootstrap centred at the observed statistic).""" + if len(rows) < 5: + return (rows[col].mean() if len(rows) else np.nan), np.nan + obs = rows[col].mean() + grouped = rows.groupby(rows["t"].map(block_key))[col] + sums = grouped.sum().to_numpy() + counts = grouped.count().to_numpy() + nb = len(sums) + pick = rng.integers(0, nb, size=(N_BOOT, nb)) + means = sums[pick].sum(axis=1) / np.maximum(counts[pick].sum(axis=1), 1) + means = means - means.mean() # recentre on zero + p = (np.abs(means) >= abs(obs)).mean() + return obs, p + + +def summarize(events_df, label, rng): + print(f"\n### {label} (n = {len(events_df)})") + if len(events_df) == 0: + print(" no events") + return + hdr = f" {'horizon':<9} {'mean u':>8} {'median u':>9} {'p_null':>8} {'n':>5}" + print(hdr + "\n " + "-" * (len(hdr) - 2)) + for h in HORIZONS: + col = f"fu{h}" + rows = events_df.dropna(subset=[col]) + obs, p = boot_p(rows, col, rng) + med = rows[col].median() if len(rows) else np.nan + print(f" {h:>2}w {obs:8.3f} {med:9.3f} {p:8.4f} {len(rows):5d}") + + +# ------------------------------------------------------------------ pipeline +def run(cot, prices, price_log): + rng = np.random.default_rng(SEED) + have_prices = prices is not None and any(v is not None for v in prices.values()) + + print("=" * 78) + print("OFFSIDE-CAPITULATION EVALUATION - run report") + print("=" * 78) + if price_log: + print("\nPrice sources:") + for line in price_log: + print(line) + if not have_prices: + print("\nNO PRICE SOURCE RESOLVED. Degraded COT-only mode: capitulation") + print("frequency/size only; offside conditioning and outcomes are") + print("impossible without prices, and are NOT reported. This is a") + print("blocked run, not a null result.") + + pooled = {("long", g): [] for g in ("event", "off_nocap", "cap_nooff")} + pooled.update({("short", g): [] for g in ("event", "off_nocap", "cap_nooff")}) + per_market_sign = [] + + for code, (name, _) in MARKETS.items(): + cot_sub = cot[cot["CFTC_Contract_Market_Code"] == code] + if cot_sub.empty: + print(f"\n{name}: no COT rows, skipped") + continue + price = prices.get(code) if have_prices else None + d = build_market_panel(cot_sub, price) + ncap = int(d["cap_long"].sum() + d["cap_short"].sum()) + print(f"\n{name}: {len(d)} report weeks " + f"({d.index[0].date()} .. {d.index[-1].date()}), " + f"{ncap} capitulation weeks " + f"({ncap / max(len(d), 1) * 100:.1f}%)") + if price is None: + continue + + for side in ("long", "short"): + # sign convention: positive forward u = CONTINUATION of the + # adverse move (against the ex-crowd). Crowd long -> adverse is + # down -> multiply raw forward u by -1; crowd short -> +1. + sgn = -1.0 if side == "long" else 1.0 + ev, off_recent, cap = flag_events(d, side) + + def collect(mask, group, side=side, sgn=sgn): + rows = d.loc[mask, [f"fu{h}" for h in HORIZONS]].copy() * sgn + rows["t"] = rows.index + rows["market"] = name + pooled[(side, group)].append(rows) + + collect(ev, "event") + collect(off_recent & ~cap, "off_nocap") + collect(cap & ~off_recent, "cap_nooff") + + evrows = d.loc[ev] + if len(evrows): + m4 = (evrows["fu4"] * sgn).mean() + per_market_sign.append((name, side, len(evrows), m4)) + + if not have_prices: + return + + print("\n" + "=" * 78) + print("POOLED RESULTS - forward sigma-scaled returns, signed so that") + print("POSITIVE = adverse move CONTINUES (claim B), NEGATIVE = it reverts") + print("(claim A). p_null: two-sided, 13-week calendar-block bootstrap,") + print("recentred on zero.") + print("=" * 78) + + for side in ("long", "short"): + crowd = "crowded-LONG specs" if side == "long" else "crowded-SHORT specs" + for group, label in ( + ("event", f"OFFSIDE CAPITULATION, {crowd} [primary at 4w]"), + ("off_nocap", f"control: offside WITHOUT capitulation, {crowd}"), + ("cap_nooff", f"control: capitulation WITHOUT offside, {crowd}")): + frames = pooled[(side, group)] + df = pd.concat(frames) if frames else pd.DataFrame( + columns=[f"fu{h}" for h in HORIZONS] + ["t", "market"]) + summarize(df, label, rng) + + print("\nPer-market event mean at 4w (sign robustness, FFM §5.6 style):") + print(f" {'market':<12} {'side':<6} {'n':>4} {'mean u(4w)':>11}") + for name, side, n, m4 in per_market_sign: + print(f" {name:<12} {side:<6} {n:>4} {m4:11.3f}") + neg = sum(1 for *_, m in per_market_sign if m < 0) + print(f" sign count: {neg} of {len(per_market_sign)} market-sides negative " + f"(claim A direction)") + + print("\nDeclared readings: 2 sides x 3 horizons on the event group = 6") + print("(primary = the two 4w event readings), plus the same 6 on each of") + print("two controls as contrasts. Anything else quoted from this run") + print("must be counted as an addition.") + + +# ------------------------------------------------------------------ selftest +def selftest(): + """Synthetic smoke test: exercises the full pipeline offline.""" + global MARKETS + rng = np.random.default_rng(7) + idx = pd.date_range("2000-01-04", periods=900, freq="7D") + cot_rows, prices = [], {} + codes = list(MARKETS)[:3] + for code in codes: + drift = np.cumsum(rng.normal(0, 0.02, len(idx))) + px_daily = pd.Series( + 100 * np.exp(np.interp( + np.arange(len(idx) * 7), np.arange(len(idx)) * 7, drift) + + rng.normal(0, 0.005, len(idx) * 7)), + index=pd.date_range(idx[0] - pd.Timedelta(days=3), + periods=len(idx) * 7, freq="D")) + prices[code] = px_daily + npf = np.tanh(np.convolve(rng.normal(0, 1, len(idx)), + np.ones(10) / 10, mode="same")) + # inject occasional large unwinds so events exist + jumps = rng.choice(len(idx) - 30, size=25, replace=False) + 20 + npf = pd.Series(npf, index=idx) + for j in jumps: + npf.iloc[j] = npf.iloc[j - 1] * 0.2 + oi = pd.Series(100000.0, index=idx) + net = npf * oi + for t in idx: + cot_rows.append({ + "Market_and_Exchange_Names": code, + "Report_Date_as_MM_DD_YYYY": t, + "CFTC_Contract_Market_Code": code, + "Open_Interest_All": oi[t], + "NonComm_Positions_Long_All": max(net[t], 0) + 10000, + "NonComm_Positions_Short_All": max(-net[t], 0) + 10000, + }) + cot = pd.DataFrame(cot_rows) + MARKETS = {c: (f"SYN-{c}", []) for c in codes} + run(cot, prices, [" synthetic data, selftest mode"]) + print("\nSELFTEST COMPLETE - pipeline ran end to end on synthetic data.") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--selftest", action="store_true") + ap.add_argument("--cot-only", action="store_true") + args = ap.parse_args() + + if args.selftest: + selftest() + return + + import requests + CACHE.mkdir(exist_ok=True) + cot = load_cot(requests) + if cot is None: + print("FATAL: no COT data reachable. This environment cannot run the " + "evaluation; report the block rather than fabricating results.", + file=sys.stderr) + sys.exit(2) + + prices, price_log = (None, []) + if not args.cot_only: + prices = load_prices_local() + if prices is not None: + price_log = [" local occ_prices_daily.csv"] + else: + prices, price_log = load_prices_stooq(requests) + run(cot, prices, price_log) + + +if __name__ == "__main__": + main() From eda4a3cb408cf838815dac6df8b11e8940cafcab Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 21:41:03 +0000 Subject: [PATCH 2/3] Sort the check script's imports for ruff Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01RMg9W6WCCNS69zx3cEaCKb --- scripts/offside_capitulation_check.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/offside_capitulation_check.py b/scripts/offside_capitulation_check.py index 8d6e917..77ec439 100644 --- a/scripts/offside_capitulation_check.py +++ b/scripts/offside_capitulation_check.py @@ -35,8 +35,8 @@ import argparse import io import sys -import zipfile import warnings +import zipfile from pathlib import Path import numpy as np From dc8f21f0eedb52010c628c23e5430c1efd714d0b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 22:17:11 +0000 Subject: [PATCH 3/3] Read the stores first, download only where they are absent The author's point: the data already lives in the stores this repo produces. COT now comes from COTDATA_STORE via cotdata.get_cot and daily closes from MARKETDATA_STORE via marketdata.get_bars (propadj, then backadj, logged), with the cftc.gov and stooq downloads demoted to fallback for store-less environments. Verified against the npf fixture store: gold loads 1,829 weeks through the real read path. Section 4 of the handoff is untouched; the amendment is recorded in section 5, dated and pre-execution. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01RMg9W6WCCNS69zx3cEaCKb --- ...6-08-23-offside-capitulation-evaluation.md | 41 ++++-- scripts/offside_capitulation_check.py | 134 ++++++++++++++---- 2 files changed, 137 insertions(+), 38 deletions(-) diff --git a/docs/handoffs/2026-08-23-offside-capitulation-evaluation.md b/docs/handoffs/2026-08-23-offside-capitulation-evaluation.md index 52563a5..e1c845b 100644 --- a/docs/handoffs/2026-08-23-offside-capitulation-evaluation.md +++ b/docs/handoffs/2026-08-23-offside-capitulation-evaluation.md @@ -125,14 +125,30 @@ publication lag deliberately). ## 5. Execution constraints, measured -This session's sandbox **cannot run the evaluation**: the proxy answers 403 to -CONNECT for `www.cftc.gov`, and `stooq.com`, `query1.finance.yahoo.com`, -`fred.stlouisfed.org` and `publicreporting.cftc.gov` are all unreachable -(probed 2026-08-23). This is the same block the NPF variance check hit, and -the same remedy is used: execution handed to a trusted-network session via a -one-shot trigger, per the CLAUDE.md precedent. - -Price caveats the executor must carry into the results file: +> **Amended 2026-08-23, pre-execution, at the author's direction, before any +> live number existed.** The first draft sourced data by download only. The +> author pointed out the obvious: the data already lives in the stores this +> repo produces. The script is now **store-first** — COT from +> `$COTDATA_STORE` via `cotdata.get_cot` (code stitching included), daily +> closes from `$MARKETDATA_STORE` via `marketdata.get_bars` (propadj, then +> backadj, logged) — and downloads only where the stores are absent. §4 is +> untouched; this changes where bytes come from, not what is computed. On a +> store machine (the Mac, the Windows producer) the run needs no network at +> all, and store prices are Norgate back-adjusted series, which retires the +> stooq splicing caveat below for those runs. + +The git repo carries no data (the store is external and gitignored), so a +cloud session holds only code. This session's sandbox additionally **cannot +download**: the proxy answers 403 to CONNECT for `www.cftc.gov`, and +`stooq.com`, `query1.finance.yahoo.com`, `fred.stlouisfed.org` and +`publicreporting.cftc.gov` are all unreachable (probed 2026-08-23). That is +the same block the NPF variance check hit, and the same remedy applies where +no store is mounted: execution handed to a trusted-network session via a +one-shot trigger, per the CLAUDE.md precedent. **A machine with the stores is +the better executor.** + +Price caveats the executor must carry into the results file (download-path +runs only; store-path runs replace them with the Norgate series' own terms): - stooq continuous futures are spliced, not back-adjusted; roll carry inflates adverse-move and forward-return magnitudes in high-carry markets @@ -146,9 +162,12 @@ Price caveats the executor must carry into the results file: ## 6. Instructions to the executing session 1. Check out this branch (`claude/offside-capitulation-evaluation-b55iv9`). -2. `pip install pandas numpy requests xlrd`, then - `python scripts/offside_capitulation_check.py` (optionally `--selftest` - first; it must pass before a live run is attempted). +2. `pip install pandas numpy requests xlrd` (plus `pyyaml python-dateutil` + and `crucible-marketdata` for the store path), then run + `python scripts/offside_capitulation_check.py` with `COTDATA_STORE` and + `MARKETDATA_STORE` exported if this machine has the stores (preferred — + no network needed); without them the script downloads. Run `--selftest` + first; it must pass before a live run is attempted. 3. Commit the full stdout as `docs/analysis/2026-08-23-offside-capitulation-run.txt` and write `docs/analysis/2026-08-23-offside-capitulation-results.md`: the tables, the §4 pre-committed reading applied literally, per-market diff --git a/scripts/offside_capitulation_check.py b/scripts/offside_capitulation_check.py index 77ec439..142aa54 100644 --- a/scripts/offside_capitulation_check.py +++ b/scripts/offside_capitulation_check.py @@ -19,10 +19,14 @@ is an event study on report-date-aligned data (the FFM test's §3.2 scope refusal is inherited). -Data: CFTC Legacy futures-only annual zips (same URLs as -src/cotdata/providers/cftc.py) and daily continuous-futures closes from -stooq.com, sampled at each report date. If no price source is reachable the -script degrades to COT-only descriptives and says so; it never fabricates. +Data, store first: on a machine with the real stores, COT comes from +$COTDATA_STORE via the repo's own `cotdata.get_cot` (code stitching included) +and daily closes from $MARKETDATA_STORE via `marketdata.get_bars` (propadj, +falling back to backadj) — no network needed. Where the stores are absent +(cloud sessions), it falls back to downloading the CFTC Legacy annual zips +(same URLs as src/cotdata/providers/cftc.py) and stooq.com continuous +closes. If no price source resolves the script degrades to COT-only +descriptives and says so; it never fabricates. Usage: python offside_capitulation_check.py # full run (needs network) @@ -34,6 +38,7 @@ import argparse import io +import os import sys import warnings import zipfile @@ -46,16 +51,17 @@ CACHE = Path(__file__).resolve().parent / "occ_cache" -# Legacy futures-only market codes -> (name, stooq symbol candidates, tried in order) +# Legacy futures-only market codes -> +# (name, stooq symbol candidates tried in order, pipeline symbol per registry.yaml) MARKETS = { - "088691": ("Gold", ["gc.f"]), - "084691": ("Silver", ["si.f"]), - "067651": ("Crude WTI", ["cl.f"]), - "023651": ("Nat gas", ["ng.f"]), - "002602": ("Corn", ["c.f", "zc.f"]), - "099741": ("EUR FX", ["6e.f", "eurusd"]), - "043602": ("10Y Note", ["zn.f", "ty.f"]), - "13874A": ("E-mini S&P", ["es.f", "sp.f", "^spx"]), + "088691": ("Gold", ["gc.f"], "GC"), + "084691": ("Silver", ["si.f"], "SI"), + "067651": ("Crude WTI", ["cl.f"], "CL"), + "023651": ("Nat gas", ["ng.f"], "NG"), + "002602": ("Corn", ["c.f", "zc.f"], "ZC"), + "099741": ("EUR FX", ["6e.f", "eurusd"], "6E"), + "043602": ("10Y Note", ["zn.f", "ty.f"], "ZN"), + "13874A": ("E-mini S&P", ["es.f", "sp.f", "^spx"], "ES"), } COLS = ["Market_and_Exchange_Names", "Report_Date_as_MM_DD_YYYY", @@ -81,6 +87,70 @@ # ---------------------------------------------------------------- data layer +def load_cot_store(): + """COT from $COTDATA_STORE via the repo's own consumer API. Returns the + same long frame as the download path, or None if the store is not set.""" + if not os.environ.get("COTDATA_STORE", "").strip(): + return None, [] + sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + from cotdata.cot import get_cot + frames, log = [], [] + for code, (name, _, sym) in MARKETS.items(): + try: + df = get_cot(sym, report="legacy") + if df.empty: + raise ValueError("empty frame") + except Exception as e: # noqa: BLE001 + log.append(f" {name}: store cot {sym} FAILED ({e!r})") + continue + sub = df.reset_index() + sub = sub.rename(columns={sub.columns[0]: "Report_Date_as_MM_DD_YYYY"}) + sub["Report_Date_as_MM_DD_YYYY"] = pd.to_datetime( + sub["Report_Date_as_MM_DD_YYYY"]) + sub = sub[sub["Report_Date_as_MM_DD_YYYY"] >= f"{START_YEAR}-01-01"] + sub["CFTC_Contract_Market_Code"] = code + sub["Market_and_Exchange_Names"] = name + frames.append(sub[COLS]) + log.append(f" {name}: store cot {sym}, {len(sub)} weeks " + f"{sub['Report_Date_as_MM_DD_YYYY'].min().date()} .. " + f"{sub['Report_Date_as_MM_DD_YYYY'].max().date()}") + if not frames: + return None, log + return pd.concat(frames, ignore_index=True), log + + +def load_prices_marketdata(): + """Daily closes from $MARKETDATA_STORE via marketdata.get_bars. + propadj first (correct for log returns), backadj as a logged fallback.""" + if not os.environ.get("MARKETDATA_STORE", "").strip(): + return None, [] + try: + import marketdata + except ImportError: + return None, [" MARKETDATA_STORE set but crucible-marketdata not " + "installed; falling through"] + out, log = {}, [] + for code, (name, _, sym) in MARKETS.items(): + got = None + for adj in ("propadj", "backadj"): + try: + bars = marketdata.get_bars(sym, adj) + close_col = next(c for c in ("Close", "close") if c in bars.columns) + s = bars[close_col].astype(float).sort_index().dropna() + s = s[s > 0] # log returns need positive prices + if len(s) < 500: + raise ValueError(f"unusable series ({len(s)} rows)") + s.index = pd.to_datetime(s.index) + got = s + log.append(f" {name}: marketdata {sym} {adj}, {len(s)} bars " + f"{s.index[0].date()} .. {s.index[-1].date()}") + break + except Exception as e: # noqa: BLE001 + log.append(f" {name}: marketdata {sym} {adj} FAILED ({e!r})") + out[code] = got + return out, log + + def cot_year_frame(year, requests): if year < 2004: url = f"https://www.cftc.gov/files/dea/history/deafut_xls_{year}.zip" @@ -119,7 +189,7 @@ def load_cot(requests): def load_prices_stooq(requests): """Daily closes per market from stooq; returns dict code -> Series, and a log.""" out, log = {}, [] - for code, (name, syms) in MARKETS.items(): + for code, (name, syms, _) in MARKETS.items(): got = None for sym in syms: url = f"https://stooq.com/q/d/l/?s={sym}&i=d" @@ -285,7 +355,7 @@ def run(cot, prices, price_log): pooled.update({("short", g): [] for g in ("event", "off_nocap", "cap_nooff")}) per_market_sign = [] - for code, (name, _) in MARKETS.items(): + for code, (name, *_rest) in MARKETS.items(): cot_sub = cot[cot["CFTC_Contract_Market_Code"] == code] if cot_sub.empty: print(f"\n{name}: no COT rows, skipped") @@ -393,7 +463,7 @@ def selftest(): "NonComm_Positions_Short_All": max(-net[t], 0) + 10000, }) cot = pd.DataFrame(cot_rows) - MARKETS = {c: (f"SYN-{c}", []) for c in codes} + MARKETS = {c: (f"SYN-{c}", [], c) for c in codes} run(cot, prices, [" synthetic data, selftest mode"]) print("\nSELFTEST COMPLETE - pipeline ran end to end on synthetic data.") @@ -408,22 +478,32 @@ def main(): selftest() return - import requests - CACHE.mkdir(exist_ok=True) - cot = load_cot(requests) + cot, cot_log = load_cot_store() + if cot is None: + import requests + CACHE.mkdir(exist_ok=True) + cot = load_cot(requests) + cot_log = cot_log + [" COT via cftc.gov download (no COTDATA_STORE)"] if cot is None: - print("FATAL: no COT data reachable. This environment cannot run the " - "evaluation; report the block rather than fabricating results.", - file=sys.stderr) + print("FATAL: no COT source. COTDATA_STORE is unset (or unreadable) " + "and cftc.gov is unreachable; report the block rather than " + "fabricating results.", file=sys.stderr) sys.exit(2) + print("COT source:") + for line in cot_log: + print(line) prices, price_log = (None, []) if not args.cot_only: - prices = load_prices_local() - if prices is not None: - price_log = [" local occ_prices_daily.csv"] - else: - prices, price_log = load_prices_stooq(requests) + prices, price_log = load_prices_marketdata() + if prices is None: + prices = load_prices_local() + price_log = price_log + ( + [" local occ_prices_daily.csv"] if prices is not None else []) + if prices is None: + import requests + prices, stooq_log = load_prices_stooq(requests) + price_log = price_log + stooq_log run(cot, prices, price_log)