Repository navigation
Claude/phase 8.4 cold start - #162
Merged
Merged
Conversation
…ion reset Glenn (2026-10-05): not a cold-start test but a production reset. - D-01 freeze under tag archive/pre-reset-2026-10, then everything generated leaves git (data incl. raw, outputs, notebook outputs); data/ and outputs/ gitignored wholesale. - D-02 fail-loud build from empty (absorbs DECISIONS D-08) + provenance manifest. - D-03 registry true-zero epoch (Glenn's choice), ADR-0005 with the caveat and a disclosure line on every post-reset DSR. - D-04 CI on fixtures; real-data tests marked and local-only; record tests retired to the archive tag, each listed. - D-05/D-06 MVP-1 no-regime page from a fresh clone, run twice identical, N4 sign-off. - D-07 learnings + Phase 9 restructured into a 9.x module-by-module rebuild in the target repo (E-11 at M3, L2 recipe at M6, CR-02 at M5). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…nto 08.5 Glenn (2026-10-05): archive datasets, outputs and models so the platform can cold-start/reset at any time, and git-version-control tagged models. - 08.4 (re-scoped): state tooling — archive/list/reset/restore/promote. Local archives (tar.gz + sha256 manifest: code commit, config hash, registry count, date range); promote commits one and tags model/<name>; reset always auto-archives and never touches the registry; no pickles in any archive (restore refits and verifies, P27); live book state never promoted (G-11). Plus the fail-loud empty build (D-08) and a no_regime page without regime models. Budget 0. - 08.5 (inserted): the production clean-slate reset decided earlier today, now done with the tool (promote pre-reset-2026-10 → tag model/pre-reset-2026-10), registry true-zero epoch (ADR-0005), tests, learnings, Phase 9.x order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…ly, D-T9 fail_loud switch, D-T10 full data/outputs scope) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Glenn 2026-10-05: human-readable, human-editable, human-testable by one person with Python scripts and notebooks; the simplest honest design wins; rigor only for decision-bearing numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Glenn 2026-10-05 (P-07): two plans; one-file state tool; plain manifest and tar; no pickles in any archive; restore without refit/1e-9 verification; two build config keys; no extra provenance file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
… CONCERNS) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Delete code the weekly product never runs: - platform/parked/ (classifier #2, joint driver, stability suite) - parked-only helpers: allocation/joint_tilt.py, evaluation/dependence.py (orphan), the features/ package (classifier #2's substrate), labeling_2 config - research scripts: run_joint_lift, run_subsample_stability, terminal_month_diagnostic, joint_lift_diagnostics, diagnose_s1_truncation - their tests, incl. the parked-boundary and pooling-consumers audits Remaining references repointed (weekly cold-start rule now checks driver + weekly; two invariant-ratio tests removed with their function). MODULE-MAP and DECISIONS updated (G-13; G-08 marked changed). Legacy-import ratchet stays 31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…tive platform commands Deleted (no tests, no live doc references): diagnose_yahoo_tls.py, diagnose_yfinance.py, diagnose_cpi_handoff.py, run_policy_trials.py, smoke_step5.sh, egress_test.sh. The macrotrends diagnostics stay for Mac fetch failures. README: new 'Platform (active)' section; fixed install script name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
The 1,991-line guide was mostly the frozen legacy quarterly pipeline. It is archived verbatim at docs/archive/LEGACY-CLAUDE.md; the new CLAUDE.md (~110 lines) covers what this is, KISS, where decisions live, how to run, honesty rules, hard rules and conventions. Retired tests/unit/test_docs_recorded_counts.py (forced the test count to be restated at four doc sites on every suite change); README no longer hard-codes the count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
….K-11) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…en; roadmap goal matches K-1..K-11 Plan 02 waits for Plan 01 (wave 2) so D-08 is marked built after it lands and the full suite runs on both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…uperseded by K-9/K-5,K-6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…ript exits 1 - build.fail_loud / build.allow_missing_sources in platform_settings.yaml - BuildFailed, expected_source_columns, build_problems gate build_monthly_spine before any write - build_platform_data.py returns 1 on BuildFailed - _pit_cfg turns the gate off (dated note); new test_platform_build_fail_loud.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…ily fetch fails the build - merge_preserving / CheckpointManager.save take replace_columns (default () is unchanged) - build_monthly_spine saves monthly_raw with every splice research_name replaced (221defc) - build_platform_data.py: under build.fail_loud a raising or empty DAAA/DBAA fetch exits 1 after the other checks; the closing text names the weekly page, not the budgeted backtest - A1 guard test flips to test_a_failed_fred_daily_fetch_fails_the_build (dated note) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
- weekly._model_free_inputs: with no nowcaster in no_regime mode the page builds from monthly_raw and monthly_features alone and persists the executed book (regime_tilt still raises) - _NO_MODEL_SENTENCE via assemble_weekly_report(suspended_sentence=...); the pinned sentence is unchanged - scoreboard line reads 'Scoreboard: not yet measured (reason)' (pinned test updated, dated) - new test_platform_weekly_model_free.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…ts as not delivered Review finding: an all-NaN source column passed the gate, and an all-NaN yield column crashed the splice with a bare IndexError before the gate ran. The gate now checks sources right after fetching (missing_sources), and splice fallbacks after the splice (fallback_splices). New test covers a silently empty fred_monthly column. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…ough one CLI
- state.py: archive NAME writes archives/NAME/{state.tar.gz,manifest.json} (no pickles, sha256 per file)
- restore NAME needs empty trees, extracts with filter="data" into staging, checks every sha256 before moving
- checkpoints.py: PLATFORM_ROOT_DIR / PLATFORM_DATA_DIR / PLATFORM_OUTPUT_DIR on the existing import line
- tests: CLI round trip, bad/existing names, non-empty tree, tampered hash, traversal, no data_filter
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…promoted ones - reset NAME? --yes archives first, then empties data/ and outputs/ (keeps .gitkeep), names deleted pickles - promote NAME re-packs without G-11 live state into archives/promoted/NAME, commits and tags model/NAME, never pushes - .gitignore: archives/* and !archives/promoted/; pre-commit large-file hook skips archives/promoted/ - tests: reset/list in tmp dirs, promote in a throwaway git repo (dirty, oversize, failed commit refusals) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
- README: archive / reset / restore / promote commands, the clean-slate order, step 2 is regime_tilt only, the build fails loud unless a column is listed under build.allow_missing_sources - DECISIONS: row G-12 (state tooling, one module); D-08 now "built" (08.4-01) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…d not parse Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…archive before reset, no notebook scratch in promoted archives - WR-01: names reject '..', trailing '.', '.lock'; promote checks the tag ref and its absence before committing (no orphan commit on a bad tag) - WR-02: archive verifies every member's sha256 before the folder exists, so reset deletes only after a verified archive; symlinks listed as not archived - CR-01 (part): data/checkpoints/platform_notebook/ scratch is never promoted - WR-03: the model-free page says which month it scored - WR-05: list marks an unreadable archive instead of crashing - WR-06: scoped monkeypatch in the promote test Suite 2271 (py3.11/pandas 3 and py3.10/pandas 2.3). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…e four Mac checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
UAT test 1 on the corporate VPN: macrotrends answered HTTP 403 and Yahoo rate-limited ^GSPC, so the fail-loud build correctly wrote nothing. The weekly page then crashed with a FileNotFoundError traceback. - weekly.main checks that monthly_raw and monthly_features exist. If they don't, it names the missing files and the build command and exits 1, with no traceback. - The macrotrends fetch WARNING now names an HTTP 403 block or a rate limit in plain words (network_hint). - The ^GSPC warning says Yahoo throttles shared networks. - BuildFailed adds a one-line VPN/firewall hint when a macrotrends or index_monthly column is missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
On Glenn's Mac, pytest ran in a shell that still exported TC_DATA_DIR=$HOME/tc-cold/data from the cold-start UAT. The session fixture copies its read-only real data from DATA_DIR/checkpoints, which followed that variable to an empty tree, so 8 tests failed: - seven real-data tests found no checkpoints; - the TC_ROOT_DIR cascade test saw the exported TC_DATA_DIR win. conftest now pops TC_ROOT_DIR, TC_CONFIG_DIR, TC_DATA_DIR, TC_OUTPUT_DIR and TC_CHECKPOINT_DIR before any test module imports the package. The run header names any variable it ignored. Tests that exercise the overrides still set them through monkeypatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…ource On the corporate VPN, macrotrends answers HTTP 403 (gold_spot, wti_crude) and Yahoo rate-limits ^GSPC (sp500_close_me), so the fail-loud build stops every time. The only way through was editing the tracked config. - build_platform_data.py --allow-missing COL[,COL...] adds the columns to build.allow_missing_sources for this run only. It refuses a name the build doesn't fetch (exit 2) and logs what it allows. - The BuildFailed message now ends with the exact command, e.g. --allow-missing gold_spot,wti_crude,sp500_close_me. - A test pins the VPN case end to end: the build writes, gold falls back to IAU from its start (month-end, nothing invented before it), oil keeps its FRED primary, and sp500_close_me is simply absent (P&L-only, unread by the weekly page). - Config comment fixed: the macrotrends gold scrape measures 1985-02+, not 1915+. D-08 row and scripts/README updated. Considered and not taken: World Bank Pink Sheet gold (1960+, reachable behind the VPN). It is a monthly average, and gold has no pnl_splice overlay, so it would put averaged prices into gold's P&L (against E-08) and understate gold's volatility in the no-regime book. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…S D-09) Glenn (2026-10-09): use a monthly-average gold price, to get a consistent, long-history source that works on his corporate VPN. Macrotrends gold_spot answers HTTP 403 there, and on the tracked data it covered only 1985-02+ (not the 1915+ the config claimed), sampled near month-end. - New source worldbank_monthly: the World Bank "Pink Sheet" (monthly averages of daily London prices, nominal USD, 1960-01+, free, no key). The download link changes with each monthly issue, so the commodity-markets page is read for the current one; parsed by column name with openpyxl. Reachable from networks where macrotrends gives 403. - splice.gold: [gold_wb, IAU]. gold_spot is dropped from the macrotrends fetch (only the wti_crude oil fallback is left there). - E-08 kept: gold's P&L reads month-end prices through pnl_splice.gold = ratio_splice(gold_wb -> IAU, join 2005-01-31), the same pattern as oil's D-02 join. On real data the seam is exact and post-2005 lag-1 autocorrelation is -0.10 (passes); pre-2005 P&L is averaged, as oil's is pre-1986. - publication_lags: gold_wb 0 (observed by the month's close; the workbook is re-issued about the 2nd business day of m+1). - The lag-marker guard now raises BuildFailed (a RuntimeError), so the build script exits 1 with its message instead of a traceback. - openpyxl added to the [ingestion] extra and requirements.txt. Known traits, recorded in D-09: whole-dollar values, flat at the $35 peg until 1968-01. Tests: fetcher (parse, link discovery, soft failure, offline), gold's live P&L overlay, and the VPN build case (gold keeps its full span). Tests that re-derive the tracked 8.3 record now read it under the gold source it was built with (tests/unit/tracked_record.py), until 08.5 takes the tracked data out of git. No data or output files changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…on a VPN Glenn (2026-10-09): bring gold_spot back, and make it something a VPN build can --allow-missing. - macrotrends_monthly fetches gold_spot again; publication_lags gold_spot 0. - splice.gold chain: [gold_wb, gold_spot, IAU]. The World Bank stays primary; gold's P&L overlay (gold_wb -> IAU, join 2005-01) is unchanged. - gold_spot is an expected source again, so --allow-missing accepts it. On the VPN: --allow-missing gold_spot,wti_crude,sp500_close_me. - If every FRED series fails (the UAT drill, FRED_API_KEY=bogus), the build now says it is the key and stops suggesting --allow-missing for all 19 FRED columns. - The weekly page prints where it wrote the page and how to view it. Tests: chain middle link (gold_wb lost -> gold_spot, not IAU), the IAU-only scenario now loses both primaries, the VPN case blocks gold_spot again, and the FRED-key message. D-09, splicing_rules, build docstring and the UAT note updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
…urce
- UAT: all four checks passed on Glenn's Mac (2026-10-09, corporate VPN,
throwaway ~/tc-cold):
- build from empty with --allow-missing gold_spot,wti_crude,sp500_close_me,
and the bogus-key drill (exit 1, names FRED_API_KEY);
- the no-model page;
- archive -> reset -> restore, with an identical page;
- promote with the real pre-commit hooks (tag and branch deleted, not pushed).
- VERIFICATION: status passed. The human checks are recorded, with what UAT
found and fixed: no-data page exit, pytest TC_*_DIR isolation,
--allow-missing, D-09 World Bank gold, and the FRED-key message.
- ROADMAP/STATE: 08.4 complete (2/2 plans, 2026-10-09). Next is 08.5: a KISS
pass on its CONTEXT, then plan. Suite 2283, registry 48.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uY2PDfrfhs9KL6vSbBAvB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.