Rolling Normalization: A Simple Correction for Temporal Distributional Shift in Language Model Metrics

Companion code for the paper of the same name. Fine-tuned GPT-2 perplexity on Federal Reserve speeches (1996–2023) produces a statistically significant, Bonferroni-surviving correlation with S&P 500 volatility — and it's entirely spurious. It's driven by Alan Greenspan's deliberately ambiguous "Fed speak" coinciding with a historically volatile period of his tenure, not by speech-level signal. Rolling window normalization — scoring each speech against a trailing window of recent speeches instead of the full corpus — is a one-line, retraining-free fix that eliminates the correlation completely, across every horizon tested.

Why this matters

Any paper correlating a language model metric with an outcome over a multi-year sample can have this problem, and standard robustness checks (surviving Bonferroni correction, holding across asset classes, surviving crisis exclusion, beating linguistic baselines) are not sufficient to catch it — all of those checks pass here, and the result is still noise. Section 5.5 confirms the source directly: retraining the model on post-Greenspan speeches alone makes the raw correlation nearly disappear.

Repository structure

├── src/
│   ├── config.py                 # shared paths, constants, chair-era lookup
│   ├── scrape_speeches.py        # scrapes federalreserve.gov (3 site eras)
│   ├── clean_data.py             # dedupe / length-filter / normalize text
│   ├── finetune.py               # LoRA fine-tune GPT-2 on the speech corpus
│   ├── compute_perplexity.py     # base + fine-tuned perplexity per speech
│   ├── rolling_normalization.py  # ← the paper's core contribution
│   ├── analysis.py               # Table 3: raw correlation + Bonferroni
│   ├── baselines.py              # word count / TTR / sentiment / word length
│   ├── robustness.py             # Tables 4-9: chair-era, crisis exclusion,
│   │                             #   alt assets, rolling-normalization contrast
│   ├── visualize.py              # Figures 1-2
│   └── generate_sample.py        # sanity-check text generation
├── notebooks/
│   └── data_cleaning.ipynb       # exploratory notebook behind clean_data.py
├── models/                       # final LoRA adapters (checkpoints not included)
│   ├── gpt2-finetuned-fedspeeches/
│   └── gpt2-finetuned-fedspeeches-without-greenspan/
├── data/
│   └── fed_speeches_index.csv    # scraped index (URLs/dates/titles only)
├── results/figures/              # generated plots
└── requirements.txt

The core idea

Raw perplexity conflates two things: long-run drift in the underlying language distribution, and short-run, document-level atypicality. Only the second is meaningful signal. rolling_normalization.py isolates it:

z_t = (PPL_t - mean(PPL_{t-W..t-1})) / std(PPL_{t-W..t-1})

using only the preceding W = 40 speeches (about a year of Fed communications) so there's no look-ahead bias. It's a few lines, runs at inference with no retraining, and generalizes to any perplexity-outcome correlation over a long time series — the same correction removes the spurious correlation in the average-word-length baseline too, even though that baseline shows no obvious difference across chair eras on its own.

Quickstart

pip install -r requirements.txt

Full pipeline (scraping takes a few hours; see data/README.md for what's already included vs. what you'll need to regenerate):

python src/scrape_speeches.py
python src/clean_data.py
python src/finetune.py                      # or use the included adapters in models/
python src/compute_perplexity.py
python src/analysis.py                      # Table 3
python src/baselines.py
python src/robustness.py                    # Tables 4-9, incl. rolling normalization
python src/visualize.py                     # Figures 1-2

Just want the headline result? Once you have fed_speeches_perplexity.csv:

python src/rolling_normalization.py

prints the raw-vs-normalized correlation contrast directly.

Results summary

Horizon	Raw perplexity R	p (Bonferroni)	Rolling z-score R	p
21-day	0.091	< 0.001	0.015	0.48
180-day	0.113	< 0.001	0.022	0.31

Full tables (era breakdown, alternative assets, crisis exclusion, top speeches, baseline comparison) are in the paper and reproduced by analysis.py, baselines.py, and robustness.py.

Data and model notes

• Corpus: 1,697 Federal Reserve speeches, 1996–2023, scraped from federalreserve.gov across three site layout eras.
• Model: GPT-2 small fine-tuned via LoRA (r=16, α=32, on c_attn/c_proj), ~2.4M trainable params.
• Not included in this repo: raw scraped text and downstream CSVs (tens of MB each, regenerable via the pipeline above — see data/README.md), and training checkpoints (only the final adapters are kept). The market data is pulled live via yfinance and isn't cached.

Limitations

This is a single-corpus validation on a small model (GPT-2 small); the paper's Section 7 discusses generalization to other domains (political speech, earnings calls, scientific abstracts) and larger models as future work, and is explicit that the analysis is correlational, not causal.

License

MIT — see LICENSE.