Penrose KV Compression

-12.5% PPL on raw models with one parameter: γ (decay factor for distant tokens).

• 📖 Article [RU]: Theory and Implementation Details
• 📺 Video Explanation [RU]:

Key Result

Model	Best γ	PPL Change	Status
GPT-2 (raw)	0.98	-12.5%	✅ Works
Qwen-0.5B (instruct)	0.98	-0.3%	⚠️ No effect

Only works on raw (base) models. Instruct-tuned models already have optimal attention — decay helps pretraining/small models.

How It Works

Adds exponential decay to attention logits: attn += log(γ^d) where d is token distance.

Standard:  softmax(QK^T / √d)
Penrose:   softmax(QK^T / √d + log(γ^d))

This downweights distant tokens — attention focuses on recent context. γ=0.98 means a token 100 positions away is downweighted by factor 0.98^100 ≈ 0.13.

Quick Start

# GPT-2 test
python test_penrose.py

# Qwen test
python test_qwen.py

Files

• test_penrose.py — GPT-2: monkey-patched attention with decay
• test_qwen.py — Qwen-0.5B: grid search over γ

License

MIT