[ICLR 2025] When Attention Sink Emerges in Language Models: An Empirical View (Spotlight)
-
Updated
Jul 8, 2025 - Python
#
attention-sink
Here are 7 public repositories matching this topic...
Official repository for RegToken - ECCV 2026
-
Updated
Jul 21, 2026 - Python
🐙 Implements Flash Attention with sink for gpt-oss-20b; includes test.py. WIP backward pass, varlen support, and community sync to return softmax_lse only.
flash attention sink with flash-attention attention-sink flash-with flash-sink attention-with with-sink flash-attention-with flash-attention-sink flash-with-sink attention-with-sink flash-attention-with-sink
-
Updated
Jul 27, 2026 - Python
Effective rank, RankMe, E1, CKA and anisotropy on transformer hidden states are determined by one direction. The exact identity, and the attention sink behind it.
transformers language-models reproducibility interpretability training-dynamics attention-sink effective-rank representation-geometry
-
Updated
Jul 30, 2026 - Python
Benchmark of KV-cache eviction policies on GPT-2-medium showing two clear regimes: sink-preserving heuristics dominate at tiny budgets, while attention-based eviction becomes near-lossless at moderate budgets. attention_384 matches full-cache quality with ~33% less cache.
benchmarking transformers pytorch perplexity kv-cache llm-inference attention-sink cache-eviction gpu-profiling
-
Updated
Jul 14, 2026 - Python
Empirical comparison of Absolute PE vs RoPE showing that absolute PE is 63× more sensitive to sink token removal than RoPE. Includes attention sink measurement, KV-cache eviction tolerance, masked-key ablation, perplexity scaling, and decode throughput.
benchmarking transformers pytorch rope perplexity kv-cache positional-encoding llm-inference attention-sink
-
Updated
Jul 15, 2026 - Python
Empirical measurement of attention sink in GPT-2: attention map analysis, per-head classification, and masked-key ablation to assess functional impact on tail perplexity and output distribution.
nlp transformers pytorch interpretability gpt2 huggingface kv-cache llm-inference attention-sink attention-analysis
-
Updated
Jul 13, 2026 - Python
Improve this page
Add a description, image, and links to the attention-sink topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the attention-sink topic, visit your repo's landing page and select "manage topics."