Neural Inverse Depth Scaling: Most Layers Update Most Tokens via Ensemble Averaging

This repository contains code to reproduce the experiments in the paper Inverse Depth Scaling From Most Layers Being Similar, Yizhou Liu, Sara Kangaslahti, Ziming Liu, and Jeff Gore, ICML 2026.

Overview

Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies.

Here, we quantify how depth affects loss via analysis of LLMs and toy residual networks. We find loss scales inversely proportional to depth in LLMs, probably due to functionally similar layers reducing error through ensemble averaging rather than compositional learning or discretizing smooth dynamics.

This regime is inefficient yet robust and may arise from the architectural bias of residual networks and target functions incompatible with smooth dynamics. The findings suggest that improving LLM efficiency may require architectural innovations to encourage compositional use of depth.

Repository Structure

LLM hidden state experiments can be found in LLMs, whose description is in Appendix A. Scaling analysis of Chinchilla models are in Scaling, whose description is in Appendix B. Toy model experiments are in exp, whose description is in Appendix C.

Experiment	Where in Paper	Code
LLM Evaluations	Figure 2, a-e	LLMs folder
Chinchilla Data Fitting	Figure 2f	Scaling folder fit-x.ipynb
Toy Model	Figure 3b and Figure 4c	exp-9 and exp-9-1
Toy Model Longer Training	Figure 4, a and b	exp-9-3
Toy Model With MSE and Longer Training	Figure 5	exp-9-6
Causal Trace	Figure 6	causal-trace folder
Additional Analysis on Fitting	Appendix B.3	Scaling folder
Toy Model With High-order Intergration Scheme	Appendix C.3	exp-9-4

Citation

@article{liu2026inverse,
  title={Inverse Depth Scaling From Most Layers Being Similar},
  author={Yizhou Liu and Sara Kangaslahti and Ziming Liu and Jeff Gore},
  journal={arXiv preprint arXiv:2602.05970},
  year={2026}
}

Interested in Other Neural Scaling Laws?

• Width Scaling Due to Limited Representation: Superposition Yields Robust Neural Scaling (paper link, code link)
• Time Scaling Due to Limited Training: Universal One-third Time Scaling in Learning Peaked Distributions (paper link, code link)