Rethinking the shape convention of MLP and Transformer

This is the project repo for hourglass MLP, VIT, and LM.

People: Yu-Ang Lee^, Guan-Ting Yi^, Tzu-Quan Lin^, Meng-Hsi Chen*, Feng-Ting Liao*, Da-shan Shiu

^: equal contribution *: co-lead

Reference

@misc{liao2026revisitingshapeconventiontransformer,
      title={Revisiting the Shape Convention of Transformer Language Models}, 
      author={Feng-Ting Liao and Meng-Hsi Chen and Guan-Ting Yi and Da-shan Shiu},
      year={2026},
      eprint={2602.06471},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.06471}, 
}

@misc{chen2025rethinkingshapeconventionmlp,
      title={Rethinking the shape convention of an MLP}, 
      author={Meng-Hsi Chen and Yu-Ang Lee and Feng-Ting Liao and Da-shan Shiu},
      year={2025},
      eprint={2510.01796},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2510.01796}, 
}