Some minimal code to pretrain a GPT-2-like model from scratch on the FineWeb-Edu 10B dataset.
The code in this repo is basically taken from build-nanogpt, just a little bit restructured and with minor code changes.
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Some minimal code to pretrain a GPT-2-like model from scratch on the FineWeb-Edu 10B dataset.
The code in this repo is basically taken from build-nanogpt, just a little bit restructured and with minor code changes.