| Configuration | Default |
|---|---|
| Model | GPT2LMHeadModel with randomly initialized weights |
| Parameters | Approximately 50M: 49,350,656 at the full 32,768-token vocabulary; fewer if the learned vocabulary is smaller |
| Transformer layers | 10 |
| Attention heads | 8 per layer; 64 dimensions per head |
| Hidden / feed-forward width | 512 / 2,048 |
| Context window | 2,048 tokens |
| Vocabulary | Up to 32,768 tokens, including special tokens |
| Dropout | 0.1 for embeddings, attention, and residual connections |
| Effective batch | Up to 32 windows per optimizer update: microbatch 4 × accumulation 8; up to 65,504 target tokens |
| Training hardware / precision | One A100 GPU; BF16 mixed precision with TF32 enabled |
| Optimizer / schedule | AdamW, betas (0.9, 0.95), weight decay 0.1; 3% warmup, cosine decay from 3e-4 toward 3e-5 |
| Training length | Up to 12 epochs; stop after 3 full epochs without a validation-loss improvement of 0.002 nats/token |
| Session budget | No fixed time limit; optional MAX_SESSION_SECONDS, with resumable local checkpoints |
| Gradient checkpointing | Off by default; enable if GPU memory is tight |
| Validation | Stride-1,024 context windows every 500 updates and at each epoch end; best checkpoint selected |
| Progress samples | Three fixed prompts every 100 optimizer updates; printed and saved |
| Corpus snapshot | 600 unique Gutenberg IDs from CorpusCollection; corrected Sir Tom URL; accepted counts depend on cleaning |
A model trained from scratch on a corpus of female works from the 18th and 19th centuries using a randomly initialized GPT 2 config.
Once finished training, it's looking to be around 50M in size. I might put it on Hugging Face. In the future, I might experiment with further splitting (and perhaps enhancing) this corpus between anonymous female authors/female authors with male pseudonyms and female authors who wrote and published as women while they were alive.
After training on a smaller dataset (around 72 books) for a few epochs, I decided to put the actual run in abeyance while gathering more books. I also have to wait for my compute units to refresh, so you will not be seeing anything from this repository for quite some time.