Before training: '''
Hello, I'm a language model, Definitive lined 214 1924 Techniques Jenkins Presbytershadowribe constituent moratorium examine bucket Jacenkowikipedia oakón goalt crucial "lé Hello, I'm a language model,Importproblem'tKnow balcon doomed NB Runesidablelegate dogma Post uncsterdam Lone Indonesian applicant speakovo martialdeep barred Hello, I'm a language model, symbolicCommunexistence Adventuresgres Speed message Martialasure Roosevelt bring LG mapped grittyire Constantiggleaddin Matter most Bollclud Hello, I'm a language model, detectordisc Wiz separates Franz Leone vicious desertSETasis lur Dres sensitivity Pants CAN757 identifiable Shapiro spontEnglish overwhelminglyConn Hello, I'm a language model,book 978Ins challenging Cemetery Qinants adoptedometimes aquatic Discipline unbeaten flakes wasting015 br HobbitMaterials outer rive Garrett venues '''
After training: '''
Hello, I'm a language model, that has been shown in most cases on the web, the language will be the target language of general education. But, Hello, I'm a language model, what is a picture on a computer is an embedded system, is that a computer's input is what's been asked to Hello, I'm a language model, and using that model to model complex situations using the same model in three dimensions: One of the most common examples is Hello, I'm a language model, but you can find the default settings for an interface. In a previous tutorial on desktop computer, this is a default- '''
for faster training:
- Add compiler
- Include Flash Attention
- add norm
- Add dynamic learning rate (decay)
- Add decay to >= 2 dimension parameters for AdamW optimizer
-
add feature of expanding small batch size to equivalent of ~0.5M batch size from GPT2 paper while using small core GPU
-
Add DDP (Data distributed parallel) feature to enable parallel training on multiple GPU
https://github.com/karpathy/char-rnn/blob/master/data/tinyshakespeare/input.txt
- other datasetes for LLM training: FineWeb_Edu