nanoGPT
Building and training a GPT-style transformer from scratch in PyTorch.
Smaller model experimentTrain a smaller model with same dataset to see if overfitting is reduced What I did: Trained a smaller (5M param) model on the 200M word wikipedia dataset Learned a bit more about…
Larger model trainingTrain larger model on larger dataset for longer, trying to acheive more coherent output What I did: Updated wikipedia scraper algorithm to use priority queue instead of greedy sear…
GPT training — first coherent sentences!Debug model output and train semi coherent model What I did: Debugged model training loop implemented automated, dockerized training setup Trained first coherent model on Vast.ai S…
GPT architectureBuild and test GPT architecture (not training yet) What I did: Finished attention head design finished implemeing full GPT architecture Implemented basic tokenizer and model infere…Starting the transformer implementationBegin implementing transformer paper + gpt1 in pytorch What I did: Revisted attention is all you need learned about layer normalization implemented attention head from scratch in p…