nanoGPT · 2025-11-24 · 3 hours
Goal: Begin implementing transformer paper + gpt1 in pytorch
What I did:
- Revisted attention is all you need
- learned about layer normalization
- implemented attention head from scratch in pytorch
What worked:
- attention head shapes look correct
- understanding of transformer arch is solidifying
What failed:
- Slower at hand writing pytorch than I'd like, but also delayed by trying to fully understand the architecture
Key learning:
- In the decoder, we mask the attention matrix pre-softmax so that the output is autoregressive
- need to pad outputs to ensure entire batch is of the same shape, because thats the only way to make gpu run the computations in parrallel
Next session:
- Implement full transformer block
- Implement GPT model
- Port wikipedia data scraping from wikipedia graph project for data collection
Time spent: 3 hours