GPT training — first coherent sentences!

Goal: Debug model output and train semi coherent model

What I did:
- Debugged model training loop
- implemented automated, dockerized training setup
- Trained first coherent model on Vast.ai
- Set up automated aws s3 dataset fetching and model weight saving
- Started overnight wikipedia scraping job to collect 75000 pages or ~300M words
- Wasted time trying to get the wikipedia scraper to be faster

What worked:
- Vast.ai training on 1 4090 cost less than 1$ and took around a hour to acheive log loss ~1 on a 5000 document dataset (10M words)
-

What failed:
- Wikipedia scraper slow, cannot figure out how to optimize it

Key learning:
- One insight that matters

Next session:
- Train on vast ai multi gpu machine using >100M word dataset
- update code to use Distributed training setup

Time spent: 8 hours