Larger model training

Goal: Train larger model on larger dataset for longer, trying to acheive more coherent output

What I did:
- Updated wikipedia scraper algorithm to use priority queue instead of greedy search
- Updated tokenization and data preprocessing system to allow for more efficient retokenization of large datasets
- Scraped 200M word dataset and trained 30M parameter model on cloud

What worked:
-
- Got Vast.ai training instance going with full utilization of 5090 GPU
- estimated training time ~30 hours for 20 epochs,

What failed:
- Tuning training config to fully utilize GPU + have decent model size took a few retries
- even with 8 hours of scraping, could only get ~200M words.

Key learning:
- Data requirements for transformer seems very high, even a 10M word dataset from yesterday wasnt very good
- I'm starting to see why we use pre trained weights for VLA models (and others)
- May want to create a script that automatically finds a good batch size (~95-99% VRAM utilization)
- Would save some time and money when dealing with larger cloud instances

Next session:
- Plan out project direction considering new knowledge
- Test model inferencing!

Time spent: 4 hours