Pretraining data: from the web to token sequences
A · 10-second intuition A base model is shaped by what it reads, so most pretraining engineering is deciding which text to keep and how to pack it.
Predict before you read
Why is near-duplicate removal done before pretraining?