Scaling laws and compute-optimal training
A · 10-second intuition Loss falls smoothly and predictably as you add parameters, data and compute, so you can plan a training run before spending the money.
Predict before you read
Roughly how many FLOPs does training a dense model with N parameters on D tokens cost?