Matrices are functions
A matrix is a machine that turns one vector into another. Multiplying by it stretches, rotates and mixes the coordinates.
Play with it
Change the four numbers of the matrix W. The grid shows what W does to the whole plane; the arrow shows one vector x and its image xW.
Area scale (determinant): 1.32
The idea
Written as xW (the convention in PyTorch code), each output coordinate is the dot product of x with one column of W. A 4,096 × 12,288 matrix, like Qwen's up_proj, makes 12,288 such dot products at once.
Matrices can stretch (scale), rotate, shear, flatten a dimension away, or change the number of dimensions entirely. A neural network layer is a matrix followed by a simple nonlinearity.
Almost all of an LLM's 8 billion parameters are entries of matrices like this. “Training” means adjusting those entries.
Matrix multiplication is not elementwise multiplication. xW mixes every input coordinate into every output coordinate; elementwise (⊙) multiplies matching positions only, which is what the MLP's gate does.
Check yourself
x has 4,096 numbers and W is 4,096 × 1,024. How many numbers does xW have?
1,024. The output size is the number of columns.
Which W leaves every vector unchanged?
The identity: 1 on the diagonal, 0 elsewhere.