Vectors and the dot product
A vector is a list of numbers you can picture as an arrow. The dot product measures how much two arrows point the same way.
Play with it
Drag the tips of the two arrows. Watch the dot product and the angle.
- a
- (3, 1)
- b
- (1, 2.5)
- a · b
- 5.50
- angle
- 50°
cos(angle) = a·b / (|a| |b|) = 5.50 / (3.16 × 2.69) = 0.65
The arrows are somewhat aligned.
The idea
Every token inside an LLM is a vector: Qwen3-8B uses 4,096 numbers per token. We can only draw 2 of them, but every rule you see here holds in 4,096 dimensions.
The dot product multiplies matching coordinates and adds them up: a·b = a₁b₁ + a₂b₂ + … It equals |a| |b| cos(angle), so it grows when the arrows are long and when they point the same way.
Dividing by both lengths leaves only the direction part, the cosine similarity, between −1 and 1. Embedding search and “these two words are similar” are this number.
A big dot product does not always mean “similar direction”: a very long vector gives big dot products with almost anything. That is one reason Qwen normalizes queries and keys (QK-norm) before comparing them.
Check yourself
What is (1, 2) · (3, −1)?
1×3 + 2×(−1) = 1.
Two vectors at a right angle: what is their dot product?
Zero, whatever their lengths. cos(90°) = 0.