Serving systems: batching, paged attention, prefix caching

A · 10-second intuition A model server is a scheduler that keeps the GPU busy by mixing many requests while managing KV-cache memory like an operating system manages pages.