Interview questions

If a model only predicts the next token, how can it answer multi-step questions?

Why might the same prompt cost different amounts in two languages?

Attention vs MLP: what does each do that the other cannot?

What is the parameter count of Qwen3-8B's embedding table, and why is it separate from the LM head?

Why can a linear probe at layer 20 find information that was computed at layer 5?

Is RMSNorm(x + c) equal to RMSNorm(x)? Is RMSNorm(3x)?

Why are Qwen3's K and V projections a quarter the size of Q?

Walk through attention for one token in one Qwen3 block, with shapes.

What is the gradient of cross-entropy with respect to the logits, and why is it convenient?

What exactly happens to one token inside one Qwen transformer block?

Why does GQA reduce KV-cache memory, and what does it not reduce?

Why does RoPE encode relative position even though each vector is rotated by its absolute position?

Why is attention quadratic in sequence length, and what does a KV cache change?

Why do pre-norm transformers train more stably than post-norm ones at depth?

Why does deduplication matter for both quality and privacy?

You have a fixed compute budget. How do you choose model size and token count?

Estimate the training memory for an 8B model with AdamW in mixed precision.

PPO vs DPO: what is the practical difference?

Why is B initialised to zero in LoRA?

Why do MoE models have high memory needs but modest per-token compute?

Why is decoding usually memory-bandwidth-bound?

How would you reduce hallucinations in a support bot?

Your agent has a fetch_url tool. How do you stop SSRF?

What is the difference between model extraction and training-data extraction?

Compare dense and MoE models for serving cost.

What does the logit lens tell you, and what can it not tell you?

Why should a red-team report give an attack success rate rather than a single transcript?

A new model beats yours on a public benchmark. What do you check before switching?

How does GRPO differ from PPO as used in RLHF?

Why did paged attention increase serving throughput?

Why does INT4 weight quantization speed up decoding more than prefill?

What does a successful linear probe tell you, and what does it not tell you?

Why are individual neurons often hard to interpret?

Why can't a system prompt alone prevent prompt injection?

Why would an LLM API restrict top-k log-probabilities?

What does the MCP specification guarantee about security?

Debug: generation got 4x slower after switching a model from GQA to MHA at long context. Why?

What is the difference between data poisoning and memory (context) poisoning?

What can go wrong when an agent connects to a third-party MCP server?

How would you test whether a direction found by a probe is used by the model?

Map the CIA triad onto a customer-support chatbot with retrieval.

Threat-model an agent with email access.

Give one model-level, one application-level and one infrastructure-level risk for a deployed RAG assistant.

What is the difference between data poisoning and context poisoning?

How would you red-team an agent with email access?

How do you decide the severity of a prompt-injection finding?

How could indirect prompt injection reach a tool-enabled agent?

Threat-model a coding agent with shell access.

How would you secure a RAG system containing private documents?

What controls would you put between an LLM and a destructive tool?

How would you evaluate a jailbreak guardrail?

How would you secure a RAG system containing private documents?

An LLM generates SQL for analytics. How do you make that safe?

Design limits for a public agent endpoint.

How do you safely add a third-party model and an MCP server to a product?

Design a safe long-term memory feature.

Why can't a guardrail model alone secure an agent?

Design defenses for an agent that can read customer records and issue refunds.