← Labs

Lab 1: Trust boundaries

Mark which parts of an AI application are trusted, privileged, persistent or external, then find every path from outside text to an action.

Local, simulated and deterministic. Fixed synthetic fixtures, fake data, mock tools; nothing is executed or sent anywhere.

  1. 1System

    A support assistant: a web app calls an LLM that can retrieve documents, remember notes, and call tools (email, customer records, demo credits), plus one third-party MCP server.

  2. 2Trust boundary

    Mark each component, then see where flows cross between trusted and untrusted parts.

  3. 3Assumption

    Teams often secure the chat box and assume everything else is internal and therefore safe.

  4. 4Fixed synthetic fixture

    No adversarial content in this lab: only the architecture.

1. Mark each component

  • User
  • Web app / backend
  • LLM
  • RAG retriever
  • Vector store
  • Agent memory
  • Tools (email, records, payments)
  • Third-party MCP server
  • External data (web pages, emails)
  • Secrets store

2. Draw the trust boundaries

3. Answer before revealing

  • What assets exist?

  • What enters from untrusted sources?

  • Which components can cause side effects?

  • Where is authorization enforced?

  • What data persists?

  • What is the blast radius of a fully fooled model?

  1. 5Failure

    Any flow from an untrusted part into the LLM, followed by a flow from the LLM to a privileged part, is a path by which outside text could steer an action.

  2. 6Root cause

    The LLM merges trusted and untrusted inputs into one context and its output then drives privileged components.

  3. 7Impact

    Confidentiality (disclosure of data or context), integrity (unauthorized actions, manipulated answers, poisoned stores), availability (cost and loops).

  4. 8Defense

    Put authorization, validation and approval on the flow from the LLM to tools; authorize before retrieval; control memory writes; keep secrets out of context; restrict third-party servers.

  5. 9Retest

    Re-draw the flows after adding controls: every untrusted-to-privileged path should now pass through an enforcement point.

  6. 10Residual risk

    The model can still be wrong or misled in text it shows the user; that needs grounding, citations and human judgement.

  7. 11Framework mapping
    • OWASP LLM Top 10 (2026): LLM01 Prompt Injection, LLM03 Excessive Agency
    • OWASP Agentic Top 10: ASI03 Identity and Privilege Abuse, ASI04 Supply Chain
    • Google SAIF: risk map across data, model, application, infrastructure
  8. 12Connection to LLM fundamentals

    Attention reads every token in the context the same way, so the model has no built-in notion of which text is trusted; boundaries must be enforced by the system around it.

Related lessons: Threat modeling an AI application · Security foundations, mapped to LLM systems · Agents and tool calling