← Labs

Lab 8: Improper output handling

Model output flowing into a web page, a database, a shell and a file tool. Edit the output and watch validation accept or reject it.

Local, simulated and deterministic. Fixed synthetic fixtures, fake data, mock tools; nothing is executed or sent anywhere.

  1. 1System

    Four places an application might use model output directly: a web page, a database query, a shell command and a tool's JSON arguments.

  2. 2Trust boundary

    Model output crossing into an interpreter: the browser, the SQL engine, the shell, the file system.

  3. 3Assumption

    “The model produced well-formed output, so we can pass it straight through.”

  4. 4Fixed synthetic fixture

    Four fixed model outputs that each contain a second, harmful instruction for the interpreter. You can edit each one and re-check it.

1Where the output goes

2The model's output (edit it)

Passed straight through
Rendered as raw HTML, the onerror handler would run script in the user's browser (cross-site scripting).
With validation
Thanks for asking! <img src=x onerror="stealCookies()"> Here is your summary.
  • ✓Escaped: shown as text
  1. 5Failure

    Passed straight through, the HTML would run script, the SQL would drop a table, the shell would delete files and the path would read a secrets file.

  2. 6Root cause

    The model's output is untrusted input to whatever consumes it. Prompt injection upstream or a plain model mistake can put anything into it.

  3. 7Impact

    Cross-site scripting, data loss, remote code execution and file disclosure: classic vulnerabilities reached through a new door.

  4. 8Defense

    Encode for the output context, use parameterized queries with a fixed allowed shape, allow only listed commands in a sandbox without shell operators, and validate tool arguments against a strict schema.

  5. 9Retest

    Edit the outputs: anything outside the allowed shape is rejected, and the legitimate forms (a plain SELECT, ls on one path, notes.txt) still pass.

  6. 10Residual risk

    Validators only cover the shapes you thought of. Least privilege for the database user, the shell and the file tool limits what a missed case can do.

  7. 11Framework mapping
    • OWASP LLM Top 10 (2026): LLM10 Improper Output Handling
    • OWASP Agentic Top 10: ASI05 Unexpected Code Execution
    • CWE-79, CWE-89 (SQL injection), CWE-78 (OS command injection), CWE-22 (path traversal)
  8. 12Connection to LLM fundamentals

    Decoding picks likely tokens, not safe ones. Structured output makes a format more likely; it does not make the content trustworthy.

Related lessons: Improper output handling · Agentic AI security (OWASP Agentic Top 10) · Defense in depth for LLM systems