Chapter 13 of 14 · Today's models

When machines make mistakes

Language models are amazing, but they can be wrong.

They can say something that sounds very sure, but is not true.

We tested a real model, Qwen, on a famous old speech, the Gettysburg Address.

It guessed the next word right about 77 times out of 100.

And some of its mistakes were not really mistakes! Abraham Lincoln wrote the speech out five times, a little differently each time.

So check important things, and ask a grown-up you trust.

Your turn

Look at what Qwen guessed, and what Lincoln wrote. Then flip the safety cards.

Recorded Qwen measurement· Qwen3-8B-Base on the Gettysburg Address
about 77 in 100 next words right about 3 more match another Lincoln copy the rest were wrong

Lincoln wrote out his speech five times. When Qwen was "wrong", sometimes it said exactly what another copy says:

  • …ago our fathers brought forth ___

    copy we checked
    on
    Qwen guessed
    upon
    Nicolay copy
    upon ✓

  • …a new nation, conceived in ___

    copy we checked
    Liberty
    Qwen guessed
    liberty
    Nicolay copy
    liberty ✓

  • …long endure. We are met ___

    copy we checked
    on
    Qwen guessed
    here
    Hay copy
    here ✓

Safety cards: tap to flip

Grown-ups test models too. OpenAI says GPT-6 Astra was its first model to reach its "critical" level for computer security skills, so at first it let fewer people use it. Anthropic says Claude Opus 5.5 is much less likely than recent models to do things that are hard to undo. These are the companies' own statements: source: TechCrunch, source: Anthropic announcement.
What you learned

Models can be wrong, so we always check.

Finish "Your turn" to earn the Fact checker sticker.
Grown-up corner: the precise version
  • sounding sure but wrong = hallucination

Teacher-forced top-1 recall of Qwen3-8B-Base on the Bliss copy of the Gettysburg Address was 76.9% (bfloat16). Of its 73 misses, 9 are the wording of another Lincoln manuscript (79.7% if any manuscript counts).

Labs also test models for dangerous abilities. OpenAI says GPT-6 Astra met its "critical" cybersecurity threshold, so it limited early access; Anthropic reports Claude Opus 5.5 is much less likely than recent models to take hard-to-reverse actions. These are the companies' own statements.