a real example of an ai getting something wrong, and what it teaches about trusting output

ai · 2025-08-02 · 6 min

a few months ago i asked an ai to summarize a contract clause. the clause was one sentence long. the summary was one sentence long. the summary was also wrong in a way that changed the meaning, and i did not notice until i read the original again.

the clause said the tenant shall give notice no later than thirty days before the end of the term. the summary said the tenant may give notice thirty days after the end of the term. two words changed, and both changes pointed in the same direction: from an obligation to a permission, and from before to after. if i had trusted the summary, i would have believed the opposite of the contract.

that is the whole lesson, but the interesting part is why it happened.

why the model got it wrong

the model was not asked to preserve legal meaning. it was asked to summarize, and a summary is a compression. compression rewards fluency: a sentence that reads naturally, uses the same vocabulary, and keeps the shape of the original. "shall" and "may" have almost the same shape. "before" and "after" have almost the same shape. a model predicting the next likely word has very little signal that one is the opposite of the other.

this is not a rare failure mode. it is the normal behavior of a system that models language rather than facts. the model knows what a contract clause usually sounds like. it does not know which word the parties actually signed. when the two disagree, the plausible version wins unless something forces the source back into view.

there is a second layer: summaries hide their work. a wrong answer in a chat is visible as an answer. a wrong summary looks like a shorter version of the truth. the format itself invites trust, because it is the format we use when we have already done the reading.

the habit that catches it

the habit is simple and slightly annoying: keep the source open next to the summary, and compare them sentence by sentence. not word by word, sentence by sentence. for a one-sentence clause that takes ten seconds. for a ten-page document it takes longer, which is exactly why people skip it.

the better version of the habit is to ask for the summary and the evidence together. not "summarize this", but "quote the sentence you are summarizing, then summarize it". a model that has to produce the source sentence is much less likely to drift, and when it does drift you can see the drift immediately.

a third version is to ask for the exceptions. "list anything in this clause that changes an obligation, a deadline, or a permission". that question is harder to answer fluently, because the answer has to name the specific words. it is not foolproof, but it moves the task from compression to extraction, and extraction is where the model is most useful.

what this changes about trusting output

it does not mean the model is useless. i still use it to summarize things. i use it more carefully, and i use it for a different job: not to replace the source, but to decide which parts of the source deserve a closer read. a summary is an index. an index is useful. an index is not the book.

the rule i keep is boring: the model can draft, but it cannot be the last reader. for anything with a deadline, a signature, a price, or a legal consequence, the last reader is me. that rule costs a few minutes per document. the alternative costs a lot more.

if you are building a workflow around this, the useful question is not "which model is most accurate". it is "where in this workflow is a wrong answer cheap to catch". put the model there. keep it away from the places where a wrong answer is expensive and invisible.

for the other side of the same problem, see one year of chatgpt. for the plain-language version of what these systems are doing, see what is an llm.