Another lawyer just tried to outsource a murder appeal to a chatbot, and the results went about as poorly as you would expect.
New Mexico attorney Stephen Aarons recently found himself slapped with a five-thousand-dollar fine and a contempt of court citation after submitting a legal brief that featured completely fabricated police testimony and entirely fictional witnesses. His excuse? He fed a rough transcript and case notes into OpenAI's ChatGPT to generate a summary, trusted the output blindly, and didn't realize artificial intelligence could invent details out of thin air.
If this story sounds like a repeat of past legal blunders, that is because it is. Years after attorneys in New York got caught submitting briefs loaded with fake court citations generated by AI, members of the bar are still treating predictive text engines like hyper-intelligent paralegals. They treat a statistical word-probability machine like an infallible oracle, and the courts are officially out of patience.
When the New Mexico Supreme Court questioned Aarons about the hallucinated details—including fictional accounts of what the shooter was wearing—justices asked the obvious question: Did he pay attention to any news over the last few years? The bench noted that lawyers relying on unvetted AI output is an everyday cautionary tale. Yet, attorneys keep stepping into the same trap.
The Danger of Trusting the Machine
Large language models are designed to sound confident. When a chatbot does not know a specific fact, it does not raise its hand and say it is clueless. It extrapolates. It generates a statistically pleasing sequence of words that reads like a coherent sentence, even if the underlying reality is pure fiction.
In a casual setting, an AI hallucination is a minor annoyance. In a courtroom, it is career suicide. When you submit a court filing, your signature is a personal guarantee that you checked the facts. You cannot point at a server farm in California and say the computer made you do it.
The defense lawyer in this case was handling the appeal for Oscar Renee Sandoval, who was convicted of murder. By stuffing the primary brief with fake testimony, the attorney actively jeopardized a high-stakes criminal defense. The state supreme court pointedly remarked that Aarons showed a distinct lack of concern for his own client, which is arguably a worse offense than mere technological ignorance.
Why This Keeps Happening
Many practicing attorneys suffer from a fundamental misunderstanding of how software works. They confuse search engines with generative text tools. A traditional search engine points you toward an existing document. A generative AI model builds a response token by token based on patterns.
Lawyers are busy. Dockets are packed, billable hour pressures are intense, and the temptation to automate tedious drafting is massive. Taking a messy, multi-thousand-word trial transcript and asking an AI to summarize it sounds like a smart time-saver.
The mistake isn't using the tool. The mistake is treating the tool as a finished product rather than a rough first draft that requires heavy editing. If you do not independently verify every single case citation, quote, and witness statement against primary sources, you are handing the court a loaded gun and pointing it at your own foot.
How to Avoid Professional Disaster
If you work in a field where accuracy dictates outcomes, you need strict internal rules for handling machine-generated text. Treat every output from an AI tool as an untrustworthy intern who is prone to making things up to impress you.
- Never copy and paste directly. Any block of text generated by an AI must be treated as suspect until cross-referenced with real documents.
- Check the primary source. If a tool cites a witness statement, pull up the actual transcript. If it cites a precedent, read the actual reporter.
- Assume hallucination by default. Build your workflow around the certainty that the model will invent at least one incorrect detail.
The courts are done accepting "honest mistake" as a defense for sloppy digital research. Disciplinary boards are handing out fines, and judges are publicly dismantling lawyers who fail to do basic due diligence. Technology will keep evolving, but the core duty of a professional remains the same: verify your work before it hits the record.
