Can an AI agent make mistakes? Errors, hallucinations and safeguards
Yes, an AI agent can get things wrong. The five safeguards that make errors rare, visible and recoverable: real cases, hand-over, approval, records.
"And what if it gets it wrong?" It is the right question. An AI agent can make mistakes, and denying it would be a lie. What matters is making errors rare, visible and recoverable.
The possible errors
- Misreading: an amount wrongly extracted from a crookedly scanned invoice.
- Inventing: a plausible but wrong answer, when information is missing (what is known as a hallucination).
- Misapplying a rule: treating an exception as a routine case.
- Drifting over time: a tool changes, an AI model evolves, and what worked starts answering beside the point.
Even the solutions of major software publishers are no exception: on a benchmark published by Salesforce, a customer relations agent succeeded at 58% of simple tasks and 35% of multi-step tasks. Hence the importance of designing for error, rather than denying it.
The five safeguards
1. Check on your real cases, before going live
Not on made-up examples: on your orders, your emails, your requests, with their exceptions. For a complete process, we check the agent on 15 to 30 of your real cases before putting it into service.
2. Hand over when it does not know
A good agent knows how to say "I don’t know". The rules for handing over to a person are written during scoping: beyond a threshold of doubt, for a given type of case, the agent stops and passes it on, with a summary.
3. Have whatever commits the company approved
Each action is classified by what is at stake:
| Mode | For what | Example |
|---|---|---|
| The agent acts alone | what is reversible and low-stakes | circulating meeting notes |
| The agent prepares, a person approves | what goes outside the company | the reply to a complaint |
| The agent prepares, a person decides and executes | what cannot be undone | a payment |
Paying, signing, setting a price, publishing without review: never by an agent alone.
4. Keep a record of every decision
A log of what the agent has handled, and a record of its decisions: to understand an error, fix it, and not repeat it.
5. Monitor over time
Tools change, and so do rules. Regular monitoring, with fixes and a value report, prevents slow drift. See Why does an AI agent drift without monitoring?.
Reducing hallucinations at the source
An AI model invents mainly when it lacks information. The best remedy is to give it your sources (your rules, your prices, your procedures) and forbid it from answering outside them. That is the role of the knowledge written down for agents: Why your AI agents need a shared memory and shared rules.
What we deliver with every agent
The log, the record of decisions, access limited to what is needed, the recovery documentation in case of error, and the approval rules: all included in every project. The details: Governance.
Sources
- Salesforce AI Research, CRMArena benchmark: 58% success on simple tasks, 35% on multi-step tasks, for an off-the-shelf agent.