Different tasks can tolerate wildly different error budgets
A first-draft summary with a human reviewer downstream can tolerate a meaningfully higher error rate than an unreviewed answer surfaced directly to a customer. The budget should be set per use case, against the actual cost of an error in that specific context, not as one blanket target across every feature.
Measuring the actual rate requires a held-out evaluation set reviewed by humans on a cadence — not vibes, and not the vendor's own benchmark numbers, which rarely reflect your specific data and task.
What happens when you exceed the budget
An SLO is only useful if breaching it triggers something — more human review, a fallback to a simpler deterministic path, or pulling the feature back to an internal-only tier. Define that response before launch, not as an afterthought once the first embarrassing error reaches a customer.
Zero hallucination is not an engineering target. An acceptable, monitored, and budgeted error rate is.MNB AI Practice
Key actions
- Set an error budget per use case based on the real cost of a mistake
- Measure against a human-reviewed evaluation set, not vendor benchmarks
- Define the automatic response when the budget is breached
- Re-measure on a fixed cadence as the model or data changes