Why Stopping ChatGPT From Lying Could Make It Useless

Michael Koch

3 days ago

In recent weeks, OpenAI published a research paper exposing a key structural flaw in how large language models (LLMs) — including ChatGPT — hallucinate (i.e. produce confident but false statements). Their proposed “fix” is elegant in theory: adjust evaluation metrics so that abstaining (saying “I don’t know” or expressing uncertainty) is rewarded more, and wrong confident guesses are penalized harder. But here’s the twist — some experts argue this very fix could cripple ChatGPT’s appeal overnight.

Let’s unpack why.

What’s causing hallucinations, according to OpenAI

OpenAI’s core insight is that hallucinations are not a glitch so much as a side effect of how models are trained and evaluated:

During training, LLMs learn to predict the most likely next word given context. This probabilistic approach is agnostic to truth; it just cares about plausibility.
In evaluations, models are often judged on accuracy, not whether they express uncertainty. Thus, models get rewarded for guessing rather than refusing when unsure.
Over time, models are optimized to avoid silence. They tend to produce confident outputs—even if they’re mistaken—because test performance favors that.

In effect, AI systems learn to “bluff” when unsure, because in past benchmarks that strategy wins.

OpenAI suggests that if we change these evaluation incentives — so that abstention (with proper caveats) is recognized instead of penalized — models would learn to temper their confidence and reduce hallucinations.

Why the “fix” might kill ChatGPT’s appeal

That’s where the problem lies: users generally expect an AI assistant to *answer* questions, not say “I don’t know” all the time. The proposed trade-off could backfire.

A skeptic, Wei Xing from the University of Sheffield, warns:

If ChatGPT starts refusing to answer 30% of queries (a plausible figure under new metrics), many users would abandon it.
Estimating uncertainty reliably may require exploring multiple candidate responses or running extra computations—hiking costs.
For large-scale consumer services, adding so much overhead while reducing immediate user gratification may break the economics.

In short: the confidence that ChatGPT conveys—rightly or wrongly—is part of what makes it compelling. If you remove that, you might remove usability as well.

The deeper barrier: some hallucinations are irreducible

It’s not just about incentives. The paper also points out that some hallucination risk is inherent due to the nature of language modeling:

When generating sentences, errors cumulate — the probability of a flawless multiword response is much lower than for a simple yes/no.
If a fact is rare or weakly represented in training data, the model has little ground to stand on.

Thus, even with perfect incentive alignment, some level of hallucination might remain. The fix can reduce but not eliminate the problem.

So, what might realistically happen?

Given these trade-offs, here are possible futures:

Hybrid approach: ChatGPT might adopt “confidence thresholds”—only answer when above a threshold, otherwise refuse. But if the threshold is too high, user frustration grows; too low, hallucinations persist.
Tiered models: For mission-critical domains (e.g., law, medicine), use stricter evaluation and higher compute cost; for casual usage, maintain looser standards.
External fact checking/retrieval: Use external knowledge bases or verification modules to validate responses. That way, the model doesn’t just “guess blindly.” (Many current mitigation techniques follow a similar hybrid design.)
New metrics & user education: Shift expectations—users must accept that AI is fallible and should treat it as an advisor, not an oracle.

Conclusion

OpenAI’s diagnosis is bold and likely correct: current practices incentivize confident errors. Their remedy — reward uncertainty — is theoretically sound. But applying it wholesale risks undermining what draws people to ChatGPT in the first place.

We’re left in a bind: do we prefer a more cautious, truthful AI that sometimes refuses to answer, or a bold, more usable AI that occasionally fabricates? The answer may lie in balancing both—and letting different use cases adopt different trade-offs.

Where to go from here

Why Language Models Hallucinate
India Today: OpenAI says it has found why AI chatbots hallucinate and the surprising fix to stop it
Futurism: Fixing Hallucinations Would Destroy ChatGPT, Expert Finds
youth class culture: Why OpenAI’s solution to AI hallucinations would kill ChatGPT …
Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback