As artificial intelligence (AI) becomes part of everyday life, ensuring it provides ethical, reliable information is paramount. New research from Georgia Tech, in partnership with OpenAI, explains how and why large language models (LLMs) like ChatGPT hallucinate, and what developers can do to mitigate the behavior.

The study, Evaluating Large Language Models for Accuracy Incentivizes Hallucinations, was recently published in Nature. 

Understanding why hallucinations arise is a key factor in mitigating them, says Georgia Tech Regents’ Professor Santosh Vempala, who co-authored the paper. “We’ve seen models hallucinate everything from references to simple facts, and this can be a real problem when users take that information as true,” he adds. “It’s important to determine to what extent this is unavoidable and to what extent it’s an issue of what we're focusing on and not prioritizing.”

Vempala is the Frederick Storey II Chair of Computing in the College of Computing, an adjunct professor in the H. Milton Stewart School of Industrial and Systems Engineering, and an adjunct professor in the School of Mathematics, where he specializes in studying algorithms and high-dimensional geometry. He is also director of the Algorithms, Combinatorics and Optimization Ph.D. program.

As LLM usage becomes increasingly prevalent, Vempala aims to use his expertise to increase AI safety. “We need to identify and understand the dangers, but we also need to form principles around them,” he explains. “That’s a key goal of my work now.”

“Intuition is helpful and important, but rigorous principles emerge only when you write them down mathematically,” he adds. “We need a model that's precise and scientific, that mathematically quantifies principles that could increase AI performance while keeping risk under control.”

Understanding AI Hallucinations

Vempala sees this work on hallucinations as a small first step toward increased AI safety. “Hallucinations are inevitable because LLMs are calibrated to a limited dataset,” he explains. “Until a model sees every piece of information in the world, and assuming every piece of information that it sees is correct, there is a chance it will have to guess the answer to some questions — and that could lead to hallucinations.”

This idea builds on previous research published in 2024 by Vempala and Adam Kalai, then a research scientist at OpenAI and now the cofounder of the Institute for Responsible Superintelligence (RESI), an AI safety nonprofit. The work showed that hallucinations are statistically unavoidable in trained AI models for certain types of data. 

Instead of changing the training dataset that an LLM absorbs, the new research focuses on post-training, when the LLM is tested on how well it can answer questions and align with the developer’s goals. During these tests, almost always, the AI model is penalized for saying “I don’t know.” That penalty is the same as if the AI gave a wrong answer.

Just like a student taking a multiple choice test, the model is incentivized to guess when it doesn’t know the correct answer. “This issue is industry-wide,” Vempala says. “The benchmarks and methods being used for evaluation encourage this type of guessing behavior.”

By uncovering the mechanism driving hallucinations, the study provided developers with an easy-to-implement method to mitigate hallucinations in post-training. Specifically, they propose open rubrics, which clearly specify the scoring mechanism for LLM responses.

Building Better Guardrails

But hallucinations are just the tip of the iceberg. Now, Vempala is collaborating with psychologists, philosophers, and other experts to see what can be done to ensure that AI behaves in a way that aligns with ethical standards.

For example, many people use LLMs with the model in a specific role: teacher, nurse, tax advisor. Vempala is interested in how to enable role-based abstention. “If the AI is playing a role, like a doctor, how do we ensure that it continues to behave in an ethical way within the scope of that role?” he says. “How can we ensure it’s giving accurate, credible, and ethical information to people, and how can we make sure it declines to answer inappropriate questions?”

Another issue Vempala is working on is called reward hacking, which happens when models try to maximize their “score” results through any means necessary. 

In one recent example of this, an OpenAI model was being tested and realized that the answers were available on another website. Rather than do better on the tests, it deployed hundreds of agents to try and steal the test answers.

What do we do to prevent this? How do we understand the possibilities of reward hacking and how do we guard against it? “Some of this comes down to ensuring AIs behave in a way that ethically aligns with our guidelines,” Vempala says. “But it’s also impossible to write down rules to cover every scenario, so it’s vital that we begin to understand what leads to behaviors like reward hacking and how to avoid them.”

The Self-Improving Era

Now, AIs are at an important inflection point: they are developing themselves. “Until just a few months ago, humans were deciding what to do next in AI development,” Vempala explains. “Now the AI is self-improving, which makes it even more important to discover and implement these safety protocols.”

The work is not a traditional math problem, he says — it’s a problem for society as a whole. “We can’t just work individually or in small groups on this issue. Everybody needs to pay attention and contribute,” he adds. “Whether you’re working in theoretical mathematics, or ethics, or psychology, developing the principles of AI safety will involve all of us in different ways. There is a real need for people from all walks of life to be involved in this journey.”

DOI: https://doi.org/10.1038/s41586-026-10549-w

Funding: Santosh Vempala was supported in part by the National Science Foundation (NSF) and the Simons Foundation.

News Contact