As more people turn to large-language models (LLMs) during personal crises, the companies behind them are engineering these AI-powered models to be more personal, empathetic, and understanding when prompts indicate a user is in emotional distress.
The trend is raising concern about how well prepared ChatGPT, Claude, Gemini, and other popular LLMs are to provide appropriate emotional support.
To address this concern, Georgia Tech researchers have developed a new neural framework that examines how different training sources influence empathy and social reasoning within these AI-powered models.
“A lot of people are using LLMs to help process the death of a loved one, or maybe their girlfriend broke up with them, or their boyfriend cheated on them,” said Glenn Matlin, a Ph.D. student in the School of Interactive Computing.
Matlin said the new framework he and a Georgia Tech-led research team have developed audits neural networks with an LLM and estimates the influence documents, transcripts, and other data sources have on a model’s output.
“We’re trying to connect the data to outputs, which is helpful for us in understanding where the information a language model produces comes from,” he said.
Matlin is the lead author of a new paper examining model sourcing for social reasoning and how models mimic human behavior in conversations. The paper will be presented this week at the Conference on Language Modeling in San Francisco.
Matlin said the findings suggest that an objective tone in its output indicates that a model places greater importance on textbooks or similar neutral training sources. An empathetic tone suggests the model values sources with dialogue and conversational examples more.
“We find that models tend to learn better from interpersonal dialogue when it comes to social reasoning,” he said. “Models that are good at social reasoning will rely less on textbooks about human emotions and more on FAQs or customer service dialogue.
“It can read all it wants from texts that explain anger, but that wouldn’t help it determine whether someone is angry.”
Professor Mark Riedl, Matlin’s advisor, said that knowing which sources models value most could enable engineers to alter their behavior.
“The question is, in this vast ocean of documents in this neural network, which ones become important in certain situations, and which ones don’t,” Riedl said.
“This leads to the potential for interventions in how we train the model. If we want more empathy, we might look to see where it learns empathy and give it more of those documents, fewer of them, or other documents.”
Riedl said the framework can audit an LLM’s sourcing for advice on any topic, from financial to medical.
“We want to help people make more informed decisions about whether they should use AI for a particular task,” he said.
“You might make a different decision about whether to trust the AI’s recommendation if you knew how a skill was learned and which documents influenced that decision. The first step is knowing how it learns, where it learns from, and what documents tend to lead to certain behaviors.”
The findings that Matlin, Riedl, and their collaborators have made so far came through a partnership with the Allen AI Institute (Ai2), which allowed them to audit their open-sourced LLM, Olmo 3.
While not as large as ChatGPT or Claude, Olmo 3’s training data comprises over a billion documents.
Matilin said it’s impossible to know why ChatGPT or Claude behave in certain ways because their parent companies do not disclose their training data. However, Matlin’s study may help researchers to make better guesses about what’s happening behind the scenes.
He added that the study is the first of its kind and scale on an academic level.
“This is the first one that’s ever been done for an open data model of that type,” he said. “We’re providing a view into neural networks that is hard to come by without being in the corporate setting.”
