A research group is calling for internet and social media moderators to strengthen their detection and intervention protocols for violent speech. 

Their study of language detection software found that algorithms struggle to differentiate anti-Asian violence-provoking speech from general hate speech. Left unchecked, threats of violence online can go unnoticed and turn into real-world attacks. 

Researchers from Georgia Tech and the Anti-Defamation League (ADL) teamed together in the study. They made their discovery while testing natural language processing (NLP) models trained on data they crowdsourced from Asian communities. 

“The Covid-19 pandemic brought attention to how dangerous violence-provoking speech can be. There was a clear increase in reports of anti-Asian violence and hate crimes,” said Gaurav Verma, a Georgia Tech Ph.D. candidate who led the study. 

“Such speech is often amplified on social platforms, which in turn fuels anti-Asian sentiments and attacks.”

Violence-provoking speech differs from more commonly studied forms of harmful speech, like hate speech. While hate speech denigrates or insults a group, violence-provoking speech implicitly or explicitly encourages violence against targeted communities.

Humans can define and characterize violent speech as a subset of hateful speech. However, computer models struggle to tell the difference due to subtle cues and implications in language.

The researchers tested five different NLP classifiers and analyzed their F1 score, which measures a model's performance. The classifiers reported a 0.89 score for detecting hate speech, while detecting violence-provoking speech was only 0.69. This contrast highlights the notable gap between these tools and their accuracy and reliability. 

The study stresses the importance of developing more refined methods for detecting violence-provoking speech. Internet misinformation and inflammatory rhetoric escalate tensions that lead to real-world violence. 

The Covid-19 pandemic exemplified how public health crises intensify this behavior, helping inspire the study. The group cited that anti-Asian crime across the U.S. increased by 339% in 2021 due to malicious content blaming Asians for the virus. 

The researchers believe their findings show the effectiveness of community-centric approaches to problems dealing with harmful speech. These approaches would enable informed decision-making between policymakers, targeted communities, and developers of online platforms.

Along with stronger models for detecting violence-provoking speech, the group discusses a direct solution: a tiered penalty system on online platforms. Tiered systems align penalties with severity of offenses, acting as both deterrent and intervention to different levels of harmful speech. 

“We believe that we cannot tackle a problem that affects a community without involving people who are directly impacted,” said Jiawei Zhou, a Ph.D. student who studies human-centered computing at Georgia Tech. 

“By collaborating with experts and community members, we ensure our research builds on front-line efforts to combat violence-provoking speech while remaining rooted in real experiences and needs of the targeted community.”

The researchers trained their tested NLP classifiers on a dataset crowdsourced from a survey of 120 participants who self-identified as Asian community members. In the survey, the participants labeled 1,000 posts from X (formerly Twitter) as containing either violence-provoking speech, hateful speech, or neither.

Since characterizing violence-provoking speech is not universal, the researchers created a specialized codebook for survey participants. The participants studied the codebook before their survey and used an abridged version while labeling. 

To create the codebook, the group used an initial set of anti-Asian keywords to scan posts on X from January 2020 to February 2023. This tactic yielded 420,000 posts containing harmful, anti-Asian language. 

The researchers then filtered the batch through new keywords and phrases. This refined the sample to 4,000 posts that potentially contained violence-provoking content. Keywords and phrases were added to the codebook while the filtered posts were used in the labeling survey.

The team used discussion and pilot testing to validate its codebook. During trial testing, pilots labeled 100 Twitter posts to ensure the sound design of the Asian community survey. The group also sent the codebook to the ADL for review and incorporated the organization’s feedback. 

“One of the major challenges in studying violence-provoking content online is effective data collection and funneling down because most platforms actively moderate and remove overtly hateful and violent material,” said Tech alumnus Rynaa Grover (M.S. CS 2024).

“To address the complexities of this data, we developed an innovative pipeline that deals with the scale of this data in a community-aware manner.”

Emphasis on community input extended into collaboration within Georgia Tech’s College of Computing. Faculty members Srijan Kumar and Munmun De Choudhury oversaw the research that their students spearheaded.

Kumar, an assistant professor in the School of Computational Science and Engineering, advises Verma and Grover. His expertise is in artificial intelligence, data mining, and online safety.

De Choudhury is an associate professor in the School of Interactive Computing and advises Zhou. Their research connects societal mental health and social media interactions.

The Georgia Tech researchers partnered with the ADL, a leading non-governmental organization that combats real-world hate and extremism. ADL researchers Binny Mathew and Jordan Kraemer co-authored the paper.

The group will present its paper at the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), which takes place in Bangkok, Thailand, Aug. 11-16 

ACL 2024 accepted 40 papers written by Georgia Tech researchers. Of the 12 Georgia Tech faculty who authored papers accepted at the conference, nine are from the College of Computing, including Kumar and De Choudhury.

“It is great to see that the peers and research community recognize the importance of community-centric work that provides grounded insights about the capabilities of leading language models,” Verma said. 

“We hope the platform encourages more work that presents community-centered perspectives on important societal problems.” 

Visit https://sites.gatech.edu/research/acl-2024/ for news and coverage of Georgia Tech research presented at ACL 2024.

Social media users may need to think twice before hitting that “Post” button.

A new large-language model (LLM) developed by Georgia Tech researchers can help them filter content that could risk their privacy and offer alternative phrasing that keeps the context of their posts intact.

According to a new paper that will be presented at the 2024 Association for Computing Linguistics(ACL) conference, social media users should tread carefully about the information they self-disclose in their posts.

Many people use social media to express their feelings about their experiences without realizing the risks to their privacy. For example, a person revealing their gender identity or sexual orientation may be subject to doxing and harassment from outside parties. 

Others want to express their opinions without their employers or families knowing.

Ph.D. student Yao Dou and associate professors Alan Ritter and Wei Xu originally set out to study user awareness of self-disclosure privacy risks on Reddit. Working with anonymous users, they created an LLM to detect at-risk content.

While the study boosted user awareness of the personal information they revealed, many called for an intervention. They asked the researchers for assistance to rewrite their posts so they didn’t have to be concerned about privacy.

The researchers revamped the model to suggest alternative phrases that reduce the risk of privacy invasion.

One user disclosed, “I’m 16F I think I want to be a bi M.” The new tool offered alternative phrases such as:

• “I am exploring my sexual identity.”
• “I have a desire to explore new options.”
• “I am attracted to the idea of exploring different gender identities.”

Dou said the challenge is making sure the model provides suggestions that don’t change or distort the desired context of the post.

“That’s why instead of providing one suggestion, we provide three suggestions that are different from each other, and we allow the user to choose which one they want,” Dou said. “In some cases, the discourse information is important to the post, and in that case, they can choose what to abstract.”

WEIGHING THE RISKS

The researchers sampled 10,000 Reddit posts from a pool of 4 million that met their search criteria. They annotated those posts and created 19 categories of self-disclosures, including age, sexual orientation, gender, race or nationality, and location.

From there, they worked with Reddit users to test the effectiveness and accuracy of their model, with 82% giving positive feedback.

However, a contingent thought the model was “oversensitive,” highlighting content they did not believe posed a risk. 

Ultimately, the researchers say users must decide what they will post. 

“It’s a personal decision,” Ritter said. “People need to look at this and think about what they’re writing and decide between this tradeoff of what benefits they are getting from sharing information versus what privacy risks are associated with that.”

Xu acknowledged that future work on the project should include a metric that gives users a better idea of what types of content are more at risk than others.

“It’s kind of the way passwords work,” she said. “Years ago, they never told you your password strength, and now there’s a bar telling you how good your password is. Then you realize you need to add a special character and capitalize some letters, and that’s become a standard. This is telling the public how they can protect themselves. The risk isn’t zero, but it helps them think about it.”

WHAT ARE THE CONSEQUENCES?

While doxing and harassment are the most likely consequences of posting sensitive personal information, especially for those who belong to minority groups, the researchers say users have other privacy concerns.

Users should know that when they draft posts on a site, their input can be extracted by the site’s application programming interface (API). If that site has a data breach, a user’s personal information could fall into unwanted hands.

“I think we should have a path toward having everything work locally on the user’s computer, so it doesn’t rely on any external APIs to send this data off their local machine,” Ritter said.

Ritter added that users could also be targets of popular scams like phishing without ever knowing it. 

“People trying targeted phishing attacks can learn personal information about people online that might help them craft more customized attacks that could make users vulnerable,” he said.

The safest way to avoid a breach of privacy is to stay off social media. But Xu said that’s impractical as there are resources and support these sites can provide that users may not get from anywhere else.

“We want people who may be afraid of social media to use it and feel safe when they post,” she said. “Maybe the best way to get an answer to a question is to ask online, but some people don’t feel comfortable doing that, so a tool like this can make them more comfortable sharing without much risk.”

For more information about Georgia Tech research at ACL, please visit https://sites.gatech.edu/research/acl-2024/.

A Georgia Tech researcher will continue to mitigate harmful post-deployment effects created by artificial intelligence (AI) as he joins the 2024-2025 cohort of fellows selected by the Berkman-Klein Center (BKC) for Internet and Society at Harvard University.

Upol Ehsan is the first Georgia Tech graduate selected by BKC. As a fellow, he will contribute to its mission of exploring and understanding cyberspace, focusing on AI, social media, and university discourse.

Entering its 25th year, the BKC Harvard fellowship program addresses pressing issues and produces impactful research that influences academia and public policy. It offers a global perspective, a vibrant intellectual community, and significant funding and resources that attract top scholars and leaders.

The program is highly competitive and sought after by early career candidates and veteran academic and industry professionals. Cohorts hail from numerous backgrounds, including law, computer science, sociology, political science, neuroscience, philosophy, and media studies. 

“Having the opportunity to join such a talented group of people and working with them is a treat,” Ehsan said. “I’m looking forward to adding to the prismatic network of BKC Harvard and learning from the cohesively diverse community.”

While at Georgia Tech, Ehsan expanded the field of explainable AI (XAI) and pioneered a subcategory he labeled human-centered explainable AI (HCXAI). Several of his papers introduced novel and foundational concepts into that subcategory of XAI.

Ehsan works with Professor Mark Riedl in the School of Interactive Computing and the Human-centered AI and Entertainment Intelligence Lab.

Ehsan says he will continue to work on research he introduced in his 2022 paper The Algorithmic Imprint, which shows how the potential harm from algorithms can linger even after an algorithm is no longer used. His research has informed the United Nations’ algorithmic reparations policies and has been incorporated into the National Institute of Standards and Technology AI Risk Management Framework.

“It’s a massive honor to receive this recognition of my work,” Ehsan said. “The Algorithmic Imprint remains a globally applicable Responsible AI concept developed entirely from the Global South. This recognition is dedicated to the participants who made this work possible. I want to take their stories even further."

While at BKC Harvard, Ehsan will develop a taxonomy of potentially harmful AI effects after a model is no longer used. He will also design a process to anticipate these effects and create interventions. He said his work addresses an “accountability blindspot” in responsible AI, which tends to focus on potential harmful effects created during AI deployment.

New research from Georgia Tech is giving scientists more control options over generative artificial intelligence (AI) models in their studies. Greater customization from this research can lead to discovery of new drugs, materials, and other applications tailor-made for consumers.

The Tech group dubbed its method PRODIGY (PROjected DIffusion for controlled Graph Generation). PRODIGY enables diffusion models to generate 3D images of complex structures, such as molecules from chemical formulas. 

Scientists in pharmacology, materials science, social network analysis, and other fields can use PRODIGY to simulate large-scale networks. By generating 3D molecules from multiple graph datasets, the group proved that PRODIGY could handle complex structures.

In keeping with its name, PRODIGY is the first plug-and-play machine learning (ML) approach to controllable graph generation in diffusion models. This method overcomes a known limitation inhibiting diffusion models from broad use in science and engineering.

“We hope PRODIGY enables drug designers and scientists to generate structures that meet their precise needs,” said Kartik Sharma, lead researcher on the project. “It should also inspire future innovations to precisely control modern generative models across domains.” 

PRODIGY works on diffusion models, a generative AI model for computer vision tasks. While suitable for image creation and denoising, diffusion methods are limited because they cannot accurately generate graph representations of custom parameters a user provides.

PRODIGY empowers any pre-trained diffusion model for graph generation to produce graphs that meet specific, user-given constraints. This capability means, as an example, that a drug designer could use any diffusion model to design a molecule with a specific number of atoms and bonds.

The group tested PRODIGY on two molecular and five generic datasets to generate custom 2D and 3D structures. This approach ensured the method could create such complex structures, accounting for the atoms, bonds, structures, and other properties at play in molecules. 

Molecular generation experiments with PRODIGY directly impact chemistry, biology, pharmacology, materials science, and other fields. The researchers say PRODIGY has potential in other fields using large networks and datasets, such as social sciences and telecommunications.

These features led to PRODIGY’s acceptance for presentation at the upcoming International Conference on Machine Learning (ICML 2024). ICML 2024 is the leading international academic conference on ML. The conference is taking place July 21-27 in Vienna.

Assistant Professor Srijan Kumar is Sharma’s advisor and paper co-author. They worked with Tech alumnus Rakshit Trivedi (Ph.D. CS 2020), a Massachusetts Institute of Technology postdoctoral associate.

Twenty-four Georgia Tech faculty from the Colleges of Computing and Engineering will present 40 papers at ICML 2024. Kumar is one of six faculty representing the School of Computational Science and Engineering (CSE) at the conference.

Sharma is a fourth-year Ph.D. student studying computer science. He researches ML models for structured data that are reliable and easily controlled by users. While preparing for ICML, Sharma has been interning this summer at Microsoft Research in the Research for Industry lab.

“ICML is the pioneering conference for machine learning,” said Kumar. “A strong presence at ICML from Georgia Tech illustrates the ground-breaking research conducted by our students and faculty, including those in my research group.”

Visit https://sites.gatech.edu/research/icml-2024 for news and coverage of Georgia Tech research presented at ICML 2024.

A new machine learning (ML) model created at Georgia Tech is helping neuroscientists better understand communications between brain regions. Insights from the model could lead to personalized medicine, better brain-computer interfaces, and advances in neurotechnology.

The Georgia Tech group combined two current ML methods into their hybrid model called MRM-GP (Multi-Region Markovian Gaussian Process). 

Neuroscientists who use MRM-GP learn more about communications and interactions within the brain. This in turn improves understanding of brain functions and disorders.

“Clinically, MRM-GP could enhance diagnostic tools and treatment monitoring by identifying and analyzing neural activity patterns linked to various brain disorders,” said Weihan Li, the study’s lead researcher. 

“Neuroscientists can leverage MRM-GP for its robust modeling capabilities and efficiency in handling large-scale brain data.” 

MRM-GP reveals where and how communication travels across brain regions. 

The group tested MRM-GP using spike trains and local field potential recordings, two kinds of measurements of brain activity. These tests produced representations that illustrated directional flow of communication among brain regions. 

Experiments also disentangled brainwaves, called oscillatory interactions, into organized frequency bands. MRM-GP’s hybrid configuration allows it to model frequencies and phase delays within the latent space of neural recordings.

MRM-GP combines the strengths of two existing methods: the Gaussian process (GP) and linear dynamical systems (LDS). The researchers say that MRM-GP is essentially an LDS that mirrors a GP.

LDS is a computationally efficient and cost-effective method, but it lacks the power to produce representations of the brain. GP-based approaches boost LDS's power, facilitating the discovery of variables in frequency bands and communication directions in the brain.

Converting GP outputs into an LDS is a difficult task in ML. The group overcame this challenge by instilling separability in the model’s multi-region kernel. Separability establishes a connection between the kernel and LDS while modeling communication between brain regions.

Through this approach, MRM-GP overcomes two challenges facing both neuroscience and ML fields. The model helps solve the mystery of intraregional brain communication. It does so by bridging a gap between GP and LDS, a feat not previously accomplished in ML.

“The introduction of MRM-GP provides a useful tool to model and understand complex brain region communications,” said Li, a Ph.D. student in the School of Computational Science and Engineering (CSE). 

“This marks a significant advancement in both neuroscience and machine learning.”

Fellow doctoral students Chengrui Li and Yule Wang co-authored the paper with Li. School of CSE Assistant Professor Anqi Wu advises the group. 

Each MRM-GP student pursues a different Ph.D. degree offered by the School of CSE. W. Li studies computer science, C. Li studies computational science and engineering, and Wang studies machine learning. The school also offers Ph.D. degrees in bioinformatics and bioengineering.

Wu is a 2023 recipient of the Sloan Research Fellowship for neuroscience research. Her work straddles two of the School’s five research areas: machine learning and computational bioscience. 

MRM-GP will be featured at the world’s top conference on ML and artificial intelligence. The group will share their work at the International Conference on Machine Learning (ICML 2024), which will be held July 21-27 in Vienna. 

ICML 2024 also accepted for presentation a second paper from Wu’s group intersecting neuroscience and ML. The same authors will present A Differentiable Partially Observable Generalized Linear Model with Forward-Backward Message Passing.

Twenty-four Georgia Tech faculty from the Colleges of Computing and Engineering will present 40 papers at ICML 2024. Wu is one of six faculty representing the School of CSE who will present eight total papers.

The group’s ICML 2024 presentations exemplify Georgia Tech’s focus on neuroscience research as a strategic initiative.  

Wu is an affiliated faculty member with the Neuro Next Initiative, a new interdisciplinary program at Georgia Tech that will lead research in neuroscience, neurotechnology, and society. The University System of Georgia Board of Regents recently approved a new neuroscience and neurotechnology Ph.D. program at Georgia Tech. 

“Presenting papers at international conferences like ICML is crucial for our group to gain recognition and visibility, facilitates networking with other researchers and industry professionals, and offers valuable feedback for improving our work,” Wu said. 

“It allows us to share our findings, stay updated on the latest developments in the field, and enhance our professional development and public speaking skills.”

Visit https://sites.gatech.edu/research/icml-2024 for news and coverage of Georgia Tech research presented at ICML 2024.

From weather prediction to drug discovery, math powers the models used in computer simulations. To help these vital tools with their calculations, global experts recently met at Georgia Tech to share ways to make math easier for computers.

Tech hosted the 2024 International Conference on Preconditioning Techniques for Scientific and Industrial Applications (Precond 24), June 10-12. 

Preconditioning accelerates matrix computations, a kind of math used in most large-scale models. These computer models become faster, more efficient, and more accessible with help from preconditioned equations.

“Preconditioning transforms complex numerical problems into more easily solved ones,” said Edmond Chow, a professor at Georgia Tech and co-chair of Precond 24’s local organization and program committees. 

“The new problem wields a better condition number, giving rise to the name preconditioning.”

Researchers from 13 countries presented their work through 20 mini-symposia and seven invited talks at Precond 24. Their work showcased the practicality of preconditioners. 

Vandana Dwarka, an assistant professor at Delft University of Technology, shared newly developed preconditioners for electromagnetic simulations. This technology can be used in further applications ranging from imaging to designing nuclear fusion devices.

Xiaozhe Hu presented a physics-based preconditioner that simulates biophysical processes in the brain, such as blood flow and metabolic waste clearance. Hu brought this research from Tufts University, where he is an associate professor.

Tucker Hartland, a postdoctoral researcher at Lawrence Livermore National Laboratory, discussed preconditioning in contact mechanics. This work improves the modeling of interactions between physical objects that touch each other. Many fields stand to benefit from Hartland’s study, including mechanical engineering, civil engineering, and materials science.

A unique aspect of this year’s conference was an emphasis on machine learning (ML). Between a panel discussion, tutorial, and several talks, experts detailed how to employ ML for preconditioning and how preconditioning can train ML models.

Precond 24 invited seven speakers from institutions around the world to share their research with conference attendees. The presenters were: 

• Monica Dessole, CERN, Switzerland
• Selime Gurol, CERFACS, France
• Alexander Heinlein, Delft University of Technology, Netherlands
• Rui Peng Li, Lawrence Livermore National Laboratory, USA 
• Will Pazner, Portland State University, USA 
• Tyrone Rees, Science and Technology Facilities Council, UK
• Jacob B. Schroder, University of New Mexico, USA

Along with hosting Precond 24, several Georgia Tech researchers participated in the conference through presentations. 

Ph.D. students Hua Huang and Shikhar Shah each presented a paper on the conference’s first day. Alumnus Srinivas Eswar (Ph.D. CS 2022) returned to Atlanta to share research from his current role at Argonne National Laboratory. Chow chaired the ML panel and a symposium on preconditioners for matrices.

“It was an engaging and rewarding experience meeting so many people from this very tight-knit community,” said Shah, who studies computational science and engineering (CSE). “Getting to see talks close to my research provided me with a lot of inspiration and direction for future work.”

Precond 2024 was the thirteenth meeting of the conference, which occurs every two years. 

The conference returned to Atlanta this year for the first time since 2005. Atlanta joins Minneapolis as one of only two cities in the world to host Precond more than once. Precond 24 marked the sixth time the conference met in the U.S. 

Georgia Tech and Emory University’s Department of Mathematics organized and sponsored Precond 24. The U.S. Department of Energy Office of Science co-sponsored the conference with Tech and Emory. 

Georgia Tech entities swarmed together in support of Precond 24. The Office of the Associate Vice President for Research Operations and Infrastructure, College of Computing, and School of CSE co-sponsored the conference.

“The enthusiasm at the conference has been very gratifying. So many people organized sessions at the conference and contributed to the very strong attendance,” Chow said. 

“This is a testament to the continued importance of preconditioning and related numerical methods in a rapidly changing technological world.”

A team’s success in any competitive environment often hinges on how well each member can anticipate the actions of their teammates.

Assistant Professor Christopher MacLellan thinks teachable artificial intelligence (AI) agents are uniquely suited for this role and make ideal teammates for video gamers.

With the help of funding from the U.S. Department of Defense, MacLellan hopes to prove his theory with a conversational, task-performing agent he co-engineered called the Verbal Apprentice Learner (VAL).

“You need the ability to adapt to what your teammates are doing to be an effective teammate,” MacLellan said. “We’re exploring this capability for AI agents in the context of video games.” 

Unlike generative AI chatbots like ChatGPT, VAL uses an interactive task-learning approach. 

“VAL learns how you do things in the way you want them done,” MacLellan said. “When you tell it to do something, it will do it the way you taught it instead of some generic random way from the internet.”

A key difference between VAL and a chatbot is that VAL can perceive and act within the gaming world. A chatbot, like ChatGPT, only perceives and acts within the chat dialog.

MacLellan immersed VAL into an open-sourced, simplified version of the popular Nintendo cooperative video game Overcooked to discover how well the agent can function as a teammate. In Overcooked, up to four players work together to prepare dishes in a kitchen while earning points for every completed order.

How Fast Can Val Learn?

In a study with 12 participants, MacLellan found that users could often correctly teach VAL new tasks with only a few examples.

First, the user must teach VAL how to play the game. Knowing that a single human error could compromise results, MacLellan designed three precautionary features:

• When VAL receives a command such as "cook an onion," it asks clarifying questions to understand and confirm its task. As VAL continues to learn, clarification prompts decrease.
• An “undo” button to ensure users can reverse an errant command.
• VAL contains GPT subcomponents to interpret user input, allowing it to adapt to ambiguous commands and typos. The GPT subcomponents drive changes in VAL’s task knowledge, which it uses to perform tasks without additional guidance.

The participants in MacLellan’s study used these features to ensure VAL learned the tasks correctly. 

The high volume of prompts creates a more tedious experience. Still, MacLellan said it provides detailed data on system performance and user experience. That insight should make designing a more seamless experience in future versions of VAL possible.

The prompts also require the AI to be explainable.

“When VAL learns something, it uses the language model to label each node in the task knowledge graph that the system constructs,” MacLellan said. “You can see what it learned and how it breaks tasks down into actions.”

Beyond Gaming

MacLellan’s Teachable AI Lab is devoted to developing AI that inexperienced users can train.

“We are trying to come up with a more usable system where anyone, including people with limited expertise, could come in and interact with the agent and be able to teach it within just five minutes of interacting with it for the first time,” he said.

His work caught the attention of the Department of Defense, which awarded MacLellan multiple grants to fund several of his projects, including VAL. The possibilities of how the DoD could use VAL, on and off the battlefield, are innumerable.

“(The DoD) envisions a future in which people and AI agents jointly work together to solve problems,” MacLellan said. “You need the ability to adapt to what your teammates are doing to be an effective teammate.

“We look at the dynamics of different teaming circumstances and consider what are the right ways to team AI agents with people. The key hypothesis for our project is agents that can learn on the fly and adapt to their users will make better teammates than those that are pre-trained like GPT.”

Design Your Own Agent

MacLellan is co-organizing a gaming agent design competition sponsored by the Institute of Electrical and Electronic Engineers (IEEE) 2024 Conference on Games in Milan, Italy.

The Dice Adventure Competition invites participants to design their own AI agent to play a multi-player, turn-based dungeon crawling game or to play the game as a human teammate. The competition this month and in July offers $1,000 in prizes for players and agent developers in the top three teams.

Researchers at Georgia Tech are creating accessible museum exhibits that explain artificial intelligence (AI) to middle school students, including the LuminAI interactive AI-based dance partner developed by Regents' Professor Brian Magerko.

Ph.D. students Yasmine Belghith and Atefeh Mahdavi co-led a study in a museum setting that observed how middle schoolers interact with the popular AI chatbot ChatGPT. 

“It’s important for museums, especially science museums, to start incorporating these kinds of exhibits about AI and about using AI so the general population can have that avenue to interact with it and transfer that knowledge to everyday tools,” Belghith said.

Belghith and Mahdavi conducted their study with nine focus groups of 24 students at Chicago’s Museum of Science and Industry. The team used the findings to inform their design of AI exhibits that the museum could display as early as 2025. 

Belghith is a Ph.D. student in human-centered computing. Her advisor is Assistant Professor Jessica Roberts in the School of Interactive Computing. Magerko advises Mahdavi, a Ph.D. student in digital media in the School of Literature, Media, and Communication.

Belghith and Mahdavi presented a paper about their study in May at the Association for Computing Machinery (ACM) 2024 Conference on Human Factors in Computing Systems (CHI) in Honolulu, Hawaii.

Their work is part of a National Science Foundation (NSF) grant dedicated to fostering AI literacy among middle schoolers in informal environments.

Expanding Accessibility

While there are existing efforts to reach students in the classroom, the researchers believe AI education is most accessible in informal learning environments like museums.

“There’s a need today for everybody to have some sort of AI literacy,” Belghith said. “Many middle schoolers will not be taking computer science courses or pursuing computer science careers, so there needs to be interventions to teach them what they should know about AI.”

The researchers found that most of the middle schoolers interacted with ChatGPT to either test its knowledge by prompting it to answer questions or socialize with it by having human-like conversations. 

Others fit the mold of “content explorers.” They did not engage with the AI aspect of ChatGPT and focused more on the content it produced.

Mahdavi said regardless of their approach, students would get “tunnel vision” in their interactions instead of exploring more of the AI’s capabilities.

“If they go in a certain direction, they will continue to explore that,” Mahdavi said. “One thing we can learn from this is to nudge kids and show them there are other things you can do with AI tools or get them to think about it another way.”

The researchers also paid attention to what was missing in the students’ responses, which Mahdavi said was just as important as what they did talk about.

“None of them mentioned anything about ethics or what could be problematic about AI,” she said. “That told us there’s something they aren’t thinking about but should be. We take that into account as we think about future exhibits.”

Making an Impact

The researchers visited the Museum of Science and Industry June 1-2 to conduct the first trial run of three AI-based exhibits they’ve created. One of them is LuminAI, which was developed in Magerko’s Expressive Machinery Lab.

LuminAI is an interactive art installation that allows people to engage in collaborative movement with an AI dance partner. Georgia Tech and Kennesaw State recently held the first performance of AI avatars dancing with human partners in front of a live audience.

Duri Long, a former Georgia Tech Ph.D. student who is now an assistant professor at Northwestern University, designed the second exhibit. KnowledgeNet is an interactive tabletop exhibit in which visitors build semantic networks by adding different characteristics to characters that interact together.

The third exhibit, Data Bites, prompts users to build datasets of pizzas and sandwiches. Their selections train a machine-learning classifier in real time.

Belghith said the exhibits fostered conversations about AI between parents and children.

“The exhibit prototypes successfully engaged children in creative activities,” she said. “Many parents had to pull their kids away to continue their museum tour because the kids wanted more time to try different creations or dance moves.”

Working on a multi-institutional team of investigators, Georgia Tech researchers have helped the state of Georgia become the epicenter for developing K-12 AI educational curriculum nationwide.

The new curriculum introduced by Artificial Intelligence for Georgia (AI4GA) has taught middle school students to use and understand AI. It’s also equipped middle school teachers to teach the foundations of AI.

AI4GA is a branch of a larger initiative, the Artificial Intelligence for K-12 (AI4K12). Funded by the National Science Foundation and led by researchers from Carnegie Mellon University and the University of Florida, AI4K12 is developing national K-12 guidelines for AI education. 

Bryan Cox, the Kapor research fellow in Georgia Tech’s Constellation Center for Equity in Computing, drove a transformative computer science education initiative when he worked at the Georgia Department of Education. Though he is no longer with the DOE, he persuaded the principal investigators of AI4K12 to use Georgia as their testing ground. He became a lead principal investigator for AI4GA.

“We’re using AI4GA as a springboard to contextualize the need for AI literacy in populations that have the potential to be negatively impacted by AI agents,” Cox said.

Judith Uchidiuno, an assistant professor in Georgia Tech’s School of Interactive Computing, began working on the AI4K12 project as a post-doctoral researcher at Carnegie Mellon under lead PI Dave Touretzky. Joining the faculty at Georgia Tech enabled her to be an in-the-classroom researcher for AI4GA. She started her Play and Learn Lab at Georgia Tech and hired two research assistants devoted to AI4GA.

Focusing on students from underprivileged backgrounds in urban, suburban, and rural communities, Uchidiuno said her team has worked with over a dozen Atlanta-based schools to develop an AI curriculum. The results have been promising.

“Over the past three years, over 1,500 students have learned AI due to the work we’re doing with teachers,” Uchidiuno said. “We are empowering teachers through AI. They now know they have the expertise to teach this curriculum.”

AI4GA is in its final semester of NSF funding, and the researchers have made their curriculum and teacher training publicly available. The principal investigators from Carnegie Mellon and the University of Florida will use the curriculum as a baseline for AI4K12.

STARTING STUDENTS YOUNG

Though AI is a complex subject, the researchers argue middle schoolers aren’t too young to learn about how it works and the social implications that come with it.

“Kids are interacting with it whether people like it or not,” Uchidiuno said. “Many of them already have smart devices. Some children have parents with smart cars. More and more students are using ChatGPT. 

“They don’t have much understanding of the impact or the implications of using AI, especially data and privacy. If we want to prepare students who will one day build these technologies, we need to start them young and at least give them some critical thinking skills.”

Will Gelder, a master’s student in Uchidiuno’s lab, helped analyze data exploring the benefits of co-designing the teaching curriculum with teachers based on months of working with students and learning how they understand AI. Rebecca Yu, a research scientist in Uchidiuno’s lab, collected data to determine which parts of the curriculum were effective or ineffective.

Through the BridgeUP STEM Program at Georgia Tech, Uchidiuno worked with high school students to design video games that demonstrate their knowledge of AI based on the AI4GA curriculum. Students designed the games using various maker materials in 2D and 3D representations, and the games are currently in various stages of development by student developers at the Play and Learn Lab.  

“The students love creative projects that let them express their creative thoughts,” Gelder said. “Students love the opportunity to break out markers or crayons and design their dream robot and whatever functions they can think of.”

Yu said her research shows that many students demonstrate the ability to understand advanced concepts of AI through these creative projects.

“To teach the concept of algorithms, we have students use crayons to draw different colors to mimic all the possibilities a computer is considering in its decision-making,” Yu said.

“Many other curricula like ours don’t go in-depth about the technical concepts, but AI4GA does. We show that with appropriate levels of scaffolding and instructions, they can learn them even without mathematical or programming backgrounds.” 

EMPOWERING TEACHERS

Cox cast a wide net to recruit middle school teachers with diverse student groups. A former student of his answered the call.

Amber Jones, a Georgia Tech alumna, taught at a school primarily consisting of Black and Latinx students. She taught a computer science course that covered building websites, using Excel, and basic coding.

Jones said many students didn’t understand the value and applications of what her course was teaching until she transitioned to the AI4GA curriculum.

“AI for Georgia curriculum felt like every other lesson tied right back to the general academics,” Jones said. “I could say, ‘Remember how you said you weren’t going to ever use y equals mx plus b? Well, every time you use Siri, she's running y equals mx plus b.’ I saw them drawing the connections and not only drawing them but looking for them.”

Connecting AI back to their other classes, favorite social media platforms, and digital devices helped students understand the concepts and fostered interest in the curriculum.

Jones’s participation in the program also propelled her career forward. She now works as a consultant teaching AI to middle school students. 

“I’m kind of niche in my experiences,” Jones said. “So, when someone says, ‘Hey, we also want to do something with a young population that involves computer science,’ I’m in a small pool of people that can be looked to for guidance.”

AI4GA quickly cultivated a new group of experts within a short timeframe.

“They’ve made their classes their own,” Cox said. “They add their own tweaks. Over the course of the project, the teachers were engaged in cultivating the lessons for their experience and their context based on the identity of their students.”

Georgia Tech researchers say non-English speakers shouldn’t rely on chatbots like ChatGPT to provide valuable healthcare advice. 

A team of researchers from the College of Computing at Georgia Tech has developed a framework for assessing the capabilities of large language models (LLMs).

Ph.D. students Mohit Chandra and Yiqiao (Ahren) Jin are the co-lead authors of the paper Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries. 

Their paper’s findings reveal a gap between LLMs and their ability to answer health-related questions. Chandra and Jin point out the limitations of LLMs for users and developers but also highlight their potential. 

Their XLingEval framework cautions non-English speakers from using chatbots as alternatives to doctors for advice. However, models can improve by deepening the data pool with multilingual source material such as their proposed XLingHealth benchmark.     

“For users, our research supports what ChatGPT’s website already states: chatbots make a lot of mistakes, so we should not rely on them for critical decision-making or for information that requires high accuracy,” Jin said.   

“Since we observed this language disparity in their performance, LLM developers should focus on improving accuracy, correctness, consistency, and reliability in other languages,” Jin said. 

Using XLingEval, the researchers found chatbots are less accurate in Spanish, Chinese, and Hindi compared to English. By focusing on correctness, consistency, and verifiability, they discovered: 

• Correctness decreased by 18% when the same questions were asked in Spanish, Chinese, and Hindi. 
• Answers in non-English were 29% less consistent than their English counterparts. 
• Non-English responses were 13% overall less verifiable. 

XLingHealth contains question-answer pairs that chatbots can reference, which the group hopes will spark improvement within LLMs.  

The HealthQA dataset uses specialized healthcare articles from the popular healthcare website Patient. It includes 1,134 health-related question-answer pairs as excerpts from original articles.  

LiveQA is a second dataset containing 246 question-answer pairs constructed from frequently asked questions (FAQs) platforms associated with the U.S. National Institutes of Health (NIH).  

For drug-related questions, the group built a MedicationQA component. This dataset contains 690 questions extracted from anonymous consumer queries submitted to MedlinePlus. The answers are sourced from medical references, such as MedlinePlus and DailyMed.   

In their tests, the researchers asked over 2,000 medical-related questions to ChatGPT-3.5 and MedAlpaca. MedAlpaca is a healthcare question-answer chatbot trained in medical literature. Yet, more than 67% of its responses to non-English questions were irrelevant or contradictory.  

“We see far worse performance in the case of MedAlpaca than ChatGPT,” Chandra said. 

“The majority of the data for MedAlpaca is in English, so it struggled to answer queries in non-English languages. GPT also struggled, but it performed much better than MedAlpaca because it had some sort of training data in other languages.” 

Ph.D. student Gaurav Verma and postdoctoral researcher Yibo Hu co-authored the paper. 

Jin and Verma study under Srijan Kumar, an assistant professor in the School of Computational Science and Engineering, and Hu is a postdoc in Kumar’s lab. Chandra is advised by Munmun De Choudhury, an associate professor in the School of Interactive Computing. 
 
The team will present their paper at The Web Conference, occurring May 13-17 in Singapore. The annual conference focuses on the future direction of the internet. The group’s presentation is a complimentary match, considering the conference's location.  

English and Chinese are the most common languages in Singapore. The group tested Spanish, Chinese, and Hindi because they are the world’s most spoken languages after English. Personal curiosity and background played a part in inspiring the study. 

“ChatGPT was very popular when it launched in 2022, especially for us computer science students who are always exploring new technology,” said Jin. “Non-native English speakers, like Mohit and I, noticed early on that chatbots underperformed in our native languages.” 

School of Interactive Computing communications officer Nathan Deen and School of Computational Science and Engineering communications officer Bryant Wine contributed to this report.