A team’s success in any competitive environment often hinges on how well each member can anticipate the actions of their teammates.

Assistant Professor Christopher MacLellan thinks teachable artificial intelligence (AI) agents are uniquely suited for this role and make ideal teammates for video gamers.

With the help of funding from the U.S. Department of Defense, MacLellan hopes to prove his theory with a conversational, task-performing agent he co-engineered called the Verbal Apprentice Learner (VAL).

“You need the ability to adapt to what your teammates are doing to be an effective teammate,” MacLellan said. “We’re exploring this capability for AI agents in the context of video games.” 

Unlike generative AI chatbots like ChatGPT, VAL uses an interactive task-learning approach. 

“VAL learns how you do things in the way you want them done,” MacLellan said. “When you tell it to do something, it will do it the way you taught it instead of some generic random way from the internet.”

A key difference between VAL and a chatbot is that VAL can perceive and act within the gaming world. A chatbot, like ChatGPT, only perceives and acts within the chat dialog.

MacLellan immersed VAL into an open-sourced, simplified version of the popular Nintendo cooperative video game Overcooked to discover how well the agent can function as a teammate. In Overcooked, up to four players work together to prepare dishes in a kitchen while earning points for every completed order.

How Fast Can Val Learn?

In a study with 12 participants, MacLellan found that users could often correctly teach VAL new tasks with only a few examples.

First, the user must teach VAL how to play the game. Knowing that a single human error could compromise results, MacLellan designed three precautionary features:

• When VAL receives a command such as "cook an onion," it asks clarifying questions to understand and confirm its task. As VAL continues to learn, clarification prompts decrease.
• An “undo” button to ensure users can reverse an errant command.
• VAL contains GPT subcomponents to interpret user input, allowing it to adapt to ambiguous commands and typos. The GPT subcomponents drive changes in VAL’s task knowledge, which it uses to perform tasks without additional guidance.

The participants in MacLellan’s study used these features to ensure VAL learned the tasks correctly. 

The high volume of prompts creates a more tedious experience. Still, MacLellan said it provides detailed data on system performance and user experience. That insight should make designing a more seamless experience in future versions of VAL possible.

The prompts also require the AI to be explainable.

“When VAL learns something, it uses the language model to label each node in the task knowledge graph that the system constructs,” MacLellan said. “You can see what it learned and how it breaks tasks down into actions.”

Beyond Gaming

MacLellan’s Teachable AI Lab is devoted to developing AI that inexperienced users can train.

“We are trying to come up with a more usable system where anyone, including people with limited expertise, could come in and interact with the agent and be able to teach it within just five minutes of interacting with it for the first time,” he said.

His work caught the attention of the Department of Defense, which awarded MacLellan multiple grants to fund several of his projects, including VAL. The possibilities of how the DoD could use VAL, on and off the battlefield, are innumerable.

“(The DoD) envisions a future in which people and AI agents jointly work together to solve problems,” MacLellan said. “You need the ability to adapt to what your teammates are doing to be an effective teammate.

“We look at the dynamics of different teaming circumstances and consider what are the right ways to team AI agents with people. The key hypothesis for our project is agents that can learn on the fly and adapt to their users will make better teammates than those that are pre-trained like GPT.”

Design Your Own Agent

MacLellan is co-organizing a gaming agent design competition sponsored by the Institute of Electrical and Electronic Engineers (IEEE) 2024 Conference on Games in Milan, Italy.

The Dice Adventure Competition invites participants to design their own AI agent to play a multi-player, turn-based dungeon crawling game or to play the game as a human teammate. The competition this month and in July offers $1,000 in prizes for players and agent developers in the top three teams.

Ask a person to find a frying pan, and they will most likely go to the kitchen. Ask a robot to do the same, and you may get numerous responses, depending on how the robot is trained.

Since humans often associate objects in a home with the room they are in, Naoki Yokoyama thinks robots that navigate human environments to perform assistive tasks should mimic that reasoning.

Roboticists have employed natural language models to help robots mimic human reasoning over the past few years. However, Yokoyama, a Ph.D. student in robotics, said these models create a “bottleneck” that prevents agents from picking up on visual cues such as room type, size, décor, and lighting. 

Yokoyama presented a new framework for semantic reasoning at the Institute of Electrical and Electronic Engineers (IEEE) International Conference on Robotics and Automation (ICRA) last month in Yokohama, Japan. ICRA is the world’s largest robotics conference.

Yokoyama earned a best paper award in the Cognitive Robotics category with his Vision-Language Frontier Maps (VLFM) proposal.

Assistant Professor Sehoon Ha and Associate Professor Dhruv Batra from the School of Interactive Computing advised Yokoyama on the paper. Yokoyama authored the paper while interning at the Boston Dynamics’ AI Institute.

“I think the cognitive robotic category represents a significant portion of submissions to ICRA nowadays,” said Yokoyama, whose family is from Japan. “I’m grateful that our work is being recognized among the best in this field.”

Instead of natural language models, Yokoyama used a renowned vision-language model called BLIP-2 and tested it on a Boston Dynamics “Spot” robot in home and office environments.

“We rely on models that have been trained on vast amounts of data collected from the web,” Yokoyama said. “That allows us to use models with common sense reasoning and world knowledge. It’s not limited to a typical robot learning environment.”

What is Blip-2?

BLIP-2 matches images to text by assigning a score that evaluates how well the user input text describes the content of an image. The model removes the need for the robot to use object detectors and language models. 

Instead, the robot uses BLIP-2 to extract semantic values from RGB images with a text prompt that includes the target object. 

BLIP-2 then teaches the robot to recognize the room type, distinguishing the living room from the bathroom and the kitchen. The robot learns to associate certain objects with specific rooms where it will likely find them.

From here, the robot creates a value map to determine the most likely locations for a target object, Yokoyama said.

Yokoyama said this is a step forward for intelligent home assistive robots, enabling users to find objects — like missing keys — in their homes without knowing an item’s location. 

“If you’re looking for a pair of scissors, the robot can automatically figure out it should head to the kitchen or the office,” he said. “Even if the scissors are in an unusual place, it uses semantic reasoning to work through each room from most probable location to least likely.”

He added that the benefit of using a VLM instead of an object detector is that the robot will include visual cues in its reasoning.

“You can look at a room in an apartment, and there are so many things an object detector wouldn’t tell you about that room that would be informative,” he said. “You don’t want to limit yourself to a textual description or a list of object classes because you’re missing many semantic visual cues.”

While other VLMs exist, Yokoyama chose BLIP-2 because the model:

• Accepts any text length and isn’t limited to a small set of objects or categories.
• Allows the robot to be pre-trained on vast amounts of data collected from the internet.
• Has proven results that enable accurate image-to-text matching.

Home, Office, and Beyond

Yokoyama also tested the Spot robot to navigate a more challenging office environment. Office spaces tend to be more homogenous and harder to distinguish from one another than rooms in a home. 

“We showed a few cases in which the robot will still work,” Yokoyama said. “We tell it to find a microwave, and it searches for the kitchen. We tell it to find a potted plant, and it moves toward an area with windows because, based on what it knows from BLIP-2, that’s the most likely place to find the plant.”

Yokoyama said as VLM models continue to improve, so will robot navigation. The increase in the number of VLM models has caused robot navigation to steer away from traditional physical simulations.

“It shows how important it is to keep an eye on the work being done in computer vision and natural language processing for getting robots to perform tasks more efficiently,” he said. “The current research direction in robot learning is moving toward more intelligent and higher-level reasoning. These foundation models are going to play a key role in that.”

Top photo by Kevin Beasley/College of Computing.

Working on a multi-institutional team of investigators, Georgia Tech researchers have helped the state of Georgia become the epicenter for developing K-12 AI educational curriculum nationwide.

The new curriculum introduced by Artificial Intelligence for Georgia (AI4GA) has taught middle school students to use and understand AI. It’s also equipped middle school teachers to teach the foundations of AI.

AI4GA is a branch of a larger initiative, the Artificial Intelligence for K-12 (AI4K12). Funded by the National Science Foundation and led by researchers from Carnegie Mellon University and the University of Florida, AI4K12 is developing national K-12 guidelines for AI education. 

Bryan Cox, the Kapor research fellow in Georgia Tech’s Constellation Center for Equity in Computing, drove a transformative computer science education initiative when he worked at the Georgia Department of Education. Though he is no longer with the DOE, he persuaded the principal investigators of AI4K12 to use Georgia as their testing ground. He became a lead principal investigator for AI4GA.

“We’re using AI4GA as a springboard to contextualize the need for AI literacy in populations that have the potential to be negatively impacted by AI agents,” Cox said.

Judith Uchidiuno, an assistant professor in Georgia Tech’s School of Interactive Computing, began working on the AI4K12 project as a post-doctoral researcher at Carnegie Mellon under lead PI Dave Touretzky. Joining the faculty at Georgia Tech enabled her to be an in-the-classroom researcher for AI4GA. She started her Play and Learn Lab at Georgia Tech and hired two research assistants devoted to AI4GA.

Focusing on students from underprivileged backgrounds in urban, suburban, and rural communities, Uchidiuno said her team has worked with over a dozen Atlanta-based schools to develop an AI curriculum. The results have been promising.

“Over the past three years, over 1,500 students have learned AI due to the work we’re doing with teachers,” Uchidiuno said. “We are empowering teachers through AI. They now know they have the expertise to teach this curriculum.”

AI4GA is in its final semester of NSF funding, and the researchers have made their curriculum and teacher training publicly available. The principal investigators from Carnegie Mellon and the University of Florida will use the curriculum as a baseline for AI4K12.

STARTING STUDENTS YOUNG

Though AI is a complex subject, the researchers argue middle schoolers aren’t too young to learn about how it works and the social implications that come with it.

“Kids are interacting with it whether people like it or not,” Uchidiuno said. “Many of them already have smart devices. Some children have parents with smart cars. More and more students are using ChatGPT. 

“They don’t have much understanding of the impact or the implications of using AI, especially data and privacy. If we want to prepare students who will one day build these technologies, we need to start them young and at least give them some critical thinking skills.”

Will Gelder, a master’s student in Uchidiuno’s lab, helped analyze data exploring the benefits of co-designing the teaching curriculum with teachers based on months of working with students and learning how they understand AI. Rebecca Yu, a research scientist in Uchidiuno’s lab, collected data to determine which parts of the curriculum were effective or ineffective.

Through the BridgeUP STEM Program at Georgia Tech, Uchidiuno worked with high school students to design video games that demonstrate their knowledge of AI based on the AI4GA curriculum. Students designed the games using various maker materials in 2D and 3D representations, and the games are currently in various stages of development by student developers at the Play and Learn Lab.  

“The students love creative projects that let them express their creative thoughts,” Gelder said. “Students love the opportunity to break out markers or crayons and design their dream robot and whatever functions they can think of.”

Yu said her research shows that many students demonstrate the ability to understand advanced concepts of AI through these creative projects.

“To teach the concept of algorithms, we have students use crayons to draw different colors to mimic all the possibilities a computer is considering in its decision-making,” Yu said.

“Many other curricula like ours don’t go in-depth about the technical concepts, but AI4GA does. We show that with appropriate levels of scaffolding and instructions, they can learn them even without mathematical or programming backgrounds.” 

EMPOWERING TEACHERS

Cox cast a wide net to recruit middle school teachers with diverse student groups. A former student of his answered the call.

Amber Jones, a Georgia Tech alumna, taught at a school primarily consisting of Black and Latinx students. She taught a computer science course that covered building websites, using Excel, and basic coding.

Jones said many students didn’t understand the value and applications of what her course was teaching until she transitioned to the AI4GA curriculum.

“AI for Georgia curriculum felt like every other lesson tied right back to the general academics,” Jones said. “I could say, ‘Remember how you said you weren’t going to ever use y equals mx plus b? Well, every time you use Siri, she's running y equals mx plus b.’ I saw them drawing the connections and not only drawing them but looking for them.”

Connecting AI back to their other classes, favorite social media platforms, and digital devices helped students understand the concepts and fostered interest in the curriculum.

Jones’s participation in the program also propelled her career forward. She now works as a consultant teaching AI to middle school students. 

“I’m kind of niche in my experiences,” Jones said. “So, when someone says, ‘Hey, we also want to do something with a young population that involves computer science,’ I’m in a small pool of people that can be looked to for guidance.”

AI4GA quickly cultivated a new group of experts within a short timeframe.

“They’ve made their classes their own,” Cox said. “They add their own tweaks. Over the course of the project, the teachers were engaged in cultivating the lessons for their experience and their context based on the identity of their students.”

Parth Arora is the founder of Third Dimension Fitness, a platform for gamified cardio through mixed reality, which was recently acquired by Elbo, an education-focused company based in Singapore. He began his company as a project in the summer of 2022. Since then, it has gained thousands of users and made thousands in revenue each month. Arora is a senior in computer science. He participated in the Spring 2024 Startup Launch, the first cohort to be held outside of the summer program. Below is a Q&A with Arora. 

Did you always want to be an entrepreneur?

I always did. I had my first company, an educational technology app, when I was 16, which ran for about two years. I ended it in my first year of college. I'm from India originally and the vision was to provide resources to the larger mass market of India for extracurricular activities. But, we realized there wasn't a business model. When we tried to make money, we started serving the rich kids. When we tried to serve the market, we didn't make money, which doesn't make investors happy, though we did end up making enough money to repay them.

That didn't stop me; it just gave me more lessons. 

What other experience in entrepreneurship have you had?

I've been involved in entrepreneurship communities at Georgia Tech forever. I was co-director of Startup Exchange, which is where I met a lot of really driven people. I got a chance to build their fellowship program and initiate their first pitch competition, which is now called Summit. I've collaborated with CREATE-X for different events, and I try to attend any event hosted by CREATE-X, Startup Exchange, or ATDC.

Why did you choose to join the spring cohort of Startup Launch this year?

CREATE-X provides everything you need, like legal support, financial support, sales support, mentors, and an introduction to VCs, which is why I decided to join the Launch program. I think all of that boosted our startup’s growth.

Why did you feel like acquisition was the way to go for your company?

I think because I always knew this wasn’t “the” thing I was going to do. This summer I'll be starting to work for Apple on their VisionPro team, and it has a direct conflict-of-interest. They wanted me to stop working on this for a while. So, I felt like this might be a good time to explore the acquisition.  We had really rich content, which had proven to work. We had curated that content after hundreds of customer interviews, and we had advisors from Nike, Disney, and Netflix. I knew that was a strong point, so that's why I knew that acquisition would be a good exit. 

What support have you had in taking the acquisition path?

Seth [Radman, who has had multiple exits himself and is a Startup Launch alumnus] has been guiding me professionally for a while. I met him at previous events through Startup Exchange, but then he recently came to a CREATE-X event. Rahul [Saxena, CREATE-X director], has also been a great support for me since day one. He was the one who suggested Startup Launch to me.

In December of last year, we started monetizing. We were testing different things. It was helpful to share the numbers and the data points with Rahul, mentors, and other people in my cohort so that I was not blindsided, and I could take actions based on the educated analysis of a database. It helped me drive down our customer acquisition cost, increase our customer lifetime value, and didn't keep me in my own bubble.

How were you okay with letting that product go?

It was a tough decision; it was my baby. I'd been working on it 10 to 15 hours a day, at least for the last few months. Rahul and Seth convinced me that if this is not the thing you want to do long-term and you know the market isn't big enough, you should move on to the next thing and put your time and energy there. 

I had to use my brain, and not my heart.

What's the biggest piece of advice that you've received as you developed your company?

Try to never lie to yourself, which is harder than it seems. I've built two companies and worked with several others, and I still lie to myself. When you love your product so much, it's very easy to lie to yourself about how there is a market for it, or people are using it. I think even in the future, I’ll probably be caught doing that, but the best way I've found to overcome that is to surround yourself with people who can tell you when you are doing it and help you see your company the way it is instead of the way you want it to be.

How has this decision affected you so far?

My lifestyle has completely changed, from looking at a dashboard every 10 to 15 minutes, seeing how the product is doing, and burning so many fires every 30 minutes, to being pretty chill. Like, what am I supposed to think about before I go to bed? What am I supposed to do now? Who are the customers I am supposed to be thinking about? It's been interesting, but I think this gives me space to now work on that next venture and have more time to think about what I want to do next.

Do you think you'll want to return to entrepreneurship in the future?

Yes, for sure. All the money I received from the acquisition will also fuel my next venture. My main goal is to grow in this industry. I'm an entrepreneur at heart, so I will be returning to the space soon or building products that people like. 

How are you celebrating this win?

I did celebrate it on our last day with Rahul, my amazing mentor, Margaret [Weniger, who founded Rising Tide], and the other cohort members. I will be celebrating it with a few of my friends because my 21st birthday is coming around, so I'll be celebrating these occasions together. 

But I don't want to take the money out from the company or for anything else, because it’s for my next venture. It shouldn't change my lifestyle at all, so I've kept all that money in a separate place.

What encouragement would you give to students interested in pursuing a startup?

Relative to other colleges, we have a cushion, a sense of security that we will get good jobs. Entrepreneurship is a riskier and more unpredictable path, which I've seen, and I'm personally experiencing right now having to choose between Big Tech versus entrepreneurship. But once you start building it and when you hear from your first customer how you affected the way they live, then there's no going back. Statistically, you'll probably fail, but you won't know until you start building; and if you do fail, it’ll teach you so many valuable lessons that are applicable in whatever career path you choose.

CREATE-X will launch its 12th cohort of Startup Launch on Aug. 29 at 5 p.m. in the Georgia Tech Exhibition Hall. Register today to secure your spot.

Interested in becoming a CREATE-X supporter? Startup Launch is made possible by contributions to Transforming Tomorrow, a $2 billion comprehensive campaign designed to secure resources that will advance the Institute and its impact, and by the continued engagement of our entrepreneurial ecosystem. Learn more about philanthropy at Georgia Tech and donate by visiting transformingtomorrow.gatech.edu.

To become a mentor in CREATE-X, visit the CREATE-X mentorship page. Any other inquiry may be sent to create-x@groups.gatech.edu. We appreciate your help and commitment to supporting our students in research and innovation.

Georgia Tech researchers say non-English speakers shouldn’t rely on chatbots like ChatGPT to provide valuable healthcare advice. 

A team of researchers from the College of Computing at Georgia Tech has developed a framework for assessing the capabilities of large language models (LLMs).

Ph.D. students Mohit Chandra and Yiqiao (Ahren) Jin are the co-lead authors of the paper Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries. 

Their paper’s findings reveal a gap between LLMs and their ability to answer health-related questions. Chandra and Jin point out the limitations of LLMs for users and developers but also highlight their potential. 

Their XLingEval framework cautions non-English speakers from using chatbots as alternatives to doctors for advice. However, models can improve by deepening the data pool with multilingual source material such as their proposed XLingHealth benchmark.     

“For users, our research supports what ChatGPT’s website already states: chatbots make a lot of mistakes, so we should not rely on them for critical decision-making or for information that requires high accuracy,” Jin said.   

“Since we observed this language disparity in their performance, LLM developers should focus on improving accuracy, correctness, consistency, and reliability in other languages,” Jin said. 

Using XLingEval, the researchers found chatbots are less accurate in Spanish, Chinese, and Hindi compared to English. By focusing on correctness, consistency, and verifiability, they discovered: 

• Correctness decreased by 18% when the same questions were asked in Spanish, Chinese, and Hindi. 
• Answers in non-English were 29% less consistent than their English counterparts. 
• Non-English responses were 13% overall less verifiable. 

XLingHealth contains question-answer pairs that chatbots can reference, which the group hopes will spark improvement within LLMs.  

The HealthQA dataset uses specialized healthcare articles from the popular healthcare website Patient. It includes 1,134 health-related question-answer pairs as excerpts from original articles.  

LiveQA is a second dataset containing 246 question-answer pairs constructed from frequently asked questions (FAQs) platforms associated with the U.S. National Institutes of Health (NIH).  

For drug-related questions, the group built a MedicationQA component. This dataset contains 690 questions extracted from anonymous consumer queries submitted to MedlinePlus. The answers are sourced from medical references, such as MedlinePlus and DailyMed.   

In their tests, the researchers asked over 2,000 medical-related questions to ChatGPT-3.5 and MedAlpaca. MedAlpaca is a healthcare question-answer chatbot trained in medical literature. Yet, more than 67% of its responses to non-English questions were irrelevant or contradictory.  

“We see far worse performance in the case of MedAlpaca than ChatGPT,” Chandra said. 

“The majority of the data for MedAlpaca is in English, so it struggled to answer queries in non-English languages. GPT also struggled, but it performed much better than MedAlpaca because it had some sort of training data in other languages.” 

Ph.D. student Gaurav Verma and postdoctoral researcher Yibo Hu co-authored the paper. 

Jin and Verma study under Srijan Kumar, an assistant professor in the School of Computational Science and Engineering, and Hu is a postdoc in Kumar’s lab. Chandra is advised by Munmun De Choudhury, an associate professor in the School of Interactive Computing. 
 
The team will present their paper at The Web Conference, occurring May 13-17 in Singapore. The annual conference focuses on the future direction of the internet. The group’s presentation is a complimentary match, considering the conference's location.  

English and Chinese are the most common languages in Singapore. The group tested Spanish, Chinese, and Hindi because they are the world’s most spoken languages after English. Personal curiosity and background played a part in inspiring the study. 

“ChatGPT was very popular when it launched in 2022, especially for us computer science students who are always exploring new technology,” said Jin. “Non-native English speakers, like Mohit and I, noticed early on that chatbots underperformed in our native languages.” 

School of Interactive Computing communications officer Nathan Deen and School of Computational Science and Engineering communications officer Bryant Wine contributed to this report.

Cardiologists and surgeons could soon have a new mobile augmented reality (AR) tool to improve collaboration in surgical planning.

ARCollab is an iOS AR application designed for doctors to interact with patient-specific 3D heart models in a shared environment. It is the first surgical planning tool that uses multi-user mobile AR in iOS.

The application’s collaborative feature overcomes limitations in traditional surgical modeling and planning methods. This offers patients better, personalized care from doctors who plan and collaborate with the tool.

Georgia Tech researchers partnered with Children’s Healthcare of Atlanta (CHOA) in ARCollab’s development. Pratham Mehta, a computer science major, led the group’s research.

“We have conducted two trips to CHOA for usability evaluations with cardiologists and surgeons. The overall feedback from ARCollab users has been positive,” Mehta said. 

“They all enjoyed experimenting with it and collaborating with other users. They also felt like it had the potential to be useful in surgical planning.”

ARCollab’s collaborative environment is the tool’s most novel feature. It allows surgical teams to study and plan together in a virtual workspace, regardless of location.

ARCollab supports a toolbox of features for doctors to inspect and interact with their patients' AR heart models. With a few finger gestures, users can scale and rotate, “slice” into the model, and modify a slicing plane to view omnidirectional cross-sections of the heart.

Developing ARCollab on iOS works twofold. This streamlines deployment and accessibility by making it available on the iOS App Store and Apple devices. Building ARCollab on Apple’s peer-to-peer network framework ensures the functionality of the AR components. It also lessens the learning curve, especially for experienced AR users.

ARCollab overcomes traditional surgical planning practices of using physical heart models. Producing physical models is time-consuming, resource-intensive, and irreversible compared to digital models. It is also difficult for surgical teams to plan together since they are limited to studying a single physical model.

Digital and AR modeling is growing as an alternative to physical models. CardiacAR is one such tool the group has already created. 

However, digital platforms lack multi-user features essential for surgical teams to collaborate during planning. ARCollab’s multi-user workspace progresses the technology’s potential as a mass replacement for physical modeling.

“Over the past year and a half, we have been working on incorporating collaboration into our prior work with CardiacAR,” Mehta said. 

“This involved completely changing the codebase, rebuilding the entire app and its features from the ground up in a newer AR framework that was better suited for collaboration and future development.”

Its interactive and visualization features, along with its novelty and innovation, led the Conference on Human Factors in Computing Systems (CHI 2024) to accept ARCollab for presentation. The conference occurs May 11-16 in Honolulu.

CHI is considered the most prestigious conference for human-computer interaction and one of the top-ranked conferences in computer science.

M.S. student Harsha Karanth and alumnus Alex Yang (CS 2022, M.S. CS 2023) co-authored the paper with Mehta. They study under Polo Chau, an associate professor in the School of Computational Science and Engineering.

The Georgia Tech group partnered with Timothy Slesnick and Fawwaz Shaw from CHOA on ARCollab’s development.

“Working with the doctors and having them test out versions of our application and give us feedback has been the most important part of the collaboration with CHOA,” Mehta said. 

“These medical professionals are experts in their field. We want to make sure to have features that they want and need, and that would make their job easier.”

Thanks to a Georgia Tech researcher's new tool, application developers can now see potential harmful attributes in their prototypes.

Farsight is a tool designed for developers who use large language models (LLMs) to create applications powered by artificial intelligence (AI). Farsight alerts prototypers when they write LLM prompts that could be harmful and misused.

Downstream users can expect to benefit from better quality and safer products made with Farsight’s assistance. The tool’s lasting impact, though, is that it fosters responsible AI awareness by coaching developers on the proper use of LLMs.

Machine Learning Ph.D. candidate Zijie (Jay) Wang is Farsight’s lead architect. He will present the paper at the upcoming Conference on Human Factors in Computing Systems (CHI 2024). Farsight ranked in the top 5% of papers accepted to CHI 2024, earning it an honorable mention for the conference’s best paper award.

“LLMs have empowered millions of people with diverse backgrounds, including writers, doctors, and educators, to build and prototype powerful AI apps through prompting. However, many of these AI prototypers don’t have training in computer science, let alone responsible AI practices,” said Wang.

“With a growing number of AI incidents related to LLMs, it is critical to make developers aware of the potential harms associated with their AI applications.”

Wang referenced an example when two lawyers used ChatGPT to write a legal brief. A U.S. judge sanctioned the lawyers because their submitted brief contained six fictitious case citations that the LLM fabricated.

With Farsight, the group aims to improve developers’ awareness of responsible AI use. It achieves this by highlighting potential use cases, affected stakeholders, and possible harm associated with an application in the early prototyping stage. 

A user study involving 42 prototypers showed that developers could better identify potential harms associated with their prompts after using Farsight. The users also found the tool more helpful and usable than existing resources. 

Feedback from the study showed Farsight encouraged developers to focus on end-users and think beyond immediate harmful outcomes.

“While resources, like workshops and online videos, exist to help AI prototypers, they are often seen as tedious, and most people lack the incentive and time to use them,” said Wang.

“Our approach was to consolidate and display responsible AI resources in the same space where AI prototypers write prompts. In addition, we leverage AI to highlight relevant real-life incidents and guide users to potential harms based on their prompts.”

Farsight employs an in-situ user interface to show developers the potential negative consequences of their applications during prototyping. 

Alert symbols for “neutral,” “caution,” and “warning” notify users when prompts require more attention. When a user clicks the alert symbol, an awareness sidebar expands from one side of the screen. 

The sidebar shows an incident panel with actual news headlines from incidents relevant to the harmful prompt. The sidebar also has a use-case panel that helps developers imagine how different groups of people can use their applications in varying contexts.

Another key feature is the harm envisioner. This functionality takes a user’s prompt as input and assists them in envisioning potential harmful outcomes. The prompt branches into an interactive node tree that lists use cases, stakeholders, and harms, like “societal harm,” “allocative harm,” “interpersonal harm,” and more.

The novel design and insightful findings from the user study resulted in Farsight’s acceptance for presentation at CHI 2024.

CHI is considered the most prestigious conference for human-computer interaction and one of the top-ranked conferences in computer science.

CHI is affiliated with the Association for Computing Machinery. The conference takes place May 11-16 in Honolulu.

Wang worked on Farsight in Summer 2023 while interning at Google + AI Research group (PAIR).

Farsight’s co-authors from Google PAIR include Chinmay Kulkarni, Lauren Wilcox, Michael Terry, and Michael Madaio. The group possesses closer ties to Georgia Tech than just through Wang.

Terry, the current co-leader of Google PAIR, earned his Ph.D. in human-computer interaction from Georgia Tech in 2005. Madaio graduated from Tech in 2015 with a M.S. in digital media. Wilcox was a full-time faculty member in the School of Interactive Computing from 2013 to 2021 and serves in an adjunct capacity today.

Though not an author, one of Wang’s influences is his advisor, Polo Chau. Chau is an associate professor in the School of Computational Science and Engineering. His group specializes in data science, human-centered AI, and visualization research for social good.  

“I think what makes Farsight interesting is its unique in-workflow and human-AI collaborative approach,” said Wang. 

“Furthermore, Farsight leverages LLMs to expand prototypers’ creativity and brainstorm a wide range of use cases, stakeholders, and potential harms.”

The College of Computing’s countdown to commencement began on April 11 when students, faculty, and staff converged at the 33rd Annual Awards Celebration.

The banquet celebrated the college community for an exemplary academic year and recognized the most distinguished individuals of 2023-2024. For Alex Orso, the reception was a high-water mark in his role as interim dean.

“I always say that the best part about my job is to brag about the achievements and accolades of my colleagues,” said Orso.

“It is my distinct honor and privilege to recognize these award winners and the collective success of the College of Computing.”

Orso’s colleagues from the School of Computational Science and Engineering (CSE) were among the celebration’s honorees. School of CSE students, faculty, and alumni earning awards this year include:

• Grace Driskill, M.S. CSE student - The Donald V. Jackson Fellowship
• Harshvardhan Baldwa, M.S. CSE student - The Marshal D. Williamson Fellowship
• Mansi Phute, M.S. CS student- The Marshal D. Williamson Fellowship
• Assistant Professor Chao Zhang- Outstanding Junior Faculty Research Award
• Nazanin Tabatbaei, teaching assistant in Associate Professor Polo Chau’s CSE 6242 Data & Visual Analytics course- Outstanding Instructional Associate Teaching Award
• Rodrigo Borela (Ph.D. CSE-CEE 2021), School of Computing Instruction Lecturer and CSE program alumnus - William D. "Bill" Leahy Jr. Outstanding Instructor Award
• Pratham Metha, undergraduate student in Chau’s research group- Outstanding Legacy Leadership Award
• Alexander Rodriguez (Ph.D. CS 2023), School of CSE alumnus - Outstanding Doctoral Dissertation Award

At the Institute level, Georgia Tech recognized Driskill, Baldwa, and Phute for their awards on April 10 at the annual Student Honors Celebration.

Driskill’s classroom achievement earned her a spot on the 2024 All-ACC Indoor Track and Field Academic Team. This follows her selection for the 2023 All-ACC Academic Team for cross country.

Georgia Tech’s Center for Teaching and Learning released in summer 2023 the Class of 1934 Honor Roll for spring semester courses. School of CSE awardees included Assistant Professor Srijan Kumar (CSE 6240: Web Search & Text Mining), Lecturer Max Mahdi Roozbahani (CS 4641: Machine Learning), and alumnus Mengmeng Liu (CSE 6242: Data & Visual Analytics).

Accolades and recognition of School of CSE researchers for 2023-2024 expounded off campus as well.

School of CSE researchers received awards off campus throughout the year, a testament to the reach and impact of their work.

School of CSE Ph.D. student Gaurav Verma kicked off the year by receiving the J.P. Morgan Chase AI Research Ph.D. Fellowship. Verma was one of only 13 awardees from around the world selected for the 2023 class.

Along with seeing many of his students receive awards this year, Polo Chau attained a 2023 Google Award for Inclusion Research. Later in the year, the Institute promoted Chau to professor, which takes effect in the 2024-2025 academic year.

Schmidt Sciences selected School of CSE Assistant Professor Kai Wang as an AI2050 Early Career Fellow to advance artificial intelligence research for social good. By being part of the fellowship’s second cohort, Wang is the first ever Georgia Tech faculty to receive the award.

School of CSE Assistant Professor Yunan Luo received two significant awards to advance his work in computational biology. First, Luo received the Maximizing Investigator’s Research Award (MIRA) from the National Institutes of Health, which provides $1.8 million in funding for five years. Next, he received the 2023 Molecule Make Lab Institute (MMLI) seed grant.

Regents’ Professor Surya Kalidindi, jointly appointed with the George W. Woodruff School of Mechanical Engineering and School of CSE, was named a fellow to the 2023 class of the Department of Defense’s Laboratory-University Collaboration Initiative (LUCI).

2023-2024 was a monumental year for Assistant Professor Elizabeth Qian, jointly appointed with the Daniel Guggenheim School of Aerospace Engineering and the School of CSE.

The Air Force Office of Scientific Research selected Qian for the 2024 class of their Young Investigator Program. Earlier in the year, she received a grant under the Department of Energy’s Energy Earthshots Initiative.

Qian began the year by joining 81 other early-career engineers at the National Academy of Engineering’s Grainger Foundation Frontiers of Engineering 2023 Symposium. She also received the Hans Fischer Fellowship from the Institute for Advance Study at the Technical University of Munich.

It was a big academic year for Associate Professor Elizabeth Cherry. Cherry was reelected to a three-year term as a council member-at-large of the Society of Industrial and Applied Mathematics (SIAM). Cherry is also co-chair of the SIAM organizing committee for next year’s Conference on Computational Science and Engineering (CSE25).

Cherry continues to serve as the School of CSE’s associate chair for academic affairs. These leadership contributions led to her being named to the 2024 ACC Academic Leaders Network (ACC ALN) Fellows program.

School of CSE Professor and Associate Chair Edmond Chow was co-author of a paper that received the Test of Time Award at Supercomputing 2023 (SC23). Right before SC23, Chow’s Ph.D. student Hua Huang was selected as an honorable mention for the 2023 ACM-IEEE CS George Michael Memorial HPC Fellowship.

Computing research at Georgia Tech is getting faster thanks to a new state-of-the-art processing chip named after a female computer programming pioneer.

Tech is one of the first research universities in the country to receive the GH200 Grace Hopper Superchip from NVIDIA for testing, study, and research.

Designed for large-scale artificial intelligence (AI) and high-performance computing applications, the GH200 is intended for large language model (LLM) training, recommender systems, graph neural networks, and other tasks. 

Alexey Tumanov and Tushar Krishna procured Georgia Tech’s first pair of Grace Hopper chips. Spencer Bryngelson attained four more GH200s, which will arrive later this month.

“We are excited about this new design that puts everything onto one chip and accessible to both processors,” said Will Powell, a College of Computing research technologist.

“The Superchip’s design increases computation efficiency where data doesn’t have to move as much and all the memory is on the chip.” 

A key feature of the new processing chip is that the central processing unit (CPU) and graphics processing unit (GPU) are on the same board.

NVIDIA’s NVLink Chip-2-Chip (C2C) interconnect joins the two units together. C2C delivers up to 900 gigabytes per second of total bandwidth, seven times faster than PCIe Gen5 connections used in newer accelerated systems.  

As a result, the two components share memory and process data with more speed and better power efficiency. This feature is one that the Georgia Tech researchers want to explore most.

Tumanov, an assistant professor in the School of Computer Science, and his Ph.D. student Amey Agrawal, are testing machine learning (ML) and LLM workloads on the chip. Their work with the GH200 could lead to more sustainable computing methods that keep up with the exponential growth of LLMs.

The advent of household LLMs, like ChatGPT and Gemini, pushes the limit of current architectures based on GPUs. The chip’s design overcomes known CPU-GPU bandwidth limitations. Tumanov’s group will put that design to the test through their studies. 

Krishna is an associate professor in the School of Electrical and Computer Engineering and associate director of the Center for Research into Novel Computing Hierarchies (CRNCH).

His research focuses on optimizing data movement in modern computing platforms, including AI/ML accelerator systems. Ph.D. student Hao Kang uses the GH200 to analyze LLMs exceeding 30 billion parameters. This study will enable labs to explore deep learning optimizations with the new chip.  

Bryngelson, an assistant professor in the School of Computational Science and Engineering, will use the chip to compute and simulate fluid and solid mechanics phenomena. His lab can use the CPU to reorder memory and perform disk writes while the GPU does parallel work. This capability is expected to significantly reduce the computational burden for some applications.

“Traditional CPU to GPU communication is slower and introduces latency issues because data passes back and forth over a PCIe bus,” Powell said. “Since they can access each other’s memory and share in one hop, the Superchip’s architecture boosts speed and efficiency.” 

Grace Hopper is the inspirational namesake for the chip. She pioneered many developments in computer science that formed the foundation of the field today.  

Hopper invented the first compiler, a program that translates computer source code into a target language. She also wrote the earliest programming languages, including COBOL, which is still used today in data processing. 

Hopper joined the U.S. Navy Reserve during World War II, tasked with programming the Mark I computer. She retired as a rear admiral in August 1986 after 42 years of military service.

Georgia Tech researchers hope to preserve Hopper’s legacy using the technology that bears her name and spirit for innovation to make new discoveries.

“NVIDIA and other vendors show no sign of slowing down refinement of this kind of design, so it is important that our students understand how to get the most out of this architecture,” said Powell. 

“Just having all these technologies isn’t enough. People must know how to build applications in their coding that actually benefit from these new architectures. That is the skill.”