From weather prediction to drug discovery, math powers the models used in computer simulations. To help these vital tools with their calculations, global experts recently met at Georgia Tech to share ways to make math easier for computers.

Tech hosted the 2024 International Conference on Preconditioning Techniques for Scientific and Industrial Applications (Precond 24), June 10-12. 

Preconditioning accelerates matrix computations, a kind of math used in most large-scale models. These computer models become faster, more efficient, and more accessible with help from preconditioned equations.

“Preconditioning transforms complex numerical problems into more easily solved ones,” said Edmond Chow, a professor at Georgia Tech and co-chair of Precond 24’s local organization and program committees. 

“The new problem wields a better condition number, giving rise to the name preconditioning.”

Researchers from 13 countries presented their work through 20 mini-symposia and seven invited talks at Precond 24. Their work showcased the practicality of preconditioners. 

Vandana Dwarka, an assistant professor at Delft University of Technology, shared newly developed preconditioners for electromagnetic simulations. This technology can be used in further applications ranging from imaging to designing nuclear fusion devices.

Xiaozhe Hu presented a physics-based preconditioner that simulates biophysical processes in the brain, such as blood flow and metabolic waste clearance. Hu brought this research from Tufts University, where he is an associate professor.

Tucker Hartland, a postdoctoral researcher at Lawrence Livermore National Laboratory, discussed preconditioning in contact mechanics. This work improves the modeling of interactions between physical objects that touch each other. Many fields stand to benefit from Hartland’s study, including mechanical engineering, civil engineering, and materials science.

A unique aspect of this year’s conference was an emphasis on machine learning (ML). Between a panel discussion, tutorial, and several talks, experts detailed how to employ ML for preconditioning and how preconditioning can train ML models.

Precond 24 invited seven speakers from institutions around the world to share their research with conference attendees. The presenters were: 

• Monica Dessole, CERN, Switzerland
• Selime Gurol, CERFACS, France
• Alexander Heinlein, Delft University of Technology, Netherlands
• Rui Peng Li, Lawrence Livermore National Laboratory, USA 
• Will Pazner, Portland State University, USA 
• Tyrone Rees, Science and Technology Facilities Council, UK
• Jacob B. Schroder, University of New Mexico, USA

Along with hosting Precond 24, several Georgia Tech researchers participated in the conference through presentations. 

Ph.D. students Hua Huang and Shikhar Shah each presented a paper on the conference’s first day. Alumnus Srinivas Eswar (Ph.D. CS 2022) returned to Atlanta to share research from his current role at Argonne National Laboratory. Chow chaired the ML panel and a symposium on preconditioners for matrices.

“It was an engaging and rewarding experience meeting so many people from this very tight-knit community,” said Shah, who studies computational science and engineering (CSE). “Getting to see talks close to my research provided me with a lot of inspiration and direction for future work.”

Precond 2024 was the thirteenth meeting of the conference, which occurs every two years. 

The conference returned to Atlanta this year for the first time since 2005. Atlanta joins Minneapolis as one of only two cities in the world to host Precond more than once. Precond 24 marked the sixth time the conference met in the U.S. 

Georgia Tech and Emory University’s Department of Mathematics organized and sponsored Precond 24. The U.S. Department of Energy Office of Science co-sponsored the conference with Tech and Emory. 

Georgia Tech entities swarmed together in support of Precond 24. The Office of the Associate Vice President for Research Operations and Infrastructure, College of Computing, and School of CSE co-sponsored the conference.

“The enthusiasm at the conference has been very gratifying. So many people organized sessions at the conference and contributed to the very strong attendance,” Chow said. 

“This is a testament to the continued importance of preconditioning and related numerical methods in a rapidly changing technological world.”

Artificial intelligence and machine learning techniques are infused across the College of Engineering’s education and research.

From safer roads to new fuel cell technology, semiconductor designs to restoring bodily functions, Georgia Tech engineers are capitalizing on the power of AI to quickly make predictions or see danger ahead.

Explore some of the ways we are using AI to create a better future on the College's website.

This story was featured in the spring 2024 issue of Helluva Engineer magazine, produced biannually by the College of Engineering.

It’s tempting to think that the artificial intelligence revolution is coming — for good or ill — and that AI will soon be baked into every facet of our lives. With generative AI tools suddenly available to anyone and seemingly every company scrambling to leverage AI for their business, it can feel like the AI-dominated future is just over the horizon.

The truth is, that future is already here. Most of us just didn’t notice.

Every time you unlock your smartphone or computer with a face scan or fingerprint. Every time your car alerts you that you’re straying from your lane or automatically adjusts your cruise control speed. Every time you ask Siri for directions or Alexa to turn on some music. Every time you start typing in the Google search box and suggestions or the outright answer to your question appear. Every time Netflix recommends what you should watch next.

All driven by AI. And all a regular part of most people’s days.

But what is “artificial intelligence”? What about “machine learning” and “algorithms”? How are they different and how do they work?

We asked two of the many Georgia Tech engineers working in these areas to help us understand the basic concepts so we’re all better prepared for the AI future — er, present.

Read the full crash course on the College of Engineering website.

This story was featured in the spring 2024 issue of Helluva Engineer magazine, produced biannually by the College of Engineering.

A team’s success in any competitive environment often hinges on how well each member can anticipate the actions of their teammates.

Assistant Professor Christopher MacLellan thinks teachable artificial intelligence (AI) agents are uniquely suited for this role and make ideal teammates for video gamers.

With the help of funding from the U.S. Department of Defense, MacLellan hopes to prove his theory with a conversational, task-performing agent he co-engineered called the Verbal Apprentice Learner (VAL).

“You need the ability to adapt to what your teammates are doing to be an effective teammate,” MacLellan said. “We’re exploring this capability for AI agents in the context of video games.” 

Unlike generative AI chatbots like ChatGPT, VAL uses an interactive task-learning approach. 

“VAL learns how you do things in the way you want them done,” MacLellan said. “When you tell it to do something, it will do it the way you taught it instead of some generic random way from the internet.”

A key difference between VAL and a chatbot is that VAL can perceive and act within the gaming world. A chatbot, like ChatGPT, only perceives and acts within the chat dialog.

MacLellan immersed VAL into an open-sourced, simplified version of the popular Nintendo cooperative video game Overcooked to discover how well the agent can function as a teammate. In Overcooked, up to four players work together to prepare dishes in a kitchen while earning points for every completed order.

How Fast Can Val Learn?

In a study with 12 participants, MacLellan found that users could often correctly teach VAL new tasks with only a few examples.

First, the user must teach VAL how to play the game. Knowing that a single human error could compromise results, MacLellan designed three precautionary features:

• When VAL receives a command such as "cook an onion," it asks clarifying questions to understand and confirm its task. As VAL continues to learn, clarification prompts decrease.
• An “undo” button to ensure users can reverse an errant command.
• VAL contains GPT subcomponents to interpret user input, allowing it to adapt to ambiguous commands and typos. The GPT subcomponents drive changes in VAL’s task knowledge, which it uses to perform tasks without additional guidance.

The participants in MacLellan’s study used these features to ensure VAL learned the tasks correctly. 

The high volume of prompts creates a more tedious experience. Still, MacLellan said it provides detailed data on system performance and user experience. That insight should make designing a more seamless experience in future versions of VAL possible.

The prompts also require the AI to be explainable.

“When VAL learns something, it uses the language model to label each node in the task knowledge graph that the system constructs,” MacLellan said. “You can see what it learned and how it breaks tasks down into actions.”

Beyond Gaming

MacLellan’s Teachable AI Lab is devoted to developing AI that inexperienced users can train.

“We are trying to come up with a more usable system where anyone, including people with limited expertise, could come in and interact with the agent and be able to teach it within just five minutes of interacting with it for the first time,” he said.

His work caught the attention of the Department of Defense, which awarded MacLellan multiple grants to fund several of his projects, including VAL. The possibilities of how the DoD could use VAL, on and off the battlefield, are innumerable.

“(The DoD) envisions a future in which people and AI agents jointly work together to solve problems,” MacLellan said. “You need the ability to adapt to what your teammates are doing to be an effective teammate.

“We look at the dynamics of different teaming circumstances and consider what are the right ways to team AI agents with people. The key hypothesis for our project is agents that can learn on the fly and adapt to their users will make better teammates than those that are pre-trained like GPT.”

Design Your Own Agent

MacLellan is co-organizing a gaming agent design competition sponsored by the Institute of Electrical and Electronic Engineers (IEEE) 2024 Conference on Games in Milan, Italy.

The Dice Adventure Competition invites participants to design their own AI agent to play a multi-player, turn-based dungeon crawling game or to play the game as a human teammate. The competition this month and in July offers $1,000 in prizes for players and agent developers in the top three teams.

Researchers at Georgia Tech are creating accessible museum exhibits that explain artificial intelligence (AI) to middle school students, including the LuminAI interactive AI-based dance partner developed by Regents' Professor Brian Magerko.

Ph.D. students Yasmine Belghith and Atefeh Mahdavi co-led a study in a museum setting that observed how middle schoolers interact with the popular AI chatbot ChatGPT. 

“It’s important for museums, especially science museums, to start incorporating these kinds of exhibits about AI and about using AI so the general population can have that avenue to interact with it and transfer that knowledge to everyday tools,” Belghith said.

Belghith and Mahdavi conducted their study with nine focus groups of 24 students at Chicago’s Museum of Science and Industry. The team used the findings to inform their design of AI exhibits that the museum could display as early as 2025. 

Belghith is a Ph.D. student in human-centered computing. Her advisor is Assistant Professor Jessica Roberts in the School of Interactive Computing. Magerko advises Mahdavi, a Ph.D. student in digital media in the School of Literature, Media, and Communication.

Belghith and Mahdavi presented a paper about their study in May at the Association for Computing Machinery (ACM) 2024 Conference on Human Factors in Computing Systems (CHI) in Honolulu, Hawaii.

Their work is part of a National Science Foundation (NSF) grant dedicated to fostering AI literacy among middle schoolers in informal environments.

Expanding Accessibility

While there are existing efforts to reach students in the classroom, the researchers believe AI education is most accessible in informal learning environments like museums.

“There’s a need today for everybody to have some sort of AI literacy,” Belghith said. “Many middle schoolers will not be taking computer science courses or pursuing computer science careers, so there needs to be interventions to teach them what they should know about AI.”

The researchers found that most of the middle schoolers interacted with ChatGPT to either test its knowledge by prompting it to answer questions or socialize with it by having human-like conversations. 

Others fit the mold of “content explorers.” They did not engage with the AI aspect of ChatGPT and focused more on the content it produced.

Mahdavi said regardless of their approach, students would get “tunnel vision” in their interactions instead of exploring more of the AI’s capabilities.

“If they go in a certain direction, they will continue to explore that,” Mahdavi said. “One thing we can learn from this is to nudge kids and show them there are other things you can do with AI tools or get them to think about it another way.”

The researchers also paid attention to what was missing in the students’ responses, which Mahdavi said was just as important as what they did talk about.

“None of them mentioned anything about ethics or what could be problematic about AI,” she said. “That told us there’s something they aren’t thinking about but should be. We take that into account as we think about future exhibits.”

Making an Impact

The researchers visited the Museum of Science and Industry June 1-2 to conduct the first trial run of three AI-based exhibits they’ve created. One of them is LuminAI, which was developed in Magerko’s Expressive Machinery Lab.

LuminAI is an interactive art installation that allows people to engage in collaborative movement with an AI dance partner. Georgia Tech and Kennesaw State recently held the first performance of AI avatars dancing with human partners in front of a live audience.

Duri Long, a former Georgia Tech Ph.D. student who is now an assistant professor at Northwestern University, designed the second exhibit. KnowledgeNet is an interactive tabletop exhibit in which visitors build semantic networks by adding different characteristics to characters that interact together.

The third exhibit, Data Bites, prompts users to build datasets of pizzas and sandwiches. Their selections train a machine-learning classifier in real time.

Belghith said the exhibits fostered conversations about AI between parents and children.

“The exhibit prototypes successfully engaged children in creative activities,” she said. “Many parents had to pull their kids away to continue their museum tour because the kids wanted more time to try different creations or dance moves.”

Naiya Salinas and her instructor, Deryk Stoops, looked back and forth between the large screen on the wall and a hand-held monitor.

Tracing between the lines of code, Salinas made a discovery: A character was missing.

The lesson was an important, real-world example of the problem-solving skills required when working in robotics. Salinas is one of a half-dozen students enrolled in the new AI Enhanced Robotic Manufacturing program at the Georgia Veterans Education Career Transition Resource (VECTR) Center, which is setting a new standard for technology-focused careers.

The set-up of the lab was intentional, said Stoops, who designed the course modules and worked with local industry to determine their manufacturing needs. Then, with funding from the Georgia Tech Manufacturing Institute's (GTMI) Georgia Artificial Intelligence in Manufacturing (Georgia AIM) project, Stoops worked with administrators at Central Georgia Technical College to purchase robotics and other cutting-edge manufacturing tools.

As a result, the VECTR Center’s AI-Enhanced Robotic Manufacturing Studio trains veterans in industry-standard robotics, manufacturing modules, cameras, and network systems. This equipment gives students experience in a variety of robotics-based manufacturing applications. Graduates can also finish the 17-credit course with two certifications that carry some weight in the manufacturing world.

“After getting the Georgia AIM grant, we pulled together a roundtable with industry. And then we did site visits to see how they pulled AI and robotics into the space,” said Stoops. “All the equipment in here is the direct result of industry feedback.”

Statewide Strategic Effort

Funded by a $65 million grant from the federal Economic Development Administration, Georgia AIM is a network of projects across the state born out of GTMI and led by Georgia Tech’s Enterprise Innovation Institute. These projects work to connect the manufacturing community with smart technologies and a ready workforce. Central Georgia received around $4 million as part of the initiative to advance innovation, workforce development and STEM education in support of local manufacturing and Robins Air Force Base.

Georgia AIM pulls together a host of regional partners all working toward a common goal of increasing STEM education, access to technology and enhancing AI among local manufacturers. This partnership includes Fort Valley State University, the Middle Georgia Innovation Project led by the Development Authority of Houston County, Central Georgia Technical College, which administers the VECTR Center, and the 21st Century Partnership.

“This grant will help us turn our vision for both the Middle Georgia Innovation Project and the Middle Georgia STEM Alliance, along with our partners, into reality, advancing this region and supporting the future of Robins AFB,” said Brig. Gen. John Kubinec, USAF (ret.), president and chief executive officer of the 21st Century Partnership.

Georgia AIM funding for Central Georgia Technical College and Fort Valley State focused on enhancing technology and purchasing new components to assist in education. At Fort Valley State, a mobile lab will launch later this year to take AI-enhanced technologies to underserved parts of the state, while Central Georgia Tech invested in an AI-enhanced robotics manufacturing lab at the VECTR Center.

“This funding will help bring emerging technology throughout our service area and beyond, to our students, economy, and Robins Air Force Base,” said Dr. Ivan Allen, president of Central Georgia Technical College. “Thanks to the power of this partnership, our faculty and students will have the opportunity to work directly with modern manufacturing technology, giving our students the experience and education needed to transition from the classroom to the workforce in an in-demand industry.”

New Gateway for Vets

The VECTR Center’s AI-Enhanced Robotics Manufacturing Studio includes FANUC robotic systems, Rockwell Automation programmable logic controllers, Cognex AI-enabled machine vision systems, smart sensor networks, and a MiR autonomous mobile robot.

The studio graduated its first cohort of students in February and celebrated its ribbon-cutting ceremony on April 17 with a host of local officials and dignitaries. It was also an opportunity to celebrate the students, who are transitioning from a military career to civilian life.

The new technologies at the VECTR Center lab are opening new doors to a growing, cutting-edge field.

“From being in this class, you really start to see how the world is going toward AI. Not just Chat GPT, but everything — the world is going toward AI for sure now,” said Jordan Leonard, who worked in logistics and as a vehicle mechanic in the U.S. Army. Now, he’s upskilling into robotics and looking forward to using his new skills in maintenance. “What I want to do is go to school for instrumentation and electrical technician. But since a lot of industrial plants are trying to get more robots, for me this will be a step up from my coworkers by knowing these things.”

Whether it’s typing an email or guiding travel from one destination to the next, artificial intelligence (AI) already plays a role in simplifying daily tasks.

But what if it could also help people live more efficiently — that is, more sustainably, with less waste?

It’s a concept that often runs through the mind of Iesha Baldwin, the inaugural Georgia AIM Fellow with the Partnership for Inclusive Innovation (PIN) at the Georgia Institute of Technology’s Enterprise Innovation Institute. Born out of the Georgia Tech Manufacturing Institute, the Georgia AIM (Artificial Intelligence in Manufacturing) project works with PIN fellows to advance the project's mission of equitably developing and deploying talent and innovation in AI for manufacturing throughout the state of Georgia.

When she accepted the PIN Fellowship for 2023, she saw an opportunity to learn more about the nexus of artificial intelligence, manufacturing, waste, and education. With a background in environmental studies and science, Baldwin studied methods for waste reduction, environmental protection, and science education.

“I took an interest in AI technology because I wanted to learn how it can be harnessed to solve the waste problem and create better science education opportunities for K-12 and higher education students,” said Baldwin.

This type of unique problem-solving is what defines the PIN Fellowship programs. Every year, a cohort of recent college graduates is selected, and each is paired with an industry that aligns with their expertise and career goals — specifically, cleantech, AI manufacturing, supply chain and logistics, and cybersecurity/information technology. Fellowships are one year, with fellows spending six months with a private company and then six months with a public organization.

Through the experience, fellows expand their professional network and drive connections between the public and private sectors. They also use the opportunity to work on special projects that involve using new technologies in their area of interest.

With a focus on artificial intelligence in manufacturing, Baldwin led an inventory management project at the Georgia manufacturer Freudenberg-NOK, where the objective was to create an inventory management system that reduced manufacturing downtime and, as a result, increased efficiency, and reduced waste.

She also worked in several capacities at Georgia Tech: supporting K-12 outreach programs at the Advanced Manufacturing Pilot Facility, assisting with energy research at the Marcus Nanotechnology Research Center, and auditing the infamous mechanical engineering course ME2110 to improve her design thinking and engineering skills.

“Learning about artificial intelligence is a process, and the knowledge gained was worth the academic adventure,” she said. “Because of the wonderful support at Georgia Tech, Freudenberg NOK, PIN, and Georgia AIM, I feel confident about connecting environmental sustainability and technology in a way that makes communities more resilient and sustainable.”

Since leaving the PIN Fellowship, Baldwin connected her love for education, science, and environmental sustainability through her new role as the inaugural sustainability coordinator for Spelman College, her alma mater.  In this role, she is responsible for supporting campus sustainability initiatives.

Ask a person to find a frying pan, and they will most likely go to the kitchen. Ask a robot to do the same, and you may get numerous responses, depending on how the robot is trained.

Since humans often associate objects in a home with the room they are in, Naoki Yokoyama thinks robots that navigate human environments to perform assistive tasks should mimic that reasoning.

Roboticists have employed natural language models to help robots mimic human reasoning over the past few years. However, Yokoyama, a Ph.D. student in robotics, said these models create a “bottleneck” that prevents agents from picking up on visual cues such as room type, size, décor, and lighting. 

Yokoyama presented a new framework for semantic reasoning at the Institute of Electrical and Electronic Engineers (IEEE) International Conference on Robotics and Automation (ICRA) last month in Yokohama, Japan. ICRA is the world’s largest robotics conference.

Yokoyama earned a best paper award in the Cognitive Robotics category with his Vision-Language Frontier Maps (VLFM) proposal.

Assistant Professor Sehoon Ha and Associate Professor Dhruv Batra from the School of Interactive Computing advised Yokoyama on the paper. Yokoyama authored the paper while interning at the Boston Dynamics’ AI Institute.

“I think the cognitive robotic category represents a significant portion of submissions to ICRA nowadays,” said Yokoyama, whose family is from Japan. “I’m grateful that our work is being recognized among the best in this field.”

Instead of natural language models, Yokoyama used a renowned vision-language model called BLIP-2 and tested it on a Boston Dynamics “Spot” robot in home and office environments.

“We rely on models that have been trained on vast amounts of data collected from the web,” Yokoyama said. “That allows us to use models with common sense reasoning and world knowledge. It’s not limited to a typical robot learning environment.”

What is Blip-2?

BLIP-2 matches images to text by assigning a score that evaluates how well the user input text describes the content of an image. The model removes the need for the robot to use object detectors and language models. 

Instead, the robot uses BLIP-2 to extract semantic values from RGB images with a text prompt that includes the target object. 

BLIP-2 then teaches the robot to recognize the room type, distinguishing the living room from the bathroom and the kitchen. The robot learns to associate certain objects with specific rooms where it will likely find them.

From here, the robot creates a value map to determine the most likely locations for a target object, Yokoyama said.

Yokoyama said this is a step forward for intelligent home assistive robots, enabling users to find objects — like missing keys — in their homes without knowing an item’s location. 

“If you’re looking for a pair of scissors, the robot can automatically figure out it should head to the kitchen or the office,” he said. “Even if the scissors are in an unusual place, it uses semantic reasoning to work through each room from most probable location to least likely.”

He added that the benefit of using a VLM instead of an object detector is that the robot will include visual cues in its reasoning.

“You can look at a room in an apartment, and there are so many things an object detector wouldn’t tell you about that room that would be informative,” he said. “You don’t want to limit yourself to a textual description or a list of object classes because you’re missing many semantic visual cues.”

While other VLMs exist, Yokoyama chose BLIP-2 because the model:

• Accepts any text length and isn’t limited to a small set of objects or categories.
• Allows the robot to be pre-trained on vast amounts of data collected from the internet.
• Has proven results that enable accurate image-to-text matching.

Home, Office, and Beyond

Yokoyama also tested the Spot robot to navigate a more challenging office environment. Office spaces tend to be more homogenous and harder to distinguish from one another than rooms in a home. 

“We showed a few cases in which the robot will still work,” Yokoyama said. “We tell it to find a microwave, and it searches for the kitchen. We tell it to find a potted plant, and it moves toward an area with windows because, based on what it knows from BLIP-2, that’s the most likely place to find the plant.”

Yokoyama said as VLM models continue to improve, so will robot navigation. The increase in the number of VLM models has caused robot navigation to steer away from traditional physical simulations.

“It shows how important it is to keep an eye on the work being done in computer vision and natural language processing for getting robots to perform tasks more efficiently,” he said. “The current research direction in robot learning is moving toward more intelligent and higher-level reasoning. These foundation models are going to play a key role in that.”

Top photo by Kevin Beasley/College of Computing.