Cardiologists and surgeons could soon have a new mobile augmented reality (AR) tool to improve collaboration in surgical planning.

ARCollab is an iOS AR application designed for doctors to interact with patient-specific 3D heart models in a shared environment. It is the first surgical planning tool that uses multi-user mobile AR in iOS.

The application’s collaborative feature overcomes limitations in traditional surgical modeling and planning methods. This offers patients better, personalized care from doctors who plan and collaborate with the tool.

Georgia Tech researchers partnered with Children’s Healthcare of Atlanta (CHOA) in ARCollab’s development. Pratham Mehta, a computer science major, led the group’s research.

“We have conducted two trips to CHOA for usability evaluations with cardiologists and surgeons. The overall feedback from ARCollab users has been positive,” Mehta said. 

“They all enjoyed experimenting with it and collaborating with other users. They also felt like it had the potential to be useful in surgical planning.”

ARCollab’s collaborative environment is the tool’s most novel feature. It allows surgical teams to study and plan together in a virtual workspace, regardless of location.

ARCollab supports a toolbox of features for doctors to inspect and interact with their patients' AR heart models. With a few finger gestures, users can scale and rotate, “slice” into the model, and modify a slicing plane to view omnidirectional cross-sections of the heart.

Developing ARCollab on iOS works twofold. This streamlines deployment and accessibility by making it available on the iOS App Store and Apple devices. Building ARCollab on Apple’s peer-to-peer network framework ensures the functionality of the AR components. It also lessens the learning curve, especially for experienced AR users.

ARCollab overcomes traditional surgical planning practices of using physical heart models. Producing physical models is time-consuming, resource-intensive, and irreversible compared to digital models. It is also difficult for surgical teams to plan together since they are limited to studying a single physical model.

Digital and AR modeling is growing as an alternative to physical models. CardiacAR is one such tool the group has already created. 

However, digital platforms lack multi-user features essential for surgical teams to collaborate during planning. ARCollab’s multi-user workspace progresses the technology’s potential as a mass replacement for physical modeling.

“Over the past year and a half, we have been working on incorporating collaboration into our prior work with CardiacAR,” Mehta said. 

“This involved completely changing the codebase, rebuilding the entire app and its features from the ground up in a newer AR framework that was better suited for collaboration and future development.”

Its interactive and visualization features, along with its novelty and innovation, led the Conference on Human Factors in Computing Systems (CHI 2024) to accept ARCollab for presentation. The conference occurs May 11-16 in Honolulu.

CHI is considered the most prestigious conference for human-computer interaction and one of the top-ranked conferences in computer science.

M.S. student Harsha Karanth and alumnus Alex Yang (CS 2022, M.S. CS 2023) co-authored the paper with Mehta. They study under Polo Chau, an associate professor in the School of Computational Science and Engineering.

The Georgia Tech group partnered with Timothy Slesnick and Fawwaz Shaw from CHOA on ARCollab’s development.

“Working with the doctors and having them test out versions of our application and give us feedback has been the most important part of the collaboration with CHOA,” Mehta said. 

“These medical professionals are experts in their field. We want to make sure to have features that they want and need, and that would make their job easier.”

From her home more than 800 miles away, Georgia Tech online master's student Jasmine Tata is monitoring fish in aquariums at Georgia Tech.

Tata is a New York-based QA analyst and project manager. She started the Online Master of Science in Computer Science (OMSCS) program in Fall 2022 and joined FishStalkers last year.

The student-led research program is part of the School of Biological Sciences' McGrath Lab. Its researchers use machine learning, computer vision, and other technologies to better understand the evolution of animal behaviors.

One of the lab's research projects studies Lake Malawi cichlids to explore connections between observed behavior and brain function.

The FishStalkers are vital to the project. They collect video, depth, and other data from individual fish using Raspberry Pi single-board computers. This information, coupled with open-source code they developed, allows the group to track, monitor, and classify the behaviors of a fish as it builds and maintains its bower, which is a sand structure these cichlids use to attract mates.

Along with monitoring the research tanks, Tata's contributions include improving the automated collection and analysis of data streaming from the Pis. She's also helping to adapt the data pipeline to work with yellow-head, orange-cap, and other cichlid species.

[RELATED: Georgia Tech's OMSCS Program Celebrates 10th Anniversary]

"I've enjoyed learning more about new problems in a relatively unfamiliar field. In a pure computer science-focused lab, I never would experience the frustrations of data collection that come with biological subjects," said Tata.

"The fish builds bowers on its own schedule, and data collection must accurately capture this, regardless of weekends or holidays."

Tata says her experience with FishStalkers has given her new ideas about presenting data to non-technical team members. The team uses a spreadsheet integrated with data collection scripts running on the Raspberry Pis. The spreadsheet allows someone without technical knowledge to pause, upload data, or start new trials simply by toggling a dropdown.

"This has given me a lot of ideas about how to meet people where they are in terms of technical skills when it comes to user interface design and has encouraged me to learn more about human-computer interaction," said Tata.

Tata learned about the FishStalkers research group when its founder, Breanna Shi, reached out through the OMSCS Slack study channel. Shi developed the group through Georgia Tech's Vertically Integrated Projects (VIP) program as a mentorship program.

"Given their real-world computer science experience, I wanted to see if there were OMSCS students interested in collaborating on FishStalkers projects and assisting in the mentorship of undergraduate researchers," said Shi. 

Shi is a third-year Ph.D. student studying bioinformatics with minors in machine learning and higher education. She created FishStalkers as a mentorship program because she recognized that undergraduate and masters-level students could feel less valued or isolated in research environments.

"The FishStalkers model empowers all its researchers with the respect and responsibility as a full team member. Whether it's your first week as a FishStalker or your last, you will complete tasks that benefit the research team and yourself," said Shi.

[RELATED: Women-Centered Mentorship Provides Empowerment to Conquer Ph.D.]

Tata's experience in the business world made her a good fit for the FishStalkers program. Shi says Tata contributes valuable insight to the group as a mentor because most students approach the program from a purely academic viewpoint.

"Jasmine, like other OMSCS students, works full-time and attends the OMSCS program part-time. Her roles as a project manager and a software QA analyst allow her to contribute a unique perspective to the FishStalkers group," said Shi.

In addition to sharing her experience mentoring two OMSCS students this semester, Tata has helped Shi overcome some of the inherent challenges of long-distance collaboration. These include creating a sense of interpersonal connection among in-person and remote research team members.

Group meetings host a virtual link to enhance the online research experience. Every member provides progress updates during the sessions. The researchers also virtually check in and out of their research hours in a shared group chat and describe the work completed during their check-out.

"FishStalkers also runs a monthly lab-buddy program where a researcher is paired with a new buddy each month to schedule a 30-minute meeting to chat and learn about each other's work," said Shi.

"These strategies benefit OMSCS students in our group and provide a positive research environment for junior researchers. We seek to incorporate innovative strategies to create an accessible research environment for all students interested in participating in our research," said Shi.

FishStalkers has been such a success that Shi is expanding the model. This fall, Shi will work with OMSCS Executive Director David Joyner and OMSCS Associate Director of Research Nick Lytle to connect OMSCS students with interdisciplinary research projects in labs across campus.

"My role will be to establish relationships between data collectors and data analyzers to provide a service to non-technical labs across campus and a valuable research experience for OMSCS students," said Shi.

"We will be building from my existing work in image processing in the McGrath Lab and expanding to other labs with data analysis needs. I am very excited to have the experience of growing as a collaborator."