Monterey Bay Aquarium Research Institute app gets gamers to help explore the ocean

How well do you know your ocean creatures? Can you identify a sea anemone? How about a fanfin anglerfish, a sea gooseberry, a red spotted siphonophore, a flapjack octopus or a hagfish?

Marine researchers have collected photos and video of ocean life for decades. But to get useful information from those thousands of terabytes of photos and videos, someone’s got to document what’s in every single one. In the past, expert scientists have carried that burden, spending long hours in dark rooms tagging turtles, tunas and tiger sharks one photo at a time.

But now, the Moss Landing-based Monterey Bay Aquarium Research Institute is letting artificial intelligence — and gamers — take the reins. This summer, MBARI launched a beta version of its new game app, FathomVerse. The game drops users into an aquatic world with tasks and missions to complete. For example you might need to collect 16 squids. “As you’re navigating around in the ocean, you’re going to come up on different orbs, and as you flash your light on them they emerge as animals,” explained Kakani Katija, the project’s principal engineer. “You get to make this decision — ‘Is this an animal that I’m searching for? Or is this animal just cool, and I want to learn more about it?’”

As users learn more they get better at classifying animals. And as they classify animals, they’re actually training machine learning algorithms. Eventually, Katija said, those algorithms will help scientists sort through the massive backlogs of aquatic visual data. She said knowing what creatures live in the ocean and where they are will speed up research and help to inform activities — like ocean exploration, offshore wind, deep sea mining and aquaculture — that need to understand and monitor their environmental and biological impacts.

“This project could be a game-changer in how marine life is monitored,” said Claudie Beaulieu, an assistant professor in the Ocean Sciences Department at UC Santa Cruz who is not involved in the project. “Visual data in the ocean takes a lot of time and effort to process (and is) usually labeled manually by experts. With these images coming in fast, it’s impossible to keep up the labeling and processing of these big data. Using AI to process (it) is so much faster, and means a lot of data is available for research and monitoring.”

Ocean Vision AI – a three-armed project

FathomVerse is just one of three arms of a larger project called Ocean Vision AI, a program that aims to use artificial intelligence to make visual data — and the tools to  analyze it — more accessible to researchers.

Ocean Vision AI is one year in to a $5 million grant from the National Science Foundation’s Convergence Accelerator program, which funds projects to “solve societal challenges.”

Its three arms include: FathomNet, a repository for large data sets of ocean images,  “the portal,” an AI-based tool for processing those large data sets, and the FathomVerse game app. Each product is targeted to a different audience, with the overarching goal of connecting people who care about the ocean.

“We’re trying to offer three different products, which capture interactions and offer services on this spectrum of experts to enthusiasts,” said Kevin Barnard, a software engineer at MBARI and co-lead on the program. “FathomNet is certainly on the expert side, the portal sort of sits in the middle, and then the game, FathomVerse, lands on the enthusiast side.”

“The ocean research community doesn’t have a central place like this yet,” said Lilli Carlsen, the project’s engagement coordinator. “We’re creating a space that brings together experts from machine learning and also the ocean research community to work together on this huge challenge of studying the ocean and of creating a better understanding of what’s out there so that we can inform responsible decision making and ocean stewardship.”

FathomNet and the Ocean Vision AI portal

FathomNet, the repository arm of the project, started as a place to gather big ocean data.

“The ocean this is this massive place — you’ve got at least 200,000 animals that have been described as marine species,” Katija said, “Plus whatever we haven’t found yet. As an institution, we’ve had kind of a 30-year head start when it comes to managing visual data — annotating it, organizing it, creating databases around this information — and so we see this as a way to give back to the community, to make our data more accessible, but also hopefully kickstart all of these important collaborations in this space.”

Most of the data currently in the repository is from MBARI, but the team hopes to gain participation from others over time.

Megan Cromwell, assistant chief data officer for NOAA’s National Ocean Service, has contributed. She worked on three major NOAA projects that produced massive amounts of visual data. She needed to figure out how to get those photos and videos — mostly recorded on hard drives now, but previously on physical media like mini-DVs and DVDs — to researchers in a useful way.

“Artificial intelligence and machine learning is the solution,” she said. “The entire ocean image community needs this for one reason or another, whether it be for the analysis and quantification, or whether it be, in my case, to make the data available, where people can find it and where it’s easy to use.”

“I get really passionate about it because FathomNet and Ocean Vision AI and all the work that they’re doing is a game changer for the entire community,” she said.

The team released FathomNet 1.0 this summer and is now turning their focus to the portal — the AI that can sort through massive amounts of data quickly, and accurately tag the creatures lurking in the photos.

FathomVerse

While FathomNet and the portal are targeted to experts, the team also wants to draw from the expertise of the public. “There’s this huge community of people who aren’t academic experts, but who have this wealth of knowledge and also a wealth of enthusiasm and motivation to help us,” Carlsen said.

FathomVerse, developed in partnership with the Kenya-based Internet of Elephants and Netherlands-based &ranj game development companies, is intended to tap into that rich resource and expand inclusion in ocean research.

Katija wants to, “break down barriers for the public to understand what’s it like to work in ocean science. Through (users’) contributions, we can not only improve the AI that’s used to identify all the life of the ocean, but potentially link them with real live expeditions and people who are doing the work.”

Beta testing for FathomVerse is now closed to the public, but the team hopes to publicly release the full game in the first quarter of 2024. Sign up to get updates at: www.fathomverse.game

If all goes really well, Katija thinks the project could serve as an example for visually exploring life in other places, like space, too. “One day we can have robots just going out and and searching for life that we know nothing about,” she said. We could send robots on missions, and if they encountered an animal, they’d check through the library of life they’d been trained on. “If it doesn’t match anything in the library, then [they’d flag] us onshore to let us know that something is different or something is new,” she said. “That sounds very sci-fi, but we’re not that far away from that, frankly.”