We introduce SeaSplat, a method to enable real-time rendering of underwater scenes leveraging recent advances in 3D radiance fields. Underwater scenes are challenging visual environments, as rendering through a medium such as water introduces both range and color dependent effects on image capture. We constrain 3D Gaussian Splatting (3DGS), a recent advance in radiance fields enabling rapid training and real-time rendering of full 3D scenes, with a physically grounded underwater image formation model. Applying SeaSplat to the real-world scenes from SeaThru-NeRF dataset, a scene collected by an underwater vehicle in the US Virgin Islands, and simulation-degraded real-world scenes, not only do we see increased quantitative performance on rendering novel viewpoints from the scene with the medium present, but are also able to recover the underlying true color of the scene and restore renders to be without the presence of the intervening medium. We show that the underwater image formation helps learn scene structure, with better depth maps, as well as show that our improvements maintain the significant computational improvements afforded by leveraging a 3D Gaussian representation.

Click below for more details.

SeaSplat: Representing Underwater Scenes with 3D Gaussian Splatting and a Physically Grounded Image Formation Model

Website for paper SeaSplat: Representing Underwater Scenes with 3D Gaussian Splatting and a Physically Grounded Image Formation Model.

Daniel Yang

We are excited to announce a new workshop titled "Robots for Understanding Natural Ecosystems" that we're hosting at the upcoming ICRA 2024, in Yokohama, Japan on May 17th! The workshop invites roboticists who work in ecology or ecologists who work with robots to come and share their findings and discuss the future of these technologies and their role in ecology and conservation topics! https://warp.whoi.edu/rune

Successful applications of complex vision-based behaviours underwater have lagged behind progress in terrestrial and aerial domains. This is largely due to the degraded image quality resulting from the physical phenomena involved in underwater image formation. Spectrally-selective light attenuation drains some colors from underwater images while backscattering adds others, making it challenging to perform vision-based tasks underwater.  State-of-the-art methods for underwater color correction optimize the parameters of image formation models to restore the full spectrum of color to underwater imagery.  However, these methods have high computational complexity that is unfavourable for realtime use by autonomous underwater vehicles (AUVs), as a result of having been primarily designed for offline color correction.  Here, we present DeepSeeColor, a novel algorithm that combines a state-of-the-art underwater image formation model with the computational efficiency of deep learning frameworks.  In our experiments, we show that DeepSeeColor offers comparable performance to the popular "Sea-Thru" algorithm [Akkaynak et al. 2019] while being able to rapidly process images at up to 60Hz, thus making it suitable for use onboard AUVs as a preprocessing step to enable more robust vision-based behaviours.

Jamieson, Stewart, Jonathan P. How, and Yogesh Girdhar. “DeepSeeColor: Realtime Adaptive Color Correction for Autonomous Underwater Vehicles via Deep Learning Methods.” In IEEE International Conference on Robotics and Automation, 2023.

We are pleased to have the DeepSeeColor source code and data available at: https://github.com/warplab/DeepSeeColor

Coral reefs worldwide are threatened by anthropogenic disturbances and climate change. New tools are needed to scale up monitoring of coral reefs to understand reef ecosystems, rapidly assess biodiversity, and measure the efficacy of interventions. This interdisciplinary project will address this need by creating an autonomous robotic system that can navigate a complex ecosystem and intelligently sample its environment to estimate local biodiversity and ecosystem health.

CUREE is a robot designed to explore underwater ecosystems, observe complex interactions between the organisms that live there and their habitats, and use these observations in real-time to adapt its behavior as an intelligent partner for marine science.  As a compact system designed to be deployed and operated by teams as small as a single person, CUREE can be taken anywhere in the world in checked luggage on commercial airlines and deployed without a need for significant supporting infrastructure.  In experiments in the U.S. Virgin Islands, we demonstrated how CUREE can be used to study coral reefs, by combining audio and visual observations of a coral reef to infer the preferred habitat of snapping shrimp, or by tracking a barracuda as it hunts above a reef.

**Project Updates **

VMAT Dataset link

Supplementary videos link

In-situ visual observations of marine organisms is crucial to developing behavioural understandings and their relations to their surrounding ecosystem. Typically, these observations are collected via divers, tags, and remotely-operated or human-piloted vehicles. Recently, however, autonomous underwater vehicles equipped with cameras and embedded computers with GPU capabilities are being developed for a variety of applications, and in particular, can be used to supplement these existing data collection mechanisms where human operation or tags are more difficult. Existing approaches have focused on using fully-supervised tracking methods, but labelled data for many underwater species are severely lacking. Semi-supervised trackers may offer alternative tracking solutions because they require less data than fully-supervised counterparts. However, because there are not existing realistic underwater tracking datasets, the performance of semi-supervised tracking algorithms in the marine domain is not well understood. To better evaluate their performance and utility, in this paper we provide (1) a novel dataset specific to marine animals, (2) an evaluation of state-of-the-art semi-supervised algorithms in the context of underwater animal tracking, and (3) an evaluation of real-world performance through demonstrations using a semi-supervised algorithm on-board an autonomous underwater vehicle to track marine animals in the wild.

WARPLab is in St. Johns, USVI from July 25 - Aug 7 2022. The focus of this op is to test new WARPAUV capabilities including audio-visual benthic surveys of coral reefs and animal tracking in visually complex environments typically found in coral reefs.

We conducted robot field trials in USVI from Oct 17-Oct 30, 2021. The goal of the mission was to test our new robots, new robot behaviors, deploy acoustic sensors, and collect samples, all aimed at improving our capability to monitor coral reefs and understanding the impact of climate change. Our team is a diverse mix of  roboticists, biologists, and ecologists, all united over the common theme saving the reefs.

We presented a novel marine animal tracking dataset with evaluations of semi-supervised trackers as a poster, "Evaluation of Semi-supervised Methods for In-situ, Visual Tracking of Marine Animals", at the CV4Animals Workshop at CVPR 2021.

The workshop details are here: https://www.cv4animals.com/paper

The full poster is here: https://drive.google.com/file/d/1lXYhvzBBV5cjCabB9OGMUzece_GnE0Hp/view

Results of the trackers can be found here: https://drive.google.com/drive/folders/1Or2B4Yv6uwMgAFzvOHCA9giwTodXM977

We propose a generative model for the spatio-temporal distribution of high dimensional categorical observations. These are commonly produced by robots equipped with an imaging sensor such as a camera, paired with an image classifier, potentially producing observations over thousands of categories. The proposed approach combines the use of Dirichlet distributions to model sparse co-occurrence relations between the observed categories using a latent variable, and Gaussian processes to model the latent variable's spatio-temporal distribution. Experiments in this paper show that the resulting model is able to efficiently and accurately approximate the temporal distribution of high dimensional categorical measurements such as taxonomic observations of microscopic organisms in the ocean, even in unobserved (held out) locations, far from other samples. This work's primary motivation is to enable deployment of informative path planning techniques over high dimensional categorical fields, which until now have been limited to scalar or low dimensional vector observations.

Soucie, J. S., Sosik, H., & Girdhar, Y. (2020). Gaussian-Dirichlet Random Fields for Inference over High Dimensional Categorical Observations. [To appear in] International Conference on Robotics and Automation (ICRA).
[ArXiv Preprint]

We present a novel POMDP problem formulation for a robot that must autonomously decide where to go to collect new and scientifically relevant images given a limited ability to communicate with its human operator. From this formulation we derive constraints and design principles for the observation model, reward model, and communication strategy of such a robot, exploring techniques to deal with the very high-dimensional observation space and scarcity of relevant training data. We introduce a novel active reward learning strategy based on making queries to help the robot minimize path ``regret'' online, and evaluate it for suitability in autonomous visual exploration through simulations. We demonstrate that, in some bandwidth-limited environments, this novel regret-based criterion enables the robotic explorer to collect up to 17% more reward per mission than the next-best criterion.

Jamieson, S., How, J. P., & Girdhar, Y. (2020). Active Reward Learning for Co-Robotic Vision Based Exploration in Bandwidth Limited Environments. [To appear in] IEEE International Conference on Robotics and Automation.
[PDF]

We present Plume Localization under Uncertainty using Maximum-ValuE information and Search (PLUMES), a planner for localizing and collecting samples at the global maximum of an a priori unknown and partially observable continuous environment. This “maximum seek-and-sample” (MSS) problem is pervasive in the environmental and earth sciences. Experts want to collect scientifically valuable samples at an environmental maximum (e.g., an oil-spill source), but do not have prior knowledge about the phenomenon's distribution. We formulate the MSS problem as a partially-observable Markov decision process (POMDP) with continuous state and observation spaces, and a sparse reward signal. To solve the MSS POMDP, PLUMES uses an information-theoretic reward heuristic with continuous-observation Monte Carlo Tree Search to efficiently localize and sample from the global maximum. In simulation and field experiments, PLUMES collects more scientifically valuable samples than state-of-the-art planners in a diverse set of environments, with various platforms, sensors, and challenging real-world conditions.

https://doi.org/10.1109/LRA.2019.2929997

Bibtex:

@article{Flaspohler2019,
author = {Flaspohler, Genevieve and Preston, Victoria and Michel, Anna Pauline Miranda and Girdhar, Yogesh and Roy, Nicholas},
doi = {10.1109/LRA.2019.2929997},
issn = {2377-3766},
journal = {IEEE Robotics and Automation Letters},
month = {oct},
number = {4},
pages = {3782--3789},
title = {{Information-Guided Robotic Maximum Seek-and-Sample in Partially Observable Continuous Environments}},
url = {https://ieeexplore.ieee.org/document/8767964/},
volume = {4},
year = {2019}}

Abstract

This work proposes a bandwidth tunable technique for real-time probabilistic scene modeling and mapping to enable co-robotic exploration in communication constrained environments such as the deep sea. The parameters of the system enable the user to characterize the scene complexity represented by the map, which in turn determines the bandwidth requirements. The approach is demonstrated using an underwater robot that learns an unsupervised scene model of the environment and then uses this scene model to communicate the spatial distribution of various high-level semantic scene constructs to a human operator. Preliminary experiments in an artificially constructed tank environment, as well as simulated missions over a 10m×10m coral reef using real data, show the tunability of the maps to different bandwidth constraints and science interests. To our knowledge, this is the first paper to quantify how the free parameters of the unsupervised scene model impact both the scientific utility of and bandwidth required to communicate the resulting scene model.

https://doi.org/10.1109/ICRA.2019.8794132

https://arxiv.org/pdf/1903.03214.pdf

We just successfully completed our 2019 robot field trials at the Bellairs Research Institute 2019 Sea Trials in Holetown, Barbados. The main goals of the trials were to test our new ASV and AUV co-operating robot system and use them to demonstrate and evaluate our scene characterization and hotspot detection algorithms.

Here are a few select photos from the trip:

https://photos.app.goo.gl/f75Ss9WFzNzvMzNt8

Our paper on enabling distributed learning in bandwidth limited environments was one of the finalists
for the best paper award at IROS 2018 (6 finalists among 1,254 accepted).

Abstract:
Unsupervised learning techniques, such as Bayesian topic models, are capable of discovering latent structure directly from raw data. These unsupervised models can endow robots with the ability to learn from their observations without human supervision, and then use the learned models for tasks such as autonomous exploration, adaptive sampling, or surveillance. This paper extends single-robot topic models to the domain of multiple robots. The main difficulty of this extension lies in achieving and maintaining global consensus among the unsupervised models learned locally by each robot. This is especially challenging for multi-robot teams operating in communication-constrained environments, such as marine robots.

This paper presents a novel approach for multi-robot distributed learning in which each robot maintains a local topic model to categorize its observations and model parameters are shared to achieve global consensus. We apply a combinatorial optimization procedure that combines local robot topic distributions into a globally consistent model based on topic similarity, which we find mitigates topic drift when compared to a baseline approach that matches topics naively. We evaluate our methods experimentally by demonstrating multi-robot underwater terrain characterization using simulated missions on real seabed imagery. Our proposed method achieves similar model quality under bandwidth-constraints to that achieved by models that continuously communicate, despite requiring less than one percent of the data transmission needed for continuous communication.