ROST can automatically characterize large image datasets, identify various scene components and detect context aware anomalies. These automatically annotated datasets can then be visualized using various techniques.

Interactive mission visualizations

Here are some visualizations of the image data streams collected by the SeaBED AUV, during a missions to explore Hannibal Seamount in Panama.

Following example shows results of automatic unsupervised characterization of substrate types using spatiotemporal topic modeling of a image data, collected in a transect by an AUV off the west coast of Panama (depths ca. 200 - 400 m). The plot shows the distribution of topics at each timestep. Different colors in the timeline correspond to different topic labels.

Anomaly detection

Using proposed topic modeling on a crab dataset, off the west coast of Panama. (a) A stacked plot showing distribution of topic labels at each time step in the dataset, computed using the proposed technique. We see that Topic 0 and 1 are characteristic of the underlying terrain, whereas other topic labels are representative of other phenomena such as galatheid crab aggregations (topic 4). (b) Shows the normalized perplexity scores for each time step, with the highest peak at t=116. (c) Shows examples of images with high perplexity scores, corresponding to anomalous observations. Image with $t=116$ shows various animal species feeding on a fish caracas, and is the most anomalous scene in the dataset. Other anomalous observations are of galatheid crabs. (d) Shows examples of some typical images in the dataset, represented by their low perplexity scores

Substrate Characterization

In this example we see four different substrate types being represented distinctly by the topic model, without any prior training.


  • Y. Girdhar, D. Whitney, and G. Dudek, “Curiosity based exploration for learning terrain models,” in 2014 IEEE International Conference on Robotics and Automation (ICRA), 2014, pp. 578–584.
  • Y. Girdhar, W. Cho, M. Campbell, J. Pineda, E. Clarke, and H. Singh, “Anomaly Detection in Unstructured Environments using Bayesian Nonparametric Scene Modeling,” in IEEE International Conference on Robotics and Automation (ICRA) (accepted), 2016.
  • A. Kalmbach, M. Hoeberechts, A. B. Albu, H. Glotin, S. Paris, and Y. Girdhar, “Learning Deep-Sea Substrate Types With Visual Topic Models,” in IEEE Winter Conference on Applications of Computer Vision (accepted), 2016.