DeepScene

Semantic Segmentation using Deep Convolutional Neural Networks

DeepScene contains our unimodal AdapNet++ and multimodal SSMA models trained on various datasets. Select a dataset and a corresponding model to load from the drop down box below, and click on Random Example to see the live segmentation results. The code for these models is available on our Github repository(https://github.com/DeepSceneSeg).


Input Image


Segmentation Output



  • Building

  • Bicycle

  • Fence

  • Unlabeled

  • Wall

  • Sky

  • Traffic Light

  • Terrain

  • Bus

  • Traffic Sign

  • Person

  • Pole

  • Train

  • Truck

  • Vegetation

  • Car

  • Sidewalk

  • Rider

  • Road

  • Motorcycle

Technical Approach

Coming soon!

Datasets

Overview


The Freiburg Forest dataset was collected using our Viona autonomous mobile robot platform equipped with cameras for capturing multi-spectral and multi-modal images. The dataset may be used for evaluation of different perception algorithms for segmentation, detection, classification, etc. All scenes were recorded at 20 Hz with a camera resolution of 1024x768 pixels. The data was collected on three different days to have enough variability in lighting conditions as shadows and sun angles play a crucial role in the quality of acquired images. The robot traversed about 4.7 km each day. We provide manually annotated pixel-wise ground truth segmentation masks for 6 classes: Obstacle, Trail, Sky, Grass, Vegetation, and Void.

For each spectrum/modality, we provide one zip file containing all the sequences. Each sequence is a continous stream of camera frames. All the multi-spectral images are in the PNG format and the depth images are in the 16-bit TIFF format. For the evaluations mentioned in the paper, we provide two text files containing the train and test splits. If you would like to contribute to the annotations, please contact us. More details and evaluations can be found in our papers listed under publications.

BibTeX


Please cite our work if you use the Freiburg Forest Dataset or report results based on it.

@InProceedings{Valada_2016_ISER,
author = {Abhinav Valada and Gabriel Oliveira and Thomas Brox and Wolfram Burgard},
title = {Deep Multispectral Semantic Scene Understanding of Forested Environments using Multimodal Fusion},
booktitle = {The 2016 International Symposium on Experimental Robotics (ISER 2016)},
year = 2016,
month = oct,
url = {http://ais.informatik.uni-freiburg.de/publications/papers/valada16iser.pdf},
address = {Tokyo, Japan}
}

License Agreement


The data is provided for non-commercial use only. By downloading the data, you accept the license agreement which can be downloaded here. If you report results based on the Freiburg Forest datasets, please consider citing the first paper mentioned under publications.

Downloads


Freiburg Forest Raw

The raw dataset contains over 15,000 images of unstructued forest environments, captured at 20Hz using our Viona autonomous robot platform equipped with a Bumblebee2 stereo vision camera.

Freiburg Forest Multi-Modal/Spectral Annotated

The dataset contains the following multi-modal/spectral images with groundtruth annotations: RGB, Depth, NIR, NRG, NDVI, EVI and their variants. Pixel-level annotations are provided for 6 semantic classes: Trail, Grass, Vegetation, Obstacle, Sky, Void.

Code

A software implementation of this project (AdapNet++, SSMA, AdapNet, CMoDE) based on TensorFlow can be found on our GitHub repository.

Video Demos





Images

Modality 1

Modality 2

Segmentation Output

Summer
Winter
Fall
Sunset
Night
Rain
Glare
Motion Blur
Snow & Shadows

Publications

  • Abhinav Valada, Rohit Mohan, Wolfram Burgard
    Self-Supervised Model Adaptation for Multimodal Semantic Segmentation
    arXiv preprint arXiv:1808.03833, 2018.

  • Abhinav Valada, Johan Vertens, Ankit Dhall, Wolfram Burgard
    AdapNet: Adaptive Semantic Segmentation in Adverse Environmental Conditions
    Proceedings of the IEEE International Conference on Robotics and Automation, Singapore, 2017.

  • Abhinav Valada, Ankit Dhall, Wolfram Burgard
    Convoluted Mixture of Deep Experts for Robust Semantic Segmentation
    IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop, State Estimation and Terrain Perception for All Terrain Mobile Robots, Daejeon, Korea, 2016.

  • Abhinav Valada, Gabriel L. Oliveira, Thomas Brox, Wolfram Burgard
    Deep Multispectral Semantic Scene Understanding of Forested Environments using Multimodal Fusion
    The International Symposium on Experimental Robotics (ISER), Tokyo, Japan, 2016.

  • Abhinav Valada, Gabriel L. Oliveira, Thomas Brox, Wolfram Burgard
    Robust Semantic Segmentation using Deep Fusion
    Robotics: Science and Systems (RSS) Workshop, Limits and Potentials of Deep Learning in Robotics, Ann Arbor, USA, 2016.

  • Gabriel L. Oliveira, Abhinav Valada, Wolfram Burgard, Thomas Brox
    Deep Learning for Human Part Discovery in Images
    Proceedings of the IEEE International Conference on Robotics and Automation, Stockholm, Sweden, 2016.
  • People