You are presented for object instance segmentation a conceptually simple, flexible, and general framework. The approach looked at here while simultaneously generating a high-quality segmentation mask for each instance is efficient enough to detect objects in an image.
This method, named Mask R-CNN, by addition of a branch in order to predict an object mask in parallel with the existing branch for bounding box recognition extends Faster R-CNN. Mask R-CNN, running at 5 fps is relatively simple to train and adds only a small overhead to Faster R-CNN.
Above all, Mask R-CNN is easy to generalize to other tasks, for example: Allowing the estimation of human poses in the same framework.
The results on the top are put ahead in all three tracks of the COCO suite of challenges, that include the following:
- Instance Segmentation,
- Bounding box Object Detection,
- Person Key Point Detection.
Mask R-CNN, including the COCO 2016 challenge winners outperforms all existing, single-model entries on every task.
The vision community over a short period of time has rapidly improved object detection as well as semantic segmentation results. In a huge part, these advances have been driven by powerful baseline systems, for object detection and semantic segmentation, respectively such as the Fast/Faster RCNN and Fully Convolutional Network (FCN) frameworks. These methods together with fast training and inference time are conceptually intuitive and offer flexibility as well as robustness.
Their Main Goal:
Their main goal in this work is to develop for instance segmentation a comparably enabling framework. which is challenging because it while also precisely segmenting each instance requires the correct detection of all objects in an image.
However, you are shown that a surprisingly simple, flexible, and a fast system can surpass prior state-of-the-art instance segmentation results. Hence presenting their method, called Mask.
Since Mask R-CNN when given the Faster R-CNN framework turns out to be pretty simple to implement as well as train, it, as a result, facilitates a wide range of flexible architecture designs.
Mask R-CNN in principle is an intuitive extension of Faster R-CNN, yet for good results the construction of the mask branch properly is critical. Most importantly, Faster RCNN was not designed for alignment of pixel-to-pixel between network inputs and outputs. In order to fix the misalignment caused, in this paper, they propose RoIAlign: A layer that is simple, quantization-free and preserves exact spatial locations faithfully.
- Instance Segmentation
If looked at conceptually, Mask R-CNN is pretty simple: Faster R-CNN for each candidate object has
- Two Outputs,
- A Class Label
- A Bounding-Box Offset;
To this, there is an addition of a third branch that outputs the object mask. Mask R-CNN thus turns out to be a natural and intuitive idea.
But the additional mask output is distinct from the class and box outputs, therefore, requiring extraction of a much finer spatial layout of an object. Next, you are introduced to the key elements of Mask R-CNN, including, the main missing piece of Fast/Faster R-CNN: pixel-to-pixel alignment.
It is hoped that this simple and effective approach presented in the paper will serve as a solid baseline in instance-level recognition and help ease any future research in the area.
This was only a brief insight into the paper. For more information regarding the same, go through the links mentioned below.
This is an implementation of Mask R-CNN on Python 3, Keras, and TensorFlow. The model generates bounding boxes and segmentation masks for each instance of an object in the image. It’s based on Feature Pyramid Network (FPN) and a ResNet101 backbone.
- Source code of Mask R-CNN built on FPN and ResNet101.
- Training code for MS COCO
- Pre-trained weights for MS COCO
- Jupyter notebooks to visualize the detection pipeline at every step
- ParallelModel class for multi-GPU training
- Evaluation on MS COCO metrics (AP)
- Example of training on your own dataset
The code is documented and designed to be easy to extend. If you use it in your research, please consider referencing this repository. If you work on 3D vision, you might find our recently released Matterport3D dataset useful as well. This dataset was created from 3D-reconstructed spaces captured by our customers who agreed to make them publicly available for academic use. You can see more examples here.
- demo.ipynb Is the easiest way to start. It shows an example of using a model pre-trained on MS COCO to segment objects in your own images. It includes code to run object detection and instance segmentation on arbitrary images.
- train_shapes.ipynb shows how to train Mask R-CNN on your own dataset. This notebook introduces a toy dataset (Shapes) to demonstrate training on a new dataset.
- (model.py, utils.py, config.py): These files contain the main Mask RCNN implementation.
- inspect_data.ipynb. This notebook visualizes the different pre-processing steps to prepare the training data.
- inspect_model.ipynb This notebook goes in depth into the steps performed to detect and segment objects. It provides visualizations of every step of the pipeline.
- inspect_weights.ipynb This notebooks inspects the weights of a trained model and looks for anomalies and odd patterns.
Step by Step Detection
To help with debugging and understanding the model, there are 3 notebooks (inspect_data.ipynb, inspect_model.ipynb,inspect_weights.ipynb) that provide a lot of visualizations and allow running the model step by step to inspect the output at each point. Here are a few examples:
1. Anchor sorting and filtering
2. Bounding Box Refinement
3. Mask Generation
Examples of generated masks. These then get scaled and placed on the image in the right location.
Often it’s useful to inspect the activations at different layers to look for signs of trouble (all zeros or random noise).
5. Weight Histograms
Another useful debugging tool is to inspect the weight histograms. These are included in the inspect_weights.ipynb notebook.
6. Logging to TensorBoard
TensorBoard is another great debugging and visualization tool. The model is configured to log losses and save weights at the end of every epoch.
6. Composing the different pieces into a final result
Training on MS COCO
We’re providing pre-trained weights for MS COCO to make it easier to start. You can use those weights as a starting point to train your own variation on the network. Training and evaluation code is in
samples/coco/coco.py. You can import this module in Jupyter notebook (see the provided notebooks for examples) or you can run it directly from the command line as such:
# Train a new model starting from pre-trained COCO weights python3 samples/coco/coco.py train --dataset=/path/to/coco/ --model=coco # Train a new model starting from ImageNet weights python3 samples/coco/coco.py train --dataset=/path/to/coco/ --model=imagenet # Continue training a model that you had trained earlier python3 samples/coco/coco.py train --dataset=/path/to/coco/ --model=/path/to/weights.h5 # Continue training the last model you trained. This will find # the last trained weights in the model directory. python3 samples/coco/coco.py train --dataset=/path/to/coco/ --model=last
You can also run the COCO evaluation code with:
# Run COCO evaluation on the last trained model python3 samples/coco/coco.py evaluate --dataset=/path/to/coco/ --model=last
The training schedule, learning rate, and other parameters should be set in
Training on Your Own Dataset
Start by reading this blog post about the balloon color splash sample. It covers the process starting from annotating images to training to using the results in a sample application.
In summary, to train the model on your own dataset you’ll need to extend two classes:
Config This class contains the default configuration. Subclass it and modify the attributes you need to change.
Dataset This class provides a consistent way to work with any dataset. It allows you to use new datasets for training without having to change the code of the model. It also supports loading multiple datasets at the same time, which is useful if the objects you want to detect are not all available in one dataset.
See examples in
Differences from the Official Paper
This implementation follows the Mask RCNN paper for the most part, but there are a few cases where we deviated in favor of code simplicity and generalization. These are some of the differences we’re aware of. If you encounter other differences, please do let us know.
- Image Resizing: To support training multiple images per batch we resize all images to the same size. For example, 1024x1024px on MS COCO. We preserve the aspect ratio, so if an image is not square we pad it with zeros. In the paper the resizing is done such that the smallest side is 800px and the largest is trimmed at 1000px.
- Bounding Boxes: Some datasets provide bounding boxes and some provide masks only. To support training on multiple datasets we opted to ignore the bounding boxes that come with the dataset and generate them on the fly instead. We pick the smallest box that encapsulates all the pixels of the mask as the bounding box. This simplifies the implementation and also makes it easy to apply image augmentations that would otherwise be harder to apply to bounding boxes, such as image rotation.To validate this approach, we compared our computed bounding boxes to those provided by the COCO dataset. We found that ~2% of bounding boxes differed by 1px or more, ~0.05% differed by 5px or more, and only 0.01% differed by 10px or more.
- Learning Rate: The paper uses a learning rate of 0.02, but we found that to be too high, and often causes the weights to explode, especially when using a small batch size. It might be related to differences between how Caffe and TensorFlow compute gradients (sum vs mean across batches and GPUs). Or, maybe the official model uses gradient clipping to avoid this issue. We do use gradient clipping, but don’t set it too aggressively. We found that smaller learning rates converge faster anyway so we go with that.
Contributions to this repository are welcome. Examples of things you can contribute:
- Speed Improvements. Like re-writing some Python code in TensorFlow or Cython.
- Training on other datasets.
- Accuracy Improvements.
- Visualizations and examples.
You can also join our team and help us build even more projects like this one.
Python 3.4, TensorFlow 1.3, Keras 2.0.8 and other common packages listed in
MS COCO Requirements:
To train or test on MS COCO, you’ll also need:
- pycocotools (installation instructions below)
- MS COCO Dataset
- Download the 5K minival and the 35K validation-minus-minival subsets. More details in the original Faster R-CNN implementation.
If you use Docker, the code has been verified to work on this Docker container.
- Install dependencies
pip3 install -r requirements.txt
- Clone this repository
- Run setup from the repository root directory
python3 setup.py install
- Download pre-trained COCO weights (mask_rcnn_coco.h5) from the releases page.
- (Optional) To train or test on MS COCO install
pycocotoolsfrom one of these repos. They are forks of the original pycocotools with fixes for Python3 and Windows (the official repo doesn’t seem to be active anymore).
Projects Using this Model
If you extend this model to other datasets or build projects that use it, we’d love to hear from you.
4K Video Demo by Karol Majek.
Images to OSM: Improve OpenStreetMap by adding baseball, soccer, tennis, football, and basketball fields.
Splash of Color. A blog post explaining how to train this model from scratch and use it to implement a color splash effect.
Segmenting Nuclei in Microscopy Images. Built for the 2018 Data Science Bowl
Code is in the
Mapping Challenge: Convert satellite imagery to maps for use by humanitarian organisations.
Link To The PDF: Click Here