Multi-modality Anomaly Segmentation on the Road

Heng Gao, Zhuolin He, Shoumeng Qiu, Xiangyang Xue*, Jian Pu
Fudan University

Abstract

Semantic segmentation allows autonomous driving cars to understand the surroundings of the vehicle comprehensively. However, it is also crucial for the model to detect obstacles that may jeopardize the safety of autonomous driving systems. Based on our experiments, we find that current uni-modal anomaly segmentation frameworks tend to produce high anomaly scores for non-anomalous regions in images. Motivated by this empirical finding, we develop a multi-modal uncertainty-based anomaly segmentation framework, named MMRAS+, for autonomous driving systems. MMRAS+ effectively reduces the high anomaly outputs of non-anomalous classes by introducing text-modal using the CLIP text encoder. Indeed, MMRAS+ is the first multi-modal anomaly segmentation solution for autonomous driving. Moreover, we develop an ensemble module to further boost the anomaly segmentation performance. Experiments on RoadAnomaly, SMIYC, and Fishyscapes validation datasets demonstrate the superior performance of our method.

Installation

Please follow the instructions given in Mask2Anomaly.

Get Text Embeddings

python tools/prompt_engeering.py

The results will be saved in './pretrain/'. We utilize these text embeddings to segment anomalies in road scenes. :)

Anomaly Inference

The anomaly inference datasets are given in './datasets/Validation_Dataset', which can be downloaded from this link.

CUDA_VISIBLE_DEVICES=1 python anomaly_utils/anomaly_inference.py --score mmras --exp_name the_exp_name

The pre-trained model is given in './output/m2unk_coco_supervised_v1/best_contrastive.pth', one can also download the model weights from this link.

After running the code above, the inference log file will be saved in './results/'

Acknowledgement

Our code is developed based on Mask2Anomaly and MaskCLIP. Thanks to their great work!

Citation

If you find MMRAS+ useful in your research, please consider citing:

@article{gao2025multi,
  title={Multi-modality Anomaly Segmentation on the Road},
  author={Gao, Heng and He, Zhuolin and Qiu, Shoumeng and Xue, Xiangyang and Pu, Jian},
  journal={arXiv preprint arXiv:2503.17712},
  year={2025}
}