NUCSEG: An Effective Detectron2 Architecture for Nuclei Instance Segmentation in Histological Images

Authors

• Payman Hussein Hussan, Department of Computer Networks and Software Techniques, Babylon Technical Institute, Al-Furat Al-Awsat Technical University, Kufa, IraqFollow
• Syefy Mohammed Mangj, Department of Computer Networks and Software Techniques, Babylon Technical Institute, Al-Furat Al-Awsat Technical University, Kufa, IraqFollow

Abstract

Nuclei are often minuscule objects within histological whole slide imaging (WSI), presenting a labor-intensive task for pathologists to identify and analyze their characteristics. The precise identification of nuclei is essential in cancer pathology to conduct both quantitative and qualitative assessments of cancerous cells found within a slice of a tumor in order to assess the level of malignancy of the tumor; and to predict the outcome of the tumor. Manual identification is labor-intensive and prone to errors, whereas automatic recognition is complicated by the varied appearances and overlapping characteristics of nuclei, along with artifacts from digital photography. To address this challenge, a deep learning model called the NUCSEG model is suggested for automatic detection, which helps pathologists distinguish instances of nuclear objects within the WSI images. The NUCSEG model, inspired by the Detectron2 framework, utilizes Mask R-CNN with various enhanced ResNet backbone networks and BiFPN to accurately detect the nuclei boundaries and overlapping instances at multiple resolution levels to effectively identify the boundaries between the nucleus and cytoplasm. The experimental results show that the best NUCSEG configuration using the deep ResNet with a 101-layer backbone achieved a performance with a BBOX AP⁵⁰ and AP⁷⁵ of 80.141% and 55.278%, respectively. Furthermore, NUCSEG outperformed YOLOv11, particularly demonstrating superior performance at high IoU for 0.75, indicating its ability to propose regions and localize regions with consistent, high confidence ratings, even when nuclei overlap or have irregular shapes.

Keywords

Detectron2, Whole slide imaging (WSI), Bidirectional feature pyramid network (BiFPN), YOLOv11, Nuclei instance segmentation, Mask R-CNN

Subject Area

Computer Science

Article Type

Article

First Page

3412

Last Page

3437

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite this Article

Hussan, Payman Hussein and Mangj, Syefy Mohammed (2026) "NUCSEG: An Effective Detectron2 Architecture for Nuclei Instance Segmentation in Histological Images," Baghdad Science Journal: Vol. 23: Iss. 9, Article 25.
DOI: https://doi.org/10.21123/2411-7986.5419