Detecting cell nuclei on oral cytology using Mask R-CNN
Application of the Sliding Window Method and Mask-RCNN Method to Nuclear Recognition in Oral Cytology
Authors: Eiji Mitate, Kirin Inoue, Retsushi Sato, Youichi Shimomoto, Seigo Ohba, Kinuko Ogata, Tomoya Sakai, Jun Ohno, Ikuo Yamamoto, Izumi Asahina
Summary
This study from Nagasaki University (Japan) developed an AI-assisted cytology framework for detecting cell nuclei in oral cytological images, comparing two methods: the Sliding Window Method (SWM) and Mask-RCNN.
Early detection of oral cancer relies on recognizing nuclear atypia (increased nucleus-to-cytoplasm ratio). Thus, accurate nuclei segmentation is essential for AI-based cytology screening.
Methodology
- Dataset: 30 cases of liquid-based oral cytology, Papanicolaou-stained.
- Image capture: Nikon Eclipse Ti-S microscope (40× objective, 1280×1024 px).
- Sliding Window Method: Images divided into 96×96 px tiles (591 train / 197 test = 1,576 total). CNN trained to classify tiles with vs without nuclei.
- Mask-RCNN: 130 images (Class II and III lesions) used to create masks of nuclear regions for instance segmentation.
- Hardware: NVIDIA RTX 2080Ti GPU + Intel i9 CPU + 64 GB RAM.
Results
- SWM: Best test accuracy = 93.1 % (No. 2 model, 20 epochs, 18 layers).
- Mask-RCNN: Detected 37 cell nuclei with only 1 false positive (error rate = 0.027).
- Loss improved from 0.89 → 0.45 over 40 epochs.
- Mask-RCNN showed superior performance in reducing false detections and segmenting nuclei without overlaps.
- Visually, Mask-RCNN accurately outlined nuclei while omitting background and non-cellular regions.
Conclusion: Mask-RCNN outperformed the Sliding Window Method in both accuracy and efficiency for detecting cell nuclei in oral cytology.
This approach lays the foundation for AI-assisted oral cytology screening analogous to cervical cytology.
DOI: https://doi.org/10.1186/s13000-022-01245-0
Key Words
Oral Cytology, Mask RCNN, Sliding Window Method, Deep Learning, Nuclear Segmentation, Papanicolaou Staining, Cytopathology, AI Assisted Diagnosis, Computer Vision, Dental AI
Extracted Data
- Year: 2022
- Modality: Cytology (Microscopic Liquid-Based Preparation)
- Dataset: 1,576 tiles (SWM) + 130 images (Mask-RCNN)
- Dataset Split: 591 train | 197 test (SWM); 130 train (Mask-RCNN)
- Network Architecture: CNN for SWM (18–24 layers) and Mask-RCNN (backbone + RPN + Head modules)
- Metrics: SWM accuracy = 93.1 %; Mask-RCNN error rate = 0.027 (37/38 true detections)
- AP – Strategy: Manual masking of nuclei from blue-stained cells (Class II–III lesions)
- AP – Professional Qty: 1
- AP – Supervisor Presence: No information
- AP – Experience Level: No information
- AP – Expertise Area: OMFP
- AP – Tool or System: No information
- Task: Classification and Instance Segmentation of Cell Nuclei in Oral Cytology
- Project Objective: To compare SWM and Mask-RCNN for automatic nuclear detection and enable AI-assisted screening for early oral cancer diagnosis.
Clinical Relevance
- Clinical importance: Early oral cancer diagnosis depends on identifying nuclear atypia and N/C ratio changes in cytology smears.
- Innovation: First comparison of Sliding Window and Mask-RCNN for oral cytology; demonstrated Mask-RCNN superiority in precision and false positive reduction.
- Practical impact: Supports development of automated screening systems to reduce pathologist workload and improve diagnostic consistency in oral oncology.
Youtube Video: https://youtu.be/n2KksRV61fo
