Comitês de Redes Neurais Convolucionais para a Segmentação de Núcleos em Imagens Histológicas de Displasias Epiteliais Orais

Abstract

Early diagnosis of potentially malignant disorders is the most reliable way to reduce the incidence of carcinoma in the oral cavity. Conditions such as leukoplakia and erythroplakia carry a risk of malignant transformation and often manifest alongside oral epithelial dysplasias (OEDs), which are alterations in the size and shape of cellular structures. The segmentation of cell nuclei is essential for diagnosis, allowing the extraction of information from these structures without interference from the epithelial region. In recent years, convolutional neural networks (CNN) have gained prominence due to their performance in nucleus segmentation tasks. However, achieving high segmentation accuracy remains a challenge. This work proposes a methodology based on ensemble neural network systems to improve the segmentation of cell nuclei in OED images. Segmentation models were combined using rules such as weighted sum, aiming to minimize individual errors of CNN models and enhance overall segmentation performance. Three types of ensembles were proposed: one composed of semantic segmenters, another using instance segmenters, and a hybrid ensemble combining both approaches. The proposed approach was evaluated on a newly created dataset for this study, containing OED cases, as well as on public datasets. Additionally, data augmentation techniques, RCAug and HistoGAN, were investigated to analyze their impact on segmentation. The methodology achieved remarkable results, with an accuracy of 95.07% and a Dice coefficient of 0.91 on the OED dataset. On public datasets, the method obtained a Dice coefficient of 0.89, demonstrating competitiveness compared to other methods in the literature. These results highlight the robustness of the proposed approach and its potential as a diagnostic support tool for OED cases, contributing to the early identification of cellular disorders.