Segmentação semântica de núcleos em leucoplasia bucal por meio de comitês de modelos de aprendizagem profunda

Abstract

The histopathological assessment of oral potentially malignant disorders, particularly oral leukoplakia, relies heavily on the analysis of nuclear morphology. However, this process is marked by subjectivity and substantial interobserver variability. In this context, accurate cell nuclei segmentation is a fundamental step for computer-aided histopathological analysis systems. Despite its relevance, nuclei segmentation specifically in oral leukoplakia images remains underexplored, as does the use of ensembles built from heterogeneous architectures. This dissertation proposes and evaluates ensembles of heterogeneous deep learning models for the semantic segmentation of cell nuclei in histopathological images of oral leukoplakia. Ten architectures representing three paradigms were investigated: convolutional, hybrid, and attention-based. The models were evaluated individually and combined using five \textit{ensemble learning} strategies under different data augmentation configurations. The evaluation employed traditional segmentation metrics and statistical validation. As a methodological contribution, three individual-contribution metrics were proposed and formalized in this dissertation: \textit{kept rate}, \textit{discarded rate}, and \textit{error mitigation}. These metrics quantify, respectively, the preservation of an individual model's correct predictions, the loss of those correct predictions after combination, and the correction of individual errors by the ensemble. Among the individual models, the best performance was achieved by ResNeSt-Unet++, with a Dice coefficient of 0.8853. Combination strategies based on probabilistic averaging proved to be the most effective, with \textit{soft voting} achieving a Dice coefficient of 0.8924 and statistically significant gains over the individual models. The application of the proposed metrics indicated that architectures from distinct paradigms exhibit complementary error patterns, favoring the preservation of correct predictions and the mitigation of individual errors. In addition, the gain provided by the ensembles tended to saturate when three to four models were combined. These results demonstrate that heterogeneous ensembles constitute an effective strategy for nuclei segmentation in oral leukoplakia.