Transformer-based automatic weed segmentation in multispectral UAV imagery

Abstract

Weed mismanagement causes significant annual losses to Brazilian agriculture, with some estimates reaching billions of reais. Given the vital role agriculture plays in Brazil's economy, efficient weed management is essential. Over the years, advances in technology, such as the use of multispectral and hyperspectral sensors coupled with Unmanned Aerial Vehicles (UAVs), have enhanced weed management strategies, enabling precise monitoring and decision-making. However, the critical challenge in developing automated weed management systems lies in accurately segmenting weeds from crops. This task is inherently complex due to various factors, such as the phenotypic similarity between weeds and crops, occlusions, shadows, variations in weather conditions, differences in plant growth stages, and image capture inconsistencies during UAV flights, including motion blur and noise. This research proposes a novel segmentation framework based on vision transformers architectures, leveraging vegetation indices and alternative color spaces to achieve precise weed segmentation across diverse scenarios and datasets. The proposed methodology integrates spectral, spatial, and contextual information using early and late fusion strategies, significantly enhancing segmentation accuracy. Experimental results demonstrate the superiority of the approach compared to state-of-the-art methods. On the Sugarcane and RedEdge datasets, late fusion strategies achieved F1-scores of 0.840 and 0.857, respectively. For the challenging Sequoia dataset, the methodology achieved an F1-score of 0.736, highlighting its robustness in complex agricultural scenarios.