Uma metodologia comparativa para avaliação de LLMs na detecção de phishing: um estudo envolvendo DeepSeek, Gemini e ChatGPT

Abstract

Phishing detection is a growing challenge in the digital security field, due to the sophistication of malicious messages and the difficulty in identifying attacks in an automated and reliable manner. Although Large Language Models (LLMs) have been investigated as potential solutions, their effectiveness still needs to be validated in real scenarios, especially for newer models. This study proposes a comparative evaluation between the Gemini, DeepSeek and ChatGPT models, focusing on the classification of legitimate and malicious emails. Based on 1,009 analyzed samples, the study characterized the emails based on the red flags (key features) extracted by each model, and applied standardized prompts to ensure consistency in the tests. The evaluation metrics included precision, recall, accuracy and F1-Score, complemented by a qualitative analysis of the responses. The results indicate that DeepSeek stands out in robustness, and obtained the best overall performance, with an F1-Score of 92.38%.