Avaliação do uso de grandes modelos de linguagem na análise de riscos em segurança da informação: um estudo comparativo com especialistas humanos

Abstract

The increasing sophistication of cyber threats and the continuous evolution of the technological landscape impose significant challenges on organizations, especially given the global shortage of qualified cybersecurity professionals. In this context, Generative Artificial Intelligence and Large Language Models emerge as promising alternatives to support cybersecurity risk analysis and assessment activities. Despite their potential, Large Language Models also present limitations, such as generating inaccurate or ’hallucinated’ information capable of negatively influencing decision-making processes. This thesis investigates the behavior of Large Language Models in cybersecurity risk assessment scenarios compared to human experts, highlighting the importance of human oversight and the integration of humans and machines in this process. To achieve this, a questionnaire containing risk scenarios was applied to human participants, whose responses were compared with those generated by five popular Large Language Models. The results suggest that LLMs tended to underestimate security risks compared to human participants. Therefore, although they may be useful as support tools, their use as autonomous evaluators in information security risk assessment processes should be considered with caution.