Desenvolvimento de um framework de benchmarking para avaliação de modelos de linguagem proprietários e open source

Abstract

Generative artificial intelligence technologies, especially Large Language Models (LLMs), have emerged as central tools in computational applications. However, the predominance of proprietary solutions raises questions about transparency, flexibility, and technological dependency. Therefore, the objective of this work is to present and evaluate an experimental framework for comparing proprietary and open-source language models, using quantitative metrics and standardized benchmarks. Models from the Gemini family (Google) and opensource alternatives such as LLaMA, GPT-OSS and Qwen were employed, evaluated through the Google Generative AI API and Groq API, respectively. The study adopts a quantitative and experimental approach, using the ROUGE, BLEU, and BERTScore metrics, as well as the MMLU and HellaSwag benchmarks, complemented by analyses with the EvidentlyAI tool. The results demonstrate the feasibility of the system as a benchmarking tool, highlighting the potential of open solutions and the importance of transparent methodologies for LLM evaluation.