Desenvolvimento e avaliação de chatbot pedagógico open source com RAG para educação online

Abstract

This paper details a comprehensive methodology for the development and evaluation of a pedagogical chatbot for educational platforms, prioritizing accessibility and reproducibility with open-source tools. The process began with the construction of a detailed dataset from the CS50 course, employing web scraping and text processing techniques to structure the knowledge base. Based on this dataset, rich and diverse synthetic Question-Answer (QA) data were generated, formulated according to the cognitive levels of Bloom's Taxonomy. The quality of this data was rigorously evaluated using automated metrics from the RAGAS framework, such as Faithfulness, Question Quality (via Simple Criteria Score), and Response Relevancy. This approach was complemented by human supervision, and the subsequent analysis revealed moderate, yet statistically significant, correlations between the automated metrics and human assessments, thereby validating the utility of RAGAS. The next stage consisted of evaluating the pedagogical chatbot, implemented with the OpenWebUI software and powered by the CS50 knowledge base. A comparison was made between the Gemini-2.0-Flash (proprietary) and GPT-OSS-20B (open-weight) models in response generation. The results indicated no statistically significant difference in the Factual Correctness metric between the proprietary and the open-weight models. This finding reinforces the technical viability of open-source solutions for providing factually correct pedagogical support, mitigating costs and promoting a more inclusive and equitable educational landscape.