Desenvolvimento de processo ETL automatizado em dados de pesquisas de uma empresa na área de tecnologia.

Abstract

ETL – which stands for data extraction, transformation and loading – is an integration process between different data sources capable of generating insights through possible transformations and storing the data in a single location. This technical report aims to describe the steps developed in an automated ETL process, created to optimize activities performed manually in the treatment of research data, applied by a technology company. Thus, it was proposed to use the Knime tool because it is intuitive, quick to implement and is open-source, enabling the implementation of the process without additional costs. The results obtained with the application of the process and its automation brought an increase in productivity, improved data accuracy and consumer experience, with agility as the main factor in the results, generating a gain of 23,8 hours on average for each execution of the process. However, this report can serve as an example for implementation in other projects and in other areas present in the job market.