Solução integrada de engenharia de dados para gestão e análise, utilizando uma abordagem open source e de baixo custo

Abstract

The increasing generation of data, driven by digital transformation and the spread of technologies like the Internet of Things, demands innovative and accessible solutions for efficient information management. The democratization of data analysis through the adoption of open source and low-cost technologies enables organizations of all sizes and sectors to extract valuable insights to optimize processes, make more accurate decisions, and drive business growth. By combining the scalability and flexibility of open-source solutions with accessibility and reduced costs, this approach allows a greater number of businesses to harness the potential of their data, fostering innovation and competitiveness. This work presents the implementation of a data management environment based on open-source tools, aiming to create an efficient and low-cost solution for building a lakehouse. The study addresses the growing need to manage large volumes of data, proposing an accessible ecosystem for organizations in various sectors, integrating the functionalities of both a Data Lake and a Data Warehouse to enable the collection, organization, and analysis of data in a structured manner. During the development, one of the challenges was integrating the various open-source tools, ensuring compatibility and security in the data management process. The modular approach adopted facilitated the configuration and automation of the stages, ensuring data integrity and optimizing the overall performance of the environment, consolidating itself as an effective solution for storing and processing large volumes of data. The results demonstrate the technical and economic feasibility of the environment, which was applied to the Electronic Information System (SEI) as a case study. The environment enabled the creation of a reliable infrastructure for the organization, storage, and efficient processing of data, providing an optimized foundation to support detailed analyses and improve information management. The comparison with Azure Synapse, a widely used solution in the market for data management, highlighted not only significant cost savings but also the ability of the open-source environment to offer a robust solution without requiring large investments, making it a viable alternative for institutions with limited resources.