Uma abordagem não intrusiva e automática para configuração do Hadoop

Abstract

The amount of digital data produce in the last years has increased significantly. MapRe- duce framework such as Hadoop have been widely used for processing big data on top of

cloud resources. In spite of these advances, contemporary systems are complex and dy- namic which makes them hard to configure in order to improve application performance.

Software auto-tuning is a solution to this problem as it helps developers and system ad- ministrators to handle hundreds of system parameters. For example, current work in

the literature use machine learning algorithms for Hadoop automatic configuration to

improve performance. However, these solutions use single machine learning algorithms,

thus making unfeasible to compare these solutions with each other to understand which

approach is best suited given an application and its input. In addition, current work is

intrusive or expose operational details for developers and/or system administrators. This

work proposes a transparent, modular and hybrid approach to improve the performance

of Hadoop applications. The approach proposes an architecture and implementation of

transparent software that automatically configures the Hadoop. Furthermore, this ap- proach proposes a hybrid solution that combines genetic algorithms with various machine

learning techniques as separate modules. A research prototype was implemented and eval- uated proving that the proposed approach can significantly reduce the execution time of

applications Hadoop WordCount and Terasort autonomously. Furthermore, the approach

converges quickly to the most suitable configuration application with low overhead.