É uma exceção testar comportamento excepcional? Uma avaliação empírica utilizando testes automatizados em Java

Abstract

Software testing is a crucial activity to check the internal quality of a software. During testing, developers often create tests for the normal behavior of a particular functionality (e.g., was this file properly uploaded to the cloud?). However, little is known whether developers also create tests for the exceptional behavior (e.g., what happens if the network fails during the file upload?). To minimize this knowledge gap, in this work we designed and performed a mixed-method study to understand whether and to what extent 417 open source Java projects are testing the exceptional behavior using the JUnit and TestNG frameworks, and the AssertJ library. Through static analysis, we found that 254 (60.91%) projects have at least one test method dedicated to test the exceptional behavior. We also found that the number of test methods for exceptional behavior with respect to the total number of test methods lies between 0% and 10% in 317 (76.02%) projects. Also, 239 (57.31%) projects test only up to 10% of the used exceptions in the System Under Test (SUT). When it comes to mobile apps, we found that, in general, developers pay less attention to exceptional behavior tests when compared to desktop/server and multi-platform developers. In general, we found more test methods covering custom exceptions (the ones created in the own project) when compared to standard exceptions available in the Java Development Kit (JDK) or in third-party libraries. Moreover, we also performed a dynamic analysis on a subset of 39 projects, with publicly available line coverage reports, to investigate whether and to what extent the test suites exercises the throw statements found in the SUT. The results of the dynamic analysis indicate that the test suites do not exercise the most of throw statements, even in projects where there seem to have concerns about code coverage. We also enriched the understanding of how developers write their exceptional behavior tests in terms of constructs from JUnit, TestNG, and AssertJ. To triangulate the results, we conduct a survey with 66 developers from the projects we study. In general, the survey results confirm our findings. In particular, the majority of the respondents agrees that developers often neglect exceptional behavior tests. As implications, our numbers might be important to alert developers that more effort should be placed on creating tests for the exceptional behavior.