TY - JOUR
T1 - Testing with Fewer Resources
T2 - An Adaptive Approach to Performance-Aware Test Case Generation
AU - Grano, Giovanni
AU - Laaber, Christoph
AU - Panichella, Annibale
AU - Panichella, Sebastiano
PY - 2020
Y1 - 2020
N2 - Automated test case generation is an effective technique to yield high-coverage test suites. While the majority of research effort has been devoted to satisfying coverage criteria, a recent trend emerged towards optimizing other non-coverage aspects. In this regard, runtime and memory usage are two essential dimensions: less expensive tests reduce the resource demands for the generation process and later regression testing phases. This study shows that performance-aware test case generation requires solving two main challenges: providing a good approximation of resource usage with minimal overhead and avoiding detrimental effects on both final coverage and fault detection effectiveness. To tackle these challenges, we conceived a set of performance proxies - inspired by previous work on performance testing - that provide a reasonable estimation of the test execution costs (i.e., runtime and memory usage). Thus, we propose an adaptive strategy, called aDynaMOSA, which leverages these proxies by extending DynaMOSA, a state-of-the-art evolutionary algorithm in unit testing. Our empirical study - involving 110 non-trivial Java classes - reveals that our adaptive approach generates test suite with statistically significant improvements in runtime (-25 percent) and heap memory consumption (-15 percent) compared to DynaMOSA. Additionally, aDynaMOSA has comparable results to DynaMOSA over seven different coverage criteria and similar fault detection effectiveness. Our empirical investigation also highlights that the usage of performance proxies (i.e., without the adaptiveness) is not sufficient to generate more performant test cases without compromising the overall coverage.
AB - Automated test case generation is an effective technique to yield high-coverage test suites. While the majority of research effort has been devoted to satisfying coverage criteria, a recent trend emerged towards optimizing other non-coverage aspects. In this regard, runtime and memory usage are two essential dimensions: less expensive tests reduce the resource demands for the generation process and later regression testing phases. This study shows that performance-aware test case generation requires solving two main challenges: providing a good approximation of resource usage with minimal overhead and avoiding detrimental effects on both final coverage and fault detection effectiveness. To tackle these challenges, we conceived a set of performance proxies - inspired by previous work on performance testing - that provide a reasonable estimation of the test execution costs (i.e., runtime and memory usage). Thus, we propose an adaptive strategy, called aDynaMOSA, which leverages these proxies by extending DynaMOSA, a state-of-the-art evolutionary algorithm in unit testing. Our empirical study - involving 110 non-trivial Java classes - reveals that our adaptive approach generates test suite with statistically significant improvements in runtime (-25 percent) and heap memory consumption (-15 percent) compared to DynaMOSA. Additionally, aDynaMOSA has comparable results to DynaMOSA over seven different coverage criteria and similar fault detection effectiveness. Our empirical investigation also highlights that the usage of performance proxies (i.e., without the adaptiveness) is not sufficient to generate more performant test cases without compromising the overall coverage.
KW - Evolutionary testing
KW - many-objective optimization
KW - performance
UR - http://www.scopus.com/inward/record.url?scp=85073545104&partnerID=8YFLogxK
U2 - 10.1109/TSE.2019.2946773
DO - 10.1109/TSE.2019.2946773
M3 - Article
AN - SCOPUS:85073545104
SN - 0098-5589
VL - 47 (2021)
SP - 2332
EP - 2347
JO - IEEE Transactions on Software Engineering
JF - IEEE Transactions on Software Engineering
IS - 11
ER -