Regression testing is a core activity that allows developers to ensure that source code changes do not introduce bugs. An important prerequisite then is that test cases are deterministic. However, this is not always the case as some tests suffer from socalled flakiness. Flaky tests have serious consequences, as they can hide real bugs and increase software inspection costs. Existing research has focused on understanding the root causes of test flakiness and devising techniques to automatically fix flaky tests; a key area of investigation being concurrency. In this paper, we investigate the relationship between flaky tests and three previously defined test smells, namely Resource Optimism, Indirect Testing and Test Run War.We have set up a study involving 19,532 JUnit test methods belonging to 18 software systems. A key result of our investigation is that 54% of tests that are flaky contain a test code smell that can cause the flakiness. Moreover, we found that refactoring the test smells not only removed the design flaws, but also fixed all 54% of flaky tests causally co-occurring with test smells.
Original languageEnglish
Title of host publicationProceedings - 2017 IEEE International Conference on Software Maintenance and Evolution, ICSME 2017
Place of PublicationLos Alamitos, CA
Number of pages11
ISBN (Electronic)978-1-5386-0992-7
Publication statusPublished - 2017
EventICSME 2017: 33rd International Conference on Software Maintenance and Evolution - Shanghai, China
Duration: 17 Sep 201724 Sep 2017
Conference number: 33


ConferenceICSME 2017
Abbreviated titleICSME
Internet address

    Research areas

  • Test Smells, Flaky Tests, Refactoring

ID: 32869195