2002), here we illustrate how it is possible increasing the robustness of the findings by running replications that overcome previous experiments’ threats to validity. Even though it is not possible to rule out all the validity threats in any experiment (Shull et al. Erdogmus et al.’s experiment has already been replicated by Fucci and Turhan ( 2013) retaining the same experimental design, but changing the operationalization of the response variable (i.e., quality). Specifically, we run a replication of Erdogmus et al.’s experiment on TDD (Erdogmus et al. Learning about TDD’s effectiveness in terms of quality may assist practitioners when deciding which development approach to use in their daily practice. This is because, according to TDD advocates, TDD’s continuous in-built testing and refactoring cycles create an ever-growing safety net (i.e., a test bed) on which software systems rest (Beck 2003). TDD advocates have attributed it several benefits, being the most claimed its ability to deliver high-quality Footnote 1 software (Beck 2003). TDD is an agile software development process that enforces the creation of software systems by means of small testing-coding-refactoring cycles (Beck 2003). This allows studying the effect of such changes on results, and thus, elicit moderator variables (i.e., variables rather than the treatments impacting the results (Juristo and Vegas 2009)).Īs an illustrative example, here we show how we run a replication that overcame the weaknesses of a series of experiments on test-driven development (TDD) (Beck 2003). Besides, replications allow the systematic variation of specific elements from baseline experiments’ configurations (e.g., the experimental task).

This allows comparing the replications’ results and hypothesizing on whether the replications’ threats to validity might have materialized or not.

For example, replications’ designs can be tweaked to overcome the threats to validity of previous experiments. Replication of experiments may help to overcome such limitations (Shepperd 2016). 2014): (1) results may be imprecise (as the number of participants is typically low in SE experiments) (2) results might be artifactual (e.g., influenced by the programming environment rather than the treatments themselves) (3) results cannot be generalized to different contexts rather than those of the experiment (4) results may be impacted by the materialization of unforeseen threats to validity (e.g., rivalry threat, when the effectiveness of the less “desirable” treatment gets penalized by the participants’ disinterest). However, isolated experiments suffer from several shortcomings (Gómez et al. Isolated experiments are being run in software engineering (SE) to assess the performance of different treatments (i.e., tools, technologies, or processes).

Reliability of an experiment