A call for virtual experiments: accelerating the scientific process

Abstract

Experimentation is fundamental to the scientific method, whether for exploration, description or explanation. In the exploration of a novel system, children and researchers alike will mess about with things just to see what happens. More formalized experimental protocols ensure reproducible results and form a basis for comparing systems in terms of their response to a specific stimulus. Finally, experiments can be carefully designed to distinguish between competing causal hypotheses based on their different testable predictions about the outcome of the experimental manipulation. One would therefore expect experiments to be central in computational biology too. Indeed, a mathematical model embodies a thought experiment, a causal hypothesis, and its falsifiable predictions. It is easy to ask "what if" we were to change a parameter, an initial state, or the model structure. Papers in computational biology focus on describing and analyzing the effects of such changes, and on confronting models with experimental data. This confrontation often generates new hypotheses, and many if not most new models arise by modification of existing ones. However, most virtual experiments are not built to be reproducible, and thus die with the paper they are published in. This inhibits the critical scrutiny of models, as models are seldom subjected to the same simulation experiments as their predecessors, or revisited later in the light of new data. Perhaps worse, the status quo fails to take full advantage of experiments as a common language between modellers and experimentalists. Despite the growing availability of data and model repositories, there has been only a slow uptake of emerging tools and standards for documenting and sharing the protocols for simulation experiments and their results. We argue that promoting the reuse of virtual experiments would vastly improve the usefulness and relevance of computational models, including in biomedical endeavours such as the Virtual Physiological Human and the Human Brain Project. We review the benefits of reusable virtual experiments: in specifying, assaying, and comparing the behavioural repertoires of models; as prerequisites for reproducible research; to guide model reuse and composition; and for quality assurance in the application of computational biology models. Next, we discuss potential approaches for implementing virtual experiments, arguing that models and experimental protocols should be represented separately, but annotated so as to facilitate the linking of models to experiments and data. We follow with some consideration of open questions and challenges that remain before the use of virtual experiments can become widespread. Lastly, we outline a vision for how the rigorous, streamlined confrontation between experimental datasets and candidate models would enable a "continuous integration" of biological knowledge, akin to the strategy used in software development.