Questioning Standardization

In the article titled "Environmental standardization: cure or cause of poor reproducibility in animal experiments?" by S.H. Richter, J. P. Garner, and H. Würbel, published in Nature Methods (2009 April 6(4):257-61), we read:


"It is widely believed that environmental standardization is the best way to guarantee reproducible results ..... However, mounting evidence indicates that even subtle differences in laboratory or test conditions can lead to conflicting test outcomes. Because experimental treatments may interact with environmental conditions, experiments conducted under highly standardized conditions may reveal local 'truths' with little external validity. .... Environmental standardization can contribute to spurious and conflicting findings ..... This conclusion calls for research into practicable and effective ways of systematic environmental heterogenization ...."

There are 2 things noteworthy about this work.

The first is the application of data mining techniques to the results of scientific experiments to test hypothesis about how science is done - a baby step towards a Science of Science.

The second is the possible relevance of this approach to system (software/hardware) testing. That is, does our insistence on standardized test environments for software testing and validation inevitably leads us to amplify the quirks and specificalities of each (presumably) standardized test environment?