Error and the Growth of Experimental Knowledge (Science & Its Conceptual Foundations) [Paperback]

In the last half of this century, the debate in this field has centered around the "Falsification" methodology of Karl Popper. In testing deterministic hypotheses, Popper observed that hypotheses can be proven wrong with falsifying evidence by applying the logical rule of modus tollens. However, when experimental evidence is consistent with the hypothesis, believing that this confirms the hypothesis is committing the "affirming the consequent" fallacy. Hence, scientific "theories" cannot be verified, but only falsified. Subsequent philosophers pointed out a number of shortcomings with this method: it cannot be applied to statistical hypotheses (which is the area were methodology is most important to practicing scientists), the Duhem problem- an observation can only falsify a hypothesis if it is conclusively certain, and the problem of auxiliary assumptions- when data is inconsistent with the hypothesis, the test is not informative regarding whether the main hypothesis is false or whether the problem resides in the ever-present auxiliary assumptions that are necessary to connect the hypothesis with experimental implications.

Subsequent philosophers addressed these methodological faults by applying the principles of Bayesian statistics to the problem of testing hypotheses; using experimental evidence to transform an a priori probability of a hypothesis being valid to an a posteriori probability. Ideas from a number of Bayesian philosophers were synthesized by Howson and Urbach in their excellent book, Scientific Reasoning: The Bayesian Approach. This method applies equally well to both statistical and deterministic hypotheses. Also, the Bayesians claim neat solutions to other problems with Popper, e.g., auxiliary assumptions and Duhem. Predictably, other philosophers were quick to find faults with this approach. The most common objections are: the necessity of subjective prior probabilities and the "Problem of Old Evidence."

Mayo, in her book Error and the Growth of Experimental Knowledge, presents an Error-Statistical approach to scientific inference. She confronts head-on three important issues that any normative method must address: What counts as experimental evidence?, How to assess the acceptability of auxiliary assumptions and how to rule out alternative hypotheses?, and How to falsify statistical hypotheses? The first is addressed by explicitly identifying and justifying assumptions of the experimental data. The second issue is handled by the careful design and control of experiments. Statistical hypotheses are falsified using Neyman-Pearson type statistical tests.

The concept of "severe tests" is developed which is very powerful and widely applicable. Experiments are designed to ensure that the test of a hypothesis is severe and informative. These tests are useful in both testing auxiliary hypotheses and ruling out alternate hypotheses. A hypothesis can be confirmed if it passes a severe test (enabling the "Growth of Knowledge.") Mayo effectively uses the concept of severe tests to shed light on the philosophical problem of the acceptability of ad hoc (use-constructed) hypotheses. Common or canonical types of error are assembled into an "error repertoire" which are used to design experiments. When hypotheses fail, these well design tests will yield information as to which type of error was committed. Failed hypotheses are subsequently improved by applying the Peircean idea of "listening to error patterns."

Deterministic hypotheses are brought into this framework by the observation that in testing such hypotheses, inevitable approximations, inaccuracies and uncertainties enable the application of standard statistical tests. Also, Mayo's Error Statistical approach seems to address many of the other shortcomings of both Popper and the Bayesians. Perhaps the most important aspect of Mayo's methodology to practitioners as myself, is that these principles are of practical use to scientists, as opposed to merely impressive theoretical constructs. Her statement "you cannot just throw some evidence at the error-statistician and expect an informative answer" rings true to any scientist that has tried to draw conclusions from poorly designed experiments. I can strongly recommend this book to practitioners of both the "hard" sciences and certainly the "soft" sciences where methodology is of critical importance.

D.S. Fraedrich

Research Physicist

Naval Research Laboratory