(Essay found in Nesselroade & Grimm, 2019; pgs. 166 – 167)

An aspect of probabilistic thinking that seems to have been lost in most modern discussions of probability, and which may be partly responsible for the reproducibility problem in the social and medical sciences, is the realization that uncertainty is not merely the quantification of likelihood, but is also influenced by a clear understanding of the situation; let us use the term “clarity” for a lack of a better one. Now “likelihood” (or “risk” as it is sometimes called) is usually understood as something that can be quantified numerically; like the odds of selecting a spade from a deck of cards. This predisposes that the conditions are well understood as well as the relative frequencies of favorable and unfavorable options to be known. Every judgment call dealing only with likelihood has some known degree of risk to be wrong. Modern probability theory, since the 1930’s, has almost exclusively focused on only this aspect of uncertainty. However, uncertainty, which is what probability theory was designed to address, is not restricted to merely this more objective quantification. Clarity is also a necessary component. By clarity, it is meant the degree to which the situation one is speculating about is being properly conceptualized.

For instance, when we say that the likelihood of getting snake eyes when rolling two dice is 1/36, we are assuming that we have full clarity regarding the situation (e.g., what is known as “snake eyes,” the features of the two dice, and so on). We realize that gambling on this outcome has a certain amount of risk associated with it (actually a lot of risk associated with it!) but this risk is at least quantifiable. However, if we think about it for a few minutes it should be easy to see that we may be overlooking several aspects of the probabilistic situation out of convenience. For instance, what if each side of each die is not equally likely? What if there are actually ten sides to each die? What if the numbers on the die do not start with 1? Now, this is a simple example where there is most likely a lot of initial clarity regarding the situation; or if not, issues revolving around the clarity of the situation could be easily resolved by simply looking at the dice to be rolled. However, in many research situations, the degree of clarity that a researcher has regarding the situation is not as easy as inspecting the dice. Further compounding the situation, a researcher’s lack of clarity may be best understood as a qualitative variable as opposed to a quantitative one; and as a result, not something that could even be factored into modern probability theory. Acknowledging this limitation, however, might leave the researcher without a clear path forward in their efforts to investigate. So, oftentimes researchers, perhaps unknowingly, engage in what some theorists call “willful ignorance” regarding issues of clarity so as to focus only on the issue of likelihood when testing their hypotheses. By essentially ignoring issues of clarity and focusing exclusively on likelihood, modern probability theory has created a false sense of confidence in the stated outcomes of some statistical analyses. Some argue that our failure to address this fundamental issue has led to the challenging state in which many areas of scientific investigation currently find themselves (e.g., Byers, 2011; Weisberg, 2014).

To help illustrate this further, imagine a medical situation where a patient has arrived with a set of symptoms and an unknown diagnosis. There are two judgments that need to be made. First, what is the nature of the illness; and second, what form of treatment would be best. It might be helpful to consider the first of these two judgements as being representative of the issues of clarity. The doctor, relying on clinical expertise, past experiences, logic, and judgment, moves to increase clarity by rendering a diagnosis. Yes, some probabilistic information may have been used in this analysis, but primarily this was not an exercise in probability. Now, the second question of treatment may be much more representative of the issues related to likelihood and probability. Based on the outcomes of previous studies employing different treatments, the doctor may choose to prescribe one form of treatment over another due to the likelihood of a favorable outcome derived from a quantification of relevant data. Research situations oftentimes involve both forms of uncertainty; likelihood and clarity. Moreover, when research replication efforts take place, issues of clarity have oftentimes not properly been noted or documented by the original researchers and are not properly considered by the replicating researcher. This disconnect can lead to contradictory findings when replication takes place and thus great confusion as to what is actually going on. In the interest of accuracy and the public confidence in science, it would be wise for researchers to begin to consider seriously the issue of clarity when setting up their investigations, writing up their procedures, and drawing conclusions from their data.

Find this and other essays regarding “Is the Scientific Method Broken?” in the Nesselroade & Grimm textbook.

Byers, W. (2011). The blind spot: Science and the crisis of uncertainty. Princeton, NJ: Princeton University Press.

Weisberg, H. I. (2014). Willful ignorance: The mismeasure of uncertainty. Hoboken, N.J.: Wiley and Sons.