(Essay found in Nesselroade & Grimm, 2019; pg. 115)

Throughout the text a series of several “Boxes” are asking whether the scientific method is broken in light of the non-reproducibility problem currently plaguing the social, behavioral, and medical sciences. In Box 1.1 we looked at the “Wallpaper Effect” and the difficulty in identifying and controlling all extraneous variables. In Box 2.3 we looked at, among other things, different ways the collection of data may be biased through wording effects and order effects. In this box, let us explore some of the problems that occur in the data gathering process.

Sometimes researchers, even those with the best of intentions and who take precautions to remove their own biases, can influence how others respond merely by their involvement in the study. In general, these are referred to as “demand characteristics” or “experimenter effects.” Sometimes the unintentional involvement of the researcher can take a response which might ordinarily be quite varied in a population and shrink it to almost nothing. Perhaps the most famous historical example of demand characteristics features “Clever Hans,” the horse that could do mathematics (Pfungst, 1911). Hans was owned by Wilhelm Von Osten, a German schoolteacher who had claimed to teach his horse to add, subtract, multiple, divide, and work with fractions. Hans would answer a question by tapping out numbers with his hoof; his accuracy, though not perfect, was remarkable. So remarkable that the initial 1904 investigation by a team of academics concluded that it was not a trick. Three years later, however, a local psychologist, Oskar Pfungst, concluded that Hans was indeed clever, but was not doing mathematics. Instead, Hans had learned a “start” cue and a “stop” cue. The “start” cue was the act of being addressed by Von Osten who then looked down at his hooves. As Hans approached the proper answer, Von Osten, believing Hans was about to stop stomping, would make subtle straightening movements of the body and head. This served as the “stop” cue for the clever horse; his reward of food would be forthcoming. Hans usually landed on the right answer or very close to it. The normal variation of a horse periodically stomping its hooves was now being unintentionally managed by cues from the handler such that the variance of stomps less than or greater than the “right” number shrunk dramatically.

The same problem can take place when researchers are gathering data from other people. Numerous studies suggest that the hopes, expectations, and fears of data gatherers can subtly influence people’s responses (e.g., Nichols & Maner, 2008; Rosenthal & Fode, 1963; Rubin, Paolini, & Crisp, 2010); taking what might ordinarily be a very diffuse set of responses and pulling them tightly together around the researchers desired or expected response.

Demand characteristics and the role of the experimenter in the data gathering process are potential problems for many issues that social and behavioral scientists investigate. Even under the best of conditions, responses that may naturally be quite varied can begin to narrow around a certain response and mislead us about the true nature of reality. Perhaps those of us in the social and behavioral sciences need to take the human element into more careful consideration when gathering and interpreting data.

Find this and other essays regarding “Is the Scientific Method Broken?” in the Nesselroade & Grimm textbook.

Nichols, A. L., & Maner, J. K. (2008). The good subject effect: Investigating participant demand characteristics. Journal of General Psychology, 135, 151-165.

Pfungst, O. (1911). Clever Hans (The horse of Mr. Von Osten): A contribution to experimental animal and human psychology. New York: Holt and Co.

Rosenthal, R., & Fode, K. (1963). The effect of experimenter bias on performance of the albino rat. Behavioral Science, 8, 183-189.

Rubin, M., Paolini, S., & Crisp, R. (2010). A processing fluency explanation of bias against migrants Journal of Experimental Social Psychology, 46(1), 21-28.