Gerd Gigerenzer is Director at the Max Planck Institute for Human Development and Director of the Harding Center for Risk Literacy in Berlin. His web page is HERE.
He is also the author of an entertaining paper from 2004 called Mindless Statistics (HERE).
The Abstract reads:
Statistical rituals largely eliminate statistical thinking in the social sciences. Rituals are indispensable for identification with social groups, but they should be the subject rather than the procedure of science. What I call the “null ritual” consists of three steps: (1) set up a statistical null hypothesis, but do not specify your own hypothesis nor any alternative hypothesis, (2) use the 5% significance level for rejecting the null and accepting your hypothesis, and (3) always perform this procedure. I report evidence of the resulting collective confusion and fears about sanctions on the part of students and teachers, researchers and editors, as well as textbook writers.
Gigerenzer takes apart what he calls the 'null ritual' that scientists are taught about in statistics lessons. In particular psychologists.
One of the great pieces of evidence Gigerenzer presents is the result of a test that was set by Haller and Krauss (2002). In this test the researchers posed a question about null hypothesis testing to 30 statistics teachers, including professors of psychology, lecturers, and teaching assistants, 39 professors and lecturers of psychology (not teaching statistics), and 44 psychology students. Teachers and students were from the psychology departments at six German universities. Each statistics teacher taught null hypothesis testing, and each student had successfully passed one or more statistics courses in which it was taught. The question was followed by 6 statements and the people taking the test were asked to mark which of the statements they believed to be true or false.
In fact all 6 of the statements were false. But all 6 of the statements erred "in the same direction of wishful thinking: They make a p-value look more informative than it is".
The results of this study were presented by Gigerenzer:
Gigerenzer also goes on to quote Richard Feynman on hypothesis testing and states Feynman’s conjecture:
To report a significant result and reject the null in favor of an alternative hypothesis is meaningless unless the alternative hypothesis has been stated before the data was obtained.
And quotes Feynman's anecdotal story about his interaction with a psychology researcher at Princeton whilst he was a student.
And it’s a general principle of psychologists that in these tests they arrange so that the odds that the things that happen happen by chance is small, in fact, less than one in twenty. . . . And then he ran to me, and he said, “Calculate the probability for me that they should alternate, so that I can see if it is less than one in twenty.” I said, “It probably is less than one in twenty, but it doesn’t count.” He said, “Why?” I said, “Because it doesn’t make any sense to calculate after the event. You see, you found the peculiarity, and so you selected the peculiar case.” . . . If he wants to test this hypothesis, one in twenty, he cannot do it from the same data that gave him the clue. He must do another experiment all over again and then see if they alternate. He did, and it didn’t work.
References
Feynman, R., 1998. The Meaning of it All: Thoughts of a Citizen-Scientist. Perseus Books, Reading, MA, pp. 80–81.
Haller, H., Krauss, S., 2002. Misinterpretations of significance: a problem students share with their teachers? Methods of Psychological Research—Online [On-line serial], 7, pp 1–20.
