Saturday 29 June 2013

MIndless Statistics and Feynman's Conjecture

Gerd Gigerenzer is Director at the Max Planck Institute for Human Development and Director of the Harding Center for Risk Literacy in Berlin. His web page is HERE.

He is also the author of an entertaining paper from 2004 called Mindless Statistics (HERE).

The Abstract reads:
Statistical rituals largely eliminate statistical thinking in the social sciences. Rituals are indispensable for identification with social groups, but they should be the subject rather than the procedure of science. What I call the “null ritual” consists of three steps: (1) set up a statistical null hypothesis, but do not specify your own hypothesis nor any alternative hypothesis, (2) use the 5% significance level for rejecting the null and accepting your hypothesis, and (3) always perform this procedure. I report evidence of the resulting collective confusion and fears about sanctions on the part of students and teachers, researchers and editors, as well as textbook writers.
Gigerenzer takes apart what he calls the 'null ritual' that scientists are taught about in statistics lessons. In particular psychologists. 

One of the great pieces of evidence Gigerenzer presents is the result of a test that was set by Haller and Krauss (2002). In this test the researchers posed a question about null hypothesis testing to 30 statistics teachers, including professors of psychology, lecturers, and teaching assistants, 39 professors and lecturers of psychology (not teaching statistics), and 44 psychology students. Teachers and students were from the psychology departments at six German universities. Each statistics teacher taught null hypothesis testing, and each student had successfully passed one or more statistics courses in which it was taught. The question was followed by 6 statements and the people taking the test were asked to mark which of the statements they believed to be true or false.

In fact all 6 of the statements were false. But all 6 of the statements erred "in the same direction of wishful thinking: They make a p-value look more informative than it is".

The results of this study were presented by Gigerenzer:

Gigerenzer also goes on to quote Richard Feynman on hypothesis testing and states Feynman’s conjecture:
To report a significant result and reject the null in favor of an alternative hypothesis is meaningless unless the alternative hypothesis has been stated before the data was obtained.
And quotes Feynman's anecdotal story about his interaction with a psychology researcher at Princeton whilst he was a student.
And it’s a general principle of psychologists that in these tests they arrange so that the odds that the things that happen happen by chance is small, in fact, less than one in twenty. . . . And then he ran to me, and he said, “Calculate the probability for me that they should alternate, so that I can see if it is less than one in twenty.” I said, “It probably is less than one in twenty, but it doesn’t count.” He said, “Why?” I said, “Because it doesn’t make any sense to calculate after the event. You see, you found the peculiarity, and so you selected the peculiar case.” . . . If he wants to test this hypothesis, one in twenty, he cannot do it from the same data that gave him the clue. He must do another experiment all over again and then see if they alternate. He did, and it didn’t work. 

Feynman, R., 1998. The Meaning of it All: Thoughts of a Citizen-Scientist. Perseus Books, Reading, MA, pp. 80–81.
Haller, H., Krauss, S., 2002. Misinterpretations of significance: a problem students share with their teachers? Methods of Psychological Research—Online [On-line serial], 7, pp 1–20.

Tuesday 25 June 2013

Its white foot tripped, Then dropped from sight.

The wreck of a small boat on the Caldy shore this morning.

 Copyright M.G. Reed 2013

Thursday 20 June 2013

Ashmolean Mummy & Penicillin electron density plots

The Ashmolean museum in Oxford is just one of the delights of this brilliant city of learning. 

One of the unexpected exhibits is the gallery of Egyptian mummy's, some of which include elaborate body wrappings and painted portraits (HERE). 

In addition there is a mummy of a 2 year old boy. Next to the mummy and about life size, is a fantastic glass sculpture artwork by Angela Palmer. The 2000 year old mummy was scanned with a CT scanner to create a full 3D data set. Palmer then created a physical representation of this virtual data set by tracing details from the 3D data set onto a set of 111 sheets of glass. The sculpture now sits near to the boy and is a permanent part of the exhibition.

Image copyright Angela Palmer

The technique of drawing contours on parallel sheets of glass that Angela Palmer uses was inspired by her seeing the work of Nobel prize winning crystallographer Dorothy Hodgkin-Crowfoot. Below is an image of her work created in the mid 1940's (from HERE).  An original of this artefact is in the Oxford Museum of the History of Science (HERE).

From the 1949 penicillin monograph published by Princeton University Press. A photograph of electron-density contours drawn on sheets of perspex showing the thiazolidine and beta-lactam of potassium benzylpenicillin. 

Owing to the urgency created by the `extreme importance of penicillin as a military weapon' in WWII, determining the crystal structure of penicillin involved the first crystallographic use in Britain of a Hollerith computer; with Dorothy Hodgkin-Crowfoot as lead crystallographer and with Leslie J. Comrie's Scientific Computing Services Ltd developing the computing methodology. 

As noted in the penicillin monograph: `Under the terms of the contract for the publication of the Chemistry of Penicillin the publisher has agreed to waive its rights under the copyright after five years from the date of publication. Thereafter this volume will be in the public domain and dedicated to the public.' 

Thursday 13 June 2013

Data Analysis for Politics and Policy

Before Edward Tufte became very well known for his work on data visualisation, he wrote on the application of statistical methods in the social sciences. 

One of his best books is a slim volume published in 1974 by Prentice Hall, Data Analysis for Politics and Policy.  Even if you have no interest in either Politics or Policy, this is a great book for learning about data analysis. 

The book was previously available to download for free from Tufte's site, but is now available there for $2 as an e-book in PDF format (HERE)

Below is one of the figures from the book, which shows a data set on US death rates from motor vehicle accidents that Tufte analyses in some detail through the book. The plot shows the distribution in death rate across each of the states in the US. The graphic doesn't reach the high standards that he set himself later in his career, but it already shows a dedication to showing the data in a clear visual manner. 

Images Copyright E.Tufte.

Sunday 2 June 2013

On the unreliability of (some) neuroscience

Raymond Tallis has a superb piece in today's Guardian on the ludicrousness of some of the claims that are made by those who use functional brain imaging to try and explain that who we are and what we do is somehow revealed by the particular bits of the brain that 'light up' in big imaging bits of kit. HERE.

In the piece he references a Nature Neuroscience Review paper by Katherine Button and friends. The abstract reads as follows:

A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles.
 See  also previous posts HERE and HERE.