Jonathan Lehrer
13 December 2010
The test of replicability, as it's known, is the foundation of modern research. It's a safeguard for the creep of subjectivity.
Francis Bacon, early modern philosopher and pioneer of the scientific method, declared that experiments allowed us to "put nature into question". But it appears that nature often gives us different answers.
Jonathan Schooler - University of Washington graduate in the 1980s.
- It was believed that the act of describing our memories improved them.
- Verbal Overshadowing - Schooler demonstrated that subjects shown and asked to describe a face were much less likely to recognize the face when shown it later than those who had simply looked at it.
- As he kept replicating these studies later on, there was a 30% decrease in results confirmed by other researchers.
Joseph Rines at Duke
- performed a similar experiment with ESP and trying to recognize cards. Most subjects correctly picked out 25% of 25 cards. One person consistently picked out 50%.
- Once there was a stack of 1000 cards, this subject's success rate was barely above chance.
- Rhine called this trend the decline effect which is more or less, a regression to the mean as statistical flukes get cancelled out.
After a new scientific paradigm is proposed, the peer review process is tilted toward supporting the earlier results. But after a few years, the academic incentives shift so that the most notable results are those that disprove the theory.
Michael Jennions at Australian National University argues that the decline effect is largely a product of publication bias, of the tendency of scientists to prefer positive data over null results.
Richard Palmer, a biologist at the University of Alberta
- suspects that an equally significant issue is the selective reporting of results.
- When a large number of studies with a large number of sample size results should cluster around a common value and those with a small sample size should be scattered randomly.
- When Palmer studied the results of fluctuating asymmetry, which had a small sample size, he found that the results skewed towards positive results.
Acupuncture studies in Asian countries always concluded a positive result and that it was effective. During the same period, there were studies in Western countries that found only 56% proved therapeutic. Palmer suggests that scientists find ways to confirm their preferred hypothesis.
Decline Effect
- Human fallibility of science, in which data is tweaked and beliefs shape perception.
- Many of our most exciting theories may just be fads.
- Reminds us of how difficult it is to prove anything.
- Just because an idea is true doesn't mean it can be proved and just because an idea can be proved, doesn't mean it's true.
