Monday, January 3, 2011

Why Science Rules (Even When it Sucks)

New Yorker Magazine published an article last month that on its surface seems like a resounding condemnation of science, the scientific method and the scientists who purport to follow it. It seems that in spite of the claims that science is neutral and objective, the pressures of money, fame and egotism still create bad science ... a lot of bad science. And we're not talking about trivial stuff. This affects people's lives, fortunes and health.

Author Jonah Lehrer's article The Truth Wears Off reveals some shockingly bad science:
  • The drugs Abilify, Seroquel, and Zyprexa were shown to dramatically improve the condition of schizophrenics and are now heavily prescribed. But follow-up studies simply haven't been able to reproduce the original results. The drugs are effective, but not to the extent originally claimed.
  • Zoologists showed that asymmetry in birds was strongly linked to the number of genetic mutations in the birds, and others went on to find similar results many other species including flies and even humans. But later studies weren't able to reproduce the results. It was as though the asymmetry effect had evaporated.
  • Dozens of acupuncture studies in Japan and China, conducted by reputable scientists and reviewed by qualified peers, showed 100% positive results. But the same studies, when conducted by American scientists, were ambivalent; only 56% of the studies showed positive results.
  • Out of 432 scientific claims about a link between genetics and a mental disease, only one was consistently replicable. The rest were junk science and wishful thinking.
The list goes on and on. It's enough to make reasonable people question and even reject science entirely.

According to Lehrer, the problem seems to be that science heavily favors positive results, to the point where science has trouble staying honest.

It starts with the scientists. There are no accolades in saying you didn't discover something. Nobody ever got famous for not discovering DNA, or not inventing a drug, or not figuring why people who leave a light on at night are nearsighted. Scientists get famous for positive, dramatic results.

And the scientific journals, in spite of their peer-review process, are heavily biased toward publishing exciting new results. When they publish a dull non-discovery nobody reads it, nobody cites the article, and nobody tries to replicate the dull results.

These two effects seem to be multiplicative. The scientists unconsciously bias their results to positive, exciting new discoveries, and the publications skim the very most fascinating and exciting discoveries off the top. Out of the millions of studies done each year by reputable, honest scientists, there are bound to be stasitical anomalies, errors and results that are confounded by unexpected side effects. But these are the very results that the positive-results bias pushes to the front.

But who is trying to fix this problem? Who is holding scientists' feet to the fire? Is it politicians? Is it the young-earth creationists or even the intelligent-design proponents who are always looking for the flaws in modern science? Is it the homeopathics and herbal remedy manufacturers? No, no and no. It is the scientists themselves.

This is where science itself comes to the rescue. Scientists understand that they have a problem, and they are the ones working hardest to fix it. As the great physicist Richard Feynman famously said:
"The first principle [of science] is that you must not fool yourself – and you are the easiest person to fool."
And that, in a nutshell, is what the scientists are now doing.

It's the scientists who discovered the problem, and scientists who will change their own culture to try to overcome the bias that they discovered. Because that's how science works. You can fool yourself and your colleagues for a while, but sooner or later the scientific method works. The bad data just don't add up and can't stand the test of time. The scientists take another look, discover why they were wrong the first time, and fix it. As Carl Sagan said:
In science it often happens that scientists say, 'You know that's a really good argument; my position is mistaken,' and then they actually change their minds and you never hear that old view from them again. ... It happens every day. I cannot recall the last time something like that happened in politics or religion.

4 comments:

  1. Today's Google quote of the day: "I never guess. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts." - Sir Arthur Conan Doyle

    The great thing about science is that over time, results improve as a result of the scientific method. Even if a few studies lead us astray, others will bring us back to the correct conclusion eventually.

    Unlike other schools of "thought."

    I finished and enjoyed your book. The way it really picked up momentum once the core ideas were in place meant that the back half especially was surprisingly thrilling for a non-fiction work.

    ReplyDelete
  2. All of those examples are clearly science in action. Science continues to test and re-test it's data because we can never be positive that things might not change. It's consistency in the testing that matters, not single results. We still test gravity, just to make sure nothing has changed.

    The only people who would start questioning science over results like those are people who don't understand science to begin with.

    ReplyDelete
  3. JonL – great quote, I hadn't heard that one before. And thanks for your kind words about The Religion Virus. That's what makes the whole effort worthwhile.

    Cephus – That's really the core message of science, isn't it? Scientists are never above questioning their own truths, but at the same time, they understand which truths are solid and which ones are still being established.

    ReplyDelete
  4. I have a feeling that the dangerous and ineffective drugs going to market is more to do with an employer's pressures to get a product out there than with the science.

    ReplyDelete

Dear readers -- I am no longer blogging and after leaving these blogs open for two years have finally stopped accepting comments due to spammers. Thanks for your interest. If you'd like to write to me, click on the "Contact" link at the top. Thanks! -- CJ.

Note: Only a member of this blog may post a comment.