|
|
|
|
HalfMooner
Dingaling
Philippines
15831 Posts |
 Posted - 07/06/2006 : 16:00:44
|
This is an opinion essay, published in PLoS Medicine. On the face of it, this is damning stuff.
What is your opinion?quote:
Why Most Published Research Findings Are False
John P. A. Ioannidis
[John P. A. Ioannidis is in the Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece, and Institute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts-New England Medical Center, Tufts University School of Medicine, Boston, Massachusetts, United States of America. E-mail: jioannid@cc.uoi.gr]
Summary
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
. . .
[Copyright: © 2005 John P. A. Ioannidis. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.]
[My emphasis.]
The rest of the essay is here
|
“Biology is just physics that has begun to smell bad.” —HalfMooner
Here's a link to Moonscape News, and one to its Archive. |
Edited by - HalfMooner on 07/06/2006 16:01:27
|
|
|
Zebra
Skeptic Friend
USA
354 Posts |
Posted - 07/06/2006 : 19:49:02 [Permalink]
|
I'm not getting a good sense of what he means by "false" - this is about medical research, and very often those studies compare groups, for example taking a group of men who have known heart disease and treating half with high-dose statin of brand A and the other half with usual-dose statin of brand B, then concluding that brand A is better than brand B high-dose statin (of brand A) is better than usual-dose statin of brand B at achieving outcome X (fewer deaths, fewer hospitalizations, fewer cardiac "events" including heart attacks, whatever - depends what outcomes they looked at and what they found). One major (known) problem comes when people assume that those results must apply to other groups (men without heart disease, women, children, dogs) or to other interventions (bring LDL below 70, statin A must be better than statin C also, whatever), when that's just not what the study could possibly show. Also, as he says, the bias against publishing "negative" results - results that show no difference, & thus seem less interesting - that does also affect the availability of potentially really useful information.
Here's one of the rebuttal letters this generated: quote:
Truth, Probability, and Frameworks
Jonathan D. Wren
James T. Kirk: Harry lied to you, Norman. Everything Harry says is a lie. Remember that, Norman: Everything he says is a lie.
Harry Mudd: Now I want you to listen to me very carefully, Norman: I… am… lying.
—Star Trek, the episode “I, Mudd”
Although John P. A. Ioannidis [1] brings up several good points about over-reliance on formal—yet arbitrary—statistical cutoffs and bias against the reporting of negative results, his claim that most published research findings are false is somewhat paradoxical. Ironically, the truer his premise is, the less likely his conclusions are. He, after all, relies heavily on other studies to support his premise, so if most (i.e., greater than 50%) of his cited studies are themselves false (including the eight of 37 that pertain to his own work), then his argument is automatically on shaky ground. As mentioned in the PLoS Medicine Editorial [2], scientific studies don't offer truth, per se. Even when studies appear in the best journals, they offer probabilistic assertions. Ioannidis's statement that “the probability that a research finding is indeed true depends on the prior probability of it being true” [1] is really begging the question; this, after all, is the problem. We cannot know such probabilities a priori, and guessing at such probabilities and/or parameters (as he does in his single nucleotide polymorphism [SNP] association example) surely could not be less biased than any statistical test of significance. ...
and it closes with: quote:
If Norman, the android from Star Trek mentioned in the beginning quote, had been equipped with the capacity to evaluate statements within a framework, he never would have short-circuited as a result of Kirk's paradox. He could have entertained the possibility that either Kirk was lying about Harry or Harry's statement was incomplete (i.e., lying about what?) Similarly, repeatedly re-examining any particular finding to resolve the true/not true paradox via statistical arguments alone can short-circuit our patience. We should instead seek to identify the framework by which implications of the finding can be tested, and I would argue that the more important the finding, the more testable implications it has.
|
I think, you know, freedom means freedom for everyone* -Dick Cheney
*some restrictions may apply |
 |
|
|
Dave W.
Info Junkie
USA
26022 Posts |
Posted - 07/06/2006 : 21:02:21 [Permalink]
|
quote:
Originally posted by HalfMooner
What is your opinion?
My opinion is that you've found yet another article capable of destroying irony meters all over the world.
I've purposefully not read the full article, because Zebra's posting of a portion of Wren's rebuttal says quite a lot. And it leads me to formulate several possibilities regarding Ioannidis' article.
He's attempting to show that we cannot possibly know the Truth-with-a-capital-T, which is trivially true and thus his title is wrong. And I doubt many editors would let it through like that.
Or, he's trying to cast doubt on the currently "standard" p-value of 0.05 to tell us anything useful about medicine, which, given physicists' use of 0.001 as their "standard" is quite probably true. But, then one would expect him to spend a large amount of text talking about the trade-offs of the monetary expense of trials with large numbers of participants (to drop the p-value) with the benefits of getting some idea of whether a hypothesis is right or wrong. (Experiments in physics, since they usually are not life-threatening by nature, don't suffer the same trade-off quandry.)
Or, he's railing against medical dogma ("institutional bias"), which is always good but badly aimed if one is going to smear everyone equally, including oneself.
While the irony - as pointed out by Wren - is simply delicious, I do plan on reading Ioannidis' whole article and see if any of my hypotheses seem to fit the bill (I'd thought of a couple more, but forgot 'em already, dangit). |
- Dave W. (Private Msg, EMail)
Evidently, I rock!
Why not question something for a change?
Visit Dave's Psoriasis Info, too. |
|
|
|
 |
|
|
|
Topic Locked
