|
|
|
markie
Skeptic Friend
Canada
356 Posts |
 Posted - 09/16/2005 : 12:40:33
|
Note by me: This is extracted from the Public Libary of Science, Medicine, and so is more in the context of medical studies, but it has general application I think to some other areas of science.
The full article is at
http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0020124
and provides enough detail to cause blurryeyedosis to all but the most interested of readers :)
Mark
quote:
Why Most Published Research Findings Are False
John P. A. Ioannidis
Summary
There is increasing concern that most current published research findings are false. ... In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. ... There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims [6–8]. However, this should not be surprising. It can be proven that most claimed research findings are false. Here I will examine the key factors that influence this problem and some corollaries thereof.
|
|
|
H. Humbert
SFN Die Hard
USA
4574 Posts |
Posted - 09/16/2005 : 14:41:14 [Permalink]
|
If most published studies are false, then odds are that this study is also false.
|
"A man is his own easiest dupe, for what he wishes to be true he generally believes to be true." --Demosthenes
"The first principle is that you must not fool yourself - and you are the easiest person to fool." --Richard P. Feynman
"Face facts with dignity." --found inside a fortune cookie |
| Edited by - H. Humbert on 09/16/2005 14:41:31 |
 |
|
|
marfknox
SFN Die Hard
USA
3739 Posts |
Posted - 09/16/2005 : 16:16:53 [Permalink]
|
If most published studies are false, then odds are that this study is also false.
This isn't a study. This is a critical analysis that supports the claim that most published research findings are false and includes proposals on how to improve the situation. |
"Too much certainty and clarity could lead to cruel intolerance" -Karen Armstrong
Check out my art store: http://www.marfknox.etsy.com
|
|
|
|
H. Humbert
SFN Die Hard
USA
4574 Posts |
Posted - 09/16/2005 : 16:45:55 [Permalink]
|
quote:
Originally posted by marfknox
If most published studies are false, then odds are that this study is also false.
This isn't a study. This is a critical analysis that supports the claim that most published research findings are false and includes proposals on how to improve the situation.
Main Entry: 1study
Pronunciation: 'st&-dE
Function: noun
Inflected Form(s): plural stud·ies
Etymology: Middle English studie, from Old French estudie, from Latin studium, from studEre to devote oneself, study; probably akin to Latin tundere to beat -- more at CONTUSION
1 : a state of contemplation : REVERIE
2 a : application of the mental faculties to the acquisition of knowledge <years of study> b : such application in a particular field or to a specific subject <the study of Latin> c : careful or extended consideration <the proposal is under study> d (1) : a careful examination or analysis of a phenomenon, development, or question (2) : the published report of such a study
|
"A man is his own easiest dupe, for what he wishes to be true he generally believes to be true." --Demosthenes
"The first principle is that you must not fool yourself - and you are the easiest person to fool." --Richard P. Feynman
"Face facts with dignity." --found inside a fortune cookie |
| Edited by - H. Humbert on 09/16/2005 16:49:50 |
|
|
|
Hawks
SFN Regular
Canada
1383 Posts |
Posted - 09/16/2005 : 18:20:02 [Permalink]
|
The article mentions a few biases which might influence the validity of research and I think we should be aware of these.. The significance of most studies is only evaluated using the p-value (type I errors - false positives - the chance that a significant result is achieved although in reality it wasn't significant), while totally omitting to present the power of the experiment (type II errors - false negatives - the chance of a reaching a non-significant result which in fact really is significant). The extent to which this actually contributes to published research results being wrong is questionable, however, since journals generally prefer to publish positive rather than negative results (ie, there is a bias against type II errors occuring in journals).
Other biases such as poorly conducted experiments and deliberately/undeliberatly omitting/misinterpreting data to achieve more significant results do unfortunately happen. Some blatant crap, such as that advocated here: http://www.skepticfriends.org/forum/topic.asp?TOPIC_ID=4946, fortunately never gets published in reputed journals, but scientific fraud and omissions will also happen with "conventional" research. That's human nature for you. What can be done about it? Well, conduct more science. Science is self-correcting and as the evidence accumulates, confounding factors are identified and frauds are exposed, we are continually more likely to "be right" (or less likely to be wrong).
|
METHINKS IT IS LIKE A WEASEL
It's a small, off-duty czechoslovakian traffic warden! |
|
|
|
marfknox
SFN Die Hard
USA
3739 Posts |
Posted - 09/16/2005 : 22:37:56 [Permalink]
|
Humbert,
While I stand corrected - this article is indeed a study - if you read the title and the first sentence of it, it isn't saying that most studies are false. It says that most "published research findings" are false. And the things it critizes them for mostly don't apply to the type of study that it is. |
"Too much certainty and clarity could lead to cruel intolerance" -Karen Armstrong
Check out my art store: http://www.marfknox.etsy.com
|
|
|
|
H. Humbert
SFN Die Hard
USA
4574 Posts |
Posted - 09/16/2005 : 23:20:41 [Permalink]
|
quote:
Originally posted by marfknox
Humbert,
While I stand corrected - this article is indeed a study - if you read the title and the first sentence of it, it isn't saying that most studies are false. It says that most "published research findings" are false. And the things it critizes them for mostly don't apply to the type of study that it is.
Yes, but what you don't understand is that Markie has a history here of denigrating "know-it-all" scientists and skeptics who are too blinded by bias to acknowledge the validity of his personal religious convictions.
My response, which was simply an attempt at humor, was prejudiced by these previous exchanges. I was not about to indulge Markie in another round of science-bashing.
However, Hawks makes an excellent point. The only real solution to the problem of bias in science is to conduct more science.
|
"A man is his own easiest dupe, for what he wishes to be true he generally believes to be true." --Demosthenes
"The first principle is that you must not fool yourself - and you are the easiest person to fool." --Richard P. Feynman
"Face facts with dignity." --found inside a fortune cookie |
| Edited by - H. Humbert on 09/16/2005 23:34:26 |
|
|
|
markie
Skeptic Friend
Canada
356 Posts |
Posted - 09/17/2005 : 05:55:04 [Permalink]
|
quote:
Originally posted by H. Humbert
quote:
Originally posted by marfknox
Humbert,
While I stand corrected - this article is indeed a study - if you read the title and the first sentence of it, it isn't saying that most studies are false. It says that most "published research findings" are false. And the things it critizes them for mostly don't apply to the type of study that it is.
Yes, but what you don't understand is that Markie has a history here of denigrating "know-it-all" scientists and skeptics who are too blinded by bias to acknowledge the validity of his personal religious convictions.
My response, which was simply an attempt at humor, was prejudiced by these previous exchanges. I was not about to indulge Markie in another round of science-bashing.
However, Hawks makes an excellent point. The only real solution to the problem of bias in science is to conduct more science.
Humbert, the same humourous side of it crossed my mind as well as soon as I read the article summary. Sort of like one of those old logic teasers which goes something like, "a Cretan philosopher once opined that all Cretans were liars". Similarly, was this study's results also incorrect? (short answer, no.)
I wasn't sure, however, if your response was meant in humour or not. I guess I've been getting conditioned to your replies being, as I perceive them, somewhat hostile.
I *love* science, I really do, and I wish I could convey my deep feelings that science can truly uncover the descriptive truth of how the material mechanism of the universe works. I merely deny that the universe is nothing but material mechanism, while obviously some skeptics and scientists have the opposite 'belief'. If I sometimes point out the failings of skeptics and science (and I consider myself both a skeptical and scientific type), just consider it an attempt for some balance, nothing personal.
As Hawks and Marfknox point out, the article does present constructive ways to fix a very real problems in the science methodology of looking for 'significance'. What we need imo is more science, sure, but science which is more correct.
Mark
|
|
|
|
R.Wreck
SFN Regular
USA
1191 Posts |
Posted - 09/17/2005 : 07:55:09 [Permalink]
|
quote:
Originally posted by markie:
I merely deny that the universe is nothing but material mechanism, while obviously some skeptics and scientists have the opposite 'belief'.
There is no good evidence for the universe being anything other than "material mechanism". Therefore that is a resonable conclusion, not a "belief", regarding the universe.
quote:
belief
the feeling of certainty that something exists or is true:
All non-violent religious and political beliefs should be respected equally.
[+ that] It is my (firm) belief that nuclear weapons are immoral.
His belief in God gave him hope during difficult times.
Recent revelations about corruption have shaken many people's belief in (= caused people to have doubts about) the police.
The brutality of the murders was beyond belief (= too difficult to be imagined).
He called at her house in the belief that (= confident that) she would lend him the money.
|
The foundation of morality is to . . . give up pretending to believe that for which there is no evidence, and repeating unintelligible propositions about things beyond the possibliities of knowledge.
T. H. Huxley
The Cattle Prod of Enlightened Compassion
|
|
|
|
markie
Skeptic Friend
Canada
356 Posts |
Posted - 09/17/2005 : 09:28:58 [Permalink]
|
quote:
Originally posted by markie
I merely deny that the universe is nothing but material mechanism, while obviously some skeptics and scientists have the opposite 'belief'.[quote] [quote]Originally posted by R. Wreck
There is no good evidence for the universe being anything other than "material mechanism". Therefore that is a resonable conclusion, not a "belief", regarding the universe.
Hmmm, Wreck it's a little fishy: The only evidence you give credence to concerns material mechanism, and then you conclude that material mechanism is all that there is.
I'll tell you what. When you can logically derive, or even approximate, from a set of mechanistic first principles, all the phenomena in the universe today (including life and consciousness) then you will have shown that your conclusion of a strictly materialistic universe is indeed quite 'reasonable'. Until then, I feel quite justified in calling it a belief.
Mark
|
|
|
|
R.Wreck
SFN Regular
USA
1191 Posts |
Posted - 09/17/2005 : 09:59:40 [Permalink]
|
quote:
Originally posted by markie:
--------------------------------------------------------------------------------
Originally posted by markie
I merely deny that the universe is nothing but material mechanism, while obviously some skeptics and scientists have the opposite 'belief'.
Originally posted by R. Wreck
There is no good evidence for the universe being anything other than "material mechanism". Therefore that is a resonable conclusion, not a "belief", regarding the universe.
--------------------------------------------------------------------------------
Hmmm, Wreck it's a little fishy: The only evidence you give credence to concerns material mechanism, and then you conclude that material mechanism is all that there is.
What other credible evidence is there?
quote:
I'll tell you what. When you can logically derive, or even approximate, from a set of mechanistic first principles, all the phenomena in the universe today (including life and consciousness) then you will have shown that your conclusion of a strictly materialistic universe is indeed quite 'reasonable'. Until then, I feel quite justified in calling it a belief.
That's a bit of a strawman. I didn't say that we were able to explain "all the phenomena in the universe" via any mechanism. We don't even know what "all the phenomena in the universe" are. What I said was that the evidence we have is consistent with the universe being a material mechanism and nothing more.
But since you bring it up, science has consistently used materialistic, mechanistic evidence, and reason, to explain many phenomena that were previously believed to be some manifestation of the supernatural. Based on the track records of materialistic science and supernatural belief, it is reasonable to conclude that if an explanation for a phenomenon is to be found, it will be through science rather than belief.
(edited to correct formatting) |
The foundation of morality is to . . . give up pretending to believe that for which there is no evidence, and repeating unintelligible propositions about things beyond the possibliities of knowledge.
T. H. Huxley
The Cattle Prod of Enlightened Compassion
|
| Edited by - R.Wreck on 09/17/2005 11:14:07 |
|
|
|
marfknox
SFN Die Hard
USA
3739 Posts |
Posted - 09/17/2005 : 18:03:28 [Permalink]
|
OK, I think markie is right when he says that claiming there is only the material universe is a "belief". I don't think beliefs are inherently bad - we all must have beliefs when we lack sufficient information to know.
That said, most nontheists are agnostics - they admit a possibility of more than the material, but shrug their shoulders in the absence of any evidence. Sure, there could be metaphysical stuff. But apparently we can't percieve it in any verifiable and objective way, so why bother pondering it? How much can you ponder a vague notion that you have zero information about?
When people start describing these supposed metaphysical realities (usually through theology) they have leaped across a chasm that skeptics just won't leap across, and for good reason. My theological ramblings pretty much stop at: If there is a God that gives all humans some sort of purpose, apparently that God wanted me to be an atheist. |
"Too much certainty and clarity could lead to cruel intolerance" -Karen Armstrong
Check out my art store: http://www.marfknox.etsy.com
|
| Edited by - marfknox on 09/17/2005 18:05:44 |
|
|
|
markie
Skeptic Friend
Canada
356 Posts |
Posted - 09/17/2005 : 20:12:35 [Permalink]
|
Marfknox and Wrecks bring up good points which however are going off the topic of the thread. (Me bad, since I think I veered it off in that direction.) Perhaps I shall pick it up in the religion section, a section I've tried my best to ignore recently.
Anyways, I have a better idea now just why it is that scientists can claim, say, something about the health effects of food X one year and the next year come up with what seems an entirely opposite conclusion. It generates an incredible waste of time and energy for all parties.
I'm going to have to brush up on my statistics to more fully appreciate the article, but it appears that most fully functioning and publishing scientists are, more than anything, in need of having a fuller grasp on statistical concepts so they can be wiser in their mathematical analysis and conclusions. As it is, a little bit of knowledge can be maddeningly misleading.
Mark
|
|
|
|
Hawks
SFN Regular
Canada
1383 Posts |
Posted - 09/17/2005 : 20:53:58 [Permalink]
|
quote:
Originally posted by markie
Anyways, I have a better idea now just why it is that scientists can claim, say, something about the health effects of food X one year and the next year come up with what seems an entirely opposite conclusion.
"Bad" science is certainly part of the problem, but even in the absense of this, papers with "faulty" conclusions will still be published. Considering that we don't (and can't) know everything, we can't control for all confounding factors when conducting our research. Therefore, any conclusions we draw will be tentative and subject to revision.
quote:
...it appears that most fully functioning and publishing scientists are, more than anything, in need of having a fuller grasp on statistical concepts so they can be wiser in their mathematical analysis and conclusions.
One of my old professors even went so far as to say that experiments should be designed so that statistics should not have to be used at all when interpreting the results. Ie, the results should be clear-cut every time. In a perfect world we would have the resources (time and money) to do this. |
METHINKS IT IS LIKE A WEASEL
It's a small, off-duty czechoslovakian traffic warden! |
|
|
|
marfknox
SFN Die Hard
USA
3739 Posts |
Posted - 09/18/2005 : 01:07:32 [Permalink]
|
Part of the problem isn't even the scientists. It's the damn media and our poorly educated masses. Scientists do a study on something that has broad interest, the media picks it up and makes a big hubbub of it, and most people buy into it as confirmed fact, rather than an addition to a growing body of knowledge that has the potential to be flawed or flat out wrong. And then if it is later proven wrong, really stupid people stop trusting science as if it was the fault of the scientific method and not peoples' biases and mistakes.
Thus, I agree with and happily repeat Humbert's sentiment: "However, Hawks makes an excellent point. The only real solution to the problem of bias in science is to conduct more science." |
"Too much certainty and clarity could lead to cruel intolerance" -Karen Armstrong
Check out my art store: http://www.marfknox.etsy.com
|
|
|
|
Dude
SFN Die Hard
USA
6891 Posts |
Posted - 09/18/2005 : 21:49:27 [Permalink]
|
quote:
OK, I think markie is right when he says that claiming there is only the material universe is a "belief".
markie has been down this road with us before... and he still doesn't comprehend.
Nobody would rationally make the claim that the material universe is "all there is", unless there was some evidence to prove the claim.
Ignorant people take that and jump over a cliff with it though. They use that statement as evidence for something beyond the material universe. Which is a grevious error of logic on their part. But you just can't communicate with people who cling tightly to that irrational ignorance. Its like taking a security blanket away from a child, you get no rational response from them.
|
Ignorance is preferable to error; and he is less remote from the truth who believes nothing, than he who believes what is wrong.
-- Thomas Jefferson
"god :: the last refuge of a man with no answers and no argument." - G. Carlin
Hope, n.
The handmaiden of desperation; the opiate of despair; the illegible signpost on the road to perdition. ~~ da filth |
|
|
|
 |
|
|
|
Topic Locked
