|
|
|
JustMe
Skeptic Friend
64 Posts |
 Posted - 03/06/2009 : 12:51:09
|
The 'Black Box' of Peer Review
March 4, 2009
Countless decisions in academe are based on the quest for excellence. Which professors to hire and promote. Which grants to fund. Which projects to pursue. Everyone wants to promote excellence. But what if academe actually doesn't know what excellence is?
Michèle Lamont decided to explore excellence by studying one of the primary mechanisms used by higher education to -- in theory -- reward excellence: scholarly peer review. Applying sociological and other disciplinary approaches to her study, Lamont won the right to observe peer review panels that are normally closed to all outsiders. And she was able to interview peer review panelists before and after their meetings, examine notes of reviewers before and after decision-making meetings, and gain access to information on the outcomes of these decisions.
The result is How Professors Think: Inside the Curious World of Academic Judgment (Harvard University Press), which aims to expose what goes on behind the closed doors where funds are allocated and careers can be made. For those who have always wondered why they missed out on that grant or fellowship, the book may or may not provide comfort. Lamont describes processes in which most peer reviewers take their responsibilities seriously, and devote considerable time and attention to getting it right.
She also finds plenty of flaws -- professors whose judgment on proposals is clouded by their own personal interests, deal making among panelists to make sure decisions are made in time for panelists to catch their planes, and an uneven and somewhat unpredictable efforts by panelists to reward personal drive and determination over qualities that a grant program says are the actual criteria.
Full article: http://www.insidehighered.com/news/2009/03/04/peerreview
|
|
|
Dave W.
Info Junkie
USA
26022 Posts |
Posted - 03/06/2009 : 13:53:39 [Permalink]
|
The 'Black Box' of Peer Review... of grant applications.
That's an important distinction, which will probably be lost on much of the anti-science crowd, who are going to read the first few paragraphs and run with this as evidence as to why their articles have been rejected by peer reviewers for journals like Science and Nature. |
- Dave W. (Private Msg, EMail)
Evidently, I rock!
Why not question something for a change?
Visit Dave's Psoriasis Info, too. |
 |
|
|
JustMe
Skeptic Friend
64 Posts |
Posted - 03/06/2009 : 15:34:18 [Permalink]
|
I think this study formalizes something that is intuitively predictable. Systems and processes that involve humans are vulnerable to human nature. Even science. I was on a cosmology reading binge a year or two ago and stumbled on a book called 'Hunting Down the Universe' that had a lot of thought provoking material on some of the pressures that can steer scientific debate and progress. The following review is an accurate reflection of the content of the book, and though I can't speak to the objective veracity of that content, it was certainly compelling and very interesting to contemplate.
http://www.informatics.sussex.ac.uk/research/groups/nlp/gazdar/teach/atc/1998/revman/lawford.html |
|
|
|
tomk80
SFN Regular
Netherlands
1278 Posts |
Posted - 03/06/2009 : 18:50:59 [Permalink]
|
Originally posted by JustMe
I think this study formalizes something that is intuitively predictable. Systems and processes that involve humans are vulnerable to human nature. Even science. I was on a cosmology reading binge a year or two ago and stumbled on a book called 'Hunting Down the Universe' that had a lot of thought provoking material on some of the pressures that can steer scientific debate and progress. The following review is an accurate reflection of the content of the book, and though I can't speak to the objective veracity of that content, it was certainly compelling and very interesting to contemplate.
http://www.informatics.sussex.ac.uk/research/groups/nlp/gazdar/teach/atc/1998/revman/lawford.html
|
You didn't really answer Dave W.'s point with this. The article concerns grant applications. This means it handles the directions scientific investigation takes, not the outcomes of the investigation.
Now, I am very aware that all peer-reviewers have their own biases in that as well, but what do you think the result of those biases is on the validity of the research? |
Tom
`Contrariwise,' continued Tweedledee, `if it was so, it might be; and if it were so, it would be; but as it isn't, it ain't. That's logic.'
-Through the Looking Glass by Lewis Caroll- |
|
|
|
Dave W.
Info Junkie
USA
26022 Posts |
Posted - 03/06/2009 : 19:26:00 [Permalink]
|
Originally posted by tomk80
You didn't really answer Dave W.'s point with this. |
I wasn't actually expecting an answer, my reply was more-or-less just a prediction of what the nutbags of the world are going to do with the OP's book review (without having read the book, or even the whole review, of course).| The article concerns grant applications. This means it handles the directions scientific investigation takes, not the outcomes of the investigation. |
And judging from the comments on the review, it's not even the "hard" sciences that were studied, but "soft" sciences and even liberal-arts grants. But apparently there have been other studies of, say, the NIH granting process. |
- Dave W. (Private Msg, EMail)
Evidently, I rock!
Why not question something for a change?
Visit Dave's Psoriasis Info, too. |
|
|
|
tomk80
SFN Regular
Netherlands
1278 Posts |
Posted - 03/07/2009 : 05:08:27 [Permalink]
|
Originally posted by Dave W.
Originally posted by tomk80
You didn't really answer Dave W.'s point with this. |
I wasn't actually expecting an answer, my reply was more-or-less just a prediction of what the nutbags of the world are going to do with the OP's book review (without having read the book, or even the whole review, of course).| The article concerns grant applications. This means it handles the directions scientific investigation takes, not the outcomes of the investigation. |
And judging from the comments on the review, it's not even the "hard" sciences that were studied, but "soft" sciences and even liberal-arts grants. But apparently there have been other studies of, say, the NIH granting process.
|
Looking at the granting processes I've seen, I'd say there is indeed a matter of subjectivity, if not some kind of partisanship. This is because a number of reviewers (not all scientists) need to decide on the question "Is this subject worth studying and is the study design and structure sufficient to do so?" This is an open-ended question to which different scientists and non-scientists have different answers. That's the whole point of what makes science, that different people have different answers to the same question and that this can be discussed. With grants this, as well as the politics (in the end the money comes from politicians) will play a role in the process.
But there is a huge difference between this and commenting on an article in peer-review, where the study design and direction of study already are determined on previously. There the question no longer is "what knowledge do we expect to gain from this study" but "is the study performed and reported accurately and do the conclusions of the study logically follow from the results obtained in it". |
Tom
`Contrariwise,' continued Tweedledee, `if it was so, it might be; and if it were so, it would be; but as it isn't, it ain't. That's logic.'
-Through the Looking Glass by Lewis Caroll- |
|
|
|
Simon
SFN Regular
USA
1992 Posts |
Posted - 03/07/2009 : 10:15:15 [Permalink]
|
Originally posted by tomk80
This is because a number of reviewers (not all scientists) need to decide on the question "Is this subject worth studying and is the study design and structure sufficient to do so?"
|
I am not sure I agree with that.
As far as I know, everybody on a board is a scientist and his field should be that of the grant they cover (no astrophysicists to give out microbiology grants, for example), in fact, it is common practice for the board members to have received grants from the board in the previous years.
But, of course, the system is not perfect.
They are some subjectivity (is it worth investigating, is it really within the -often very precise- framework of this particular funding) as well as human factors (the process is not anonymous).
The boards can also make technical mistake, I remember them refusing to fund a project because it was not technically feasible, and then, the researcher achieving the infeasible a few months later.
But all in all, despite its shortcoming becoming more clear cuts in the recent years, when the lack of funding forced them to reject a much higher proportion of projects, I think that funding agencies are doing a pretty good job.
I, for one, certainly can not think of a better system. |
Look again at that dot. That's here. That's home. That's us. On it everyone you love, everyone you know, everyone you ever heard of, every human being who ever was, lived out their lives. The aggregate of our joy and suffering, thousands of confident religions, ideologies, and economic doctrines, every hunter and forager, every hero and coward, every creator and destroyer of civilization, every king and peasant, every young couple in love, every mother and father, hopeful child, inventor and explorer, every teacher of morals, every corrupt politician, every "superstar," every "supreme leader," every saint and sinner in the history of our species lived there – on a mote of dust suspended in a sunbeam.
Carl Sagan - 1996 |
|
|
|
Dave W.
Info Junkie
USA
26022 Posts |
Posted - 03/07/2009 : 10:34:23 [Permalink]
|
Originally posted by Simon
I, for one, certainly can not think of a better system. |
One of the commenters to the original review wrote, in part:Often in peer review panels I have been reminded of Winston Churchill's characterization of deomcracy, adjusted to the current situation: peer review as the worst form of selecting grant recipients, except for all the others. |
- Dave W. (Private Msg, EMail)
Evidently, I rock!
Why not question something for a change?
Visit Dave's Psoriasis Info, too. |
|
|
|
Ricky
SFN Die Hard
USA
4907 Posts |
|
|
Zebra
Skeptic Friend
USA
354 Posts |
Posted - 03/07/2009 : 15:29:54 [Permalink]
|
Originally posted by Dave W.
The 'Black Box' of Peer Review... of grant applications.
That's an important distinction, which will probably be lost on much of the anti-science crowd, who are going to read the first few paragraphs and run with this as evidence as to why their articles have been rejected by peer reviewers for journals like Science and Nature.
|
I'll just add my 2 cents about peer review for journals, from personal experience of having done reviews for some medical journals (less prestigious, widely known journals than Science & Nature - but several of them, over the years.)
Reviewers are chosen for having some experience in the topic area; typically 3 reviewers per article. (Though just because the Journal of 9/11 Studies has 3 "peer reviewers" does not mean they otherwise follow standard peer review protocol...in part because their 3 reviewers are also their 3 editors...who have been the authors of many of the articles they've published.)
The paper(s) are sent for review without inclusion of any information about the authors, though sometimes the reviewer has some clues based on the work, or specific mention of the city or institution if included in the paper, or stilted translation into English (clueing one in that the authors are non-English-speakers).
The reviewers are given specific criteria on which to comment (but can be detailed or brief in their responses, & can add comments in other areas, for example in their answer to the question as to whether this paper adds to scholarly knowledge about the topic & should be published).
The paper's authors receive the reviewers' comments anonymously, and are asked by the editors to respond to the comments, either by making changes to the article, or by responding to the editors if the authors think that no change is needed/warranted in response to some of the comments (in which case they have to explain why they think this).
After revision, at least in some cases, the paper is sent out again for review. At least, this has been my experience as an (occasional) author of journal papers...one of the first reviewers wants you to "expand discussion of X and add a graph" and one of the next round of reviewers wants you to "shorten the discussion of X, & take out the graph."
But, as has been discussed here, this process is different from grant application review, in part because there's no money involved, and in part because it's not done as a group process (more subject to hijacking & interpersonal considerations?).
|
I think, you know, freedom means freedom for everyone* -Dick Cheney
*some restrictions may apply |
|
|
|
tomk80
SFN Regular
Netherlands
1278 Posts |
Posted - 03/07/2009 : 18:48:31 [Permalink]
|
Originally posted by Simon
Originally posted by tomk80
This is because a number of reviewers (not all scientists) need to decide on the question "Is this subject worth studying and is the study design and structure sufficient to do so?"
|
I am not sure I agree with that.
As far as I know, everybody on a board is a scientist and his field should be that of the grant they cover (no astrophysicists to give out microbiology grants, for example), in fact, it is common practice for the board members to have received grants from the board in the previous years. |
That might be the case, I'm not sure. It may be different for different institutions awarding the grants. IIRC some dutch institutions have a number of non-scientists in the application process, but now you've made me doubt my statement.
But, of course, the system is not perfect.
They are some subjectivity (is it worth investigating, is it really within the -often very precise- framework of this particular funding) as well as human factors (the process is not anonymous).
The boards can also make technical mistake, I remember them refusing to fund a project because it was not technically feasible, and then, the researcher achieving the infeasible a few months later.
But all in all, despite its shortcoming becoming more clear cuts in the recent years, when the lack of funding forced them to reject a much higher proportion of projects, I think that funding agencies are doing a pretty good job.
I, for one, certainly can not think of a better system.
|
I agree with that.
The main point that I was trying to make is that in a grant application the reviewers need to decide on the question whether some grant is worth studying. And while the question that are unanswered in a certain field may be objectively determined, which of those questions has priority is a lot fuzzier.
But alas, I cannot think of a better system either. In the end I think that this fuzziness is inherent in any system you'd come up with, precisely because of the nature of research. You don't know what you're going to get, even if you have some suspicions. Otherwise you wouldn't need to do the research in the first place. |
Tom
`Contrariwise,' continued Tweedledee, `if it was so, it might be; and if it were so, it would be; but as it isn't, it ain't. That's logic.'
-Through the Looking Glass by Lewis Caroll- |
|
|
|
Zebra
Skeptic Friend
USA
354 Posts |
Posted - 03/07/2009 : 21:29:51 [Permalink]
|
Originally posted by tomk80
...
But alas, I cannot think of a better system either.
...
|
I know - how about a popular vote by the American public?  |
I think, you know, freedom means freedom for everyone* -Dick Cheney
*some restrictions may apply |
|
|
|
JustMe
Skeptic Friend
64 Posts |
Posted - 03/07/2009 : 23:01:00 [Permalink]
|
There is a difference between saying that peer review is a failure, and saying that peer review has failures. There has been anecdotal evidence for some time that the type of subjective judgements noted in the study are a systemic problem affecting multiple facets of the inquiry process across multiple disciplines.
Editor of the British medical journal The Lancet, Richard Horton, is claimed to have said "The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong."
Oddly, when I was looking for more context on this quote I came across it again at an online open access peer reviewed journal site that highlighted a quote from the prof who taught the most useful and fascinating university class I've ever taken. He said:
"Universal and free access to knowledge has been our only protection against the types of intellectual tyranny that are frequently presented as absolute authority and representation of consensus. Open access journals like SJI have the potential to assess a manuscript based upon its accuracy and potential rather than the prestige of the institution and the pen from which it originates. We are emerging into an international community and a global consciousness where versatility and multiple approaches to the dissemination of knowledge, traditional and novel, are essential. SJI and other open access journals are the expression of this evolution." -- Dr. Michael A. Persinger, Full Professor of Behavioural Neuroscience, Psychology and Biology
http://www.scientificjournals.org/index.php |
|
|
|
Simon
SFN Regular
USA
1992 Posts |
Posted - 03/07/2009 : 23:44:02 [Permalink]
|
Originally posted by Zebra
Originally posted by tomk80
...
But alas, I cannot think of a better system either.
...
|
I know - how about a popular vote by the American public?
|
That'd be quite a terrible method as the general public has little understanding about the Science, not to mention, cutting edge Science (as illustrated by Palin's indignation at Scientists studying fruit-flies).
Not to mention, it would be cumbersome to vote for every single project.
What the public can do, through its elected officials, is too give the direction in which Science should go ('Work on curing cancer; develop new sources of energy'...) by attributing money to these broader categories and then have a panel of specialists which are familiar with cutting edge science, to give the money to the projects which fulfil these objectives best.
But that's already the way the grant process works. |
Look again at that dot. That's here. That's home. That's us. On it everyone you love, everyone you know, everyone you ever heard of, every human being who ever was, lived out their lives. The aggregate of our joy and suffering, thousands of confident religions, ideologies, and economic doctrines, every hunter and forager, every hero and coward, every creator and destroyer of civilization, every king and peasant, every young couple in love, every mother and father, hopeful child, inventor and explorer, every teacher of morals, every corrupt politician, every "superstar," every "supreme leader," every saint and sinner in the history of our species lived there – on a mote of dust suspended in a sunbeam.
Carl Sagan - 1996 |
|
|
|
Zebra
Skeptic Friend
USA
354 Posts |
Posted - 03/08/2009 : 00:02:20 [Permalink]
|
Note the rolling eyes. I, too, think it would be a horrible method.
Another route by which the public influences research direction is by supporting foundations.
I have to grit my teeth every time I hear "Fund the Cure!" (running theme of the Susan G. Komen foundation, which per their website is "the largest source of nonprofit funds dedicated to the fight against breast cancer in the world" - $1 billion in the past 27 yrs).
Every October, one of the nearby grocery stores asks people (every time you go in) to donate $1 "to fund the cure." I've told them I'll donate when they stop selling cigarettes, but they just look at me funny & haven't taken me up on the offer yet.
"Fund the Cure" a catchy phrase, & taps into public fear about breast cancer (which is, of course, a common and sometimes fatal condition), but IMO it also implies (a) that breast cancer is one disease (it's not) and (b) that The Cure is right around the corner (it's not).
Researchers sitting on a grant review panel have a much better understanding of the big picture, and also the details.
|
I think, you know, freedom means freedom for everyone* -Dick Cheney
*some restrictions may apply |
|
|
|
Dave W.
Info Junkie
USA
26022 Posts |
Posted - 03/08/2009 : 01:11:57 [Permalink]
|
Originally posted by JustMe
There has been anecdotal evidence for some time that the type of subjective judgements noted in the study are a systemic problem affecting multiple facets of the inquiry process across multiple disciplines. |
Anyone who understands that peer review is performed by human beings should consider "subjective judgements" to be endemic within the process, and not merely something suggested by "anecdotal evidence." Open access or not. This is not a difficult fact around which one might wrap one's head.
The problem only occurs when someone suggests that peer review is some sort of guarantee of accuracy. And, believe it or not, the people most often doing so are the anti-science cranks who point out some failure of peer review and go on to claim that such a failure proves that their non-peer-reviewed work should be given much more consideration than it is due. By analogy, what the nutbars are saying is, "despite a century of experience and safety improvements in the automobile, people still die in them, so you should buy our $3,000 bicycle instead." It's a transparently bad argument.
As soon as human nature is removed from the process of doing science, we will attain perfect science. Until then, the fact that problems exist shouldn't be a matter of discussion. What should matter is how serious the problems are and how often they occur.
In short, "there are problems in peer review" is not new or interesting. But "this process reduces problems in peer review" would be. |
- Dave W. (Private Msg, EMail)
Evidently, I rock!
Why not question something for a change?
Visit Dave's Psoriasis Info, too. |
|
|
 |
|
|
|
The ‘Black Box’ of Peer Review
