Peer Review

Transcription

Peer Review

Technische Universität Graz, Institut für Maschinelles Sehen und Darstellen
Peer Review
Horst Bischof
H. Bischof
Professor
Horst Cerjak, 19.12.2005
Wissenschaftl. Arbeiten
1
Overview
•
•
•
•
•
•
Peer Review
Why Peer Review
How it works
Problems
How to write reviews
Alternatives
Some material is based on Slides from:
Gerry McKiernan (Iowa State University Library),
Peggy Dominy & Jay Bhatt
Sarah Rockwell
H. Bischof
Professor
2
PEER REVIEW: DEFINITION
“Peer review is the assessment by
an expert of material submitted for
publication.”
Carin M. Olson, “Peer Review of the Biomedical Literature,” American
Journal of Emergency Medicine 8 no.4 (July 1990): 356-358.
H. Bischof
Professor
3
When did peer review start?
Some would say that “Peer
Review” goes back as far
as the 17th century, when it
was known as “The
Inquisition of the Holy
Roman and Catholic
Church”. Scholars’ works
were examined for any hints
of “heresy”.
Galileo
Wikipedia
http://en.wikipedia.org/wiki/Peer_review
H. Bischof
Professor
4
Peer review in “modern times”
Peer review (known as refereeing
in some academic fields) is
used in:
1. Publication process
2. Awarding of funding for
research
3. Patents
4. Standards
Each of these involve slightly
different practices, but
ultimately colleagues are
evaluating each other.
H. Bischof
Professor
5
Process of peer review
Once a paper has been
submitted for consideration
of publication, the editor will
select 1-2 or 3 scholars from
a pool of volunteers to read
and evaluate the paper.
Typically it is a double blind process: the reviewers do not
know who the author is and the author does not know who
the reviewers are. That way only the merits of the paper are
evaluated.
H. Bischof
Professor
6
Process of peer review (cont.)
The reviewers (within a reasonable time period)
respond with their comments which are then
forwarded to the author for response to or
compliance with reviewer’s suggestions. In the days
before the “Internet”, this added weeks (months?) to
the publishing process.
Today, moving text back and forth electronically has
dramatically accelerated the process, though the
imposition on an overburden volunteer researcher
has not changed much.
H. Bischof
Professor
7
Review Conferences
•
•
•
•
•
•
•
Conference with/without Area Chair
PC assigns to ACs
2-3 Reviewers (selected by AC or PC)
Reviews are handed in
Maybe Rebuttal
Decision by AC or PC
Camera Ready Copy if accepted
H. Bischof
Professor
8
Overview of review process
(considerable variation between journals,
conferences)
• Potential reviewer contacted by journal
• Given authors, title, abstract, and time frame for
review
• Reviewer agrees to review paper (or declines)
• Reviewer receives paper
• Reviewer performs review
• Reviewer submits review to editors
• Editors examine reviews, obtain additional reviews if
needed, and make decision
• Decision goes to author, with comments from
reviewers
• Reviewer thanked; may be informed of decision; may
receive
copy of comments sent to author
H. Bischof
Professor
9
Content of reviews
• Review form
• Comments to editor
• Comments to authors
– General comments
– Specific recommendations
• Journal may ask specific questions to ensure that
specific points are addressed
H. Bischof
Professor
10
Lets look at an Example
eccv-rev.html
H. Bischof
Professor
11
Reviews are generally blinded
• Reviewer’s identity is known to editors and
journal staff
• Reviewer’s identity usually will not be
released to authors
• Reviewer’s identity usually will not be
released to third parties
• Intended to shield reviewers and allow them
to provide critical and honest reviews
• No system is perfect - authors sometimes
discover the identities of reviewers
H. Bischof
Professor
12
Why do peer review?
• Filter
– More papers submitted than could be
“printed”
– Eliminate “bad” science, pseudo-science,
harmful science...
• Aura of “quality” (only the best gets in)
• Collegial stamp of approval
• Professional obligation to the principles of one’s
discipline
H. Bischof
Professor
13
PEER REVIEW: PURPOSES
Peer review helps to ensure that published
research is:
Important
Original
Timely
Technically-reliable
Internally-consistent Well-presented
Benefited from guidance by experts
Carin M. Olson, “Peer Review of the Biomedical Literature,” American Journal of
Emergency Medicine 8 no.4 (July 1990): 356-358.
H. Bischof
Professor
14
PEER REVIEW: STRENGTHS
The underlying strength of peer review
is“…the concerted effort by large
numbers of researchers and scholars
who work to assure that valid and
valuable works are published and
conversely to assure that invalid or nonvaluable works are not published … .”
Anne C. Weller, Editorial Peer Review: Its Strengths and Weaknesses.
(Medford, NJ: Information Today, 2001).
H. Bischof
Professor
15
So, what’s the problem?
•
•
•
Famous papers that were published and did NOT get peer reviewed:
– Watson & Crick’s 1951 paper on the structure of DNA in Nature
– Abdus Salam’s paper “Weak and electromagnetic interactions” (1968). Led to Nobel Prize
– Alan Sokal’s “Transgressing the Boundaries...” in 1996 turned out to be a hoax. Now known
as the Sokal Affair.
Famous papers that were published and passed peer review that later proved to be fraudulent:
– Jan Hendrik Schon (Bell Labs) submitted and passed peer review 15 papers published in
Science and Nature (1998-2001) found to be fraudulent.
– Igor and Grichka Bogdanov 1999 & 2002 published papers in theoretical physics believed
by many to be jargon-rich nonsense.
Famous papers that got rejected that later turned out to be seminal works:
– Krebs & Johnson’s 1937 paper on the role of citric acid on metabolism was rejected by
Nature as being of “insufficient importance”, was eventually published in the Dutch journal
Enzymologia. This discovery, now known as the Krebs Cycle, was recognized with a Nobel
prize in 1953.
– Black & Scholes 1973 paper on “the pricing of options and corporate liabilities”, rejected
many times, was eventually published at the intersession of Merton Miller to get it accepted by
the Journal of Political Economy. This work led to the Nobel Prize.
H. Bischof
Professor
16
PEER REVIEW PROBLEMS:
PROBLEMS
•
•
•
•
•
•
Subjectivity
Bias
Abuse
Detecting defects
Fraud and Misconduct
Delay
Fytton Rowland, “The Peer-Review Process,”
Learned Publishing 15 no. 4 (October 2002): 247-258.
Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf
H. Bischof
Professor
17
SUBJECTIVITY
• Summary rejections by editor without sending the
paper to referees
• Choice of referees by the editor (choosing for
example, a known harsh referee for a paper the
editor wishes to see rejected)
H. Bischof
Professor
18
BIAS
• Discrimination against authors because of their
nationality, native language, gender or host institution
• Situations where author and referee are competitors
in some sense, or belong to warring schools of
thought
H. Bischof
Professor
19
ABUSE
• Too many articles out of one piece of research, or
duplicate publication
• Intellectual theft: omission or downgrading of junior
staff by senior authors
• Plagiarism (stealing others yet unpublished work that
has been sent for review)
• Delaying publication of potentially competing
research
H. Bischof
Professor
20
FRAUD and MISCONDUCT
• Fabrication of results
• Falsification of data
• False claim of authorship for results
H. Bischof
Professor
21
DELAY
“There is much muttering about publication delay, a real enough
problem, especially in paper publication, but peer review itself is
often responsible for as much of the delay as the paper publication
and distribution process itself.”
Stevan Harnad
Stevan Harnad, “Implementing Peer Review on the Net: Scientific Quality Control in
Scholarly Electronic Journals, in Scholarly Publication: The Electronic Frontier, edited
by Robin P. Peek and Gregory B. Newby (Cambridge MA: MIT Press, 1996).
http://www.ecs.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html
H. Bischof
Professor
22
Peer Review in Practice
H. Bischof
Professor
23
From an editor’s point of view the
ideal reviewer
• Is a researcher who is working in the same
discipline as the subject of the paper yet is not in
direct competition with the authors
• Will understand the hypotheses underlying the work
• Will be familiar with the model systems and
methods used in the project
• Will be able to judge the quality of the data and
analyses and assess the validity of the conclusions
• Will be able to assess the significance of the work
H. Bischof
Professor
24
Questions to consider when deciding
whether to review a paper
Do you have appropriate expertise?
– Ideal reviewers seldom exist
– Editors often send papers to multiple reviewers, with
different areas of expertise and different perspectives
– Young reviewers tend to underestimate their
expertise
– If in doubt, contact the editor and discuss your
concerns
H. Bischof
Professor
25
Is the work too close to your own?
• Example: paper contains experiments that overlap
with those you are performing, planning, or preparing
for publication
• Decline to review paper
– Conflict of interest precludes review
– There would be a danger of the appearance of
misconduct, even if you acted ethically throughout
the review process
• Make every effort to avoid receiving the full paper – if
you receive it, return it immediately and discuss this
problem with editor
H. Bischof
Professor
26
How do you handle the paper?
• Manuscripts under review are confidential documents.
• They contain unpublished data and ideas, which must be
kept confidential.
• You cannot share the paper or its contents with your
colleagues.
• Manuscripts should be kept in a secure place, where they
will not be readily accessible to the curious or
unscrupulous.
H. Bischof
Professor
27
Confidentiality is critical
• Not only the paper, but also the outcome and content
of the review are confidential.
• Lapses in the confidentiality undermine the review
process, betray the trust of the authors and the
editors, and can create serious problems for
everyone involved in the reviews.
H. Bischof
Professor
28
Peer Review?
• It’s not perfect—grist for a lot of mills
• The Web has made it less of a obstacle to access
• Different disciplines have different perspectives—
different issues
• Pedagogical yardstick for students
H. Bischof
Professor
29
Peer review on the Internet
• Using email
email based online peer review
See Peer Review of Scholarly Publications in Health,
Online Manuscript Peer Review and Tracking Systems and
Physics of Plasmas Online Manuscript Submission and Peer
Review
• Discussion approach
better interaction among authors, reviewers and the editorial
body
JIME – Open Peer Review Process
• Wikis
Immense potential to conduct peer review
• Blogs
post publication comments
See Article Note: On Blogging as Tool, but Really About Using
RSS
H. Bischof
Professor
30
Access to Scientific Literature
• Author home pages linking their papers
Google finds them.
• Institutional Repositories
Provide access to faculty authored research papers.
See Publisher Policies that shows listing of which
publishers allow either publisher or post print version
on IRs
• Indexed by Google; increases visibility of scholarly
material
H. Bischof
Professor
31
Access to Scientific Literature
• SHERPA: Securing a Hybrid Environment for
Research Preservation and Access. It is developing
open-access institutional repositories in a number of
research universities to disseminate research
findings worldwide
• Preserving EPrints:Scaling the Preservation
Mountain
• DSpace at Drexel
• University of Pennsylvania Institutional Repository
• Institutional Repositories are increasing and hence
open access to scholarly literature increasing
H. Bischof
Professor
32
Global benefits
• Worldwide increase in access to scientific literature
• Increased opportunities for collaboration among
experts worldwide
• Increased speed to disseminate scientific literature
with electronic communities
• More informal peer reviews
• Quality needs to be maintained
• See Scholarly Electronic Publishing Weblog
H. Bischof
Professor
33

Peer Review

Transcription

Documents pareils

Examples for Benzoic acid derivatives - Wirtz

ISO 9001:2008

Properties - Walter Schottky Institut

Modul 1Talking Pictures - Institut für Kulturdidaktik

evolaris englisch

changing family issues - Hu

Klimalog Research and dialogue for a climate