A reader writes:
Dan, in the introduction to your blog, you state that the chemical analysis of the radiocarbon samples is “all nicely peer-reviewed in scientific journals.” You go on to state that the chemical and physical properties of the images are also documented, “in peer-reviewed scientific journals.”
Such touting of the two words, “peer reviewed,” in a sense analogous to some sort of seal of approval by scientists, made good sense years ago. Now, to the minds of scientists everywhere in every field, it sounds hollow, almost comically supercilious. Without knowing that the peer review was conducted by a reputable and ethical journal, the approbation is meaningless. Paperless and on-demand publishing, virtual internet offices, pay-per-click advertising and the concept called open access have created more bad journals than good ones. Anonymous peer review, which should be the ideal, can and often means two guys with laptops in a Tiki bar.
For a more complete picture you might want to read some of the material at Nature’s Peer Review Debates.
Here is the Table of Contents from those debates. In and of itself it is very interesting to read:
Overview
Nature’s trial of open peer review
Despite enthusiasm for the concept, open peer review was not widely popular, either among authors or by scientists invited to comment. by Sarah Greaves, Joanna Scott, Maxine Clarke, Linda Miller, Timo Hannay, Annette Thomas, Philip Campbell / doi:10.1038/nature05535 Full Text
Systems
Online frontiers of the peer-reviewed literature
The Internet is allowing much more interactive science publishing by Theodora Bloom / doi:10.1038/nature05030 Full Text
Trusting data’s quality
Database publication presents unique challenges for the peer reviewer by Brenda Riley / doi:10.1038/nature04993 Full Text
Opening up the process
A hybrid system of peer review by Erik Sandewall / doi:10.1038/nature04994 Full Text
An open, two-stage peer-review journal
The editors of Atmospheric Chemistry and Physics explain their journal’s approach. by Thomas Koop and Ulrich Pöschl / doi:10.1038/nature04988 Full Text
Reviving a culture of scientific debate
Can ‘open peer review’ work for biologists? Biology Direct is hopeful. by Eugene Koonin, Laura Landweber, David Lipman and Ros Dignon / doi:10.1038/nature05005 Full Text
Quality and value
The true purpose of peer review
1) What you can’t measure, you can’t manage: the need for quantitative indicators in peer review by Charles Jennings / doi:10.1038/nature05032 Full Text
2) Models of quality control for scientific research by Tom Jefferson / doi:10.1038/nature05031 Full Text
How can we get the best out of peer review?
A recipe for good peer review by Trish Groves / doi:10.1038/nature04995 Full Text
Statistics in peer review
Researchers need reviewers to check their stats. by David Ozonoff / oi:10.1038/nature04989 Full Text
How can we research peer review?
Improving the peer-review process relies on understanding its context and culture. by Joan E. Sieber / doi:10.1038/nature05006 Full Text
Ethics
Trust and reputation on the web
Online publications have several ways to give themselves a good name. by William Arms / doi:10.1038/nature05035 Full Text
Detecting misconduct
Does a digital workflow make it easier to detect ethical breeches in peer review? by Dale Benos / doi:10.1038/nature04996 Full Text
What is it for?
Analysing the purpose of peer review. by Elizabeth Wager /doi:10.1038/nature04990 Full Text
Increasing accountability
What authors, editors and reviewers should do to improve peer review. by Kirby Lee and Lisa Bero / doi:10.1038/nature05007 Full Text
Evolving peer review for the internet
Peer review needs to adapt to the pace and volume of information published online by Richard Akerman / doi:10.1038/nature04997 Full Text
Wisdom of the crowds
Scientific publishers should let their online readers become reviewers. by Chris Anderson / doi:10.1038/nature04992 Full Text
Certification in a digital era
What functions do we take for granted in print? by Herbert Van de Sompel / doi:10.1038/nature05008 Full Text
The case for group review
Peer review would be improved by discussions across the lab. by Debomoy Lahiri / doi:10.1038/nature05033 Full Text
Peer review of interdisciplinary scientific papers
Boundary-crossing research meets border patrol by Christopher Lee / doi:10.1038/nature05034 Full Text
‘I don�t know what to believe’
Understanding peer review is key to developing informed opinions about scientific research. by Tracey Brown / doi:10.1038/nature04998 Full Text
The pros and cons of open peer review
Should authors be told who their reviewers are? by Thomas DeCoursey / doi:10.1038/nature04991 Full Text
Does peer review mean the same to the public as it does to scientists?
Even reviewed literature can be cherry-picked to support any argument. by John Moore / doi:10.1038/nature05009 Full Text
I have personal experience with the peer-review process. At the suggestion of the editor, I submitted a manuscript (ID No. 25055) on February 24, 2012, explaining why an article titled “Entropy and evolution” (Am. J. Phys., Vol. 76, No. 11, November 2008) was absurd. An anonymous reviewer said I was wrong. In this way, the editor justified not retracting the article. I re-wrote my submission and posted it here: http://creationwiki.org/Pseudoscience_in_the_American_Journal_of_Physics.
But David, everyone says you are wrong.
Dear Frank Ly,
Not quite. Stephen Barr and Randy Isaac are two Christian physicists who write about evolution. Barr told me angrily that I was wrong and harming the Catholic Church. I accused him of not reading my explanation and demanded an apology. He never explained why I was wrong. Isaac offered to “walk me through” the matter on the open forum of the American Scientific Affiliation, but his comments were inane. For example: David: A Boeing 747 does not have a temperature. Randy: Yes it does. I asked the president of the ASA to assign a moderator, but he declined to do it. Why don’t you read my Creationwiki. org article and tell me what part you don’t understand? You don’t need even a high school course in physics to see how absurd the AJP article is, believe it or not.
For a good illustration of the problems, have a look at this:
http://www.youtube.com/watch?v=C42AwvaZ-04
The problem is not so much with the ‘review’ part, but mainly with the ‘peer’ part.
Being reviewed by ‘peers’ means being subjected to censorship by those dominating the field. ‘Peer pressure’ or ‘peer control’ is to be avoided. New types of review should be introduced. For instance, a book such as the recent one by Fanti & Gaeta could go with the online publication of all the materials, methods, codes and data, and a subsequent open (but moderated) online review process.
I am not sure if this will post anonymously, that is my intention, I have entered anonymous in the blank for the name. If it doesn’t, oh well-I included an informal resume because I believe it might give some credence to having been there, done that-it is not intended as a personal trumpeting exercise.
I have personal experience with peer-review-I have authored, co-authored numerous manuscripts (>40) in scientific journals. I have submitted and been awarded multiple national research grants I have served as a reviewer on various journals and on various grant review panels. I am still invited to review articles for scientific journals on an ad-hoc basis, several times a year. Peer-review has its issues, it is not immune to bias-egos and personal vendettas can enter into the equation, when they shouldn’t be a part of it. Overall, I believe the merits outweigh the bad things. A certain amount of respectability is awarded with peer-review, as it should be-it is difficult to imagine not having such a system in place to evaluate scientific findings.
Peer review does not equal infallibility-it does not guarantee that the results or the interpretation given to them are beyond reproach, never to be questioned-it is the very nature of science to reevaluate and reinterpret previous results. Online publication has helped to speed up the peer-review process, and make results more readily available, which in my opinion, is a good thing. Certainly, ideas and results can be presented elsewhere (non-peer reviewed) that are scientifically sound. There is unquestionably a sense of bias in the scientific community at large when it comes to the science of the Shroud, but I don’t think writing off peer-review is the way to go. In the end, good science stands the test of time, and will be extended upon-results will either be supported or they will fall, irregardless of where the results are published.
Overconfident academic expert falasy. Perfect!
The last reference “Does peer review mean the same to the public as it does to scientists?” is a good *short* description of what is peer-review. I quote one paragraph from it:
“It’s been peer reviewed, so it must be right, right? Wrong! Not everything in the peer-reviewed literature is correct. Indeed, some of it is downright bad science. Professional scientists usually know how to rate papers within their own fields of expertise (all too often very narrow ones nowadays). We realize that some journals are more stringent than others in what they will accept, and that peer-review standards can unfortunately be too flexible. A lust for profit has arguably led to the appearance of too many journals, and so it can be all too easy to find somewhere that will publish poor-quality work.”
I think it summarizes the big picture. But there are so many parameters to take into consideration that to really get a sense of them you need to publish and review many papers.
As I see it, “peer-review” is a subset of “auditing processes”. The last ten years of my engineering career were spent in the Internal Audit department of a major national organisation in New Zealand. It was one of the most enjoyable and rewarding times of my working life. Conventionally, Auditing is often thought of as focusing on monetary issues, but during the 1990s I noticed that its scope tended to become extended to a wide variety of other important public activities, with recruiting of a variety of professions well beyond its traditional focus on mere accounting. I attended a number of auditing conferences, including an international conference in Auckland. One of the conference highlights I recall was an audit paper focusing on environmental activities for example. The matter of licensing for various patented processes and franchises is commonly a subject for professional audit, where the reputation of the process may be at stake. Essentially auditing is concerned with providing assurance, and this as I see it is the purpose of “peer-review” of scientific papers and their related claims. Auditing has its own literature on technical processes, sampling methods, and standards. It is supported by a vast network of a wide variety of general and specialised auditing associations directed at auditing education and standards.
My own work was focused on providing assurances for various major capital-intensive engineering projects, as to their feasibility, the associated risks, and the likelihood of realising the benefits claimed for them (pre-audit), as well as post-audits to provide a data-base of experience, and current audits, ungluing stalled projects.
However, auditors (read “peer-reviewers”) are all-too human and the audit process can be corrupted by failures of objectivity, personal interests and bias, or telling the customer what they would prefer to hear, rather than telling it as it is. Historically in NZ, we have even had an Auditor-General who was sacked on a corruption charge because of misuse of a credit-card. I also observed that there seemed to be a few major international auditing organisations whose methods and training seemed somewhat lax and did not meet what I believed to be adequately rigorous professional standards. The adverse non-objective reaction to Yves Delage’s paper in 1902 will be well-known to Shroudies.
Proper Peer-review processes needs to meet the same high standards that are aspired to in every other branch of auditing and provision of professional assurances.
Concerning the question of the importance for a scientist to get his work published in a peer-reviewed journal, I think you should read this recent comment of mine: http://shroudstory.com/2013/04/05/a-committee-far-removed-from-padua/#comment-29504
Even if this is a MUST for any credible and profesionnal scientist to get his work published in a CREDIBLE peer-reviewed journal, the only thing that can give us some insurrance about the validity of his work is if another scientist, working completely independently from the first scientist, came to the same conclusions. And, in my mind, if this second scientist used a different protocol and/or methodology and came nevertheless to the same conclusions than the first scientist, this has even more value. The same is true if a third scientist, working also completely independently from the first two scientists, came also to the same conclusion.
The best example of this kind of insurance factor regarding the Shroud can be found in the conclusions reached independently by Adler and Heller in the US and Baima Bollone in Italy versus the question of the nature of what appeared to be bloodstains on the Shroud. Both came to the same conclusion (i.e. that the blood on the Shroud is really blood and that it mostly came from exudates of humid blood clots instead of coming from complete blood), even though they sometimes used different methods and protocols. That’s the kind of conclusions we can trust.
As I write as a historian for two university presses (OUP and Yale University Press), I am always subject to peer review, on my proposals themselves, on my drafts and on my final versions. I am also asked to comment, as probably one of three ‘reviewers’, on proposals sent in. In the vast majority of cases I see the peer reviews on my own work although they remain strictly anonymous and often i am able to clear up points reviewers have disagreed with to the satisfaction of my editor (who makes the final decision on publication as with most academic journals). It is a system that works well.
I am not a scientist but there does seem to be a genuine problem, accepted by many contributors to this site, relating to the provenance of samples from the Shroud. I leave it to others to comment but I am always amazed that Ray Rogers’ work on the 1988 samples is described as ‘peer-reviewed’ when he fails to provide any supporting evidence that the fibres he used were genuinely those cut off by Gonella. He suggests that they have not come to him directly so there is every possibility that they might not be the originals or have been contaminated on the way. There is even talk on this site of an intermediary who passed on materials. So who are the ‘peers’ who considered Rogers’ paper acceptable? Was he able to provide them with further evidence he chose not to publish publicly that convinced them of the authenticity of his fibres?
On a separate note, it would seem important to know where these fibres are now, following Rogers’ death. Who actually owns them? Is anyone authorised to pass them on to an independent laboratory specialising in textiles to see if Rogers’ findings might be replicated?
I have read your interesting discussions …
I agree with some comment sent about the “commercial book”.
But Giulio Fanti is a Professors of Mechanical Engineering
and we presume he was able to work with the adequate experiments .
Then we have to wait before to read the report about the inherent experiments.
In my opinion it’s impossible to condemn (… perhaps out of season ?) a work (as some
of you are trying to do ) before to read that work !
Then there is the possibility to improve the discussion about the important
point indicated by Hugh Farey, probably :
“the shroud fragments sucked out with a vacuum cleaner are likely to be
more degraded, and thus give an apparently older age for the shroud than it really is…”
Then :
Where are the right indications ?
Where is the right work ?
In my idea we have to work using the fibrils taken in the past and keeped
in a safe place. Then we can compare the results obtained using the material taken by Riggi ..
— —
In any case (if we want to work in a proper manner) we have to try
to control the certified fibrils using the AFM techniques.
Do you agree on that question ?
— —
In my opinion the five keys (= the Matters) useful to work, in order to obtain a result,
are the following :
– Materials Science
– Applied Mechanics and Structural Mechanics
– Textile Chemistry
– Nanotechnology
– Thermodynamics
– Statistics
— —
You have to discuss the results obtained from (for example) the AFM three-point
bending test (on ancient and modern linen fibrils), etc. … !
Where are the useful textile works to take into account in order to be right with the claims ?
The careful studies of the pollen require the adequate knowledges
in the field of the
B I O L O G Y ! …
and see also the blood crust
… the figures 23-24 in that book (= under the optical microscope and under the SEM).
See also : the inherent spectra about the chemical composition.
I believe that using the AFM techniques is possible to improve these controls !
— —
Sorry for that unpleasant forgetfulness !
I beg your pardon.
Here is a link to a recent article (April 7, 2013) in the Science Section of the New York Times describing the plague of bogus journals and conferences:
http://www.nytimes.com/2013/04/08/health/for-scientists-an-exploding-world-of-pseudo-academia.html?ref=science
Sadly, It is now common for researchers to receive daily offer to publish their work in journals and at conferences that have barely any serious peer-review. This was unknown in the 90s.