[3dem] sharing em maps during peer-review process

Sat May 2 09:39:28 PDT 2015

I have been a supporter of open sharing of data for many years, and I am
very happy that deposition of maps and, more recently also entire
datasets has become the standard in our field. I appreciate the forceful
statements made by Ed and others and, of course, I support the spirit in
which they are made. Nevertheless, even though it is important to remain
positive, I feel compelled to add to this discussion that there is also
abuse of the anonymous reviewing system. While this fact is not disputed
by anybody, and it does not automatically mean that there is a different
system we should adopt, it validates the worries that some have voiced.
I also would like to point out that there is some importance in ensuring
fair and unbiased reviews. It is true that for many of us one of the
main reasons for doing science is the enjoyment we get from our work.
However, I think that there are also a number of other reasons for being
in science including, for example, a job with a career, financial gain
(especially in scientific publishing) and curing a disease. Whether or
not a paper is reviewed while observing high ethical standards may
therefore have a real-life impact on people’s trajectories, both in
science and in areas benefiting from it. For those of us already
established this may be less of a worry but those trying to get in, a
lot could be riding even on a single publication.

What I find lacking a bit in our current system is the accountability of
the reviewers. I agree that trust is part of the reviewing process,
ideally in both directions, but I find it a bit single-sided to see the
need for accountability only on the side of the scientific contributors.
If social media has taught us one thing, it is that people who do not
need to worry about accountability sometimes also feel free to abandon
ethical standards. For the reviewing system to work, we also have to
think of ways to hold reviewers and editors accountable. I mention
editors because they play an important role as the stewards of the
reviewing process. For obvious reasons their interests may not always
align with the noble pursuit of science but focus more on making money
for the journal they are working for. In this respect I have come to
appreciate the system adopted by eLife. I agree with Sjors that it is
not always helpful to reveal your identity as a reviewer, especially if
this may prevent you from giving your honest and unbiased opinion.
However, at eLife, even if a reviewer chooses not to reveal their name
they are known during the review process to the other reviewers.
Furthermore, the editor of a paper is a practicing scientist with a
general understanding of the subject of the paper under review, and
without an obvious conflict of interest that an editor at a for-profit
journal may face. Finally, besides being open-access, eLife also
publishes the exchange between reviewers and authors for papers that are
ultimately published. I therefore think that eLife has made a
substantial step towards a more transparent and fairer system. In this
regard, a recent blog post by Science Careers may also be interesting to
read:

http://sciencecareers.sciencemag.org/career_magazine/previous_issues/articles/2015_01_27/caredit.a1500025

Regarding the sharing of 3D maps with the reviewers, one way to reduce
the potential risk of abuse might be to share only boxed-out regions of
the map that are needed to support the claims made in the publication.
This would allow reviewers to inspect these parts of the map while the
map fragments could not so easily be used as a reference structure for
another project. This is still not foolproof, of course, but it may help
balance a little more the positions of authors and reviewers.

     Niko

On 5/2/2015 7:49 AM, Edward Egelman wrote:
> I agree strongly with Sjors, and think that more openness and more data
> availability can only strengthen science. It was not too long ago where
> some people refused to make reconstruction volumes available AFTER
> publication, since there was no journal policy dictating submission to a
> repository. As for double-blind reviewing, in a recent NSMB there is an
> editorial about the new double blind submission possibility. They
> provide a link to a rather equivocal editorial evaluation of the system
> that appeared in one of the first Nature journals to institute this,
> Nature Nanotechnology:
> 
> http://www.nature.com/nnano/journal/v9/n11/full/nnano.2014.265.html
> 
> In that editorial, they state:
> 
> "However, the/American Economic Review/ dropped double-blind review in
> 2011, citing the role of search engines in limiting their capacity to
> maintain author anonymity as a major reason for the decision."
> 
> In our field (as opposed to economics) it would be even harder to have
> anonymous authors, as one needs to describe the microscope and camera,
> cite previous publications, etc. One could not say "the protein was
> prepared as previously described" and give a citation.
> They also state that although 80% of respondents to some survey
> supported double-blind, less than 20% of submitted manuscripts chose
> that option (what they actually state is that is that the monthly
> percentage has never risen above 20% for the each of the two journals,
> so this is the maximum of something that is fluctuating and the overall
> percentage may be 10% or less!).
> Regards,
> Ed
> 
> On 5/2/15 7:10 AM, Sjors Scheres wrote:
>> Dear all,
>>
>> I think moving towards more openness will be a good thing. As a reviewer
>> it is often hard to assess the quality of a map from 2D still pictures
>> alone, and having a look at the map will surely help to assess its
>> quality. Trust already is at the foundation of the review process. Whether
>> the reviewer also gets access to the map and the model does not change
>> much in that respect.
>>
>> Reversed anonymity in the review process as suggested by Matthijn is
>> problematic because the identity of the authors is usually very easy to
>> distill from the paper itself. A useful comment on this (in the context of
>> double-blind review) is in Nature this week. Going the other way, with
>> open identities of both authors and reviewers, is attractive in some ways,
>> but also has its difficulties. I personally tend to tick the option to
>> reveal my identity as a reviewer in eLife, and have also had my papers
>> reviewed by people who revealed their identity. Last year I wrote one
>> negative review for a journal that did not have this option, and later
>> revealed myself to one of the authors and had an open, face-to-face
>> discussion with the intention to be helpful. However, revealing one's
>> identity as a reviewer also has a potential danger of bias towards being
>> less critical (perhaps especially for less established reviewers?), or the
>> danger of a culture where positive reviews are expected to be 'paid back'
>> later. Again, I think trust is the key word, and recognizing the
>> difficulties of every option is a good start.
>>
>> Perhaps I'm naive, but I think most of us are in this for the fun of it,
>> and to ultimately change the world for the better. More openness will
>> probably help in speeding up the scientific process. In that context I
>> also agree with Grant that discussing options like (bio)-arxiv may be
>> useful.
>>
>> Sjors
>>
>>> dear colleagues,
>>>
>>> i would be interested in experiences / suggestions / views of others in
>>> the
>>> field on  the following issue that may be of interest to many of us:
>>> the editor of our manuscript forwarded the request of a peer-reviewer to
>>> access the cryo-em map of our beloved complex. this has never happened to
>>> us, but to our surprise the editor did not consider the request to be
>>> unusual.
>>> of course, we share the point that the map would be of great help in
>>> judging the interpretation of the data. however, we also feel very
>>> uncomfortable sending the condensed result of lengthy research to an
>>> anonymous colleague, who could theoretically make considerable misuse of
>>> it. nevertheless, the policy of the journal seems to let us little
>>> choice: "Supporting
>>> data must be made available to editors and peer-reviewers at the time of
>>> submission for the purposes of evaluating the manuscript. Peer-reviewers
>>> may be asked to comment on the terms of access to materials, methods
>>> and/or
>>> data sets".
>>> in any case we would be curious whether others indeed got similar requests
>>> and how they dealt with it. a good solution for (paranoid?) people like us
>>> could be a good web-based viewer that lets others view our map, but i
>>> would
>>> not know of such a tool.
>>>
>>> Thanks
>>>
>>> Friedrich
>>>
>>> --
>>> Dr. Friedrich Foerster
>>> Max-Planck Institut fuer Biochemie
>>> Am Klopferspitz 18
>>> D-82152 Martinsried
>>>
>>> Tel: +49 89 8578 2632
>>> Fax: +49 89 8578 2641
>>>
>>> www.biochem.mpg.de/foerster
>>> _______________________________________________
>>> 3dem mailing list
>>> 3dem at ncmir.ucsd.edu
>>> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
>>>
>>
> 
> -- 
> Edward H. Egelman, Ph.D.
> Professor
> Dept. of Biochemistry and Molecular Genetics
> University of Virginia
> 
> President
> Biophysical Society
> 
> phone: 434-924-8210
> fax: 434-924-5069
> egelman at virginia.edu
> http://www.people.virginia.edu/~ehe2n
> 
> 
> 
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
> 