[3dem] Don't blame your thermometer...

Penczek, Pawel A Pawel.A.Penczek at uth.tmc.edu
Wed Aug 28 07:29:10 PDT 2013


Hi,

"cheating" is a strong a loaded term.  Most mistakes are made by overzealous and sometimes not fully knowledgeable users.
Given a tool that is not entirely correctly advertised as a panacea against mistake the dangers paradoxically increase.

I am not aware of any statements that would specify what resolution a reference should be in "golden standard" procedure.

Regards,
Pawel.


On Aug 28, 2013, at 9:20 AM, Steven Ludtke <sludtke at bcm.edu> wrote:

> Ouch, now I can see a real 'discussion' arising.
> 
> One minor correction to your statement. They did, indeed, claim to do a 'gold standard' refinement in that paper, however, they used the same (high resolution) reference for both of the 'independent' refinements. They claimed a final resolution of ~6 Å which is higher than the '11 Å reference'. Then again, I suspect that the '11 Å resolution' reference was not rigorously filtered to 11 Å. So, they used the word 'gold standard' without following the procedure it is supposed to embody.
> 
> You are correct, of course, even so that there are many many ways to 'cheat' any standard. Nonetheless, it isn't a bad idea to develop standards which at least make it more difficult to do so, and also dramatically reduce the possibility of _unintentional_ cheating. There is very little anyone can do about someone who's determined to cheat. (That isn't intended as a comment about the PNAS manuscript, which I suspect could only be resolved at this point if they decided to make their raw data publicly available).
> 
> On Aug 28, 2013, at 8:56 AM, "Penczek, Pawel A" <Pawel.A.Penczek at uth.tmc.edu> wrote:
> 
>> 
>> My reading of Marin's parabole is that he is trying to clarify issues surrounding some of the exaggerated claims that surfaced
>> in few recent years and espoused in the papers listed by Steve.
>> 
>> While by themselves they are innocuous the danger is that for some they can provide a false sense of self-confidence: I used
>> the well-documented "prevention" method therefore my structure is beyond doubt and reproach.
>> 
>> Regrettably, it is not quite so and while additional validation never does any harm by itself it is not sufficient to prove the
>> results are correct.  A determined person can alway squeeze out helices and phosphate bonds from the data.
>> 
>> A very good illustration of perils associated with the uncritical use of "golden standard" is provided by Richard Henderson interviewed in Science:
>>        Is High-Tech View of HIV Too Good to Be True?
>> 
>> *   Jon Cohen
>> 
>> *   Science 2 August 2013: 443-444.
>> 
>> Regards,
>> Pawel A. Penczek, Ph.D.
>> Professor
>> Department of Biochemistry and Molecular Biology
>> The University of Texas - Houston Medical School
>> MSB 6.220
>> 6431 Fannin
>> Houston, TX 77030
>> USA
>> 
>> phone: 713-500-5416
>> fax: 713-500-6297
>> pawel.a.penczek at uth.tmc.edu<mailto:pawel.a.penczek at uth.tmc.edu>
>> http://www.uth.tmc.edu/bmb/faculty/pawel-penczek.html
>> 
>> 
>> On Aug 28, 2013, at 7:54 AM, Steven Ludtke <sludtke at bcm.edu<mailto:sludtke at bcm.edu>> wrote:
>> 
>> Ahh, but does your thermostat have a wet bulb or a dry bulb thermometer  ;^)
>> 
>> First I thought, "No, what an obvious trolling attempt", then I decided there may be some actual confusion, so perhaps we should resolve the semantics. So I will succumb to the urge to reply this one time (and almost certainly regret doing so later).
>> 
>> True, a "gold standard FSC" means "gold standard refinement, followed by a normal FSC", since that is too long to say, people have started saying "gold standard FSC".  Neither does cryoEM refer to microscopy with cold electrons.
>> 
>> For anyone who is confused, "gold standard refinement" simply refers to the process of splitting ones data in half at the very beginning, and performing two completely independent refinements (with independent starting models). It is, of course, not the only way of preventing resolution exaggeration due to model/noise bias, but (barring the use of other artifact inducers like hard spherical masks), it is certainly one robust technique for doing so. If you follow this technique honestly, then you can be confident that your resolution is not over-estimated. However, most people apply some sort of mask to reduce noise on the final structure before FSC computation (otherwise small box sizes lead to better resolutions but worse structures). This can raise concerns about exaggeration again, if masking isn't done with an appropriate bias-minimizing mask.
>> 
>> * Scheres, S. H. & Chen, S. (2012) Prevention of overfitting in cryo-EM structure determination. Nat. Methods. 9, 853-854.
>> 
>> * Murray, S. C., Flanagan, J., Popova, O. B., Chiu, W., Ludtke, S. J. & Serysheva, I. I. (2013) Validation of Cryo-EM Structure of IP3R1 Channel. Structure. 21, 1-10. PMC3696195
>> 
>> So, if you have any doubts, or are using some other method which you believe avoids bias, Richard's recent suggestion to randomize the phases in your raw data, then rerefine to prove that your algorithm doesn't claim to have achieved resolutions beyond this point can be used to really test your process.
>> 
>> * Chen, S., McMullan, G., Faruqi, A. R., Murshudov, G. N., Short, J. M., Scheres, S. H. & Henderson, R. (2013) High-resolution noise substitution to measure overfitting and validate resolution in 3D structure determination by single particle electron cryomicroscopy. Ultramicroscopy.
>> 
>> 
>> On Aug 28, 2013, at 6:27 AM, Marin van Heel <marin.vanheel at googlemail.com<mailto:marin.vanheel at googlemail.com>> wrote:
>> 
>> 
>> 
>> Thought of the day:
>> 
>> If you overheat your house, do you blame your thermometer?
>> It is rather your own responsibly to change your thermostat settings!
>> If you over-fit your 3D reconstruction, do you blame the FSC resolution measure?
>> It is rather your own responsibility to avoid overfitting by reference bias.
>> There is no such thing as a biased FSC (Fourier Shell Correlation). Neither is there a “gold standard” FSC!
>> Yes, you CAN do deviously biased data processing to try to prove you are better than everybody else.
>> But what ever you do, please don’t blame your thermometer.
>> 
>> Cheers,
>> Marin
>> 
>> 
>> --
>> ================================================================
>> 
>>   Marin van Heel
>> 
>>   Professor of Cryo-EM Data Processing
>> 
>>   Leiden University
>>   NeCEN Building Room 05.27
>>   Einsteinweg 55
>>   2333 CC Leiden
>>   The Netherlands
>> 
>>   Skype:    Marin.van.Heel
>>   email:  marin.vanheel(A_T)gmail.com<http://gmail.com/>
>>   and:    mvh.office(A_T)gmail.com<http://gmail.com/>
>> 
>> ----------------------------------------------
>>   and:
>> 
>>   Professor of Structural Biology
>> 
>>   Imperial College London
>>   Faculty of Natural Sciences
>>   Biochemistry Building (Room 512)
>>   South Kensington Campus
>>   London SW7 2AZ,  UK
>>   email:  m.vanheel(A_T)ic.ac.uk<http://ic.ac.uk/>
>> 
>> ----------------------------------------------
>>   Currently visiting Professor at:
>> 
>>   Laboratório Nacional de Nanotecnologia - LNNano
>>   CNPEM/ABTLuS, Campinas, Brazil
>> 
>> ------------------------------------------------------------------
>> 
>> I receive many emails per day and, although I try,
>> there is no guarantee that I will actually read each incoming email.
>> Moreover, our Spam filters can be strikt and sometimes make
>> legitimate emails disappear (try the gmail accounts, alternatively)
>> 
>> 
>> _______________________________________________
>> 3dem mailing list
>> 3dem at ncmir.ucsd.edu<http://ncmir.ucsd.edu/>
>> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
>> 
>> ----------------------------------------------------------------------------
>> Steven Ludtke, Ph.D.
>> Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem<http://www.bcm.edu/biochem>)
>> Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu<http://ncmi.bcm.edu/>)
>> Co-Director CIBR Center                          (www.bcm.edu/research/cibr<http://www.bcm.edu/research/cibr>)
>> Baylor College of Medicine
>> sludtke at bcm.edu<mailto:sludtke at bcm.edu>
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> 3dem mailing list
>> 3dem at ncmir.ucsd.edu<http://ncmir.ucsd.edu>
>> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
>> 
>> _______________________________________________
>> 3dem mailing list
>> 3dem at ncmir.ucsd.edu
>> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
> 
> ----------------------------------------------------------------------------
> Steven Ludtke, Ph.D.
> Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)
> Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)
> Co-Director CIBR Center                          (www.bcm.edu/research/cibr)
> Baylor College of Medicine                             
> sludtke at bcm.edu
> 
> 
> 
> 
> 



More information about the 3dem mailing list