[3dem] Don't blame your thermometer...

Penczek, Pawel A Pawel.A.Penczek at uth.tmc.edu
Wed Aug 28 06:56:41 PDT 2013


My reading of Marin's parabole is that he is trying to clarify issues surrounding some of the exaggerated claims that surfaced
in few recent years and espoused in the papers listed by Steve.

While by themselves they are innocuous the danger is that for some they can provide a false sense of self-confidence: I used
the well-documented "prevention" method therefore my structure is beyond doubt and reproach.

Regrettably, it is not quite so and while additional validation never does any harm by itself it is not sufficient to prove the
results are correct.  A determined person can alway squeeze out helices and phosphate bonds from the data.

A very good illustration of perils associated with the uncritical use of "golden standard" is provided by Richard Henderson interviewed in Science:
         Is High-Tech View of HIV Too Good to Be True?

 *   Jon Cohen

 *   Science 2 August 2013: 443-444.

Regards,
Pawel A. Penczek, Ph.D.
Professor
Department of Biochemistry and Molecular Biology
The University of Texas - Houston Medical School
MSB 6.220
6431 Fannin
Houston, TX 77030
USA

phone: 713-500-5416
fax: 713-500-6297
pawel.a.penczek at uth.tmc.edu<mailto:pawel.a.penczek at uth.tmc.edu>
http://www.uth.tmc.edu/bmb/faculty/pawel-penczek.html


On Aug 28, 2013, at 7:54 AM, Steven Ludtke <sludtke at bcm.edu<mailto:sludtke at bcm.edu>> wrote:

Ahh, but does your thermostat have a wet bulb or a dry bulb thermometer  ;^)

First I thought, "No, what an obvious trolling attempt", then I decided there may be some actual confusion, so perhaps we should resolve the semantics. So I will succumb to the urge to reply this one time (and almost certainly regret doing so later).

True, a "gold standard FSC" means "gold standard refinement, followed by a normal FSC", since that is too long to say, people have started saying "gold standard FSC".  Neither does cryoEM refer to microscopy with cold electrons.

For anyone who is confused, "gold standard refinement" simply refers to the process of splitting ones data in half at the very beginning, and performing two completely independent refinements (with independent starting models). It is, of course, not the only way of preventing resolution exaggeration due to model/noise bias, but (barring the use of other artifact inducers like hard spherical masks), it is certainly one robust technique for doing so. If you follow this technique honestly, then you can be confident that your resolution is not over-estimated. However, most people apply some sort of mask to reduce noise on the final structure before FSC computation (otherwise small box sizes lead to better resolutions but worse structures). This can raise concerns about exaggeration again, if masking isn't done with an appropriate bias-minimizing mask.

* Scheres, S. H. & Chen, S. (2012) Prevention of overfitting in cryo-EM structure determination. Nat. Methods. 9, 853-854.

* Murray, S. C., Flanagan, J., Popova, O. B., Chiu, W., Ludtke, S. J. & Serysheva, I. I. (2013) Validation of Cryo-EM Structure of IP3R1 Channel. Structure. 21, 1-10. PMC3696195

So, if you have any doubts, or are using some other method which you believe avoids bias, Richard's recent suggestion to randomize the phases in your raw data, then rerefine to prove that your algorithm doesn't claim to have achieved resolutions beyond this point can be used to really test your process.

* Chen, S., McMullan, G., Faruqi, A. R., Murshudov, G. N., Short, J. M., Scheres, S. H. & Henderson, R. (2013) High-resolution noise substitution to measure overfitting and validate resolution in 3D structure determination by single particle electron cryomicroscopy. Ultramicroscopy.


On Aug 28, 2013, at 6:27 AM, Marin van Heel <marin.vanheel at googlemail.com<mailto:marin.vanheel at googlemail.com>> wrote:



Thought of the day:

If you overheat your house, do you blame your thermometer?
It is rather your own responsibly to change your thermostat settings!
If you over-fit your 3D reconstruction, do you blame the FSC resolution measure?
It is rather your own responsibility to avoid overfitting by reference bias.
There is no such thing as a biased FSC (Fourier Shell Correlation). Neither is there a “gold standard” FSC!
Yes, you CAN do deviously biased data processing to try to prove you are better than everybody else.
But what ever you do, please don’t blame your thermometer.

Cheers,
Marin


--
================================================================

    Marin van Heel

    Professor of Cryo-EM Data Processing

    Leiden University
    NeCEN Building Room 05.27
    Einsteinweg 55
    2333 CC Leiden
    The Netherlands

    Skype:    Marin.van.Heel
    email:  marin.vanheel(A_T)gmail.com<http://gmail.com/>
    and:    mvh.office(A_T)gmail.com<http://gmail.com/>

----------------------------------------------
    and:

    Professor of Structural Biology

    Imperial College London
    Faculty of Natural Sciences
    Biochemistry Building (Room 512)
    South Kensington Campus
    London SW7 2AZ,  UK
    email:  m.vanheel(A_T)ic.ac.uk<http://ic.ac.uk/>

----------------------------------------------
    Currently visiting Professor at:

    Laboratório Nacional de Nanotecnologia - LNNano
    CNPEM/ABTLuS, Campinas, Brazil

------------------------------------------------------------------

I receive many emails per day and, although I try,
there is no guarantee that I will actually read each incoming email.
Moreover, our Spam filters can be strikt and sometimes make
legitimate emails disappear (try the gmail accounts, alternatively)


_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu<http://ncmir.ucsd.edu/>
https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem

----------------------------------------------------------------------------
Steven Ludtke, Ph.D.
Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem<http://www.bcm.edu/biochem>)
Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu<http://ncmi.bcm.edu/>)
Co-Director CIBR Center                          (www.bcm.edu/research/cibr<http://www.bcm.edu/research/cibr>)
Baylor College of Medicine
sludtke at bcm.edu<mailto:sludtke at bcm.edu>





_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu<http://ncmir.ucsd.edu>
https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem



More information about the 3dem mailing list