<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Marin,<br>
<br>
So many tasty worms in there. As you & others already know, I
agree with you on the dangers of fixed-threshold criteria. <br>
<br>
However, on the topic of the “SNR = (CCC/(1-CCC))” formula, I am not
convinced by your argument involving CC=-1.<br>
<br>
The reason is that this formula is really an <i>estimator</i> for
the true, unknown, SNR. This is explicitly stated by Bershard &
Rockmore (1974), whose work Frank & Al-Ali (1975) builds on as
well as by Frank & Al-Ali themselves. See in B&R (1974)
equation 6, where the left-hand-side is an estimate for SNR (alpha
circumflex in their notation) based on the right hand-side, which
involves the sample cross-correlation (r in their notation). Or,
indeed, see the title: "On estimating signal-to-noise ratio using
the sample correlation coefficient
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css">
p, li { white-space: pre-wrap; }
</style>".<br>
<br>
To put it bluntly, estimators should be expected to "fail" or "get
it wrong" sometimes (i.e. if used after a single, one-off
experiment). Thankfully, B&R derived estimates for the variance
(error) of their estimator. Saxton (1978) also derives this, and
Pawel Penczek has a nice & detailed review (2010 Methods Enzym)
of confidence intervals that can be derived from such estimator
variances. <br>
<br>
If the sample CC (FSC in a particular shell) comes out as -1, either
(1) the fundamental assumption B&R used to derive the estimator,
namely that we are measuring two noise-corrupted versions of the
same signal, was violated and we shouldn't be using this estimator
at all, or (2) the specific occurrences of the noise in our two
measurements conspired to give us exactly anti-correlated
measurements. If the number of measurements is not tiny, this is
incredibly, incredibly unlikely. Therefore, no matter what the SNR
estimator says (-0.5 in your example), it's OK that the truth is
very different since if we were to repeat the experiment we would
almost never, ever get the same (CC=-1) result again. <br>
<br>
In fact, according to B&R, if we repeated the experiment an
infinite number of times, the average estimate would be exactly
correct (if you used their unbiased estimator, but even the one you
mention is basically fine). The CC=-1 measurement would just be seen
as a freak outlier. The distribution of estimates can be
characterized, and this freak measurement would be way out in the
tail.<br>
<br>
I find no reason (yet?) to believe that B&R's estimator is
wrong.<br>
<br>
Cheers,<br>
Alexis<br>
<br>
<br>
<br>
-- <br>
Alexis Rohou<br>
<br>
Research Specialist<br>
Grigorieff Lab<br>
<a href="http://grigoriefflab.janelia.org">http://grigoriefflab.janelia.org</a><br>
Tel. +1 571 209 4000 x3485<br>
<br>
<br>
<div class="moz-cite-prefix">On 08/12/2015 08:19 AM, Marin van Heel
wrote:<br>
</div>
<blockquote cite="mid:55CB39E0.7020306@googlemail.com" type="cite">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
<div class="moz-cite-prefix"><br>
Dear Smith Liu,<br>
<br>
You have hit upon a can of worms here… Although the FRC/FSC
metrics we introduced in 1982/1986 [1, 2] are now considered the
"gold standard" cryo-EM resolution criterion, these resolution
issues continue to be heavily debated [3]. Many FSC
add-ons/variants and tangential issues such as “reference bias”
have been inserted into the resolution criterion discussion.
These discussions unfortunately confuse even established
researchers (referees of major journals…), let alone newcomers
to the field. Many believe the resolution issue is better
resolved in X-crystallography. In fact, the FSC is arguably a
better metric than the R-factor, the generally accepted
resolution metric in X-ray crystallography [4]. Fortunately,
FRC/FSC criteria are now slowly also becoming the standard in
optical microscopy, X-ray microscopy, X-ray crystallography, and
other fields of 2D/3D imaging.<br>
<br>
The most controversial part of the FSC discussion is the FSC
threshold value to serve as a resolution criterion (such as the
FSC 0.5 value you mention). It took more than a decade to remove
the mathematically flawed DPR (Differential Phase Residual) from
the literature, after I explicitly discussed its shortcomings
and proposed a corrected phase residual in 1987 [3]. The
discussion in the field was then deviated towards the FSC
threshold at which one defines the average resolution of a 3D
structure. The “0.5” “criterion” was just postulated ad hoc,
without any scientific justification. Ten years ago, we argued
that all fixed-valued FSC threshold criteria (such as: “0.5” and
“0.143”) are based on flawed statistics [5]. Virtually all more
formal justifications for resolution criteria start off
referring to the old formula “SNR = (CCC/(1-CCC))” by Frank
& Al-Ali 1975 [6]. Unfortunately this formula is also
mathematically incorrect as was discussed previously [5]. <br>
<br>
Here is another very simple argument to illustrate its flawed
definition: the normalised CCC (or FSC) has values in the
range: -1<=CCC<=+1, whereas the SNR (=S2/N2) is, per
definition, positive. Now insert the value CCC= -1, the case of
perfectly anti-correlated data, into the formula. This yields:
SNR = “-0.5”, a rampant violation of the SNR definition range.
The formula could be valid for the limiting case of CCC is close
to unity, but such high correlation values are not relevant in
the resolution-threshold context. For uncorrelated signals/noise
the CCC oscillates around the zero mark and, through the flawed
Frank & Al-Ali formula, produces as many positive as it does
erroneous negative SNR values.<br>
<br>
Unfortunately, virtually all (~100?) papers on resolution
criteria and validation tests in cryo-EM (from friends and foes)
are based on this formula and are thus based on “flawed
statistics” to say the least. With the great recent success of
cryo-EM, everybody appears to have stopped thinking about the
basics, and merrily continue to refer to incorrect stuff while
focusing on “my resolution is better than yours”. After decades
of funny jokes and verbal FSC controversies at GRC meetings, I
don’t find it so funny anymore: it is time to clean up the mess.
I have lost the patience to discuss these issues with referees
who continue to consider the subject as debatable. Questionable
actions are sometimes hidden behind this controversy such as in
Mao & Sodrosky [7], who cynically accuse us - their critics
- of not knowing how to interpret the FSC: “FSC estimates of
resolution are known to be quite sensitive to statistical bias
…” etc. etc. As I said, this whole issue is no longer amusing;
it has become a matter of the debatable scientific culture
(integrity?) in the field of the cryo-EM field. <br>
<br>
Oh, by the way, Smith Liu, what I really was going to say when I
started typing an answer to your question is that if you are new
to the field it is a good idea to read some basic literature in
Fourier Optics. Maybe my lecture notes can help [8]. The
horizontal axis in the FSC is 1/spatial-frequency (we are in
Fourier space) and the FSC values in the curve indicate the
cross-correlation level at that level of resolution (= inside
that specific 3D Fourier shell).<br>
<br>
Hope this helps,<br>
<br>
Marin<br>
<br>
[1] Van Heel M, Keegstra W, Schutter W, van Bruggen EFJ:
Arthropod hemocyanin structures studied by image analysis <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://singleparticles.org/methodology/MvH_FRC_Leeds_1982.pdf">http://singleparticles.org/methodology/MvH_FRC_Leeds_1982.pdf</a><br>
[2] Harauz G & van Heel M: Exact filters for general
geometry three dimensional reconstruction, Optik 73 (1986)
146-156<br>
[3]Van Heel M: Similarity measures between images.
Ultramicroscopy 21 (1987) 95-100.]. [4] Van Heel: Unveiling
ribosomal structures: the final phases. Current Opinions in
Structural Biology 10 (2000) 259-264.<br>
[5] Van Heel M & Schatz M: Fourier Shell Correlation
Threshold Criteria, J. Struct. Biol. 151 (2005) 250-262<br>
[6] Frank J & Al-Ali L: Signal-to-noise ratio of electron
micrographs obtained by cross correlation. Nature (1975)<br>
[7] Mao Y, Castillo-Menendeza LR, Sodroski JG: Reply to
Subramaniam, van Heel, and Henderson: Validity of the
cryo-electron microscopy structures of the HIV-1 envelope
glycoprotein complex. PNAS 2013 <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="http://www.pnas.org/cgi/doi/10.1073/pnas.1316666110">www.pnas.org/cgi/doi/10.1073/pnas.1316666110</a><br>
[8] Van Heel: Principles of Phase Contrast (Electron)
Microscopy. <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://www.single-particles.org/methodology/MvH_Phase_Contrast.pdf">http://www.single-particles.org/methodology/MvH_Phase_Contrast.pdf</a><br>
<br>
===========================================<br>
<br>
<br>
<br>
On 08/08/2015 07:45, Smith Liu wrote:<br>
</div>
<blockquote
cite="mid:10144882.10d59.14f0ceab82b.Coremail.smith_liu123@163.com"
type="cite">
<div
style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial">
<div style="LINE-HEIGHT: 1.7; FONT-FAMILY: Arial; COLOR:
#000000; FONT-SIZE: 14px">
<div>Dear All, </div>
<div><br>
</div>
<div>I know the x-axis of the FSC curve is on the reverse of
the resolution, and the value in the x-axis corresponding
FSC 0.5 is usually regarded as the reverse of the
resolution of the whole EM map.</div>
<div><br>
</div>
<div>Here I do not know the meaning of the resolution in the
X-axis. The Whole map has only one resolution
corresponding FSC 0.5, then why the x-axis is on different
resolutions (for example the x-axis is from resolution 0
to 20 A, or the reverse of that scope)? Is it because
different parts of the map have different resolutions
(caused by different parts of map have different
quality), or it is because the X-axis of the FSC curve has
some relation with Fourier shell? If the X-axis of the FSC
is on the property related to Fourier shell, then what is
in the relation of resolution (or the reverse of it) in
the x-axis with Fourier shell (in addition, what is the
Fourier shell)?</div>
<div><br>
</div>
<div>Best regards.</div>
<div><br>
</div>
<div>Smith</div>
<div> </div>
</div>
<br>
</div>
<br>
<br>
<span title="neteasefooter"><span id="netease_mail_footer"></span></span>
</blockquote>
<br>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
3dem mailing list
<a class="moz-txt-link-abbreviated" href="mailto:3dem@ncmir.ucsd.edu">3dem@ncmir.ucsd.edu</a>
<a class="moz-txt-link-freetext" href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a>
</pre>
</blockquote>
<br>
</body>
</html>