[3dem] [ccpem] on FSC curve (A can of worms...)

Wed Aug 12 11:17:10 PDT 2015

Hi Marin,

So many tasty worms in there. As you & others already know, I agree with 
you on the dangers of fixed-threshold criteria.

However, on the topic of the “SNR = (CCC/(1-CCC))” formula, I am not 
convinced by your argument involving CC=-1.

The reason is that this formula is really an /estimator/ for the true, 
unknown, SNR. This is explicitly stated by Bershard & Rockmore (1974), 
whose work Frank & Al-Ali (1975) builds on as well as by Frank & Al-Ali 
themselves. See in B&R (1974) equation 6, where the left-hand-side is an 
estimate for SNR (alpha circumflex in their notation) based on the right 
hand-side, which involves the sample cross-correlation (r in their 
notation). Or, indeed, see the title: "On estimating signal-to-noise 
ratio using the sample correlation coefficient ".

To put it bluntly, estimators should be expected to "fail" or "get it 
wrong" sometimes (i.e. if used after a single, one-off experiment). 
Thankfully, B&R derived estimates for the variance (error) of their 
estimator.  Saxton (1978) also derives this, and Pawel Penczek has a 
nice & detailed review (2010 Methods Enzym) of confidence intervals that 
can be derived from such estimator variances.

If the sample CC (FSC in a particular shell) comes out as -1, either (1) 
the fundamental assumption B&R used to derive the estimator, namely that 
we are measuring two noise-corrupted versions of the same signal, was 
violated and we shouldn't be using this estimator at all, or (2) the 
specific occurrences of the noise in our two measurements conspired to 
give us exactly anti-correlated measurements. If the number of 
measurements is not tiny, this is incredibly, incredibly unlikely. 
Therefore, no matter what the SNR estimator says (-0.5 in your example), 
it's OK that the truth is very different since if we were to repeat the 
experiment we would almost never, ever get the same (CC=-1) result again.

In fact, according to B&R, if we repeated the experiment an infinite 
number of times, the average estimate would be exactly correct (if you 
used their unbiased estimator, but even the one you mention is basically 
fine). The CC=-1 measurement would just be seen as a freak outlier. The 
distribution of estimates can be characterized, and this freak 
measurement would be way out in the tail.

I find no reason (yet?) to believe that B&R's estimator is wrong.

Cheers,
Alexis

-- 
Alexis Rohou

Research Specialist
Grigorieff Lab
http://grigoriefflab.janelia.org
Tel. +1 571 209 4000 x3485

On 08/12/2015 08:19 AM, Marin van Heel wrote:
>
> Dear Smith Liu,
>
> You have hit upon a can of worms here… Although the FRC/FSC metrics we 
> introduced in 1982/1986 [1, 2] are now considered the "gold standard" 
> cryo-EM resolution criterion, these resolution issues continue to be 
> heavily debated [3]. Many FSC add-ons/variants and tangential issues 
> such as “reference bias” have been inserted into the resolution 
> criterion discussion. These discussions unfortunately confuse even 
> established researchers (referees of major journals…), let alone 
> newcomers to the field. Many believe the resolution issue is better 
> resolved in X-crystallography. In fact, the FSC is arguably a better 
> metric than the R-factor, the generally accepted resolution metric in 
> X-ray crystallography [4]. Fortunately, FRC/FSC criteria are now 
> slowly also becoming the standard in optical microscopy, X-ray 
> microscopy, X-ray crystallography, and other fields of 2D/3D imaging.
>
> The most controversial part of the FSC discussion is the FSC threshold 
> value to serve as a resolution criterion (such as the FSC 0.5 value 
> you mention). It took more than a decade to remove the mathematically 
> flawed DPR (Differential Phase Residual) from the literature, after I 
> explicitly discussed its shortcomings and proposed a corrected phase 
> residual in 1987 [3]. The discussion in the field was then deviated 
> towards the FSC threshold at which one defines the average resolution 
> of a 3D structure. The “0.5” “criterion” was just postulated ad hoc, 
> without any scientific justification. Ten years ago, we argued that 
> all fixed-valued FSC threshold criteria (such as: “0.5” and “0.143”) 
> are based on flawed statistics [5]. Virtually all more formal 
> justifications for resolution criteria start off referring to the old 
> formula “SNR = (CCC/(1-CCC))” by Frank & Al-Ali  1975 [6]. 
> Unfortunately this formula is also mathematically incorrect as was 
> discussed previously [5].
>
> Here is another very simple argument to illustrate its flawed 
> definition: the normalised CCC (or FSC) has values in the range:  
> -1<=CCC<=+1, whereas the SNR (=S2/N2) is, per definition, positive. 
> Now insert the value CCC= -1, the case of perfectly anti-correlated 
> data, into the formula. This yields: SNR = “-0.5”, a rampant violation 
> of the SNR definition range. The formula could be valid for the 
> limiting case of CCC is close to unity, but such high correlation 
> values are not relevant in the resolution-threshold context. For 
> uncorrelated signals/noise the CCC oscillates around the zero mark 
> and, through the flawed Frank & Al-Ali formula, produces as many 
> positive as it does erroneous negative SNR values.
>
> Unfortunately, virtually all (~100?) papers on resolution criteria and 
> validation tests in cryo-EM (from friends and foes) are based on this 
> formula and are thus based on “flawed statistics” to say the least. 
> With the great recent success of cryo-EM, everybody appears to have 
> stopped thinking about the basics, and merrily continue to refer to 
> incorrect stuff while focusing on “my resolution is better than 
> yours”. After decades of funny jokes and verbal FSC controversies at 
> GRC meetings, I don’t find it so funny anymore: it is time to clean up 
> the mess. I have lost the patience to discuss these issues with 
> referees who continue to consider the subject as debatable. 
> Questionable actions are sometimes hidden behind this controversy such 
> as in Mao & Sodrosky [7], who cynically accuse us - their critics - of 
> not knowing how to interpret the FSC: “FSC estimates of resolution are 
> known to be quite sensitive to statistical bias …” etc. etc.  As I 
> said, this whole issue is no longer amusing; it has become a matter of 
> the debatable scientific culture (integrity?) in the field of the 
> cryo-EM field.
>
> Oh, by the way, Smith Liu, what I really was going to say when I 
> started typing an answer to your question is that if you are new to 
> the field it is a good idea to read some basic literature in Fourier 
> Optics. Maybe my lecture notes can help [8]. The horizontal axis in 
> the FSC is 1/spatial-frequency (we are in Fourier space) and the FSC 
> values in the curve indicate the cross-correlation level at that level 
> of resolution (= inside that specific 3D Fourier shell).
>
> Hope this helps,
>
> Marin
>
> [1] Van Heel M, Keegstra W, Schutter W, van Bruggen EFJ: Arthropod 
> hemocyanin structures studied by image analysis 
> http://singleparticles.org/methodology/MvH_FRC_Leeds_1982.pdf
> [2] Harauz G & van Heel M: Exact filters for general geometry three 
> dimensional reconstruction, Optik 73 (1986) 146-156
> [3]Van Heel M: Similarity measures between images. Ultramicroscopy 21 
> (1987) 95-100.]. [4] Van Heel: Unveiling ribosomal structures: the 
> final phases. Current Opinions in Structural Biology 10 (2000) 259-264.
> [5] Van Heel M & Schatz M:  Fourier Shell Correlation Threshold 
> Criteria, J. Struct. Biol. 151 (2005) 250-262
> [6] Frank J & Al-Ali L:  Signal-to-noise ratio of electron micrographs 
> obtained by cross correlation. Nature (1975)
> [7] Mao Y, Castillo-Menendeza LR, Sodroski JG: Reply to Subramaniam, 
> van Heel, and Henderson: Validity of the cryo-electron microscopy 
> structures of the HIV-1 envelope glycoprotein complex. PNAS 2013 
> www.pnas.org/cgi/doi/10.1073/pnas.1316666110
> [8] Van Heel:  Principles of Phase Contrast (Electron) Microscopy. 
> http://www.single-particles.org/methodology/MvH_Phase_Contrast.pdf
>
> ===========================================
>
>
>
> On 08/08/2015 07:45, Smith Liu wrote:
>> Dear All,
>>
>> I know the x-axis of the FSC curve is on the reverse of the 
>> resolution, and the value in the x-axis corresponding FSC 0.5 is 
>> usually regarded as the reverse of the resolution of the whole EM map.
>>
>> Here I do not know the meaning of the resolution in the X-axis. The 
>> Whole map has only one resolution corresponding FSC 0.5, then why 
>> the x-axis is on different resolutions (for example the x-axis is 
>> from resolution 0 to 20 A, or the reverse of that scope)? Is it 
>> because different parts of the map have different resolutions (caused 
>> by different parts of map  have different quality), or it is because 
>> the X-axis of the FSC curve has some relation with Fourier shell? If 
>> the X-axis of the FSC is on the property related to Fourier shell, 
>> then what is in the relation of resolution (or the reverse of it) in 
>> the x-axis with Fourier shell (in addition, what is the Fourier shell)?
>>
>> Best regards.
>>
>> Smith
>>
>>
>>
>
>
>
>
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20150812/39797a2e/attachment.html>