<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Movies from an electron-counting camera (e.g. K2) can be compressed losslessly quite nicely. We use the option in SerialEM to write the raw counts (*not* the gain-corrected data) as compressed TIFF files and they are stored at about 4 bits per pixel.
<div class=""><br class="">
</div>
<div class="">As Marin says, reducing the resolution of “integrating-mode” movie or image data can be dangerous because the quantizing noise (it is theoretically 1/sqrt(12) least-significant bit, RMS) can become comparable to the original noise and thus degrade
the SNR. This is particularly serious when the MTF of the camera dampens the high-frequency signal and noise a lot so they are not faithfully represented with a few bits of resolution.</div>
<div class=""><br class="">
</div>
<div class="">I’ve been experimenting with lossy compression of Falcon-2 and DD movies. Theoretically the information content is only a few bits per pixel. Filtering can be used to counteract the MTF so that the quantizing noise spectral density is kept 100x
or 1000x below the noise spectral density. This means that the SSNR is degraded by only 1% or 0.1% at all frequencies. For these levels of compression the output files are about 4.5 or 5.5 bits per pixel, respectively. It seems to me that archiving movies
with compression of this sort, especially in online repositories, could be useful.</div>
<div class=""><br class="">
</div>
<div class="">Fred Sigworth</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div apple-content-edited="true" class="">Fred J. Sigworth<br class="">
Professor<br class="">
Yale University<br class="">
Cell Molec Physiology and<br class="">
Dept Biomedical Engineering<br class="">
<a href="mailto:fred.sigworth@yale.edu" class="">fred.sigworth@yale.edu</a><br class="">
<br class="">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Jun 16, 2015, at 11:20 AM, Marin van Heel <<a href="mailto:marin.vanheel@googlemail.com" class="">marin.vanheel@googlemail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="moz-cite-prefix">Dear All,<br class="">
<br class="">
For various reasons I don’t think this line of reasoning is very productive. The data compression to 8 or even 4 bits as has been suggested in this discussion can only lead to loss of data (see below). It may also represent poor management of the available
EM resources.<br class="">
<br class="">
Point by point:<br class="">
<br class="">
A) Advanced cryo-EM equipment costs of the order of ~5000 AUs (Arbitrary Units: $/Eu/£) per day to own and operate, and will generate up to ~ 2Tbyte of cryo-EM data per 24h. The costs of storing this precious data for “eternity” will not exceed 100 AUs per
day, that is, one or two percent of the tax-payers total investment in your data collection. NOT storing that raw data may NOT be a good idea for economic reasons alone (just in case you, for example, need to repeat the experiment to get the data back).<br class="">
<br class="">
B) Compressing all the raw data to save space can make sense as long as the compression is loss-less (<a class="moz-txt-link-freetext" href="https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Lossless-5Fcompression&d=AwMF-g&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=0CYI4NKLGjHxGsE4MJzRWuiVDcoaSNJAozflvQKB2rk&m=2B-TZWkT1dYnk74TiLHGbEDjJaQHQiWgdF2nfQvcPsA&s=V1LK6pWb7pWmncSqqo-pE109153jQGKzqdXcv7i66PQ&e=">https://en.wikipedia.org/wiki/Lossless_compression</a>).
The compression (after movie alignment) as suggested, however, may lead to a significant information loss.
<br class="">
<br class="">
C) The dynamic range of a raw image is mainly determined by the low-frequency components of the data. Scaling the min-max densities from 0-255 for compression/truncation to 8 bit data, changes the data representation from image to image. The high-resolution
information we are interested is has a contrast of probably less than 0.1% of the strong low-frequency components. The signal we are interested in is thus already much smaller than the discretisation error of 1:256 of the A-to-D conversion. That does not mean
one will not be able to fish that information from the discretisation and Poisson noise in the raw data… But it will certainly suffer. The grey scales will change from image to image purely dependent on whether there is, for example, an ice crystal somewhere
in the field of view. High-pass filtering will remove the large-scale details thus also increase the dynamic range available for the high-res frequency data components.
<br class="">
<br class="">
D) Note that the fact that you manage to get a 3D structure out is no proof that you have not lost information. It is merely proof for the fact that there was enough left over to create a reasonable 3D that satisfies you.
<br class="">
<br class="">
E) There are also other reasons for never deleting the original data such as validation! You may be challenged – as has happened in the recent past (PNAS 2013) - to show the original data set to prove it is what you claim it is and was collected on the instrumentation
you claim it was taken on. (In the PNAS cases the original data has still not been released).<br class="">
<br class="">
F) What one can or wants to do with the raw data changes over time. Many new movie alignment algorithms have been proposed recently; access to exactly the same raw data is essential for validation of the new algorithms. (You may even get more out of your data!)<br class="">
<br class="">
G) The raw data characterizes the camera (and validates the data set as per E) and allow you to correct for its flaws (<a class="moz-txt-link-freetext" href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.nature.com_srep_2015_150611_srep10317_full_srep10317.html&d=AwMF-g&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=0CYI4NKLGjHxGsE4MJzRWuiVDcoaSNJAozflvQKB2rk&m=2B-TZWkT1dYnk74TiLHGbEDjJaQHQiWgdF2nfQvcPsA&s=mK5yRXErw3CaBNPIpc_skVjUgfa1zLp_1B9iodehASE&e=">http://www.nature.com/srep/2015/150611/srep10317/full/srep10317.html</a>).
You may also want to see whether the camera itself deteriorated over time.<br class="">
<br class="">
H) Especially when the raw data are of some integer type, (and you are using data with a limited dynamic range), the data on disk will be written in a highly redundant fashion. You may then use loss-less compression algorithms to reduce the size of your data
without suffering any information loss. You may always compress the data, you may never compromise on its information content!<br class="">
<br class="">
Cheers, Marin<br class="">
<br class="">
========================================<br class="">
<br class="">
On 04/06/2015 00:15, Tom Houweling wrote:<br class="">
</div>
<blockquote cite="mid:1BFCB93C-8D3B-4B12-BEAE-4816AB6B7CDC@berkeley.edu" type="cite" class="">
<div class="">What I meant is that Relion appears to have no problem reading 16 bit and 8 bit formats, therefore converting to 32bit floating point images should not be necessary.</div>
<div class=""><br class="">
</div>
<div class="">However, the verdict on loss of resolution reducing the data to 8 bits is still out. I’m motivated by conserving disk space. </div>
<div class=""><br class="">
</div>
<div class="">I’m currently reprocessing a good dataset that yielded a high resolution structure. But this time I converted the aligned stacks of 32bit per pixel to just 8 by the following method:</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>1)<span class="Apple-tab-span" style="white-space:pre">
</span>Calculate the mean and std. deviation</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>2)<span class="Apple-tab-span" style="white-space:pre">
</span>Cutoff at +/- 3 std dev</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>3)<span class="Apple-tab-span" style="white-space:pre">
</span>Set lowest value to 0 and highest to 255 </div>
<div class=""><br class="">
</div>
<div class="">Tom</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Jun 3, 2015, at 10:58 AM, Amedee des Georges <<a moz-do-not-send="true" href="mailto:adesgeorges@GMAIL.COM" class="">adesgeorges@GMAIL.COM</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode:
space; -webkit-line-break: after-white-space;" class="">
Dear Tom,
<div class=""><br class="">
</div>
<div class="">Did you see any decrease in resolution with 8bit vs 16? How did it look? </div>
<div class="">It’s obviously an advantage to use 8bits for storage if it doesn’t decrease image quality significantly. </div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class=""><br class="">
</div>
<div class="">Amedee</div>
<div class=""><br class="">
<div class="">
<div class="">On Jun 3, 2015, at 1:44 PM, Tom Houweling <<a moz-do-not-send="true" href="mailto:tom.houweling@BERKELEY.EDU" class="">tom.houweling@BERKELEY.EDU</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite" class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space;" class="">
<div class="">We have successfully processed MRC images and stacks in Relion that were in 16 bit mode 6 and also in the non MRC sanctioned mode 5 (8 bit unsigned).</div>
<div class=""><br class="">
</div>
<div class="">—Tom</div>
<div class=""><br class="">
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Jun 3, 2015, at 10:22 AM, Rémi Fronzes <<a moz-do-not-send="true" href="mailto:remi.fronzes@PASTEUR.FR" class="">remi.fronzes@PASTEUR.FR</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">
Dear All,
<div class=""><br class="">
</div>
<div class="">Maybe a silly question but still worth asking. </div>
<div class="">Is it a problem to extract and use in relion particles from 16bits MRC images (i.e. collected using EPU) ?</div>
<div class="">Or do we have to convert the micrographs in 32 bits MRC format.</div>
<div class=""><br class="">
</div>
<div class="">Cheers</div>
<div class=""><br class="">
</div>
<div class="">Rémi<br class="">
<div class="">
<div style="font-family: Helvetica;
font-size: inherit; font-style:
normal; font-variant: normal;
font-weight: normal;
letter-spacing: normal;
line-height: normal; orphans: 2;
text-align: -webkit-auto;
text-indent: 0px; text-transform:
none; white-space: normal; widows:
2; word-spacing: 0px;
-webkit-text-stroke-width: 0px;" class="">
<div style="margin: 0px;
font-size: 18px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class="">
<span style="letter-spacing: 0px; " class=""><br class="Apple-interchange-newline">
<br class="">
</span></div>
<div style="margin: 0px;
font-size: 18px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class="">
<span style="letter-spacing: 0px; " class="">Rémi Fronzes</span></div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class="">
<span style="letter-spacing: 0px; " class="">G5 biologie structurale de la sécrétion bactérienne, institut Pasteur</span></div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class="">
<span style="letter-spacing: 0px; " class="">CNRS UMR 3528, institut Pasteur</span></div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); min-height: 13px; " class="">
<span style="letter-spacing: 0px; " class=""></span><br class="">
</div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class="">
<span style="letter-spacing: 0px; " class="">Office: +33 (0)145688864</span></div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class="">
<span style="letter-spacing: 0px; " class="">Lab: +33 (0) 145688863</span></div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class="">
<span style="letter-spacing: 0px; " class="">Mobile: +33 (0) 688263992</span></div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(2,
30, 170); " class="">
<span style="letter-spacing: 0px;
color: rgb(0, 78, 135); " class="">Email:<span class="Apple-converted-space"> </span><span style="letter-spacing: 0px;
color: rgb(2, 30, 170); " class=""><a moz-do-not-send="true" href="mailto:remi.fronzes@pasteur.fr" class="">remi.fronzes@pasteur.fr</a></span></span></div>
</div>
<div style="font-size: 12px;
font-style: normal; font-variant:
normal; font-weight: normal;
letter-spacing: normal;
line-height: normal; orphans: 2;
text-align: -webkit-auto;
text-indent: 0px; text-transform:
none; white-space: normal; widows:
2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px;
margin: 0px; font-family:
RotisSansSerif; color: rgb(2, 30,
170); " class="">
<span style="letter-spacing: 0px;
color: rgb(0, 78, 135); " class=""><br class="">
</span></div>
<div style="font-size: 12px;
font-style: normal; font-variant:
normal; font-weight: normal;
letter-spacing: normal;
line-height: normal; orphans: 2;
text-align: -webkit-auto;
text-indent: 0px; text-transform:
none; white-space: normal; widows:
2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px;
margin: 0px; font-family:
RotisSansSerif; color: rgb(2, 30,
170); " class="">
<div style="margin: 0px; color:
rgb(0, 86, 148); " class="">
<span style="letter-spacing: 0px; " class="">25 rue du Docteur Roux </span></div>
<div style="margin: 0px; color:
rgb(0, 86, 148); " class="">
<span style="letter-spacing: 0px; " class="">Bâtiment Metchnikoff, 3ème étage</span></div>
<div style="margin: 0px; color:
rgb(0, 86, 148); " class="">
<span style="letter-spacing: 0px; " class="">75015 Paris, France </span></div>
</div>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div class="">--<br class="">
Tom Houweling - QB3 Nogales Lab Computer Analyst @ Howard Hughes Medical Institute<br class="">
University of California Berkeley, 708D Stanley Hall, Berkeley, CA 94720<br class="">
<br class="">
</div>
</div>
<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div class="">--<br class="">
Tom Houweling - QB3 Nogales Lab Computer Analyst @ Howard Hughes Medical Institute<br class="">
University of California Berkeley, 708D Stanley Hall, Berkeley, CA 94720<br class="">
<br class="">
</div>
</div>
<br class="">
</div>
</blockquote>
<br class="">
<br class="">
<pre class="moz-signature" cols="72">--
================================================================
Prof Dr Ir Marin van Heel
Professor of Cryo-EM Data Processing
Leiden University
</pre>
</div>
_______________________________________________<br class="">
3dem mailing list<br class="">
<a href="mailto:3dem@ncmir.ucsd.edu" class="">3dem@ncmir.ucsd.edu</a><br class="">
https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=AwICAg&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=0CYI4NKLGjHxGsE4MJzRWuiVDcoaSNJAozflvQKB2rk&m=2B-TZWkT1dYnk74TiLHGbEDjJaQHQiWgdF2nfQvcPsA&s=SdoimShX4zOjgEPBY7RmgaE4HRENF_PBykDlWjMImA8&e=
<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>