<html><head></head><body bgcolor="#FFFFFF" text="#000000" style="background-color: rgb(255, 255, 255); line-height: initial;"> <div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">Dear Marin and Pawel,</div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><br></div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">For the K2 camera that outputs counts (e.g. 1, 2, 3, 4, etc) there is no loss of information in storing these numbers as 4 bit or 8 bit as long as you don't exceed the highest integer that the data type can hold. Any excess bits just hold 0s. Storing as 32 bits does not cost much but it also has no purpose.</div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><br></div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">Best wishes, </div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">John</div> <div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><br style="display:initial"></div> <div style="font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">Sent from my BlackBerry 10 smartphone.</div> <table width="100%" style="background-color:white;border-spacing:0px;"> <tbody><tr><td colspan="2" style="font-size: initial; text-align: initial; background-color: rgb(255, 255, 255);"> <div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in; font-family: Tahoma, 'BB Alpha Sans', 'Slate Pro'; font-size: 10pt;"> <div><b>From: </b>Marin van Heel</div><div><b>Sent: </b>Tuesday, June 16, 2015 11:22 AM</div><div><b>To: </b>Tom Houweling; CCPEM@JISCMAIL.AC.UK</div><div><b>Cc: </b>3DEM</div><div><b>Subject: </b>Re: [3dem] [ccpem] MRC file format (Compressing cryo-EM data to 8-bits/pix and beyond)</div></div></td></tr></tbody></table><div style="border-style: solid none none; border-top-color: rgb(186, 188, 209); border-top-width: 1pt; font-size: initial; text-align: initial; background-color: rgb(255, 255, 255);"></div><br><div id="_originalContent" style="background-color: rgb(255, 255, 255);">
<meta content="text/html; charset=windows-1252" http-equiv="Content-Type">
<div class="moz-cite-prefix">Dear All,<br>
<br>
For various reasons I don’t think this line of reasoning is very
productive. The data compression to 8 or even 4 bits as has been
suggested in this discussion can only lead to loss of data (see
below). It may also represent poor management of the available EM
resources.<br>
<br>
Point by point:<br>
<br>
A) Advanced cryo-EM equipment costs of the order of ~5000 AUs
(Arbitrary Units: $/Eu/£) per day to own and operate, and will
generate up to ~ 2Tbyte of cryo-EM data per 24h. The costs of
storing this precious data for “eternity” will not exceed 100 AUs
per day, that is, one or two percent of the tax-payers total
investment in your data collection. NOT storing that raw data may
NOT be a good idea for economic reasons alone (just in case you,
for example, need to repeat the experiment to get the data back).<br>
<br>
B) Compressing all the raw data to save space can make sense as
long as the compression is loss-less
(<a class="moz-txt-link-freetext" href="https://en.wikipedia.org/wiki/Lossless_compression">https://en.wikipedia.org/wiki/Lossless_compression</a>). The
compression (after movie alignment) as suggested, however, may
lead to a significant information loss. <br>
<br>
C) The dynamic range of a raw image is mainly determined by the
low-frequency components of the data. Scaling the min-max
densities from 0-255 for compression/truncation to 8 bit data,
changes the data representation from image to image. The
high-resolution information we are interested is has a contrast of
probably less than 0.1% of the strong low-frequency components.
The signal we are interested in is thus already much smaller than
the discretisation error of 1:256 of the A-to-D conversion. That
does not mean one will not be able to fish that information from
the discretisation and Poisson noise in the raw data… But it will
certainly suffer. The grey scales will change from image to image
purely dependent on whether there is, for example, an ice crystal
somewhere in the field of view. High-pass filtering will remove
the large-scale details thus also increase the dynamic range
available for the high-res frequency data components. <br>
<br>
D) Note that the fact that you manage to get a 3D structure out is
no proof that you have not lost information. It is merely proof
for the fact that there was enough left over to create a
reasonable 3D that satisfies you. <br>
<br>
E) There are also other reasons for never deleting the original
data such as validation! You may be challenged – as has happened
in the recent past (PNAS 2013) - to show the original data set to
prove it is what you claim it is and was collected on the
instrumentation you claim it was taken on. (In the PNAS cases the
original data has still not been released).<br>
<br>
F) What one can or wants to do with the raw data changes over
time. Many new movie alignment algorithms have been proposed
recently; access to exactly the same raw data is essential for
validation of the new algorithms. (You may even get more out of
your data!)<br>
<br>
G) The raw data characterizes the camera (and validates the data
set as per E) and allow you to correct for its flaws
(<a class="moz-txt-link-freetext" href="http://www.nature.com/srep/2015/150611/srep10317/full/srep10317.html">http://www.nature.com/srep/2015/150611/srep10317/full/srep10317.html</a>).
You may also want to see whether the camera itself deteriorated
over time.<br>
<br>
H) Especially when the raw data are of some integer type, (and you
are using data with a limited dynamic range), the data on disk
will be written in a highly redundant fashion. You may then use
loss-less compression algorithms to reduce the size of your data
without suffering any information loss. You may always compress
the data, you may never compromise on its information content!<br>
<br>
Cheers, Marin<br>
<br>
========================================<br>
<br>
On 04/06/2015 00:15, Tom Houweling wrote:<br>
</div>
<blockquote cite="mid:1BFCB93C-8D3B-4B12-BEAE-4816AB6B7CDC@berkeley.edu" type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div class="">What I meant is that Relion appears to have no
problem reading 16 bit and 8 bit formats, therefore converting
to 32bit floating point images should not be necessary.</div>
<div class=""><br class="">
</div>
<div class="">However, the verdict on loss of resolution reducing
the data to 8 bits is still out. I’m motivated by conserving
disk space. </div>
<div class=""><br class="">
</div>
<div class="">I’m currently reprocessing a good dataset that
yielded a high resolution structure. But this time I converted
the aligned stacks of 32bit per pixel to just 8 by the following
method:</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre">
</span>1)<span class="Apple-tab-span" style="white-space:pre"> </span>Calculate
the mean and std. deviation</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre">
</span>2)<span class="Apple-tab-span" style="white-space:pre"> </span>Cutoff
at +/- 3 std dev</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre">
</span>3)<span class="Apple-tab-span" style="white-space:pre"> </span>Set
lowest value to 0 and highest to 255 </div>
<div class=""><br class="">
</div>
<div class="">Tom</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Jun 3, 2015, at 10:58 AM, Amedee des
Georges <<a moz-do-not-send="true" href="mailto:adesgeorges@GMAIL.COM" class="">adesgeorges@GMAIL.COM</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode:
space; -webkit-line-break: after-white-space;" class="">Dear
Tom,
<div class=""><br class="">
</div>
<div class="">Did you see any decrease in resolution
with 8bit vs 16? How did it look? </div>
<div class="">It’s obviously an advantage to use 8bits
for storage if it doesn’t decrease image quality
significantly. </div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class=""><br class="">
</div>
<div class="">Amedee</div>
<div class=""><br class="">
<div class="">
<div class="">On Jun 3, 2015, at 1:44 PM, Tom
Houweling <<a moz-do-not-send="true" href="mailto:tom.houweling@BERKELEY.EDU" class="">tom.houweling@BERKELEY.EDU</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite" class="">
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252" class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space;" class="">
<div class="">We have successfully processed MRC
images and stacks in Relion that were in 16
bit mode 6 and also in the non MRC sanctioned
mode 5 (8 bit unsigned).</div>
<div class=""><br class="">
</div>
<div class="">—Tom</div>
<div class=""><br class="">
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Jun 3, 2015, at 10:22 AM,
Rémi Fronzes <<a moz-do-not-send="true" href="mailto:remi.fronzes@PASTEUR.FR" class="">remi.fronzes@PASTEUR.FR</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">Dear All,
<div class=""><br class="">
</div>
<div class="">Maybe a silly question but
still worth asking. </div>
<div class="">Is it a problem to extract
and use in relion particles from
16bits MRC images (i.e. collected
using EPU) ?</div>
<div class="">Or do we have to convert
the micrographs in 32 bits MRC format.</div>
<div class=""><br class="">
</div>
<div class="">Cheers</div>
<div class=""><br class="">
</div>
<div class="">Rémi<br class="">
<div class="">
<div style="font-family: Helvetica;
font-size: inherit; font-style:
normal; font-variant: normal;
font-weight: normal;
letter-spacing: normal;
line-height: normal; orphans: 2;
text-align: -webkit-auto;
text-indent: 0px; text-transform:
none; white-space: normal; widows:
2; word-spacing: 0px;
-webkit-text-stroke-width: 0px;" class="">
<div style="margin: 0px;
font-size: 18px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class=""><span style="letter-spacing: 0px; " class=""><br class="Apple-interchange-newline">
<br class="">
</span></div>
<div style="margin: 0px;
font-size: 18px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class=""><span style="letter-spacing: 0px; " class="">Rémi Fronzes</span></div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class=""><span style="letter-spacing: 0px; " class="">G5 biologie
structurale de la sécrétion
bactérienne, institut Pasteur</span></div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class=""><span style="letter-spacing: 0px; " class="">CNRS UMR 3528,
institut Pasteur</span></div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); min-height: 13px; " class=""><span style="letter-spacing: 0px; " class=""></span><br class="">
</div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class=""><span style="letter-spacing: 0px; " class="">Office: +33
(0)145688864</span></div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class=""><span style="letter-spacing: 0px; " class="">Lab: +33 (0)
145688863</span></div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(0,
78, 135); " class=""><span style="letter-spacing: 0px; " class="">Mobile: +33 (0)
688263992</span></div>
<div style="margin: 0px;
font-size: 12px; font-family:
RotisSansSerif; color: rgb(2,
30, 170); " class=""><span style="letter-spacing: 0px;
color: rgb(0, 78, 135); " class="">Email:<span class="Apple-converted-space"> </span><span style="letter-spacing: 0px;
color: rgb(2, 30, 170); " class=""><a moz-do-not-send="true" href="mailto:remi.fronzes@pasteur.fr" class="">remi.fronzes@pasteur.fr</a></span></span></div>
</div>
<div style="font-size: 12px;
font-style: normal; font-variant:
normal; font-weight: normal;
letter-spacing: normal;
line-height: normal; orphans: 2;
text-align: -webkit-auto;
text-indent: 0px; text-transform:
none; white-space: normal; widows:
2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px;
margin: 0px; font-family:
RotisSansSerif; color: rgb(2, 30,
170); " class=""><span style="letter-spacing: 0px;
color: rgb(0, 78, 135); " class=""><br class="">
</span></div>
<div style="font-size: 12px;
font-style: normal; font-variant:
normal; font-weight: normal;
letter-spacing: normal;
line-height: normal; orphans: 2;
text-align: -webkit-auto;
text-indent: 0px; text-transform:
none; white-space: normal; widows:
2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px;
margin: 0px; font-family:
RotisSansSerif; color: rgb(2, 30,
170); " class="">
<div style="margin: 0px; color:
rgb(0, 86, 148); " class=""><span style="letter-spacing: 0px; " class="">25 rue du Docteur
Roux </span></div>
<div style="margin: 0px; color:
rgb(0, 86, 148); " class=""><span style="letter-spacing: 0px; " class="">Bâtiment Metchnikoff,
3ème étage</span></div>
<div style="margin: 0px; color:
rgb(0, 86, 148); " class=""><span style="letter-spacing: 0px; " class="">75015 Paris, France </span></div>
</div>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div class="">--<br class="">
Tom Houweling - QB3 Nogales Lab Computer
Analyst @ Howard Hughes Medical Institute<br class="">
University of California Berkeley,
708D Stanley Hall, Berkeley, CA 94720<br class="">
<br class="">
</div>
</div>
<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div class="">--<br class="">
Tom Houweling - QB3 Nogales Lab Computer Analyst @ Howard
Hughes Medical Institute<br class="">
University of California Berkeley, 708D Stanley Hall,
Berkeley, CA 94720<br class="">
<br class="">
</div>
</div>
<br class="">
</div>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
================================================================
Prof Dr Ir Marin van Heel
Professor of Cryo-EM Data Processing
Leiden University
</pre>
<br><!--end of _originalContent --></div><br></body></html>