<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix"><br>
Yes that was understood John. The point is that with loss-less
compression you can achieve even smaller files than with 4/8/16/32
bit integers, because those, as you state correctly, are largely
filled with zeroes.<br>
<br>
Cheers<br>
<br>
Marin<br>
<br>
<br>
On 16/06/2015 18:02, John Rubinstein wrote:<br>
</div>
<blockquote
cite="mid:20150616160248.185593978.63297.1647@utoronto.ca"
type="cite">
<div style="width: 100%; font-size: initial; font-family: Calibri,
'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align:
initial; background-color: rgb(255, 255, 255);">Dear Marin and
Pawel,</div>
<div style="width: 100%; font-size: initial; font-family: Calibri,
'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align:
initial; background-color: rgb(255, 255, 255);"><br>
</div>
<div style="width: 100%; font-size: initial; font-family: Calibri,
'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align:
initial; background-color: rgb(255, 255, 255);">For the K2
camera that outputs counts (e.g. 1, 2, 3, 4, etc) there is no
loss of information in storing these numbers as 4 bit or 8 bit
as long as you don't exceed the highest integer that the data
type can hold. Any excess bits just hold 0s. Storing as 32 bits
does not cost much but it also has no purpose.</div>
<div style="width: 100%; font-size: initial; font-family: Calibri,
'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align:
initial; background-color: rgb(255, 255, 255);"><br>
</div>
<div style="width: 100%; font-size: initial; font-family: Calibri,
'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align:
initial; background-color: rgb(255, 255, 255);">Best wishes, </div>
<div style="width: 100%; font-size: initial; font-family: Calibri,
'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align:
initial; background-color: rgb(255, 255, 255);">John</div>
<div style="width: 100%; font-size: initial; font-family: Calibri,
'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align:
initial; background-color: rgb(255, 255, 255);"><br
style="display:initial">
</div>
<div style="font-size: initial; font-family: Calibri, 'Slate Pro',
sans-serif; color: rgb(31, 73, 125); text-align: initial;
background-color: rgb(255, 255, 255);">Sent from my BlackBerry 10 smartphone.</div>
<table style="background-color:white;border-spacing:0px;"
width="100%">
<tbody>
<tr>
<td colspan="2" style="font-size: initial; text-align:
initial; background-color: rgb(255, 255, 255);">
<div style="border-style: solid none none;
border-top-color: rgb(181, 196, 223); border-top-width:
1pt; padding: 3pt 0in 0in; font-family: Tahoma, 'BB
Alpha Sans', 'Slate Pro'; font-size: 10pt;">
<div><b>From: </b>Marin van Heel</div>
<div><b>Sent: </b>Tuesday, June 16, 2015 11:22 AM</div>
<div><b>To: </b>Tom Houweling; <a class="moz-txt-link-abbreviated" href="mailto:CCPEM@JISCMAIL.AC.UK">CCPEM@JISCMAIL.AC.UK</a></div>
<div><b>Cc: </b>3DEM</div>
<div><b>Subject: </b>Re: [3dem] [ccpem] MRC file format
(Compressing cryo-EM data to 8-bits/pix and beyond)</div>
</div>
</td>
</tr>
</tbody>
</table>
<br>
<div id="_originalContent" style="background-color: rgb(255, 255,
255);">
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
<div class="moz-cite-prefix">Dear All,<br>
<br>
For various reasons I don’t think this line of reasoning is
very productive. The data compression to 8 or even 4 bits as
has been suggested in this discussion can only lead to loss of
data (see below). It may also represent poor management of the
available EM resources.<br>
<br>
Point by point:<br>
<br>
A) Advanced cryo-EM equipment costs of the order of ~5000 AUs
(Arbitrary Units: $/Eu/£) per day to own and operate, and will
generate up to ~ 2Tbyte of cryo-EM data per 24h. The costs of
storing this precious data for “eternity” will not exceed 100
AUs per day, that is, one or two percent of the tax-payers
total investment in your data collection. NOT storing that raw
data may NOT be a good idea for economic reasons alone (just
in case you, for example, need to repeat the experiment to get
the data back).<br>
<br>
B) Compressing all the raw data to save space can make sense
as long as the compression is loss-less (<a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://en.wikipedia.org/wiki/Lossless_compression">https://en.wikipedia.org/wiki/Lossless_compression</a>).
The compression (after movie alignment) as suggested, however,
may lead to a significant information loss. <br>
<br>
C) The dynamic range of a raw image is mainly determined by
the low-frequency components of the data. Scaling the min-max
densities from 0-255 for compression/truncation to 8 bit data,
changes the data representation from image to image. The
high-resolution information we are interested is has a
contrast of probably less than 0.1% of the strong
low-frequency components. The signal we are interested in is
thus already much smaller than the discretisation error of
1:256 of the A-to-D conversion. That does not mean one will
not be able to fish that information from the discretisation
and Poisson noise in the raw data… But it will certainly
suffer. The grey scales will change from image to image
purely dependent on whether there is, for example, an ice
crystal somewhere in the field of view. High-pass filtering
will remove the large-scale details thus also increase the
dynamic range available for the high-res frequency data
components. <br>
<br>
D) Note that the fact that you manage to get a 3D structure
out is no proof that you have not lost information. It is
merely proof for the fact that there was enough left over to
create a reasonable 3D that satisfies you. <br>
<br>
E) There are also other reasons for never deleting the
original data such as validation! You may be challenged – as
has happened in the recent past (PNAS 2013) - to show the
original data set to prove it is what you claim it is and was
collected on the instrumentation you claim it was taken on.
(In the PNAS cases the original data has still not been
released).<br>
<br>
F) What one can or wants to do with the raw data changes over
time. Many new movie alignment algorithms have been proposed
recently; access to exactly the same raw data is essential for
validation of the new algorithms. (You may even get more out
of your data!)<br>
<br>
G) The raw data characterizes the camera (and validates the
data set as per E) and allow you to correct for its flaws (<a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://www.nature.com/srep/2015/150611/srep10317/full/srep10317.html">http://www.nature.com/srep/2015/150611/srep10317/full/srep10317.html</a>).
You may also want to see whether the camera itself
deteriorated over time.<br>
<br>
H) Especially when the raw data are of some integer type, (and
you are using data with a limited dynamic range), the data on
disk will be written in a highly redundant fashion. You may
then use loss-less compression algorithms to reduce the size
of your data without suffering any information loss. You may
always compress the data, you may never compromise on its
information content!<br>
<br>
Cheers, Marin<br>
<br>
========================================<br>
<br>
On 04/06/2015 00:15, Tom Houweling wrote:<br>
</div>
<blockquote
cite="mid:1BFCB93C-8D3B-4B12-BEAE-4816AB6B7CDC@berkeley.edu"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<div class="">What I meant is that Relion appears to have no
problem reading 16 bit and 8 bit formats, therefore
converting to 32bit floating point images should not be
necessary.</div>
<div class=""><br class="">
</div>
<div class="">However, the verdict on loss of resolution
reducing the data to 8 bits is still out. I’m motivated by
conserving disk space. </div>
<div class=""><br class="">
</div>
<div class="">I’m currently reprocessing a good dataset that
yielded a high resolution structure. But this time I
converted the aligned stacks of 32bit per pixel to just 8 by
the following method:</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span"
style="white-space:pre"> </span>1)<span
class="Apple-tab-span" style="white-space:pre"> </span>Calculate
the mean and std. deviation</div>
<div class=""><span class="Apple-tab-span"
style="white-space:pre"> </span>2)<span
class="Apple-tab-span" style="white-space:pre"> </span>Cutoff
at +/- 3 std dev</div>
<div class=""><span class="Apple-tab-span"
style="white-space:pre"> </span>3)<span
class="Apple-tab-span" style="white-space:pre"> </span>Set
lowest value to 0 and highest to 255 </div>
<div class=""><br class="">
</div>
<div class="">Tom</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Jun 3, 2015, at 10:58 AM, Amedee des
Georges <<a moz-do-not-send="true"
href="mailto:adesgeorges@GMAIL.COM" class="">adesgeorges@GMAIL.COM</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode:
space; -webkit-line-break: after-white-space;"
class="">Dear Tom,
<div class=""><br class="">
</div>
<div class="">Did you see any decrease in resolution
with 8bit vs 16? How did it look? </div>
<div class="">It’s obviously an advantage to use
8bits for storage if it doesn’t decrease image
quality significantly. </div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class=""><br class="">
</div>
<div class="">Amedee</div>
<div class=""><br class="">
<div class="">
<div class="">On Jun 3, 2015, at 1:44 PM, Tom
Houweling <<a moz-do-not-send="true"
href="mailto:tom.houweling@BERKELEY.EDU"
class="">tom.houweling@BERKELEY.EDU</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite" class="">
<meta http-equiv="Content-Type"
content="text/html; charset=utf-8" class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;"
class="">
<div class="">We have successfully processed
MRC images and stacks in Relion that were
in 16 bit mode 6 and also in the non MRC
sanctioned mode 5 (8 bit unsigned).</div>
<div class=""><br class="">
</div>
<div class="">—Tom</div>
<div class=""><br class="">
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Jun 3, 2015, at 10:22
AM, Rémi Fronzes <<a
moz-do-not-send="true"
href="mailto:remi.fronzes@PASTEUR.FR"
class="">remi.fronzes@PASTEUR.FR</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type"
content="text/html; charset=utf-8"
class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break:
after-white-space;" class="">Dear
All,
<div class=""><br class="">
</div>
<div class="">Maybe a silly question
but still worth asking. </div>
<div class="">Is it a problem to
extract and use in relion
particles from 16bits MRC images
(i.e. collected using EPU) ?</div>
<div class="">Or do we have to
convert the micrographs in 32 bits
MRC format.</div>
<div class=""><br class="">
</div>
<div class="">Cheers</div>
<div class=""><br class="">
</div>
<div class="">Rémi<br class="">
<div class="">
<div style="font-family:
Helvetica; font-size: inherit;
font-style: normal;
font-variant: normal;
font-weight: normal;
letter-spacing: normal;
line-height: normal; orphans:
2; text-align: -webkit-auto;
text-indent: 0px;
text-transform: none;
white-space: normal; widows:
2; word-spacing: 0px;
-webkit-text-stroke-width:
0px;" class="">
<div style="margin: 0px;
font-size: 18px;
font-family: RotisSansSerif;
color: rgb(0, 78, 135); "
class=""><span
style="letter-spacing:
0px; " class=""><br
class="Apple-interchange-newline">
<br class="">
</span></div>
<div style="margin: 0px;
font-size: 18px;
font-family: RotisSansSerif;
color: rgb(0, 78, 135); "
class=""><span
style="letter-spacing:
0px; " class="">Rémi
Fronzes</span></div>
<div style="margin: 0px;
font-size: 12px;
font-family: RotisSansSerif;
color: rgb(0, 78, 135); "
class=""><span
style="letter-spacing:
0px; " class="">G5
biologie structurale de la
sécrétion bactérienne,
institut Pasteur</span></div>
<div style="margin: 0px;
font-size: 12px;
font-family: RotisSansSerif;
color: rgb(0, 78, 135); "
class=""><span
style="letter-spacing:
0px; " class="">CNRS UMR
3528, institut Pasteur</span></div>
<div style="margin: 0px;
font-size: 12px;
font-family: RotisSansSerif;
color: rgb(0, 78, 135);
min-height: 13px; " class=""><span
style="letter-spacing:
0px; " class=""></span><br
class="">
</div>
<div style="margin: 0px;
font-size: 12px;
font-family: RotisSansSerif;
color: rgb(0, 78, 135); "
class=""><span
style="letter-spacing:
0px; " class="">Office: +33
(0)145688864</span></div>
<div style="margin: 0px;
font-size: 12px;
font-family: RotisSansSerif;
color: rgb(0, 78, 135); "
class=""><span
style="letter-spacing:
0px; " class="">Lab: +33
(0) 145688863</span></div>
<div style="margin: 0px;
font-size: 12px;
font-family: RotisSansSerif;
color: rgb(0, 78, 135); "
class=""><span
style="letter-spacing:
0px; " class="">Mobile:
+33 (0) 688263992</span></div>
<div style="margin: 0px;
font-size: 12px;
font-family: RotisSansSerif;
color: rgb(2, 30, 170); "
class=""><span
style="letter-spacing:
0px; color: rgb(0, 78,
135); " class="">Email:<span
class="Apple-converted-space"> </span><span style="letter-spacing: 0px;
color: rgb(2, 30, 170);
" class=""><a
moz-do-not-send="true"
href="mailto:remi.fronzes@pasteur.fr" class="">remi.fronzes@pasteur.fr</a></span></span></div>
</div>
<div style="font-size: 12px;
font-style: normal;
font-variant: normal;
font-weight: normal;
letter-spacing: normal;
line-height: normal; orphans:
2; text-align: -webkit-auto;
text-indent: 0px;
text-transform: none;
white-space: normal; widows:
2; word-spacing: 0px;
-webkit-text-size-adjust:
auto;
-webkit-text-stroke-width:
0px; margin: 0px; font-family:
RotisSansSerif; color: rgb(2,
30, 170); " class=""><span
style="letter-spacing: 0px;
color: rgb(0, 78, 135); "
class=""><br class="">
</span></div>
<div style="font-size: 12px;
font-style: normal;
font-variant: normal;
font-weight: normal;
letter-spacing: normal;
line-height: normal; orphans:
2; text-align: -webkit-auto;
text-indent: 0px;
text-transform: none;
white-space: normal; widows:
2; word-spacing: 0px;
-webkit-text-size-adjust:
auto;
-webkit-text-stroke-width:
0px; margin: 0px; font-family:
RotisSansSerif; color: rgb(2,
30, 170); " class="">
<div style="margin: 0px;
color: rgb(0, 86, 148); "
class=""><span
style="letter-spacing:
0px; " class="">25 rue du
Docteur Roux </span></div>
<div style="margin: 0px;
color: rgb(0, 86, 148); "
class=""><span
style="letter-spacing:
0px; " class="">Bâtiment
Metchnikoff, 3ème étage</span></div>
<div style="margin: 0px;
color: rgb(0, 86, 148); "
class=""><span
style="letter-spacing:
0px; " class="">75015
Paris, France </span></div>
</div>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div class="">--<br class="">
Tom Houweling - QB3 Nogales
Lab Computer Analyst @ Howard
Hughes Medical Institute<br class="">
University of California Berkeley,
708D Stanley Hall, Berkeley, CA 94720<br
class="">
<br class="">
</div>
</div>
<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div class="">--<br class="">
Tom Houweling - QB3 Nogales Lab Computer Analyst @
Howard Hughes Medical Institute<br class="">
University of California Berkeley, 708D Stanley Hall,
Berkeley, CA 94720<br class="">
<br class="">
</div>
</div>
<br class="">
</div>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
================================================================
Prof Dr Ir Marin van Heel
Professor of Cryo-EM Data Processing
Leiden University
</pre>
<br>
<!--end of _originalContent --></div>
<br>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
================================================================
Prof Dr Ir Marin van Heel
Professor of Cryo-EM Data Processing
Leiden University
NeCEN Building Room 05.27
Einsteinweg 55
2333 CC Leiden
The Netherlands
Tel. NL: +31(0)715271424 // Mobile NL: +31(0)652736618
Skype: Marin.van.Heel
email: marin.vanheel(A_T)gmail.com
and: mvh.office(A_T)gmail.com
----------------------------------------------
Emeritus Professor of Structural Biology
Imperial College London
Faculty of Natural Sciences
Biochemistry Building (Room 512)
South Kensington Campus
London SW7 2AZ, UK
email: m.vanheel(A_T)ic.ac.uk
Tel. UK: +44(0)2075945316 //Mobile: +44(0)7941540625
----------------------------------------------
Visiting Professor at:
Laboratório Nacional de Nanotecnologia - LNNano
CNPEM/ABTLuS, Campinas, Brazil
Brazilian mobile phone +55-19-983189143
------------------------------------------------------------------
I receive many emails per day and, although I try,
there is no guarantee that I will actually read each incoming email.
Moreover, our Spam filters can be strikt and sometimes make
legitimate emails disappear (try the gmail accounts, alternatively)</pre>
</body>
</html>