<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:337654244;
mso-list-template-ids:1793723820;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="en-SE" link="blue" vlink="purple" style="word-wrap:break-word;overflow-wrap: break-word;-webkit-nbsp-mode: space;line-break:after-white-space">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thank you for the reply Steven. Obviously greater minds than mine has thought about this before. EMAN2 has been a champion in file-formats and the HDF5 format
is as you say a very flexible general-purpose data container format. I quickly skimmed your paper. From my understanding, one could reduce the bit depth from 32 bit to 12 bit while still being able to maintain the information contents of the file for single-particle
data. That is a significant save. Even going to 16 bit, as Guillaume suggested on the CCPEM list, would cut the file size in half. Adding lossless compression algorithms could save you even more, albeit it is difficult to efficiently compress noisy solvent
regions. Offlist, it was suggested that the bzip algorithm is able to compress further. /Daniel<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-family:"Calibri",sans-serif;color:black">From:
</span></b><span style="font-family:"Calibri",sans-serif;color:black">"Ludtke, Steven J." <sludtke@bcm.edu><br>
<b>Date: </b>Tuesday, 28 April 2026 at 15:07<br>
<b>To: </b>Daniel Larsson <daniel.larsson@icm.uu.se><br>
<b>Cc: </b>"3dem@ncmir.ucsd.edu" <3dem@ncmir.ucsd.edu>, Collaborative Computational Project in Electron cryo-Microscopy <CCPEM@JISCMAIL.AC.UK><br>
<b>Subject: </b>Re: [3dem] volume compression standard<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">Just for the sake of argument, I'll throw this out there: <o:p>
</o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://urldefense.com/v3/__https://pmc.ncbi.nlm.nih.gov/articles/PMC9645247/__;!!Mih3wA!AoH5fbQG1jQGXjFWdi2eqrvNtM_lkBaL2s6RSD_gsQr_zL8wcnSG4fC8m77_2DJ3aEQ5XeSIZArxU43TgwCxPYBrsHDLTQ$">https://pmc.ncbi.nlm.nih.gov/articles/PMC9645247/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">gzipping only saves significant space if one discritizes the values stored in a float format or uses an int representation. <o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Apr 28, 2026, at 7:38<span style="font-family:"Arial",sans-serif"> </span>AM, Daniel Larsson via 3dem <3dem@ncmir.ucsd.edu> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:Helvetica">***CAUTION:*** This email is not from a BCM Source. Only click links or open attachments you know are safe.</span></b>
<o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Would it not be very beneficiary of the community could agree on a loss-less compression standard for EM volumes (maps and masks)?</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">High-resolution maps take up significant disk space, often around 500-800 MB for large complexes. Considering that there are many volumes associated with each
reconstruction (half-maps, full map, sharpened map, masks, etc) and multiply this by all the reconstructions for a typical project, and it adds up significantly. In addition, both RELION and CryoSPARC save volumes for intermediate iterations. These large files
take a long time to download/upload, e.g. transferring between computers or when depositing structures. Opening maps for visualization in ChimeraX and Coot is also slowed down by having to read large files from disk.</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">So, we need a compression standard. Zip compression is fast and can reduce the file size considerably without changing the information in the file. Mask files
are reduced to just a few percent of the original size (example 537 MB to 9 MB). What we need is:<br>
<br>
<br>
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<ol style="margin-top:0cm" start="1" type="1">
<li class="MsoListParagraph" style="margin-top:0cm;margin-bottom:0cm;mso-list:l0 level1 lfo1">
<span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Software generating volumes should by default write these in a lossless compressed format</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></li><li class="MsoListParagraph" style="margin-top:0cm;margin-bottom:0cm;mso-list:l0 level1 lfo1">
<span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Software visualizing volumes should accept and on-the-fly uncompress maps in memory</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></li><li class="MsoListParagraph" style="margin-top:0cm;margin-bottom:0cm;mso-list:l0 level1 lfo1">
<span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Refinement packages should accept and on-the-fly uncompress maps in memory</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></li><li class="MsoListParagraph" style="margin-top:0cm;margin-bottom:0cm;mso-list:l0 level1 lfo1">
<span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Repositories should accept compressed maps and un/recompress according to their own needs</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></li></ol>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Actors that immediately comes to my mind are CryoSPARC, RELION, Warp, CryoDRNG, IMOD, ChimeraX, PyMOL, Coot, Blender Molecular nodes, Phenix, Servalcat, wwPDB/EMDB.
The exact format is up for debate, but why not something simple such as gzipped mrc files?</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Best regards,</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Daniel (written while impatiently waiting for files being uploaded to the wwPDB)<br>
<br>
<br>
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:Helvetica"><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här:<span class="apple-converted-space"> </span></span><a href="https://urldefense.com/v3/__http:/www.uu.se/om-uu/dataskydd-personuppgifter/__;!!Mih3wA!BUNKAv9U59Pq6jXGpg_HasLYL0bl-XFcbmYiAET8ZiKt_ru2lzuFBJJ-SqqvDQ3tw1khyJypThew9ThNQozonIt7z2lJrA$"><span style="font-size:13.5pt;font-family:Helvetica">http://www.uu.se/om-uu/dataskydd-personuppgifter/</span></a><span style="font-size:13.5pt;font-family:Helvetica"><br>
<br>
E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here:<span class="apple-converted-space"> </span></span><a href="https://urldefense.com/v3/__http:/www.uu.se/en/about-uu/data-protection-policy__;!!Mih3wA!BUNKAv9U59Pq6jXGpg_HasLYL0bl-XFcbmYiAET8ZiKt_ru2lzuFBJJ-SqqvDQ3tw1khyJypThew9ThNQozonIusG45pUg$"><span style="font-size:13.5pt;font-family:Helvetica">http://www.uu.se/en/about-uu/data-protection-policy</span></a><span class="apple-converted-space"><span style="font-size:13.5pt;font-family:Helvetica"> </span></span><span style="font-size:13.5pt;font-family:Helvetica">_______________________________________________<br>
3dem mailing list<br>
</span><a href="mailto:3dem@ncmir.ucsd.edu"><span style="font-size:13.5pt;font-family:Helvetica">3dem@ncmir.ucsd.edu</span></a><span style="font-size:13.5pt;font-family:Helvetica"><br>
</span><a href="https://urldefense.com/v3/__https:/mail.ncmir.ucsd.edu/mailman/listinfo/3dem__;!!GKvG1w!c4KeXym_7XyZmm8YaD37WyGM_5btxOKmUoJ71fQnhHEHYyN7mE3s75UPV0cWhg8nk-lDiwJcXfWQbcBjv_kd$"><span style="font-size:13.5pt;font-family:Helvetica">https://urldefense.com/v3/__https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem__;!!GKvG1w!c4KeXym_7XyZmm8YaD37WyGM_5btxOKmUoJ71fQnhHEHYyN7mE3s75UPV0cWhg8nk-lDiwJcXfWQbcBjv_kd$</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p><o:p> </o:p></p>
<div style="border:solid black 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="line-height:12.0pt;background:white"><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:black">VARNING: Klicka inte på länkar och öppna inte bilagor om du inte känner igen avsändaren och vet att innehållet är säkert.<br>
CAUTION: Do not click on links or open attachments unless you recognise the sender and know the content is safe.<o:p></o:p></span></p>
</div>
<p><o:p> </o:p></p>
</div>
</body>
</html>