<html aria-label="message body">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Also, just to point out that float16 is not a great choice for final maps for several different reasons:
<div><br>
</div>
<div>- there are competing 16 bit float standards (bfloat vs float16). bfloat is used heavily for machine learning, but has only 7 bit precision, making it a very poor choice for CryoEM volumes. It's just asking for problems.</div>
<div><br>
</div>
<div>- CryoEM/ET volumes don't span a wide enough dynamic range to make float representation necessary, and it causes precision changes with magnitude</div>
<div><br>
</div>
<div>- float16 is _much_ less compressible than a bit truncated 12 bit number stored as a 16 bit integer</div>
<div><br>
</div>
<div>- float16/bfloat isn't ideal for work in Fourier space due to the high dynamic range of most CryoEM data</div>
<div><br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Apr 28, 2026, at 10:24 AM, Ludtke, Steven J. via 3dem <3dem@ncmir.ucsd.edu> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<font size="2"><b>***CAUTION:*** This email is not from a BCM Source. Only click links or open attachments you know are safe.</b></font>
<hr>
Yes, I'm not trying to make a big push for HDF5. It does work for this purpose, it is a usable cross-discipline standard, and CryoEM lacks a format which supports compression natively, but there are tradeoffs. I know CZII is trying really hard to push for ZARR,
but I can't imagine they will get much more uptake than HDF5 ever did, as in several respects it is even worse than HDF.
<div><br>
</div>
<div>However, it is quite possible to get similar results by using bit truncation to render volumes as 10 or 12 bit in MRC format then using your compression algorithm of choice. The key factor is that by bit reduction you get rid of the pure noise bits which
also makes the data dramatically more compressible. For masks, they compress down to virtually nothing, which is critical if you want to do something like a binary mask representation of tomogram segmentation.</div>
<div><br>
</div>
<div>We used GZIP (mainly) in that paper largely because it is so standard, and from a librarian/archivist perspective long term support is the most important aspect of an archive, but clearly there are better (faster and more effective) compression algorithms
out there.</div>
<div><br>
</div>
<div>The major takeaway is that you can both prove and demonstrate that 4-5 bits is sufficient to retain all information in noisy CryoEM images, and for final reconstructions 10-12 bits is demonstrably sufficient.<br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Apr 28, 2026, at 9:42 AM, Daniel Larsson <daniel.larsson@icm.uu.se> wrote:</div>
<br class="Apple-interchange-newline">
<div><font size="2" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-line: none; text-decoration-thickness: auto; text-decoration-style: solid;"><b>***CAUTION:***
This email is not from a BCM Source. Only click links or open attachments you know are safe.</b></font><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"></span>
<hr style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-line: none; text-decoration-thickness: auto; text-decoration-style: solid;">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-line: none; text-decoration-thickness: auto; text-decoration-style: solid;">
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Thank you for the reply Steven. Obviously greater minds than mine has thought about this before. EMAN2 has
been a champion in file-formats and the HDF5 format is as you say a very flexible general-purpose data container format. I quickly skimmed your paper. From my understanding, one could reduce the bit depth from 32 bit to 12 bit while still being able to maintain
the information contents of the file for single-particle data. That is a significant save. Even going to 16 bit, as Guillaume suggested on the CCPEM list, would cut the file size in half. Adding lossless compression algorithms could save you even more, albeit
it is difficult to efficiently compress noisy solvent regions. Offlist, it was suggested that the bzip algorithm is able to compress further. /Daniel<o:p></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></span></div>
<div style="border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; border-image: none; padding: 3pt 0cm 0cm;">
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-family: Calibri, sans-serif;">From:<span class="Apple-converted-space"> </span></span></b><span style="font-family: Calibri, sans-serif;">"Ludtke, Steven J." <<a href="mailto:sludtke@bcm.edu" style="color: blue; text-decoration: underline;">sludtke@bcm.edu</a>><br>
<b>Date:<span class="Apple-converted-space"> </span></b>Tuesday, 28 April 2026 at 15:07<br>
<b>To:<span class="Apple-converted-space"> </span></b>Daniel Larsson <<a href="mailto:daniel.larsson@icm.uu.se" style="color: blue; text-decoration: underline;">daniel.larsson@icm.uu.se</a>><br>
<b>Cc:<span class="Apple-converted-space"> </span></b>"<a href="mailto:3dem@ncmir.ucsd.edu" style="color: blue; text-decoration: underline;">3dem@ncmir.ucsd.edu</a>" <<a href="mailto:3dem@ncmir.ucsd.edu" style="color: blue; text-decoration: underline;">3dem@ncmir.ucsd.edu</a>>,
Collaborative Computational Project in Electron cryo-Microscopy <<a href="mailto:CCPEM@JISCMAIL.AC.UK" style="color: blue; text-decoration: underline;">CCPEM@JISCMAIL.AC.UK</a>><br>
<b>Subject:<span class="Apple-converted-space"> </span></b>Re: [3dem] volume compression standard<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div>
</div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Just for the sake of argument, I'll throw this out there:<span class="Apple-converted-space"> </span><o:p></o:p></div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><a href="https://urldefense.com/v3/__https://pmc.ncbi.nlm.nih.gov/articles/PMC9645247/__;!!GKvG1w!eBokV4Uz0MQ0H7rP-_4SpLdSKD6JCgyd6qeHtJxFgdMb3gL1XqQtvVWXjcd1A9A44Ls6snSC8JJWjvWxCmSQaW8l3w$" originalsrc="https://urldefense.com/v3/__https://pmc.ncbi.nlm.nih.gov/articles/PMC9645247/__;!!GKvG1w!eBokV4Uz0MQ0H7rP-_4SpLdSKD6JCgyd6qeHtJxFgdMb3gL1XqQtvVWXjcd1A9A44Ls6snSC8JJWjvWxCmSQaW8l3w$" style="color: blue; text-decoration: underline;">https://pmc.ncbi.nlm.nih.gov/articles/PMC9645247/</a><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">gzipping only saves significant space if one discritizes the values stored in a float format or uses an int representation. <o:p></o:p></div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br>
<br>
<o:p></o:p></div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On Apr 28, 2026, at 7:38<span style="font-family: Arial, sans-serif;"> </span>AM, Daniel Larsson via 3dem <3dem@ncmir.ucsd.edu> wrote:<o:p></o:p></div>
</div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 10pt; font-family: Helvetica;">***CAUTION:*** This email is not from a BCM Source. Only click links or open attachments you know are safe.</span></b><o:p></o:p></div>
<div class="MsoNormal" align="center" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; text-align: center;">
<hr size="0" width="100%" align="center">
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Would it not be very beneficiary of the community could agree on a loss-less compression standard for EM
volumes (maps and masks)?</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;"> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">High-resolution maps take up significant disk space, often around 500-800 MB for large complexes. Considering
that there are many volumes associated with each reconstruction (half-maps, full map, sharpened map, masks, etc) and multiply this by all the reconstructions for a typical project, and it adds up significantly. In addition, both RELION and CryoSPARC save volumes
for intermediate iterations. These large files take a long time to download/upload, e.g. transferring between computers or when depositing structures. Opening maps for visualization in ChimeraX and Coot is also slowed down by having to read large files from
disk.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;"> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">So, we need a compression standard. Zip compression is fast and can reduce the file size considerably without
changing the information in the file. Mask files are reduced to just a few percent of the original size (example 537 MB to 9 MB). What we need is:<br>
<br>
<br>
</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div>
</div>
<ol start="1" type="1" style="margin-bottom: 0cm; margin-top: 0cm;">
<li class="MsoListParagraph" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">
<span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Software generating volumes should by default write these in a lossless compressed format</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></li><li class="MsoListParagraph" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">
<span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Software visualizing volumes should accept and on-the-fly uncompress maps in memory</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></li><li class="MsoListParagraph" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">
<span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Refinement packages should accept and on-the-fly uncompress maps in memory</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></li><li class="MsoListParagraph" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">
<span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Repositories should accept compressed maps and un/recompress according to their own needs</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></li></ol>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;"> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Actors that immediately comes to my mind are CryoSPARC, RELION, Warp, CryoDRNG, IMOD, ChimeraX, PyMOL, Coot,
Blender Molecular nodes, Phenix, Servalcat, wwPDB/EMDB. The exact format is up for debate, but why not something simple such as gzipped mrc files?</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;"> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Best regards,</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Daniel (written while impatiently waiting for files being uploaded to the wwPDB)<br>
<br>
<br>
</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div>
</div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13.5pt; font-family: Helvetica;"><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här:<span class="apple-converted-space"> </span></span><a href="https://urldefense.com/v3/__http:/www.uu.se/om-uu/dataskydd-personuppgifter/__;!!Mih3wA!BUNKAv9U59Pq6jXGpg_HasLYL0bl-XFcbmYiAET8ZiKt_ru2lzuFBJJ-SqqvDQ3tw1khyJypThew9ThNQozonIt7z2lJrA$" originalsrc="https://urldefense.com/v3/__http:/www.uu.se/om-uu/dataskydd-personuppgifter/__;!!Mih3wA!BUNKAv9U59Pq6jXGpg_HasLYL0bl-XFcbmYiAET8ZiKt_ru2lzuFBJJ-SqqvDQ3tw1khyJypThew9ThNQozonIt7z2lJrA$" style="color: blue; text-decoration: underline;"><span style="font-size: 13.5pt; font-family: Helvetica;">http://www.uu.se/om-uu/dataskydd-personuppgifter/</span></a><span style="font-size: 13.5pt; font-family: Helvetica;"><br>
<br>
E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here:<span class="apple-converted-space"> </span></span><a href="https://urldefense.com/v3/__http:/www.uu.se/en/about-uu/data-protection-policy__;!!Mih3wA!BUNKAv9U59Pq6jXGpg_HasLYL0bl-XFcbmYiAET8ZiKt_ru2lzuFBJJ-SqqvDQ3tw1khyJypThew9ThNQozonIusG45pUg$" originalsrc="https://urldefense.com/v3/__http:/www.uu.se/en/about-uu/data-protection-policy__;!!Mih3wA!BUNKAv9U59Pq6jXGpg_HasLYL0bl-XFcbmYiAET8ZiKt_ru2lzuFBJJ-SqqvDQ3tw1khyJypThew9ThNQozonIusG45pUg$" style="color: blue; text-decoration: underline;"><span style="font-size: 13.5pt; font-family: Helvetica;">http://www.uu.se/en/about-uu/data-protection-policy</span></a><span class="apple-converted-space"><span style="font-size: 13.5pt; font-family: Helvetica;"> </span></span><span style="font-size: 13.5pt; font-family: Helvetica;">_______________________________________________<br>
3dem mailing list<br>
</span><a href="mailto:3dem@ncmir.ucsd.edu" style="color: blue; text-decoration: underline;"><span style="font-size: 13.5pt; font-family: Helvetica;">3dem@ncmir.ucsd.edu</span></a><span style="font-size: 13.5pt; font-family: Helvetica;"><br>
</span><a href="https://urldefense.com/v3/__https:/mail.ncmir.ucsd.edu/mailman/listinfo/3dem__;!!GKvG1w!c4KeXym_7XyZmm8YaD37WyGM_5btxOKmUoJ71fQnhHEHYyN7mE3s75UPV0cWhg8nk-lDiwJcXfWQbcBjv_kd$" originalsrc="https://urldefense.com/v3/__https:/mail.ncmir.ucsd.edu/mailman/listinfo/3dem__;!!GKvG1w!c4KeXym_7XyZmm8YaD37WyGM_5btxOKmUoJ71fQnhHEHYyN7mE3s75UPV0cWhg8nk-lDiwJcXfWQbcBjv_kd$" style="color: blue; text-decoration: underline;"><span style="font-size: 13.5pt; font-family: Helvetica;">https://urldefense.com/v3/__https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem__;!!GKvG1w!c4KeXym_7XyZmm8YaD37WyGM_5btxOKmUoJ71fQnhHEHYyN7mE3s75UPV0cWhg8nk-lDiwJcXfWQbcBjv_kd$</span></a><o:p></o:p></div>
</div>
</blockquote>
</div>
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div>
</div>
<p><o:p> </o:p></p>
<div style="border: 1pt solid black; padding: 2pt;">
<div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 12pt; background: white;">
<span style="font-size: 10pt; font-family: Calibri, sans-serif;">VARNING: Klicka inte på länkar och öppna inte bilagor om du inte känner igen avsändaren och vet att innehållet är säkert.<br>
CAUTION: Do not click on links or open attachments unless you recognise the sender and know the content is safe.</span></div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
_______________________________________________<br>
3dem mailing list<br>
3dem@ncmir.ucsd.edu<br>
https://urldefense.com/v3/__https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem__;!!GKvG1w!YLWwAeH4JivjPEY8PUCIYQ9Lq7vcasp4J2OO48i5IugUf9zx_Gp4gtW3zfrHR2x7C5nfv_6dGxlk0eWibyDz$
<br>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>