[3dem] Data storage and compression

[Matthew H. Cahn]

Thanks, Henning.  Do any of the commonly-used processing programs (e.g. RELION, CryoSPARC) read or write mrcz format?

Matthew

||  Matthew Cahn  |  Linux Administrator  |  Dept. of Molecular Biology / Research Computing  |  Princeton University  |  (609) 258-5404  |  mcahn at princeton.edu<mailto:mcahn at princeton.edu>  ||


On Aug 29, 2019, at 5:04 PM, Henning Stahlberg <henning.stahlberg at unibas.ch<mailto:henning.stahlberg at unibas.ch>> wrote:

Hi,
Gain corrected images are floating point. Loss-less compression of noisy floats is not trivial. The MRCZ format of Rob McLeod deals with that, see
 https://doi.org/10.1016/j.jsb.2017.11.012
Henning.

Henning Stahlberg
http://c-cina.org<http://c-cina.org/>
(sent from a mobile device)

On Aug 29, 2019, at 22:51, Matthew H. Cahn <mcahn at princeton.edu<mailto:mcahn at princeton.edu>> wrote:

By the way, why don’t gain-corrected images compress well?  Doesn’t gain-correction just change the dynamic range of the pixels?

Matthew

||  Matthew Cahn  |  Linux Administrator  |  Dept. of Molecular Biology / Research Computing  |  Princeton University  |  (609) 258-5404  |  mcahn at princeton.edu<mailto:mcahn at princeton.edu>  ||


On Aug 29, 2019, at 3:35 PM, Ali Punjani <alipunjani at cs.toronto.edu<mailto:alipunjani at cs.toronto.edu>> wrote:

Hi Matthew,

In cryoSPARC, LZW compressed tiff files are handled natively and decompressed in parallel on the fly as needed. They are never stored in decompressed form, so the original tiff files (created by IMOD, serialEM, etc) are retained throughout a project. This is a very common workflow, though as noted, for effective tiff compression the movies should not be gain corrected - the gain reference must be stored separately and provided at import time.

Ali

On Thu, Aug 29, 2019 at 3:19 PM Matthew H. Cahn <mcahn at princeton.edu<mailto:mcahn at princeton.edu>> wrote:
Is it possible to convert mrc files to compressed tiff immediately after data collection, and use that compressed tiff for all the downstream processing?  For example, if I compress the mrc files from our Krios to tiff with LZW (lossless) compression, using IMOD’s mrc2tif, can motioncor2, RELION, CryoSPARC, etc. handle that format and keep everything compressed throughout the process?

— Matthew

||  Matthew Cahn  |  Linux Administrator  |  Dept. of Molecular Biology / Research Computing  |  Princeton University  |  (609) 258-5404  |  mcahn at princeton.edu<mailto:mcahn at princeton.edu>  ||


_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem

_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20190829/1188f46e/attachment-0001.html>