[3dem] Data storage and compression

[Craig Yoshioka]

Interested in all of your takes on 2 things:

(1) I think libtiff has supported building with Zstandard support for a bit (though it is not a common compilation option).  My guess is TIFF+ZStandard will do quite a bit better in the general-case compression dept. than LZW.  Does anyone expect that software packages would automatically work with such images if I built libtiff with Zstandard and linked against it?  I suppose I could just try it, but have not had the chance yet…

(2) For cameras like the F3 that output movies with gains pre-applied- it would seem some amount of lossy compression is required to achieve reasonable compression savings- has anyone looked at using “error-bounded” lossy compression algorithms in this use case?  My very naive attempt at quantizing the pixel values (drop 5 LSB) was promising but needs a lot more characterization:

    original Falcon 3 MRC to TIFF-LZW : 1.9GB -> 1.3GB
quantized Falcon 3 MRC to TIFF-LZW : 1.9GB -> 295MB

both movies got ~ the same CTF fit limit after motion correction and fitting (3.3 vs 3.1), the RMSD between the two final aligned averages was 0.041.







On Aug 29, 2019, at 8:53 PM, Carlos Oscar S. Sorzano <coss at cnb.csic.es<mailto:coss at cnb.csic.es>> wrote:


Dear all,

as it seems to be the logical trend now, Scipion also keeps the movies in the original (compressed or not) format, and decompress them if necessary on the fly (the gain is also considered after this decompression). In this way, there is no need for extra space.

Kind regards, Carlos Oscar

El 29/08/2019 a las 21:35, Ali Punjani escribió:
Hi Matthew,

In cryoSPARC, LZW compressed tiff files are handled natively and decompressed in parallel on the fly as needed. They are never stored in decompressed form, so the original tiff files (created by IMOD, serialEM, etc) are retained throughout a project. This is a very common workflow, though as noted, for effective tiff compression the movies should not be gain corrected - the gain reference must be stored separately and provided at import time.

Ali

On Thu, Aug 29, 2019 at 3:19 PM Matthew H. Cahn <mcahn at princeton.edu<mailto:mcahn at princeton.edu>> wrote:
Is it possible to convert mrc files to compressed tiff immediately after data collection, and use that compressed tiff for all the downstream processing?  For example, if I compress the mrc files from our Krios to tiff with LZW (lossless) compression, using IMOD’s mrc2tif, can motioncor2, RELION, CryoSPARC, etc. handle that format and keep everything compressed throughout the process?

— Matthew

||  Matthew Cahn  |  Linux Administrator  |  Dept. of Molecular Biology / Research Computing  |  Princeton University  |  (609) 258-5404  |  mcahn at princeton.edu<mailto:mcahn at princeton.edu>  ||


_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem



_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem


--
------------------------------------------------------------------------
Carlos Oscar Sánchez Sorzano                  e-mail:   coss at cnb.csic.es<mailto:coss at cnb.csic.es>
Biocomputing unit                             http://i2pc.es/coss
National Center of Biotechnology (CSIC)
c/Darwin, 3
Campus Universidad Autónoma (Cantoblanco)     Tlf: 34-91-585 4510
28049 MADRID (SPAIN)                          Fax: 34-91-585 4506
------------------------------------------------------------------------


_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20190830/f8713ee3/attachment.html>