[3dem] EER home-made?

[Stahlberg Henning]

Thanks, Takanori. Very helpful.
Best, Henning.


On 28.10.22, 10:42, "Takanori Nakane" <tnakane.protein at osaka-u.ac.jp> wrote:

    Hi,

     > This is for diffractive imaging using a hybrid pixel detector camera, 
    running at 8000 fps or much faster (Dectris' ARINA will go up to 120000 
    fps). The frames are small and usually contain only few electrons.
     > Saving 512x512px @ 8bit with 8000 fps to disk will result in 4.2 
    Gigabytes per second, or 15 Terabytes per hour, of data that mostly 
    contain zeros. EER is preferred.

    This is not correct. Such sparse images compress better with
    bitshuffle+lz4 (Dectris EIGER uses this), bz2 etc. EER is a very simple
    run-length encoding with a limited maximum repeat length of 7 bits
    (= 127). If there are 1024 consecutive empty pixels, it has to store
    [127] [127] [127] ... instead of just 10 bits. It also uses 4 bits per
    pixel to encode 4x super-resolution information, which is not relevant
    for other detectors. EER is designed for easy implementation in FPGA,
    not for an optimal compression rate.

    Also note that EER images have a bit depth of 1, i.e. binary.
    Each pixel can have either zero or one electron, not more.
    So you cannot save "8 bit" images.

    Another issue with EER is that EER uses TIFF as a container but
    libtiff does not support more than 65535 IFDs (i.e. frames) per file.
    This is not a limitation in the TIFF specification, but that of libtiff.
    (https://urldefense.com/v3/__https://www.asmail.be/msg0055011809.html__;!!Mih3wA!FqBf-TfrYuAA1lEFiHZ-uN_pIspeDXZrqM-kxmQPstAH8KIx7Kud8bitWscgUmG1I1g1PkN6u38uDj8ggUdXW8pqwqqveoY$  )
    So one has to implement a dedicated TIFF routines in addition to the EER
    encoder/decoder itself. Even in this case, indexing a TIFF movie with
    so many IFDs will be slow because there is no central index of IFDs in
    TIFF. Use other containers, such as HDF5.

    Do you really need to keep raw frames at 8000 fps?
    EER is actually an IMAGE format, not a real event format.
    For a real event format, see for example that of MediPix  detectors.

    Best regards,

    Takanori Nakane

    On 2022/10/28 17:11, Stahlberg Henning wrote:
    > Dear Takanori,
    > This is for diffractive imaging using a hybrid pixel detector camera, running at 8000 fps or much faster (Dectris' ARINA will go up to 120000 fps). The frames are small and usually contain only few electrons.
    > Saving 512x512px @ 8bit with 8000 fps to disk will result in 4.2 Gigabytes per second, or 15 Terabytes per hour, of data that mostly contain zeros. EER is preferred.
    > Best, Henning.
    > 
    > Henning Stahlberg
    > Laboratory of Biological Electron Microscopy
    > Institute of Physics, School of Basic Sciences, EPFL, and
    > Dep. of Fund. Microbiology, Faculty of Biology and Medicine, UNIL,
    > Cubotron, BSP421, 1015 Lausanne, Switzerland
    > lbem.epfl.ch <https://urldefense.com/v3/__http://lbem.epfl.ch__;!!Mih3wA!FqBf-TfrYuAA1lEFiHZ-uN_pIspeDXZrqM-kxmQPstAH8KIx7Kud8bitWscgUmG1I1g1PkN6u38uDj8ggUdXW8pq2exi6Gs$  > ,   +41 21 693 45 07
    > 
    > On 27.10.22, 15:38, "3dem on behalf of Takanori Nakane" <3dem-bounces at ncmir.ucsd.edu on behalf of tnakane.protein at osaka-u.ac.jp> wrote:
    > 
    >      Hi,
    > 
    >      Why do you want to do this?
    > 
    >      EER is not an efficient format in terms of the compression ratio.
    > 
    >      Also note that RELION's EER parser is hard coded for 4096 pixels and does not accept
    >      other sizes.
    > 
    >      Best regards,
    > 
    >      Takanori Nakane
    > 
    >      On 2022/10/27 22:25, Stahlberg Henning wrote:
    >      > Hi,
    >      >
    >      > Does anybody know if there is a public domain software available that can translate an EM image into a list of electron event recordings (EER file),
    >      > assuming the image is mostly black and has only a hand-full of electron impacts on it?
    >      >
    >      > Obviously, that should at least run on a GPU, if not more specialized hardware. Dark-field, flat-field and PSF should be refined on the fly.
    >      >
    >      > Henning.
    >      >
    >      > *Henning Stahlberg*
    >      > Laboratory of Biological Electron Microscopy
    >      > Institute of Physics, School of Basic Sciences, EPFL, and
    >      > Dep. of Fund. Microbiology, Faculty of Biology and Medicine, UNIL,
    >      > Cubotron, BSP421, 1015 Lausanne, Switzerland
    >      > lbem.epfl.ch
    >      > <https://urldefense.com/v3/__http://lbem.epfl.ch__;!!Mih3wA!EgSsjtPxILdQGBLYCe6YvOsUEdERiq0QhPq_T9khPOdej8-aPP_32UVqQGh1YUX8Nm__I_Zq2QW4B9Tll2iUskEiAQTVUQc$>
    >      >
    >      >
    >      > _______________________________________________
    >      > 3dem mailing list
    >      > 3dem at ncmir.ucsd.edu
    >      > https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
    > 
    >      _______________________________________________
    >      3dem mailing list
    >      3dem at ncmir.ucsd.edu
    >      https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
    >