[3dem] EER home-made?
Tim Grüne
tim.gruene at univie.ac.at
Fri Oct 28 10:28:57 PDT 2022
Dear Henning,
you can check with the crystallographic community, e.g. ccp4bb. What you
name EER has been in use for neutron data for more than 1 1/2 decades
and may have been developed even much before. Alternatively, you can
convert to cbf. It comes with several (open source, lossless)
compression schemes like lz4 and packbits, tuned to low intensity data.
Decompression is on the fly. I have had the impression that compression
is not popular within EM community, which is puzzling considering the
speed of many available algorithms.
Best wishes from Walhalla,
Tim
Am 28.10.2022 11:21, schrieb Stahlberg Henning:
> Thanks, Takanori. Very helpful.
> Best, Henning.
>
>
> On 28.10.22, 10:42, "Takanori Nakane" <tnakane.protein at osaka-u.ac.jp>
> wrote:
>
> Hi,
>
> > This is for diffractive imaging using a hybrid pixel detector
> camera,
> running at 8000 fps or much faster (Dectris' ARINA will go up to
> 120000
> fps). The frames are small and usually contain only few electrons.
> > Saving 512x512px @ 8bit with 8000 fps to disk will result in 4.2
> Gigabytes per second, or 15 Terabytes per hour, of data that mostly
> contain zeros. EER is preferred.
>
> This is not correct. Such sparse images compress better with
> bitshuffle+lz4 (Dectris EIGER uses this), bz2 etc. EER is a very
> simple
> run-length encoding with a limited maximum repeat length of 7 bits
> (= 127). If there are 1024 consecutive empty pixels, it has to
> store
> [127] [127] [127] ... instead of just 10 bits. It also uses 4 bits
> per
> pixel to encode 4x super-resolution information, which is not
> relevant
> for other detectors. EER is designed for easy implementation in
> FPGA,
> not for an optimal compression rate.
>
> Also note that EER images have a bit depth of 1, i.e. binary.
> Each pixel can have either zero or one electron, not more.
> So you cannot save "8 bit" images.
>
> Another issue with EER is that EER uses TIFF as a container but
> libtiff does not support more than 65535 IFDs (i.e. frames) per
> file.
> This is not a limitation in the TIFF specification, but that of
> libtiff.
>
> (https://urldefense.com/v3/__https://www.asmail.be/msg0055011809.html__;!!Mih3wA!FqBf-TfrYuAA1lEFiHZ-uN_pIspeDXZrqM-kxmQPstAH8KIx7Kud8bitWscgUmG1I1g1PkN6u38uDj8ggUdXW8pqwqqveoY$
> )
> So one has to implement a dedicated TIFF routines in addition to
> the EER
> encoder/decoder itself. Even in this case, indexing a TIFF movie
> with
> so many IFDs will be slow because there is no central index of IFDs
> in
> TIFF. Use other containers, such as HDF5.
>
> Do you really need to keep raw frames at 8000 fps?
> EER is actually an IMAGE format, not a real event format.
> For a real event format, see for example that of MediPix
> detectors.
>
> Best regards,
>
> Takanori Nakane
>
> On 2022/10/28 17:11, Stahlberg Henning wrote:
> > Dear Takanori,
> > This is for diffractive imaging using a hybrid pixel detector
> camera, running at 8000 fps or much faster (Dectris' ARINA will go up
> to 120000 fps). The frames are small and usually contain only few
> electrons.
> > Saving 512x512px @ 8bit with 8000 fps to disk will result in 4.2
> Gigabytes per second, or 15 Terabytes per hour, of data that mostly
> contain zeros. EER is preferred.
> > Best, Henning.
> >
> > Henning Stahlberg
> > Laboratory of Biological Electron Microscopy
> > Institute of Physics, School of Basic Sciences, EPFL, and
> > Dep. of Fund. Microbiology, Faculty of Biology and Medicine,
> UNIL,
> > Cubotron, BSP421, 1015 Lausanne, Switzerland
> > lbem.epfl.ch
> <https://urldefense.com/v3/__http://lbem.epfl.ch__;!!Mih3wA!FqBf-TfrYuAA1lEFiHZ-uN_pIspeDXZrqM-kxmQPstAH8KIx7Kud8bitWscgUmG1I1g1PkN6u38uDj8ggUdXW8pq2exi6Gs$
> > , +41 21 693 45 07
> >
> > On 27.10.22, 15:38, "3dem on behalf of Takanori Nakane"
> <3dem-bounces at ncmir.ucsd.edu on behalf of
> tnakane.protein at osaka-u.ac.jp> wrote:
> >
> > Hi,
> >
> > Why do you want to do this?
> >
> > EER is not an efficient format in terms of the compression
> ratio.
> >
> > Also note that RELION's EER parser is hard coded for 4096
> pixels and does not accept
> > other sizes.
> >
> > Best regards,
> >
> > Takanori Nakane
> >
> > On 2022/10/27 22:25, Stahlberg Henning wrote:
> > > Hi,
> > >
> > > Does anybody know if there is a public domain software
> available that can translate an EM image into a list of electron event
> recordings (EER file),
> > > assuming the image is mostly black and has only a
> hand-full of electron impacts on it?
> > >
> > > Obviously, that should at least run on a GPU, if not more
> specialized hardware. Dark-field, flat-field and PSF should be refined
> on the fly.
> > >
> > > Henning.
> > >
> > > *Henning Stahlberg*
> > > Laboratory of Biological Electron Microscopy
> > > Institute of Physics, School of Basic Sciences, EPFL, and
> > > Dep. of Fund. Microbiology, Faculty of Biology and
> Medicine, UNIL,
> > > Cubotron, BSP421, 1015 Lausanne, Switzerland
> > > lbem.epfl.ch
> > >
> <https://urldefense.com/v3/__http://lbem.epfl.ch__;!!Mih3wA!EgSsjtPxILdQGBLYCe6YvOsUEdERiq0QhPq_T9khPOdej8-aPP_32UVqQGh1YUX8Nm__I_Zq2QW4B9Tll2iUskEiAQTVUQc$>
> > >
> > >
> > > _______________________________________________
> > > 3dem mailing list
> > > 3dem at ncmir.ucsd.edu
> > > https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
> >
> > _______________________________________________
> > 3dem mailing list
> > 3dem at ncmir.ucsd.edu
> > https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
> >
>
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
--
--
Tim Gruene
Head of the Centre for X-ray Structure Analysis
Faculty of Chemistry
University of Vienna
Phone: +43-1-4277-70202
GPG Key ID = A46BEE1A
More information about the 3dem
mailing list