[3dem] EER home-made?

[Schenk, Andreas]

Hi Henning,

I am not aware of any public domain library for conversion to EER. 
But if saving to disk is the main bottleneck zstd compression might be an option. It is a fast parallel compressor, which is also supported in the NVIDIA nvCOMP library.  
A hardware implementation for zstd on FPGA exists as well. 

Best regards,
Andreas

> Date: Fri, 28 Oct 2022 08:11:57 +0000
> From: Stahlberg Henning <henning.stahlberg at epfl.ch>
> To: Takanori Nakane <tnakane.protein at osaka-u.ac.jp>,
>         "3dem at ncmir.ucsd.edu" <3dem at ncmir.ucsd.edu>
> Subject: Re: [3dem] EER home-made?
> Message-ID: <33BE67FD-2666-4045-89D6-99D820DEA8A2 at epfl.ch>
> Content-Type: text/plain; charset="utf-8"
> 
> Dear Takanori,
> This is for diffractive imaging using a hybrid pixel detector camera, running at 8000
> fps or much faster (Dectris' ARINA will go up to 120000 fps). The frames are small
> and usually contain only few electrons.
> Saving 512x512px @ 8bit with 8000 fps to disk will result in 4.2 Gigabytes per
> second, or 15 Terabytes per hour, of data that mostly contain zeros. EER is
> preferred.
> Best, Henning.
> 
> Henning Stahlberg
> Laboratory of Biological Electron Microscopy Institute of Physics, School of Basic
> Sciences, EPFL, and Dep. of Fund. Microbiology, Faculty of Biology and
> Medicine, UNIL, Cubotron, BSP421, 1015 Lausanne, Switzerland
> 
> ?On 27.10.22, 15:38, "3dem on behalf of Takanori Nakane" <3dem-
> bounces at ncmir.ucsd.edu on behalf of tnakane.protein at osaka-u.ac.jp> wrote:
> 
>     Hi,
> 
>     Why do you want to do this?
> 
>     EER is not an efficient format in terms of the compression ratio.
> 
>     Also note that RELION's EER parser is hard coded for 4096 pixels and does not
> accept
>     other sizes.
> 
>     Best regards,
> 
>     Takanori Nakane
> 
>     On 2022/10/27 22:25, Stahlberg Henning wrote:
>     > Hi,
>     >
>     > Does anybody know if there is a public domain software available that can
> translate an EM image into a list of electron event recordings (EER file),
>     > assuming the image is mostly black and has only a hand-full of electron
> impacts on it?
>     >
>     > Obviously, that should at least run on a GPU, if not more specialized
> hardware. Dark-field, flat-field and PSF should be refined on the fly.
>     >
>     > Henning.
>     >
>     > *Henning Stahlberg*
>     > Laboratory of Biological Electron Microscopy
>     > Institute of Physics, School of Basic Sciences, EPFL, and
>     > Dep. of Fund. Microbiology, Faculty of Biology and Medicine, UNIL,
>     > Cubotron, BSP421, 1015 Lausanne, Switzerland