[3dem] Falcon II uses only half of the 1Gb/s bandwidth.

Wed Oct 17 14:02:20 PDT 2018

Hi Eugene,

the Falcon II most certainly writes the intermediate frames locally (to the microscope computer) before it writes the stacked file to the end destination (in your case the support PC).  You can determine the location of the intermediate frame directory by looking at the falcon frame xml file (I have forgotten the full name).  

Additionally, the microscope PC has a 1 Gbit ethernet port connected to your support PC *at best*.  Don’t forget that, the computer is trying to write across the network to your support PC as its trying to get data from the camera on another card, so there’s a resource issue.  Depending on what operating system your microscope has, you may be running a 32-bit system.  It is my understanding that, even on some 64bit OSs, FEI was emulating 32bit so they could use older software.  I’m guessing the official reason has something to do with retaining compatibility, or something.  I don’t know.  All I know is that moving data from, or near, a microscope PC should be minimised.

There is no easy fix to this, that I’m aware of.  Installing a 10Gbit card doesn’t work for most older systems (no suitable PCI slots on the motherboard).  This leaves trying to “hack” the Falcon, or intercept the image signal with a computer of your own choosing and under your control.  You can make your life a little bit easier by assigning the intermediate frames to be saved and stacked on an SSD, which has a faster write speed, but even still, you will always be limited by the speed of the network cards from the camera, and from your microscope PC.  You may actually be limiting your acquisition speed by writing across the network.  That’s something you should check. 

Feel free to ping me off-list if you want more specific details. Good luck.

mike
-------
Mike Strauss - former cryoEM Facility Manager 
MPI Biochemistry
mike at biochem.mpg.de
tel: +49 89-8578-2474
mob: +49 151 55 308040

> On 17. Oct 2018, at 16:40, <3dem-request at ncmir.ucsd.edu> <3dem-request at ncmir.ucsd.edu> wrote:
> 
> Send 3dem mailing list submissions to
> 	3dem at ncmir.ucsd.edu
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
> or, via email, send a message with subject or body 'help' to
> 	3dem-request at ncmir.ucsd.edu
> 
> You can reach the person managing the list at
> 	3dem-owner at ncmir.ucsd.edu
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of 3dem digest..."
> 
> 
> Today's Topics:
> 
>   1. Re: Falcon II uses only half of the 1Gb/s bandwidth.
>      (Eugene Pichkur)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Wed, 17 Oct 2018 23:40:38 +0300
> From: Eugene Pichkur <eugene.pichkur at gmail.com>
> To: sludtke at bcm.edu
> Cc: sargis at nysbc.org, 3dem at ncmir.ucsd.edu
> Subject: Re: [3dem] Falcon II uses only half of the 1Gb/s bandwidth.
> Message-ID:
> 	<CAJJJP0=jhwuTeNGfzyLOBEArS_N+pMhMyxKaM+qtJTm5KCZXKg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> Dear Sargis and Steven,
> 
> Thank you for your help.
> 
> Here is a map to provide a better understanding of the problem.
> [image: Network_map.jpeg]
> We transfer data to Support PC that has a RAID 0 (sequential write speed
> ~600MB/s) through a simple Samba share. Unless FEI's EPU writes stacks to
> Microscope PC's disk prior to transfer, which is unlikely, i don't see a
> problem with a disk access speed.
> 
> I do have doubts about the ethernet controller used in the Camera Support
> Main Unit, but I don't see a way to test it.
> 
> IOPS is definitely a problem when there are multiple simultaneous
> writes/reads, which is probably a case for large CryoEM facilities like
> SEMC.
> 
> 
> 
> 
> 
> 
> 
> On Wed, Oct 17, 2018 at 4:48 AM Ludtke, Steven J <sludtke at bcm.edu> wrote:
> 
>> Ok, can't comment specifically on the Falcon, but this seems more a
>> computer issue than a camera issue.
>> 
>> Unless you are storing your data on an external drive (USB portable drive
>> or similar) it is almost certainly not the disk speed. Even a cheap
>> spinning platter internal drive can manage 100-120 MB/s sustained write,
>> and most will do ~150 nowadays. The theoretical peak for a 1 Gbps network
>> is 120 MB/s and with overhead, 100 - 110 MB/s is a practical limit for the
>> network. Non-SSD external USB drives (yes, even USB3/C) typically max out
>> at ~50 MB/s.
>> 
>> It is almost certainly not the network switch. There really aren't any
>> half-duplex switches any more, and it looks like the one you have has a 32
>> Gbps backplane, so should not be a bottleneck.
>> 
>> In most cases, the real issue is the configuration of the shared
>> filesystem on the Windows PC.  The K2 computers have a 10 Gbps port
>> generally used for transfer to the microscope PC, and there, tuning the
>> filesystem parameters (and having a sufficiently fast set of drives on the
>> microscope PC) are critical to getting good throughput. So:
>> 
>> 1) no writing directly to shared external drives
>> 2) check your shared filesystem configuration (if that's what you're using
>> to move the images)
>> 3) if you're moving the images some other way (sftp/scp or somesuch), we
>> need to know that to comment
>> 
>> 
>> 
>> --------------------------------------------------------------------------------------
>> Steven Ludtke, Ph.D. <sludtke at bcm.edu>                      Baylor
>> College of Medicine
>> Charles C. Bell Jr., Professor of Structural Biology
>> Dept. of Biochemistry and Molecular Biology                      (
>> www.bcm.edu/biochem)
>> Academic Director, CryoEM Core                                        (
>> cryoem.bcm.edu)
>> Co-Director CIBR Center                                    (
>> www.bcm.edu/research/cibr)
>> 
>> 
>> 
>> On Oct 16, 2018, at 8:02 PM, Sargis Dallakyan <sargis at nysbc.org> wrote:
>> 
>> ****CAUTION:*** This email is not from a BCM Source. Only click links or
>> open attachments you know are safe.*
>> ------------------------------
>> Just to break an ice on this topic, what kind of disk is on a PC that you
>> are transferring the data to? It's been mentioned in one of the latest
>> DDN videos
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DNabTsYl-5FmIQ-26list-3DPLohLhg8SlVmbG8W-5FTFIWQFFhg1LJ0-2Dxf1&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=KUru33P_2mVMifpXMF33Nyz25BpRHmbk-Tg64ZHkg2I&e=>
>> that no matter how big is your bandwidth, you'll be disk IO limited if
>> your storage can't support enough IOPS (https://en.wikipedia.org/wiki/IOPS
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_IOPS&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=0wiVG1LvTlAYZBXD_HJ_va_utU5R50hVDaW1XSqarA8&e=>).
>> Maybe getting an SSD might solve this bottleneck, if I understand the
>> problem correctly.
>> 
>> Respectfully,
>> Sargis Dallakyan, PhD - Systems Administrator
>> National Resource for Automated Molecular Microscopy
>> Simons Electron Microscopy Center
>> New York Structural Biology Center
>> http://emg.nysbc.org/redmine/users/105
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__emg.nysbc.org_redmine_users_105&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=DK8kFnGbasHV5hu6_1CAytIhdp2d9bx8DhSXEt0od1U&e=>
>> 
>> 
>> Date: Tue, 16 Oct 2018 16:15:00 +0300
>> From: Eugene Pichkur <eugene.pichkur at gmail.com>
>> To: 3dem at ncmir.ucsd.edu
>> Subject: [3dem] Falcon II uses only half of the 1Gb/s bandwidth.
>> Message-ID:
>>        <
>> CAJJJP0=9cSHBRxKLOQkaYmqvg3X6Ym4iq-DT_oMCvNYERErr4w at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>> 
>> Hello 3DEM community,
>> 
>> We've noticed that our Falcon II uses only half of the 1Gbps bandwidth to
>> transfer data from the Camera Support Rack to the Microscope PC (I've
>> attached a screenshot to illustrate the issue).
>> 
>> Microscope PC runs Windows 7 and we are able to get the full frame rate
>> (20fps) from the detector, however, it takes ~20s to transfer a typical
>> movie (40frames, 1.2GB) to the PC which is a very frustrating bottleneck.
>> 
>> Both Falcon II and Ceta are connected through the same 1Gb switch (HPE
>> OfficeConnect 1420 24G 2SFP), so my guess is that the bandwidth is split
>> 50/50. However, during data collection only Falcon II is used which makes
>> this limitation pointless, not to mention that Falcon II and Ceta can't be
>> used simultaneously.
>> 
>> Does anyone have a workaround or any info on the topic?
>> 
>> 
>> Thanks in advance,
>> Eugene Pichkur
>> 
>> National Research Centre ?Kurchatov Institute?
>> Akademika Kurchatova, 1, Moscow, Russia, 123098
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181016/a9ba7356/attachment.html
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.ncmir.ucsd.edu_pipermail_3dem_attachments_20181016_a9ba7356_attachment.html&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ORrcT2ZtD3pxG1SOnfC4g8e6CT_-aLkzonc2aVUQsC0&e=>
>>> 
>> -------------- next part --------------
>> A non-text attachment was scrubbed...
>> Name: Data_transfer_Falcon2_crop.png
>> Type: image/png
>> Size: 36229 bytes
>> Desc: not available
>> URL: <
>> http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181016/a9ba7356/attachment.png
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.ncmir.ucsd.edu_pipermail_3dem_attachments_20181016_a9ba7356_attachment.png&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=_tqIiPe4uB8Wu-2WRJmE3oz5GQqMxfyFfUP-nL5xIvU&e=>
>>> 
>> 
>> ------------------------------
>> 
>> Subject: Digest Footer
>> 
>> _______________________________________________
>> 3dem mailing list
>> 3dem at ncmir.ucsd.edu
>> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ivLCaBkOqzNDdkBqKbq_PHM4v4QIK6Qn1V0pq2BZ2OI&e=>
>> 
>> 
>> ------------------------------
>> 
>> End of 3dem Digest, Vol 134, Issue 28
>> *************************************
>> _______________________________________________
>> 3dem mailing list
>> 3dem at ncmir.ucsd.edu
>> 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ivLCaBkOqzNDdkBqKbq_PHM4v4QIK6Qn1V0pq2BZ2OI&e=
>> 
>> 
>> _______________________________________________
>> 3dem mailing list
>> 3dem at ncmir.ucsd.edu
>> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
>> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181017/7a3259eb/attachment.html>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: Network_map.jpeg
> Type: image/jpeg
> Size: 589915 bytes
> Desc: not available
> URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181017/7a3259eb/attachment.jpeg>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: Network_map.jpeg
> Type: image/jpeg
> Size: 589915 bytes
> Desc: not available
> URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181017/7a3259eb/attachment-0001.jpeg>
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
> 
> 
> ------------------------------
> 
> End of 3dem Digest, Vol 134, Issue 36
> *************************************