[3dem] Falcon II uses only half of the 1Gb/s bandwidth.

Wed Oct 17 13:40:38 PDT 2018

Dear Sargis and Steven,

Thank you for your help.

Here is a map to provide a better understanding of the problem.
[image: Network_map.jpeg]
We transfer data to Support PC that has a RAID 0 (sequential write speed
~600MB/s) through a simple Samba share. Unless FEI's EPU writes stacks to
Microscope PC's disk prior to transfer, which is unlikely, i don't see a
problem with a disk access speed.

I do have doubts about the ethernet controller used in the Camera Support
Main Unit, but I don't see a way to test it.

IOPS is definitely a problem when there are multiple simultaneous
writes/reads, which is probably a case for large CryoEM facilities like
SEMC.

On Wed, Oct 17, 2018 at 4:48 AM Ludtke, Steven J <sludtke at bcm.edu> wrote:

> Ok, can't comment specifically on the Falcon, but this seems more a
> computer issue than a camera issue.
>
> Unless you are storing your data on an external drive (USB portable drive
> or similar) it is almost certainly not the disk speed. Even a cheap
> spinning platter internal drive can manage 100-120 MB/s sustained write,
> and most will do ~150 nowadays. The theoretical peak for a 1 Gbps network
> is 120 MB/s and with overhead, 100 - 110 MB/s is a practical limit for the
> network. Non-SSD external USB drives (yes, even USB3/C) typically max out
> at ~50 MB/s.
>
> It is almost certainly not the network switch. There really aren't any
> half-duplex switches any more, and it looks like the one you have has a 32
> Gbps backplane, so should not be a bottleneck.
>
> In most cases, the real issue is the configuration of the shared
> filesystem on the Windows PC.  The K2 computers have a 10 Gbps port
> generally used for transfer to the microscope PC, and there, tuning the
> filesystem parameters (and having a sufficiently fast set of drives on the
> microscope PC) are critical to getting good throughput. So:
>
> 1) no writing directly to shared external drives
> 2) check your shared filesystem configuration (if that's what you're using
> to move the images)
> 3) if you're moving the images some other way (sftp/scp or somesuch), we
> need to know that to comment
>
>
>
> --------------------------------------------------------------------------------------
> Steven Ludtke, Ph.D. <sludtke at bcm.edu>                      Baylor
> College of Medicine
> Charles C. Bell Jr., Professor of Structural Biology
> Dept. of Biochemistry and Molecular Biology                      (
> www.bcm.edu/biochem)
> Academic Director, CryoEM Core                                        (
> cryoem.bcm.edu)
> Co-Director CIBR Center                                    (
> www.bcm.edu/research/cibr)
>
>
>
> On Oct 16, 2018, at 8:02 PM, Sargis Dallakyan <sargis at nysbc.org> wrote:
>
> ****CAUTION:*** This email is not from a BCM Source. Only click links or
> open attachments you know are safe.*
> ------------------------------
> Just to break an ice on this topic, what kind of disk is on a PC that you
> are transferring the data to? It's been mentioned in one of the latest
> DDN videos
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DNabTsYl-5FmIQ-26list-3DPLohLhg8SlVmbG8W-5FTFIWQFFhg1LJ0-2Dxf1&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=KUru33P_2mVMifpXMF33Nyz25BpRHmbk-Tg64ZHkg2I&e=>
>  that no matter how big is your bandwidth, you'll be disk IO limited if
> your storage can't support enough IOPS (https://en.wikipedia.org/wiki/IOPS
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_IOPS&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=0wiVG1LvTlAYZBXD_HJ_va_utU5R50hVDaW1XSqarA8&e=>).
> Maybe getting an SSD might solve this bottleneck, if I understand the
> problem correctly.
>
> Respectfully,
> Sargis Dallakyan, PhD - Systems Administrator
> National Resource for Automated Molecular Microscopy
> Simons Electron Microscopy Center
> New York Structural Biology Center
> http://emg.nysbc.org/redmine/users/105
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__emg.nysbc.org_redmine_users_105&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=DK8kFnGbasHV5hu6_1CAytIhdp2d9bx8DhSXEt0od1U&e=>
>
>
> Date: Tue, 16 Oct 2018 16:15:00 +0300
> From: Eugene Pichkur <eugene.pichkur at gmail.com>
> To: 3dem at ncmir.ucsd.edu
> Subject: [3dem] Falcon II uses only half of the 1Gb/s bandwidth.
> Message-ID:
>         <
> CAJJJP0=9cSHBRxKLOQkaYmqvg3X6Ym4iq-DT_oMCvNYERErr4w at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hello 3DEM community,
>
> We've noticed that our Falcon II uses only half of the 1Gbps bandwidth to
> transfer data from the Camera Support Rack to the Microscope PC (I've
> attached a screenshot to illustrate the issue).
>
> Microscope PC runs Windows 7 and we are able to get the full frame rate
> (20fps) from the detector, however, it takes ~20s to transfer a typical
> movie (40frames, 1.2GB) to the PC which is a very frustrating bottleneck.
>
> Both Falcon II and Ceta are connected through the same 1Gb switch (HPE
> OfficeConnect 1420 24G 2SFP), so my guess is that the bandwidth is split
> 50/50. However, during data collection only Falcon II is used which makes
> this limitation pointless, not to mention that Falcon II and Ceta can't be
> used simultaneously.
>
> Does anyone have a workaround or any info on the topic?
>
>
> Thanks in advance,
> Eugene Pichkur
>
> National Research Centre ?Kurchatov Institute?
> Akademika Kurchatova, 1, Moscow, Russia, 123098
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181016/a9ba7356/attachment.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.ncmir.ucsd.edu_pipermail_3dem_attachments_20181016_a9ba7356_attachment.html&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ORrcT2ZtD3pxG1SOnfC4g8e6CT_-aLkzonc2aVUQsC0&e=>
> >
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: Data_transfer_Falcon2_crop.png
> Type: image/png
> Size: 36229 bytes
> Desc: not available
> URL: <
> http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181016/a9ba7356/attachment.png
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.ncmir.ucsd.edu_pipermail_3dem_attachments_20181016_a9ba7356_attachment.png&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=_tqIiPe4uB8Wu-2WRJmE3oz5GQqMxfyFfUP-nL5xIvU&e=>
> >
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ivLCaBkOqzNDdkBqKbq_PHM4v4QIK6Qn1V0pq2BZ2OI&e=>
>
>
> ------------------------------
>
> End of 3dem Digest, Vol 134, Issue 28
> *************************************
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ivLCaBkOqzNDdkBqKbq_PHM4v4QIK6Qn1V0pq2BZ2OI&e=
>
>
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181017/7a3259eb/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Network_map.jpeg
Type: image/jpeg
Size: 589915 bytes
Desc: not available
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181017/7a3259eb/attachment-0002.jpeg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Network_map.jpeg
Type: image/jpeg
Size: 589915 bytes
Desc: not available
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181017/7a3259eb/attachment-0003.jpeg>