[3dem] Falcon II uses only half of the 1Gb/s bandwidth.

Wed Oct 17 14:45:38 PDT 2018

Dear Bradford, Mike and Steven,

Bypassing the switch is an excellent idea, I will try it. It might change
the IP of the camera and make it inaccessible to the software, but one
problem at a time, as they say.

Mike, thank you for providing some insight into the way FEI's software
works. Our Microscope PC has 16GB of RAM, so it probably runs a 64bit Win
7.
I remember reading about Falcon hack at bio.brandeis.edu, but I think,
they've took down the page. As far as I remember, they used BlackBox
optical tap to split signal and record frames on the separate computer.

I will also dig into Samba related mess, thanks for sharing!

On Thu, Oct 18, 2018 at 12:16 AM Ludtke, Steven J <sludtke at bcm.edu> wrote:

> IOPS may be an issue, but I suspect it isn't really the cause of your
> slowdown. Certainly you could install a SSD in the Support PC and test, but
> I'm skeptical.
>
> The Samba share is almost certainly the culprit in limited transfer rates
> here. I am not aware of any network cards anyone ships in a "full computer"
> nowadays which cannot trivially handle 1Gbps full duplex. 10Gbps can still
> have some hardware bottlenecks, but not 1.  Most of the tuning I've done
> has been for Linux <-> Windows, not Windows to Windows, but these may help:
>
>
> https://www.experts-exchange.com/articles/3545/Samba-3-for-Windows-Vista-and-later.html
>
> https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/role/file-server/smb-file-server
>
>
> --------------------------------------------------------------------------------------
> Steven Ludtke, Ph.D. <sludtke at bcm.edu>                      Baylor
> College of Medicine
> Charles C. Bell Jr., Professor of Structural Biology
> Dept. of Biochemistry and Molecular Biology                      (
> www.bcm.edu/biochem)
> Academic Director, CryoEM Core                                        (
> cryoem.bcm.edu)
> Co-Director CIBR Center                                    (
> www.bcm.edu/research/cibr)
>
>
>
> On Oct 17, 2018, at 3:40 PM, Eugene Pichkur <eugene.pichkur at gmail.com>
> wrote:
>
> Dear Sargis and Steven,
>
> Thank you for your help.
>
> Here is a map to provide a better understanding of the problem.
> <Network_map.jpeg>
> We transfer data to Support PC that has a RAID 0 (sequential write speed
> ~600MB/s) through a simple Samba share. Unless FEI's EPU writes stacks to
> Microscope PC's disk prior to transfer, which is unlikely, i don't see a
> problem with a disk access speed.
>
> I do have doubts about the ethernet controller used in the Camera Support
> Main Unit, but I don't see a way to test it.
>
> IOPS is definitely a problem when there are multiple simultaneous
> writes/reads, which is probably a case for large CryoEM facilities like
> SEMC.
>
>
>
>
>
>
>
> On Wed, Oct 17, 2018 at 4:48 AM Ludtke, Steven J <sludtke at bcm.edu> wrote:
>
>> Ok, can't comment specifically on the Falcon, but this seems more a
>> computer issue than a camera issue.
>>
>> Unless you are storing your data on an external drive (USB portable drive
>> or similar) it is almost certainly not the disk speed. Even a cheap
>> spinning platter internal drive can manage 100-120 MB/s sustained write,
>> and most will do ~150 nowadays. The theoretical peak for a 1 Gbps network
>> is 120 MB/s and with overhead, 100 - 110 MB/s is a practical limit for the
>> network. Non-SSD external USB drives (yes, even USB3/C) typically max out
>> at ~50 MB/s.
>>
>> It is almost certainly not the network switch. There really aren't any
>> half-duplex switches any more, and it looks like the one you have has a 32
>> Gbps backplane, so should not be a bottleneck.
>>
>> In most cases, the real issue is the configuration of the shared
>> filesystem on the Windows PC.  The K2 computers have a 10 Gbps port
>> generally used for transfer to the microscope PC, and there, tuning the
>> filesystem parameters (and having a sufficiently fast set of drives on the
>> microscope PC) are critical to getting good throughput. So:
>>
>> 1) no writing directly to shared external drives
>> 2) check your shared filesystem configuration (if that's what you're
>> using to move the images)
>> 3) if you're moving the images some other way (sftp/scp or somesuch), we
>> need to know that to comment
>>
>>
>>
>> --------------------------------------------------------------------------------------
>> Steven Ludtke, Ph.D. <sludtke at bcm.edu>                      Baylor
>> College of Medicine
>> Charles C. Bell Jr., Professor of Structural Biology
>> Dept. of Biochemistry and Molecular Biology                      (
>> www.bcm.edu/biochem)
>> Academic Director, CryoEM Core                                        (
>> cryoem.bcm.edu)
>> Co-Director CIBR Center                                    (
>> www.bcm.edu/research/cibr)
>>
>>
>>
>> On Oct 16, 2018, at 8:02 PM, Sargis Dallakyan <sargis at nysbc.org> wrote:
>>
>> ****CAUTION:*** This email is not from a BCM Source. Only click links or
>> open attachments you know are safe.*
>> ------------------------------
>> Just to break an ice on this topic, what kind of disk is on a PC that you
>> are transferring the data to? It's been mentioned in one of the latest
>> DDN videos
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DNabTsYl-5FmIQ-26list-3DPLohLhg8SlVmbG8W-5FTFIWQFFhg1LJ0-2Dxf1&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=KUru33P_2mVMifpXMF33Nyz25BpRHmbk-Tg64ZHkg2I&e=>
>>  that no matter how big is your bandwidth, you'll be disk IO limited if
>> your storage can't support enough IOPS (
>> https://en.wikipedia.org/wiki/IOPS
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_IOPS&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=0wiVG1LvTlAYZBXD_HJ_va_utU5R50hVDaW1XSqarA8&e=>).
>> Maybe getting an SSD might solve this bottleneck, if I understand the
>> problem correctly.
>>
>> Respectfully,
>> Sargis Dallakyan, PhD - Systems Administrator
>> National Resource for Automated Molecular Microscopy
>> Simons Electron Microscopy Center
>> New York Structural Biology Center
>> http://emg.nysbc.org/redmine/users/105
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__emg.nysbc.org_redmine_users_105&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=DK8kFnGbasHV5hu6_1CAytIhdp2d9bx8DhSXEt0od1U&e=>
>>
>>
>> Date: Tue, 16 Oct 2018 16:15:00 +0300
>> From: Eugene Pichkur <eugene.pichkur at gmail.com>
>> To: 3dem at ncmir.ucsd.edu
>> Subject: [3dem] Falcon II uses only half of the 1Gb/s bandwidth.
>> Message-ID:
>>         <
>> CAJJJP0=9cSHBRxKLOQkaYmqvg3X6Ym4iq-DT_oMCvNYERErr4w at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hello 3DEM community,
>>
>> We've noticed that our Falcon II uses only half of the 1Gbps bandwidth to
>> transfer data from the Camera Support Rack to the Microscope PC (I've
>> attached a screenshot to illustrate the issue).
>>
>> Microscope PC runs Windows 7 and we are able to get the full frame rate
>> (20fps) from the detector, however, it takes ~20s to transfer a typical
>> movie (40frames, 1.2GB) to the PC which is a very frustrating bottleneck.
>>
>> Both Falcon II and Ceta are connected through the same 1Gb switch (HPE
>> OfficeConnect 1420 24G 2SFP), so my guess is that the bandwidth is split
>> 50/50. However, during data collection only Falcon II is used which makes
>> this limitation pointless, not to mention that Falcon II and Ceta can't be
>> used simultaneously.
>>
>> Does anyone have a workaround or any info on the topic?
>>
>>
>> Thanks in advance,
>> Eugene Pichkur
>>
>> National Research Centre ?Kurchatov Institute?
>> Akademika Kurchatova, 1, Moscow, Russia, 123098
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181016/a9ba7356/attachment.html
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.ncmir.ucsd.edu_pipermail_3dem_attachments_20181016_a9ba7356_attachment.html&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ORrcT2ZtD3pxG1SOnfC4g8e6CT_-aLkzonc2aVUQsC0&e=>
>> >
>> -------------- next part --------------
>> A non-text attachment was scrubbed...
>> Name: Data_transfer_Falcon2_crop.png
>> Type: image/png
>> Size: 36229 bytes
>> Desc: not available
>> URL: <
>> http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181016/a9ba7356/attachment.png
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.ncmir.ucsd.edu_pipermail_3dem_attachments_20181016_a9ba7356_attachment.png&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=_tqIiPe4uB8Wu-2WRJmE3oz5GQqMxfyFfUP-nL5xIvU&e=>
>> >
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> 3dem mailing list
>> 3dem at ncmir.ucsd.edu
>> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ivLCaBkOqzNDdkBqKbq_PHM4v4QIK6Qn1V0pq2BZ2OI&e=>
>>
>>
>> ------------------------------
>>
>> End of 3dem Digest, Vol 134, Issue 28
>> *************************************
>> _______________________________________________
>> 3dem mailing list
>> 3dem at ncmir.ucsd.edu
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ivLCaBkOqzNDdkBqKbq_PHM4v4QIK6Qn1V0pq2BZ2OI&e=
>>
>>
>> _______________________________________________
>> 3dem mailing list
>> 3dem at ncmir.ucsd.edu
>> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwMFaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=L8Esi10beB1R-ERlRCSQfmHZy_5muHF1yPo7souDjQI&s=xwSQw1wJyYBDsPfmQNSjL71FdnsXhuymUJJC7HgMgd4&e=>
>>
> <Network_map.jpeg>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181018/b6cccabd/attachment-0001.html>