<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Dear Bradford, Mike and Steven,<br><br></div><div dir="ltr"><br></div><div dir="ltr">Bypassing the switch is an excellent idea, I will try it. It might change the IP of the camera and make it inaccessible to the software, but one problem at a time, as they say. </div><div dir="ltr"><br></div><div>Mike, thank you for providing some insight into the way FEI's software works. Our Microscope PC has 16GB of RAM, so it probably runs a 64bit Win 7. </div><div>I remember reading about Falcon hack at <a href="http://bio.brandeis.edu">bio.brandeis.edu</a>, but I think, they've took down the page. As far as I remember, they used BlackBox optical tap to split signal and record frames on the separate computer. </div><div><br></div><div>I will also dig into Samba related mess, thanks for sharing!</div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Oct 18, 2018 at 12:16 AM Ludtke, Steven J <<a href="mailto:sludtke@bcm.edu">sludtke@bcm.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;line-break:after-white-space">
IOPS may be an issue, but I suspect it isn't really the cause of your slowdown. Certainly you could install a SSD in the Support PC and test, but I'm skeptical.
<div><br>
</div>
<div>The Samba share is almost certainly the culprit in limited transfer rates here. I am not aware of any network cards anyone ships in a "full computer" nowadays which cannot trivially handle 1Gbps full duplex. 10Gbps can still have some hardware
bottlenecks, but not 1. Most of the tuning I've done has been for Linux <-> Windows, not Windows to Windows, but these may help:</div>
<div><br>
</div>
<div><a href="https://www.experts-exchange.com/articles/3545/Samba-3-for-Windows-Vista-and-later.html" target="_blank">https://www.experts-exchange.com/articles/3545/Samba-3-for-Windows-Vista-and-later.html</a></div>
<div><a href="https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/role/file-server/smb-file-server" target="_blank">https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/role/file-server/smb-file-server</a></div>
<div><br>
</div>
<div>
<div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word;line-break:after-white-space">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word;line-break:after-white-space">
<div dir="auto" style="word-wrap:break-word;line-break:after-white-space">
<div style="color:rgb(0,0,0);font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<font face="Courier"><span style="font-size:14px">--------------------------------------------------------------------------------------<br>
Steven Ludtke, Ph.D. <<a href="mailto:sludtke@bcm.edu" target="_blank">sludtke@bcm.edu</a>> Baylor College of Medicine <br>
Charles C. Bell Jr., Professor of Structural Biology<br>
Dept. of Biochemistry and Molecular Biology (<a href="http://www.bcm.edu/biochem" target="_blank">www.bcm.edu/biochem</a>)<br>
Academic Director, CryoEM Core (<a href="http://cryoem.bcm.edu" target="_blank">cryoem.bcm.edu</a>)<br>
Co-Director CIBR Center (<a href="http://www.bcm.edu/research/cibr" target="_blank">www.bcm.edu/research/cibr</a>)<br>
<br>
</span></font><br>
</div>
</div>
</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Oct 17, 2018, at 3:40 PM, Eugene Pichkur <<a href="mailto:eugene.pichkur@gmail.com" target="_blank">eugene.pichkur@gmail.com</a>> wrote:</div>
<br class="m_3465790147179592932Apple-interchange-newline">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Dear Sargis and Steven,
<div><br>
</div>
<div>Thank you for your help.<br>
</div>
<div><br>
</div>
<div>Here is a map to provide a better understanding of the problem. <br>
</div>
<div>
<div><span id="m_3465790147179592932cid:ii_jndjgnhg0"><Network_map.jpeg></span><br>
</div>
</div>
<div>We transfer data to Support PC that has a RAID 0 (sequential write speed ~600MB/s) through a simple Samba share. Unless FEI's EPU writes stacks to Microscope PC's disk prior to transfer, which is unlikely, i don't see a problem with a disk access
speed. </div>
<div><br>
</div>
<div>I do have doubts about the ethernet controller used in the Camera Support Main Unit, but I don't see a way to test it. </div>
<div><br>
</div>
<div>IOPS is definitely a problem when there are multiple simultaneous writes/reads, which is probably a case for large CryoEM facilities like SEMC. </div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Wed, Oct 17, 2018 at 4:48 AM Ludtke, Steven J <<a href="mailto:sludtke@bcm.edu" target="_blank">sludtke@bcm.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;line-break:after-white-space">Ok, can't comment specifically on the Falcon, but this seems more a computer issue than a camera issue.
<div><br>
</div>
<div>Unless you are storing your data on an external drive (USB portable drive or similar) it is almost certainly not the disk speed. Even a cheap spinning platter internal drive can manage 100-120 MB/s sustained write, and most will do ~150 nowadays.
The theoretical peak for a 1 Gbps network is 120 MB/s and with overhead, 100 - 110 MB/s is a practical limit for the network. Non-SSD external USB drives (yes, even USB3/C) typically max out at ~50 MB/s. </div>
<div><br>
</div>
<div>It is almost certainly not the network switch. There really aren't any half-duplex switches any more, and it looks like the one you have has a 32 Gbps backplane, so should not be a bottleneck.</div>
<div><br>
</div>
<div>In most cases, the real issue is the configuration of the shared filesystem on the Windows PC. The K2 computers have a 10 Gbps port generally used for transfer to the microscope PC, and there, tuning the filesystem parameters (and having a sufficiently
fast set of drives on the microscope PC) are critical to getting good throughput. So:
<div><br>
</div>
<div>1) no writing directly to shared external drives</div>
<div>2) check your shared filesystem configuration (if that's what you're using to move the images)</div>
<div>3) if you're moving the images some other way (sftp/scp or somesuch), we need to know that to comment<br>
<div><br>
</div>
<div><br>
<div>
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word;line-break:after-white-space">
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word;line-break:after-white-space">
<div dir="auto" style="word-wrap:break-word;line-break:after-white-space">
<div style="font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<font face="Courier"><span style="font-size:14px">--------------------------------------------------------------------------------------<br>
Steven Ludtke, Ph.D. <<a href="mailto:sludtke@bcm.edu" target="_blank">sludtke@bcm.edu</a>> Baylor College of Medicine <br>
Charles C. Bell Jr., Professor of Structural Biology<br>
Dept. of Biochemistry and Molecular Biology (<a href="http://www.bcm.edu/biochem" target="_blank">www.bcm.edu/biochem</a>)<br>
Academic Director, CryoEM Core (<a href="http://cryoem.bcm.edu/" target="_blank">cryoem.bcm.edu</a>)<br>
Co-Director CIBR Center (<a href="http://www.bcm.edu/research/cibr" target="_blank">www.bcm.edu/research/cibr</a>)<br>
<br>
</span></font><br>
</div>
</div>
</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Oct 16, 2018, at 8:02 PM, Sargis Dallakyan <<a href="mailto:sargis@nysbc.org" target="_blank">sargis@nysbc.org</a>> wrote:</div>
<br class="m_3465790147179592932m_6426025386482338208Apple-interchange-newline">
<div><font size="2" style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><b>***CAUTION:***
This email is not from a BCM Source. Only click links or open attachments you know are safe.</b></font><span style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline!important"></span>
<hr style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div id="m_3465790147179592932m_6426025386482338208divtagdefaultwrapper" dir="ltr" style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-size:12pt;font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div style="margin-top:0px;margin-bottom:0px">Just to break an ice on this topic, what kind of disk is on a PC that you are transferring the data to? It's been mentioned in one of the<span class="m_3465790147179592932m_6426025386482338208Apple-converted-space"> </span><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DNabTsYl-5FmIQ-26list-3DPLohLhg8SlVmbG8W-5FTFIWQFFhg1LJ0-2Dxf1&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=KUru33P_2mVMifpXMF33Nyz25BpRHmbk-Tg64ZHkg2I&e=" class="m_3465790147179592932m_6426025386482338208OWAAutoLink" id="m_3465790147179592932m_6426025386482338208LPlnk186991" disabled target="_blank">latest
DDN videos</a><span class="m_3465790147179592932m_6426025386482338208Apple-converted-space"> </span>that no matter how big is your bandwidth, you'll be disk IO limited if your storage can't support enough IOPS (<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_IOPS&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=0wiVG1LvTlAYZBXD_HJ_va_utU5R50hVDaW1XSqarA8&e=" class="m_3465790147179592932m_6426025386482338208OWAAutoLink" id="m_3465790147179592932m_6426025386482338208LPlnk502046" disabled target="_blank">https://en.wikipedia.org/wiki/IOPS</a>).
Maybe getting an SSD might solve this bottleneck, if I understand the problem correctly.<br>
</div>
<div style="margin-top:0px;margin-bottom:0px"><br>
</div>
<div id="m_3465790147179592932m_6426025386482338208Signature">
<div id="m_3465790147179592932m_6426025386482338208divtagdefaultwrapper" dir="ltr" style="font-size:12pt;background-color:rgb(255,255,255);font-family:Calibri,Arial,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px;margin-bottom:0px"></p>
<div><span id="m_3465790147179592932m_6426025386482338208ms-rterangepaste-start"></span><span>Respectfully,</span><span id="m_3465790147179592932m_6426025386482338208ms-rterangepaste-end"></span><br>
Sargis Dallakyan, PhD - Systems Administrator<br>
National Resource for Automated Molecular Microscopy<br>
Simons Electron Microscopy Center<br>
New York Structural Biology Center<br>
<a title="http://emg.nysbc.org/redmine/users/105" href="https://urldefense.proofpoint.com/v2/url?u=http-3A__emg.nysbc.org_redmine_users_105&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=DK8kFnGbasHV5hu6_1CAytIhdp2d9bx8DhSXEt0od1U&e=" id="m_3465790147179592932m_6426025386482338208LPNoLP" disabled target="_blank">http://emg.nysbc.org/redmine/users/105</a></div>
<br>
<p style="margin-top:0px;margin-bottom:0px"></p>
</div>
</div>
<br>
<div>
<div class="m_3465790147179592932m_6426025386482338208BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="m_3465790147179592932m_6426025386482338208PlainText">Date: Tue, 16 Oct 2018 16:15:00 +0300<br>
From: Eugene Pichkur <<a href="mailto:eugene.pichkur@gmail.com" target="_blank">eugene.pichkur@gmail.com</a>><br>
To:<span class="m_3465790147179592932m_6426025386482338208Apple-converted-space"> </span><a href="mailto:3dem@ncmir.ucsd.edu" target="_blank">3dem@ncmir.ucsd.edu</a><br>
Subject: [3dem] Falcon II uses only half of the 1Gb/s bandwidth.<br>
Message-ID:<br>
<<a href="mailto:CAJJJP0=9cSHBRxKLOQkaYmqvg3X6Ym4iq-DT_oMCvNYERErr4w@mail.gmail.com" target="_blank">CAJJJP0=9cSHBRxKLOQkaYmqvg3X6Ym4iq-DT_oMCvNYERErr4w@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hello 3DEM community,<br>
<br>
We've noticed that our Falcon II uses only half of the 1Gbps bandwidth to<br>
transfer data from the Camera Support Rack to the Microscope PC (I've<br>
attached a screenshot to illustrate the issue).<br>
<br>
Microscope PC runs Windows 7 and we are able to get the full frame rate<br>
(20fps) from the detector, however, it takes ~20s to transfer a typical<br>
movie (40frames, 1.2GB) to the PC which is a very frustrating bottleneck.<br>
<br>
Both Falcon II and Ceta are connected through the same 1Gb switch (HPE<br>
OfficeConnect 1420 24G 2SFP), so my guess is that the bandwidth is split<br>
50/50. However, during data collection only Falcon II is used which makes<br>
this limitation pointless, not to mention that Falcon II and Ceta can't be<br>
used simultaneously.<br>
<br>
Does anyone have a workaround or any info on the topic?<br>
<br>
<br>
Thanks in advance,<br>
Eugene Pichkur<br>
<br>
National Research Centre ?Kurchatov Institute?<br>
Akademika Kurchatova, 1, Moscow, Russia, 123098<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.ncmir.ucsd.edu_pipermail_3dem_attachments_20181016_a9ba7356_attachment.html&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ORrcT2ZtD3pxG1SOnfC4g8e6CT_-aLkzonc2aVUQsC0&e=" id="m_3465790147179592932m_6426025386482338208LPlnk596493" class="m_3465790147179592932m_6426025386482338208OWAAutoLink" disabled target="_blank">http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181016/a9ba7356/attachment.html</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: Data_transfer_Falcon2_crop.png<br>
Type: image/png<br>
Size: 36229 bytes<br>
Desc: not available<br>
URL: <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.ncmir.ucsd.edu_pipermail_3dem_attachments_20181016_a9ba7356_attachment.png&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=_tqIiPe4uB8Wu-2WRJmE3oz5GQqMxfyFfUP-nL5xIvU&e=" id="m_3465790147179592932m_6426025386482338208LPlnk400284" class="m_3465790147179592932m_6426025386482338208OWAAutoLink" disabled target="_blank">http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181016/a9ba7356/attachment.png</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
3dem mailing list<br>
<a href="mailto:3dem@ncmir.ucsd.edu" target="_blank">3dem@ncmir.ucsd.edu</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ivLCaBkOqzNDdkBqKbq_PHM4v4QIK6Qn1V0pq2BZ2OI&e=" id="m_3465790147179592932m_6426025386482338208LPlnk220647" class="m_3465790147179592932m_6426025386482338208OWAAutoLink" disabled target="_blank">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><br>
<br>
<br>
------------------------------<br>
<br>
End of 3dem Digest, Vol 134, Issue 28<br>
*************************************<br>
</div>
</span></font></div>
</div>
</div>
<span style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline!important">_______________________________________________</span><br style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<span style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline!important">3dem
mailing list</span><br style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<a href="mailto:3dem@ncmir.ucsd.edu" style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">3dem@ncmir.ucsd.edu</a><br style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ivLCaBkOqzNDdkBqKbq_PHM4v4QIK6Qn1V0pq2BZ2OI&e=" style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ivLCaBkOqzNDdkBqKbq_PHM4v4QIK6Qn1V0pq2BZ2OI&e=</a></div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
_______________________________________________<br>
3dem mailing list<br>
<a href="mailto:3dem@ncmir.ucsd.edu" target="_blank">3dem@ncmir.ucsd.edu</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwMFaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=L8Esi10beB1R-ERlRCSQfmHZy_5muHF1yPo7souDjQI&s=xwSQw1wJyYBDsPfmQNSjL71FdnsXhuymUJJC7HgMgd4&e=" rel="noreferrer" target="_blank">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><br>
</blockquote>
</div>
<span id="m_3465790147179592932cid:f_jndm88tp1"><Network_map.jpeg></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote></div>