<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0; margin-bottom:0">You are welcome Eugene. Nice diagram and glad to hear discussions on this topic. I assume RAID 0 storage on Support PC and the disk on Microscope PC are both SSD. What type of disks are on Camera Support Rack #1? If
there are additional computations done on it, besides data transfer, you can stop them to see if you get better transfer rate.
<br>
<br>
</p>
<p style="margin-top:0; margin-bottom:0">We have two storage servers at two different locations that are connected to different K2 PC through fiber. Anchi from our lab noticed that transfer rate on one is twice as much as on the other. I was thinking that the
distance and the switches might be causing this, but now that I think of it, the one that has twice the rate has hardware RAID6 with SSD cache. The one that is slower has software RAID6. There is an interesting discussion
<span>on software vs hardware RAID performance</span> at <a href="https://serverfault.com/questions/685289/software-vs-hardware-raid-performance-and-cache-usage" class="OWAAutoLink" id="LPlnk403362" previewremoved="true">
https://serverfault.com/questions/685289/software-vs-hardware-raid-performance-and-cache-usage.</a></p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255); font-family:Calibri,Arial,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p></p>
<div><span>Respectfully,</span><br>
Sargis Dallakyan, PhD - Systems Administrator<br>
National Resource for Automated Molecular Microscopy<br>
Simons Electron Microscopy Center<br>
New York Structural Biology Center<br>
<a title="http://emg.nysbc.org/redmine/users/105" href="http://emg.nysbc.org/redmine/users/105" id="LPNoLP">http://emg.nysbc.org/redmine/users/105</a></div>
<br>
<p></p>
</div>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Eugene Pichkur <eugene.pichkur@gmail.com><br>
<b>Sent:</b> Wednesday, October 17, 2018 1:40 PM<br>
<b>To:</b> sludtke@bcm.edu<br>
<b>Cc:</b> Sargis Dallakyan; 3dem@ncmir.ucsd.edu<br>
<b>Subject:</b> Re: [3dem] Falcon II uses only half of the 1Gb/s bandwidth.</font>
<div> </div>
</div>
<meta content="text/html; charset=utf-8">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Dear Sargis and Steven,
<div><br>
</div>
<div>Thank you for your help.<br>
</div>
<div><br>
</div>
<div>Here is a map to provide a better understanding of the problem. <br>
</div>
<div>
<div><img alt="Network_map.jpeg" tabindex="0" width="538" height="303" data-outlook-trace="F:1|T:1" src="cid:ii_jndjgnhg0"><br>
</div>
</div>
<div>We transfer data to Support PC that has a RAID 0 (sequential write speed ~600MB/s) through a simple Samba share. Unless FEI's EPU writes stacks to Microscope PC's disk prior to transfer, which is unlikely, i don't see a problem with a disk access speed. </div>
<div><br>
</div>
<div>I do have doubts about the ethernet controller used in the Camera Support Main Unit, but I don't see a way to test it. </div>
<div><br>
</div>
<div>IOPS is definitely a problem when there are multiple simultaneous writes/reads, which is probably a case for large CryoEM facilities like SEMC. </div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr">On Wed, Oct 17, 2018 at 4:48 AM Ludtke, Steven J <<a href="mailto:sludtke@bcm.edu" id="LPlnk677624" class="OWAAutoLink" previewremoved="true">sludtke@bcm.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div style="word-wrap:break-word; line-break:after-white-space">Ok, can't comment specifically on the Falcon, but this seems more a computer issue than a camera issue.
<div><br>
</div>
<div>Unless you are storing your data on an external drive (USB portable drive or similar) it is almost certainly not the disk speed. Even a cheap spinning platter internal drive can manage 100-120 MB/s sustained write, and most will do ~150 nowadays. The theoretical
peak for a 1 Gbps network is 120 MB/s and with overhead, 100 - 110 MB/s is a practical limit for the network. Non-SSD external USB drives (yes, even USB3/C) typically max out at ~50 MB/s. </div>
<div><br>
</div>
<div>It is almost certainly not the network switch. There really aren't any half-duplex switches any more, and it looks like the one you have has a 32 Gbps backplane, so should not be a bottleneck.</div>
<div><br>
</div>
<div>In most cases, the real issue is the configuration of the shared filesystem on the Windows PC. The K2 computers have a 10 Gbps port generally used for transfer to the microscope PC, and there, tuning the filesystem parameters (and having a sufficiently
fast set of drives on the microscope PC) are critical to getting good throughput. So:
<div><br>
</div>
<div>1) no writing directly to shared external drives</div>
<div>2) check your shared filesystem configuration (if that's what you're using to move the images)</div>
<div>3) if you're moving the images some other way (sftp/scp or somesuch), we need to know that to comment<br>
<div><br>
</div>
<div><br>
<div>
<div dir="auto" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" style="word-wrap:break-word; line-break:after-white-space">
<div style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">
<font face="Courier"><span style="font-size:14px">--------------------------------------------------------------------------------------<br>
Steven Ludtke, Ph.D. <<a href="mailto:sludtke@bcm.edu" target="_blank" id="LPlnk612509" class="OWAAutoLink" previewremoved="true">sludtke@bcm.edu</a>> Baylor College of Medicine <br>
Charles C. Bell Jr., Professor of Structural Biology<br>
Dept. of Biochemistry and Molecular Biology (<a href="http://www.bcm.edu/biochem" target="_blank" id="LPlnk427697" class="OWAAutoLink" previewremoved="true">www.bcm.edu/biochem</a>)<br>
Academic Director, CryoEM Core (<a href="http://cryoem.bcm.edu" target="_blank" id="LPlnk271090" class="OWAAutoLink" previewremoved="true">cryoem.bcm.edu</a>)<br>
Co-Director CIBR Center (<a href="http://www.bcm.edu/research/cibr" target="_blank" id="LPlnk254246" class="OWAAutoLink" previewremoved="true">www.bcm.edu/research/cibr</a>)<br>
<br>
</span></font><br>
</div>
</div>
</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Oct 16, 2018, at 8:02 PM, Sargis Dallakyan <<a href="mailto:sargis@nysbc.org" target="_blank" id="LPlnk52095" class="OWAAutoLink" previewremoved="true">sargis@nysbc.org</a>> wrote:</div>
<br class="x_m_6426025386482338208Apple-interchange-newline">
<div><font style="font-family:Helvetica; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none" size="2"><b>***CAUTION:*** This email
is not from a BCM Source. Only click links or open attachments you know are safe.</b></font><span style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<hr style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div id="x_m_6426025386482338208divtagdefaultwrapper" dir="ltr" style="font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-size:12pt; font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div style="margin-top:0px; margin-bottom:0px">Just to break an ice on this topic, what kind of disk is on a PC that you are transferring the data to? It's been mentioned in one of the<span class="x_m_6426025386482338208Apple-converted-space"> </span><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DNabTsYl-5FmIQ-26list-3DPLohLhg8SlVmbG8W-5FTFIWQFFhg1LJ0-2Dxf1&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=KUru33P_2mVMifpXMF33Nyz25BpRHmbk-Tg64ZHkg2I&e=" class="x_m_6426025386482338208OWAAutoLink OWAAutoLink" id="LPlnk731263" disabled="" target="_blank" previewremoved="true">latest
DDN videos</a><span class="x_m_6426025386482338208Apple-converted-space"> </span>that no matter how big is your bandwidth, you'll be disk IO limited if your storage can't support enough IOPS (<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_IOPS&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=0wiVG1LvTlAYZBXD_HJ_va_utU5R50hVDaW1XSqarA8&e=" class="x_m_6426025386482338208OWAAutoLink OWAAutoLink" id="LPlnk98064" disabled="" target="_blank" previewremoved="true">https://en.wikipedia.org/wiki/IOPS</a>).
Maybe getting an SSD might solve this bottleneck, if I understand the problem correctly.<br>
</div>
<div style="margin-top:0px; margin-bottom:0px"><br>
</div>
<div id="x_m_6426025386482338208Signature">
<div id="x_m_6426025386482338208divtagdefaultwrapper" dir="ltr" style="font-size:12pt; background-color:rgb(255,255,255); font-family:Calibri,Arial,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px"></p>
<div><span id="x_m_6426025386482338208ms-rterangepaste-start"></span><span>Respectfully,</span><span id="x_m_6426025386482338208ms-rterangepaste-end"></span><br>
Sargis Dallakyan, PhD - Systems Administrator<br>
National Resource for Automated Molecular Microscopy<br>
Simons Electron Microscopy Center<br>
New York Structural Biology Center<br>
<a title="http://emg.nysbc.org/redmine/users/105" href="https://urldefense.proofpoint.com/v2/url?u=http-3A__emg.nysbc.org_redmine_users_105&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=DK8kFnGbasHV5hu6_1CAytIhdp2d9bx8DhSXEt0od1U&e=" id="LPlnk153413" disabled="" target="_blank" class="OWAAutoLink" previewremoved="true">http://emg.nysbc.org/redmine/users/105</a></div>
<br>
<p style="margin-top:0px; margin-bottom:0px"></p>
</div>
</div>
<br>
<div>
<div class="x_m_6426025386482338208BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="x_m_6426025386482338208PlainText">Date: Tue, 16 Oct 2018 16:15:00 +0300<br>
From: Eugene Pichkur <<a href="mailto:eugene.pichkur@gmail.com" target="_blank" id="LPlnk408451" class="OWAAutoLink" previewremoved="true">eugene.pichkur@gmail.com</a>><br>
To:<span class="x_m_6426025386482338208Apple-converted-space"> </span><a href="mailto:3dem@ncmir.ucsd.edu" target="_blank" id="LPlnk257324" class="OWAAutoLink" previewremoved="true">3dem@ncmir.ucsd.edu</a><br>
Subject: [3dem] Falcon II uses only half of the 1Gb/s bandwidth.<br>
Message-ID:<br>
<<a href="mailto:CAJJJP0=9cSHBRxKLOQkaYmqvg3X6Ym4iq-DT_oMCvNYERErr4w@mail.gmail.com" target="_blank" id="LPlnk682858" class="OWAAutoLink" previewremoved="true">CAJJJP0=9cSHBRxKLOQkaYmqvg3X6Ym4iq-DT_oMCvNYERErr4w@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hello 3DEM community,<br>
<br>
We've noticed that our Falcon II uses only half of the 1Gbps bandwidth to<br>
transfer data from the Camera Support Rack to the Microscope PC (I've<br>
attached a screenshot to illustrate the issue).<br>
<br>
Microscope PC runs Windows 7 and we are able to get the full frame rate<br>
(20fps) from the detector, however, it takes ~20s to transfer a typical<br>
movie (40frames, 1.2GB) to the PC which is a very frustrating bottleneck.<br>
<br>
Both Falcon II and Ceta are connected through the same 1Gb switch (HPE<br>
OfficeConnect 1420 24G 2SFP), so my guess is that the bandwidth is split<br>
50/50. However, during data collection only Falcon II is used which makes<br>
this limitation pointless, not to mention that Falcon II and Ceta can't be<br>
used simultaneously.<br>
<br>
Does anyone have a workaround or any info on the topic?<br>
<br>
<br>
Thanks in advance,<br>
Eugene Pichkur<br>
<br>
National Research Centre ?Kurchatov Institute?<br>
Akademika Kurchatova, 1, Moscow, Russia, 123098<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.ncmir.ucsd.edu_pipermail_3dem_attachments_20181016_a9ba7356_attachment.html&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ORrcT2ZtD3pxG1SOnfC4g8e6CT_-aLkzonc2aVUQsC0&e=" id="LPlnk863791" class="x_m_6426025386482338208OWAAutoLink OWAAutoLink" disabled="" target="_blank" previewremoved="true">http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181016/a9ba7356/attachment.html</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: Data_transfer_Falcon2_crop.png<br>
Type: image/png<br>
Size: 36229 bytes<br>
Desc: not available<br>
URL: <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.ncmir.ucsd.edu_pipermail_3dem_attachments_20181016_a9ba7356_attachment.png&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=_tqIiPe4uB8Wu-2WRJmE3oz5GQqMxfyFfUP-nL5xIvU&e=" id="LPlnk54551" class="x_m_6426025386482338208OWAAutoLink OWAAutoLink" disabled="" target="_blank" previewremoved="true">http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181016/a9ba7356/attachment.png</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
3dem mailing list<br>
<a href="mailto:3dem@ncmir.ucsd.edu" target="_blank" id="LPlnk87167" class="OWAAutoLink" previewremoved="true">3dem@ncmir.ucsd.edu</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwMFAw&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ivLCaBkOqzNDdkBqKbq_PHM4v4QIK6Qn1V0pq2BZ2OI&e=" id="LPlnk759669" class="x_m_6426025386482338208OWAAutoLink OWAAutoLink" disabled="" target="_blank" previewremoved="true">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><br>
<br>
<br>
------------------------------<br>
<br>
End of 3dem Digest, Vol 134, Issue 28<br>
*************************************<br>
</div>
</span></font></div>
</div>
</div>
<span style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important">_______________________________________________</span><br style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<span style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important">3dem
mailing list</span><br style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<a href="mailto:3dem@ncmir.ucsd.edu" target="_blank" id="LPlnk833707" class="OWAAutoLink" style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px" previewremoved="true">3dem@ncmir.ucsd.edu</a><br style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ivLCaBkOqzNDdkBqKbq_PHM4v4QIK6Qn1V0pq2BZ2OI&e=" target="_blank" id="LPlnk864912" class="OWAAutoLink" style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px" previewremoved="true">https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=Dk5VoQQ-wINYVssLMZihyC5Dj_sWYKxCyKz9E4Lp3gc&m=4ZvqXtesRM4IBU-zUAPbIdFLZVEo1F1tixUiF6vUDKE&s=ivLCaBkOqzNDdkBqKbq_PHM4v4QIK6Qn1V0pq2BZ2OI&e=</a></div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
_______________________________________________<br>
3dem mailing list<br>
<a href="mailto:3dem@ncmir.ucsd.edu" target="_blank" id="LPlnk59047" class="OWAAutoLink" previewremoved="true">3dem@ncmir.ucsd.edu</a><br>
<a href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem" rel="noreferrer" target="_blank" id="LPlnk487260" class="OWAAutoLink" previewremoved="true">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>