<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Arial Narrow";
panose-1:2 11 6 6 2 2 2 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Charles,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My Tyan FreeNAS box has a quad port 10G SFP+ baseboard-mounted NIC (AOC brand, purchased from Tyan through Superbiiz with the server). I have configured two of its ports for port aggregation with LAGP which is supported by FreeNAS. Two
optical fibers are connected to a floor switch managed by IT, which also supports LAGP. The uplink from the floor switch to a core switch in the datacenter has much higher bandwidth (100Gb/s, I think) and this is an all-optical network. My workstations in
the lab have 10Gb copper ethernet NICs (Solarflare or Intel) – they are connected to the floor switch with SFP-UTP transceivers on the switch side, because the cable in the building are all ethernet. Fortunately, these are all CAT6a or CAT7, since the buildings
are all less than 10 years old. The highest transfer rates between a cluster front end and our FreeNAS box by rsync I have seen was about 1GB/s (~10 Gb/s) disk-to-disk. I have not measured the network speed yet over this route with a tool like iperf. Between
this storage server and our lab workstations it is less (200-300Mb/s or so), but that’s probably limited by the local RAID5 array on the workstations, or because no network tuning of the workstations that run CentOS8.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We have a rack enclosure in a side room to the Titan Krios that hosts the K2 summit processor, Tyan FreeNAS box, Warp PC and two FEI Falcon storage servers. All of theses have 10G interfaces and are connected to a local optical switch managed
by IT (also in that rack). The FreeNAS also has a BMC. Out HPC people did not want the Tyan server in their datacenter, because they think Tyan/Supermicro etc are inferior hardware without a service contract. They said that there were many reports of supermicro
PSUs catching fire and they could not host such hardware in the datacenter. I understand this and I agree, but I bought the Tyan server because it was much cheaper than “professional” datacenter-grade servers.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Before the FreeNAS, I had bought a DDN storage server (55 drives, which was later expanded with an additional 80-drive expansion chassis), which is administered by IT and hosted directly in the OIST datacenter. Because the DDN has smaller
drives, is a couple years older and has ridiculous redundancy (wasting more than 30% of the raw capacity), the total usable capacity is pretty much the same as the FreeNAS. The annual service contract of the DDN alone is 1/3 of the total cost of the Tyan FreeNAS
box. I have therefore decided to let the DDN service contract lapse until something fails… it’s not a comfortable solution, but I just cannot afford this any more. I am considering swapping its drives against new large ones and installing FreeNAS on it.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">For file access we use NFS and CIFS. But I still have not found enough time to figure out NIS on Centos8. This was no problem on Centos7, but for some reason CentOS8 is much harder in this respect. I have one linux box for user authentication
in the lab using LDAP, because I want control over my local segment and IT demand to give up root access when authenticating through their central AD domain controller. FreeNAS has all the major network services preconfigured and it is a simple matter of enabling
the service. I like it quite a bit. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I highly recommend to work with your IT team to have them lock down your cryoem network on the router level – for many trouble-free years, ours has been a dedicated network segment isolated from internet and the rest of the university,
only allowing routing of the teamviewer and RAPID ports. This way we got rid of the FEI service PCs, which used to act as firewalled gateways, are crappy hardware, and kill your file transfer bandwidth. The Thermo FSE’s actually prefer teamviewer over RAPID
(which you can still run directly on the MPC on demand). All the while we can access our scope PCs and workstations through teamviewer and I could even remotely reboot my FreeNAS via its BMC and a web interface once connected to an internal workstation.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hope this provides some inspiration…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Matthias<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Charles Bayly-Jones <charles.bayly-jones@monash.edu>
<br>
<b>Sent:</b> Saturday, April 03, 2021 9:12 AM<br>
<b>To:</b> Xu, Chen <Chen.Xu@umassmed.edu><br>
<b>Cc:</b> Matthias Wolf <matthias.wolf@oist.jp>; Krishan Pandey <krishan.pandey@health.slu.edu>; 3dem@ncmir.ucsd.edu<br>
<b>Subject:</b> Re: [3dem] Data storage system<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi all.<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Our lab is also currently making similar considerations for a moderately large server. So this email chain has been quite serendipitous.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm wondering whether those (Chen, Steve, Matthias, Tim) with external servers (45drives, trueNAS, synology, etc) might comment on their network setup? Did you opt for a special network switch? Has this been necessary or can these servers
directly host the workstations? What kind of local network infrastructure do you use? Connections, SFP+/RJ45/some fancy optical thing?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I note Steve has said, in the small 12-drive synologyNAS that a 10Gbe (RJ45?) card has been sufficient for ~950/600 MB/s IO. I recall the synology can be networked directly to workstations without the need for a switch, if I'm not mistaken.
So what is people's experience with the other larger servers? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Perhaps you might also comment on your typical use case? E.g. is your server for long term storage of processed or raw data? Or is it rather a scratch space for analysis with read & write? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Very best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Charles<o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">_________________</span><span style="font-size:9.5pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial Narrow",sans-serif">Charles Bayly-Jones</span></b><span style="font-size:9.5pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial Narrow",sans-serif">BSc(ScSchProg)(Hons)</span><span style="font-size:9.5pt"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt"><o:p> </o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial Narrow",sans-serif">Monash Biomedicine Discovery Institute & </span></b><b><span style="font-size:12.0pt;font-family:"Arial Narrow",sans-serif">ARC CoE in advanced molecular imaging</span></b><span style="font-size:9.5pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:9.5pt;font-family:"Arial Narrow",sans-serif">Department of Biochemistry and Molecular Biology </span></b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial Narrow",sans-serif">Rm 218, Level 2, Building 77</span><span style="font-size:9.5pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial Narrow",sans-serif">23 Innovation Walk</span><span style="font-size:9.5pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial Narrow",sans-serif">Clayton VIC 3800</span><span style="font-size:9.5pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial Narrow",sans-serif">Australia</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Sat, 3 Apr 2021 at 04:23, Xu, Chen <<a href="mailto:Chen.Xu@umassmed.edu">Chen.Xu@umassmed.edu</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">We also have very positive experience with FreeNAS/TruNAS system. Here is the shop for their<br>
on-the-shelf units.<br>
<br>
<a href="https://urldefense.com/v3/__https:/www.amazon.com/stores/iXsystems/page/C88EFDE3-3E4C-4951-860E-0E8A8BD91BF9?ref_=ast_bln__;!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCOnSJxj9w$" target="_blank">https://urldefense.com/v3/__https://www.amazon.com/stores/iXsystems/page/C88EFDE3-3E4C-4951-860E-0E8A8BD91BF9?ref_=ast_bln__;!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCOnSJxj9w$</a>
<br>
<br>
-Chen<br>
<br>
________________________________________<br>
From: 3dem <<a href="mailto:3dem-bounces@ncmir.ucsd.edu" target="_blank">3dem-bounces@ncmir.ucsd.edu</a>> on behalf of Matthias Wolf <<a href="mailto:matthias.wolf@oist.jp" target="_blank">matthias.wolf@oist.jp</a>><br>
Sent: Friday, April 2, 2021 11:27 AM<br>
To: Krishan Pandey; <a href="mailto:3dem@ncmir.ucsd.edu" target="_blank">3dem@ncmir.ucsd.edu</a><br>
Subject: Re: [3dem] Data storage system<br>
<br>
Hi Krishan,<br>
<br>
Our 60-drive 760 TB FreeNAS has been running flawless since more than 1 year. One failed drive was easily hot swapped. If you want capacity, this does not cost more per TB than inferior hardware, yet you get good redundancy and performance.<br>
<br>
See my old tweet here <a href="https://urldefense.com/v3/__https:/twitter.com/hicryoem/status/1223966559976083458?s=21__;!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCPmSkFojQ$" target="_blank">
https://urldefense.com/v3/__https://twitter.com/hicryoem/status/1223966559976083458?s=21__;!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCPmSkFojQ$</a> <<a href="https://urldefense.com/v3/__https:/nam10.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Ftwitter.com*2Fhicryoem*2Fstatus*2F1223966559976083458*3Fs*3D21__*3B!!Mih3wA!TpItSV7E6sE1qSO0JgCwUfuLYuNhW92PMSZUaIF70zdj2z4BE_zg2izktUsKL5clRQ*24&data=04*7C01*7CChen.Xu*40umassmed.edu*7Ca3ecd1c220144d5c148208d8f5ebdb95*7Cee9155fe2da34378a6c44405faf57b2e*7C0*7C1*7C637529740649781691*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C2000&sdata=kHkXkbLqxxMR*2F*2FgoMCQQQxjSAvsIXrzIxrf93xWYXrE*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCOaPADKzw$" target="_blank">https://urldefense.com/v3/__https://nam10.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Ftwitter.com*2Fhicryoem*2Fstatus*2F1223966559976083458*3Fs*3D21__*3B!!Mih3wA!TpItSV7E6sE1qSO0JgCwUfuLYuNhW92PMSZUaIF70zdj2z4BE_zg2izktUsKL5clRQ*24&data=04*7C01*7CChen.Xu*40umassmed.edu*7Ca3ecd1c220144d5c148208d8f5ebdb95*7Cee9155fe2da34378a6c44405faf57b2e*7C0*7C1*7C637529740649781691*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C2000&sdata=kHkXkbLqxxMR*2F*2FgoMCQQQxjSAvsIXrzIxrf93xWYXrE*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCOaPADKzw$</a>
><br>
<br>
Matthias<br>
<br>
________________________________<br>
From: 3dem <<a href="mailto:3dem-bounces@ncmir.ucsd.edu" target="_blank">3dem-bounces@ncmir.ucsd.edu</a>> on behalf of Krishan Pandey <<a href="mailto:krishan.pandey@health.slu.edu" target="_blank">krishan.pandey@health.slu.edu</a>><br>
Sent: Friday, April 2, 2021 03:59<br>
To: <a href="mailto:3dem@ncmir.ucsd.edu" target="_blank">3dem@ncmir.ucsd.edu</a><br>
Subject: [3dem] Data storage system<br>
<br>
Hello,<br>
<br>
I am requesting suggestions and cost estimates about off the shelf data storage systems to store raw cryo-EM movies and processed data for our lab. Our initial target is 150-200 TB with options to expand it in future.<br>
We don't have much local IT support for Linux based systems, that's why I am asking for an off-the shelf system which should be easier to install and manage.<br>
<br>
Thank you<br>
best regards<br>
<br>
Krishan Pandey<br>
<br>
_______________________________________________<br>
3dem mailing list<br>
<a href="mailto:3dem@ncmir.ucsd.edu" target="_blank">3dem@ncmir.ucsd.edu</a><br>
<a href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem" target="_blank">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</body>
</html>