[3dem] Data storage system

Fri Apr 2 19:52:03 PDT 2021

Make sure you get a server that has tool-free HDD / SSD caddies. The older
servers use caddies that require 4 screws to secure each HDD. Why is this
relevant? The HDDs you buy today will not have enough capacity in 3 - 4
years. At that time, you'll need to upgrade tens of HDDs, which is a breeze
if the caddies are tool free, but a huge time sink if they're not. The
Supermicro servers that Matthias and Craig mentioned should have tool free
caddies. If you order a different model, double check because some of the
older style servers are still being sold. In addition to Supermicro, you
check this website, which has good reviews of modern servers that point out
important features like the caddies:
https://urldefense.com/v3/__https://www.servethehome.com/category/storage/__;!!Mih3wA!XrhN182RbdR5C-yMny1ZshMkzyzigNP9QlO3eQsDtd_Mzbb6gpRnGONjFJGv42S7-A$ 

Regarding backups, a slower but cheaper alternative is to use
Dropbox's advanced business plan. It's < USD 1K per year for "as much as
you need" storage. The downside is that transfer to/from is slow. In
Singapore, we get 20 - 40 MB/sec. To put this number range in perspective,
we just had to restore all our backed up data (35 TB) to our local server
and it took 2 weeks.

Cheers.
Lu

--
Lu Gan
Associate Professor
Department of Biological Sciences
Centre for BioImaging Sciences
National University of Singapore
14 Science Drive 4
S1A, Lvl 2
Singapore 117543

https://urldefense.com/v3/__http://www.anaphase.org__;!!Mih3wA!XrhN182RbdR5C-yMny1ZshMkzyzigNP9QlO3eQsDtd_Mzbb6gpRnGONjFJGqJUQ5DQ$ 

Tel: (65) 6516 8868

On Sat, Apr 3, 2021 at 10:14 AM Matthias Wolf <matthias.wolf at oist.jp> wrote:

> Hi Charles,
>
>
>
> My Tyan FreeNAS box has a quad port 10G SFP+ baseboard-mounted NIC (AOC
> brand, purchased from Tyan through Superbiiz with the server). I have
> configured two of its ports for port aggregation with LAGP which is
> supported by FreeNAS. Two optical fibers are connected to a floor switch
> managed by IT, which also supports LAGP. The uplink from the floor switch
> to a core switch in the datacenter has much higher bandwidth (100Gb/s, I
> think) and this is an all-optical network. My workstations in the lab have
> 10Gb copper ethernet NICs (Solarflare or Intel) – they are connected to the
> floor switch with SFP-UTP transceivers on the switch side, because the
> cable in the building are all ethernet. Fortunately, these are all CAT6a or
> CAT7, since the buildings are all less than 10 years old. The highest
> transfer rates between a cluster front end and our FreeNAS box by rsync I
> have seen was about 1GB/s (~10 Gb/s) disk-to-disk. I have not measured the
> network speed yet over this route with a tool like iperf. Between this
> storage server and our lab workstations it is less (200-300Mb/s or so), but
> that’s probably limited by the local RAID5 array on the workstations, or
> because no network tuning of the workstations that run CentOS8.
>
>
>
> We have a rack enclosure in a side room to the Titan Krios that hosts the
> K2 summit processor, Tyan FreeNAS box, Warp PC and two FEI Falcon storage
> servers. All of theses have 10G interfaces and are connected to a local
> optical switch managed by IT (also in that rack). The FreeNAS also has a
> BMC. Out HPC people did not want the Tyan server in their datacenter,
> because they think Tyan/Supermicro etc are inferior hardware without a
> service contract. They said that there were many reports of supermicro PSUs
> catching fire and they could not host such hardware in the datacenter. I
> understand this and I agree, but I bought the Tyan server because it was
> much cheaper than “professional” datacenter-grade servers.
>
>
>
> Before the FreeNAS, I had bought a DDN storage server (55 drives, which
> was later expanded with an additional 80-drive expansion chassis), which is
> administered by IT and hosted directly in the OIST datacenter. Because the
> DDN has smaller drives, is a couple years older and has ridiculous
> redundancy (wasting more than 30% of the raw capacity), the total usable
> capacity is pretty much the same as the FreeNAS. The annual service
> contract of the DDN alone is 1/3 of the total cost of the Tyan FreeNAS box.
> I have therefore decided to let the DDN service contract lapse until
> something fails… it’s not a comfortable solution, but I just cannot afford
> this any more. I am considering swapping its drives against new large ones
> and installing FreeNAS on it.
>
>
>
> For file access we use NFS and CIFS. But I still have not found enough
> time to figure out NIS on Centos8. This was no problem on  Centos7, but for
> some reason CentOS8 is much harder in this respect. I have one linux box
> for user authentication in the lab using LDAP, because I want control over
> my local segment and IT demand to give up root access when authenticating
> through their central AD domain controller. FreeNAS has all the major
> network services preconfigured and it is a simple matter of enabling the
> service. I like it quite a bit.
>
>
>
> I highly recommend to work with your IT team to have them lock down your
> cryoem network on the router level – for many trouble-free years, ours has
> been a dedicated network segment isolated from internet and the rest of the
> university, only allowing routing of the teamviewer and RAPID ports. This
> way we got rid of the FEI service PCs, which used to act as firewalled
> gateways, are crappy hardware, and kill your file transfer bandwidth. The
> Thermo FSE’s actually prefer teamviewer over RAPID (which you can still run
> directly on the MPC on demand). All the while we can access our scope PCs
> and workstations through teamviewer and I could even remotely reboot my
> FreeNAS via its BMC and a web interface once connected to an internal
> workstation.
>
>
>
> Hope this provides some inspiration…
>
>
>
>    Matthias
>
>
>
> *From:* Charles Bayly-Jones <charles.bayly-jones at monash.edu>
> *Sent:* Saturday, April 03, 2021 9:12 AM
> *To:* Xu, Chen <Chen.Xu at umassmed.edu>
> *Cc:* Matthias Wolf <matthias.wolf at oist.jp>; Krishan Pandey <
> krishan.pandey at health.slu.edu>; 3dem at ncmir.ucsd.edu
> *Subject:* Re: [3dem] Data storage system
>
>
>
> Hi all.
>
>
>
> Our lab is also currently making similar considerations for a moderately
> large server. So this email chain has been quite serendipitous.
>
>
>
> I'm wondering whether those (Chen, Steve, Matthias, Tim) with external
> servers (45drives, trueNAS, synology, etc) might comment on their network
> setup? Did you opt for a special network switch? Has this been necessary or
> can these servers directly host the workstations? What kind of local
> network infrastructure do you use? Connections, SFP+/RJ45/some fancy
> optical thing?
>
>
>
> I note Steve has said, in the small 12-drive synologyNAS that a 10Gbe
> (RJ45?) card has been sufficient for ~950/600 MB/s IO. I recall the
> synology can be networked directly to workstations without the need for a
> switch, if I'm not mistaken. So what is people's experience with the other
> larger servers?
>
>
>
> Perhaps you might also comment on your typical use case? E.g. is your
> server for long term storage of processed or raw data? Or is it rather a
> scratch space for analysis with read & write?
>
>
>
> Very best,
>
> Charles
>
> _________________
>
>
>
> *Charles Bayly-Jones*
>
> BSc(ScSchProg)(Hons)
>
>
>
> *Monash Biomedicine Discovery Institute & **ARC CoE in advanced molecular
> imaging*
>
> *Department of Biochemistry and Molecular Biology *
>
> Rm 218, Level 2, Building 77
>
> 23 Innovation Walk
>
> Clayton VIC 3800
>
> Australia
>
>
>
> On Sat, 3 Apr 2021 at 04:23, Xu, Chen <Chen.Xu at umassmed.edu> wrote:
>
> We also have very positive experience with FreeNAS/TruNAS system. Here is
> the shop for their
> on-the-shelf units.
>
>
> https://urldefense.com/v3/__https://www.amazon.com/stores/iXsystems/page/C88EFDE3-3E4C-4951-860E-0E8A8BD91BF9?ref_=ast_bln__;!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCOnSJxj9w$
> <https://urldefense.com/v3/__https:/www.amazon.com/stores/iXsystems/page/C88EFDE3-3E4C-4951-860E-0E8A8BD91BF9?ref_=ast_bln__;!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCOnSJxj9w$>
>
> -Chen
>
> ________________________________________
> From: 3dem <3dem-bounces at ncmir.ucsd.edu> on behalf of Matthias Wolf <
> matthias.wolf at oist.jp>
> Sent: Friday, April 2, 2021 11:27 AM
> To: Krishan Pandey; 3dem at ncmir.ucsd.edu
> Subject: Re: [3dem] Data storage system
>
> Hi Krishan,
>
> Our 60-drive 760 TB FreeNAS has been running flawless since more than 1
> year. One failed drive was easily hot swapped. If you want capacity, this
> does not cost more  per TB than inferior hardware, yet you get good
> redundancy and performance.
>
> See my old tweet here
> https://urldefense.com/v3/__https://twitter.com/hicryoem/status/1223966559976083458?s=21__;!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCPmSkFojQ$
> <https://urldefense.com/v3/__https:/twitter.com/hicryoem/status/1223966559976083458?s=21__;!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCPmSkFojQ$>
> <
> https://urldefense.com/v3/__https://nam10.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Ftwitter.com*2Fhicryoem*2Fstatus*2F1223966559976083458*3Fs*3D21__*3B!!Mih3wA!TpItSV7E6sE1qSO0JgCwUfuLYuNhW92PMSZUaIF70zdj2z4BE_zg2izktUsKL5clRQ*24&data=04*7C01*7CChen.Xu*40umassmed.edu*7Ca3ecd1c220144d5c148208d8f5ebdb95*7Cee9155fe2da34378a6c44405faf57b2e*7C0*7C1*7C637529740649781691*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C2000&sdata=kHkXkbLqxxMR*2F*2FgoMCQQQxjSAvsIXrzIxrf93xWYXrE*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCOaPADKzw$
> <https://urldefense.com/v3/__https:/nam10.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Ftwitter.com*2Fhicryoem*2Fstatus*2F1223966559976083458*3Fs*3D21__*3B!!Mih3wA!TpItSV7E6sE1qSO0JgCwUfuLYuNhW92PMSZUaIF70zdj2z4BE_zg2izktUsKL5clRQ*24&data=04*7C01*7CChen.Xu*40umassmed.edu*7Ca3ecd1c220144d5c148208d8f5ebdb95*7Cee9155fe2da34378a6c44405faf57b2e*7C0*7C1*7C637529740649781691*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C2000&sdata=kHkXkbLqxxMR*2F*2FgoMCQQQxjSAvsIXrzIxrf93xWYXrE*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCOaPADKzw$>
> >
>
>    Matthias
>
> ________________________________
> From: 3dem <3dem-bounces at ncmir.ucsd.edu> on behalf of Krishan Pandey <
> krishan.pandey at health.slu.edu>
> Sent: Friday, April 2, 2021 03:59
> To: 3dem at ncmir.ucsd.edu
> Subject: [3dem] Data storage system
>
> Hello,
>
> I am requesting suggestions and cost estimates about off the shelf data
> storage systems to store raw cryo-EM movies and processed data for our lab.
> Our initial target is 150-200 TB with options to expand it in future.
> We don't have much local IT support for Linux based systems, that's why I
> am asking for an off-the shelf system which should be easier to install and
> manage.
>
> Thank you
> best regards
>
> Krishan Pandey
>
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
>
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20210403/23ca9e81/attachment-0001.html>