[3dem] Data storage system

Sat Apr 3 09:21:35 PDT 2021

Hi Charles,

In our case, as facility and a research lab we need a decent capacity for
data storage. We have a FreeNAS system with a head unit and two expansion
shelves, so three chases with > 100+ drives. There are a 10GbE switch with
SFP+ and an 1GbE RJ45 switch.

Our FreeNAS is configured as mirrored running ZFS as default. And Steve is
right - it is not worry-free. In the first five years, we had to hot-swap
5-6 drives. We had unexpected power outrage twice (no UPS) and the system
came up fine without any data loss. However, when the system booted up and
one of the shelves not seen, it was indeed a scary experience. But so far in
five years, we haven't had any data loss. I like ZFS a lot in that sense.
Writing to FreeNAS storage is very fast, almost saturating the bandwidth,
but reading is slightly slower.

For facility user data, we store for them about 100 days before purging out.
Usually, the data is also transferred to user as soon as collection session is
started. Our own data is stored for long term.

Workstations all have SFP+ (we like Chelsio cards) connecting to the fiber switch.
I think for small storage unit, it can be directly connected to a single workstation
without needing a switch.

Best regards,
-Chen

________________________________________
From: Charles Bayly-Jones <charles.bayly-jones at monash.edu>
Sent: Friday, April 2, 2021 8:12 PM
To: Xu, Chen
Cc: Matthias Wolf; Krishan Pandey; 3dem at ncmir.ucsd.edu
Subject: Re: [3dem] Data storage system

Hi all.

Our lab is also currently making similar considerations for a moderately large server. So this email chain has been quite serendipitous.

I'm wondering whether those (Chen, Steve, Matthias, Tim) with external servers (45drives, trueNAS, synology, etc) might comment on their network setup? Did you opt for a special network switch? Has this been necessary or can these servers directly host the workstations? What kind of local network infrastructure do you use? Connections, SFP+/RJ45/some fancy optical thing?

I note Steve has said, in the small 12-drive synologyNAS that a 10Gbe (RJ45?) card has been sufficient for ~950/600 MB/s IO. I recall the synology can be networked directly to workstations without the need for a switch, if I'm not mistaken. So what is people's experience with the other larger servers?

Perhaps you might also comment on your typical use case? E.g. is your server for long term storage of processed or raw data? Or is it rather a scratch space for analysis with read & write?

Very best,
Charles
_________________

Charles Bayly-Jones
BSc(ScSchProg)(Hons)

Monash Biomedicine Discovery Institute & ARC CoE in advanced molecular imaging
Department of Biochemistry and Molecular Biology
Rm 218, Level 2, Building 77
23 Innovation Walk
Clayton VIC 3800
Australia
[Image result for monash university logo]

On Sat, 3 Apr 2021 at 04:23, Xu, Chen <Chen.Xu at umassmed.edu<mailto:Chen.Xu at umassmed.edu>> wrote:
We also have very positive experience with FreeNAS/TruNAS system. Here is the shop for their
on-the-shelf units.

https://urldefense.com/v3/__https://www.amazon.com/stores/iXsystems/page/C88EFDE3-3E4C-4951-860E-0E8A8BD91BF9?ref_=ast_bln__;!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCOnSJxj9w$<https://urldefense.com/v3/__https://nam10.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fwww.amazon.com*2Fstores*2FiXsystems*2Fpage*2FC88EFDE3-3E4C-4951-860E-0E8A8BD91BF9*3Fref_*3Dast_bln__*3B!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCOnSJxj9w*24&data=04*7C01*7CChen.Xu*40umassmed.edu*7Cd343c3e044014ec146a108d8f6352a35*7Cee9155fe2da34378a6c44405faf57b2e*7C0*7C0*7C637530055509932883*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=xGdB*2FdPlKm*2FnDlBo6nVyrNqCNmUsTkOq8eeqLLa4Xu4*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!Mih3wA!T7oAvx3lrqO6gZdsEwTFFxNFaBUIlixCdVZ_jI4bU3NXloo_5zsfU6C6ml8D9Q9Yjg$ >

-Chen

________________________________________
From: 3dem <3dem-bounces at ncmir.ucsd.edu<mailto:3dem-bounces at ncmir.ucsd.edu>> on behalf of Matthias Wolf <matthias.wolf at oist.jp<mailto:matthias.wolf at oist.jp>>
Sent: Friday, April 2, 2021 11:27 AM
To: Krishan Pandey; 3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
Subject: Re: [3dem] Data storage system

Hi Krishan,

Our 60-drive 760 TB FreeNAS has been running flawless since more than 1 year. One failed drive was easily hot swapped. If you want capacity, this does not cost more  per TB than inferior hardware, yet you get good redundancy and performance.

See my old tweet here https://urldefense.com/v3/__https://twitter.com/hicryoem/status/1223966559976083458?s=21__;!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCPmSkFojQ$<https://urldefense.com/v3/__https://nam10.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Ftwitter.com*2Fhicryoem*2Fstatus*2F1223966559976083458*3Fs*3D21__*3B!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCPmSkFojQ*24&data=04*7C01*7CChen.Xu*40umassmed.edu*7Cd343c3e044014ec146a108d8f6352a35*7Cee9155fe2da34378a6c44405faf57b2e*7C0*7C0*7C637530055509932883*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=or4AzmaH3kRTzvJxFCtXFdpCjoHTxBqKUROWxYVfFco*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!Mih3wA!T7oAvx3lrqO6gZdsEwTFFxNFaBUIlixCdVZ_jI4bU3NXloo_5zsfU6C6ml9YOVVMaw$ > <https://urldefense.com/v3/__https://nam10.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Ftwitter.com*2Fhicryoem*2Fstatus*2F1223966559976083458*3Fs*3D21__*3B!!Mih3wA!TpItSV7E6sE1qSO0JgCwUfuLYuNhW92PMSZUaIF70zdj2z4BE_zg2izktUsKL5clRQ*24&data=04*7C01*7CChen.Xu*40umassmed.edu*7Ca3ecd1c220144d5c148208d8f5ebdb95*7Cee9155fe2da34378a6c44405faf57b2e*7C0*7C1*7C637529740649781691*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C2000&sdata=kHkXkbLqxxMR*2F*2FgoMCQQQxjSAvsIXrzIxrf93xWYXrE*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCOaPADKzw$<https://urldefense.com/v3/__https://nam10.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fnam10.safelinks.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Ftwitter.com*2Fhicryoem*2Fstatus*2F1223966559976083458*3Fs*3D21__*3B!!Mih3wA!TpItSV7E6sE1qSO0JgCwUfuLYuNhW92PMSZUaIF70zdj2z4BE_zg2izktUsKL5clRQ*24*26data*3D04*7C01*7CChen.Xu*40umassmed.edu*7Ca3ecd1c220144d5c148208d8f5ebdb95*7Cee9155fe2da34378a6c44405faf57b2e*7C0*7C1*7C637529740649781691*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C2000*26sdata*3DkHkXkbLqxxMR*2F*2FgoMCQQQxjSAvsIXrzIxrf93xWYXrE*3D*26reserved*3D0__*3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!Mih3wA!XMzGwejDpzMf8LVPKLKH7ywovtHBrbzH4RFa7s3_HCHKfln9EJAtGF_JwCOaPADKzw*24&data=04*7C01*7CChen.Xu*40umassmed.edu*7Cd343c3e044014ec146a108d8f6352a35*7Cee9155fe2da34378a6c44405faf57b2e*7C0*7C0*7C637530055509942876*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=JDHHDuE7pqL6pDkj7xuugs8Q7*2FsePjbRpvAxE7n2kTk*3D&reserved=0__;JSUlJSUlJSUlJSUqKioqKioqKioqKioqKiolJSoqKioqKioqKioqKiUlKioqJSUlJSUlJSUlJSUlJSUlJSUl!!Mih3wA!T7oAvx3lrqO6gZdsEwTFFxNFaBUIlixCdVZ_jI4bU3NXloo_5zsfU6C6ml-O_lfHFA$ > >

   Matthias

________________________________
From: 3dem <3dem-bounces at ncmir.ucsd.edu<mailto:3dem-bounces at ncmir.ucsd.edu>> on behalf of Krishan Pandey <krishan.pandey at health.slu.edu<mailto:krishan.pandey at health.slu.edu>>
Sent: Friday, April 2, 2021 03:59
To: 3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
Subject: [3dem] Data storage system

Hello,

I am requesting suggestions and cost estimates about off the shelf data storage systems to store raw cryo-EM movies and processed data for our lab. Our initial target is 150-200 TB with options to expand it in future.
We don't have much local IT support for Linux based systems, that's why I am asking for an off-the shelf system which should be easier to install and manage.

Thank you
best regards

Krishan Pandey

_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem<https://urldefense.com/v3/__https://nam10.safelinks.protection.outlook.com/?url=https*3A*2F*2Fmail.ncmir.ucsd.edu*2Fmailman*2Flistinfo*2F3dem&data=04*7C01*7CChen.Xu*40umassmed.edu*7Cd343c3e044014ec146a108d8f6352a35*7Cee9155fe2da34378a6c44405faf57b2e*7C0*7C0*7C637530055509942876*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=HmJlP2rLOwwWvoB9q7p02zsBHgv41FIWLlZNHQf*2BdKg*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSU!!Mih3wA!T7oAvx3lrqO6gZdsEwTFFxNFaBUIlixCdVZ_jI4bU3NXloo_5zsfU6C6ml8NT8HDhA$ >