[3dem] Data storage system

[Craig Yoshioka]

We’ve bought the 60 drive SuperMicro chassis, I like them a lot.  Good redundancy and build quality- remote management and KVM included.  Currently have them configured as 6x10 RAIDZ2 ZFS pool, which provides 480TB of useable disk space and decent performance per server.  A very fast Intel optane SSD boosts synchronous write performance, +768GB of RAM and 36 cpu cores, for around ~40K each (prices ~2-3 years old).  It can hit ~4GB/sec in linear read/writes and do about 15-20K IOPS (not great, but not bad).  I got it with 4 10GbE ethernet ports but have only used two so far aggregated to the switch.  In the last three years we haven’t yet had to replace any drives.

For off-site cloud backup I think Wasabi seems pretty reasonable at $70/TB/yr for offsite, they don’t have ingress/egress taxes.  This will be way less IT support needed on your part, and likely better redundancy/availability story.

Currently looking into Quobyte and WekaIO to provide an all encompassing data collection and data processing HPC storage solution, but it’s expensive.  The dream is to not have to compromise between storage capacity, throughput and high IOPS.  Both systems serve primarily from a cluster of all SSD servers, but can transparently tier data to a large object store.  If anyone has experiences with these I’d love to hear.

For large archival storage, I like the combination of performance, space and cost of using an on-premise object store with disks.  Tape is still cheaper at scale, but seems to have pretty steep buy-in costs and is less convenient.






On Apr 2, 2021, at 8:27 AM, Matthias Wolf <matthias.wolf at oist.jp<mailto:matthias.wolf at oist.jp>> wrote:

Hi Krishan,

Our 60-drive 760 TB FreeNAS has been running flawless since more than 1 year. One failed drive was easily hot swapped. If you want capacity, this does not cost more  per TB than inferior hardware, yet you get good redundancy and performance.

See my old tweet here https://urldefense.com/v3/__https://twitter.com/hicryoem/status/1223966559976083458?s=21__;!!Mih3wA!VMGm1mvz3dci1eXKkSPJ6P_027UTaWEnqsL2mQA-rQ3FsrMSI4thhZI4Fxkd8kI-kw$ <https://urldefense.com/v3/__https://twitter.com/hicryoem/status/1223966559976083458?s=21__;!!Mih3wA!TpItSV7E6sE1qSO0JgCwUfuLYuNhW92PMSZUaIF70zdj2z4BE_zg2izktUsKL5clRQ$>

   Matthias

________________________________
From: 3dem <3dem-bounces at ncmir.ucsd.edu<mailto:3dem-bounces at ncmir.ucsd.edu>> on behalf of Krishan Pandey <krishan.pandey at health.slu.edu<mailto:krishan.pandey at health.slu.edu>>
Sent: Friday, April 2, 2021 03:59
To: 3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
Subject: [3dem] Data storage system

Hello,

I am requesting suggestions and cost estimates about off the shelf data storage systems to store raw cryo-EM movies and processed data for our lab. Our initial target is 150-200 TB with options to expand it in future.
We don't have much local IT support for Linux based systems, that's why I am asking for an off-the shelf system which should be easier to install and manage.

Thank you
best regards

Krishan Pandey

_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20210403/6de24491/attachment.html>