<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
We’ve bought the 60 drive SuperMicro chassis, I like them a lot. Good redundancy and build quality- remote management and KVM included. Currently have them configured as 6x10 RAIDZ2 ZFS pool, which provides 480TB of useable disk space and decent performance
per server. A very fast Intel optane SSD boosts synchronous write performance, +768GB of RAM and 36 cpu cores, for around ~40K each (prices ~2-3 years old). It can hit ~4GB/sec in linear read/writes and do about 15-20K IOPS (not great, but not bad). I got
it with 4 10GbE ethernet ports but have only used two so far aggregated to the switch. In the last three years we haven’t yet had to replace any drives.</div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">For off-site cloud backup I think Wasabi seems pretty reasonable at $70/TB/yr for offsite, they don’t have ingress/egress taxes. This will be way less IT support needed on
your part, and likely better redundancy/availability story.</span></div>
<div class=""><br class="">
</div>
<div class="">Currently looking into Quobyte and WekaIO to provide an all encompassing data collection and data processing HPC storage solution, but it’s expensive. The dream is to not have to compromise between storage capacity, throughput and high IOPS.
Both systems serve primarily from a cluster of all SSD servers, but can transparently tier data to a large object store. If anyone has experiences with these I’d love to hear.</div>
<div class=""><br class="">
</div>
<div class="">For large archival storage, I like the combination of performance, space and cost of using an on-premise object store with disks. Tape is still cheaper at scale, but seems to have pretty steep buy-in costs and is less convenient.</div>
<div class=""><br class="">
</div>
<div class=""> </div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Apr 2, 2021, at 8:27 AM, Matthias Wolf <<a href="mailto:matthias.wolf@oist.jp" class="">matthias.wolf@oist.jp</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div class=""></div>
<div class="">
<div dir="ltr" class="">Hi Krishan,</div>
<div dir="ltr" class=""><span style="font-size: inherit;" class=""><br class="">
</span></div>
<div dir="ltr" class=""><span style="font-size: inherit;" class="">Our 60-drive 760 TB FreeNAS has been running flawless since more than 1 year. One failed drive was easily hot swapped. If you want capacity, this does not cost more per TB than inferior hardware,
yet you get good redundancy and performance.</span></div>
<div dir="ltr" class=""><br class="">
</div>
<div dir="ltr" class="">See my old tweet here <a href="https://urldefense.com/v3/__https://twitter.com/hicryoem/status/1223966559976083458?s=21__;!!Mih3wA!TpItSV7E6sE1qSO0JgCwUfuLYuNhW92PMSZUaIF70zdj2z4BE_zg2izktUsKL5clRQ$" class="">https://twitter.com/hicryoem/status/1223966559976083458?s=21</a></div>
<div dir="ltr" class=""><br class="">
</div>
<div dir="ltr" class=""> Matthias</div>
<div class=""><br class="">
</div>
<div class="ms-outlook-ios-signature"></div>
</div>
<div id="id-9793da4a-196c-4b5f-907f-0af04eeba15f" class="ms-outlook-mobile-reference-message">
<hr tabindex="-1" style="display: inline-block; width: 1164.234375px; font-family: -webkit-standard; font-size: 19.44pt;" class="">
<div id="divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>3dem <<a href="mailto:3dem-bounces@ncmir.ucsd.edu" class="">3dem-bounces@ncmir.ucsd.edu</a>> on behalf of Krishan
Pandey <<a href="mailto:krishan.pandey@health.slu.edu" class="">krishan.pandey@health.slu.edu</a>><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Friday, April 2, 2021 03:59<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span><a href="mailto:3dem@ncmir.ucsd.edu" class="">3dem@ncmir.ucsd.edu</a><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>[3dem] Data storage system
<div class=""> </div>
</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Hello,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
I am requesting suggestions and cost estimates about off the shelf data storage systems to store raw <span style="background-color: rgb(255, 255, 255); display: inline !important;" class="">cryo-EM<span class="Apple-converted-space"> </span></span>movies and
processed data for our lab. Our initial target is 150-200 TB with options to expand it in future. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
We don't have much local IT support for Linux based systems, that's why I am asking for an off-the shelf system which should be easier to install and manage. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Thank you</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
best regards</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Krishan Pandey</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
</div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">3dem
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="mailto:3dem@ncmir.ucsd.edu" class="">3dem@ncmir.ucsd.edu</a></span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem" class="">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a></span></div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>