<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
In addition to the university HPC cluster, we use these in my lab at OIST for our own cryo-EM storage needs :<br>
<br>
</div>
<ul data-editing-info="{"orderedStyleType":1,"unorderedStyleType":1}" style="margin-block: 0px; list-style-type: disc;">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>Archival</b>: Qualstar Q24 LTO8 24-tape library, single drive (~100$/ 12TB uncompressed
cartridge, lifetime >30 yrs), connected to Supermicro storage server by 6 Gbps SAS PCIe card, ~20 TB/day.</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><a href="https://urldefense.com/v3/__https://www.backupworks.com/qualstar-Q24-LTO-8-Tape-libraries.aspx__;!!Mih3wA!AqQRGpcslECVpaex9vo4dVX5T8zAd44Z8fmQIknMvTitF-MbRHg6WMgKvV2L45_eUX2EqUDIFeYVPLCuJqtwD0xJHco$" id="OWA4af7b643-335e-426c-ef74-48612ad48c0a" class="OWAAutoLink" data-loopstyle="linkonly">https://www.backupworks.com/qualstar-Q24-LTO-8-Tape-libraries.aspx</a></span></div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>Near-line storage</b>: 800 TB, 60x 16TB HDD) SuperStorage 2x Xeon, 512 GB, 4x 10Gb
ethernet, TrueNAS, Archiware P5, mounted read-only by NFS on all workstations.</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><a href="https://urldefense.com/v3/__https://www.truenas.com/truenas-core/__;!!Mih3wA!AqQRGpcslECVpaex9vo4dVX5T8zAd44Z8fmQIknMvTitF-MbRHg6WMgKvV2L45_eUX2EqUDIFeYVPLCuJqtwf_G5HhQ$" id="OWAcd341203-6cd0-5b34-2fcf-9cc689de0c80" class="OWAAutoLink" data-loopstyle="linkonly">https://www.truenas.com/truenas-core/</a><br>
<a href="https://urldefense.com/v3/__https://www.archiware.com/products/p5-archive__;!!Mih3wA!AqQRGpcslECVpaex9vo4dVX5T8zAd44Z8fmQIknMvTitF-MbRHg6WMgKvV2L45_eUX2EqUDIFeYVPLCuJqtwu0b4Bos$" id="OWA7a6413d5-f88f-8ecc-0b7f-d73cf2db71c6" class="OWAAutoLink" data-loopstyle="linkonly">https://www.archiware.com/products/p5-archivehttps://www.archiware.com/products/p5-archive</a><br>
<a href="https://urldefense.com/v3/__https://www.supermicro.com/en/products/system/storage/2u/ssg-540p-e1ctr60h__;!!Mih3wA!AqQRGpcslECVpaex9vo4dVX5T8zAd44Z8fmQIknMvTitF-MbRHg6WMgKvV2L45_eUX2EqUDIFeYVPLCuJqtwhckoT10$" id="OWAf994b567-54dd-ab12-fe03-607f2fc3ef05" class="OWAAutoLink" data-loopstyle="linkonly">https://www.supermicro.com/en/products/system/storage/2u/ssg-540p-e1ctr60h</a></span></div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>Online storage</b>: DDN file server (was too expensive!): 700 TB (120 HDD x 4/6 TB),
40Gb infiniband connected to HPC cluster, read-write.</span></div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>Flash storage</b> (e.g. local GPU data cache): 8x 4TB NVMe SSD RAID0 PCIe4 card (we
have PCIe3 version and a 4-drive PCIe4) for processing to keep those GPUs busy.</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><a href="https://urldefense.com/v3/__https://www.highpoint-tech.com/nvme-individual/ssd7540__;!!Mih3wA!AqQRGpcslECVpaex9vo4dVX5T8zAd44Z8fmQIknMvTitF-MbRHg6WMgKvV2L45_eUX2EqUDIFeYVPLCuJqtwlFNIDis$" id="OWAca48ca2c-720b-7b2f-e178-df3f4f8bebba" class="OWAAutoLink" data-loopstyle="linkonly">https://www.highpoint-tech.com/nvme-individual/ssd7540</a></span></div>
</li></ul>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
I move data in stages from online > near line > tape, as they age.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
The supermicro server running TrueNAS has been very reliable. 3 drives failed over the last 4 years. Easy to hot-swap after it sends you an email. TrueNAS is highly recommended. Archiware is great for archival. It keeps a catalog of everything. Once the tape
library is full, I move the barcoded tapes to a shelf in temperature-controlled long-term storage room. If a tape is not found in the library, archiware will tell you the barcode and you simply need to load it.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Matthias</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> 3dem <3dem-bounces@ncmir.ucsd.edu> on behalf of Ludtke, Steven J. <sludtke@bcm.edu><br>
<b>Sent:</b> Thursday, February 15, 2024 12:57 PM<br>
<b>To:</b> Jobichen <jobichenc@yahoo.com><br>
<b>Cc:</b> 3DEM Mailing List <3dem@ncmir.ucsd.edu><br>
<b>Subject:</b> Re: [3dem] Advice on storage server</span>
<div> </div>
</div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">I should add that for long term backup, the most typical strategy is the convenient but unsafe "drives on a
shelf". That would be a one-time purchase of ~$2k, but the chances that all of the drives work and you can fully recover the data in 5 or 10 years may be a little marginal. Worth noting also that portable USB drives as opposed to drives designed to be internal
drives in a PC have massively lower reliability ratings in general. Also note that SSD's lose data over time if they aren't plugged in to a power source periodically for a "refresh". </span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr; text-align: left; text-indent: 0px;"><span style="font-family: "Courier New"; font-size: 14px; color: rgb(0, 0, 0);">---</span></div>
<div style="direction: ltr; text-align: left; text-indent: 0px;"><span style="font-family: "Courier New"; font-size: 14px; color: rgb(0, 0, 0);">Steven Ludtke, Ph.D. <sludtke@bcm.edu> Baylor College of Medicine<br>
Charles C. Bell Jr., Professor of Structural Biology Dept. of Biochemistry <br>
Deputy Director, Advanced Technology Cores and Molecular Pharmacology<br>
Academic Director, CryoEM Core<br>
Co-Director CIBR Center<br>
</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></div>
<blockquote>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">On Feb 14, 2024, at 8:06 PM, Ludtke, Steven J. <sludtke@bcm.edu> wrote:</span></div>
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">If you don't expect to need to access it again, ie - purely an emergency backup, Amazon Glacier is a cost effective
solution, as long as you have $ to continue paying for it. 100 TB of deep archive Glacier storage would run about $1200/year (+ additional cost if you need to retrieve it).</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">If you are storing it for possible additional processing, then you want the storage to be "close" in data transfer
terms to the processing power. ie - if you are processing in the cloud, then storing the data in the cloud makes sense. Clearly you would not want to process the data directly from cloud storage. Keep in mind the relative speeds of transfer for different devices/transfer
methods:</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">M.2 SSD -> 2-4 GB/s</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">8 drive RAID array with spinning platters directly on the machine -> ~1 GB/s</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">SATA SSD -> 0.6 GB/s</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">single spinning platter on machine -> 0.15 GB/s</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">gigabit network remote access -> 0.1 GB/s</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">less than gigabit remote access (cloud at typical institutions) -> <0.1 GB/s</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">For size comparison, a 4k x 4k x 1k tomogram at 8 bits is 16 GB, so opening that from an M.2 SSD might take
4-8 seconds, whereas opening the same file over a gigabit NAS would take almost 3 minutes.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">Personally, I have a 12 bay Synology NAS box with a 10 Gb network card in it under my desk. With 16 TB drives
and RAID6 this gives about 150 TB of usable storage space, which you can access at ~1 GB/s. Cost ~$5000, with an expected drive life of ~5 years, ie - expect you will have to periodically replace bad drives occasionally after the first few years. </span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">It's worth noting here that at $5000, with an expected life of ~5 years before you start having to pay for
more drives, this is $1000/year and gives high speed access, compared to the $1200/year for deep Glacier storage above. However, the Glacier storage has much better reliability than a single RAID6 array with no additional backup.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">Anyway, some food for thought :^)</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr; text-align: left; text-indent: 0px;"><span style="font-family: "Courier New"; font-size: 14px; color: rgb(0, 0, 0);">---</span></div>
<div style="direction: ltr; text-align: left; text-indent: 0px;"><span style="font-family: "Courier New"; font-size: 14px; color: rgb(0, 0, 0);">Steven Ludtke, Ph.D. <sludtke@bcm.edu> Baylor College of Medicine<br>
Charles C. Bell Jr., Professor of Structural Biology Dept. of Biochemistry <br>
Deputy Director, Advanced Technology Cores and Molecular Pharmacology<br>
Academic Director, CryoEM Core<br>
Co-Director CIBR Center<br>
</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></div>
<blockquote>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">On Feb 14, 2024, at 6:14 PM, Jobichen <jobichenc@yahoo.com> wrote:</span></div>
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span>
<div style="direction: ltr;"><span style="font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; color: rgb(0, 0, 0);">Dear All,</span></div>
<div style="direction: ltr;"><span style="font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; color: rgb(0, 0, 0);">We are looking for some suggestions on storing the raw datasets/movies. What will be best option for storing around
100TB of movies/processed data.</span></div>
<div style="direction: ltr;"><span style="font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; color: rgb(0, 0, 0);">What will be pros/cons of having own storage server vs cloud storage options.</span></div>
<div style="direction: ltr;"><span style="font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; color: rgb(0, 0, 0);">Thank you for your time.</span></div>
<div style="direction: ltr;"><span style="font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; color: rgb(0, 0, 0);">Jobi</span></div>
<div style="direction: ltr;"><span style="font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; color: rgb(0, 0, 0);"><br>
</span></div>
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">_______________________________________________<br>
3dem mailing list<br>
3dem@ncmir.ucsd.edu<br>
https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=-fMPusn_TT7DVAUweasDDQG4kEyzhEAyjRtShGQYPmx9cRVoBtVsmUUqEMrMPs9w&s=LBeNcMDu7IJx1_Y7BTp2_JFhuug6w0oVJobkLUozOFc&e=<br>
</span></blockquote>
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span></blockquote>
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br>
</span>
</body>
</html>