<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:310410899;
mso-list-type:hybrid;
mso-list-template-ids:-146406290 916458754 67895299 67895301 67895297 67895299 67895301 67895297 67895299 67895301;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ascii-font-family:Aptos;
mso-fareast-font-family:Aptos;
mso-hansi-font-family:Aptos;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:420490391;
mso-list-type:hybrid;
mso-list-template-ids:1194201780 67895321 67895321 67895323 67895311 67895321 67895323 67895311 67895321 67895323;}
@list l1:level1
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
</head>
<body lang="FR" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">We are relatively new in SPA, also keeping drives on shelves for the moment… I’m considering the following strategy:<o:p></o:p></span></p>
<ol style="margin-top:0cm" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Movies corresponding to published structures: ship to EMPIAR, delete from local disks unless we intend to reprocess
data in the short term;<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Output of data processing pipeline (only the jobs that lead to the good maps, or relevant observations): keep
in the drive on the shelf – as even after job selection, this can take a few Tb per dataset – mirror two disks for safety;<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Movies corresponding to unpublished, but important structures: keep in the drive on the shelf – mirror two disks
for safety;<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Movies corresponding to bad collections, or bad samples, we could never get anything useful from them: delete.<o:p></o:p></span></li></ol>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">My doubts are:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">For point 1, how hard it is to get everything uploaded? Any reasons not to do it?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">On point 2, I usually keep maps, extracted particles and aligned movies, but maybe I only need to keep particle locations and a detailed description of the pipeline
beside the maps? Then it would only take a few Mb, and I could place them in our eLabFTW. The major problem here is that selecting the jobs already takes a while, and there is no reliable way to do it automatically.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">On point 4, I’m always afraid of deleting something that some fancy new software will be able to process… we often work with flexible proteins that are really reluctant
to process to high resolution, despite good contrast sometimes. But at some point one needs to take decisions… in the end, I have to admit that I’ve only deleted one dataset, with a cold sweat running over my spine.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Please let me know of your opinions on that.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Considering the long-term storage, I’ve been told that:<o:p></o:p></span></p>
<ol style="margin-top:0cm" start="1" type="a">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo2"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">The famous tape system has a lot of logistical drawbacks: they often update software and hardware, and the old
tapes need to be converted to new formats, periodically (very time consuming, and it gets expensive if you need to replace equipment) – places that have this kind of resource usually have a dedicated crew;<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo2"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Transfer to tape is often prone to errors, and nobody is checking byte per byte if the copy went fine;<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo2"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Hard drives fail if they are used too much, and also if they are not used at all. So the best would be to plug
them every now and them, a bit like the old car in the garage. Not very time consuming, but one needs to think of doing this, and keeping track of which disk was plugged when (mental load, who hasn’t enough?) – and still, this doesn’t guarantee that they will
last 10 years;<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo2"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Some people are praying for the development of data storage in DNA, but I expect the copy to be extremely slow…<o:p></o:p></span></li></ol>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">This is a very serious issue, and I only see it getting worse as we accumulate more and more data. I know I sound pessimistic, but I wish a great day to everyone.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">---------------------------------------------------<br>
Carlos KIKUTI, PhD<br>
UMR144 - CNRS - Institut Curie<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Pavillon Trouillet Rossignol <br>
26 Rue d’Ulm - 75005 Paris, France<br>
<a href="mailto:carlos.kikuti@curie.fr" title="mailto:carlos.kikuti@curie.fr"><span style="color:#0563C1">carlos.kikuti@curie.fr</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
<div id="mail-editor-reference-message-container">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><br>
Message: 3<br>
Date: Thu, 15 Feb 2024 03:57:05 +0000<br>
From: "Ludtke, Steven J." <sludtke@bcm.edu><br>
To: Jobichen <jobichenc@yahoo.com><br>
Cc: 3DEM Mailing List <3dem@ncmir.ucsd.edu><br>
Subject: Re: [3dem] Advice on storage server<br>
Message-ID: <26C5131B-164C-4C2F-A578-87D6C5797849@bcm.edu><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
I should add that for long term backup, the most typical strategy is the convenient but unsafe "drives on a shelf".
</span><span style="font-size:11.0pt">That would be a one-time purchase of ~$2k, but the chances that all of the drives work and you can fully recover the data in 5 or 10 years may be a little marginal. Worth noting also that portable USB drives as opposed
to drives designed to be internal drives in a PC have massively lower reliability ratings in general. Also note that SSD's lose data over time if they aren't plugged in to a power source periodically for a "refresh".<br>
<br>
---<br>
Steven Ludtke, Ph.D. <sludtke@bcm.edu> Baylor College of Medicine<br>
Charles C. Bell Jr., Professor of Structural Biology Dept. of Biochemistry<br>
Deputy Director, Advanced Technology Cores and Molecular Pharmacology<br>
Academic Director, CryoEM Core<br>
Co-Director CIBR Center<br>
<br>
<br>
On Feb 14, 2024, at 8:06?PM, Ludtke, Steven J. <sludtke@bcm.edu> wrote:<br>
<br>
If you don't expect to need to access it again, ie - purely an emergency backup, Amazon Glacier is a cost effective solution, as long as you have $ to continue paying for it. 100 TB of deep archive Glacier storage would run about $1200/year (+ additional cost
if you need to retrieve it).<br>
<br>
If you are storing it for possible additional processing, then you want the storage to be "close" in data transfer terms to the processing power. ie - if you are processing in the cloud, then storing the data in the cloud makes sense. Clearly you would not
want to process the data directly from cloud storage. Keep in mind the relative speeds of transfer for different devices/transfer methods:<br>
<br>
M.2 SSD -> 2-4 GB/s<br>
8 drive RAID array with spinning platters directly on the machine -> ~1 GB/s<br>
SATA SSD -> 0.6 GB/s<br>
single spinning platter on machine -> 0.15 GB/s<br>
gigabit network remote access -> 0.1 GB/s<br>
less than gigabit remote access (cloud at typical institutions) -> <0.1 GB/s<br>
<br>
For size comparison, a 4k x 4k x 1k tomogram at 8 bits is 16 GB, so opening that from an M.2 SSD might take 4-8 seconds, whereas opening the same file over a gigabit NAS would take almost 3 minutes.<br>
<br>
Personally, I have a 12 bay Synology NAS box with a 10 Gb network card in it under my desk. With 16 TB drives and RAID6 this gives about 150 TB of usable storage space, which you can access at ~1 GB/s. Cost ~$5000, with an expected drive life of ~5 years, ie
- expect you will have to periodically replace bad drives occasionally after the first few years.<br>
<br>
It's worth noting here that at $5000, with an expected life of ~5 years before you start having to pay for more drives, this is $1000/year and gives high speed access, compared to the $1200/year for deep Glacier storage above. However, the Glacier storage has
much better reliability than a single RAID6 array with no additional backup.<br>
<br>
Anyway, some food for thought :^)<br>
<br>
---<br>
Steven Ludtke, Ph.D. <sludtke@bcm.edu> Baylor College of Medicine<br>
Charles C. Bell Jr., Professor of Structural Biology Dept. of Biochemistry<br>
Deputy Director, Advanced Technology Cores and Molecular Pharmacology<br>
Academic Director, CryoEM Core<br>
Co-Director CIBR Center<br>
<br>
<br>
On Feb 14, 2024, at 6:14?PM, Jobichen <jobichenc@yahoo.com> wrote:<br>
<br>
Dear All,<br>
We are looking for some suggestions on storing the raw datasets/movies. What will be best option for storing around 100TB of movies/processed data.<br>
What will be pros/cons of having own storage server vs cloud storage options.<br>
Thank you for your time.<br>
Jobi<br>
<br>
<br>
<br>
_______________________________________________<br>
3dem mailing list<br>
3dem@ncmir.ucsd.edu<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=-fMPusn_TT7DVAUweasDDQG4kEyzhEAyjRtShGQYPmx9cRVoBtVsmUUqEMrMPs9w&s=LBeNcMDu7IJx1_Y7BTp2_JFhuug6w0oVJobkLUozOFc&e=">https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=-fMPusn_TT7DVAUweasDDQG4kEyzhEAyjRtShGQYPmx9cRVoBtVsmUUqEMrMPs9w&s=LBeNcMDu7IJx1_Y7BTp2_JFhuug6w0oVJobkLUozOFc&e=</a><br>
<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20240215/7302abc1/attachment.html">http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20240215/7302abc1/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
3dem mailing list<br>
3dem@ncmir.ucsd.edu<br>
<a href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><br>
<br>
<br>
------------------------------<br>
<br>
End of 3dem Digest, Vol 198, Issue 21<br>
*************************************<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</body>
</html>