<html><head></head><body><div class="ydp40e44cb1yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:13px;"><div></div>
<div dir="ltr" data-setdir="false">Dear All,</div><div dir="ltr" data-setdir="false">Thank you for your suggestions. Really appreciate the quick reply to this question.</div><div dir="ltr" data-setdir="false">Jobichen</div><div dir="ltr" data-setdir="false"><br></div><div><br></div>
</div><div id="yahoo_quoted_8426797787" class="yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
On Thursday, February 15, 2024 at 07:45:59 PM GMT+11, Takanori Nakane <tnakane.protein@osaka-u.ac.jp> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div dir="ltr">Hi,<br clear="none"><br clear="none"> > 1. Movies corresponding to published structures: ship to EMPIAR, delete<br clear="none"> > from local disks unless we intend to reprocess data in the short <br clear="none">term;<br clear="none"> > For point 1, how hard it is to get everything uploaded?<br clear="none"><br clear="none">This depends on your network connection.<br clear="none"><br clear="none">Around 2020, upload from MRC-LMB to EBI was very fast,<br clear="none">reaching 250 MB/sec (not megabits, megabytes!).<br clear="none">Upload from Japan to EBI is not so fast, but we can transfer<br clear="none">one entry (say 5 TB) in two or three days.<br clear="none">For those in Japan with poor internet connection, you can send your HDDs<br clear="none">to us (PDBj/EMPIAR Japan) and we will upload files on behalf of you<br clear="none"><a shape="rect" href="https://urldefense.com/v3/__https://empiar.pdbj.org/en/deposition/__;!!Mih3wA!EmdFPHK0U3JothuSkvYDWh2poft3QGH1ZsOXnFQekU_3TUrblFl6IGaHyvVUXmLVmW5aCWhxUM3S2Uc1_O0VN8qC-jO5thwQVBY$" target="_blank">https://urldefense.com/v3/__https://empiar.pdbj.org/en/deposition/__;!!Mih3wA!EmdFPHK0U3JothuSkvYDWh2poft3QGH1ZsOXnFQekU_3TUrblFl6IGaHyvVUXmLVmW5aCWhxUM3S2Uc1_O0VN8qC-jO5thwQVBY$</a> .<br clear="none">I wish that EBI-EMPIAR offered the same mail-in deposition<br clear="none">service but they said this is impossible due to staff shortage and<br clear="none">strict security rules at EBI.<br clear="none"><br clear="none">I wouldn't delete raw data (even after upload to EMPIAR).<br clear="none">Considering the cost of protein expression,<br clear="none">purification etc and the machine time, the cost of storage is<br clear="none">relatively small. I would make two copies<br clear="none">(in physically different locations) of raw data for case 1 and 3.<br clear="none">One copy is on our cluster (but not expensive distributed file system, <br clear="none">just a JBOD RAID) and the other copy is on my shelf.<br clear="none">For 4, I might not make two copies, just one.<br clear="none"><br clear="none">The big drawback of "lots of HDDs on the shelf" is that<br clear="none">one quickly loses track of which disk contains what.<br clear="none">Once the owner leaves the group, this becomes intractable.<br clear="none">Moreover, one has to regularly check each disk is still alive<br clear="none">and make another copy when one fails. This is very tedious.<br clear="none">If you have a central storage, everything is<br clear="none">in one place and one will be notified of failing/failed disk.<br clear="none">Unless many disks fail simultaneously, the data is safe.<br clear="none">(but you might do "rm" by mistake or your computer might be<br clear="none">infected by malware; so I keep one offline disk as a backup)<br clear="none"><br clear="none">Eventually this depends on what one values.<br clear="none">I myself believe that keeping raw data and making it public is<br clear="none">essential for science. Others think it is waste of time and<br clear="none">money and one should focus on new research. Some people even say<br clear="none">it is waste of energy and bad for climate.<br clear="none"><br clear="none">Just my two cents.<br clear="none"><br clear="none">Best regards,<br clear="none"><br clear="none">Takanori Nakane<br clear="none"><br clear="none">On 2024/02/15 17:08, Kikuti Carlos wrote:<br clear="none">> We are relatively new in SPA, also keeping drives on shelves for the <br clear="none">> moment… I’m considering the following strategy:<br clear="none">> <br clear="none">> 1. Movies corresponding to published structures: ship to EMPIAR, delete<br clear="none">> from local disks unless we intend to reprocess data in the short term;<br clear="none">> 2. Output of data processing pipeline (only the jobs that lead to the<br clear="none">> good maps, or relevant observations): keep in the drive on the shelf<br clear="none">> – as even after job selection, this can take a few Tb per dataset –<br clear="none">> mirror two disks for safety;<br clear="none">> 3. Movies corresponding to unpublished, but important structures: keep<br clear="none">> in the drive on the shelf – mirror two disks for safety;<br clear="none">> 4. Movies corresponding to bad collections, or bad samples, we could<br clear="none">> never get anything useful from them: delete.<br clear="none">> <br clear="none">> My doubts are:<br clear="none">> <br clear="none">> For point 1, how hard it is to get everything uploaded? Any reasons not <br clear="none">> to do it?<br clear="none">> <br clear="none">> On point 2, I usually keep maps, extracted particles and aligned movies, <br clear="none">> but maybe I only need to keep particle locations and a detailed <br clear="none">> description of the pipeline beside the maps? Then it would only take a <br clear="none">> few Mb, and I could place them in our eLabFTW. The major problem here is <br clear="none">> that selecting the jobs already takes a while, and there is no reliable <br clear="none">> way to do it automatically.<br clear="none">> <br clear="none">> On point 4, I’m always afraid of deleting something that some fancy new <br clear="none">> software will be able to process… we often work with flexible proteins <br clear="none">> that are really reluctant to process to high resolution, despite good <br clear="none">> contrast sometimes. But at some point one needs to take decisions… in <br clear="none">> the end, I have to admit that I’ve only deleted one dataset, with a cold <br clear="none">> sweat running over my spine.<br clear="none">> <br clear="none">> Please let me know of your opinions on that.<br clear="none">> <br clear="none">> Considering the long-term storage, I’ve been told that:<br clear="none">> <br clear="none">> 1. The famous tape system has a lot of logistical drawbacks: they often<br clear="none">> update software and hardware, and the old tapes need to be converted<br clear="none">> to new formats, periodically (very time consuming, and it gets<br clear="none">> expensive if you need to replace equipment) – places that have this<br clear="none">> kind of resource usually have a dedicated crew;<br clear="none">> 2. Transfer to tape is often prone to errors, and nobody is checking<br clear="none">> byte per byte if the copy went fine;<br clear="none">> 3. Hard drives fail if they are used too much, and also if they are not<br clear="none">> used at all. So the best would be to plug them every now and them, a<br clear="none">> bit like the old car in the garage. Not very time consuming, but one<br clear="none">> needs to think of doing this, and keeping track of which disk was<br clear="none">> plugged when (mental load, who hasn’t enough?) – and still, this<br clear="none">> doesn’t guarantee that they will last 10 years;<br clear="none">> 4. Some people are praying for the development of data storage in DNA,<br clear="none">> but I expect the copy to be extremely slow…<br clear="none">> <br clear="none">> This is a very serious issue, and I only see it getting worse as we <br clear="none">> accumulate more and more data. I know I sound pessimistic, but I wish a <br clear="none">> great day to everyone.<br clear="none">> <br clear="none">> Cheers,<br clear="none">> <br clear="none">> ---------------------------------------------------<br clear="none">> Carlos KIKUTI, PhD<br clear="none">> UMR144 - CNRS - Institut Curie<br clear="none">> <br clear="none">> Pavillon Trouillet Rossignol<br clear="none">> 26 Rue d’Ulm - 75005 Paris, France<br clear="none">> <a shape="rect" ymailto="mailto:carlos.kikuti@curie.fr" href="mailto:carlos.kikuti@curie.fr">carlos.kikuti@curie.fr</a> <mailto:carlos.kikuti@curie.fr><br clear="none">> <br clear="none">> <br clear="none">> Message: 3<br clear="none">> Date: Thu, 15 Feb 2024 03:57:05 +0000<br clear="none">> From: "Ludtke, Steven J." <<a shape="rect" ymailto="mailto:sludtke@bcm.edu" href="mailto:sludtke@bcm.edu">sludtke@bcm.edu</a>><br clear="none">> To: Jobichen <<a shape="rect" ymailto="mailto:jobichenc@yahoo.com" href="mailto:jobichenc@yahoo.com">jobichenc@yahoo.com</a>><br clear="none">> Cc: 3DEM Mailing List <<a shape="rect" ymailto="mailto:3dem@ncmir.ucsd.edu" href="mailto:3dem@ncmir.ucsd.edu">3dem@ncmir.ucsd.edu</a>><br clear="none">> Subject: Re: [3dem] Advice on storage server<br clear="none">> Message-ID: <<a shape="rect" ymailto="mailto:26C5131B-164C-4C2F-A578-87D6C5797849@bcm.edu" href="mailto:26C5131B-164C-4C2F-A578-87D6C5797849@bcm.edu">26C5131B-164C-4C2F-A578-87D6C5797849@bcm.edu</a>><br clear="none">> Content-Type: text/plain; charset="utf-8"<br clear="none">> <br clear="none">> I should add that for long term backup, the most typical strategy is the <br clear="none">> convenient but unsafe "drives on a shelf". That would be a one-time <br clear="none">> purchase of ~$2k, but the chances that all of the drives work and you <br clear="none">> can fully recover the data in 5 or 10 years may be a little marginal. <br clear="none">> Worth noting also that portable USB drives as opposed to drives designed <br clear="none">> to be internal drives in a PC have massively lower reliability ratings <br clear="none">> in general. Also note that SSD's lose data over time if they aren't <br clear="none">> plugged in to a power source periodically for a "refresh".<br clear="none">> <br clear="none">> ---<br clear="none">> Steven Ludtke, Ph.D. <<a shape="rect" ymailto="mailto:sludtke@bcm.edu" href="mailto:sludtke@bcm.edu">sludtke@bcm.edu</a>> Baylor <br clear="none">> College of Medicine<br clear="none">> Charles C. Bell Jr., Professor of Structural Biology Dept. of <br clear="none">> Biochemistry<br clear="none">> Deputy Director, Advanced Technology Cores and <br clear="none">> Molecular Pharmacology<br clear="none">> Academic Director, CryoEM Core<br clear="none">> Co-Director CIBR Center<br clear="none">> <br clear="none">> <br clear="none">> On Feb 14, 2024, at 8:06?PM, Ludtke, Steven J. <<a shape="rect" ymailto="mailto:sludtke@bcm.edu" href="mailto:sludtke@bcm.edu">sludtke@bcm.edu</a>> wrote:<br clear="none">> <br clear="none">> If you don't expect to need to access it again, ie - purely an emergency <br clear="none">> backup, Amazon Glacier is a cost effective solution, as long as you have <br clear="none">> $ to continue paying for it. 100 TB of deep archive Glacier storage <br clear="none">> would run about $1200/year (+ additional cost if you need to retrieve it).<br clear="none">> <br clear="none">> If you are storing it for possible additional processing, then you want <br clear="none">> the storage to be "close" in data transfer terms to the processing <br clear="none">> power. ie - if you are processing in the cloud, then storing the data in <br clear="none">> the cloud makes sense. Clearly you would not want to process the data <br clear="none">> directly from cloud storage. Keep in mind the relative speeds of <br clear="none">> transfer for different devices/transfer methods:<br clear="none">> <br clear="none">> M.2 SSD -> 2-4 GB/s<br clear="none">> 8 drive RAID array with spinning platters directly on the machine -> ~1 GB/s<br clear="none">> SATA SSD -> 0.6 GB/s<br clear="none">> single spinning platter on machine -> 0.15 GB/s<br clear="none">> gigabit network remote access -> 0.1 GB/s<br clear="none">> less than gigabit remote access (cloud at typical institutions) -> <0.1 GB/s<br clear="none">> <br clear="none">> For size comparison, a 4k x 4k x 1k tomogram at 8 bits is 16 GB, so <br clear="none">> opening that from an M.2 SSD might take 4-8 seconds, whereas opening the <br clear="none">> same file over a gigabit NAS would take almost 3 minutes.<br clear="none">> <br clear="none">> Personally, I have a 12 bay Synology NAS box with a 10 Gb network card <br clear="none">> in it under my desk. With 16 TB drives and RAID6 this gives about 150 TB <br clear="none">> of usable storage space, which you can access at ~1 GB/s. Cost ~$5000, <br clear="none">> with an expected drive life of ~5 years, ie - expect you will have to <br clear="none">> periodically replace bad drives occasionally after the first few years.<br clear="none">> <br clear="none">> It's worth noting here that at $5000, with an expected life of ~5 years <br clear="none">> before you start having to pay for more drives, this is $1000/year and <br clear="none">> gives high speed access, compared to the $1200/year for deep Glacier <br clear="none">> storage above. However, the Glacier storage has much better reliability <br clear="none">> than a single RAID6 array with no additional backup.<br clear="none">> <br clear="none">> Anyway, some food for thought :^)<br clear="none">> <br clear="none">> ---<br clear="none">> Steven Ludtke, Ph.D. <<a shape="rect" ymailto="mailto:sludtke@bcm.edu" href="mailto:sludtke@bcm.edu">sludtke@bcm.edu</a>> Baylor <br clear="none">> College of Medicine<br clear="none">> Charles C. Bell Jr., Professor of Structural Biology Dept. of <br clear="none">> Biochemistry<br clear="none">> Deputy Director, Advanced Technology Cores and <br clear="none">> Molecular Pharmacology<br clear="none">> Academic Director, CryoEM Core<br clear="none">> Co-Director CIBR Center<br clear="none">> <br clear="none">> <br clear="none">> On Feb 14, 2024, at 6:14?PM, Jobichen <<a shape="rect" ymailto="mailto:jobichenc@yahoo.com" href="mailto:jobichenc@yahoo.com">jobichenc@yahoo.com</a>> wrote:<br clear="none">> <br clear="none">> Dear All,<br clear="none">> We are looking for some suggestions on storing the raw datasets/movies. <br clear="none">> What will be best option for storing around 100TB of movies/processed data.<br clear="none">> What will be pros/cons of having own storage server vs cloud storage <br clear="none">> options.<br clear="none">> Thank you for your time.<br clear="none">> Jobi<br clear="none">> <br clear="none">> <br clear="none">> <br clear="none">> _______________________________________________<br clear="none">> 3dem mailing list<br clear="none">> <a shape="rect" ymailto="mailto:3dem@ncmir.ucsd.edu" href="mailto:3dem@ncmir.ucsd.edu">3dem@ncmir.ucsd.edu</a><br clear="none">> <a shape="rect" href="https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=-fMPusn_TT7DVAUweasDDQG4kEyzhEAyjRtShGQYPmx9cRVoBtVsmUUqEMrMPs9w&s=LBeNcMDu7IJx1_Y7BTp2_JFhuug6w0oVJobkLUozOFc&e=" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=-fMPusn_TT7DVAUweasDDQG4kEyzhEAyjRtShGQYPmx9cRVoBtVsmUUqEMrMPs9w&s=LBeNcMDu7IJx1_Y7BTp2_JFhuug6w0oVJobkLUozOFc&e=</a> <<a shape="rect" href="https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=-fMPusn_TT7DVAUweasDDQG4kEyzhEAyjRtShGQYPmx9cRVoBtVsmUUqEMrMPs9w&s=LBeNcMDu7IJx1_Y7BTp2_JFhuug6w0oVJobkLUozOFc&e=" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=-fMPusn_TT7DVAUweasDDQG4kEyzhEAyjRtShGQYPmx9cRVoBtVsmUUqEMrMPs9w&s=LBeNcMDu7IJx1_Y7BTp2_JFhuug6w0oVJobkLUozOFc&e=</a>><br clear="none">> <br clear="none">> <br clear="none">> -------------- next part --------------<br clear="none">> An HTML attachment was scrubbed...<br clear="none">> URL: <br clear="none">> <<a shape="rect" href="http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20240215/7302abc1/attachment.html" target="_blank">http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20240215/7302abc1/attachment.html</a> <<a shape="rect" href="http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20240215/7302abc1/attachment.html" target="_blank">http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20240215/7302abc1/attachment.html</a>>><br clear="none">> <br clear="none">> ------------------------------<br clear="none">> <br clear="none">> Subject: Digest Footer<br clear="none">> <br clear="none">> _______________________________________________<br clear="none">> 3dem mailing list<br clear="none">> <a shape="rect" ymailto="mailto:3dem@ncmir.ucsd.edu" href="mailto:3dem@ncmir.ucsd.edu">3dem@ncmir.ucsd.edu</a><br clear="none">> <a shape="rect" href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem" target="_blank">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a> <br clear="none">> <<a shape="rect" href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem" target="_blank">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a>><br clear="none">> <br clear="none">> <br clear="none">> ------------------------------<br clear="none">> <br clear="none">> End of 3dem Digest, Vol 198, Issue 21<div class="yqt0516040359" id="yqtfd27633"><br clear="none">> *************************************<br clear="none">> <br clear="none">> <br clear="none">> _______________________________________________<br clear="none">> 3dem mailing list<br clear="none">> <a shape="rect" ymailto="mailto:3dem@ncmir.ucsd.edu" href="mailto:3dem@ncmir.ucsd.edu">3dem@ncmir.ucsd.edu</a><br clear="none">> <a shape="rect" href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem" target="_blank">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><br clear="none">_______________________________________________<br clear="none">3dem mailing list<br clear="none"><a shape="rect" ymailto="mailto:3dem@ncmir.ucsd.edu" href="mailto:3dem@ncmir.ucsd.edu">3dem@ncmir.ucsd.edu</a><br clear="none"><a shape="rect" href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem" target="_blank">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><br clear="none"></div></div></div>
</div>
</div></body></html>