<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
EMPIAR is great, and while they say they will take whatever people provide, if all of the raw data (even if it were limited to 'good' data) from all of the Krios around the world were archived in EMPIAR, there would be massive bandwidth and storage issues.
When used as it is now, as an archive for important reference data sets, it's great, but I'm not sure it's a viable strategy for archiving everything produced in the CryoEM community.
<div class=""><br class="">
</div>
<div class="">back of the envelope:</div>
<div class="">2000 images/day * 500 MB/compressed counting movie * 100 krios ~ 100 TB/day</div>
<div class="">
<div class=""><br class="">
</div>
<div class="">even a dedicated 10 Gb network running flat out couldn't keep up. </div>
<div class=""><br class="">
<div class="">
<div class="">
<div dir="auto" style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<font face="Courier" class=""><span style="font-size: 14px;" class="">--------------------------------------------------------------------------------------<br class="">
Steven Ludtke, Ph.D. <<a href="mailto:sludtke@bcm.edu" class="">sludtke@bcm.edu</a>> Baylor College of Medicine <br class="">
Charles C. Bell Jr., Professor of Structural Biology<br class="">
Dept. of Biochemistry and Molecular Biology (<a href="http://www.bcm.edu/biochem" class="">www.bcm.edu/biochem</a>)<br class="">
Academic Director, CryoEM Core (<a href="http://cryoem.bcm.edu" class="">cryoem.bcm.edu</a>)<br class="">
Co-Director CIBR Center (<a href="http://www.bcm.edu/research/cibr" class="">www.bcm.edu/research/cibr</a>)<br class="">
<br class="">
</span></font><br class="">
</div>
</div>
</div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 29, 2019, at 7:04 AM, Takanori Nakane <<a href="mailto:tnakane@mrc-lmb.cam.ac.uk" class="">tnakane@mrc-lmb.cam.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Hi,</span><br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<blockquote type="cite" style="font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
Most people just opt for the "hard drive on a shelf" method for completed<br class="">
projects, which has advantages (cheap/simple) and disadvantages (what<br class="">
happens if the drive dies)...<br class="">
</blockquote>
<br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">After
publication of your structures, I recommend raw data to be deposited</span><br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">in
EMPIAR.</span><br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Not
only is it useful for reproducibility, education and method development,</span><br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">it
also serves as an additional layer of backup. You might drop your disk,</span><br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">water
might leak from the ceiling, etc. Having backups in a physically</span><br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">distant</span><br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">place
is a good practice.</span><br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Best
regards,</span><br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Takanori
Nakanori</span><br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<blockquote type="cite" style="font-family: Courier; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
Julien,<br class="">
are you referring to the raw data, or are you trying to archive all of the<br class="">
files associated with a project?<br class="">
<br class="">
Counting-mode movies are generally stored and archived as compressed tiff<br class="">
stacks, though if they are collected on a Falcon, there are issues with<br class="">
this, as good compression is achieved only pre-normalization (or<br class="">
post-normalization if you decide you are willing to switch back to an<br class="">
integer format).<br class="">
<br class="">
If you want to perfectly archive everything exactly as it is (losslessly),<br class="">
some compression algorithms may do very slightly better than others, but<br class="">
pretty much any of the commonly used algorithms will do about the same.<br class="">
Usually the slower ones will do slightly better, but you have to decide if<br class="">
it's worth the CPU time the compression takes. By definition, the noisier<br class="">
the data is, the less compressible it is, unless you are willing to invoke<br class="">
"lossy" compression and throw away some of the bits of pure noise.<br class="">
<br class="">
Most people just opt for the "hard drive on a shelf" method for completed<br class="">
projects, which has advantages (cheap/simple) and disadvantages (what<br class="">
happens if the drive dies)...<br class="">
<br class="">
--------------------------------------------------------------------------------------<br class="">
Steven Ludtke, Ph.D. <<a href="mailto:sludtke@bcm.edu" class="">sludtke@bcm.edu</a><<a href="mailto:sludtke@bcm.edu" class="">mailto:sludtke@bcm.edu</a>>><br class="">
Baylor College of Medicine<br class="">
Charles C. Bell Jr., Professor of Structural Biology<br class="">
Dept. of Biochemistry and Molecular Biology<br class="">
(<a href="http://www.bcm.edu/biochem" class="">www.bcm.edu/biochem</a><<a href="http://www.bcm.edu/biochem" class="">http://www.bcm.edu/biochem</a>>)<br class="">
Academic Director, CryoEM Core<br class="">
(<a href="http://cryoem.bcm.edu/" class="">cryoem.bcm.edu</a><<a href="http://cryoem.bcm.edu/" class="">http://cryoem.bcm.edu</a>>)<br class="">
Co-Director CIBR Center<br class="">
(<a href="http://www.bcm.edu/research/cibr" class="">www.bcm.edu/research/cibr</a><<a href="http://www.bcm.edu/research/cibr" class="">http://www.bcm.edu/research/cibr</a>>)<br class="">
<br class="">
<br class="">
<br class="">
On Aug 29, 2019, at 6:30 AM, Julien Bous<br class="">
<<a href="mailto:julien.bous@etu.umontpellier.fr" class="">julien.bous@etu.umontpellier.fr</a><<a href="mailto:julien.bous@etu.umontpellier.fr" class="">mailto:julien.bous@etu.umontpellier.fr</a>>><br class="">
wrote:<br class="">
<br class="">
<br class="">
<br class="">
Dear Community,<br class="">
<br class="">
I have a question about the best way to store my data once SPA projects<br class="">
are achieved. Can you advise me about which compression format is to<br class="">
prefer?<br class="">
<br class="">
Thank you for your interest,<br class="">
<br class="">
Julien<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
3dem mailing list<br class="">
<a href="mailto:3dem@ncmir.ucsd.edu" class="">3dem@ncmir.ucsd.edu</a><<a href="mailto:3dem@ncmir.ucsd.edu" class="">mailto:3dem@ncmir.ucsd.edu</a>><br class="">
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=-Yu84q3MdcWvESYXpaK7NQdEWch6tE1eG9IVNTjLay4&s=NSrJg_YgFffwLELO1auXSC6yYLEsGHVoNV5TI_1eBqM&e=" class="">https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=-Yu84q3MdcWvESYXpaK7NQdEWch6tE1eG9IVNTjLay4&s=NSrJg_YgFffwLELO1auXSC6yYLEsGHVoNV5TI_1eBqM&e=</a><br class="">
<br class="">
_______________________________________________<br class="">
3dem mailing list<br class="">
<a href="mailto:3dem@ncmir.ucsd.edu" class="">3dem@ncmir.ucsd.edu</a><br class="">
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwIDJg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=NIpw6RIeeyKxoYDz2eZPHOcIZvNm9VytdzBFUEtQ-10&s=UG1BMTIotgpZVcqSlW0cd0tfnpxgEo9l3RLHUfU2ODc&e=" class="">https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwIDJg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=NIpw6RIeeyKxoYDz2eZPHOcIZvNm9VytdzBFUEtQ-10&s=UG1BMTIotgpZVcqSlW0cd0tfnpxgEo9l3RLHUfU2ODc&e=</a></blockquote>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</body>
</html>