<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div dir="auto" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black;">
Hi all, <br>
</div>
<div dir="auto" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black;">
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black;">
My apologies to jump in on this conversation with something outside the scope of the question, but Kilian's question about AVX-512 considerations in a build is related to an issue our lab is having with a workstation.
<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black;">
We recently got a workstation from a vendor with 2 RTX 3090s, 128GB ram, and a 10980XE processor on an ASUS x299 SAGE 10GbE. Since the start, it's had issues with running 2D and 3D classification and refinement jobs, primarily in cryosparc or programs using
Cryosparc engine (ie Scipion). The system will get 3-5 iterations into a job, before spontaneously restarting. It doesn't post a kernel panic error, nor does it post any errors in system logs besides orphaned processes and unrelated process errors from the
crash. Normal stress tests of both CPU and GPU haven't returned any issues. <br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black;">
I've traced the issue, potentially, to an issue with AVX-512 processing causing the CPU to trip some cutoff and shutdown the system. Changing some overclocking settings in the BIOS to limit voltage/clock speed has helped, but the issue still happens, just not
as soon in the job (iteration ~15). Does anyone have a recommended general BIOS configuration for computers doing Cryo-EM processing with Cryosparc and related programs? Thank you for any possible help!<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black;">
Best,<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black;">
Justas<br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> 3dem <3dem-bounces@ncmir.ucsd.edu> on behalf of Guillaume Gaullier <guillaume.gaullier@icm.uu.se><br>
<b>Sent:</b> Thursday, May 20, 2021 10:33:54 AM<br>
<b>To:</b> Israel Fernandez <israel.elotro@gmail.com><br>
<b>Cc:</b> 3dem@ncmir.ucsd.edu <3dem@ncmir.ucsd.edu><br>
<b>Subject:</b> Re: [3dem] Advice for hardware of compute nodes</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">In my case it was 768 GB total (64 GB * 12 slots I believe, in a Supermicro case). But yes, 512 GB already qualifies as a "ridiculous amount" in my book. You could also call it "future
proof" (as in: future programs implementing more sophisticated analyses than we use now may need more RAM), which is a better vocabulary to use as a justification for your spendings. :-)<br class="">
<div class="">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; word-wrap:break-word; line-break:after-white-space">
<div class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; word-wrap:break-word; line-break:after-white-space">
<div class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; word-wrap:break-word; line-break:after-white-space">
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">
<br class="x_Apple-interchange-newline">
Guillaume</div>
<div style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">
<br class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 20 May 2021, at 16:14, Israel Fernandez <<a href="mailto:israel.elotro@gmail.com" class="">israel.elotro@gmail.com</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div dir="auto" class="">Ridiculous amount of RAM could mean 512Gb?</div>
<div class=""><br class="">
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Thu, 20 May 2021 at 13:21, Guillaume Gaullier <<a href="mailto:guillaume.gaullier@icm.uu.se" class="">guillaume.gaullier@icm.uu.se</a>> wrote:<br class="">
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; padding-left:1ex; border-left-color:rgb(204,204,204)">
<div class="" style="word-wrap:break-word; line-break:after-white-space">Hello Kilian,
<div class=""><br class="">
</div>
<div class="">Regarding number of cores versus frequency, I would suggest going for more cores: some job types in RELION are not GPU-accelerated but scale very well with more MPI processes (motion correction, bayesian polishing and CTF refinement), so for those
you will benefit a lot more from having many cores than from having fewer faster cores. And 2.8-3.7 GHz is already plenty.</div>
<div class=""><br class="">
</div>
<div class="">Another general advice would be: don’t be cheap with RAM and storage, both in terms of speed and amount. Fast CPUs and GPUs are no use if you can’t feed them your data fast enough (they would spend a lot of their time waiting for inputs). The
amount of RAM is also easy to overlook, thinking you will only ever use RELION and cryoSPARC which can both use an SSD cache efficiently, but newer programs don’t always have this capability (I am thinking about cryoDRGN in particular), and it would be frustrating
to have to wait until they implement this or find workarounds (RAM upgrade, or not using your entire dataset, etc.) in the meantime. Last year I bought a workstation for the lab I work at with what seemed like a ridiculous amount of RAM, and this year I’m
thinking it was a good idea in hindsight, since I’ve been happily running cryoDRGN with large box sizes and many particles without any problem (besides that it takes a long time to run, of course… but I haven’t been limited by RAM when cryoDRGN loads the entire
dataset).<br class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">I have no idea about your other questions, but there are probably compromises to make between all options (if not technical ones, at least in terms of budget).</div>
<div class=""><br class="">
</div>
<div class="">I hope this helps, good luck!</div>
<div class="">
<div dir="auto" class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; word-wrap:break-word; line-break:after-white-space">
<div class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; word-wrap:break-word; line-break:after-white-space">
<div class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; word-wrap:break-word; line-break:after-white-space">
<div class="" style="word-wrap:break-word; line-break:after-white-space; font-family:Helvetica">
<div class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">
<br class="">
Guillaume</div>
<div class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">
<br class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div class="">
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 20 May 2021, at 11:46, Kilian Schnelle <<a href="mailto:kilian.schnelle@uni-osnabrueck.de" target="_blank" class="">kilian.schnelle@uni-osnabrueck.de</a>> wrote:</div>
<br class="">
<div class="">
<div class="" style="word-wrap:break-word; line-break:after-white-space">Hello everyone,
<div class=""><br class="">
</div>
<div class="">I am currently thinking about which hardware to get for compute nodes mostly used for cryosparc and Relion with Sturm and would be glad if someone could give me an advice.</div>
<div class=""><br class="">
</div>
<div class="">I was hovering over maybe something like the G292-Z43 servers from Gigabyte with 2-4 RTX 3090, 64-128 GB Ram and a NVMe SSD for cache. (Someone actually has experience with an exported NVMe SSD cache for all nodes? Worth over having one in every
node?) I am not sure which CPU to get though I was thinking of:</div>
<div class=""><br class="">
</div>
<div class="">- 2x <span class="">AMD Epyc 7543 32c/64t, 2,8-3,7 GHz, 256 MB L3, ~3700$/each</span></div>
<div class=""><span class=""><br class="">
</span></div>
<div class=""><font class="" style="">Or higher frequency and less cores?</font></div>
<div class=""><font class="" style=""><br class="">
</font></div>
<div class="">- 2x <span class="">AMD Epyc 74F3 24c/48t, 3,2-4,0 GHz, 256 MB L3, ~2900$/each</span></div>
<div class=""><span class=""><br class="">
</span></div>
<div class=""><font class="" style="">Or even just a single socket CPU with a different server like the 7532P?(Would have to calculate the PCIe lanes I need for everything but, would PCIe 4.0 x8 or x16 for the GPUs even make a difference?, I mean its same as
3.0 x16 in throughput in theory)</font></div>
<div class=""><br class="">
</div>
<div class="">- AMD Epyc 7543P 32c/64t, 2,8-3,7 GHz, 256 MB L3, ~2700$</div>
<div class=""><br class="">
</div>
<div class="">Is it even worth going for the 256 MB L3 cache? Or is it worth going for Team Blue CPUs because of AVX 512?</div>
<div class=""><br class="">
</div>
<div class="">Any insides would be highly appreciated.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Best wishes</div>
<div class="">Kilian</div>
</div>
_______________________________________________<br class="">
3dem mailing list<br class="">
<a href="mailto:3dem@ncmir.ucsd.edu" target="_blank" class="">3dem@ncmir.ucsd.edu</a><br class="">
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwMGaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=dFvFJX5sHRsUBJKImx2V5A9YiXfLtbd74YGjtYoLJBA&m=YjjTo2YhgevPitB-oXhkELYmq4TGhs3bQrhz-0KIKFc&s=Ok8bSNa5sw18gqEj4krxQ2m9Kjw1Zrn9dHSF1GqMEoM&e=" target="_blank" class="">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div class="">
<div class=""></div>
</div>
<u class=""></u><br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här:
<a href="https://urldefense.com/v3/__http://www.uu.se/om-uu/dataskydd-personuppgifter/__;!!Mih3wA!W1Uag_Pu-I6ca5jvZWv9Sa-me5dBrMSCy_wySw8GrmNJHJu6GbwIbrA5eNoI_OYasg$" target="_blank" class="">
http://www.uu.se/om-uu/dataskydd-personuppgifter/</a> <br class="">
<br class="">
E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here:
<a href="https://urldefense.com/v3/__http://www.uu.se/en/about-uu/data-protection-policy__;!!Mih3wA!W1Uag_Pu-I6ca5jvZWv9Sa-me5dBrMSCy_wySw8GrmNJHJu6GbwIbrA5eNoVP56dCQ$" target="_blank" class="">
http://www.uu.se/en/about-uu/data-protection-policy</a> </div>
_______________________________________________<br class="">
3dem mailing list<br class="">
<a href="mailto:3dem@ncmir.ucsd.edu" target="_blank" class="">3dem@ncmir.ucsd.edu</a><br class="">
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwMGaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=dFvFJX5sHRsUBJKImx2V5A9YiXfLtbd74YGjtYoLJBA&m=YjjTo2YhgevPitB-oXhkELYmq4TGhs3bQrhz-0KIKFc&s=Ok8bSNa5sw18gqEj4krxQ2m9Kjw1Zrn9dHSF1GqMEoM&e=" rel="noreferrer" target="_blank" class="">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><br class="">
</blockquote>
</div>
</div>
-- <br class="">
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr" class="">Israel S Fernandez
<div class="">Columbia University, New York City, NY, USA</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>