[3dem] Seeking advice on Relion/CryoSPARC workstation

Mon Jan 10 04:37:38 PST 2022

Hi all,

to add only a few comments, based on my experience in our lab.

GPU: we have systems with 2 and 4 2080 Ti, and both are used for image
processing. If you go for cryoSPARC live, the more GPUs the better for
better parallelization, as Guillaume already pointed out. Also,
depending on your available funding and the time you want to wait, you
might not go for the very latest NVIDIA GPUs, as they are hard to get.
This will most likely delay your order.

Disk space: Our machines only have 20 TB, and we're constantly at a
limit, with "just" 3-4 users only. So 40 TB is a very good starting
point.

SSD: We're running cryoSPARC without SSD, so I cannot state about the
advantages you'll get with this. But due to read/write speed of a SSD
(compared to a HDD), I would also suggest a SSD as scratch disk.

Best
Christian

>>> Guillaume Gaullier <guillaume.gaullier at icm.uu.se> 01/10/22 11:36 AM
>>>
Hi Artemis,

There are many things to consider, and you should consult with an IT
expert for the details. But the big picture is as follows:

- You need 4 GPUs if you want to use cryoSPARC Live in a way that it can
keep up with data collection (it assigns two GPUs for motion correction,
CTF estimation and particle picking, then one GPU for 2D classification,
then the 4th GPU for 3D reconstruction and refinement; if you have less
then 4 GPUs, you need to stop one of these tasks to be able to start
another, so it’s less automated).

- The model of GPU you choose will pretty much dictate which case you
need (some cases have enough room for only two of the latest GPUs). A
common choice a couple years ago was 4 Nvidia RTX 2080 Ti. I don’t know
about the latest models, but whatever is the successor of this one
should offer good performance for cryoEM data processing. In any case,
you want the "blower" type cards, that exhaust the generated heat out of
the case (as opposed to the regular type that exhausts heat around it
inside the case, which puts more strain on the case’s cooling system).

- Don’t be cheap with RAM; 128 GB is an absolute minimum, and more will
be helpful eventually (not all programs for cryoEM data analysis can use
an SSD scratch disk).

- Some job types in RELION are not GPU-accelerated (motion correction if
you don’t use MotionCor2, bayesian polishing, CTF refinement), so more
CPU cores is beneficial for these job types. If you face a compromise
between less but faster CPU cores and vice versa, it is almost certainly
more beneficial to choose slightly slower CPU cores but more of them.

- You absolutely need a dedicated SSD scratch disk. The OS is typically
on an SSD in these workstations, so some people use their account’s home
directory as a scratch space; but I would recommend against doing this!
if not managed well, this will eventually fill this disk, and you don’t
want to run out of space on the disk housing the OS. The heavy
read/write operations on the scratch space will also wear out the
underlying disk faster, so again not a great idea for the disk housing
the OS: in case of disk failure, it’s a lot easier to change a separate,
dedicated scratch disk than having to also reinstall the OS and all
programs.

- You absolutely need RAID6 for the workstation’s internal storage (or a
10 Gb/s link to the NAS, and the NAS setup in RAID6), this is the sweet
spot between hardware fault tolerance and read/write performance. A slow
storage space will be limiting for the initial step of motion
correction, where whole movies need to be read fast (and there are often
too many movies to put them all on the scratch disk). I am using two
systems, one with a 2-disk RAID1 and another one with a RAID6 (forgot
how many disks it has), and the difference in performance is very
noticeable. On the RAID1 system, most of the time during a motion
correction job is spent with the GPUs waiting for the disks; on the
RAID6 system, the GPUs are all running at full capacity during the
entire job. And don’t be cheap with total usable storage spto move datasets in and out of the workstation too often because of
limiting storage space is quickly annoying. I would say no less than 40
TB, but of course this depends a lot on how many datasets you typically
work on at any given time and how large each individual dataset can be.

These guidelines are meant to optimize for performance, and a
workstation of this type will likely be very noisy; but it should not be
a big problem if you already planned a dedicated space for it.

I hope this helps,

Guillaume

On 5 Jan 2022, at 18:44, akosta at imm.cnrs.fr<mailto:akosta at imm.cnrs.fr>
wrote:

Dear colleagues,
I'm looking for advice on building a workstation for general compute
tasks as well as cryo-EM data analysis (for example with Relion or
cryoSPARC).
The workstation will be placed next to a NAS storage device in our Data
Center and few people will have access (less that 10). The informatics
team will deal with installation and maintenance.
Are there any minimal or/and optimal requirements, especially what would
your GPU and RAM recommendation be?
I am completely ignorant on this matter, so I will appreciate any
advice!
Best regards
Artemis

--
Artemis Kosta
Service de microscopie
FR3479 Institut de Microbiologie de la Méditerranée
Aix Marseille Université
31 chemin Joseph Aiguier, Bat N
CS 70071 - 13402 Marseille cedex 09
France
akosta at imm.cnrs.fr<mailto:akosta at imm.cnrs.fr>
0033 491 164055
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imm.cnrs.fr_microscopie_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=L7-zyQ-04fFCMRqzLIOnx7H0exGZHwIQe_wMPuY600I&m=p2o4y9hYqgLOqKF0I9JDaDfJz-qh-YBAsE0yNa8KnAuQvopJQnP_drt_FwVVEslw&s=mMMscHYkoAtesPOcdF7Uewe7gPKjAmK9WQEdqKVp-Bw&e=
_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu
https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem

När du har kontakt med oss på Uppsala universitet med e-post så innebär
det att vi behandlar dina personuppgifter. För att läsa mer om hur vi
gör det kan du läsa här:
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.uu.se_om-2Duu_dataskydd-2Dpersonuppgifter_&d=DwIGaQ&c=-35OiAkTchMrZOngvJPOeA&r=L7-zyQ-04fFCMRqzLIOnx7H0exGZHwIQe_wMPuY600I&m=QZUuJijXm5Q6fNo2M_C9yWo4bFwNDZmAcd8eYR4oyCnXcRpTHJ8Gsww7JB_XFO9k&s=AbtB1bnJqmeYE6NO5LiYMQTuKNEkr33dnSlWsGrX9TE&e=

E-mailing Uppsala University means that we will process your personal
data. For more information on how this is performed, please read here:
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.uu.se_en_about-2Duu_data-2Dprotection-2Dpolicy&d=DwIGaQ&c=-35OiAkTchMrZOngvJPOeA&r=L7-zyQ-04fFCMRqzLIOnx7H0exGZHwIQe_wMPuY600I&m=QZUuJijXm5Q6fNo2M_C9yWo4bFwNDZmAcd8eYR4oyCnXcRpTHJ8Gsww7JB_XFO9k&s=HibWtvA-BgmJj3H0IAi6xENHTEGmQV6x5iFG8V4zAxs&e=