[3dem] Seeking advice on Relion/CryoSPARC workstation

Mon Jan 10 05:01:02 PST 2022

Hi all,

Towards the GPU question:

Seen from the MSRP price the 3090 is probably the best option. It gives you 24GB of VRAM, which is more then enough and also a bit future proof. The single precision (FP32) performance is great, though it takes quite a lot of power.
Though now the problems…you will hardly get one for the MSRP more likley double the price.. and as far as I know there are nearly no blower style editions available of the RTX30-XX generation, Nvidia Founder Editions are no blower style since the 20th generation and custom board partners did not really make or sell a lot of bower style for the 30th generation.
Easier to get these days are workstation/data center grade GPUs, though they are of course more expensive and not really needed. A6000 for example is near to equal/bit better then the 3090 in FP32, though significantly more expensive, but hey you get at least 48GB of VRAM and blower style cooling;) Data Center alternative would be the A40, this is passive cooled so only for servers with high airflow.
You have to check with your resellers if you may can apply to education discounts.

Here is a sheet with different GPU options I made back then when we had to decide.

Name	Chip	Cuda Cores	Base [MHz]	Boost [MHz]	Memory [GB]	Mem Bandwidth [GB/s]	VRAM Bus Width [bit]	TFLOPS FP32	TDP [W]
RTX 3080Ti	GA102	10240	1365	1665	12	912,4	384	34,1	350
RTX 3090	GA102	10496	1395	1695	24	936,2	384	35,58	350
A10	GA102	9216	885	1695	24	600,2	384	31,24	150
A4000	GA104	6144	735	1560	16	448	256	19,17	140
A5000	GA102	8192	1170	1695	24	768	384	27,77	230
A40	GA102	10752	1305	1740	48	695,8	384	37,42	300
A6000	GA102	10752	1455	1860	48	768	384	40	300
A30	GA100	3584	930	1440	24	933,1	3072	10,32	165
A100	GA100	6912	765	1410	40	1555	5120	19,49	250

cheers
Kilian

Kilian Schnelle
Department of Structural Biology
University of Osnabrueck
Barbarastr. 13
D-49076 Osnabrueck, Germany
Phone: +49 541 969 7508
Email: kilian.schnelle at uni-osnabrueck.de

> On 10. Jan 2022, at 11:35, Guillaume Gaullier <guillaume.gaullier at icm.uu.se> wrote:
> 
> Hi Artemis,
> 
> There are many things to consider, and you should consult with an IT expert for the details. But the big picture is as follows:
> 
> - You need 4 GPUs if you want to use cryoSPARC Live in a way that it can keep up with data collection (it assigns two GPUs for motion correction, CTF estimation and particle picking, then one GPU for 2D classification, then the 4th GPU for 3D reconstruction and refinement; if you have less then 4 GPUs, you need to stop one of these tasks to be able to start another, so it’s less automated).
> 
> - The model of GPU you choose will pretty much dictate which case you need (some cases have enough room for only two of the latest GPUs). A common choice a couple years ago was 4 Nvidia RTX 2080 Ti. I don’t know about the latest models, but whatever is the successor of this one should offer good performance for cryoEM data processing. In any case, you want the "blower" type cards, that exhaust the generated heat out of the case (as opposed to the regular type that exhausts heat around it inside the case, which puts more strain on the case’s cooling system).
> 
> - Don’t be cheap with RAM; 128 GB is an absolute minimum, and more will be helpful eventually (not all programs for cryoEM data analysis can use an SSD scratch disk).
> 
> - Some job types in RELION are not GPU-accelerated (motion correction if you don’t use MotionCor2, bayesian polishing, CTF refinement), so more CPU cores is beneficial for these job types. If you face a compromise between less but faster CPU cores and vice versa, it is almost certainly more beneficial to choose slightly slower CPU cores but more of them.
> 
> - You absolutely need a dedicated SSD scratch disk. The OS is typically on an SSD in these workstations, so some people use their account’s home directory as a scratch space; but I would recommend against doing this! if not managed well, this will eventually fill this disk, and you don’t want to run out of space on the disk housing the OS. The heavy read/write operations on the scratch space will also wear out the underlying disk faster, so again not a great idea for the disk housing the OS: in case of disk failure, it’s a lot easier to change a separate, dedicated scratch disk than having to also reinstall the OS and all programs.
> 
> - You absolutely need RAID6 for the workstation’s internal storage (or a 10 Gb/s link to the NAS, and the NAS setup in RAID6), this is the sweet spot between hardware fault tolerance and read/write performance. A slow storage space will be limiting for the initial step of motion correction, where whole movies need to be read fast (and there are often too many movies to put them all on the scratch disk). I am using two systems, one with a 2-disk RAID1 and another one with a RAID6 (forgot how many disks it has), and the difference in performance is very noticeable. On the RAID1 system, most of the time during a motion correction job is spent with the GPUs waiting for the disks; on the RAID6 system, the GPUs are all running at full capacity during the entire job. And don’t be cheap with total usable storage space: having to move datasets in and out of the workstation too often because of limiting storage space is quickly annoying. I would say no less than 40 TB, but of course this depends a lot on how many datasets  you typically work on at any given time and how large each individual dataset can be.
> 
> These guidelines are meant to optimize for performance, and a workstation of this type will likely be very noisy; but it should not be a big problem if you already planned a dedicated space for it.
> 
> I hope this helps,
> 
> Guillaume
> 
> 
>> On 5 Jan 2022, at 18:44, akosta at imm.cnrs.fr <mailto:akosta at imm.cnrs.fr> wrote:
>> 
>> Dear colleagues,
>> I'm looking for advice on building a workstation for general compute tasks as well as cryo-EM data analysis (for example with Relion or cryoSPARC).
>> The workstation will be placed next to a NAS storage device in our Data Center and few people will have access (less that 10). The informatics team will deal with installation and maintenance.
>> Are there any minimal or/and optimal requirements, especially what would your GPU and RAM recommendation be?
>> I am completely ignorant on this matter, so I will appreciate any advice!
>> Best regards
>> Artemis
>> 
>> -- 
>> Artemis Kosta
>> Service de microscopie
>> FR3479 Institut de Microbiologie de la Méditerranée
>> Aix Marseille Université
>> 31 chemin Joseph Aiguier, Bat N
>> CS 70071 - 13402 Marseille cedex 09
>> France
>> akosta at imm.cnrs.fr <mailto:akosta at imm.cnrs.fr>
>> 0033 491 164055
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imm.cnrs.fr_microscopie_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=L7-zyQ-04fFCMRqzLIOnx7H0exGZHwIQe_wMPuY600I&m=p2o4y9hYqgLOqKF0I9JDaDfJz-qh-YBAsE0yNa8KnAuQvopJQnP_drt_FwVVEslw&s=mMMscHYkoAtesPOcdF7Uewe7gPKjAmK9WQEdqKVp-Bw&e= _______________________________________________
>> 3dem mailing list
>> 3dem at ncmir.ucsd.edu
>> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
> 
> 
> 
> 
> 
> 
> 
> 
> 
> När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här: http://www.uu.se/om-uu/dataskydd-personuppgifter/ <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.uu.se_om-2Duu_dataskydd-2Dpersonuppgifter_&d=DwQGaQ&c=-35OiAkTchMrZOngvJPOeA&r=L7-zyQ-04fFCMRqzLIOnx7H0exGZHwIQe_wMPuY600I&m=QZUuJijXm5Q6fNo2M_C9yWo4bFwNDZmAcd8eYR4oyCnXcRpTHJ8Gsww7JB_XFO9k&s=AbtB1bnJqmeYE6NO5LiYMQTuKNEkr33dnSlWsGrX9TE&e=> 
> 
> E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here: http://www.uu.se/en/about-uu/data-protection-policy <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.uu.se_en_about-2Duu_data-2Dprotection-2Dpolicy&d=DwQGaQ&c=-35OiAkTchMrZOngvJPOeA&r=L7-zyQ-04fFCMRqzLIOnx7H0exGZHwIQe_wMPuY600I&m=QZUuJijXm5Q6fNo2M_C9yWo4bFwNDZmAcd8eYR4oyCnXcRpTHJ8Gsww7JB_XFO9k&s=HibWtvA-BgmJj3H0IAi6xENHTEGmQV6x5iFG8V4zAxs&e=>_______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20220110/6eafd355/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5926 bytes
Desc: not available
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20220110/6eafd355/attachment-0001.p7s>