<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-2022-jp">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Hideki,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I think you have a power issue.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I have two of the predecessors of these boxes since 8 years. They worked stable with 8x GTX590, but only 6x GTX690 and only 6x GTX1080Ti. The latter draws 250W of power according to specs. The rest of the hardware also needs another 600W or so.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The triple PSUs on this Tyan box yield 2000W at 100V input voltage, but 3200W at 220V. You could try connecting it to 3x 220V circuits if you have access to those (these must each have 15A rating or you will blow the breaker under full load). Otherwise I am
afraid you just don't get the electrical power required. Mind you, running a box like that under full load constantly is actually not cheap (think 20 JPY/kWhr, 3kW/hr = 60 JPY/hr = 1440 JPY/day = 43k JPY/month = 518k JPY/year). And then don't forget about
the cooling - need proper A/C, which also uses electricity. Better in a data center...</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
In addition, I removed all the fans from my GPUs in that box, because I found they run 20 degrees colder without fans. This is because the GPU fans expel the air both sides, which goes against the strong airflow by the 6 load and big case fans. You can read
out the temperature using 'nvidia-smi'. If it's more than 80C, you might have a thermal problem.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
If you can't upgrade the external power to 200-240V, try running your box with 4 GPUs first and then keep adding them until it becomes unstable.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Matthias</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> 3dem <3dem-bounces@ncmir.ucsd.edu> on behalf of Shigematsu, HIDEKI <hideki.shigematsu@riken.jp><br>
<b>Sent:</b> Tuesday, March 24, 2020 7:54 AM<br>
<b>To:</b> Liz Kellogg <lizkellogg@gmail.com><br>
<b>Cc:</b> 3dem@ncmir.ucsd.edu <3dem@ncmir.ucsd.edu><br>
<b>Subject:</b> Re: [3dem] weird server issues, suggestions/advice would be helpful!</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hi Liz,<br>
<br>
<br>
I have similar symptom with 4GPU box of 2080Ti. I put two 1000W PSU in this box one for 3 GPUs and the other for 1 GPU and Ryzen Thread ripper 32core. When I set to limit the power consumption of GPUs to 250W, it last longer. And one of my friend has the same
issue with same config but he hooked PSU to 100V power supply. Now, he switched one of the PSU to 1600W and it works fine.<br>
I think you should better try to limit the power of the GPUs by using <br>
nvidia-smi -pl 250 <br>
or remove some of GPUs from the machine to see whether it works with some GPU jobs.<br>
Worst case scenario is that you have the PSU which has limit in the power supply for the specific port for PCI-e devices, in that case, limiting the power consumption to the GPU might work but removing some of GPUs does$B!G(Bt work.<br>
<br>
Best,<br>
<br>
Hideki<br>
----<br>
Hideki Shigematsu Ph. D.<br>
<br>
RIKEN SPring-8 Center, Life Science Research Infrastructure Group<br>
1-1-1 Kouto Sayo-cho Sayo-gun, Hyogo 679-5148 Japan<br>
Phone +81-791-58-0803 (Ext.7868)<br>
FAX +81-791-58-2834<br>
<br>
> 2020/03/24 7:25$B!"(BLiz Kellogg <lizkellogg@gmail.com>$B$N%a!<%k(B:<br>
> <br>
> Hi 3dem-ers,<br>
> <br>
> I hope everyone is safely at home. I have a non COVID-19 related problem that I hope others can help advise on.<br>
> <br>
> I bought a 8-GPU server last April that would be used by my lab for image processing work. Everything seemed fine initially, however once I started getting more users the server became noticeably unstable around December and started randomly rebooting itself.
It was happening so often that at its worst we couldn$B!G(Bt get through a single refinement job without a reboot. Here are some technical details and hints at what could be going wrong:<br>
> <br>
> Configuration of the server:<br>
> TYAN Thunder HX FT77D-B7109 8GPU 2P 4x3<br>
> Intel Xeon Gold 6138 20C 2.0-3.7 GHz<br>
> 384 GB DDR4 2400/2666 ECC/REG (12x32GB)<br>
> SamSung 480GB 883 DCT SSD x 2<br>
> Seagate 12TB SAS x 16 <br>
> GeForce RTX-2080Ti 11 GB x 8<br>
> <br>
> The most noticeable errors we see when the server is up are the GPU devices becoming undetectable, along the lines of:<br>
> <br>
> $ nvidia-smi<br>
> Unable to determine the device handle for GPU 0000:B1:00.0: GPU is lost. Reboot the system to recover this GPU<br>
> <br>
> Or <br>
> <br>
> $ nvidia-smi<br>
> No devices were found<br>
> <br>
> Replacing the GPUs did not seem to help which we did back in January, we are back to the same issues.
<br>
> We also tried updating the GPU drivers to NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 (before they were 410.48)<br>
> However, we experience pretty much the same behavior before and after the driver update.
<br>
> <br>
> Since we have updated the drivers, I doubt that$B!G(Bs a driver issue. Although it could be a PCI bus issue, doesn$B!G(Bt seem likely to me because each of the 8 cards tend to go down randomly (during one strange episode, they were flickering on and off). My gut feeling
is that there is either a power issue where the system$B!G(Bs power was not dimensioned properly (though looking at the chassis specs this seems unlikely as well), or a cooling issue. I am planning on monitoring the GPU temperature (I wrote a bash script using
nvidia-smi -q) under heavy load and see if the current temp exceeds the maximum temp of each GPU.
<br>
> <br>
> Any idea of what would be going on? I think I have pretty standard server config.. has anyone experienced similar problems? Anyone that has configurations that work well for you, would you mind sharing your specs and your NVIDIA driver versions? Even if it's
exactly the same specs that would help. Any non-standard steps to configure the machine or the drivers? I am mystified as to why we are experiencing these issues.. and doesn$B!G(Bt help that we$B!G(Bre all working from home at the moment :*(<br>
> <br>
> Thanks everyone, stay safe.<br>
> <br>
> Best wishes,<br>
> <br>
> Liz<br>
> <br>
> Elizabeth H. Kellogg, Ph.D.<br>
> Assistant Professor, Cornell University<br>
> Molecular Biology and Genetics<br>
> <br>
> _______________________________________________<br>
> 3dem mailing list<br>
> 3dem@ncmir.ucsd.edu<br>
> <a href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><br>
<br>
_______________________________________________<br>
3dem mailing list<br>
3dem@ncmir.ucsd.edu<br>
<a href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><br>
</div>
</span></font></div>
</body>
</html>