<div dir="ltr">Dear all,<div><br></div><div>I can only agree with what Matthias wrote. I had a Phi 7120 in my hands for a week and tried to evaluate it for highly parallelizable algorithms that also go well with GPUs. While easier portability is one of Intel's marketing claims, it appears to be true for only very few cases. None of the popular CPU-based EM code I have seen would experience significant speed-ups without modifications. For the latter, porting from CUDA C to OpenCL (which Phi will happily execute) is much easier than modifying C++ code to make better use of Phi. However, while Phi and GPUs are similar enough to be addressed properly by OpenCL, getting the most out of any of them will require device-specific optimization. I am not sure if getting the peak performance out of a Phi is even possible with OpenCL – it certainly isn't for Nvidia GPUs (for more complex code). </div><div><br></div><div>Thus, you can choose between: No code changes and virtually no speed-ups; porting to OpenCL for reasonable performance across many platforms, but still under-utilizing every platform (and having a rather poor development environment compared to CUDA/C++); writing device-specific code to achieve the maximum.</div><div><br></div><div>Then there is Intel MKL for common tasks, such as FFT or BLAS functions. It is highly optimized for every architecture Intel currently offers, including Phi – to an extent only Intel can afford. MKL can be used as a straight drop-in for popular libraries like FFTW, so porting is just a matter of changing library names. However, using Phi only for these tasks doesn't make much sense, as it's connected over the same slow PCIe bus that makes pushing data to a GPU such a pain. You would want to do more with the data once they are in Phi's memory. It's really the same situation as with CUDA, which offers the same set of vendor-supplied, optimized libraries as Intel.</div><div><br></div><div>Going Phi causes exactly the same problems as going GPU (specifically, Nvidia & CUDA). It is only a question of whom you trust to deliver more FLOPS per $ in the long run. Right now, Nvidia is doing a much better job, but that can change if Intel ever decides to make Phi a first-class citizen (1 release in 3 years – really?).</div><div><br></div><div>As for the scalability issues on AMD chips: Their current approach to counting cores is selling a "module" with 1 float and 2 integer pipelines as 2 cores. If your code can saturate the float pipeline with 1 thread, the second "core" will be useless.</div><div><br></div><div>Best,</div><div>Dimitry</div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 18, 2014 at 11:14 AM, Matthias Wolf <span dir="ltr"><<a href="mailto:matthias.wolf@oist.jp" target="_blank">matthias.wolf@oist.jp</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="color:#1f497d">Hi Dewight, Alexis,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">Just to chime in after Alexis’ message – I did compile frealign on a Phi 7120 about one year ago during a visit by Intel. While the procedure was straight forward, I did not attempt any non-standard optimizations.
Out of the box, the performance was rather disappointing and at the time I decided it were better to use standard multi-core Xeon processors.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">Compared to Xueming Li’s GPU version of frealign, the xeon phi I tested was no competition – I use a 16-GPU box (8x nvidia GTX590 in a Tyan barebone), which accelerates the program ~1500-fold as compared to a
single 2.7 GHz Xeon core.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">While the concept of the phi is nice – it feels like having a little linux cluster in your PC to which you can ssh and run multi-threaded programs, it has clear limitations: the one I tested had only 16GB memory,
which makes large reconstructions problematic. The 61 (Intel Atom-derived) cores per board run at only 1GHz and they have a small cache. Now this is not much different to GPUs, but there are many more cores on most GPUs. Maybe with the right optimizations,
the phi would be a worthy adversary, but I did not have the time to find out.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">Regarding Intel vs AMD, I agree 100% with Steve Ludke’s statements. I tested a 32-core Opteron system against the latest quad core Xeon a couple years ago and while roughly comparable at single-threaded performance,
the Xeon scaled linearly with the number of threads (frealign-mp), whereas the Opteron quickly saturated (more than 12 cores were useless) and its performance was significantly lower. I believe this has to do with AMDs interconnects having lower bandwidth
than Intel’s hypertransport. In particular the E-series Xeons are really very good.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">Finally (this came up in a previous thread) – there is no problem operating nvidia gaming GPUs in headless mode with linux – my box is sitting in a rack in the datacenter and I simply ssh to it. Actually, the
gaming cards use the same chips as their corresponding quadro or Tesla relatives, less ECC memory. They are usually even higher clocked than the more expensive “professional” cards, but the chief difference is that the GTX series has less memory. So unless
you need quad-buffered graphics for windowed stereo and a lot of memory, there is no point in buying anything else. The main issue is to feed them with data, which requires SSD-raid, and providing sufficient current. Cooling can be alleviated by removing
their on-board fans in a good rack-mounted case, which brings the temps down by 20-30C.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"> Matthias<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1f497d">_______________________________________________________<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1f497d">Matthias Wolf, PhD MPharm - Assistant Professor<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1f497d">Molecular Cryo-Electron Microscopy Unit<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1f497d">Okinawa Institute of Science and Technology Graduate University<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1f497d">1919-1 Tancha, Onna-son, Kunigami-gun<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1f497d">Okinawa 904-0495, Japan<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1f497d">Phone <a href="tel:%2B81-%280%2998-966-8987" value="+81989668987" target="_blank">+81-(0)98-966-8987</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> <a href="mailto:3dem-bounces@ncmir.ucsd.edu" target="_blank">3dem-bounces@ncmir.ucsd.edu</a> [mailto:<a href="mailto:3dem-bounces@ncmir.ucsd.edu" target="_blank">3dem-bounces@ncmir.ucsd.edu</a>]
<b>On Behalf Of </b>Alexis Rohou<br>
<b>Sent:</b> Tuesday, November 18, 2014 2:11 PM<br>
<b>To:</b> <a href="mailto:3dem@ncmir.ucsd.edu" target="_blank">3dem@ncmir.ucsd.edu</a><br>
<b>Subject:</b> Re: [3dem] Utilizing the Xeon Phi<u></u><u></u></span></p>
</div>
</div><div><div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Dear Dewight,<br>
<br>
As far as I know, none of the 3DEM packages have been adapted to run on Phi boards. This means you could run them (provided you recompiled them using the Intel compilers) but only in native mode, which involves SSH'ing onto the boards. And even then, without
optimization, you'd probably get worse performance than on a top-of-the-range Xeon chip. However I guess if you pack enough cards per node you might get improved density for your cluster.<br>
<br>
The topic of Phi boards was brought up at the NRAMM meeting last week at Scripps and it seemed no-one had tried them yet.<br>
<br>
Here at Janelia we bought a Phi 7200 to test out, but haven't got round to doing much with it because of the time required to investigate program optimization and the relatively meager prospective gains.<br>
<br>
So, bottom line: don't go for a cluster with Phi boards, because none of the 3DEM software will be ready for them.<br>
<br>
Hope this helps,<br>
Alexis<span style="font-size:12.0pt"><u></u><u></u></span></p>
<div>
<p class="MsoNormal">On 11/12/2014 10:08 AM, Dewight R. Williams wrote:<u></u><u></u></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Dear 3dem, <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Has anyone performed 3D single particle reconstruction on the new Intel Xeon Phi boards? When you performed this work did the software need to be recompiled or was it implemented through standard openMPI? What software were you using Frealign,
Relion, Xmipp, EMAN2, etc? Thanks, I’m debating on which architecture I want to invest in for a local cluster and any feedback on these questions would be very appreciated.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Dewight<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><br>
<br>
<br>
<u></u><u></u></span></p>
<pre>_______________________________________________<u></u><u></u></pre>
<pre>3dem mailing list<u></u><u></u></pre>
<pre><a href="mailto:3dem@ncmir.ucsd.edu" target="_blank">3dem@ncmir.ucsd.edu</a><u></u><u></u></pre>
<pre><a href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem" target="_blank">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><u></u><u></u></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><br>
<br>
<br>
<u></u><u></u></span></p>
<pre>-- <u></u><u></u></pre>
<pre>Alexis Rohou<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Research Specialist<u></u><u></u></pre>
<pre><span lang="DE-AT">Grigorieff Lab<u></u><u></u></span></pre>
<pre><a href="http://grigoriefflab.janelia.org" target="_blank"><span lang="DE-AT">http://grigoriefflab.janelia.org</span></a><span lang="DE-AT"><u></u><u></u></span></pre>
<pre>Tel. <a href="tel:%2B1%20571%20209%204000%20x3485" value="+15712094000" target="_blank">+1 571 209 4000 x3485</a><u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><u></u> <u></u></span></p>
</div></div></div>
</div>
</div>
<br>_______________________________________________<br>
3dem mailing list<br>
<a href="mailto:3dem@ncmir.ucsd.edu" target="_blank">3dem@ncmir.ucsd.edu</a><br>
<a href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem" target="_blank">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a><br>
<br></blockquote></div><br></div></div>