[3dem] Multi CPU desktop

Wed Apr 9 10:35:17 PDT 2014

 This was very informative, thanks.  

It does seems that with  ~$10K you can  get a workstation capable of some serious image processing. This is about what we used to pay (or less) for an SGI in the dark ages. Also considering that such system would be dedicated 100% to your jobs this may even beat sending them to a shared cluster.   

Best

Hernando

-----Original Message-----
From: 3dem-bounces at ncmir.ucsd.edu [mailto:3dem-bounces at ncmir.ucsd.edu] On Behalf Of Steven Ludtke
Sent: Tuesday, April 08, 2014 11:14 AM
To: Pawel A. Penczek
Cc: 3dem at ncmir.ucsd.edu
Subject: Re: [3dem] Multi CPU desktop

Clearly I wasn't suggesting this as a complete replacement for clusters. However, note also that I said 5000 particles in less than an hour, so in 24 hours, you could process ~150,000 particles of the same size. Of course for more resolution you'd want finer sampling, etc.  A 64 core machine for ~$10k would almost certainly be AMD rather than intel, and each core would likely be 2 Ghz or less. That is, I would anticipate that the performance would be no more than ~1.5x the machine I quoted, and I suspect would also lack the 24 TB RAID array I included at that price (accounting for ~$3k of the $8k price I quoted).

Bottom line is that the CPU vendors are actually pretty intelligent, and have worked out chip pricing so total system cost scales with performance, becoming nonlinear in the extremes where there are few competitive options. I used to be a big fan of AMD processors, but then Intel did an end-run and has dominated ever since. The fact that the AMD processors are so much cheaper is an indication of the relative performance issues (though it scales with typical system price, not processor price).

If you're thinking about spending >$5k on a system it definitely behooves you to try and demo a machine with the same processor with a friend, vendor, etc.  The 'e2speedtest.py' included in EMAN2.1 will give you a pretty good estimate of the relative performance of a machine for most scientific image processing. It will give a value of ~1.0 on a current generation Intel processor at 2.6 Ghz. The value is the speed of a single core.

Another thing that may be worth starting to consider is the new Phi coprocessors which try to be GPU-like, but still use the standard Intel instruction set, so you don't have to recode everything in CUDA or Open-CL. The current Phi offering is 61 cores for ~$4k. I haven't experimented with them yet, though...

On Apr 8, 2014, at 9:32 AM, Penczek, Pawel A <Pawel.A.Penczek at uth.tmc.edu> wrote:

> Hi,
> 
> this is quite interesting.  However, I would say 5,000 particles only go so far.
> 
> I heard that one could get a box with 64 CPUs (I am not sure about 
> threads) and significant memory for ~10k.  I was thinking about getting two of those and hooking them up the have a mini-cluster.
> I would of course skip monitors.  
> 
> My believe is that with such a modest investment one could do decent projects "at home".
> 
> Is it really feasible?
> 
> Regards,
> -
> Pawel A. Penczek, Ph.D.
> Professor
> Structural Biology Imaging Center, Director The University of Texas
> phone: 713-500-5416
> fax: 713-500-0652
> http://www.uth.tmc.edu/bmb/faculty/pawel-penczek.html
> 
> 
> 
> 
> On Apr 8, 2014, at 9:24 AM, Steven Ludtke <sludtke at bcm.edu> wrote:
> 
>> Here is the system I was referring to:
>> 
>> Supermicro SC743-TQ-865B-SQ tower case (supports 8 hot-swap drives plus 3 fixed), MUCH quieter than similar cases, and very well designed
>> SUPERMICRO X9DAE motherboard - supports 2 processors, 16 DIMMs (up to 1 TB of RAM)
>> 2x Xeon E5-2650 (2.6 Ghz)
>> 128 GB DDR3 - registered PC12800
>> LSI MegaRAID SAS 9271-8i - Hardware RAID controller
>> 8x4TB WD Black drives
>> Any GeForce card (really no benefit in the Quadros)
>> 
>> Total cost of this config is ~$8k, and would be higher if you got higher clocked processors, but:
>> 
>> - 16 cores -> 32 threads - A few years ago threads were useless, but now you can get ~ 25% performance boost in image processing by using 24 or 32 rather than 16
>> - 128 GB RAM - suitable for very large projects and tomography processing
>> - 24 TB of usable RAID6 storage, which can read/write at ~1GB/sec, VERY useful with direct-detector data (note that 6TB drives are just emerging now, so if 24 TB isn't quite enough for you, wait a couple of months, and you can put 36 TB in the same form factor)
>> 
>> For many image processing projects, I can skip the cluster and just do the processing on this machine. I can use our old GroEL test data set (5000 particles) and get a 7.6 Å gold standard resolution structure out of this machine from scratch in under 1 hour.
>> 
>> On Apr 8, 2014, at 9:05 AM, rkhayat at ccny.cuny.edu wrote:
>> 
>>> Hi Steve,
>>> 
>>> What motherboard and CPU chipsets are you using? Thanks.
>>> 
>>> Best wishes,
>>> Reza
>>> 
>>> Reza Khayat, PhD
>>> Assistant Professor
>>> The City College of New York
>>> Department of Chemistry, MR-1135
>>> 160 Convent Avenue
>>> New York, NY  10031
>>> Tel. (212) 650-6070
>>> www.khayatlab.org
>>> 
>>> 
>>> ---- Original message ----
>>>> Date: Tue, 8 Apr 2014 09:02:25 -0500
>>>> From: 3dem-bounces at ncmir.ucsd.edu (on behalf of Steven Ludtke 
>>> <sludtke at bcm.edu>)
>>>> Subject: Re: [3dem] Multi CPU desktop  
>>>> To: Hernando J Sosa <hernando.sosa at einstein.yu.edu>
>>>> Cc: "3dem at ncmir.ucsd.edu" <3dem at ncmir.ucsd.edu>
>>>> 
>>>> I'm using a 16 core (2x8) Xeon setup at the moment,
>>>> which is quite good for a wide range of problems,
>>>> particularly if you equip it with a fast internal
>>>> RAID. The Opteron processors have remained pretty
>>>> sluggish per-core compared to the Intel processors
>>>> in recent years. That is, 16 opteron cores vs 16
>>>> Xeon cores at the same clock speed still has a
>>>> substantial performance difference for floating
>>>> point math. For most image processing applications,
>>>> performance within a specific chip series will scale
>>>> pretty linearly with clock speed, so 8x3 ghz cores
>>>> will be equivalent to 12x2 ghz cores in the same
>>>> series.  If you're thinking about the opterons, I
>>>> would definitely suggest seeing if you can run a
>>>> benchmark before purchasing, and make sure the $$$
>>>> you're forking out for a large number of cores is
>>>> going to get you a real advantage.
>>>> On Apr 8, 2014, at 7:58 AM, Hernando J Sosa
>>>> <hernando.sosa at einstein.yu.edu> wrote:
>>>> 
>>>>  Dear All,
>>>> 
>>>>  I am considering getting a desktop for  image
>>>>  processing and was looking into some 4  CPU ones
>>>>   (Opteron  16 cores each e.g.
>>>>   https://urldefense.proofpoint.com/v1/url?u=http://www.thinkmate.com/system/hpx-qs5-4410&k=yYSsEqip9%2FcIjLHUhVwIqA%3D%3D%0A&r=2aNbZJh6eMzYI7ecrNAUvrma%2F0vaXYG%2BIjyWZafay%2BU%3D%0A&m=B%2FjgPQI8mbbsJ99SWBZi%2BF4G1VUWDVs8XI%2BLBj8wCR0%3D%0A&s=30604f55c16402e5ff3c9035c82fc9fd296582ea6759d7d9d5a8932b90ac4661) . I
>>>>  was wondering  if anybody has any particular
>>>>  suggestion or better alternatives (apart from
>>>>  submitting your jobs to a central computing
>>>>  facility).
>>>> 
>>>> 
>>>>  Thanks
>>>> 
>>>>  Hernando
>>>>  _______________________________________________
>>>>  3dem mailing list
>>>>  3dem at ncmir.ucsd.edu
>>>>  https://urldefense.proofpoint.com/v1/url?u=https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem&k=yYSsEqip9%2FcIjLHUhVwIqA%3D%3D%0A&r=2aNbZJh6eMzYI7ecrNAUvrma%2F0vaXYG%2BIjyWZafay%2BU%3D%0A&m=B%2FjgPQI8mbbsJ99SWBZi%2BF4G1VUWDVs8XI%2BLBj8wCR0%3D%0A&s=2869c067675ad9735a9c498dcfcb3af19b99395f38ea8f34566c8f631c292e4e
>>>> 
>>>> ----------------------------------------------------------
>>> ------------------
>>>> Steven Ludtke, Ph.D.
>>>> Professor, Dept of Biochemistry and Mol. Biol.
>>>>  (www.bcm.edu/biochem)
>>>> Co-Director National Center For Macromolecular
>>>> Imaging        (ncmi.bcm.edu)
>>>> Co-Director CIBR Center
>>>>  (www.bcm.edu/research/cibr)
>>>> Baylor College of Medicine
>>>> 
>>>> sludtke at bcm.edu
>>>> ________________
>>>> _______________________________________________
>>>> 3dem mailing list
>>>> 3dem at ncmir.ucsd.edu
>>>> https://urldefense.proofpoint.com/v1/url?u=https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem&k=yYSsEqip9%2FcIjLHUhVwIqA%3D%3D%0A&r=2aNbZJh6eMzYI7ecrNAUvrma%2F0vaXYG%2BIjyWZafay%2BU%3D%0A&m=B%2FjgPQI8mbbsJ99SWBZi%2BF4G1VUWDVs8XI%2BLBj8wCR0%3D%0A&s=2869c067675ad9735a9c498dcfcb3af19b99395f38ea8f34566c8f631c292e4e
>> 
>> ----------------------------------------------------------------------------
>> Steven Ludtke, Ph.D.
>> Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)
>> Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)
>> Co-Director CIBR Center                          (www.bcm.edu/research/cibr)
>> Baylor College of Medicine                             
>> sludtke at bcm.edu
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> 3dem mailing list
>> 3dem at ncmir.ucsd.edu
>> https://urldefense.proofpoint.com/v1/url?u=https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem&k=yYSsEqip9%2FcIjLHUhVwIqA%3D%3D%0A&r=2aNbZJh6eMzYI7ecrNAUvrma%2F0vaXYG%2BIjyWZafay%2BU%3D%0A&m=B%2FjgPQI8mbbsJ99SWBZi%2BF4G1VUWDVs8XI%2BLBj8wCR0%3D%0A&s=2869c067675ad9735a9c498dcfcb3af19b99395f38ea8f34566c8f631c292e4e
> 
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem

----------------------------------------------------------------------------
Steven Ludtke, Ph.D.
Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)
Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)
Co-Director CIBR Center                          (www.bcm.edu/research/cibr)
Baylor College of Medicine                             
sludtke at bcm.edu

_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu
https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem