<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@等线";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:85.05pt 56.7pt 85.05pt 56.7pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Dear all,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">When I using the relion3.0’s 3d multi-body on HPC node (4X GPU 1080Ti with 385G memory), it always get out of memory error. I used almost all the default setting, just use GPU acceleration with
Number of MPI procs:5 Number of threads:2.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">I tried with/without Pre-read all particles in to RAM.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Here is the error massege:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">“ERROR: out of memory in /primary/vari/software/relion/relion-3.0_beta/src/acc/acc_projector_impl.h at line 62 (error-code 2)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">in: /primary/vari/software/relion/relion-3.0_beta/src/acc/cuda/cuda_settings.h, line 67<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">=== Backtrace ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi(_ZN11RelionErrorC1ERKSsS1_l+0x41) [0x4417c1]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi() [0x642239]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi(_ZN12AccProjector9setMdlDimEiiiiiii+0x19d) [0x642a2d]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi(_ZN14MlDeviceBundle22setupFixedSizedObjectsEv+0x30b) [0x60813b]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi(_ZN14MlOptimiserMpi11expectationEv+0x1585) [0x4475e5]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi(_ZN14MlOptimiserMpi7iterateEv+0xaa) [0x454d9a]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi(main+0xb15) [0x433175]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">/usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x2aaab9776b35]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi() [0x4364af]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">==================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">ERROR: <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">A GPU-function failed to execute.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">If this occured at the start of a run, you might have GPUs which<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">are incompatible with either the data or your installation of relion.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">If you <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> -> INSTALLED RELION YOURSELF: if you e.g. specified -DCUDA_ARCH=50<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> and are trying ot run on a compute 3.5 GPU (-DCUDA_ARCH=3.5),
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> this may happen.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> -> HAVE MULTIPLE GPUS OF DIFFERNT VERSIONS: relion needs GPUS with<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> at least compute 3.5. You may be trying to use a GPU older than<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> this. If you have multiple generations, try specifying --gpu <X><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> with X=0. Then try X=1 in a new run, and so on. The numbering of<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> GPUs may not be obvious from the driver or intuition. For a list<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> of GPU compute generations, see
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> en.wikipedia.org/wiki/CUDA#Version_features_and_specifications<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> -> ARE USING DOUBLE-PRECISION GPU CODE: relion was been written so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> as to not require this, and may thus have unforeseen requirements<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> when run in this mode. If you think it is nonetheless necessary,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> please consult the developers with this error.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">If this occurred at the middle or end of a run, it might be that<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> -> YOUR DATA OR PARAMETERS WERE UNEXPECTED: execution on GPUs is
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> subject to many restrictions, and relion is written to work within<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> common restraints. If you have exotic data or settings, unexpected<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> configurations may occur. See also above point regarding
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> double precision.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">If none of the above applies, please report the error to the relion<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">developers at github.com/3dem/relion/issues<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">The run.out message for the last Iteration is:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">“Auto-refine: Iteration= 13<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Auto-refine: Resolution= 22.3816 (no gain for 13 iter)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> Auto-refine: Changes in angles= 1.37635 degrees; and in offsets= 0.424773 pixels (no gain for 1 iter)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> Auto-refine: Refinement has converged, entering last iteration where two halves will be combined...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Auto-refine: The last iteration will use data to Nyquist frequency, which may take more CPU and RAM.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Estimating accuracies in the orientational assignment ...
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">1.33/1.33 min ............................................................~~(,_,"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Auto-refine: Estimated accuracy angles= 2.338 degrees; offsets= 0.956 pixels<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Auto-refine: Angular step= 0.9375 degrees; local searches= true<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Auto-refine: Offset search range= 2.151 pixels; offset step= 0.717 pixels<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">CurrentResolution= 22.3816 Angstroms, which requires orientationSampling of at least 10 degrees for a particle of diameter 250 Angstroms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Oversampling= 0 NrHiddenVariableSamplingPoints= 1260<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">OrientationalSampling= 1.875 NrOrientations= 140<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">TranslationalSampling= 1.434 NrTranslations= 9<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">=============================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Oversampling= 1 NrHiddenVariableSamplingPoints= 40320<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">OrientationalSampling= 0.9375 NrOrientations= 1120<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">TranslationalSampling= 0.717 NrTranslations= 36<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">=============================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">--------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">with errorcode 1.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">When I just continue from Iteration 12, it still pops out the same error:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Does anyone have come across this situation? Thank you so much.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Best Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Tinghai (Peter) Xu<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">333 Bostwick Ave., N.E., Grand Rapids, Michigan 49503<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Phone: 616-234-5787 | Email: <a href="mailto:Tinghai.Xu@vai.org">Tinghai.Xu@vai.org</a></span><span style="font-size:7.0pt;font-family:"Verdana",sans-serif"> </span><span style="font-size:11.0pt;color:#1F497D"><o:p></o:p></span></p>
</div>
</div>
</body>
</html>