<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
I’d run the last round(s) in CPU not GPU. the 385G of RAM won’t help when you are limited to the 11GB the 1080Tis have for holding working data. <br class="">
<div><br class="">
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 15, 2018, at 10:09 AM, Xu, Tinghai (Peter) <<a href="mailto:Tinghai.Xu@vai.org" class="">Tinghai.Xu@vai.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">Dear all,<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">When I using the relion3.0’s 3d multi-body on HPC node (4X GPU 1080Ti with 385G memory), it always get out of memory error. I used almost all the default setting, just use GPU acceleration with
Number of MPI procs:5 Number of threads:2.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">I tried with/without Pre-read all particles in to RAM.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">Here is the error massege:<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">“ERROR: out of memory in /primary/vari/software/relion/relion-3.0_beta/src/acc/acc_projector_impl.h at line 62 (error-code 2)<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">in: /primary/vari/software/relion/relion-3.0_beta/src/acc/cuda/cuda_settings.h, line 67<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">=== Backtrace ===<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi(_ZN11RelionErrorC1ERKSsS1_l+0x41) [0x4417c1]<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi() [0x642239]<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi(_ZN12AccProjector9setMdlDimEiiiiiii+0x19d) [0x642a2d]<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi(_ZN14MlDeviceBundle22setupFixedSizedObjectsEv+0x30b) [0x60813b]<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi(_ZN14MlOptimiserMpi11expectationEv+0x1585) [0x4475e5]<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi(_ZN14MlOptimiserMpi7iterateEv+0xaa) [0x454d9a]<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi(main+0xb15) [0x433175]<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">/usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x2aaab9776b35]<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">/primary/vari/software/relion/relion-3.0_beta/build/bin/relion_refine_mpi() [0x4364af]<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">==================<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">ERROR:<span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">A GPU-function failed to execute.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">If this occured at the start of a run, you might have GPUs which<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">are incompatible with either the data or your installation of relion.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">If you<span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> -> INSTALLED RELION YOURSELF: if you e.g. specified -DCUDA_ARCH=50<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> and are trying ot run on a compute 3.5 GPU (-DCUDA_ARCH=3.5),<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> this may happen.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> -> HAVE MULTIPLE GPUS OF DIFFERNT VERSIONS: relion needs GPUS with<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> at least compute 3.5. You may be trying to use a GPU older than<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> this. If you have multiple generations, try specifying --gpu <X><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> with X=0. Then try X=1 in a new run, and so on. The numbering of<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> GPUs may not be obvious from the driver or intuition. For a list<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> of GPU compute generations, see<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> <span class="Apple-converted-space"> </span><a href="http://en.wikipedia.org/wiki/CUDA#Version_features_and_specifications" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">en.wikipedia.org/wiki/CUDA#Version_features_and_specifications</a><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> -> ARE USING DOUBLE-PRECISION GPU CODE: relion was been written so<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> as to not require this, and may thus have unforeseen requirements<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> when run in this mode. If you think it is nonetheless necessary,<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> please consult the developers with this error.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">If this occurred at the middle or end of a run, it might be that<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> -> YOUR DATA OR PARAMETERS WERE UNEXPECTED: execution on GPUs is<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> subject to many restrictions, and relion is written to work within<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> common restraints. If you have exotic data or settings, unexpected<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> configurations may occur. See also above point regarding<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> double precision.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">If none of the above applies, please report the error to the relion<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">developers at <span class="Apple-converted-space"> </span><a href="http://github.com/3dem/relion/issues" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">github.com/3dem/relion/issues</a><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">”<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">The run.out message for the last Iteration is:<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">“Auto-refine: Iteration= 13<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Auto-refine: Resolution= 22.3816 (no gain for 13 iter)<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> Auto-refine: Changes in angles= 1.37635 degrees; and in offsets= 0.424773 pixels (no gain for 1 iter)<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> Auto-refine: Refinement has converged, entering last iteration where two halves will be combined...<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Auto-refine: The last iteration will use data to Nyquist frequency, which may take more CPU and RAM.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Estimating accuracies in the orientational assignment ...<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">1.33/1.33 min ............................................................~~(,_,"><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Auto-refine: Estimated accuracy angles= 2.338 degrees; offsets= 0.956 pixels<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Auto-refine: Angular step= 0.9375 degrees; local searches= true<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Auto-refine: Offset search range= 2.151 pixels; offset step= 0.717 pixels<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">CurrentResolution= 22.3816 Angstroms, which requires orientationSampling of at least 10 degrees for a particle of diameter 250 Angstroms<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Oversampling= 0 NrHiddenVariableSamplingPoints= 1260<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">OrientationalSampling= 1.875 NrOrientations= 140<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">TranslationalSampling= 1.434 NrTranslations= 9<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">=============================<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Oversampling= 1 NrHiddenVariableSamplingPoints= 40320<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">OrientationalSampling= 0.9375 NrOrientations= 1120<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">TranslationalSampling= 0.717 NrTranslations= 36<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">=============================<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">--------------------------------------------------------------------------<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">with errorcode 1.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">”<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">When I just continue from Iteration 12, it still pops out the same error:<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">Does anyone have come across this situation? Thank you so much.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">Best Regards,<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">Tinghai (Peter) Xu<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">333 Bostwick Ave., N.E., Grand Rapids, Michigan 49503<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">Phone: 616-234-5787 | Email: <a href="mailto:Tinghai.Xu@vai.org" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">Tinghai.Xu@vai.org</a></span><span style="font-size: 7pt; font-family: Verdana, sans-serif;" class=""> </span><span style="font-size: 11pt; color: rgb(31, 73, 125);" class=""><o:p class=""></o:p></span></div>
</div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">3dem
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="mailto:3dem@ncmir.ucsd.edu" style="color: rgb(149, 79, 114); text-decoration: underline; font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">3dem@ncmir.ucsd.edu</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem" style="color: rgb(149, 79, 114); text-decoration: underline; font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem</a></div>
</blockquote>
</div>
<br class="">
</body>
</html>