[3dem] File systems

[Sjors Scheres]

Hi Reza, Ben, Steve & the rest,

Happy New Year! That 2016 may bring you all many side chains. :-)

We've tested infiniband verus our standard ethernet and deemed the cost of
it not worth it, as we saw hardly any improvement in speed of relion. We
did however see an improvement of >30% speed when switching our cluster
from a NFS to a Fraunhofer file system. The issue arises when one starts
accessing the same disk from several hundreds of MPI processes, something
that can easily happen when dozens of users use the same system.
I think I agree with Steve that for not very large group-wide systems NFS
will be better, as it is much  easier, but for larger clusters a more
parallel file system will probably be worth the effort. At LMB, our
(small) home areas are based on NFS, while the disks on our cluster were
Fraunhofer and are now gpfs.
HTH,
Sjors


> Hi Reza,
>
> Our cluster was recently upgraded to InfiniBand and switched to gpfs.
> Since then the rate of mysterious mpi crashes/hang during relion runs has
> decreased quite a lot. And relion runs faster. Is it due to gpfs, to
> InfiniBand or to the combination of both? I do not know.
>
> Cheers
> Ben
>
> Le 3 janv. 2016 à 18:53, Reza Khayat <rkhayat at ccny.cuny.edu> a écrit :
>
> Hi,
>
> Can anyone describe some of their experience with deploying and using a
> distributed filesystem for image analysis? Is it appropriate to say that
> NFS is antiquated, slow and less secure than the younger systems like
> Lustre, Gluster, Ceph, PVFS2, or Fraunhofer?
>
> Best wishes,
> Reza
>
> Reza Khayat, PhD
> Assistant Professor
> City College of New York
> 85 St. Nicholas Terrace CDI 12308
> New York, NY 10031
> (212) 650-6070
> www.khayatlab.org
>
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
>


-- 
Sjors Scheres
MRC Laboratory of Molecular Biology
Francis Crick Avenue, Cambridge Biomedical Campus
Cambridge CB2 0QH, U.K.
tel: +44 (0)1223 267061
http://www2.mrc-lmb.cam.ac.uk/groups/scheres