[3dem] Solution to slow connection between linux and falcon3 server

benoit.zuber at ana.unibe.ch benoit.zuber at ana.unibe.ch
Mon Jan 15 04:59:21 PST 2018


Dear all,

I am reporting my experience with connexion speed between the so-called FEI server that records Falcon 3 frames (running under Windows) and an Ubuntu 16.04 workstation, connected with a direct 10G glass fiber.

As pointed out by several of you, using Samba instead of NFS is the way to go.
However Samba is a bit of a jungle as three names are used to speak sometimes of the same thing, sometimes of different things. These names are Samba, SMB, and CIFS.

It doesn’t really matter what they mean since not everyone agrees on common definitions. What matters is what you can practically do to get high transfer rates, both ways.

1st solution, pointed out by several of you: share a linux folder with windows using samba. For that you might need to install Samba (warning: this is not the same as cifs-utils):

sudo apt-get install samba

then edit /etc/samba/smb.conf and specify which folder to mount (getting everything right with respect to permissions was a bit of a pain but you’ll get there with google on your side).
Restart samba server:
sudo service smbd restart

you should then see your linux drive in windows and transfers should be fast.

2nd solution. After trying to figure out what the difference between CIFS, SMB, and Samba was I found an easier and cleaner way of setting up the connection. Here we mount a windows share on the linux workstation using mount.cifs.
As previously pointed out, if you use the regular mount.cifs (or mount –t cifs) speed will be miserable.
The trick is to specify the SMB version. To get fast connection with the FEI server you need to use at least SMB 2.1. On Ubuntu the latest SMB version available with mount is 3.0. By default, mount uses SMB version 1.0. Note that smbd uses SMB version 3.0, hence it is working fast out of the box.

With mout, you can specify SMB version with the optional keyword “vers”. You’ll find more infos about when and why to use which version in the man page of mount.cifs

Here is an example mount command:
sudo mount -t cifs -o username=feiserverusername,password=feiserverpassword,vers=3.0,uid=1004,gid=1004 //10.0.0.1/benshare testshare
This mounts the share benshare from the FEI server in the folder testshare on Ubuntu station; uid and gid ensure that user 1004 can read and write in this folder.

This solution gave the best results in our hands. We reach 1 Gbyte/sec in both directions.

Hope this will be helpful to others (this could also be integrated in Falcon3 user’s manual)

Best wishes
Ben

De : Kasim Sader <kasim.sader at gmail.com>
Date : vendredi, 22 décembre 2017 à 13:01
À : Benoît Zuber <benoit.zuber at ana.unibe.ch>
Cc : "<Sylvain.Trepout at curie.fr>" <Sylvain.Trepout at curie.fr>, Steven Ludtke <sludtke at bcm.edu>, "3dem at ncmir.ucsd.edu" <3dem at ncmir.ucsd.edu>
Objet : Re: [3dem] Slow connection linux - falcon3 server

Hi Benoit,
I use the same setup as Steve - Linux SMB shares mounted on (many) windows machines, and I get 800MB/s from the Gatan PC SSD RAID array (Windows) to my large Linux storage box. You do need to alter your windows host file though to point to the IP address, otherwise you cannot unmount the drive easily (a windows bug).
Best regards,
Kasim



Dr. Kasim Sader

Staff Scientist

Operational Lead

Cambridge CryoEM Pharmaceutical Consortium

Thermo Fisher Scientific

Tel +447342882058

kasim.sader at thermofisher.com<mailto:kasim.sader at thermofisher.com>

Sent from my iPhone

On 21 dec. 2017, at 14:44, <benoit.zuber at ana.unibe.ch<mailto:benoit.zuber at ana.unibe.ch>> <benoit.zuber at ana.unibe.ch<mailto:benoit.zuber at ana.unibe.ch>> wrote:

Thanky you Sylvain,

Added on the to do list, yes I guess we can try with SCP



De : Trepout Sylvain <Sylvain.Trepout at curie.fr<mailto:Sylvain.Trepout at curie.fr>>
Date : jeudi, 21 décembre 2017 à 15:10
À : Steven Ludtke <sludtke at bcm.edu<mailto:sludtke at bcm.edu>>, Benoît Zuber <benoit.zuber at ana.unibe.ch<mailto:benoit.zuber at ana.unibe.ch>>
Cc : "3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>" <3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>>
Objet : RE: [3dem] Slow connection linux - falcon3 server



Hi Benoit,



CIFS is a specific protocol of SAMBA that has been created to read UNIX mounted drives from Windows machines.

Since you want to perform file transfer from the opposite direction, you might want to use SAMBA instead of CIFS.

Though I'm not sure if it will drastically increase the speed.



Can you use command line operations instead ?

SCP should perform much better.



Best,

Sylvain

----------
Sylvain TREPOUT
Ingénieur de Recherche Inserm
Institut Curie / INSERM U1196
Campus Universitaire d'Orsay, Bat 112
91405 Orsay cedex FRANCE
Phone : +33 1 69 86 31 96<tel:+33%201%2069%2086%2031%2096>
Fax : +33 1 69 07 53 27<tel:+33%201%2069%2007%2053%2027>

________________________________

De : 3dem <3dem-bounces at ncmir.ucsd.edu<mailto:3dem-bounces at ncmir.ucsd.edu>> de la part de Ludtke, Steven J <sludtke at bcm.edu<mailto:sludtke at bcm.edu>>
Envoyé : jeudi 21 décembre 2017 14:56:24
À : benoit.zuber at ana.unibe.ch<mailto:benoit.zuber at ana.unibe.ch>
Cc : 3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
Objet : Re: [3dem] Slow connection linux - falcon3 server



Sorry, missed the fast windows -> windows transfer comment in the original message.



We went through similar sort of issue when configuring our K2 acquisition computer (windows) to a Linux GPU workstation with 8 drive RAID, also over 10Gb fiber. In our case we were trying to mount the Linux RAID as a volume on the K2 computer so we could just directly store frames there during acquisition. Similar to your experience we discovered that the performance of the NFS module in Windows is abysmal, even after tuning.



Strangely, the approach that gave the best performance was using SMB to share the Linux drive to the Windows machine. I don't remember the final performance numbers (we set this up some time ago), but I'm pretty sure it was coming in at around 500 MB/s.  No other approach we tried gave decent bandwidth.



--------------------------------------------------------------------------------------
Steven Ludtke, Ph.D.
Charles C. Bell Jr., Professor of Structural Biology
Dept. of Biochemistry and Molecular Biology                      (www.bcm.edu/biochem<http://www.bcm.edu/biochem>)

Erreur ! Le nom du fichier n’a pas été spécifié.<http://www.bcm.edu/biochem>


Biochemistry & Molecular Biology | Baylor College of Medicine | Houston, Texas<http://www.bcm.edu/biochem>

www.bcm.edu<http://www.bcm.edu>

The overview page for the Department of Biochemistry & Molecular Biology.



Co-Director CIBR Center                                    (www.bcm.edu/research/cibr<http://www.bcm.edu/research/cibr>)

Erreur ! Le nom du fichier n’a pas été spécifié.<http://www.bcm.edu/research/cibr>


Computational and Integrative Biomedical Research Center | Research | Baylor College of Medicine | Houston, Texas<http://www.bcm.edu/research/cibr>

www.bcm.edu<http://www.bcm.edu>

Computational and Integrative Biomedical Research Center



Co-Director National Center For Macromolecular Imaging                  (ncmi.bcm.edu<http://ncmi.bcm.edu>)

- NCMI CryoEM- Baylor College of Medicine - Houston, TX<http://ncmi.bcm.edu/>

ncmi.bcm.edu<http://ncmi.bcm.edu>

To develop technology for near-atomic resolution structure determination and modeling of molecular nanomachines by electron cryo-microscopy(cryo-EM), computer reconstruction and structure mining.



Baylor College of Medicine
sludtke at bcm.edu<mailto:sludtke at bcm.edu>



On Dec 21, 2017, at 6:32 AM, benoit.zuber at ana.unibe.ch<mailto:benoit.zuber at ana.unibe.ch> wrote:



Hi Steve,



Thanks for your reply. It has 12 disks in RAID (not sure what type). I don’t think this is the problem since when we connect another windows machine to it, files can be transfered high speed in both directions.



Benoit



De : Steven Ludtke <sludtke at bcm.edu<mailto:sludtke at bcm.edu>>
Date : jeudi, 21 décembre 2017 à 14:06
À : Benoît Zuber <benoit.zuber at ana.unibe.ch<mailto:benoit.zuber at ana.unibe.ch>>
Cc : "3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>" <3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>>
Objet : Re: [3dem] Slow connection linux - falcon3 server



Hi Benoit,

what sort of storage is being used to store the files in question on the FEI server?  Your description sounds like the files being transferred are on a single 'spinning platter' hard drive, which will generally max out in data I/O at 150-200 MB/s. Most machines used for storing high bandwidth data will either use a RAID 5/6 array of traditional hard drive (which can achieve 1000-2000 MB/s depending on the number of drives) or high speed SSDs which will max out at ~600 MB/s on a standard SATA bus, but can achieve 1000-3000 MB/s if alternative interfaces are used.



--------------------------------------------------------------------------------------
Steven Ludtke, Ph.D.
Charles C. Bell Jr., Professor of Structural Biology
Dept. of Biochemistry and Molecular Biology                      (www.bcm.edu/biochem<http://www.bcm.edu/biochem>)
Co-Director CIBR Center                                    (www.bcm.edu/research/cibr<http://www.bcm.edu/research/cibr>)
Co-Director National Center For Macromolecular Imaging                  (ncmi.bcm.edu<http://ncmi.bcm.edu/>)
Baylor College of Medicine
sludtke at bcm.edu<mailto:sludtke at bcm.edu>



On Dec 21, 2017, at 4:19 AM, benoit.zuber at ana.unibe.ch<mailto:benoit.zuber at ana.unibe.ch> wrote:



***CAUTION:*** This email is not from a BCM Source. Only click links or open attachments you know are safe.

________________________________

Dear all



We have a linux workstation connected directly through fiber optics to the so-called FEI server, i.e. the computer where Falcon3 saves frames, which runs under Windows. Strangely, file transfer from linux to windows is fast (600 MB/s) whereas file transfer from windows to linux is slow (160 MB/s). We have tried mounting the windows drive on linux through cifs or nfs, as well as mounting the linux drive on windows through nfs and we always get the same result.



We also tested windows to windows (we connected another windows machine to the fei server) : fast both ways. And we tested linux to linux : fast both ways



We tested network speed using iperf, and it is high in either direction. It is only when we transfer files that we have problems.



Any suggestion welcome.



Thanks

Benoit



--

Prof. Benoît Zuber

University of Bern

Institute of Anatomy

Baltzerstrasse 2

Postfach 922

3000 Bern 9



+41 31 631 84 40<tel:+41%2031%20631%2084%2040>

_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=iM30kDS0pjLFsoGLC5GUR_9aRscO2JMiBlLeuLAqe-k&s=c228ksyoRZn9VrDIqrz1-8wQz_j3mEiCm-er4g6mrFE&e=




_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20180115/1e720121/attachment-0001.html>


More information about the 3dem mailing list