[3dem] Falcon II uses only half of the 1Gb/s bandwidth.

Tue Oct 23 08:46:38 PDT 2018

Dear All,

I have attached a pdf version of the “Falcon Hack” document. I basically documented steps in details how to setup our Falcon II hack using Greg’s beautiful hacking. Brandeis web server took off the old set of documents suite some time ago. But the hack is still working nicely at Brandeis scope.

Like Matthias said, one likely needs updated software from Greg. FEI has changed things since the time we did this hack.

Best -

--
Chen Xu
UMass Med School

From: 3dem <3dem-bounces at ncmir.ucsd.edu> on behalf of Eugene Pichkur <eugene.pichkur at gmail.com>
Date: Tuesday, October 23, 2018 at 10:42 AM
To: Steve Ludtke <sludtke at bcm.edu>
Cc: "3dem at ncmir.ucsd.edu" <3dem at ncmir.ucsd.edu>
Subject: Re: [3dem] Falcon II uses only half of the 1Gb/s bandwidth.

Dear All,

Thank you for your excellent suggestions regarding the issue.

Mike Strauss was right about EPU writing stacks from Falcon II to the Microscope PC's HDD before transferring it to the destination folder on Support PC. This is an obvious bottleneck, but I think FEI did this for a good reason (Win 7 upgrade was probably implemented at the very end of the product lifecycle, so there was no point in making the old product almost as good as the new one).
Thanks to Mike, I was able to locate the folder where intermediate frames are saved: C:/Users/<username>/AppData/Local/Temp/FalconInterImages/
However, IntermediateConfig.xml points to the different location, not the one that is currently used (you'll find the attachment below). I've tried to find .XML files with the same name, but haven't found any, except for those that were in the backup copy made during the migration from Win XP to Win 7.

Now let me address your questions one at a time:

What type of disks are on Camera Support Rack #1?
Camera Support Rack #1 is a "black box" - I don't have a direct access to it, so no info on the type of the memory used and/or additional computations performed on it.
You are talking about the image transfer being performed by the FEI software, rather than copying a file via SMB?
Yes, correct. As indicated on the network map, 1Gbps channel ("Support network") used for transferring via SMB is fully utilised.

Matthias, thank you for pointing out the possible workarounds, I've contacted Chen Xu already. I didn't know the camera can go to 32 fps or higher, that's fascinating!

Steven, of course you are right about the switch. It isn't (probably) a cause of the problem itself (since 1Gb/s Ethernet has been around for 20 years or so). My guess was that it's configured this way to "cap" incoming 1Gbps channels because there is only one outgoing 1Gbps channel.

Regards,

Eugene

On Thu, Oct 18, 2018 at 10:40 PM Ludtke, Steven J <sludtke at bcm.edu<mailto:sludtke at bcm.edu>> wrote:
You know, I think I misunderstood the question. You are talking about the image transfer being performed by the FEI software, rather than copying a file via SMB?   If so, then almost certainly nothing to do with your hardware at all. Just poor software design in handling the network transfer by FEI.

--------------------------------------------------------------------------------------
Steven Ludtke, Ph.D. <sludtke at bcm.edu<mailto:sludtke at bcm.edu>>                      Baylor College of Medicine
Charles C. Bell Jr., Professor of Structural Biology
Dept. of Biochemistry and Molecular Biology                      (www.bcm.edu/biochem<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.bcm.edu_biochem&d=DwMFaQ&c=WJBj9sUF1mbpVIAf3biu3CPHX4MeRjY_w4DerPlOmhQ&r=QtWDcbwka7T8JQfLE66OgK0dv8RNd2CnULAJfRAhZtg&m=krVHiDK1jtUeDViN8gx4zz6ol56nLDMsNXF78IYgXvo&s=-TkZ4rFm4cTDvjh87d8PmmeOrMIse6homXbYJuUZoxw&e=>)
Academic Director, CryoEM Core                                        (cryoem.bcm.edu<https://urldefense.proofpoint.com/v2/url?u=http-3A__cryoem.bcm.edu&d=DwMFaQ&c=WJBj9sUF1mbpVIAf3biu3CPHX4MeRjY_w4DerPlOmhQ&r=QtWDcbwka7T8JQfLE66OgK0dv8RNd2CnULAJfRAhZtg&m=krVHiDK1jtUeDViN8gx4zz6ol56nLDMsNXF78IYgXvo&s=1vt079Rq5dCQEQsxYhnXsFbgOaAxkn0vGh3dNowRHcs&e=>)
Co-Director CIBR Center                                    (www.bcm.edu/research/cibr<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.bcm.edu_research_cibr&d=DwMFaQ&c=WJBj9sUF1mbpVIAf3biu3CPHX4MeRjY_w4DerPlOmhQ&r=QtWDcbwka7T8JQfLE66OgK0dv8RNd2CnULAJfRAhZtg&m=krVHiDK1jtUeDViN8gx4zz6ol56nLDMsNXF78IYgXvo&s=-5kvHi4eWZsV5AiEV73_uItel0nnaMbTWj2cZAUbt9A&e=>)

On Oct 18, 2018, at 1:29 PM, Matthias Wolf <matthias.wolf at oist.jp<mailto:matthias.wolf at oist.jp>> wrote:

***CAUTION:*** This email is not from a BCM Source. Only click links or open attachments you know are safe.
________________________________
Hi Gene,

Don't bother with the inadequate FEI microscope PC and its gigabit network.

For proper performance, you have 2 options:

  1.  get FEI's storage server and frame saving hardware (CMTS) with dedicated 10Gbit/s upload. It'll cost you $$$. Needs Win7 on the scope PC.
  2.  set up Greg McMullans brilliant Falcon frame capture, which uses a beam splitter (fiber optical tap) to duplicate the raw camera signal and capture it on a dedicated linux PC with fast SSD. Total investment ~3k US$. This works on older Win XP systems, but Greg has also written code for Win7 (FEI changes something in the Falcon data protocol upon upgrade to Win7). You may want to look up Chen Xu's "falcon hacking" page at Brandeis on Internet Waybackmachine. That way, the FEI system is completely bypassed and your movie frames will be happily saved on the capture PC before you even get an image in TIA. I have done this with good success for several years until option 1 was installed in the course of a Win7+Falcon3 upgrade. You must obtain specific Solarflare NICs for Greg's code to work and should follow his instructions to the letter (but best don't bother him and go with option 1)!

Image size is 16 Mpixel * 14bit/pixel. At 18 fps, the data rate is ~4 Gb/s.
Decent hardware can deal with this. The camera can go to 32 fps or even higher, but then everything on the linux O/S must be tuned or you drop frames.

Without frame capture, you won't get decent performance - both number of frames and frame rate will be severely limited and not up to current standards of cryo-EM movie data collection.

As Steve wrote, your gigabit switch is unlikely to throttle bandwidth.
But the raw camera data is sent in jumbo packets. Maybe your switch is using standard MTU of 1500?
You could try changing this to use jumbo packets of 9014 bytes with telnet and the CLI.

   Matthias

________________________________
From: 3dem <3dem-bounces at ncmir.ucsd.edu<mailto:3dem-bounces at ncmir.ucsd.edu>> on behalf of Eugene Pichkur <eugene.pichkur at gmail.com<mailto:eugene.pichkur at gmail.com>>
Sent: Tuesday, October 16, 2018 10:15:00 PM
To: 3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
Subject: [3dem] Falcon II uses only half of the 1Gb/s bandwidth.

Hello 3DEM community,

We've noticed that our Falcon II uses only half of the 1Gbps bandwidth to transfer data from the Camera Support Rack to the Microscope PC (I've attached a screenshot to illustrate the issue).

Microscope PC runs Windows 7 and we are able to get the full frame rate (20fps) from the detector, however, it takes ~20s to transfer a typical movie (40frames, 1.2GB) to the PC which is a very frustrating bottleneck.

Both Falcon II and Ceta are connected through the same 1Gb switch (HPE OfficeConnect 1420 24G 2SFP), so my guess is that the bandwidth is split 50/50. However, during data collection only Falcon II is used which makes this limitation pointless, not to mention that Falcon II and Ceta can't be used simultaneously.

Does anyone have a workaround or any info on the topic?

Thanks in advance,
Eugene Pichkur

National Research Centre “Kurchatov Institute”
Akademika Kurchatova, 1, Moscow, Russia, 123098

_______________________________________________
3dem mailing list
3dem at ncmir.ucsd.edu<mailto:3dem at ncmir.ucsd.edu>
https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=OolwFF628X2ftLUpz8Ras6lKj2gYG24ZfiJOoYlUzP0&s=M4xY8AZbUX0K19ncPzZDEjD1hTrbXK6xsaTCOQGNQCk&e=

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181023/b7038a00/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Falcon Hacking.pdf
Type: application/pdf
Size: 3708892 bytes
Desc: Falcon Hacking.pdf
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181023/b7038a00/attachment-0001.pdf>