[3dem] Falcon II uses only half of the 1Gb/s bandwidth.

Tue Oct 23 07:41:51 PDT 2018

Dear All,

Thank you for your excellent suggestions regarding the issue.

Mike Strauss was right about EPU writing stacks from Falcon II to the
Microscope PC's HDD before transferring it to the destination folder on
Support PC. This is an obvious bottleneck, but I think FEI did this for a
good reason (Win 7 upgrade was probably implemented at the very end of the
product lifecycle, so there was no point in making the old product almost
as good as the new one).
Thanks to Mike, I was able to locate the folder where intermediate frames
are saved: *C:/Users/<username>/AppData/Local/Temp/FalconInterImages/ *
However, *IntermediateConfig.xml* points to the different location, not the
one that is currently used (you'll find the attachment below). I've tried
to find .XML files with the same name, but haven't found any, except for
those that were in the backup copy made during the migration from Win XP to
Win 7.

Now let me address your questions one at a time:

What type of disks are on Camera Support Rack #1?

Camera Support Rack #1 is a "black box" - I don't have a direct access to
it, so no info on the type of the memory used and/or additional
computations performed on it.

> You are talking about the image transfer being performed by the FEI
> software, rather than copying a file via SMB?

Yes, correct. As indicated on the network map, 1Gbps channel ("Support
network") used for transferring via SMB is fully utilised.

Matthias, thank you for pointing out the possible workarounds, I've
contacted Chen Xu already. I didn't know the camera can go to 32 fps or
higher, that's fascinating!

Steven, of course you are right about the switch. It isn't (probably) a
cause of the problem itself (since 1Gb/s Ethernet has been around for 20
years or so). My guess was that it's configured this way to "cap" incoming
1Gbps channels because there is only one outgoing 1Gbps channel.

Regards,

Eugene

On Thu, Oct 18, 2018 at 10:40 PM Ludtke, Steven J <sludtke at bcm.edu> wrote:

> You know, I think I misunderstood the question. You are talking about the
> image transfer being performed by the FEI software, rather than copying a
> file via SMB?   If so, then almost certainly nothing to do with your
> hardware at all. Just poor software design in handling the network transfer
> by FEI.
>
>
>
> --------------------------------------------------------------------------------------
> Steven Ludtke, Ph.D. <sludtke at bcm.edu>                      Baylor
> College of Medicine
> Charles C. Bell Jr., Professor of Structural Biology
> Dept. of Biochemistry and Molecular Biology                      (
> www.bcm.edu/biochem)
> Academic Director, CryoEM Core                                        (
> cryoem.bcm.edu)
> Co-Director CIBR Center                                    (
> www.bcm.edu/research/cibr)
>
>
>
> On Oct 18, 2018, at 1:29 PM, Matthias Wolf <matthias.wolf at oist.jp> wrote:
>
> ****CAUTION:*** This email is not from a BCM Source. Only click links or
> open attachments you know are safe.*
> ------------------------------
> Hi Gene,
>
> Don't bother with the inadequate FEI microscope PC and its gigabit network.
>
> For proper performance, you have 2 options:
>
>
>    1. get FEI's storage server and frame saving hardware (CMTS) with
>    dedicated 10Gbit/s upload. It'll cost you $$$. Needs Win7 on the scope PC.
>
>    2. set up Greg McMullans brilliant Falcon frame capture, which uses a
>    beam splitter (fiber optical tap) to duplicate the raw camera signal and
>    capture it on a dedicated linux PC with fast SSD. Total investment ~3k
>    US$. This works on older Win XP systems, but Greg has also written code for
>    Win7 (FEI changes something in the Falcon data protocol upon upgrade to
>    Win7). You may want to look up Chen Xu's "falcon hacking" page at Brandeis
>    on Internet Waybackmachine. That way, the FEI system is completely bypassed
>    and your movie frames will be happily saved on the capture PC before you
>    even get an image in TIA. I have done this with good success for several
>    years until option 1 was installed in the course of a Win7+Falcon3 upgrade.
>    You must obtain specific Solarflare NICs for Greg's code to work and should
>    follow his instructions to the letter (but best don't bother him and go
>    with option 1)!
>
>
> Image size is 16 Mpixel * 14bit/pixel. At 18 fps, the data rate is ~4 Gb/s.
> Decent hardware can deal with this. The camera can go to 32 fps or
> even higher, but then everything on the linux O/S must be tuned or you drop
> frames.
>
> Without frame capture, you won't get decent performance - both number of
> frames and frame rate will be severely limited and not up to current
> standards of cryo-EM movie data collection.
>
> As Steve wrote, your gigabit switch is unlikely to throttle bandwidth.
> But the raw camera data is sent in jumbo packets. Maybe your switch is
> using standard MTU of 1500?
> You could try changing this to use jumbo packets of 9014 bytes with telnet
> and the CLI.
>
>    Matthias
>
> ------------------------------
> *From:* 3dem <3dem-bounces at ncmir.ucsd.edu> on behalf of Eugene Pichkur <
> eugene.pichkur at gmail.com>
> *Sent:* Tuesday, October 16, 2018 10:15:00 PM
> *To:* 3dem at ncmir.ucsd.edu
> *Subject:* [3dem] Falcon II uses only half of the 1Gb/s bandwidth.
>
> Hello 3DEM community,
>
> We've noticed that our Falcon II uses only half of the 1Gbps bandwidth to
> transfer data from the Camera Support Rack to the Microscope PC (I've
> attached a screenshot to illustrate the issue).
>
> Microscope PC runs Windows 7 and we are able to get the full frame rate
> (20fps) from the detector, however, it takes ~20s to transfer a typical
> movie (40frames, 1.2GB) to the PC which is a very frustrating bottleneck.
>
> Both Falcon II and Ceta are connected through the same 1Gb switch (HPE
> OfficeConnect 1420 24G 2SFP), so my guess is that the bandwidth is split
> 50/50. However, during data collection only Falcon II is used which makes
> this limitation pointless, not to mention that Falcon II and Ceta can't be
> used simultaneously.
>
> Does anyone have a workaround or any info on the topic?
>
>
> Thanks in advance,
> Eugene Pichkur
>
> National Research Centre “Kurchatov Institute”
> Akademika Kurchatova, 1, Moscow, Russia, 123098
>
>
>
>
>
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.ncmir.ucsd.edu_mailman_listinfo_3dem&d=DwICAg&c=ZQs-KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=OolwFF628X2ftLUpz8Ras6lKj2gYG24ZfiJOoYlUzP0&s=M4xY8AZbUX0K19ncPzZDEjD1hTrbXK6xsaTCOQGNQCk&e=
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181023/57ea55bf/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: FalconInterImages.png
Type: image/png
Size: 69601 bytes
Desc: not available
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181023/57ea55bf/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IntermediateConfig.png
Type: image/png
Size: 21497 bytes
Desc: not available
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20181023/57ea55bf/attachment-0003.png>