Skip to Content

Cisco UCS 6454 Fabric Interconnect Deep Dive

In our Introduction to Cisco UCS Architecture, we touched briefly on the Cisco UCS Fabric Interconnect.  Specifically we talked about the latest generation, the Cisco UCS 6454 Fabric Interconnect.  Now, I want to take a closer look at the function of the Cisco UCS Fabric Interconnect, and some things to think about when you are deploying it.

What is a Cisco UCS Fabric Interconnect?

A Cisco UCS Fabric Interconnect is the brains of the Cisco UCS platform.  If you did not already know, UCS is an acronym for “Unified Computing System”.  When we talk about the specifics of the Cisco UCS Fabric Interconnect, we are going to talk about the Cisco UCS 6454 Fabric Interconnect.

While some earlier models were larger (I’m looking at you Cisco UCS 6296), the Cisco UCS 6454 Fabric Interconnect is a 1 U Rack Mount device, and honestly, it looks like any other network switch.  Here is a diagram of the Cisco UCS 5454 Fabric Interconnect, with an expiation of what each port does:

Cisco UCS 6454 Fabric Interconnect 4th Generation

Like I said, it looks like a switch, but has so much more functionality, which we are about to go into in great detail.

What Does a Cisco UCS Fabric Interconnect Do?

The Cisco UCS Fabric Interconnect serves several main functions:

  • To uplink Cisco UCS Servers into your network
  • Provide redundancy for Cisco UCS Server connectivity
  • Manage the configuration of your Cisco UCS Servers with Cisco UCS Manager

Does it do even more than this?  Yes, but let’s start with the basics and move on from there.

Provide Cisco UCS Servers Redundant Connectivity to a Network

I have combined two of the bullet points above into a single statement to reflect the best practices of deploying a Cisco UCS environment.  While strictly speaking you could deploy a single Cisco UCS Fabric Interconnect (and I have seen it done in labs and small environments), it is not a good idea for large production environments that demand availability.

Why?  Simple.  We always want to deploy things with redundancy in mind.  If I had a single Cisco UCS Fabric Interconnect and it failed, my complete Cisco UCS environment would be down.  This is often a criticism of the Cisco UCS architecture, since it has such a large failure domain with this Cisco UCS Fabric Interconnects being so vital.

If we deploy them in pairs, and one Fabric Interconnect is down (or being rebooted to upgrade UCS Manager) we will still be operational.  If we architect things correctly, we should also NOT see any performance degradation.

A Sample Cisco UCS Architecture

Take a look at this sample Cisco UCS Architecture, where the Cisco Fabric Interconnects are deployed in pairs.  In most of my Cisco UCS experience, which relates to FlexPod, we called one Fabric Interconnect A and the other Fabric Interconnect B:

sample cisco UCS architecture ucs manager IOT nexus

 

If you look at the diagram, each component is connected to both Fabric A and Fabric B.  The connectivity flow goes something like this:

Cisco Server -> Cisco VIC A -> Cisco Blade Chassis FEX A-> Cisco Fabric Interconnect A -> Cisco Nexus Switch

It would do the same for the other side, meaning if we lost any component along that chain, the other side would simply pick up the slack, and we would not see any impact.

Sizing Considerations for the Cisco UCS Fabric Interconnect

When it comes to sizing considerations for the Cisco UCS Fabric Interconnect, there are two things to really consider:

  • Uplinks to the Cisco Nexus Infrastructure
  • Number of Ports Needed on the Cisco UCS Fabric Interconnect

Sizing the Cisco Fabric Interconnect Nexus Uplinks

The proper way to size the uplinks is to ensure all traffic can flow through a single uplink without service interruption in the event of a component failure.

This Cisco UCS will 6454 Fabric Interconnect has six 40/100 uplink ports.  This actually makes providing ample bandwidth very easy compared to previous generations.  In many cases, 40 Gbps of bandwidth (which would be two redundant 40 Gbps links, one to each Nexus, to survive failure of a Nexus device) may even be sufficient.

This is what a failure scenario looks like using redundant 40 Gbps uplinks into a Cisco Nexus environment:

cisco ucs fabric interconnect cisco nexus vpc failureThe rule of thumb is you aways want to be able to run at full bandwidth with the failure of any component along the chain, unless you accept the risk of degraded performance in the event of a component failure.

Number of Ports in the Cisco UCS 6454 Fabric Interconnect

If we make sure we can always run at peak bandwidth, you may get worried when you look at the number of ports on the Cisco UCS Fabric Interconnect.  Have no fear, Cisco has a solution for this problem, called a Cisco UCS Fabric Extender, which will tie into our second sizing discussion shortly:

Cisco FEX vs Cisco Fabric Interconnect

Sometimes, all the different components of a Cisco UCS configuration can get confusing.  Do not worry, since this is really easy.

Think of a Cisco FEX as a stupid port expander.  That is it!  By plugging a Cisco FEX into a Cisco Fabric Interconnect, you simply get more ports.  You can read more about Cisco Fabric Extender Technology (what the FEX acronym stands for) here.

There are two types of Cisco FEXs when it comes to Cisco UCS

  • Cisco UCS Top of Rack FEX
  • Cisco UCS 2200 Series Blade Chassis FEX

Cisco UCS TOR FEX

This FEX looks exactly like a Cisco Fabric Interconnect or a Cisco Nexus Switch.  It simply connects to the Fabric Interconnect to create more ports to connect things too.  That’s it!

Cisco UCS 2200 Series Blade Chassis FEX

The second type of FEX is mandatory in a Cisco Blade blade chassis, which you can see in the diagram above.  It allows the Cisco Virtual Interface Cards (Cisco VICs) an avenue to connect to the Cisco UCS Fabric Interconnect.  This brings me to my second sizing consideration…

Sizing Cisco UCS 2200 FEX Connections

Again, the most important consideration is do we accept degraded performance in the event of a failover?  Let’s take a look at how the FEX connects to the Cisco UCS Fabric Interconnect 6454 again:

cisco ucs fabric interconnect cisco 2200 blade fex

In this particular diagram, I have included the Cisco UCS 2208 FEX, which means each FEX has eight 10 Gbps connections to their respective Cisco Fabric Interconnect.  Conversely, there is also another FEX called the Cisco UCS 2204 FEX which has four 10 Gbps connections.

Remember, each Cisco UCS FEX is NOT cross connected to each Cisco Fabric Interconnect.  FEX A connects to Fabric Interconnect A, which means for maximum bandwidth during failure, the connections between FEX A and Fabric Interconnect A must be able to sustain the complete Cisco UCS chassis.

One popular method to deploying Cisco UCS is the “cable once” method where you connect all connections during initial deployment, and turn them on as needed.  This is a great way to be able to ensure you have bandwidth available if you need it.

Connecting a Cisco Blade Chassis to a Fabric Interconnect

As we mentioned, the Cisco Blade Chassis is connected to the Cisco Fabric interconnect with the Cisco UCS FEX 2200 as an intermediary.  This configuration is done via Cisco UCS Manager.

There are two ways to configure this connection:

  • PortChannel Mode – where the FEX connections are configured in a PortChannel
  • Discrete Mode – where a FEX connection is pinned to a Fabric Interconnect connection

Obviously, PortChannel Mode is the choice when it comes to availability, but you may have a use case for discrete mode.  You can learn more about how to configure Networking for the Cisco UCS Fabric Interconnect in the Cisco UCS Manager Network Management Guide.

If you want to get some practical experience working with Cisco UCS Manager, and configuring Cisco UCS Fabric Interconnect networking, be sure to take a look at the Cisco UCS Platform Emulator, which is a light weight VM that emulates a Cisco UCS Hardware environment with Cisco UCS Manager.

Want to learn more about Cisco UCS?  Check out these helpful guides:
Cisco UCS Architecture Overview
Cisco UCS C220 M5 Deep Dive (Cisco UCS Rack Server)

More Cisco UCS 6454 Fabric Interconnect Resources

Want to learn more about the Cisco UCS 6454 Fabric Interconnect?  Here are some great Cisco resources:

Cisco UCS 6454 Fabric Interconnect Spec Sheet – Everything you ever wanted to know about the Cisco Fabric Interconnect, and more!  This is your one stop shop for all the details.  If you are not familiar with Cisco Spec Sheets, be sure to check out this guide.

Cisco UCS 6454 Fabric Interconnect Hardware Installation Guide – Get the nuts and bolts on how the Cisco UCS Fabric Interconnect is physically installed.

Cisco UCS 6454 Data Sheet – Some great diagrams and specifications on the Cisco UCS 6454 Fabric Interconnect and associated components.

These are all fantastic resources to learn more about the Cisco UCS 6454 Fabric Interconnect.

Cisco UCS 6454 Fabric Interconnect Summary

The Cisco UCS 6454 Fabric Interconnect serves a few main purposes in your Cisco UCS Environment:

  • Providing redundant connectivity between components, including the upstream Cisco Nexus environment
  • Providing a point for configuration with Cisco UCS Manager (stay tuned for a Cisco UCS Manager deep dive!)

The Fabric Interconnect really is the heart and soul of your Cisco UCS environment.  Without a Fabric Interconnect, your Cisco UCS environment will cease to function.  This is why the Cisco UCS 5454 Fabric Interconnect is always deployed in pairs.  

Cisco UCS Manager makes it easy to configure the network connectivity for your Cisco UCS environment.  Cisco UCS Servers may be connected to the Fabric Interconnect directly in the case of Cisco UCS C-Series rack servers, or through a Cisco FEX, as in the case of Cisco UCS B-Series blade servers.  Either way, make sure to size your Cisco UCS network connectivity to meet your organizations requirements.