Blog‎ > ‎

CCIE DC Storage Networking Converged Ethernet (FCOE)

posted May 31, 2014, 8:25 AM by Rick McGee   [ updated Jun 3, 2014, 7:01 PM ]
Tons of terms that mean the same thing
    Unifed Fabric (Cisco)
    Unified Wire
    Converged Ethernet 
    Converged Enhanced Ethernet 
    Data Center Ethernet 
    Data Center Bridging 
        They are refer to the same infrastructure, Ethernet and Fibre Channel on the same wire

FCoE Terminology 

    FCoE Initialization Protocol (FIP)
        Control Plane for FCoE Initiator and FCoE switch 
    FCoE Forwarder (FCF) (Nexus 5K, 7K, or MDS with FCoE blade)
        Running the Ethernet and FC switching stack 
            FIP Snooping Bridge (NPV don't run full FC stack)

        FCoE node port (Ethernet attached)

    Virtual Fibre Channel (VFC) interface
        Logical interface presented to a Ethernet interface (like iSCSi interface)
        Where FIP run's between the ENode and VFC

    Virtual Port Types
             Same as N port in FC       
              Same as F port in FC
        VE_Portn (Multi Hop FCoE)
               Same as E port in FC 

How FCoE Works

    FCoE replaces Layer 1 and Layer 2 transport for FC

    All upper layer FC services remain
        Domain ID's, FSPF, FCNS, FLOGI, Zoning, etc....

    New FCoE Initialization Protocol (FIP) to negotiate between the Fabric and Node 
        Fabric is the FCF 
        Node is the ENode

    FIP is the control plane of FCoE
        New EtherType 0x8914
        Used to disvoer FCF's and perform FLOGI

    FCoE is the actual data plane
        New EtherType 0x8906
        Max length of 2240 bytes 
        FC-BB-5 T11 Org.
        Norman FC Payload is 2112
            FC Framing----> FCoE Framing-----> Ethernet 
        Implies Jumbo Frames are required

FCoE Addressing 
        Fibre Channel uses 3 byte FCID's 
            Domain, Area, WWPN

        Ethernet uses a 6 byte MAC Addresses 

        How to correlate the two?

        FCoE ENode gets a Fabric Provided MAC (FPMA) for FCoE 
            Still uses it's regular MAC for LAN traffic

Fabric Provided MAC Addresses (FPMA)
    During FIP ENode is assigned a 3 byte FCID (Domain, Area, and WWPN)
        FLOGI is part of FIP

    FCF is configured with a 3 byte FCoE MAC address Prefix (FC-MAP)
    ENodes appends FC-MAP to FCID
    3 byte FC-MAP + 3 byte FCID = 6 byte FPMA

Configure FCoE 
    Create VSAN 
     VSAN to a VLAN
    Configure a VFC 
    Associate physical Ethernet 

FCoE Topology 

Single Hop Topology 

N5K1 and N5K2 (assuming there is a basic config)
feature fcoe
feature lldp (this has a subset of DCB extensions ETS and PFC)
conf t
slot t
port 25 - 32 type fc
copy run start
Want to have the FC up first before the FCoE configuration 

conf t
int fc1/27 - 28
no shut
vsan database 
vsan 1010
zone mode enhanced vsan 1010 (make sure the trunking is working show int fc1/27 - 28 trunk vsan)
vlan 1010
fcoe vsan 1010
int vfc117
bind interface e1/17 (make sure int e1/17 is shutdown)
vsan database
vsan 1010 interface vfc117
int e1/17
switchport mode trunk
spanning-tree port type edge trunk 
switchport trunk native vlan 10
switchport trunk allowed vlan 10,1010
no shut

show flogi database 
you see after e1/17 is up and associated with vfc117 that it's interface send a flogi to join the fabric just the same as any other fc interface. the port name is the actual WWPN of the server
This also shows that FCoE FIP (control plane worked)

show int vfc 117
VSAN 1010 is up and trunking is up

device-alias database
device-alias name SERVER1_FCOE_SAN_A pwwn 10:00:00:00:c9:bb:19:9f
device-alias commit

show zoneset active vsan 1010
This zoneset was created on MDS1 and distributed through enhanced zoning fabric wide 

conf t
int fc1/29
no shut
vsan database
vsan 1020
zone mode enhanced vsan 1020 (make sure the trunking is working show int fc1/29 trunk vsan)
vlan 20
interface e1/17
switchport mode trunk
switchport trunk native vlan 20
switchport trunk allowed vlan 20,1020
spanning-tree port type edge trunk
interface vfc117
bind interface ethernet1/17
switchport trunk allowed vsan 1020
no shut
interface e1/17
no shut

zone mode enhanced vsan 1010
device-alias database
device-alias name JBOD0_DISK13_SAN_A_PWWN 21:00:00:04:cf:f3:73:5F
device-alias name JBOD0_DISK14_SAN_A_PWWN 21:00:00:04:cf:84:b7:9b
device-alias name JBOD0_DISK15_SAN_A_PWWN 21:00:00:04:cf:84:48:77
device-alias commit
vsan database 
vsan 1010 interface fc1/17
int fc1/1 -2 
switchport rate-mode dedicated
switchport mode e
no shut
zoneset name VSAN1010 vsan 1010
member device-alias JBOD0_DISK13_SAN_A
member device-alias JBOD0_DISK14_SAN_A
member device-alias JBOD0_DISK15_SAN_A
member device-alias SERVER1_FCOE-SAN_A
zoneset activate name VSAN1010 vsan 1010
zone commit vsan 1010 (you should now be able to see the active zoneset on N5K1)

zone mode enhanced vsan 1020
device-alias database
device-alias name JBOD0_DISK13_SAN_B_PWWN 22:00:00:04:cf:f3:73:5F
device-alias name JBOD0_DISK14_SAN_B_PWWN 22:00:00:04:cf:84:b7:9b
device-alias name JBOD0_DISK15_SAN_B_PWWN 22:00:00:04:cf:84:48:77
device-alias commit
vsan database 
vsan 1020 interface fc1/13
int fc1/5
switchport rate-mode dedicated
switchport mode e
no shut

overall configuration 
feature fcoe
feature lldp 
vsan databases 
    vsan 1010
vlan 1010
    fcoe vsan 1010
    vsan 1010 interface vfc117
interface vfc117
    bind interface e1/17
    switchport trunk allowed vsan 1010
    no shut
interface e1/17
    switchport mode trunk
    switchport trunk native vlan 10
    switchport trunk allowed 10,1010
    spanning-tree port type edge trunk 

FCoE status commands

show fcoe
FC-MAP is the first 3 bytes 0e:fc:00 that will make up the FPMA 

QoS for FCOE
You have to be careful because some of the NX-OS releases don't enables these service-policies by default. With version 5.1(3) when you enter "feature fcoe" it will enable the FCOE policies. There are 4 different policies that are applies

show policy-map type qos
On the ingress if it's FCOE (data plane) traffic it will be set to the qos-group to 1

show class-map type qos

show policy-map type queuing 
Queuing FCOE is for 50 percent of the link. This is only enforced when congestion is present 

show class-map type queuing

show policy-map type network-qos
If class is fcoe set to be able to pause, no drop class, and the mtu is 2158 (minimum is 2112 for FCOE ethernet herder plus  fcp and payload no greater then 2158)

for DCBX the ETS is used for the % of bandwidth that can be used per queue and the PFC allows the Pause frame to be set.

show interface ethernet 1/17 priority-flow-control

This output will show you your Pause fames sent and received. If you are sending large amounts of pause frame it could mean that your Server hardware CPUnot fast enough and or an application issue.


FIP Gen1 CNA's are not support on Nexus 2232 FEX's 

show system internal dcvx info interface e1/17
Converged Enhanced Ethernet (CEE) GEN 2 CNA 
Cisco Intel Nuovo (CIN) GEN 1 CNA