Blog‎ > ‎

Nexus 5K Fibre Channel SAN configuration

posted May 9, 2014, 1:08 PM by Rick McGee
Steps

Allocate interfaces as "type fc"
    slot 1
    port 25-32 type fc

FC Must start at the top of a module and count down
Requires config save and reload

Enable "feature fcoe"
    FC and FCOE features are not activated until the FC_FEATURES_PKG is enabled

Fibre Channel Interfaces 

Like Ethernet, FC Auto Negotiation is on by default

Configuration Parameters 
    
    Port Types
        switchport mode F,FL.E.SD,NP... 
    Trunk Mode
        switchport trunk mode on,off, or auto
    Speed
        switchport speed 1000,2000,4000, or 8000
Verify with the "show interface brier"

Fibre Channel Expansion Ports
    E_port 
    Inter Switch Link (ISL) 
On Certain MDS line cards, E pots cannot be auto-negotiated
    E_port requires dedicated rate-mode
    "switchport rate-mode dedicated"
        This may disable some ports that share the same ASIC
    Very much like the port groupings for Nexus 7K's 
    All Ports on a Nexus 5500 are dedicated rate mode
        If you set at 8Gbps they can send at 8Gbps 

FC Trunking Expansion Ports
    TE_port 
    Extended ISL and extend multiple VSAN's across links (similar to 802.1Q)
    
    Trunking negotiation is on for E ports
        trunk protocol enable
        show trunk protocol 
    Manually configured as combination of :
        switchport mode e | auto
        switchport trunk mode on | auto 
            there is no switchport mode te command
    


Overall Design

N5K1
show int brief 
 you'll see the "SFP validation failed" this is because there are Fibre Channel SFP's inserted into those slots that uses a different layer fibre channel 8b 10b encoding with a 20% overhead.

Also by default port 1-32 (on a 5548UP) are configured as Ethernet.

N5K1
show int 1/32 transceiver 

N5K1
conf t
slot 1
    port 25-32 type fc 
    end
copy run start
reload 
feature fcoe 

conf t
int  fc1/31-32 
    no shut
int fc1/27 - 28
     switchport mode e
    no shut   (remember these are all dedicated ports no need to set that mode)


N5K2
conf t
slot 1
    port 25- 32 type fc
    end
copy run start
reload
!
feature fcoe 

conf t
int  fc1/31-32 
    no shut
int fc1/29
    switchport mode e
    no shut 

MDS1 
conf t
int fc1/17 - 18
    description TO JBOD
    switchport mode fl
    no shut
int fc1/1 - 2
    switchport rate-mode dedicated
    switchport mode e 
    no shut 
int fc1/13
    switchport rate-mode dedicated
    switchport mode e
    no shut 

MDS2
conf t
int fc1/13 - 14
    description TO JBOD
    switchport mode fl
    no shut 
int fc1/5
    switchport rate-mode dedicated
    switchport mode e
    no shut 
int fc1/10
    switchport rate-mode dedicated
    switchport mode e
    no shut 
    

N5K1 
show interface brief 
You'll notice that the ports stop at 25, because we haven't enabled the feature fcoe yet.

after feature fcoe

show interface brief 

Bring up fc1/31-32 with a no shut on N5K1 and N5K2 

show int fc1/31-32 brief 
The port auto negotiated the mode to F as the Server 3 and Server 2 are N ports.

Next we bring up the MDS 1 and 2 ports to the JBODS and verify with show interface brief

fc1/17 - 18 are connecting to the JBOD that only support FC-AL (Arbitrated Loop)

After you verify that you can issue the "show flogi database" to see if the LOCAL host can be seen.

MDS2 show flogi database
remember that in this JBOD each disk sends it's own FLOGI to the SAN switch.

Now bring up the Ports between the MDS's  and N5K's 

On N5K1 issue the show interface fc1/27 - 28 
You can see the status is trunking and the Mode is TE for trunking expansion 

Now bring up the the ports between MDS1 and MDS2 

show int fc1/10 brief 
you can see that it's up as an E port and operating as a TE expansion port

Comments