Blog‎ > ‎

CCIE SAN LAB Multihop FCoE

posted Jun 4, 2014, 7:53 PM by Rick McGee   [ updated Jun 9, 2014, 7:24 PM ]
Single Hop FCoE Topology 
This is assuming that you have an investment in Fibre Channel environment and you want to start at the Servers to migrate to 10GE CNA adapters and still maintain your Fibre Channel network. This kind of design will allow you to migrate to FCoE in a phased approach.


MultiHop FCoE Topology 



MultiHop FCoE Configuration.....

N7K1-1

conf t
hostname N7K1-1
no password strength-check (wouldn't do this on a production network)
username admin password cciedc01
username admin role network-admin
int mgmt 0
ip address 192.168.0.71/24
no shut
feature telnet 

For some NX-OS software versions if you need to setup SAN and LAN administrators you will have to follow a template that can be found at FCoE RBAC Template link 

With FCoE in the Nexus 7K's you have to add a VDC just for storage

install feature-set fcoe

once you install the feature set fcoe you need to apply the license to a module

issue the "show module" command to see what module can be configured for fcoe
In this example you can install the license to F1 Module in slot 2

license fcoe module 2  ("show license usage" to make sure the license is active)


vdc N7K1-SAN type storage 
if you don't first add the FCoE QoS policy (no drop policy) you will run into this error
and it will take around 10 minutes for the VDC creation to timeout 

show policy-map type network-qos 
You will want to have all FCoE to land in the c-nq-7e-nodrop-fcoe class. You'll need to take this pre-defined policy and activate it.

conf t
system qos
service-policy type network-qos default-nq-7e-policy
You can activate on any of the nondrop CoS classes 

show class-map type network-qos
It's matching o the ether type 0x8906 (FCoE) and the CoS 3 layer 2 marking

Once the VDC is up (verify with the "show vdc") you can allocate ports to it.

conf t
no vdc combined-hostname
vdc N7K1_SAN 
allocate interface e2/3 - 6
confirm yes
allocate fcoe-vlan-range 1002 1099 from vdcx N7K1-1
end

switchto vdc N7K1-SAN
username admim password ccie01dc0
username admin role vdc-admin
feature telnet
int mgmt 0 
ip address 192.168.0.72/24
no shut
exit

vlan 1010
vsan database 
vsan 1010
fcoe vsan 1010 (combining VSAN and VLAN 1010)

feature lacp 
int e2/5 -6 
channel-group 10 mode active 
int po10 
switchport mode trunk 
switchport trunk allowed vlan 1010
int vfc 10
bind interface po10 
switchport allowed vsan 1010
switchport mode e
int po10 
no shut
int e2/5- 6
no shut


MDS1 
show int brief  | ex sfpAb
You can see form this example that fc1/17 - 18 are up to the JBOD and the FL (loop) and in VSAN 1010 

show flogi database 
As you can see from this example the disks sent their flogi to register with the fabric

N5K1 
zone mode enhanced vsan 1010
conf t
int fc1/27 - 28 
no shut 
vsan database 
vsan 1010

show fcns database vsan 1010
This example now show that you have connectivity to the fabric and can see the target disks in the JBOD


N5K2
con t
vsan database 
vsan 1010
vlan 1010
fcoe vsan 1010
int e1/17
shutdown (enable after all other configurations)
switchport mode trunk
spanning-tree port type edge trunk
switchport port trunk allowed vlan 10,1010
switchport trunk native vlan 10 (access vlan)

int vfc117
bind interface e1/17
switchport trunk allowed van 1010
no shut 
vsan database
vsan 1010 interface vfc117

int e1/17
no shut

show flogi database 
This example shows that the Enode Server1 had registered 

conf t

feature lacp
int e1/10 - 11
channel-group 10 mode active
int port10
switchport mode trunk
switchport trunk allowed vlan 1010 (only storage traffic)
int vfc 10
bind interface port-channel 10
switchport funk allowed vsan 1010
switchport mode e
no shut
int e1/10 - 11 
no shut 

show int brief 

show vfc 10

N7K1-SAN
show port-channel summary 
From this output you can see that the port-channel form N7K-SAN is up between itself and 

N5K2
show int vfc 10
Form this example you can see that the port is an TE (Virtual Trunking Expansion) to N7K1-SAN 

No we can verify with FCNS that it can see server1 FCID and PWWN

N7K1-SAN
show fcns database

N5K1

feature lacp

!

vlan 1010

     fcoe vsan 1010 

vsan database

    vsan 1010

!

interface port-channel20

switchport mode trunk

switchport trunk allowed vlan 1010

!

interface vfc20

bind interface port-channel20

switchport mode e

switchport trunk allowed vsan 1010

no shutdown

!

interface Ethernet1/8

shut 

switchport mode trunk

switchport trunk allowed vlan 1010

channel-group 20 mode active

!

interface Ethernet1/9

shut

switchport mode trunk

switchport trunk allowed vlan 1010

channel-group 20 mode active




N7K1-SAN (To N5K1)

conf t

zone mode enhanced vsan 1010

!

interface port-channel20

description TO N5K1 FCoE

switchport

switchport mode trunk

switchport trunk allowed vlan 1010

!

interface vfc20

bind interface port-channel20

switchport mode e

switchport trunk allowed vsan 1010

no shut

!

interface Ethernet2/3

switchport mode trunk 

switchport trunk allowed vlan 1010

channel-group 20 mode active

!

interface Ethernet2/4

switchport mode trunk 

switchport trunk allowed vlan 1010

        channel-group 20 mode active


Bring up N5K1 e1/8 - 9 
no shut

N7K1-SAN 


show interface brief

From the output you can see the vfc 10 and 20 are turning and VTE ports


show fcns database 

You can see all the disks from MDS1 in the Fibre Channel network 



FCoE NPV Configuration



FCoE NPV Topology 



N5K2

conf t

feature npv

save and reload yes


conf t

vsan database 

vsan 1010

vlan 1010

fcoe vsan 1010

exit


int e1/17

switchport mode trunk

spanning-tree port type edge trunk

switchport trunk all vlan 10,1010

switchport trunk native valan 10


int vfc117

bind interface e1/17

vsan database

vsan 1010 interface vfc117


int vfc117

no shut


int e1/17

no shut 


feature lacp

int e1/10 - 11

channel-group 10 mode active


int po10

switchport mode trunk

switchport trunk allowed vlan 1010


int vfc10

bind inter port-channel 10

switchport mode np 

switchport trunk allowed vsan 1010

no shut


This output shows that N5K2 is acting as a FIP snooping bridge and see's Server1 pWWN



N7K1-1

switchto vdc N7K1-SAN

conf t

int vfc10

switchport mode f

feature npiv (NPV Core Switch)


Port-Channel 10 was already created




ĉ
Rick McGee,
Jun 6, 2014, 8:34 PM
Comments