Blog‎ > ‎

CCIE DC LAB FC Fabric Service

posted May 13, 2014, 7:48 PM by Rick McGee   [ updated May 14, 2014, 7:56 PM ]
Fibre Channel networks are connection oriented
    Nodes must first register with the control plane of the fabric before sending any traffic

The Fabric Services Include
    FC Domain 
         Principle Switch Election and assignment of Domain ID's
        Fabric Routing between Domain ID's 
        End station registers with the fabric and are assigned FCID 
    FC Name Services or Directory Services
        Distributes FCID to pWWN/WWPN mappings throughout the fabric 
         Controls wich Initiators can talk to which Targets  


FC Domain 
    Principle Switch assigns Domain IDF's to Fabric 

Principle Switch Election 
    Based on lowest priority and SWWN (Switch World Wide Name)    
        Can view via "show wwn switch"
    Modify switch priority with "fcdomain priority 1 vsan 1" (PER VSAN BASIS)

Domain ID Selection 
    Dynamic form Principle Switch by default 
    Static as "fcdomain domain 0x51 [preferred | static ]
        Preferred requests Domain ID but will join without the assignment 
        Static requests Domain ID and isolates itself with the assignment 
            Would want to do this for easier Zoning 

FC Domain process restart     
    Graceful as "fcdomain restart vsan 1"
    Forced as "fcdomain restart disruptive vsan 1"
            Not Recommended, this will stop all SCSI reads and writes
Verified with:
    "show fcdomain [vsan]"
    "show fcdomain domain-list [vsan]"


show fcdomain domain-list on N5K1
Three instances of Fabric Services running on VSAN 1,10, and 30

On all switches create vsan 50

conf t
    vsan database 
    vsan 50

On TE link between MDS1 and MDS2 add to the allowed vsan list 

conf t
    int fc1/13 (MDS1) and fc1/10 (MDS2)
    switchport trunk allowed vsan add 50
Issue the command "show fcdomain domain list vsan 50"
You will see the four devices in the list N5k1, N5K2, MDS1, and MDS2

Issue the command "show fcdomain vsan 50"
This shows your local information with priority and the domain ID for vsan 50

MDS2 configure fcdomain ID
conf t
    fcdomain domain 0x54 preferred vsan 50

MDS1 configure fcdomain ID
conf t
    fcdomain domain 0x53 preferred vsan 50

N5K1 configure fcdomain ID
conf t
    fcdomain domain 0x51 preferred vsan 50

N5K2 configure fcdomain ID
conf t
    fcdomain domain 0x52 preferred vsan 50

On MDS1 
    fcdomain restart disruptive vsan 50 
        The disruptive is a hidden command, but will force all devices to log back into the fabric

show fcdomain domain-list vsan 50
You'll see the Domain ID's for all the switches have now changed


FC Fabric Services FSPF  
     Used to build an SPT (shortest path tree)through the fabric
        Similar logic to OSPF and IS-IS
        Domain-ID is the Node ID in the SPT 
        Occurs automatically on a per-VSAN basis
    Can be modified as:
        Global "fsfp config vsan 1"
        Interface fsfp [cost] dead-interval | hello-interval | passive |
              You usually don't want to or need to change these values
    Verified with:
        show fspf [database | interface] [vsan]

FSPF Topology 

    Show fspf 
You'll see each FSPF per vsan 

show fspf vsan 50 
You'll see from the output above the Local Domain is 0x51 (81) for the domain-id for N5k1 that we assigned to VSAN 50.

show fspf database vsan 50

show fcroute unicast vsan 50
The above output shows the total costs to all FC-ID's in the network 
So, to go to N5K1 (0x51) to N5K2 (0x51) it has a total cost of 1125

Fabric Login (FLOGI)
    Data Plane not routing on WWN but the FCID
    All Initiators and Target must FLOGI before sending any data into the fabric
    Verified as "show flogi database"

    No Configuration Needed

    No FLOGI indicates a basic link level negotiation problem 


show flogi database

This show 16 flogi's for the 16 Disk's in the JBOD, in a typical SAN design  you would have a RAID controller that presents RAID Volumes and present that as the LUN (logical unit)

This does show that the Layer 1 is working and that the disks are working as well.

show flogi database 
From the output you can see the WWPN of Server2. Remember you only see flogi's for devices that are local to that switch.

FC Name Services 
    FCNS (AKA Directory Services) keeps a mapping of FCID's to pWWN's/WWPN's
        Similar to IP ARP Cache
        FCID to WWPN
    End hosts/devices register with the FCNS after FLOGI
        "show flogi database" only shows local Initiators and Targets
        "show fcns database" shows everyone in the Fabric  
    If the Node did FLOGI but is not in everyone's FCNS, indicates the Fabric is broken
        E.g. VSAN is isolated, EISL allowed list is wrong, etc.

show fcns database 
The above output is a portion of the regular output that show vsan 10
    N Port is Server 2 and the NL ports are the JBOD disks