Blog‎ > ‎

CCIE DC LAB FC Fabric Service

posted May 13, 2014, 7:48 PM by Rick McGee   [ updated May 14, 2014, 7:56 PM ]
Fibre Channel networks are connection oriented
    Nodes must first register with the control plane of the fabric before sending any traffic

The Fabric Services Include
    
    FC Domain 
         Principle Switch Election and assignment of Domain ID's
    
    FSFP
        Fabric Routing between Domain ID's 
    
    FLOGI
        End station registers with the fabric and are assigned FCID 
    
    FC Name Services or Directory Services
        Distributes FCID to pWWN/WWPN mappings throughout the fabric 
    
    Zoning 
         Controls wich Initiators can talk to which Targets  

FC DOMAIN 

FC Domain 
    Principle Switch assigns Domain IDF's to Fabric 

Principle Switch Election 
    Based on lowest priority and SWWN (Switch World Wide Name)    
        Can view via "show wwn switch"
    Modify switch priority with "fcdomain priority 1 vsan 1" (PER VSAN BASIS)

Domain ID Selection 
    Dynamic form Principle Switch by default 
    
    Static as "fcdomain domain 0x51 [preferred | static ]
        Preferred requests Domain ID but will join without the assignment 
        Static requests Domain ID and isolates itself with the assignment 
            Would want to do this for easier Zoning 

FC Domain process restart     
    Graceful as "fcdomain restart vsan 1"
    Forced as "fcdomain restart disruptive vsan 1"
            Not Recommended, this will stop all SCSI reads and writes
Verified with:
    "show fcdomain [vsan]"
    "show fcdomain domain-list [vsan]"

  
Design 

show fcdomain domain-list on N5K1
Three instances of Fabric Services running on VSAN 1,10, and 30

On all switches create vsan 50

conf t
    vsan database 
    vsan 50

On TE link between MDS1 and MDS2 add to the allowed vsan list 

conf t
    int fc1/13 (MDS1) and fc1/10 (MDS2)
    switchport trunk allowed vsan add 50
    
Issue the command "show fcdomain domain list vsan 50"
You will see the four devices in the list N5k1, N5K2, MDS1, and MDS2

Issue the command "show fcdomain vsan 50"
This shows your local information with priority and the domain ID for vsan 50

MDS2 configure fcdomain ID
conf t
    fcdomain domain 0x54 preferred vsan 50

MDS1 configure fcdomain ID
conf t
    fcdomain domain 0x53 preferred vsan 50

N5K1 configure fcdomain ID
conf t
    fcdomain domain 0x51 preferred vsan 50

N5K2 configure fcdomain ID
conf t
    fcdomain domain 0x52 preferred vsan 50

On MDS1 
    fcdomain restart disruptive vsan 50 
        The disruptive is a hidden command, but will force all devices to log back into the fabric

show fcdomain domain-list vsan 50
You'll see the Domain ID's for all the switches have now changed

     
FSFP

FC Fabric Services FSPF  
   
     Used to build an SPT (shortest path tree)through the fabric
        Similar logic to OSPF and IS-IS
        Domain-ID is the Node ID in the SPT 
        Occurs automatically on a per-VSAN basis
    
    Can be modified as:
        Global "fsfp config vsan 1"
        Interface fsfp [cost] dead-interval | hello-interval | passive |
        retransmit-interval]
              You usually don't want to or need to change these values
           
    Verified with:
        show fspf [database | interface] [vsan]

FSPF Topology 


N5K1 
    Show fspf 
You'll see each FSPF per vsan 

show fspf vsan 50 
You'll see from the output above the Local Domain is 0x51 (81) for the domain-id for N5k1 that we assigned to VSAN 50.

show fspf database vsan 50

show fcroute unicast vsan 50
The above output shows the total costs to all FC-ID's in the network 
So, to go to N5K1 (0x51) to N5K2 (0x51) it has a total cost of 1125


Fabric Login (FLOGI)
    Data Plane not routing on WWN but the FCID
    All Initiators and Target must FLOGI before sending any data into the fabric
    
    Verified as "show flogi database"

    No Configuration Needed

    No FLOGI indicates a basic link level negotiation problem 

Topology 


MDS1 
show flogi database

This show 16 flogi's for the 16 Disk's in the JBOD, in a typical SAN design  you would have a RAID controller that presents RAID Volumes and present that as the LUN (logical unit)

This does show that the Layer 1 is working and that the disks are working as well.

N5K1
show flogi database 
From the output you can see the WWPN of Server2. Remember you only see flogi's for devices that are local to that switch.

FC Name Services 
        
    FCNS (AKA Directory Services) keeps a mapping of FCID's to pWWN's/WWPN's
        Similar to IP ARP Cache
        FCID to WWPN
    
    End hosts/devices register with the FCNS after FLOGI
        "show flogi database" only shows local Initiators and Targets
        "show fcns database" shows everyone in the Fabric  
    
    If the Node did FLOGI but is not in everyone's FCNS, indicates the Fabric is broken
        E.g. VSAN is isolated, EISL allowed list is wrong, etc.


N5K1 
show fcns database 
The above output is a portion of the regular output that show vsan 10
    N Port is Server 2 and the NL ports are the JBOD disks
Comments