Blog‎ > ‎

FabricPath

posted Nov 12, 2013, 10:16 AM by Rick McGee   [ updated Apr 16, 2014, 7:34 PM ]
FabricPath
    Layer 2 Routing (MAC in MAC Routing Encapsulation)

Alternative to running STP
    Even with vPC's you are still subject to Spanning-Tree
        vPC is a physical triangle and logical P2P link
    FP allows you to build arbitrary topologies
        Full mesh, partial mesh, triangle, square, etc....

FabriPath Terms

    Classical Ethernet (CE)
    Regular Ethernet with regular flooding, and running spanning-tree.

Leaf Switches
    Connect CE domain to FabricPath Domain

Spine Switch
    Fabric Path backbone switch with all ports in the FP domain only

FP Core Ports
    Links on Leaf Up to Spine, or Spine to Spine
    i.e. the "switchport mode fabricpath" links

CE Edge Ports
    Links on Leaf connecting to regular Classical Ethernet Domain
    
 
 
FabricPath Control Plane
    IS-IS is used in the FP core for L2 Routing
    Goal is to compute SPT between al FabricPath nodes (Shortest Path Tree LSA built SPT)
        IS-IS is not used for MAC address advertisements

Advantages of IS-IS
    Uses it's own Layer 3 Transport
        i.e. IP is not required
        Part of the CLNP or CLNS not IPv4 or IPv6 (Connectionless mode Network Protocol)
    Natively extensible
        i.e. support new TLV (Time Link Values)
        Advertises FabricPath Switch ID's FP Forwarding Tags
    Natively Supports ECMP
        Means Layer 2 load balancing without STP, Port-Channel, or vPC

FabricPath Switch ID
    12-Bit ID of the node in the IS-IS SPT
    Generated automatically generated
    For Verification and troubleshooting recommended to manually assign
        "fabricpath switch-id"
        "show fabricpath switch-id" 

FabricPath Data Plane
    
CE Frames are encapsulated with new FabricPath header
    FabricPath is NOT Ethernet
        Hardware the supports FabricPath is limited to Nexus F1 and F2 (own VDC) module and    
        Nexus 5500/5600 series switches
            Separate License Feature 

    FabricPath is not TRILL (Transparent interconnection of lot of links)
       TRILL is the open standard
        FabricPath is Cisco proprietary 
        Similar logic but NOT interoperable

FabricPath has the SRC and DST FabricPath Switch ID's in the Header
    Traffic is Layer 2 Routed via the SPT to DST Switch ID
    Same exact logic as Layer 3 IS-IS or OSPF routing
 
 
FabricPath Switch ID
    12-Bit ID of the node in the IS-IS SPT
    Generated automatically generated
    For Verification and troubleshooting recommended to manually assign
        "fabricpath switch-id"
        "show fabricpath switch-id"
 
FabricPath Subswitch ID
    Used for vPC+
        Combination of the vPC and FabricPath at the same time
    sSID identifies the source or destination vPC+ Port-Channel Interface
        vPC+ peers share the same SID but have a unique sSID
 
FabricPath Multi-destination Trees
    Unicast traffic is layer 2 routed based on Switch ID
        Uses the IS-IS SPT to the destination Switch ID
    Broadcast, Multicast, and Unknown unicast
        These types are forwarded based on a pre-built multi-destination trees
    Multi-destination tree root is elected based on the following:
        Highest root priority
            This is a 8 bit value between 0-255, the default value is 64
        Highest System ID
            This is a 48 bit VDC MAC address
        Highest Switch ID
            12 bit switch ID
 
FabricPath TTL
    Ethernet frames don't include TTL
        Ethernet is subject to infinite data plane loops (e.g. broadcast storms)
    FabricPath adds TTL to avoid infinite loop in the case of a control plane error
        TTL for version 5.1(6) and before is 16
        TTL for newer releases is 32
 
FabricPath MAC Learning
   
    Conversational MAC Learning (Default mode for FabricPath VLAN's)
        Only learn SRC MAC if you already know DST MAC
        Optimization of the control plane but NOT of the data plane
        Can be enabled for CE VLAN's
        "show mac address-table learning mode"
 

Traditional MAC Learning

        Learn SRC MAC of all received traffic
        Flood traffic to elicit response from DST
        Learn SRC MAC of DST from it's response

ARP Request who is 10.0.0.2 Nexus 5K1 learns MAC A (CAM Table Building). Nexus 5K 1 send out a broadcast to DST F.F.F.F (broadcast) which Nexus 5K 2 receives and sends out all ports and leans MAC A + B and sends back to Nexus 5K1

Builds Table

MAC|PORT

    A    1

    B    2

 

FabricPath doesn't knows about the local MAC, until a bidirectional communications. Nexus 5K 1 sends out to FabricPath Multipath Tree. Now Nexus 5K 2 learns S1 via Nexus 5K 1 Switch ID via unicast forwarding. Next hop is not a port, but rather a FabricPath Switch ID

 
FabricPath and STP Interaction
    FabricPath Leaf Switches MUST be STP Root for CE VLAN's 
        Ports will become Root Inconsistent via RootGuard otherwise
    Leaf switches should have the same priority and lowest (e.g. 4096)
        FabricPath switches share the common bridge ID C84C.75FA.6000
 
FabricPath Configuration with a Very few commands
       
        Enable FabricPath
                Install feature-set fabricpath
                feature-set fabricpath
        Configure FabricPath VLAN's
            mode fabricpath under the VLAN's
            (This will change to conversational
            MAC Learning)
        Configure FabricPath Core Ports
            "switchport mode fabricpath"
    
vPC + FabricPath (vPC+)
    Has to have PeerLink
    "Switchport mode FabricPath"
    !Disrupted Change!
    F1,F2, or F2E module only
 
on vPC 
    FabricPath Switch-ID 1000
    vPC+ on PeerLink seen as one node in IS-IS (FP)   



Configuration
Overall Design

Remember with the FabricPath you don't have to create port-channels

Configuration
   
    N5K1
    term mon
    feature LACP
    conf t
    int vlan 10
 int e101/1/1
    description R3
    switchport access vlan 10
    shut
    speed 1000 (2232 FEX that doesn't auto negotiate)
    no shut
 int e101/4 - 5
    description Server1 
    channel-group 10 mode active
    int po10
    switchport access vlan 10switswitchport access vlan 10
int e1/1
    description LINK TO 5K2
    switchport mode trunk    
    spanning-tree port type network
    no shut    
switchport access vlan 10switchport access vlan 10
     
   
    N5K2
    term mon
    feature LACP
int 102/1/1
    description R2
    switchport access vlan 30
    shut
    speed 1000
    no shut
int e101/1/6 -7
    description Server3
    channel-group 30 mode active
    int po30
    switchport access vlan 30
int e1/1
    description LINK TO 5K1
    switchport mode trunk    
    spanning-tree port type network
    no shut

R2
    int gig0/0
   
mac-address 0000.0000.0003mac-address 0000.0000.0002
   ip address 10.0.0.2 255.255.255.0
R3
    int gig0/0
    mac-address 0000.0000.0003
    ip address 10.0.0.3 255.255.255.0
 
Ping 10.0.0.2

N5K1
Show mac address-table dynamic vlan 10
    

N5K1
    conf t
    install feature-set fabricpath
    feature-set fabricpath
    vlan 10
    mode fabricpath
    int e1/1 (Remember this is the connection between the two N5K's)
    switchport mode fabricpath
N5K2
    conf t
    install feature-set fabricpath
    feature-set fabricpath
    vlan 10
    mode fabricpath
    int e1/1
    switchport mode fabricpath

sh run | in fabricpath



show fabricpath isis adjacency
    

You can see it's formed an adjacency with N5K2

You can see now that N5K1 only now knows about the directed connected link
for R3. This helps cut down on control plane traffic between the N5K's

The fabricpath switches will continue to flood unsolicited ARP's, but will not place the SRC in their MAC tables because it doesn't know the DST FFFF.FFFF.FFFF
    
    
Advanced Fabricpath Design

On Nexus 7K-1 through 7K-4 install the feature-set

    conf t
    install feature-set fabricpath
    feature-set fabricpath
    vlan 10
    mode fabricpath 

N5K1 (Already have FabricPath on VLAN10)
    conf t
    int e1/8 - 9
    description TO N7K1 FP
    switchport mode fabricpath
    no shut
    
N5K2 (Already have FabricPath on VLAN10)
    conf t
    int e1/8 - 9 
    description TO N7K1-2 FP
    switchport mode fabricpath 
N7K1-1
    conf t
    int e2/3 - 4, e2/7
    switchport mode fabricpath
    no shut 

N7K1-2
    conf t
    int e2/11-12, e2/15
    switchport type fabricpath 
    no shut

N7K1-3
    conf t
    int e2/23, e2/17 - 18
    switchport mode fabricpath 
    no shut

N7K1-4
    conf t
    int e2/25 - 26 , E2/31
    switchport mode fabricpath 
    no shut
    
You'll see that when you finish with the configuration and pass traffic from S3 to S1 fabricpath uses both links evenly. Using IS-IS SPT to route the traffic and not etherchannel
 
You'll also see that the upstream switch'es MAC table for vlan 10 show the following
It's doesn't show a port but the fabricpath switch ID. As a best practice one would want to change the Fabricpath Switch ID to something more descriptive for better troubleshooting

N5K1 
    conf t
    fabricpath switch-id 51
    
N5K2
    fabricpath switch-id 52

N7K1-1
conf t
    fabricpath switch-id 71

N7K1-2
conf t
    fabricpath switch-id 72

N7K1-3
conf t
    fabricpath switch-id 73

N7K1-4
conf t
    fabricpath switch-id 74

This may be event driven so you can use the "clear fabricpath isis adjacency"  to update the switch-id's 

show mac address-table dynamic vlan 10 

 
Now you can see the switch-id's for Nexus 5K1 and 5K2

IS-IS is used to exchange the switch-id and create the shortest path tree (SPT)

show fabricpath isis adjacency / show fabricpath isis database

If you want to change any of the isis behaviors you have to go under "fabricpath domain default"

Fabricpath and vPC+
    Each Fabricpath has it's own ID
        Node ID in the IS-IS SPT 
    vPC peer would normal have 2 switch ID's 
    vPC + allows Fabricpath and vPC to work together
        vPC peers shard a Fabricpath Switch ID
        Looks like one node from the IS-IS SPT
        Simplifies layer 2 multipathing

vPC+ Caveats 
    vPC peer links runs as "switchport mode fabricpath"
    implies peer link must be a F module
    vPC Peers share "fabricpath switch-id" under vPC domain
    vPC to vPC+ migration IS DISRUPTIVE
    

If you have any vdc's defined you'll see the "allowed feature-set fabricpath" under the "show run vdc" you still have to say "feature-set fabricpath" under each vdc to enable the feature"



Hint:
    When looking output from a particular interface you should change the load interval
    conf t
    int e1/x - x 
    load-interval counter 2 30 
    
    Also when looking at counter for particular interfaces it's easier to see the output with pipe
    command such as:
    sho int e1/x - x | in "output rate"


    




Comments