Blog‎ > ‎

Nexus NX-OS L2/L3 Switching

posted Nov 12, 2013, 12:30 PM by Rick McGee   [ updated Mar 25, 2014, 7:12 PM ]
Nexus NX-OS L2/L3 Switching    
At it's core it's a L2/L3 Switch
    Similar to Catalyst IOS
        VLAN's, Trunking, VTP, Rapid-PVST, MST (RPVST and MST are backwards compatible with legacy                  spanning tree protocols) etc.
    New Features
        FEX, vPC, Fabricpath, OTV, Native fiber channel switching (N5K), FCOE, and more

Nexus NX-OS vs. Catalyst IOS 
    - Default Port Modes
    - STP Port Types
    - STP Bridge Assurance 
    - Port Profiles 

    VLAN's and VTP
    Unlike IOS, NX-OS pre-reserves certain VLAN's for internal usages
        3968- 4047 and 4094 with NX-OS version before 5.2(1) 
        3968-4094 with NX-OS version 5.2(1) and after (I.E. the last 128 VLAN's)
    Can change this range with "System VLAN <#> reserve" (you cannot do this with the Cisco UCS Fabric Interconnects)

VTP is disabled by default in NX-OS (hurray)
    To enable have to turn on the feature with "feature VTP
    No additional configuration needed for extended VLAN's (1001-4044)
    Doesn't support DTP (dynamic trunking protocol)
    Two Types: 
        Static Access or Static Trunk
        NO ISL the trunks must be 802.1Q
        NO PAGP
            Channels must be statically on (LACP)
                Feature LACP
    Multi-Chassis Etherchannel

    NO CST/PVST+ (this are legacy protocols)
    Default mode is Rapid-PVST+ (802.1w)
        Can configure with MST  (802.1s)
    With the support of MST
        Group forwarding paths together with MST
        This is usually for very large deployments of 2,000 VLAN's or more

All ports are shutdown by default
    F ports wil be L2 switchports by default
    M ports are layer 3 routed interfaces by default
    Run the following commands to see the access
        - "show interface status"
        - " show interface brief"
    Default Commands on interfaces are
        - "no system default switchport"
        - "system default switchport shutdown"
        - You cannot see this under the "sh running-config" status
    Default Modes can be changed
        - "system default switchport"
            - This will change all M ports to L2 switchports 
        - "no system default switchport shutdown"
            - Changes all switch ports to an admin up status
            - This doesn't show at the link level of show e1/1
        - This commands only effect interfaces if they had not other configurations made
        - No DTP!! only 802.1q

Spanning-Tree Port Types
    NX-OS runs Rapid PVST by default
    Normal= Normal in Catalyst Switches
        - "spanning-tree port-type normal"
    Edge= Portfast in Catalyst Switches no learning, directly into a forwarding state
        - "spanning-tree port-type edge"
    Network= STP bridge Assurance
        - "spanning-tree port-type network"
        Sends out BPDU's regardless of the state, not just when a link is down like in 802.1b
        The BPDU becomes a bidirectional keep alive
            This Replaces Loopguard
            Secondary effect is VLAN pruning (act's similar to VTP pruning)
                    When seeing certain STP BPDU's across a link for VLAN's 10,20,30,40 and the other
                    side only  
                    replies with VLAN's 10,20,and 30                 
                    VLAN 40 is removed from the trunk
            Also serves as the bidirectional UDLD 
            Per VLAN basses MST per group

    config terminal 
    int e1/3 
    switchport mode trunk 
    spanning-tree port type network 
    no shutdown

    When connecting NX-OS to NX-OS switches use spanning-tree port type network. 
    When connecting NX-OS to IOS switches use spanning-tree port type normal or edge

Layer 2 Switching Configuration example switch setup 
Both R2 and R3 will be in VLAN 10
Start with N5K1 and make sure FEX are configured with Parent Switch
    Use show fex to see fex association   
    Go go global config 
    config t
    feature fex 
    int e1/4 - 5
    switchport mode fex-fabric
    fex associate 101
    no shut 
    show fex
    config t
    feature fex
    interface 1/6 - 7
    switchport mode fex-fabric
    fex assoicate 102 
    no shut

Next setup VLAN
    conf t
    vlan 10
    int e101/1/1
    switchport access vlan 10
Make sure that if are connecting these to a 2232 or 10GE FEX you shutdown the port and configure the speed manually with 
    int e101/1/1
    speed 1000
    no shut
Issue command show int e101/1/1 status 
    You'll see the following output 

Next configure N5K2
    conf t
    vlan 10 (In NX-OS you cannot assign via interface configuration as in CAT IOS)
    int e102/1/1
    switchport mode access vlan 10

Go to routers R3 and R2 and configure those interfaces g0/0
    config t
    int g0/0
    mac-address 0000.0000.0003 (for R3 for trouble shooting)
    ip address
    duplex auto
    speed auto
    media-type rj45
    no shut

show ip int brief on both R2 and R3
    You should see int g0/0 in an up/up status
On N5K's 
    show int e10x/1/1 switchport

Issue command show spanning-tree int e10x/1/1

FEX port will always be in port type edge you will not be able to change this to Network or Normal port types

Turn on 
term mon (to see logging messages to vty line)

show fex output

Next Configure N7K's 
    We are doing this in a southbound manner to the northbound switches, then eastward to the other N7K1-2 and then southbound to N5K2 and 
    to finally N2K2
This is just basic L2 connectivity 

    conf t
    vlan 10
    int e2/1 - 4
    switchport mode trunk
    spanning-tree port type network
    channel-group 1 mode on
    no shutdown

    conf t
    vlan 10
    int e2/9 - 12
    switchport mode trunk
    spanning-tree port type network 
    channel-group 1 mode on 
    no shutdown

Issue show cdp neighbors  (This is assuming N5K's ports are up)
Configure N5K's as trunk ports (vlan 10 is already configured)
    conf t 
    int e1/8 - 9
    switchport mode trunk
    spanning-tree port type network 
    channel-group 1 mode on
    conf t 
    int e1/8 - 9
    switchport mode trunk
    spanning-tree port type network 
    channel-group 1 mode on

Show interface trunk    
Show spanning-tree vlan 10
With Port-channel 1 on interfaces

For Spanning-Tree MST

    conf t
    spanning-tree mst configuration 
    revision 1
    name MST1
    instance 1 vlan 10,20,30
    instance 2 vlan 40,50,60
    conf t
    spanning-tree mst configuration 
    revision 1
    name MST1
    instance 1 vlan 10,20,30
    instance 2 vlan 40,50,60

    conf t
    spanning-tree mst configuration 
    revision 1
    name MST1
    instance 1 vlan 10,20,30
    instance 2 vlan 40,50,60

    conf t
    spanning-tree mst configuration 
    revision 1
    name MST1
    instance 1 vlan 10,20,30
    instance 2 vlan 40,50,60

Final step on all the Nexus switches would be to change the mode from RSTP to MST
    spanning-tree mode mst
Have to make sure that if you have MST working with PVST or RSTP you would have to make sure that the ROOT bridge is in the MST region 

Would use MST if you have a large amount of VLAN's     
    This would lesson the control processes 

BridgeAssurance is really for RSTP     
You can still run UDLD (feature UDLD)
    This is a global command and will be on all interfaces by default
    UDLD will be able to protect you from a L1/Physical issue but not from a spanning-tree L2 issue

NX-OS Port Profiles
    Similar to interface macros in Catalyst IOS
        - Allows a temp ate of a config to apply to  group of ports
    Main difference is that changes to port profiles are event driven
        - Catalyst IOS macros apply only once at initial application 
        - NX-OS profiles immediately re-apply any time a change is made to the profile
 Port Profile types
        - Ethernet 
        - Port-Channel
        - Interface VLAN
Verifying Port Profiles
        - "show port-profile expand-interface name [profile_name]"

Create a Profile
        -port-profile PROFILE1 
         (config-port-prof) prompt
         state enabled 
         switchport access vlan 123
         spanning-tree port type edge
         int e101/1/20 - 30
         inherit port-profile PROFILE1
show run int e101/20 - 30
To expand the name use the following
    show port-profile expand-interface name PROFILE1

So know if you change the port-profile those commands will automatically apply to the interfaces that inherit that profile 
    port-profile PROFILE1
    switchport mode trunk
    switchport trunk allowed vlan 10,20,30

You can remove by 
    no port-profile PROFILE1
        - This will remove all the configurations to the ports that inherited the port-profile
        - I really don't care for the to much because you cannot see the configuration unless you run
          the "show port-profile expand-interface name PROFILE1"

    Port-profile type [ ethernet, interface-vlan, port-channel]
    Apply configuration to profile
    E.G. switchport mode access, spanning-tree port type, ect.
    enable the profile
        state enable
    Assign to interfce
        int e1/32 inherit port-profile [name]
    show port-profile expand-interface name [name]

Nexus Layer 3
    IPv4 and IPv6
    Nexus supported modules for L3
    M1, M2, and F2/F2E
    F1 modules depend on proxy routing  from M1/2 modules
    Nexus 5K daughter card in the Nexus 5548 and a module for Nexus 5596

L3 Port types
    Same as with Catalyst IOS
    Native Layer 3 routed interface
            "no switch port"
    Switched virtual interface (SVI's)
        I.E. VLAN interface
        Feature Interface-VLAN must be enabled first

Nexus Routing protocols
    Static Routing
    RIPv2 and RIPNG (next generation IPv6)
    EIGRP and EIGRPv6
    OSFP and OSPFv3
    Policy Based Routing (PBR)
    "router rip 1" (process)
    "address-family ipv4"
    "router eigrp 1"
    "address-family ipv4"
    There are not network statement under the process, must be configured under an interface)
    "autonomous-system 1234"
OSPF has it's own feature for IPV4 and IPv6
    "feature ospf" and "feature osspv4"    
    "router ospf 1" and "router ospfv3 1"

    Used to create separate logical routing tables
    VLAN 10 and VLAN 20 kept separate 
    They don't have routed to each other
    Not active separation of control plane like VDC's 
L3 Virtulization 
    Layer 3 interfaces in a different VRF's cannot exchange traffic by default
    Slightly different then Catalyst IOS, as
    All L3 interfaces are automatically in VRF table "default"
    MGMT0 is automatically in vrf management
    VRF's are defined as a "vrf context"
    Static routes are defined under the "vrf context"
    Dynamic routing is VRP aware, but configured under the same process
    Exec mode "routing-context vrf" can change the default VRF for verifications
    Can use the "show ip route vrf management"
VRF Verifiacations
    EX. in exec mode "routing-context vrf EFG"
    Will change prompt to N7K-10%EFG
Policy Based Routing
    Will override the routing table
    Classify with an ACL
    Choose next-hop, output interface, etc...
    "feature pbr"
    route-map [name] pbr-statistics
        This will turn on pbr statistics otherwise you wont see any hits on the route maps
Dynamic routing
    Has both global and interface routing
        Global routing process affects all links or the routing instance as a whole
            E.g. EIGRP AS, OSFP SFP timers, IS-IS level
        Interface commands affect routing protocols behavior of that link
        Interface commands affect routing protocol behavior of that link
             E.g. Passive interface, EIGRP route filters, OSPF hello timmer, and so on
        Unlike IOS there is no "network" command for IGP
              Protocols are enabled directly at the link level
              network commands are still used for BGP NLRI origination 
Route Redistribution
    Unlike IOS, route-maps are required to perform redistribution on NX-OS
        Same route-map match/set logic as IOS
    Redistribution doesn't include directly connected interfaces
         required "redistribute direct route-map" command