Blog‎ > ‎

CCIE DC Nexus 1000V VM-FEX Adapter-FEX

posted Jul 17, 2014, 9:00 PM by Rick McGee
Act's as a Cisco modular chassis switch
    There is actually no hardware

Creates DVS or vDS in VMWare 

Made up of:
    Virtual Supervisor Modules (VSM) Control and mgmt. plane
    Virtual Ethernet Modules (VEM) (data plane)

Virtual Service Blades
    Virtual Security Gateway (VSG)
    ASA 1000v
        All use vPath 2.0 for data interception/control

Each server in the data center is represented as a link care in the Cisco Nexus 1000v and can be managed as if it were a line card in a physical Cisco switch.

Nexus 1000v and Cisco UCS
    It's compatible with each other
    They don't have to know about each other to work

N1KV is compatible with vPC-HM is using MAC Pinning
N1KV is not compatible with allocating "Dynamic vNIC's" in a Service Profile
    Dynamic vNIC's create VM-FEX

VM-FEX and N1KV are mutually exclusive 
    Both of these options create VDS on the hypervisor. You wouldn't run multiple VDS on the same
        VM-FEX and N1KV both use VEM's
        N1KV uses the VSM for the Control Plane
        VM-FEX uses the UCS FI's as it's Control Plane

    vPath protocol is always running in the VEM
        Directs traffic to the VSN (Virtual Services Node) applies security or optimization policy, sends 
        packet back to VEM along with the ability to fast-switch traffic now directly in the VEM

    Only new traffic flow must first be sent to VSN, subsequent traffic is forwarded directly by the 
    VEM on ESX(i) host.

    VSM install Opaque Data in VMware vCenter for its DVS
        Done using "svs connection"
        Server Virtualization Switch (SVS)

    VSM's and VEM's should all be the sam version
    Control/Mgmt. network should be low latency (more important then BW)

    vCenter download this information to ESXi for VEM's to use whenever a host is added to the
        All VES Heart Beats should increase at roughly at the same rate
            "show mod gem courters"
            If the VSM misses 6 heart beats from a VEM it considers it offline

    Always hardcode VEM to Module number before you add ESXi host to N1Kv
        It's recommended to tie to UCS Chassis and Blade
        Get UUID from ESXi host:   
            #esxcfg-info -u (must be lower case letters)

VEM Port Profiles 
    Eth (Uplink tied to HW ports)
    vEth (Virtual tied to VM's)

    System VLAN's in Eth and vEth Port Profiles
        Used to give immediate cut-through access to the vmkernel 

        VEM's must be to eh same VLSN as VSM Control VLAN

    L3 (Recommended)
        VEM traffic is encapsulated in UDP 4785
            capability l3control needed on vEth profile used for ESXi VMKernel before moving from 

    System VLAN for both vEth and Eth 

Port Channels in N1Kv

    Remember that the N1Kv doesn't have to run on UCS B series server, it can run on any manufacture 
    Server with ESXi

    Show commands
        module vem 3 execute vdmcmd show port 
        module vem 3 execute vdmcmd show pinning 

     Creates the same type of DVS in VMWare as N1Kv does
        (can also run on KVM and Hyper-V in UCS 2.1)

    UCS FI's acts as the VSM for Control and Mgmt. Plane

    Virtual Ethernet Modules (VEM) are used for the data plane.

   Used to extend a N5K down to a pizza-box C-Series rack mount servers
        Specifically, to FEX the P81E (Palo) or VIC1225 PCIe CNA
        Creates vEth and vFC port in the N5K

    2 10GE SFP physical ports on PCIe card, each with 2 channels that break out into 4 logical Channels
        Port 1, Channel 1=Ethernet with F/O to physical port 2
        Port 1, Channel 2= HBA0 (no F/O, multipathing software needed)
        Port 2, Channel 3= Ethernet with F/O to physical port 1
        Port 2, Channel 4= HBA0 (no F/O, multipathing software needed)

Can also use UCS Manager to mange your C-Series 
    Require a pair of N2232PP FEX's to act as the "IOM" in a blade chassis
        UCS 2.0 Requires 4 Cables 
        Two 1GE cables connect from C-Server LOM to 2232 FEX to provide OOB control and mgmt. 
    In UCS 2.1 Single wire mgmt. means a single pair of 10GE cable from the C-Series SFP port to the
    2232 FEX provide both mgmt. and control planes.

N1Kv Topologies 

The VSM's in this example are running on Nexus 1110 Hardware devices that can run multiple VM's for the vWAAS, VSG, vASA, or back VSM's.

Can also run VSM's on a virtual server with the VEM's

Logical View of Eth and Veth ports for the Nexus 1Kv

The Eth ports are your northbound links to the N5K switches and the vEth ports are assigned to VM hosts. If a VM host moves from one blade to another (to another VEM) it keeps the same vEth port. So all the configurations for the vEth ports doesn't have to be reconfigured again.