Home > CCNP TSHOOT, Routing > Layer 3 Switch Troubleshooting

Layer 3 Switch Troubleshooting

April 2nd, 2010 Leave a comment Go to comments

The big topics of this chapter include HSRP, VRRP, GLBP, TCAM and the port error table. I broke out my notes from the BCMSN and I’ll summarize each topic from them, then move into the troubleshooting of each protocol.

Hot Standby Router Protocol (HSRP)
Cisco proprietary protocol to allow multiple routers to appear as a single gateway IP address. EAch router is assigned to a common HSRP group one active router and one standby router, the others, up to 16 group members, just listen. There is no preemption without configuration.

  • Election of one active router — Based on priority that is assigned when the router is configured for the group. Default priority is 100, range is from 0 – 255, highest priority for election is 255. Highest IP address on HSRP interface breaks ties.

  • Election of one standby router — Second highest priority. Listens to ehloo messages, by default every 3 seconds with hold time of 10 seconds.

  • Virtual router — IP and MAC address pair that end devices use as a gateway. The active router processes all packets and frames to this address, the virtual router processes no physical frames.

    • Troubleshoot
    sh standby …
    sh standby delay

    sh standby

    DSW1#sh standby
    Vlan10 - Group 10
      State is Active
      Virtual IP address is 10.2.1.254
      Active virtual MAC address is 0000.0c07.ac0a
        Local virtual MAC address is 0000.0c07.ac0a (v1 default)
      Hello time 3 sec, hold time 10 sec
        Next hello sent in 0.704 secs
      Preemption enabled
      Active router is local
      Standby router is 10.2.1.2, priority 100 (expires in 10.480 sec)
      Priority 200 (configured 200)
      Group name is "hsrp-Vl10-10" (default)

    sh standby br

    DSW1#sh standby br
                         P indicates configured to preempt.
                         |
    Interface   Grp  Pri P State   Active          Standby         Virtual IP
    Vl10        10   200 P Active  local           10.2.1.2        10.2.1.254

    sh standby delay

    DSW1#sh standby delay
    Interface          Minimum Reload
    Vlan10             1       5

    Virtual Router Redundancy Protocol (VRRP)
    VRRP is on open protocol that provides redundancy for the real IP address of a router or the virtual address of a group. By default all VRRP routers are configured to preempt the current master if their priority is greater.

  • Election of active router — If a real IP address is used, the master is the router with that IP. If a virtual one is used the master becomes the one with the highest priority, all others are in backup state. VRRP group numbers are from 0 to 255, router priority is from 1 to 254 with 254 being the highest and default priority is 100.

    • Troubleshoot
    sh vrrp
    sh vrrp brief

    sh vrrp

    R7#sh vrrp
    FastEthernet0/0 - Group 1  
    "Just playing"
      State is Master  
      Virtual IP address is 10.2.1.253
      Virtual MAC address is 0000.5e00.0101
      Advertisement interval is 1.000 sec
      Preemption enabled
      Priority is 200
      Master Router is 10.2.1.247 (local), priority is 200
      Master Advertisement interval is 1.000 sec
      Master Down interval is 3.218 sec

    sh vrrp brie

    R8#sh vrrp br
    Interface          Grp Pri Time  Own Pre State   Master addr     Group addr
    Fa0/0              1   150 3414       Y  Backup  10.2.1.247      10.2.1.253

    Gateway Load Balancing Protocol (GLBP)
    Cisco proprietary protocol that is more robust than other redundancy protocols. When a client sends an ARP request looking for the router address, GLBP send back a reply with the virtual MAC supported by one of the routers in the GLBP group.

  • GLBP Active Virtual Gateway (GLBP-AVG) — Group members elect the AVG which replies to all host ARP requests. It has either the highest priority value or the highest IP address in the group is the tie breaker. Priority ranges from 0 to 255 with 255 being highest, 100 is the default. AVG assigns the virtual router address and assigns the secondary roles. Up to four virtual MAC address can be used in any group.
  • GLPB Active Virtual Forwarder (GLBP-AVF) — Each gateway assumes responsibility for forwarding packets that are sent to the virtual MAC address assigned to that gateway. GLBP load balances across the AVFs in round robin by default. AVFs are expected to send hellos to every other GLBP peer. If the hold time on the AVG expires without receiving a hello packet from an AVF the virtual MAC is assigned to another router.

    • Troubleshoot:
    sh glbp [brief]

    sh glbp

    R8(config-if)#do sh glbp
    FastEthernet0/0 - Group 1
      State is Active
        2 state changes, last state change 00:03:58
      Virtual IP address is 10.2.1.252
      Hello time 3 sec, hold time 10 sec
        Next hello sent in 1.776 secs
      Redirect time 600 sec, forwarder time-out 14400 sec
      Preemption enabled, min delay 0 sec
      Active is local
      Standby is 10.2.1.247, priority 150 (expires in 7.744 sec)
      Priority 200 (configured)
      Weighting 100 (default 100), thresholds: lower 1, upper 100
      Load balancing: round-robin
      Group members:
        0004.dd69.fd01 (10.2.1.248) local
        00d0.bbef.64c1 (10.2.1.247)
      There are 2 forwarders (1 active)
      Forwarder 1
        State is Active
          1 state change, last state change 00:03:48
        MAC address is 0007.b400.0101 (default)
        Owner ID is 0004.dd69.fd01
        Redirection enabled
        Preemption enabled, min delay 30 sec
        Active is local, weighting 100
      Forwarder 2
        State is Listen
        MAC address is 0007.b400.0102 (learnt)
        Owner ID is 00d0.bbef.64c1
        Redirection enabled, 597.584 sec remaining (maximum 600 sec)
        Time to live: 14397.584 sec (maximum 14400 sec)
        Preemption enabled, min delay 30 sec
        Active is 10.2.1.247 (primary), weighting 100 (expires in 7.584 sec)

    Ternary Content Addressable Memory (TCAM):
    TCAM works with CEF to make extremely fast forwarding decisions. TCAM entries are composed of Value, Mask and Result combinations.

  • Values — Always 134 bits consisting of source and destination addresses along with relevant protocol infromation.
  • Masks — Also 134 bits set to match value bits, or not match value bits that do not matter. This is so that a switch can do a bitwise and with access lists or access control entities (ACE).
  • Results — Numerical values that represent what action to take after the TCAM lookup.

    • When the TCAM cannot process the packet it is punted to the CPU for processing. Here are some reasons why a packet is punted:

  • Protocols that send multicast or broadcast packets.
  • Administrative access to a switch such as telnet or ssh.
  • Packets that use features not supported in hardware.
  • A switch’s TCAM has reached capacity. TCAM might reach capacity with large routing tables or a large number of ACLs.

    Troubleshoot TCAM:
    3550:
    show tcam
    3560:
    show platform tcam
    show platform ip unicast counts

    3550#sh tcam inacl 1 statistics

    3550#sh tcam inacl 1 statistics
    Ingress ACL TCAM#1: Number of active labels: 3
    Ingress ACL TCAM#1: Number of masks   allocated:   14, available:  194
    Ingress ACL TCAM#1: Number of entries allocated:   17, available: 1647

    3560#sh platform tcam utilization

    3560#sh platform tcam utilization

    CAM Utilization for ASIC# 0                      Max            Used
                                                 Masks/Values    Masks/values

     Unicast mac addresses:                        544/4352         25/76    
     IPv4 IGMP groups + multicast routes:          144/1152          6/26    
     IPv4 unicast directly-connected routes:       544/4352         25/76    
     IPv4 unicast indirectly-connected routes:     176/1408         15/69    
     IPv6 Multicast groups:                        544/4352         25/76    
     IPv6 unicast directly-connected routes:       544/4352         25/76    
     IPv6 unicast indirectly-connected routes:     262/2096         11/42    
     IPv4 policy based routing aces:               256/256           2/2    
     IPv4 qos aces:                                768/768         324/324  
     IPv4 security aces:                           512/512          27/27    
     IPv6 policy based routing aces:                 0/0             0/0    
     IPv6 qos aces:                                  0/0             0/0    
     IPv6 security aces:                           204/510           5/5    

    Note: Allocation of TCAM entries per feature uses
    a complex algorithm. The above information is meant
    to provide an abstract view of the current TCAM utilization

    Port Error Table:

    Error Description Cause
    Xmit-Err Transmit buffer overlflow Check for speed mismatch.
    Rcv-Err Receive buffer overflow Check for duplex mismatch.
    UnderSize Has valid checksum but less than 64 bytes. Host is sending invalid frames.
    Single-Col A single collision occurred before the port transmitted the frame. High bandwidth utilization or duplex mismatch.
    Multi-Col Multiple collisions occur before the port transmitted the frame. High bandwidth utilization or duplex mismatch.
    Late-Col Collision detected further into frame being forwarded. Cable too long or duplex mismatch.
    Excess-Col Frame has experienced sixteen successive collisions, resulting in the frame being dropped. High bandwidth, duplex mismatch or too many devices on a segment.
    Carri-Sen Normal operation on a half-duplex link. This is not a problem.
    Runts Frames smaller than 64 bytes with a bad CRC. Duplex mismatch or layer 1 problem.
    Giants Frame size greater than 1518 bytes with a bad FCS. Bad host NIC.

    Source: TSHOOT p.125

    • Categories: CCNP TSHOOT, Routing Tags:
    1. No comments yet.
    1. No trackbacks yet.