Chainring Circus

Spanning Tree Protocol Basics:
In a layer 2 environment with no routing, active redundant paths are neither allowed nor desirable, because that can cause loops. Because a switch only segments collision domains it does not segment broadcast domains. STP helps find redundant links and place one in a blocking state.

STP Definitions:

• Root bridge — Center of the spanning tree.
• Nonroot bridge — Every switch not elected the root.
• Root port — Every nonroot bridge has a single root port, decided based on root path cost.
• Designated port — Each segment has a single designated port. All ports on a root bridge are designated.
• Nondesignated port — Every switch port that is neither a root port nor designated port starts blocking.

Basic Spanning Tree Operation:
1. Elect root bridge — Lowest bridge ID wins, consists of 2 bytes from 0-65,535 defaults to 32,678 + VLAN and the MAC address of 6 bytes, for example 32769 000a.b7d1.9580 for VLAN 1.
2. Select root port — One per switch, points toward the root bridge.
3. Select designated port — One per segment with the lowest root patch cost.
4. Block ports — Block non-root and non-designated ports.

Spanning Tree Port States:

Securing STP
Root Guard — Is enabled on a per-port basis. When a port receives a superior BPDU, with a lower bridge ID, the local switch will not allow the new switch to become the root. Instead the port is changed to root-inconsistent state, no data can be sent or received until the BPDUs stop.

BPDU Guard — PortFast moves an end-user port to forwarding state without going through all of the STP checks and can induce loops in the network. If any BPDU is received on a port where BPDU guard is enabled that port is put into errdisable state. It can then be recovered manually or through the errdisable timeout function.

Strategy for troubleshooting STP:
Find the root bridge, then learn the designated ports on each subsequent switch. Cisco switches run PVST by default so you will have to work through each vlan.

Commands for STP Troubleshooting:
sh spanning-tree — View all STP parameters for all VLANs.
sh spanning-tree [int fa0/1] detail — View all STP details.
sh spanning-tree [vlan 2] summary — View ports in each of the STP states.
sh spanning-tree [vlan 2] root — Find root bridge ID, root port and root path cost.
sh spanning-tree [vlan 2] bridge — Show local switch bridge ID and STP timers.
sh spanning-tree uplinkfast — Show uplinkfast status.
sh spanning-tree backbonefast — Show the backbonefast status.

sh spanning-tree

sh spanning-tree int po23 det

sh spanning-tree vlan 10 summary

show spanning-tree root

show spanning-tree bridge

I felt like my last set of notes for the Troubleshooting Toolbox had information that that was hidden in the command output. The output is for myself to play with the commands, however, I don’t want useful information hidden. So I am going to try a different setup with this set of notes and possibly an entirely different format for the next set until I find something that is easy for me to go back and study. Bear with me.

Commands for troubleshooting MAC addresses:
sh mac address-t dyn — Whether or not a host is communicating with the switch.
clear mac address-t dyn — Clear the MAC address address of dynamically learned mac addresses.

show mac-address-table dynamic
Shows the MAC addresses learned by a switch and it’s port. Useful to see whether or not a switch is learning the mac of a host, if the MAC address is not in the table, the problem is upstream from the current switch.

clear mac address-table dynamic
Clear out the table. This way you know a mac address was learned recently. Notice in this sample that the hosts on fa0/1 and 2 had not yet been learned in the first sh mac address-table command.

Commands to troubleshoot VLANs:
sh vlan
sh vlan br
sh int tru
sh int fa0/1 swi

show vlan
Allows you to verify a VLAN exists and shows which ports belong ti which VLANs.

show int trunk
Which ports or trunk ports and what vlans are allowed.

show int switchport
Displays summary information about a port.

Etherchannel:
Ehterchannel binds multiple physical interfaces into one logical interface.

Commands to troubleshoot etherchannel:
I include couple of sh run commands because it is most common to have configuration errors when working with etherchannel as you are dealing with multiple switches.
sh etherchannel
sh etherchannel 13 summary
sh int po [13]
sh run int po [13]
sh run | inc chann

show etherchannel 13 summary
A good way to see the protocol and ports in an etherchannel.

show etherchannel
Show a brief output of the etherchannels configured on the switch.

show int port-channel 13
Shows the typical sh int output.

sh run int po13ASW1#sh run int po13

sh run | inc chann

One of my coworkers solved a problem today that I should have been able to solve. I got pulled into a project that uses a closed system with it’s own programming language that I had never seen nor programmed. My coworker was trying to figure out why his syntax was not working. So we cranked up the logging in Warn.log, Error.log and Fatal.log. We quickly figured out why it was failing, the program was looking for a file that did not exist and he was not catching that error, there is no catch statement.

So I began trying to write an IF/ELSE statement that would check for the error condition. I was given a .pdf of the manual for the system and just started trying different functions that made sense to me. I wrote a nice little one liner in the SYSTEM command that returned TRUE or FALSE if the FILE environment variable existed.

The problem was that we could not set the environment variable with a variable from the program. It has to be a string literal.

From the Trace.log:

We tried a number of different variations on that theme, imported and exported variables to a subshell, tried different return values from the exit command. We got the system to turn circles but what we really wanted was squares.

Finally my coworker called and said, why don’t we just invert the logic. If it falls through to the end of the IF/ELSE statements it must be in the format we want. Let me take a step back and say this was his code, he should have been the one to figure it out. I believe I helped him figure out what his main problem was, which helped him find the solution. I’m not trying to minimize his work, nor am I trying to minimize my input into the solution.

The moral of the story is that I should have taken a higher level look at the problem to understand the solution. I am quick to believe that I can code my way out of any problem, regardless whether or not I have ever seen the language. What I really needed to do was step back and take a higher level view of the logic, rather than dive into syntax and functions.

Let me start by what this chapter does not include, a nice set of filters for common show commands that will help you find the most pertinent information quickly. I think every networker has some of their favorite commands such as this one for BGP:

Or one of my favorite sh run commands:

Filtering the show command
Using include:

Using exclude to show the same information:

Redirecting output
I had never seen the redirect and append commands in IOS, and while I have used tee and >> or 2>&1 on UNIX servers, I had never used redirection in IOS. Redirecting with append:

Interesting, I’m not able to append to tftp while they can in the book. Let’s troubleshoot :)

So where can I append?

Lesson learned. I can append to ftp while the TSHOOT author, Kevin Wallace can append to tftp. I wonder what IOS version he is using? Just for clarity here is my version:

Ping:

Before I go into what all ping can do on IOS I want to point out a link about Mike Muus, the man who wrote ping, it’s an interesting read for some historical perspective.

What the different characters mean in the ping response field:
! — Each exclamation point indicates receipt of a reply.
. — Each period indicates the network server timed out while waiting for a reply.
U — A destination unreachable error PDU was received.
Q — Source quench (destination too busy).
M — Could not fragment.
? — Unknown packet type.
& — Packet lifetime exceeded.

Some of the ping options from the TSHOOT book:
size — Then number of bytes per datagram.
repeat — The number of ICMP Echo messages sent.
timeout — Seconds to wait for an ECMP Echo Reply.
source — Source IP of the datagrams.
df-bit — Set the do not fragment bit.

If you’re going to turn on ip packet debugging on a router, you better set up an ACL. This is me playing with debugging ICMP, remember access lists are almost always set for inbound traffic.

I also did not know about the ping sweep capability of IOS:

So you would use ping to test layer 3 and telnet to test layer 4. Let’s test ftp in our testlab. This will come back and bite me some day, why remember port numbers when they are always at your disposal.

This portion of the book is frustrating to me. I am an interactive learner and this portion is not as fun. It also feels like I am just reiterating the Key Topic points in the book. That is frustrating as well. I guess in time Cisco will have some more interesting documentation concerning troubleshooting as the pendulum swings toward documentation for the new test.

Hardware Debugging Commands:

show processes cpu — Is the switch/router able to handle the traffic?
show memory — Memory usage.
show interfaces — If needed use the clear counters command.
input queue drops — Receiving packets faster than it can process.
output queue drops — Could not send fast enough, i/o speed mismatch?
input errors — Frames were not received correctly, cabling problem?
output errors — Frames were net sent correctly, duplex mismatch?

Packet Captures:
I added another ethernet card to the server and connected it to gi0/12 on DSW2. That way I can run wireshark and capture traffic. You can download my libpcap file here.

RSPAN:
RSPAN allows you capture traffic on switch that is sent from the port on another. That way you don’t have to have a collector in every closet.

First you need to configure the vlan as a remote-span vlan:

The switches are set up with port-channels between them, however, rspan does not give an option for a portchannel interface as the reflector port. I just pointed it at one interface of the port-channel and it worked.

Here is the complete setup:

And on the destination switch:

SNMP:
Uses a pull model to collect device statistics. The command ifindex persist ensures the interface index stays consistent across reboots.

NetFlow:
Uses a push model to collect detailed traffic statistics.

Embedded Event Manager:

EEM monitors events through event detectors which then trigger an action based upon defined policies.
Event detectors can be an CLI command, interface counter, SNMP event or syslog event.
Actions occur in response to an event, examples of action include generating an SNMP trap, reloading IOS or generating a syslog message.
Policies are either an applet or a script.

As I was playing around in the EEM I came across this situation, it would be a mean way to play a game on someone.

On a more serious note:

I am going to end with some things I would like to see in the IOS CLI toolbox:
&& I should be able to see the output of this command:

Or another command:

Today on Slashdot someone had a question about getting data off of an old Xenix server. A few years ago I did a consulting job for a customer who had an old Xenix server with no ethernet card that needed to get some data for the State Police. The server ran an old database that kept track of training records and they needed that information in order to “webify” the process.

I wish I could remember the ins and outs of the problem. I gave the customer a write up of how I did it and the problems I had getting the usr directory off of the system. Unfortunately I can’t find the report.

Regardless, it was an interesting project, here is the script from my library that eventually worked:

The best reason to follow a structured approach is the shot in the dark that often happens if you don’t gather enough information or work to nail down the cause of the problem. It results in haphazard guessing with little to remind you of what you have changed and why.

A general outline of troubleshooting as described in the first paragraph of this chapter:
1. Report
2. Define the Issue
3. Gather Information
4. Better define the issue
5. Hypothesize root cause
6. Propose resolution
7. Test resolution
8. Document solution

Troubleshooting Model:

1. Problem report — Often needs further investigation.
2. Problem diagnosis
    a. Collect information — Use sh and debug to get a better understanding.
    b. Examine information
        — Look for evidence that points to the cause.
        — Look for evidence that can eliminate a vector.
        — What is happening on the network.
        — What should be happening on the network.
    c. Eliminate causes — Start to form hypotheses for what is wrong.
    d. Decide most likely cause — Once possible causes have been narrowed decide on most plausible.
    e. Verify Hypothesis — How can you test whether you have found the problem.
3. Problem resolution — Apply solution, test and document.

Troubleshooting Methods:

• Top-down Method — Start at layer 7, the application and work down.
• Bottom-up Method — Start at layer 1 and make sure the physical layer is correct. No efficient in later networks.
• Divide and Conquer — Begin by pinging and work up or down the OSI stack accordingly.
• Follow the traffic — Work through the network one switch at a time from source to destination.
• Configuration Comparison — Compare a working configuration with one that does not work.
• Component swapping — By changing components you can figure out which one is not working, either hardware or misconfigured.

Troubleshooting involves knowing what should be happening as opposed to what is happening on the network. The best way to do that is to have a baseline which involves SNMP and NetFlow data. They are covered in more depth in the next chapter.

I came across this post today by way of HackerNews. It explains some of my feelings lately as I chomp at the bit wanting the challenge of a new job. I promised my wife we would not move until we had our kids, so I have a couple more years, but I’ve been straining against the reigns.

That post also explains why I think about documentation differently than my peers. I joke about being raised in the military, I wasn’t, however, the Air Force was my first real job. In the military it is known that you will change jobs every three years and so the documentation you write is not for yourself, it is for the person you know will come behind you. In the military you do not quit, you are moved. If you stay in one place too long or “homestead” you are looked down upon and it can actually threaten your career. As a result documentation is taken seriously and my time in the Air Force shaped my understanding of documentation.

Thinking about past employees as alumni is also the same way veterans think about themselves. I am proud to have served and there have been times I felt guilty I left, other times I have wanted to return. Whenever I leave my current position I will view myself as an alumnist, I just hope my current employer does too.

In the future I would like to find an employer that shares my interest in personal development and also has the room for me to grow. I first hired on as a Programmer Analyst and was promoted three years later to Senior Network Engineer. That is the extent of advancement with my current employer, there is no Super Senior Network Engineer. I don’t mind taking a step back with a new employer as long as there is the ability for future growth as I develop.

I am going to admit that I have not been studying for the TSHOOT exam very diligently. In my mind it is a topic that we deal with on a daily basis and what could the book teach me. (sarcasm) They gave us the topology, if you understand it like your own network what more could they want? Well, I read the first few chapters and learned some new commands. Now I’m going back to start over, this time typing up notes like I did for the ONT. It keeps me headed in the right direction because other people can see my progress and that motivation keeps me on task.

The first chapter has been a struggle because it is not interesting. These notes are just me struggling through, trying not to fall asleep or get side tracked… oh look there’s a new article on slashdot…

Two network maintenance categories:
Structured tasks — Planned tasks.
Interrupt-driven tasks — Helpdesk tasks.

Network Maintenance Models:
FCAPS — Fault, Configuration, Accounting, Performance and Security management.
ITIL — IT Infrastructure Library is a series of books and checklists published by the UK Government that a company can modify to it’s needs.
TMN — The Telecommunications Management Network includes is the ITU-T version of the FCAPS model, it includes Business, Service, Network and Element management.
Cisco Lifecycle Services — Phases are Prepare, Plan, Design, Implement, Operate, Optimize.

Routine Maintenance Tasks:
Configuration changes
Hardware replacement
Scheduled backups
Software updates
Network performance monitoring

Change Management Issues:
Responsibility, who owns what processes.
Define scheduled maintenance tasks.
Change procedures to follow.
Documentation, who is responsible.
Rollback plan, what happens when a change goes south.

Documentation: — Network documentation is one of the most important tasks a network administrator does. (My opinion, not out of the book.)

Logical topology diagram — shows network interconnects and protocols
Physical topology diagram — shows physical layout and interconnects
Listing of interconnections — device and port connections with circuit IDs
Inventory of network equipment — manufacturer, serial number, model number software version
IP address assignment — describe the network numbering scheme
Configuration information — copies of current and past configurations
Original design documentation — why the network was designed that way

Troubleshooting Aids:
The terminal when logged in remotely does not display console messages by default. Use the term mon command to see messages.

Logging levels:
The logging levels listed below are the same for if you are logging console or logging buffered:

It’s always good to be able to correlate logs with the correct time, use NTP to synchronize clocks across the domain:

And if you log remotely:

Backup Tools:
At the Circus we use rancid with SVN to back up our Cisco and ProCurve devices. I never knew about the archive command, however, I still believe rancid is the better tool, with SVN it will email you diffs, set it up with tacacs+ and you know who made what changes and can be notified by email.

Let’s explore some Cisco backup options.

For standard ftp backup:

And ftp in action:

The archive command. I have it set to archive every 60 minutes and every time I write memory:

The archive command in action, when I archive from the command line, it writes r4.arch-29 and when I wri mem it archives r4.arch-30.

Both the archive and ftp sections of the R4 config:

And to restore the running config from backup use the configure replace command:

Chris pointed out that the Cisco Learning Network has posted a TSHOOT demo. I just went and played with it. Funny thing is, I wasn’t taking a real test but my pulse and blood pressure both shot up. It’s a wonder I passed any of the exams the way Cisco has got me wired.

I’ve been playing around in the TSHOOT topology and thought I would share my configuration. Here is a copy of the original topology as released on the Cisco Learning Network. You can download a .tar file of all of my configurations here. If you have it configured differently please share in the comments or a post on your own blog.

I used a few more routers for the clients and set up the BGP cloud as well. The frame relay switch configuration on R10 was left over from Narbiks labs and the extraneous configurations that you see on random ports on the client routers was testing for another project. This way I could use the lab for both scenarios. Below is my modified topology, everything else is the same as the original .pdf. Routers 1-4 are 1841s, 5-10 are 3640s, the distribution switches are a couple of 3560s and the aggregation switches are 3550s.