Chainring Circus

I have had a hard time figuring out how they are going to test us for voice troubleshooting when the only real command they cover is auto qos and and the MQC. As a result I’m going to concentrate on the definitions.

Voice Definitions
Gatekeeper — provides bandwidth management through call admission control (CAC).
Gateway — ensures interoperability between VOIP and the public switched telephone network (PSTN).
Jitter (delay variation) — When consecutive packets experience different amounts of delay. Data applications tend to be much more forgiving of jitter than voice and video.
Delay — There are multiple types of delay in a network. Some are standard or fixed and some are variable in their affects, the TSHOOT book describes delay as propagation delay, the time it takes to get a bit from one end of a link to the other.
Drops — Congested packets overflow a buffer.

Cisco Phone Boot Process
1. Power, PoE
2. Load firmware from flash.
3. Catalyst switch informs the phone it’s voice VLAN.
4. DHCP for ip address and TFTP server.
5. Downloads configuration using TFTP.
6. Registers with call agent or Call Manager.

QoS Metrics for Video

Sources:
1 — ONT Certification Guide p.62
2 — Cisco DocWiki
3 — Enabling VOIP
4 — TSHOOT Book

Multicasting
Class D IP address in the range 224.0.0.0 through 239.255.255.255. Source sends one packet stream to the multicast address and all hosts that have joined that group receive that packet.

Internet Group Management Protocol (IGMP)
Hosts join a multicast group by sending an IGMP join message to router, which then knows to send multicast messages out that interface. IGMP snooping allows a switch to learn which interfaces desire multicast traffic by listening for IGMP traffic between routers and hosts. This stops the switch from flooding multicast traffic out all ports.

IGMP Version 1 — Hosts join a multicast group by sending a membership report to its local router. Every 60 seconds the querier router sends a messages to all-hosts 224.0.0.1 to ensure that there is a host on that network segment that is still in the group. IGMPv1 does not have a mechanism for hosts to leave a group, and it takes three query intervals (3 minutes) to stop sending multicast traffic to a segment.

IGMP Version 2 — Adds the ability for routers to query a specific multicast group, elect a querier for a segment and allows a host to send a leave group message to the all routers address 224.0.0.2. All routers start as queriers, however, if a router hears a query from another router, the router with the highest IP address on the segment becomes the querier for that segment.

Reverse Path Forwarding (RPF) — Verifies that multicast traffic flows away from the source or root and is flowing toward the branch or host.

Protocol Independent Multicast (PIM) — Allows multicast to build distribution trees regardless of the unicast routing protocol which is running such as EIGRP or OSPF.

PIM Dense Mode (PIM-DM) — Uses a source distribution tree. At first all routers receive traffic for the group, but if no host joins using IGMP the router sends a prune message so that unnecessary traffic does not continue. Most often used when recipients are on every subnet, densely populated.

PIM Sparse Mode (PIM-SM) — Uses a shared tree with a root router or rendezvous point (RP) that is not necessarily the multicast source but is usually centrally located on the network. All multicast streams go through this router, hence the name share tree or shared distribution tree. A router only joins the tree when a host has joined the multicast group. It is built opposite of dense mode, the tree is built from the leaves to the root, it is only when a host joins a multicast group that the router forwards the membership report to the RP.

PIM Sparse-Dense Mode — Allows a router to use sparse or dense-mode or both at the same time. Dense mode is used to flood RP discovery and announcement messages so that the client can find the RP and use the RP to find the multicast server.

Multicast Configuration

switch(config)# ip igmp snooping
switch(config)# ip igmp snooping vlan x

router(config)# ip multicast-routing
router(config)# ip pim {dense-mode | sparse-mode | sparse-dense-mode}
router(config)# ip pim version {1 | 2}

Multicast Troubleshooting
ip igmp join-group — Let’s a router join a group in order to test.
sh ip igmp group — Shows the groups a router has joined.
sh ip igmp interface — Shows IGMP information for each interface.

sh ip mroute
ping multicast address
sh ip pim rp
sh ip rpf

sh ip igmp group

sh ip igmp interface

sh ip mroute

ping multicast address

Last week we had in a consultant from one of our electronic medical record (EMR) vendors and we were working on a RedHat cluster. He was asking me to check whether a service started at a runlevel and wanted me to ls /etc/rc.d/rc5.d, I looked at him and said, “There’s a command for that.” (The command is chkconfig.) I’ve been chuckling about it for the past few days.

The Linux and BSD foundations need to get together and start an advertising campaign, it would be a great parody.

I added some more flashcards in the files directory. These all have a file name of tshoot-topic.csv and are for the Flashcards Deluxe iPhone app.

If you notice any errors in the cards please shoot an email or make a comment. At this point I have not downloaded all of them so there may be serious errors or formatting changes. I made them from my notes so any errors are mine.

I am not going to get into a long discussion of DHCP. I run the DNS/DHCP servers at work and have a pretty good idea of how it all works. Even though the basics are the same, BIND is a different animal than IOS.

The DCHP process from debug messages
–> DHCPD: DHCPDISCOVER received from client
<-- DHCPD: Sending DHCPOFFER to client
–> DHCPD: DHCPREQUEST received from client
<-- DHCPD: Sending DHCPACK to client

And the actual debug output.

DHCP Message Table

IOS DHCP Server Configuration
The lease command is in days. Also, the excluded-address field can be a range low to high, for instance the snippet below will exclude the range of .1 to .50 on the 10.2.1.x subnet.

DHCP Server configuration:

DHCP Helper Address
DHCP uses broadcast address because the client does not have an IP address with which to communicate. Routers do not forward broadcasts, therefore routers need to set up to forward DHCP traffic, the IP helper-address command also forwards the following protocols:
TFTP
Domain Name System (DNS)
Internet Time Service (ITS)
NetBIOS name server
NetBIOS datagram server
BootP
TACACS

An IOS router as a DHCP client:

DHCP Troubleshooting
show ip dhcp conflict
show ip dhcp binding

clear ip dhcp binding *
This is not necessarily the best command to run on a production router because you clear the addresses the router knows it has given out.
clear ip dhcp conflict *
debug ip dhcp server packet
See the listing above.
debug ip dhcp server events

sh ip dhcp pool

Additional Source:
Document ID: 27470

Definitions
NAT Types
Static NAT — A one-to-one mapping of private to public IP addresses, best used for a device that needs access from outside the AS.
Dynamic NAT — A dynamic one-to-one mapping between private and public IP addresses, however, the mapping can vary and depends upon the addresses left in the pool.
NAT Overloading — PAT, allows multiple private addresses to masquerade as one public IP address by using layer 4 port numbers to differentiate sessions.
Overlapping NAT — Used when the same subnets are in use in two locations and addresses overlap.

NAT Address Types
Inside Local — A private address referencing an inside device.
Inside Global — A public address referencing an inside device.
Outside Local — A private address referencing an outside device.
Outside Global — A public address referencing an outside device.
The TSHOOT book had a good mnemonic that helps everything else fall in place, global starts with g, it means good, good being a routable address on the internet.

Order of operation for an interface, inside to outside network.
1. Decryption of IPsec traffic
2. Input ACL applied
3. Input policing applied
4. Input accounting applied
5. Policy-based routing (PBR)
6. Redirecting traffic to a web cache
7. NAT translating local to global addresses
8. Crypto map application
9. Output ACL applied
10. Cisco IOS Firewall inspection performed
11. TCP intercept feature applied
12. Encryption performed

Order of operation for an interface, outside to inside network.
1. Decryption of IPsec traffic
2. Input ACL applied
3. Input policing applied
4. Input accounting applied
5. NAT translating global to local addresses
6. Policy Based Routing (PBR)
7. Redirecting traffic to a web cache
8. Crypto map application
9. Output ACL applied
10. Cisco IOS Firewall inspection performed
11. TCP intercept feature applied
12. Encryption performed

Troubleshoot NAT
show ip nat statistics
Displays general NAT information of the router.

sh ip nat translations
Shows the current translations on the router. These can be reset with the command clear ip nat translation *.

debug ip nat
Shows real time source and destination of NAT sessions on the router.

There are three planes of a router that need to be secured, the management plane, control plane and data plane.

Management Plane
Used to access and configure a switch or router. It is secured through SNMPv3, TACACS+, VTY ACLs and SSH. It is also a best practice to have role based CLI views.

TACACS+ and RADIUS are used in part to secure user access to the management plane, the major differences between them:

Sources:
Cisco Document ID 13838
TSHOOT Book p.287

Control plane
Includes routing protocols and spanning tree used between routers and switches, it is the ability of a router to route. The control plane can be secured by the command auto secure, routing protocol authentication, and CPU/memory thresholding.

See each routing protocol discussion for troubleshooting steps.

Securing STP
Root Guard — Is enabled on a per-port basis. When a port receives a superior BPDU, with a lower bridge ID, the local switch will not allow the new switch to become the root. Instead the port is changed to root-inconsistent state, no data can be sent or received until the BPDUs stop.

BPDU Guard — PortFast moves an end-user port to forwarding state without going through all of the STP checks and can induce loops in the network. If any BPDU is received on a port where BPDU guard is enabled that port is put into errdisable state. It can then be recovered manually or through the errdisable timeout function.

Data plane
Forwards data through a router or switch. The data plane can be secured through ACLs, 802.1x, Unicast Reverse Path Forwarding (uRPF), IPsec VPN tunnels.

Securing DHCP and ARP:
DHCP snooping — With DHCP snooping enabled a switch port is either trusted or untrusted. Any DHCP replies coming from an untrusted port are discarded because they must have come from a rogue DHCP server. Additionally that switch port is shut in the errdisable state.

Dynamic ARP Inspection (DAI) — Helps to prevent ARP spoofing attacks. DHCP snooping keeps track of completed DHCP bindings including MAC address, IP address offered and lease time. This database is used by DAI to stop man-in-the-middle style attacks.

802.1X
Port-based authentication is a combination of AAA authentication and port security. An 802.1x port begins in an unauthorized state and requires the client to authenticate before it is allowed to communicate. The three components of an 802.1x network are:
Supplicant — Device trying to gain access.
Authenticator — Acts as an intermediary (proxy) between the host and the authentication server, requesting identity information from the host, verifying that information with the authentication server, and relaying a response to the host.
Authentication Server — Performs the actual authentication of the supplicant. The authentication server validates the identity of the supplicant and notifies the switch if it is authorized to communicate on the network.
Source:
TSHOOT Book
6500 802.1X Configuration Guide

Today I read about Vino’s unpopular win at L-B-L. That is fine, we can all sit and wonder why or how, but there was no outrage about Valverde getting third. Valverde could not compete in the 2008 TdF because it went through Italy and the Italians had proved he doped through a genetic match in the Puerto Affair. Now the UCI is trying get a worldwide ban on Valverde racing but no one cares.

On the same front page where they are excoriating the win by Vino they also state that Valverde’s Italian ban has been confirmed by Swiss courts. Why is Valverde still racing? Where is the outrage? Why are fans not complaining that Valverde was caught doping, has been trying to get off on a technicality, and is still racing?

Boggles the mind. Whether or not Vino is contrite has nothing to do with him paying the price. He was suspended for two years, did his time and is back racing. Valverde should not be racing at all.

Tailwinds.

Over the past couple of days I hacked an older version of DocMgr to authenticate users to Active Directory. I thought it might be interesting for some readers to see my thought process, hence this post.

All of the scripts in this post along with my test directory and my changes to DocMgr can be downloaded here.

First I started out with a simple php form for authentication. The input portion is in the download, this is the processing side. Some interesting notes about this script. At first I was using the full distinguished name (DN) of the user but that doesn’t scale. It was not until I downloaded adLDAP that I saw you could just use username@domain.org to authenticate. I don’t know why I never tried, but that stymied me.

The reason this script looks for groups the user belongs to is because originally I figured I would limit access by group. The problem is that DocMgr has it’s own permission system and you must be in the database in order to be able to login. The script at the bottom is how I dealt with that limitation.

The ldpad_set_option and ldap_get_option calls were for testing once I began hacking DocMgr. I added them in to test an error I was getting in ldap_search.

After finishing my script it was time to move on to DocMgr. I turned on DEBUG in the config.php and began placing debug output throughout DogMgr so that I could see the flow of function calls as the program ran.

I wrote down the flow of some key authentication functions and then began reading them to understand what each did and where I should start hacking. Then I went and broke the old ldap includes into open-ldap and ad-ldap includes and set up the ifdefs in the different files. Doing this I was able to still use my OpenLDAP authentication and could hack on ad-ldap.

I set up the ad-ldap-config.php file by comparing the output of ldapsearch in ldif format on both OpenLDAP and AD. We made some changes to the AD population script and moved forward using employeeNumber instead of uidNumber.

After I got the correct entries in AD and OpenLDAP figured out it was just a matter of hacking my test script into the correct functions and testing. There were a couple of errors that I needed to fix. The first was caused by searching by for sAMAccountName=* in the else portion of the ldap_account_search function of docmgr/include/ad_ldap.inc.php.

This was the first error and fix:

This was the second error and fix:

Once I finished making DocMgr authenticate to AD I wanted to make the integration into AD complete so that I did not have to manually add or remove users within the web interface. The following is the script that does that.

It is run every morning from cron.

And the code for allowing access to DocMgr. I truncate the table and just add the users each time.

I wish I knew who to attribute the quotes who have no attribution. If you know the author, please contact me. These are not my own. I will update this occasionally.

Associates do a task until they get it right, experts do a task until they can not do it wrong.
— Relevant to CCNA vs. CCIE.

I’m confronted by my own ignorance or misunderstandings all the time.
— Anthony Bourdain

The worst elements in a group reflect on everyone who chooses to associate with them.
— mtbr.com

In the modern world the stupid are cocksure while the intelligent are full of doubt.
— Bertrand Russell, this is known as the Dunning-Kruger effect.

Never attribute to malice what can be adequately explained by incompetence.
— Common saying

We are what we repeatedly do, therefore excellence is not by chance but a habit.
— I have this on my desk at work. Why settle for being mediocre.

When your IQ is higher than your body weight, you can’t always find what you need at the family bookstore.
— Chris MacAskill, SmugMug.com

The patent system should not be a roulette wheel.
— Richard Silver

If it ain’t broke, you’re not trying hard enough.
— Google Test Blog

A person who never made a mistake never tried anything new.
— Albert Einstein

Don’t worry about people stealing your ideas. If your ideas are any good, you’ll have to ram them down people’s throats.
— Howard Aiken

Strive not to be a success, but rather to be of value.
— Albert Einstein

Open source asks “How can I fix this.” Closed source asks, “How can you fix this for me.”
— Richard Silver

Information is not knowledge. The only source of knowledge is experience.
— Albert Einstein

Profit is accounting opinion, but cash is fact.
— Chris MacAskill, SmugMug.com

At the Circus we are removing the last vestige of our OpenLDAP implementation and moving it to Active Directory. As a result I’m going to document some of the odd scripts I have written as glue to help keep things running. Some of the scripts are too long to document and will just be kept in my script library, while others like the ones below, are generic enough that they might help someone else.

This is some documentation for how to query an AS400 from Linux. When I did this project I actually talked to one of the developers of the iSeries Access programs for help. He ended up sending me a snapshot rpm to get it all working.

Here are the configuration files.

This is a simple script to list the drivers available on the system. This does not look like a script I wrote so I am reticent to take ownership.

And the resulting output from the script above.

This is a simple script to query an AS400. You will have to get with your AS400 administrator because the the query is actually an odd series of fields, at least it was for our organization. I just wrote this to learn how to query the AS400, it was never used in production.