Chainring Circus

This morning I could not for the life of me remember how to read two variables from one line in bash. As a result I am putting this simple script up here so that I have an easy place to reference.

The input file was a listing of printer IP addresses that are translated is in the file /tmp/printers.txt and looks like this.

Here is the simple code to read both variables.

But it came it out in this format, not much of a problem but I prefer it more legible.

So I cleaned up the output. The first sed stanza deletes the third “.” in the output and the second sed stanza deletes the spaces.

Cisco documentation is expansive, it is both broad and has great depth, as a result finding what you need is not easy. When you watch a CCIE navigate the “Doc CD” or documentation website you realize how deeply they understand the documentation website.

I watched the free Doc CD lecture the other day at INE, I wish I could find the link sorry, then I watched the IPE documentation tutorial and have been forcing myself to use the site hierarchy rather than searching. I thought I would share some notes.

For instance, today I wanted to look at VACLs so I went:

• Cisco.com
  • Documentation
    • Products
      • Switches
        • LAN Switches Access
          • Catalyst 3560-E
            • Configuration Guides
              • Catalyst 3750-E and 3560-E Switch Configuration Guide

Then clicked around on different VLAN topics. Nothing.

So I fell back to the standard Google search:

Clicked I’m Feeling Lucky. But I wanted to learn where it was really located, so I backtracked by looking at the navigation bar on the left side.

But to really learn the documentation it doesn’t end there. I go back through the Doc CD:

• Cisco.com
  • Documentation
    • Technology
      • LAN Switching
        • LAN Security
          • VACLs
            • Securing Networks with Private VLANs and VLAN Access Control Lists

General Notes
Below are my notes on where to find different documentation on the Cisco website. I actually have this written long hand in a notebook that I still refer to when navigating around. I believe the key is repetition so I try and force myself to navigate “properly.”

This is the site I have bookmarked for the DocCD.

• Cisco.com
  • Documentation
    • Cisco IOS and NX-OS Software
      • Cisco IOS
        • Cisco IOS Software Release 12.4 Family
          • Cisco IOS Software Releases 12.4 T

You can tell from the purple links where I surf.

The links that are the most useful.

• Master Index
• Command References
• Configuration Guides

Master Index
This is useful if you know the command but just want to confirm what it does.  The other day I looked up:

Just click the on the Master Index and the correct alphabetical range, then use find in your browser to find the command you need to reference. It provides a different interface into the Command References.

Command References
Gives a short description of the command and what it does.  Then it breaks out the syntax and the options involved.  Finally it explains when the feature was added and a revision history.  I always wondered how people knew when what command had been implemented and was amazed at their recall.  Now I know, just look at the command reference.  Duh.

Configuration Guides
These are the more in depth guides. They usually start with a technology overview and provide a simple scenario and configuration. Then they get into the details of different commands and option and often include more examples with multiple routers or switches using some of the more advanced configuration options of the technology discussed. This is where I spend most of my time now, however, in the future I hope to be referencing the Master Index more than the Configuration Guides.

Below is an outline of where to find some of the technologies I reference most often.

• Dial Technologies
  • PPP
• IP
  • IP Addressing Services
    • IP Addressing
    • ARP
    • DHCP
    • DNS
    • NAT
  • Application Services
    • SLA
    • Enhanced Object Tracking
    • First Hop Redundancy Protocols
    • UDP
  • Multicast
  • IP Routing: X Protocol
  • IP Switching
    • CEF
  • IPv6
  • OER
• Long Reach Ethernet
  • Broadband Access
    • PPPOE
• MPLS
• Network Management
  • Network Management
    • EEM
• QoS
• Security and VPN
  • Securing the Control Plane
    • Control Plane Policing
  • Securing the Data Plane
    • ACLs
    • CBAC
    • IPS
• System Management
• WAN
  • Frame Relay
  • Layer 2 Tunneling Protocol Version 3
• Additional Legacy Protocols
  • Terminal Services
    • Appendix
      • Regular Expressions

Aren’t statistics wonderful. I was looking through some referrer traffic and it appears that McKesson Star and DHCP are often googled and this blog comes up as #1 with that query. So I figured I had better write a post on how to set up McKesson Star and DHCP to all play well together.

Which leads to a funny story. When I first came to my present employer all PCs that accessed Star had static IP addresses. Well to be fair not all of them, but the default-lease-time was literally set for one year and IP addresses were used in the ports table. At the time we had ~1,500 PCs and 1,000 of them were static IP addresses. Woe unto you if you had a laptop and tried to access Star.

I guess the previous administrator was thinking he would only have to change an IP address in the ports table if the PC was turned off, or once annually _if_ it got a new address upon a renewal request. My day was filled with changing DNS entries and fixing that was high on my list of priorities.

We use ISC BIND and DHCP so let me give you an example of my DHCP configuration. I have another post on DHCP here.

/etc/tcpd.conf
McKesson wrote their own telnet daemon. The reason is because the view you get in Star as well as your default printer is set according to a DNS lookup done by their daemon. The McKesson telnet daemon options are set in /etc/tcpd.conf. Let’s discuss this next because how you define name lookups also makes a big difference. As a side note, our tcpd.conf did not change when we moved from AIX to Linux.

From /etc/dhcpd.conf:

What does all of this mean? Keep in mind that UNIX is case sensitive and so is Star. What this means is that defining a computer name in Star as well as on the PC, it is important to make sure that they all match. That is why it’s easier to use an IP address. Because the default file does not specify MAPNAME and therefore whether a PC technician uses HumpBack or ALLCAPS, or lowercase makes a difference in how a host name is defined in the Star tables.

GETNAME
The GETNAME option defines whether or not the server does a query for host.chainringcircus.org or just host. If you decide to do a SIMPLE lookup make sure you have all of the possible domains listed in /etc/resolv.conf.

We use simple because a host is defined as host in the Star table and returns the correct information from an nslookup command.

MAPNAME
If you don’t set MAPNAME you will have to make sure that the PC name, DNS name and Star table name all match case. We decided to stay with all lowercase PC names. This is very important so let me explain this again, differently. Go to a windows PC and look at it’s PC name.

Click:
My Computer
–> Properties
–> Computer Name

If it is DoctorPC521 then it will register in DNS as DoctorPC521. It will return from an nslookup as DoctorPC521 and so it had better be in the Star table as DoctorPC521 not DOCTORPC521 or it will not get the correct view and printer.

I hope this helps other administrators trying to figure out how to make McKesson Star and DHCP work well together.

I’ve been playing with more clustering as I prepare for a RedHat class in August and figured I would write about ClusterIt. I was looking to run a few commands on about six servers and went looking for a simple solution. I believe ClusterIt provides an elegant solution for very little work.

Commands
Here is a list of commands and their description from their respective manpages.
dsh – Run a command on a cluster of machines as defined in the CLUSTER environmental variable.
dshbak – Takes input from the dsh command and formats it to look nicer for the user.
run – Run a command on a machine at random.
rseq – Run a command on a sequence of machines or cluster.
pcp – Copy a file to a number of machines.
pdf – Display free disk space across a number of machines, can be for a single filesystem or the entire machine.
prm – Delete a file, directory or list of files on a number of machines.
rvt – Remote terminal emulator.
clustersed – Quickly dissect cluster files, used to cut individual groups out of a cluster file.
dtop – Used to remotely monitor and display top information, this program segfaulted on my system.

There are also some more involved commands, the daemons for these must be set up on the remote machines.
barrier – Used to synchronize execution of commands on slower and faster machines. When a barrier is set, the process is not released until all of the nodes or processes have met the barrier condition.
barrierd – The daemon portion of barrier that accepts connections from the client program barrier.
jsh – Run scheduled commands on remote machines.
jsd – A simple command scheduling daemon for remote execution.

Installation
The first thing you need to do is make sure you have ssh password-less login set up. I went to our network management server and added a couple of the servers that needed to be able to run commands remotely.

In case you are doing this from scratch, here is the sequence of commands. Generate private/public keys on your management server A.

This will generate two files: ~/.ssh/id_dsa and ~/.ssh/id_dsa.pub. You now want to allow access from this server (A) to the remote server (B) by putting the contents of ~/.ssh/id_dsa.pub from A into ~/.ssh/authorized_keys2 on B.

Make sure permissions are correct and are not writable or readable except by the owner. Do this on both server A and B.

And to verify it works.

Now it’s time to install ClusterIt. I like to have a suite of programs installed in a common directory but don’t want to modify my MANPATH or worry about other nonsense. This is how I installed ClusterIt.

If you read the manpage for dsh or one of the other program in ClusterIt you can see a number of environmental variables and how to set up the ClusterIt environmental variables and files. A snippet of the manpage for dsh.

As a result I set up my .bashrc with the following options for ClusterIt.

Make sure you re-source your .bashrc.

And I have a simple /etc/clusterit/servers file:

Now to test.

Testing
And finally run some commands.

That last command you must have password-less login from the ClusterIt servers back to your management server.

I wrote a couple more scripts this week. These two little gems update my entire lab by running one command from the testlab tftp server. I pass the main script a lab name and it proceeds to update every router and switch in the testlab to the new initial lab configuration. I have tested it on at least one 1841, 2611, 3640, 3550 and 3560. I probably spent more time writing the scripts than it would have taken me to upgrade the testlab by hand for each lab, but I guess I’ll never know. You can download the scripts here.

Let me explain how it is done. Every router and switch has a configuration file named flash:def. When the whole lab has flash:def loaded they can all get to the tftp server for the lab. First side note: it was named flash:default but default is a reserved word in expect so I had to change the name, def is short for default.

In the first round of interaction the script saves every running-config to flash:new and then does a configure replace flash:def. If you just wanted to reset your lab each time, you could stop the script there. You should make sure that the switches all get changed to a VTP mode transparent and set to a new VTP domain, oddly enough I used def as my VTP domain. By making all of the switches transparent if the new config loads any VLANs or changes the domain your VTP domain will be reset. Second side note: I was going to save the original running-config as flash:save, that would still be a trivial change to make and then you could go back and review the config if you needed, but some labs build on top of the next, so I didn’t.

After each router and switch has the def configuration file loaded the script does an ls of the correct lab directory on the tftp server and proceeds to tftp the new configuration file as flash:new, overwriting the previously saved running-config. The tftp server is hooked to the switch named Cat4 which is the last host upgraded by the script so all devices can get to the tftp server.

Finally the script goes back through each device doing a round of configure replace:new. The best part of this is that there is no reload. The 2611 can take nearly 10 minutes to reload so I did not want to have to reload any devices.

In summary this is the process:
1. Save every running-config as flash:new.
2. Load the default configuration flash:def that allows for tftp access for the entire lab.
3. Copy the initial configuration for the new lab to flash:new on the routers that need it, overwriting the saved flash:new.
4. Run configure replace flash:new on ever device, bringing up the newest lab.

Here is a listing of the files on R2.

This is the tlu script. It is just the simple front end to the tlue script.

The expect script is the one that does all of the work and took the most time to write.

I wrote a couple of scripts for the testlab this week and figured I would share them. You can download them here. These could be modified for a GNS3 lab setup easily as well. Not wanting to reinvent the wheel I googled around and started hacking another script. It started off with something like this:

But gnome-terminal sets the environmental variable WINDOWID so I began by changing it to:

But then I read the gnome-terminal manpage to see what other environmental variables it set and decided all the xdotool commands were too much for what I needed. So I simplified into two main scripts, one to handle the gnome-terminal interactions and one to handle the router interactions.

The expect script to handle the interaction with the 2511-RJ getting logged into the router.

And finally the script that logs into every router in the lab, renaming the tab title to match the router name.

IPV6 Addressing
• 128 bit addresses.
• Simplified header with fewer fields; IPv4 has 12 fields, IPv6 has 5 fields;
• No checksum in the header. This results in more efficient process because in IPv4 the TTL is decremented at each hop, the checksum had to be recalculated at each hop, that is not the case with IPv6.
• No packet fragmentation done by the router, instead an ICMP “packet too big” message is sent to the client. Fragmentation information has been moved to an extension header.

Types of IPv6 Addresses
• Unicast — Send to one interface.
• Multicast — Send to many hosts in a group in the FF00::/8 address range.
• Anycast — Send to the nearest host in a group.

Abbreviate IPv6 Addresses
• Leading zeros in a field can be omitted.
• Contiguous fields containing zeros can be abbreviated with “::”.
• eui-64 addresses use the MAC address for the lower 64 bits of an IPv6 address. The MAC address is split in half and FFFE is placed between the two halves to make the 48 bit MAC into 64 bits, universal/local (U/L) flag (bit 7) in the OUI portion of the address is flipped as well.

Troubleshoot IPv6
sh ipv6 int — Validates the IPv6 and status of interfaces.
sh ipv6 routers — Displays IPv6 router advertisements.
sh ipv6 route — Shows the routing table. DUH.
sh ipv6 protocols — Shows parameters and state of the active IPv6 protocols.
debug ipv6 nd — Debug IPv6 neighbor discovery.
debug ipv6 routing — Display debugging messages for IPv6 routing table and route cache updates.
debug ipv6 packet — Displays the debugging messages for IPv6 packets.

IPv6 Configuration
ipv6 cef
ipv6 unicast-routing
ipv6 address xxxx::xxxx/xxx

OSPFv3
Configure OSPFv3
ipv6 router ospf 6
router-id 10.1.1.10
log-adjacency-changes

interface Tunnel0
no ip address
ipv6 address 2026::34:2/122
ipv6 ospf 6 area 34

Troubleshoot OSPFv3
sh ipv ospf neigh
sh ipv ospf
sh ipv ospf int

sh ipv ospf neigh

sh ipv ospf

sh ipv ospf int

RIPng
• IPv6 multicast address FF02::9 is the destination address for RIPng update messages.
• Link-local addresses used for next-hop addresses
• Metric is hop count and 15 is still the maximum, 16 is unreachable.
• Distance-vector

Configure RIPng
To set up a 3560 switch for IPv6 you must first configure the switch database management (SDM) template to one that supprts IPV6. The rest of the configuration is the same on a router and a layer 3 switch.

Troubleshoot RIPng
sh ipv6 protocols — What protocols are running on what interfaces.
sh ipv6 rip RIP_ZONE — Show general RIPng information concerning the specific RIP_ZONE.
sh ipv6 rip database — Shows the routes in the RIB.
sh ipv6 rip next-hops — Next hops out of this router as seen by RIPng.

sh ipv6 protocols

sh ipv6 rip RIP_ZONE

sh ipv6 rip database

In short, I returned my e-book to Narbik. I would recommend Micronicstraining to anyone. In fact I am now even more likely to go to Narbik’s class then I was before this incident.

The long version.
Later that day I called Micronicstraining to discuss my misgivings with them and actually spoke with Narbik. He was very helpful and understood my concerns saying that there would be no problem giving me licenses for more than one computer. With that I got off the phone placated to some extent. I tried to install LockLizard onto Wine and figured I would just deal with the inconvenience. But the installation onto Wine failed and I did not install LockLizard on Windows nor did I try to open the e-book. I didn’t even unrar the files.

That night I tossed and turned, woke up in the middle of the night and pondered my predicament. I figured I had nothing to loose by asking for my money back. That next morning I sent an email to Narbik explaining my dilemma. It is below.

Sir,

Regretfully I am writing to you to request a refund. I have not
activated my LockLizard license and am requesting that you have it
deactivated.

I would like to thank you for taking the time with me on the phone
yesterday. I had fewer misgivings concerning the number of computers
I would be allowed to study on after our conversation, however, I have
developed a study routine over the past 18 months and shoehorning
Windows into that process would not be beneficial at this time. I do
realize the lab PC runs Windows but I had already decided the last few
months of lab practice would be done in a Windows environment, not the
core of my studies.

I am truly disappointed. I downloaded the free workbook and have done
a number of labs from it. Because of that previous experience with
Micronics I did not expect the type of copy protection used in the
workbook as there is no mention of LockLizard on the Micronics
website. Over the past few months I have frequently visited the table
of contents for your workbook to map out my studies. My work
environment is based upon Linux, I do not have a Windows PC at home,
and I would be forced to change my study process in order to use the
workbook.

If you decide to change your copy protection to something more along
the lines of O’Reilly Media or Internetwork Expert please contact me,
I will be the first to purchase your workbook in a more portable
format. If you need to speak with me directly, my office phone number
is (xxx) xxx-xxxx and my cell phone number is (xxx) xxx-xxxx.

Sincerely,

Jud Bishop

Yesterday I ordered the Advanced CCIE Routing & Switching 2.0 Work Book from Narbik and figured I would share my experience.

If you have read my post on the TSHOOT book you have an understanding of my disdain for DRM and the reasons for it. It boils down to the fact that I use Linux as my primary environment at work and home, we don’t even have a Windows PC at home, and most of the DRM out there requires Windows. So then I have to load whatever I need on my work laptop, but if I am studying at home I have to make sure I bring my laptop home.

Imagine my disappointment when I got the following email from Micronics after I spent $350 on an e-book. I could not find anything on the Micronics website that says the DRM is this draconian.

If you follow the link inside the quote, it says you must use either Windows or a Mac. I am going to say this again, get SafariBooksOnline, O’Reilly is a company that understands technical people and caters to them. I am sure that Narbik is a great teacher and I hope these books are as good as they say, otherwise I will know I got ripped off.

Dear Student,

You will receive three separate emails.
1. Locklizard License
2. Vol. I & II
3. Vol. III

Since you will have only one license, choose a PC or Laptop that your Secured File will reside.

Please follow these procedures before you open the attached file:
1. First you need to open, Download and Install Lizard Safeguard Secure PDF Viewer Email (sometimes this email is considered a SPAM and if you have Gmail it goes to “7 or More” section of your Gmail).

2. Once you completed this step, you need to scroll down the page and double click on the .llv file and download.

3. After installing your PDF Viewer, open the Secured PDF Files that you have received as an attachment.

You can check the operating systems requirement on the following link:

http://www.locklizard.com/LockLizard_Secure_PDF_Viewer_v25.pdf

If you have difficulty opening your files you contact me as soon as possible so we can walk you through this process.

Thank you for your business – we appreciate it very much.

Janet Kocharians
Director of Marketing & Sales
Micronics Inc.
Mobile: (818) 331-2419
Fax: (818) 249-8388

I use the term hacking in the classic sense, not in the cracker sense.

We moved one of our enterprise electronic medical records (EMR) from AIX to Linux over the last few weeks. Go-live was last Thursday night, and I would like to take the time to discuss one of the more interesting hacks we did. It was a long project with some interesting puzzles but this was the most interesting to me.

We were told that you cannot integrate Star/MSE into active directory. As far as I was concerned that was throwing down the gauntlet of a challenge to make it work. We have had our fair share of problems with Samba and AD over the years so my boss was pushing to use Likewise rather than pure Samba. We have split infrastructure, most of the virtual servers use Likewise because my boss set them up, whereas all of the pure Linux servers use Samba because I set them up. It boiled down to my boss can hack around Likewise and I am more comfortable hacking Samba. I talked him into Samba so I had to make it work. My boss had hacked Likewise to do something similar so we discussed it and the resulting code is below.

For those who use Star/MSE you probably understand the login process, however, for those who don’t let me explain. Every user who gets a GUI interface on a Star server shares the same home directory under a restricted korn shell. We have about 1,500 users that all share one home directory but it doesn’t matter because the .profile just fires off a GUI program. In a typical setup all of the users are in the hbo group and in the password file their home points to /home/mse.

We configured winbind to use the system files first, then AD. This is so that we could have an orderly move from system authentication to AD authentication.

In AD we made two groups, hbo to map to the Linux hbo group and a nomse group. Then we forced every AD user into /home/mse directory upon login with the following configuration in /etc/samba/smb.conf.

The point of the nomse group is to be able to pick out the users who should not have the GUI fired off upon login. Even though the group numbers do not match and they are not group mapped with the net groupmap command it doesn’t matter. The trick here is that I am looking for group names in the .profile rather than gids. Below is a portion of the .profile, I would include more but I am not sure of the copyright and it is not pertinent to the discussion.