Chainring Circus

I got a new Mac Pro workstation at work and re-wrote some scripts to work on it. This morning I couldn’t find the script under the new file lay out, it was in /Applications so I decided I had better document the script so I don’t have to rewrite it if I can’t find it.

It uses the same script, tle, that I wrote a while ago, it just fires up iTerm instead of Gnome Terminal.

Every once in a while I get to write a neat piece of code that I can share. This is one of those times. I realize it is not large and by PerlMonk standards not very elegant. The problem therein lies with maintainability over the next few years. Regardless I like what I wrote and would like to share.

At the Circus we had a pretty good idea that we had some data leakage. Nothing like people taking off with everything needed to get home loans and rip off customers, just people not thinking about what they send through email. We didn’t know the extent of the problem or even if we had one. We just weren’t sure. Our C-level executives didn’t believe that employees would be so careless with customer data. We decided to find out.

I must say that the results were actually quite positive. We had a couple of people email work related data home so they could work at home over the weekend and a few emails regarding employment, but they were originated by the prospective employee.

Regardless, in order for us to find out I wrote a few scripts that hook into our email system. One that I am particularly proud of recurses through a directory of email messages and attachments scanning each file for relevant data.

Please note that by the time these scripts touch the data it has been scrubbed by the antivirus and other checks we have in place. I am only looking for keywords or regular expressions that would indicate customer related data loss.

Let me explain the directory structure. Under the email system is the directory /var/spool/filter that contains every email message that has been sent in the last 30 minutes. There is a cleanup process that erases all the files in that directory and that is actually where I wrote the hook, in the cleanup process. Here is a sample listing of the directory.

As you can see, each email header ends with a .hed extension and the message is in .txt format. The ETP.doc file is an attachment.

The subroutine I am most pleased with is the one that recurses through the directory structure. The slurp command returns a hash and if it is a subdirectory then it is a hash as well. I look for it with the following line of code.

That is how I find subdirectories to push onto the stack of recursive calls. As it traverses each directory it just looks at each file extension and makes a determination as to what to do with it.

I realize most system administrators are asking why I didn’t use the file command to make sure the script was acting appropriately for each file type but that does not work with the new Microsoft document types.

I thought it was a fun project and I enjoyed writing what I felt was an interesting piece of code.

At the Circus we have a password policy to change all passwords every 90 days. Today it was brought to my attention that one of the linux servers was not following that policy. I confirmed that was true and after a little digging I found that it was only accounts that had been migrated from AIX to linux. But we couldn’t force around 2000 users to all change their passwords at the same time because we would inundate the help desk.

This is the script that I wrote to fix the problem and distribute the password changes over a month. The result is that there are only 78 users per day that are forced to change their password each day over a 28 day period.

I’m sorry that all of these TestLab scripts are a recurring theme. Work purchased four 3560s and two 1841s for the lab so I have been updating all of my scripts. When I was working on the lab I kept having sessions hang so I wrote a quick script to clear all of the lines on the terminal server.

Now that RHEL6 is out I’ve begun playing with upgrading our older RHEL servers, starting with anything that is RHEL4 and then moving forward to RHEL5. The simplest one to start with was one of our network management servers. They are vital to our job but not customer facing and we can deal with some downtime on them.

My original plan was to install a base RHEL4 on the new server, which would give us a platform with the base system installed and the restore software hosted, then restore from backup over the top.

I took an old server we had that still had maintenance and pressed it into service. I wanted to take the opportunity to test our backups as well as the upgrade path from RHEL4 to RHEL6. So I installed RHEL4 on the new server and ran up2date to make sure it was the “latest” and greatest.

On the old server I ran up2date and then I queried the rpm database to see what packages were installed. The problem is that you cannot pipe rpm -qa output as input into an update script. Up2date wants “freetype-devel” as the package name, not the whole package name as listed in the output, “freetype-devel-2.1.9-17.el4_8.1″ note the version number. Notice also that some package names have multiple dashes while others have only one dash.

I could not easily use cut in a simple bash script so I wrote the following quick hack.

It has been interesting testing our backup software. It appears it will take some refinement for us to get the restore process worked out.

I took and did not pass my TSHOOT exam last week. Bummer. I had not studied specifically for the test very hard. In my mind I had already started thinking about the CCIE written and even started doing the “easy” CCIE labs. I have a long way to go as the best I could do was a 2 hour CCIE lab in 5 hours. That was the best. There were a few that took a couple of days to complete. So when I went to take my test I felt I had a good chance at passing but was by no means assured of a passing grade.

Without breaking the NDA here are some of my thoughts.

How I prepared for the last test:
I went and reviewed this blog. Reading my own writing and doing the commands on the lab routers refreshed some of the old memories.

I configured the TSHOOT topology from scratch again. Then I had my coworker go through and randomly break things. He would log in and break it, give me a hint and then I would go in and fix the configuration. This was easy, possibly too easy, and it was slow. We would only do this once a day and I wouldn’t worry if it took 15 or 20 minutes, I was just playing around in the lab before I went home. That all changed during the test. Because I had configured my lab I knew the configs cold, I had configured it all and I could spot typos or simple errors quickly by just viewing the running configuration. That is not the case during the test.

How I am preparing now:
I am not concerned about theory. I have been studying TCP/IP Volume I with the CCIE written in mind and am on the last chapter.

Over the weekend I made 15 different configurations with errors. I don’t even remember what each ticket does, just that it creates an error somewhere in the lab. I saved them to the flash of each switch or router and wrote the following script. It randomly selects a trouble ticket to load, then calls the testlab update expect script, tlue, and loads the bad configuration on the device.

At this point I am not concerned with the actual error. I did not have a process to follow last week during the test and was not consistent in my troubleshooting process. By being able to load a number of errors in quick succession I will be able to troubleshoot a number of errors using a consistent process. My goal is to bombard myself with random errors like I saw in the test and get my troubleshooting process down pat.

The other key here is to rely on mastering a few commands that tell me the most about the situation on a router or switch and not on reading router configurations to troubleshoot. I feel that was my weakness in my previous studies and became my weakness during the test.

Below is the script that loads random configurations with errors in the lab.

I doubt many home labs match hardware exactly with whatever the vendor has recommended. As a result most of us must script around or manually edit vendor configuration files. Here are some of my earlier scripts that I have shared that have been updated and a couple of new ones.

tl-checklist — TestLab CheckList
So that others will be able to understand the process, I have decided to walk you through the use of these scripts. The first one started out as a checklist of changes to make to each different device for every lab, but that was quickly replaced with a script. Funny thing about scripts like this, it should have probably been written in Perl but I did not expect it to grow to this size.

Each lab has its own directory of configuration files and this script works its way through each configuration file making changes as it goes. Here is an output of the usage example.

I have tried to make the script idiot proof for my own sake. The first thing it does is make a copy of the directory so that a backup of the original exists. Then it puts a marker in the directory so that the script knows whether it has updated those configuration files before. If you have run the script on that directory it will fail gracefully.

Finally it loops through each device and makes changes that are need for each device to work properly with the other scripts and for the configuration to actually match my hardware connections.

The long term reader will recognize that this script dovetails nicely with the previous testlab scripts. For instance, the backbone routers and catalyst switches are renamed to match the hostname expected in the upgrade scripts. It also uses the same command line switches so that it integrates with some of the earlier scripts.

tl-update-router — TestLab Update Router
The next script draws on tlue (testlab update expect) to go through and upgrade any number of switches or routers.

The final line of the usage example is how I normally run this script. This particular example tells the script to upgrade all of the switches and routers to the initial configuration for LAB-1.

In doing so it will first run a configure replace command loading the flash:def file, then it will load the LAB-1 configuration file from the console. The reason I moved from configure replace of a tftp copied file onto flash was because I was not getting consistent results. If, however, I went back and pasted the switch or router configuration over the console input the configuration would take and as a result I just switched to having the script input the configuration over the console session.

tlue — TestLab Update Expect
The tlue script handles the interaction of the switches and routers through expect. A friend gave me a more thorough example but it seemed overkill for my purposes. By this time I know where I usually have problems, the serial interface of R2, but other than that this script works well.

This procedure in the tlue script is the one that sends the configuration over the console cable.

If your hardware matches that of your vendor I would recommend running tl-update-route with the configure replace option, otherwise it is porbably best to configure it from the console.

tl — TestLab
Finally, this is the last script I run which logs me into each every router or switch. It uses tlr for each terminal instance to log into the correct device, then changes the title accordingly. I do not use a “&” at the end of the command to execute in a subshell because it will not automatically log out of every session when terminating the window. Regardless, control of the terminal will be returned to you.

This morning I could not for the life of me remember how to read two variables from one line in bash. As a result I am putting this simple script up here so that I have an easy place to reference.

The input file was a listing of printer IP addresses that are translated is in the file /tmp/printers.txt and looks like this.

Here is the simple code to read both variables.

But it came it out in this format, not much of a problem but I prefer it more legible.

So I cleaned up the output. The first sed stanza deletes the third “.” in the output and the second sed stanza deletes the spaces.

I wrote a couple more scripts this week. These two little gems update my entire lab by running one command from the testlab tftp server. I pass the main script a lab name and it proceeds to update every router and switch in the testlab to the new initial lab configuration. I have tested it on at least one 1841, 2611, 3640, 3550 and 3560. I probably spent more time writing the scripts than it would have taken me to upgrade the testlab by hand for each lab, but I guess I’ll never know. You can download the scripts here.

Let me explain how it is done. Every router and switch has a configuration file named flash:def. When the whole lab has flash:def loaded they can all get to the tftp server for the lab. First side note: it was named flash:default but default is a reserved word in expect so I had to change the name, def is short for default.

In the first round of interaction the script saves every running-config to flash:new and then does a configure replace flash:def. If you just wanted to reset your lab each time, you could stop the script there. You should make sure that the switches all get changed to a VTP mode transparent and set to a new VTP domain, oddly enough I used def as my VTP domain. By making all of the switches transparent if the new config loads any VLANs or changes the domain your VTP domain will be reset. Second side note: I was going to save the original running-config as flash:save, that would still be a trivial change to make and then you could go back and review the config if you needed, but some labs build on top of the next, so I didn’t.

After each router and switch has the def configuration file loaded the script does an ls of the correct lab directory on the tftp server and proceeds to tftp the new configuration file as flash:new, overwriting the previously saved running-config. The tftp server is hooked to the switch named Cat4 which is the last host upgraded by the script so all devices can get to the tftp server.

Finally the script goes back through each device doing a round of configure replace:new. The best part of this is that there is no reload. The 2611 can take nearly 10 minutes to reload so I did not want to have to reload any devices.

In summary this is the process:
1. Save every running-config as flash:new.
2. Load the default configuration flash:def that allows for tftp access for the entire lab.
3. Copy the initial configuration for the new lab to flash:new on the routers that need it, overwriting the saved flash:new.
4. Run configure replace flash:new on ever device, bringing up the newest lab.

Here is a listing of the files on R2.

This is the tlu script. It is just the simple front end to the tlue script.

The expect script is the one that does all of the work and took the most time to write.

I wrote a couple of scripts for the testlab this week and figured I would share them. You can download them here. These could be modified for a GNS3 lab setup easily as well. Not wanting to reinvent the wheel I googled around and started hacking another script. It started off with something like this:

But gnome-terminal sets the environmental variable WINDOWID so I began by changing it to:

But then I read the gnome-terminal manpage to see what other environmental variables it set and decided all the xdotool commands were too much for what I needed. So I simplified into two main scripts, one to handle the gnome-terminal interactions and one to handle the router interactions.

The expect script to handle the interaction with the 2511-RJ getting logged into the router.

And finally the script that logs into every router in the lab, renaming the tab title to match the router name.