Chainring Circus

I have debated about whether or not to post this project. I wanted this to have a sqllite backend and run on the Cisco AXP but never had the time to go back and hack more on it. I think it could have won the AXP hacking contest, but I didn’t even know about the contest at the time.

I also want to say that this project is not for the inexperienced Linux admin. This project involves a solid understanding of DHCP, DNS and iptables. I have rolled a .tar file of all of the scripts and directories needed to make this happen, but I’ve said it before and I will say it again, you got it off the web, your mileage may vary. Here is the file.

History
The basic idea here is this, we have ~375 printers that were all translated using static NAT, but most of our infrastructure runs off DHCP. So when someone moves an office they unplug their VOIP phone and computer and take it to their new office, but the printer involves manual intervention, not good.

There is one thing to say about a lazy network administrator, it usually means problems like this get fixed one way or the other. I wanted a way to be able to move printers within the “enterprise” and have their NATed address follow them. This is my solution.

NAT

A picture is worth a thousand words so here is a simple diagram.

The 192.168.24.0/24 subnet is shared between the Circus and our Electronic Medical Records vendor, EMR for short. However, the Circus is partially switched and partially routed on campus so we have to NAT IP addresses on the shared subnet into IP addresses in the enterprise. So this diagram shows a translate server with eth0 at 192.168.24.2 and eth1 as 192.168.100.58, eth0:1 is a NAT’ed IP address for a printer somewhere in the enterprise.

Let me explain this again, this time with commands:
WAN address that is shared with another company, in our case a printer:
wan printer — 192.168.24.100 — IP address EMR uses to print.
translate eth0:1 — 192.168.24.100 — IP address EMR uses to print, actually a virtual interface on translate server.
lan printer — 192.168.10.100 — IP address of printer on campus.
translate eth0 — 192.168.24.2 — WAN leg of translate server.
translate eth1 — 192.169.100.2 — LAN leg of translate server.

The wanprinter ip address 192.168.24.100 is a virtual ip address on a linux
server, translate. It would be brought up from the command line like this:

It is then natted from 192.168.24.100 to it’s local ip address:

The other rule that is added only once is to make it look like all traffic
is coming from the translate server:

rsyslog setup

You need to download rsyslog. This is because you need reliable remote logging and the normal syslog server (sysklogd) does not handle it. I am using librelp version 0.1.1 and rsyslog version 3.20.2. The problem that relp/rsyslog fixes is that standard syslog does not reliably log remotely. So what ends up happening is that dhcp log messages will be dropped, eventually it will get you because an address update message will be dropped.

The DHCP servers in our organization run Red Hat Enterprise Linux and the translate servers run Ubuntu LTS. Please check the README or INSTALL document from rsyslog for installation on your platform. The dhcp servers will be remote logging dhcp transactions to the translate servers so that a logwatch plugin can monitor address changes.

Just download these and read the doc/install.html.

To compile on RedHat I had to use –with-gnu-ld after editing etc/ld.so.conf:

I also had to add an environmental variable:

In order to get RHEL standard scripts to work you have to add a symlink to
/usr/local/sbin/rsyslogd from /usr/sbin/rsyslogd:

Make sure to edit the sysklogd init script or copy in a new one. Also make sure you get rsyslogd hacked into init. The easiest way is to run this command after putting my rewritten RH init script on the box. The rewritten script is in the tar file from above as daemon/rh-etc-init.d-rsyslogd.

If you didn’t use my init script, edit /etc/cron.daily/sysklog script to make sure rsyslog is notified about rotation.

Also edit the logrotate configuration at /etc/logrotate.conf:

Please see my dhcp-rsyslog.conf and translate.rsyslog.conf for the rsyslog
configurations for each server. If it’s not self explanatory copy the file daemon/dhcp-rsyslog.conf to the dhcp server /etc/rsyslog.conf. You will have to change the IP address of the translate server from 192.168.100.2 to whatever you are using.

DHCP Servers

Make these changes in NS1 and NS2, or if you have a setup similar to me,
just copy /etc/dhcpd/dhcpd.ad.master.conf from NS1 to NS2 and restart both DHCP servers.

In /etc/dhcpd/dhcpd.master.conf add these lines:

Restart dhcpd.

Translate Server

Our translate server is a Dell 1U 2600 or something like that. It presently
has ~375 virtual interfaces translating printers from the WAN subnet to our
internal addresses.

eth0 == External or WAN address
inet addr:172.22.25.193 Bcast:172.22.27.255 Mask:255.255.252.0

eth1 == Internal or LAN address
inet addr:172.22.100.52 Bcast:172.22.100.255 Mask:255.255.255.0

The internal IP address is only needed for the postrouting rule.

Install syslogscand on this server through CPAN and then copy DHCP.pm to the correct location. If all else fails just head DHCP.pm and see where it tells you to copy it. Then copy the file daemon/etc-syslogscand.conf to /etc/syslogscand.conf.

Finally move rc.local.firewall to a system directory and add it to rc.local
to fire off upon server start.

syslogscand Plugin Configuration

I wrote a syslogscand plugin, DHCP.pm, that watches the output of dhcp and any
time a mac address that it watches is given an IP address it queries through
dns lookups whether or not the mac address has gotten a new IP address. If it
has gotten a new address, the daemon updates the /etc/network/dhcp-nat file
as well as the /etc/network/interfaces file. It then updates iptables with the
new NAT translation.

Below is a dhcp-nat file. It is comma delimited with the first column being
the mac address of the printer, the second column is the internal LAN address,
the third column is the external WAN address and the forth column is the
hostname.

I have also added information to the /etc/network/interfaces file,
notice in the following stanza that I have added comments for the
mac and translate(d) address.

Web Interface
There is also a web interface into this so that PC technicians can interact
with the system easily. In order to make that magic happen I added the
following line to /etc/sudoers:

It allows the user www-data that apache runs under to restart syslogscand as
well as manipulate iptables and interfaces.

Copy all of the files in the cgi-bin directory to your local cgi-bin directory.

Copy all of the files in the html directory to the web server root.

You may have to tweak directory locations in the scripts.

The other day I ran across a post by Cybergurl that showed her home lab. It got me thinking that every serious Cisco candidate has some sort of lab. Some people use GNS3, some have a home rack and some have a rack provided by their employer. I have a combination of all three and figured I would share some of it.

The rack I use is provided by my employer but I have added quite a bit of my own kit. It is kept in the data center and I have the ability to log in from home over VPN or SSH.

The testlab at work is a short rack with this setup:
Work owned:
6 x 3640
2 x 3560
2 x 3550
1 x 3500yl
1 x Dell 1750 Linux server

My personal:
4 x 1841
1 x 2611-XM
1 x 3620
1 x 2511-RJ

The astute reader will notice that in this picture there is also an ASA. I removed my 3620 and added the ASA so that we could configure and test it before deployment. It is also nice to have that many routers because we had the ASA set up in a BGP environment while at the same time I had the rack configured for the CCNP TSHOOT exam.

What does your testlab look like?

This is as close to a rant as I will put on this blog. Don’t get me wrong, I can rant, but there is rarely a reason to rant in public.

I got my TSHOOT book in the mail a couple of days ago. The reading has been slow going in part because I only bought the hard cover. The last version of CCNP books included a pdf of the book on the CD-Rom. Nice. First class even. In this day and age it was refreshing to see a company not try to nickel and dime customers.

Not any more. Now you have to purchase an ebook. I would pay for a pdf of the book like what used to be provided for free. However, that is not how CiscoPress has formatted their new ebook offerings. Check out the this nice press release. Now the customer not only has to purchase the pdf that used to be provided for free, they also have to deal with Adobe DRM on their CiscoPress ebooks and I have decided I won’t pay for that.

I understand that companies need to be able to monetize products, and I agree. I also understand that many of the pdfs ended up on file sharing sites. The way SafariBooksOnline handles pdfs is a great deterrent. Just put my name and customer number all over the document as a watermark. No one would be foolish enough to share content that was so easily traced. Even if I were to edit a watermarked pdf I would be afraid I had missed some metadata.

In the end I just burned a bunch of tokens on SafariBooksOnline and downloaded the whole book, one chapter at a time. I am extremely disappointed in CiscoPress. I would love to tell you how I will never purchase another book from them but they have a captive audience. I will, however, keep a look out for Cisco books from CiscoPress competitors. If you have some favorite Cisco books that are not from CiscoPress please tell me as I am now extremely interested.

I spent the last week working on a project to convert all of the forms printing at the Circus from an AIX server to a Linux server. Because the version we are stuck on is not officially supported by the vendor I had to do some reverse engineering to figure out how things work, this article describes some of the scripts I used to understand what the software was doing and how to make it work the way we needed.

The funny thing about any printer project I get into, usage seems to explode. Yesterday we had a water cooler meeting about how to print from a completely different EMR system. I’m not sure we will do it because we need to know patient location but the printing part is easy. Who knew a paperless organization prints so much.

We have about 500 printers on the network and less than 314 forms, my guess would peg us closer to 150 forms, but when I go through the forms directory I get 314.

Regardless, the number of forms is not the issue. I changed everything I needed programmatically as you will see below. All of the scripts in the post can be downloaded here.

First I needed to get a connection from one of our electronic medical records applications to the new Linux forms print server. I wrote a short script that shows me the arguments passed to the printer as well as the standard input. This allowed me to troubleshoot what was being passed from one server to another and also gave me insight into how the whole process worked.

Once I got the two servers communicating properly it was time to troubleshoot some configuration files that were copied from the AIX server to Linux. I ended up writing the next short script to change some application configuration files that were originally made to work on AIX. This changes the print command from the AIX qprt command to Linux lpd.

Now it was time to write a short script to convert the /etc/qconfig printer configuration file from AIX and add the printers on the Linux server. So I wanted to test adding and deleting a printer from cups on the command line. I thought the following command would work.

If all else fails read the man page, or in this case the manual on the web. It’s pretty cryptic but they tell you that the -m model has to be from the model directory. Where is the model directory?

So we want the model from /usr/share/cups/model. The following command works:

Now it’s time to convert from qconfig to cups. I used the following script to read in qconfig and create the printers. I had to add the sleep because the script got ahead of the lpadmin command. Once I added the sleep it worked and ta-da, 500 printers were created. I realize that printers are added and deleted from the AIX server daily so I had to make sure it would add and delete printers so that we will get them all on go-live day. It’s a rudimentary implementation but it works.

Notes on this script, it was easier than I expected, hence I a have hash instead of just working from the original array. It’s one of those things that I’m not going to go back and clean up for a simple script, sorry.

It’s frustrating when a vendor provides an install script that does not work. Especially when they take the time to print a bunch of fancy output across the screen about how it is working. For those out there this is how you install the Symantec backup client on Linux. This is using CentOS 5.4.

Even though the script says it completed successfully, the install failed because it needed some older libraries.

Looks like we need to make it chkconfig compatible because it is a whole lot easier than creating a bunch of symlinks by hand. Add this to the top of /etc/init.d/VRTSralus.init

And make sure it works.

One of the Cisco guys has posted the topology for the new TSHOOT exam over at the Cisco Learning Network. Looks like I’ll be recabling the lab this weekend.

We have a Barracuda web filter here at the Circus. In general we are pleased with its performance, however, as our internet usage has climbed the days of history has declined. The inverse relationship is due to a “ring buffer” of 250,000 entries in the history log. There are times when we have a few hours of history and that doesn’t sit well when a manager wants to see the browsing history for a user or a pc. So we turned on syslog logging to a remote logging server and evaluated the log parsing packages that mentioned they parsed Barracuda logs. Let me give you a hint, there are not many.

My first hack was just to see what was going on and give a rudimentary understanding, it can be downloaded here.

Which gives output like this:

The basis of that script gave me more understanding to write the following script which can be downloaded here.

This script produces output like the following:

I have a friend in Chicago and another in Boulder who talk about how nice it must be to ride all winter in the south. It kills me. Check the low for each night, because that is the temperature I ride in every day. We ride at 6:00am every Tuesday-Thursday unless it’s raining too hard, it’s too cold or a combination of the two. But what is too cold?

Like a friend said, once you ride in 16 degree weather there aren’t
many of excuses left. I’m not sure many of you know what 16 degrees feels like on a bike ride, a brain freeze when you go down hill. We usually ride mountain bikes when it’s that cold, because the wind chill is too great to ride the road.

This morning was just another training ride, except it had warmed up to 26F.
It was so cold I could feel the wind on my back this morning and figured I had just not tucked in my base layer well enough. It was only when I got home and the back of my jacket was frozen solid that I understood, my bottle had been leaking down my back and the water had frozen my jersey. When it’s in the 20s your bottle freezes shut, but if you put it in your back pocket it usually won’t. Not only had my bottle frozen shut and started to freeze solid, but the water that had leaked out had frozen too.

So to all my friends who still live up north, it’s not all roses in the south either.

I live in Auburn Alabama and as you might have heard, it is a college town with a major college football team.  In the fall, socializing revolves around football.  I believe part of the reason is because there are no professional level sports in the state of Alabama.  Sure we have class 5A baseball or whatever they call the minor leagues, but in reality college football is king in the state of Alabama.  On game days this year I have been riding for TigerShaw.

In my line of work I rarely interact with the general public and don’t interact with end users very often either.  It’s nice at times because the world I work in is binary, black and white, on or off, there is no gray area.  Something works or it doesn’t.  The goal of my work is to abstract out the human element and when you don’t have that human interaction you feel rusty in social settings.

I sold suits when I was in high school and college undergrad.  When I say sold suits, I don’t mean $300 suits at Sears either, I sold suits at Bachrach back before they had a house brand. We sold Armani, Boss, Joseph Aboud and some high end Hart-Marx.  My biggest sales were in the $10,000 range and this was in the 1990’s.  I made good money for a kid.  What I learned during that job was how to work the public, how to be professional in your interactions, and how to sell.  Think about it, if an executive is going to buy a $2,000 to $3,000 suit from a kid, the kid had better be good.

The same holds true riding rickshaw.  I make good money riding rickshaw but I am professional in my dealings with clients and I work the public.  You have to be professional, work hard and sell yourself the same way to make $20 for dragging someone a mile and a half or $120 for selling a $2,000 suit.

Tailwinds.

The five elements of a Cisco wireless network read like an add right of some glossy, it’s here because there might be a question on the test:
Client devices — laptops, phones and
Mobility platform — Composed of lightweight access points (LWAP), wireless lan controller (WLC) and the wireless control system (WCS)
Network unification — WLCs allow for integration into the wired network
Network management — WCS for control and monitoring
Unified advanced services — Catch-all for marketing

There are two types of Cisco wireless implementations, Autonomous and Lightweight. As a networker who has worked on both, LWAP is the way to go. Management is much easier, mobility groups are your friend and because LWAPP is tunnels wireless interactions across the network to the WCS/WLC it makes it much easier to deploy in a split layer environment, L2 and L3 around campus depending upon location and need.

Autonomous APs — Each AP has its own configuration and operates independently. Configuration can be controlled through CiscoWorks WLSE.

LWAPs — Configuration, monitoring and security is centrally controlled through the WLC/WCS. This is the more scalable solution.

Wireless Lan Solution Engine (WLSE)

Provides basic management of autonomous APs:
• Configuration
• Fault and policy monitoring
• Reporting
• Firmware upgrades
• Radio management

Wireless Control System (WCS)

Provides advanced management of LWAPs:
• Location services
• “Self healing” — If you have a WCS you know what this means.
• Failover between WLCs
• Monitoring
• Configuration
• Firmware upgrades