Chainring Circus

Aren’t statistics wonderful. I was looking through some referrer traffic and it appears that McKesson Star and DHCP are often googled and this blog comes up as #1 with that query. So I figured I had better write a post on how to set up McKesson Star and DHCP to all play well together.

Which leads to a funny story. When I first came to my present employer all PCs that accessed Star had static IP addresses. Well to be fair not all of them, but the default-lease-time was literally set for one year and IP addresses were used in the ports table. At the time we had ~1,500 PCs and 1,000 of them were static IP addresses. Woe unto you if you had a laptop and tried to access Star.

I guess the previous administrator was thinking he would only have to change an IP address in the ports table if the PC was turned off, or once annually _if_ it got a new address upon a renewal request. My day was filled with changing DNS entries and fixing that was high on my list of priorities.

We use ISC BIND and DHCP so let me give you an example of my DHCP configuration. I have another post on DHCP here.

/etc/tcpd.conf
McKesson wrote their own telnet daemon. The reason is because the view you get in Star as well as your default printer is set according to a DNS lookup done by their daemon. The McKesson telnet daemon options are set in /etc/tcpd.conf. Let’s discuss this next because how you define name lookups also makes a big difference. As a side note, our tcpd.conf did not change when we moved from AIX to Linux.

From /etc/dhcpd.conf:

What does all of this mean? Keep in mind that UNIX is case sensitive and so is Star. What this means is that defining a computer name in Star as well as on the PC, it is important to make sure that they all match. That is why it’s easier to use an IP address. Because the default file does not specify MAPNAME and therefore whether a PC technician uses HumpBack or ALLCAPS, or lowercase makes a difference in how a host name is defined in the Star tables.

GETNAME
The GETNAME option defines whether or not the server does a query for host.chainringcircus.org or just host. If you decide to do a SIMPLE lookup make sure you have all of the possible domains listed in /etc/resolv.conf.

We use simple because a host is defined as host in the Star table and returns the correct information from an nslookup command.

MAPNAME
If you don’t set MAPNAME you will have to make sure that the PC name, DNS name and Star table name all match case. We decided to stay with all lowercase PC names. This is very important so let me explain this again, differently. Go to a windows PC and look at it’s PC name.

Click:
My Computer
–> Properties
–> Computer Name

If it is DoctorPC521 then it will register in DNS as DoctorPC521. It will return from an nslookup as DoctorPC521 and so it had better be in the Star table as DoctorPC521 not DOCTORPC521 or it will not get the correct view and printer.

I hope this helps other administrators trying to figure out how to make McKesson Star and DHCP work well together.

I’ve been playing with more clustering as I prepare for a RedHat class in August and figured I would write about ClusterIt. I was looking to run a few commands on about six servers and went looking for a simple solution. I believe ClusterIt provides an elegant solution for very little work.

Commands
Here is a list of commands and their description from their respective manpages.
dsh – Run a command on a cluster of machines as defined in the CLUSTER environmental variable.
dshbak – Takes input from the dsh command and formats it to look nicer for the user.
run – Run a command on a machine at random.
rseq – Run a command on a sequence of machines or cluster.
pcp – Copy a file to a number of machines.
pdf – Display free disk space across a number of machines, can be for a single filesystem or the entire machine.
prm – Delete a file, directory or list of files on a number of machines.
rvt – Remote terminal emulator.
clustersed – Quickly dissect cluster files, used to cut individual groups out of a cluster file.
dtop – Used to remotely monitor and display top information, this program segfaulted on my system.

There are also some more involved commands, the daemons for these must be set up on the remote machines.
barrier – Used to synchronize execution of commands on slower and faster machines. When a barrier is set, the process is not released until all of the nodes or processes have met the barrier condition.
barrierd – The daemon portion of barrier that accepts connections from the client program barrier.
jsh – Run scheduled commands on remote machines.
jsd – A simple command scheduling daemon for remote execution.

Installation
The first thing you need to do is make sure you have ssh password-less login set up. I went to our network management server and added a couple of the servers that needed to be able to run commands remotely.

In case you are doing this from scratch, here is the sequence of commands. Generate private/public keys on your management server A.

This will generate two files: ~/.ssh/id_dsa and ~/.ssh/id_dsa.pub. You now want to allow access from this server (A) to the remote server (B) by putting the contents of ~/.ssh/id_dsa.pub from A into ~/.ssh/authorized_keys2 on B.

Make sure permissions are correct and are not writable or readable except by the owner. Do this on both server A and B.

And to verify it works.

Now it’s time to install ClusterIt. I like to have a suite of programs installed in a common directory but don’t want to modify my MANPATH or worry about other nonsense. This is how I installed ClusterIt.

If you read the manpage for dsh or one of the other program in ClusterIt you can see a number of environmental variables and how to set up the ClusterIt environmental variables and files. A snippet of the manpage for dsh.

As a result I set up my .bashrc with the following options for ClusterIt.

Make sure you re-source your .bashrc.

And I have a simple /etc/clusterit/servers file:

Now to test.

Testing
And finally run some commands.

That last command you must have password-less login from the ClusterIt servers back to your management server.

In short, I returned my e-book to Narbik. I would recommend Micronicstraining to anyone. In fact I am now even more likely to go to Narbik’s class then I was before this incident.

The long version.
Later that day I called Micronicstraining to discuss my misgivings with them and actually spoke with Narbik. He was very helpful and understood my concerns saying that there would be no problem giving me licenses for more than one computer. With that I got off the phone placated to some extent. I tried to install LockLizard onto Wine and figured I would just deal with the inconvenience. But the installation onto Wine failed and I did not install LockLizard on Windows nor did I try to open the e-book. I didn’t even unrar the files.

That night I tossed and turned, woke up in the middle of the night and pondered my predicament. I figured I had nothing to loose by asking for my money back. That next morning I sent an email to Narbik explaining my dilemma. It is below.

Sir,

Regretfully I am writing to you to request a refund. I have not
activated my LockLizard license and am requesting that you have it
deactivated.

I would like to thank you for taking the time with me on the phone
yesterday. I had fewer misgivings concerning the number of computers
I would be allowed to study on after our conversation, however, I have
developed a study routine over the past 18 months and shoehorning
Windows into that process would not be beneficial at this time. I do
realize the lab PC runs Windows but I had already decided the last few
months of lab practice would be done in a Windows environment, not the
core of my studies.

I am truly disappointed. I downloaded the free workbook and have done
a number of labs from it. Because of that previous experience with
Micronics I did not expect the type of copy protection used in the
workbook as there is no mention of LockLizard on the Micronics
website. Over the past few months I have frequently visited the table
of contents for your workbook to map out my studies. My work
environment is based upon Linux, I do not have a Windows PC at home,
and I would be forced to change my study process in order to use the
workbook.

If you decide to change your copy protection to something more along
the lines of O’Reilly Media or Internetwork Expert please contact me,
I will be the first to purchase your workbook in a more portable
format. If you need to speak with me directly, my office phone number
is (xxx) xxx-xxxx and my cell phone number is (xxx) xxx-xxxx.

Sincerely,

Jud Bishop

I use the term hacking in the classic sense, not in the cracker sense.

We moved one of our enterprise electronic medical records (EMR) from AIX to Linux over the last few weeks. Go-live was last Thursday night, and I would like to take the time to discuss one of the more interesting hacks we did. It was a long project with some interesting puzzles but this was the most interesting to me.

We were told that you cannot integrate Star/MSE into active directory. As far as I was concerned that was throwing down the gauntlet of a challenge to make it work. We have had our fair share of problems with Samba and AD over the years so my boss was pushing to use Likewise rather than pure Samba. We have split infrastructure, most of the virtual servers use Likewise because my boss set them up, whereas all of the pure Linux servers use Samba because I set them up. It boiled down to my boss can hack around Likewise and I am more comfortable hacking Samba. I talked him into Samba so I had to make it work. My boss had hacked Likewise to do something similar so we discussed it and the resulting code is below.

For those who use Star/MSE you probably understand the login process, however, for those who don’t let me explain. Every user who gets a GUI interface on a Star server shares the same home directory under a restricted korn shell. We have about 1,500 users that all share one home directory but it doesn’t matter because the .profile just fires off a GUI program. In a typical setup all of the users are in the hbo group and in the password file their home points to /home/mse.

We configured winbind to use the system files first, then AD. This is so that we could have an orderly move from system authentication to AD authentication.

In AD we made two groups, hbo to map to the Linux hbo group and a nomse group. Then we forced every AD user into /home/mse directory upon login with the following configuration in /etc/samba/smb.conf.

The point of the nomse group is to be able to pick out the users who should not have the GUI fired off upon login. Even though the group numbers do not match and they are not group mapped with the net groupmap command it doesn’t matter. The trick here is that I am looking for group names in the .profile rather than gids. Below is a portion of the .profile, I would include more but I am not sure of the copyright and it is not pertinent to the discussion.

Last week we had in a consultant from one of our electronic medical record (EMR) vendors and we were working on a RedHat cluster. He was asking me to check whether a service started at a runlevel and wanted me to ls /etc/rc.d/rc5.d, I looked at him and said, “There’s a command for that.” (The command is chkconfig.) I’ve been chuckling about it for the past few days.

The Linux and BSD foundations need to get together and start an advertising campaign, it would be a great parody.

Over the past couple of days I hacked an older version of DocMgr to authenticate users to Active Directory. I thought it might be interesting for some readers to see my thought process, hence this post.

All of the scripts in this post along with my test directory and my changes to DocMgr can be downloaded here.

First I started out with a simple php form for authentication. The input portion is in the download, this is the processing side. Some interesting notes about this script. At first I was using the full distinguished name (DN) of the user but that doesn’t scale. It was not until I downloaded adLDAP that I saw you could just use username@domain.org to authenticate. I don’t know why I never tried, but that stymied me.

The reason this script looks for groups the user belongs to is because originally I figured I would limit access by group. The problem is that DocMgr has it’s own permission system and you must be in the database in order to be able to login. The script at the bottom is how I dealt with that limitation.

The ldpad_set_option and ldap_get_option calls were for testing once I began hacking DocMgr. I added them in to test an error I was getting in ldap_search.

After finishing my script it was time to move on to DocMgr. I turned on DEBUG in the config.php and began placing debug output throughout DogMgr so that I could see the flow of function calls as the program ran.

I wrote down the flow of some key authentication functions and then began reading them to understand what each did and where I should start hacking. Then I went and broke the old ldap includes into open-ldap and ad-ldap includes and set up the ifdefs in the different files. Doing this I was able to still use my OpenLDAP authentication and could hack on ad-ldap.

I set up the ad-ldap-config.php file by comparing the output of ldapsearch in ldif format on both OpenLDAP and AD. We made some changes to the AD population script and moved forward using employeeNumber instead of uidNumber.

After I got the correct entries in AD and OpenLDAP figured out it was just a matter of hacking my test script into the correct functions and testing. There were a couple of errors that I needed to fix. The first was caused by searching by for sAMAccountName=* in the else portion of the ldap_account_search function of docmgr/include/ad_ldap.inc.php.

This was the first error and fix:

This was the second error and fix:

Once I finished making DocMgr authenticate to AD I wanted to make the integration into AD complete so that I did not have to manually add or remove users within the web interface. The following is the script that does that.

It is run every morning from cron.

And the code for allowing access to DocMgr. I truncate the table and just add the users each time.

At the Circus we are removing the last vestige of our OpenLDAP implementation and moving it to Active Directory. As a result I’m going to document some of the odd scripts I have written as glue to help keep things running. Some of the scripts are too long to document and will just be kept in my script library, while others like the ones below, are generic enough that they might help someone else.

This is some documentation for how to query an AS400 from Linux. When I did this project I actually talked to one of the developers of the iSeries Access programs for help. He ended up sending me a snapshot rpm to get it all working.

Here are the configuration files.

This is a simple script to list the drivers available on the system. This does not look like a script I wrote so I am reticent to take ownership.

And the resulting output from the script above.

This is a simple script to query an AS400. You will have to get with your AS400 administrator because the the query is actually an odd series of fields, at least it was for our organization. I just wrote this to learn how to query the AS400, it was never used in production.

One of my coworkers solved a problem today that I should have been able to solve. I got pulled into a project that uses a closed system with it’s own programming language that I had never seen nor programmed. My coworker was trying to figure out why his syntax was not working. So we cranked up the logging in Warn.log, Error.log and Fatal.log. We quickly figured out why it was failing, the program was looking for a file that did not exist and he was not catching that error, there is no catch statement.

So I began trying to write an IF/ELSE statement that would check for the error condition. I was given a .pdf of the manual for the system and just started trying different functions that made sense to me. I wrote a nice little one liner in the SYSTEM command that returned TRUE or FALSE if the FILE environment variable existed.

The problem was that we could not set the environment variable with a variable from the program. It has to be a string literal.

From the Trace.log:

We tried a number of different variations on that theme, imported and exported variables to a subshell, tried different return values from the exit command. We got the system to turn circles but what we really wanted was squares.

Finally my coworker called and said, why don’t we just invert the logic. If it falls through to the end of the IF/ELSE statements it must be in the format we want. Let me take a step back and say this was his code, he should have been the one to figure it out. I believe I helped him figure out what his main problem was, which helped him find the solution. I’m not trying to minimize his work, nor am I trying to minimize my input into the solution.

The moral of the story is that I should have taken a higher level look at the problem to understand the solution. I am quick to believe that I can code my way out of any problem, regardless whether or not I have ever seen the language. What I really needed to do was step back and take a higher level view of the logic, rather than dive into syntax and functions.

I have debated about whether or not to post this project. I wanted this to have a sqllite backend and run on the Cisco AXP but never had the time to go back and hack more on it. I think it could have won the AXP hacking contest, but I didn’t even know about the contest at the time.

I also want to say that this project is not for the inexperienced Linux admin. This project involves a solid understanding of DHCP, DNS and iptables. I have rolled a .tar file of all of the scripts and directories needed to make this happen, but I’ve said it before and I will say it again, you got it off the web, your mileage may vary. Here is the file.

History
The basic idea here is this, we have ~375 printers that were all translated using static NAT, but most of our infrastructure runs off DHCP. So when someone moves an office they unplug their VOIP phone and computer and take it to their new office, but the printer involves manual intervention, not good.

There is one thing to say about a lazy network administrator, it usually means problems like this get fixed one way or the other. I wanted a way to be able to move printers within the “enterprise” and have their NATed address follow them. This is my solution.

NAT

A picture is worth a thousand words so here is a simple diagram.

The 192.168.24.0/24 subnet is shared between the Circus and our Electronic Medical Records vendor, EMR for short. However, the Circus is partially switched and partially routed on campus so we have to NAT IP addresses on the shared subnet into IP addresses in the enterprise. So this diagram shows a translate server with eth0 at 192.168.24.2 and eth1 as 192.168.100.58, eth0:1 is a NAT’ed IP address for a printer somewhere in the enterprise.

Let me explain this again, this time with commands:
WAN address that is shared with another company, in our case a printer:
wan printer — 192.168.24.100 — IP address EMR uses to print.
translate eth0:1 — 192.168.24.100 — IP address EMR uses to print, actually a virtual interface on translate server.
lan printer — 192.168.10.100 — IP address of printer on campus.
translate eth0 — 192.168.24.2 — WAN leg of translate server.
translate eth1 — 192.169.100.2 — LAN leg of translate server.

The wanprinter ip address 192.168.24.100 is a virtual ip address on a linux
server, translate. It would be brought up from the command line like this:

It is then natted from 192.168.24.100 to it’s local ip address:

The other rule that is added only once is to make it look like all traffic
is coming from the translate server:

rsyslog setup

You need to download rsyslog. This is because you need reliable remote logging and the normal syslog server (sysklogd) does not handle it. I am using librelp version 0.1.1 and rsyslog version 3.20.2. The problem that relp/rsyslog fixes is that standard syslog does not reliably log remotely. So what ends up happening is that dhcp log messages will be dropped, eventually it will get you because an address update message will be dropped.

The DHCP servers in our organization run Red Hat Enterprise Linux and the translate servers run Ubuntu LTS. Please check the README or INSTALL document from rsyslog for installation on your platform. The dhcp servers will be remote logging dhcp transactions to the translate servers so that a logwatch plugin can monitor address changes.

Just download these and read the doc/install.html.

To compile on RedHat I had to use –with-gnu-ld after editing etc/ld.so.conf:

I also had to add an environmental variable:

In order to get RHEL standard scripts to work you have to add a symlink to
/usr/local/sbin/rsyslogd from /usr/sbin/rsyslogd:

Make sure to edit the sysklogd init script or copy in a new one. Also make sure you get rsyslogd hacked into init. The easiest way is to run this command after putting my rewritten RH init script on the box. The rewritten script is in the tar file from above as daemon/rh-etc-init.d-rsyslogd.

If you didn’t use my init script, edit /etc/cron.daily/sysklog script to make sure rsyslog is notified about rotation.

Also edit the logrotate configuration at /etc/logrotate.conf:

Please see my dhcp-rsyslog.conf and translate.rsyslog.conf for the rsyslog
configurations for each server. If it’s not self explanatory copy the file daemon/dhcp-rsyslog.conf to the dhcp server /etc/rsyslog.conf. You will have to change the IP address of the translate server from 192.168.100.2 to whatever you are using.

DHCP Servers

Make these changes in NS1 and NS2, or if you have a setup similar to me,
just copy /etc/dhcpd/dhcpd.ad.master.conf from NS1 to NS2 and restart both DHCP servers.

In /etc/dhcpd/dhcpd.master.conf add these lines:

Restart dhcpd.

Translate Server

Our translate server is a Dell 1U 2600 or something like that. It presently
has ~375 virtual interfaces translating printers from the WAN subnet to our
internal addresses.

eth0 == External or WAN address
inet addr:172.22.25.193 Bcast:172.22.27.255 Mask:255.255.252.0

eth1 == Internal or LAN address
inet addr:172.22.100.52 Bcast:172.22.100.255 Mask:255.255.255.0

The internal IP address is only needed for the postrouting rule.

Install syslogscand on this server through CPAN and then copy DHCP.pm to the correct location. If all else fails just head DHCP.pm and see where it tells you to copy it. Then copy the file daemon/etc-syslogscand.conf to /etc/syslogscand.conf.

Finally move rc.local.firewall to a system directory and add it to rc.local
to fire off upon server start.

syslogscand Plugin Configuration

I wrote a syslogscand plugin, DHCP.pm, that watches the output of dhcp and any
time a mac address that it watches is given an IP address it queries through
dns lookups whether or not the mac address has gotten a new IP address. If it
has gotten a new address, the daemon updates the /etc/network/dhcp-nat file
as well as the /etc/network/interfaces file. It then updates iptables with the
new NAT translation.

Below is a dhcp-nat file. It is comma delimited with the first column being
the mac address of the printer, the second column is the internal LAN address,
the third column is the external WAN address and the forth column is the
hostname.

I have also added information to the /etc/network/interfaces file,
notice in the following stanza that I have added comments for the
mac and translate(d) address.

Web Interface
There is also a web interface into this so that PC technicians can interact
with the system easily. In order to make that magic happen I added the
following line to /etc/sudoers:

It allows the user www-data that apache runs under to restart syslogscand as
well as manipulate iptables and interfaces.

Copy all of the files in the cgi-bin directory to your local cgi-bin directory.

Copy all of the files in the html directory to the web server root.

You may have to tweak directory locations in the scripts.

I spent the last week working on a project to convert all of the forms printing at the Circus from an AIX server to a Linux server. Because the version we are stuck on is not officially supported by the vendor I had to do some reverse engineering to figure out how things work, this article describes some of the scripts I used to understand what the software was doing and how to make it work the way we needed.

The funny thing about any printer project I get into, usage seems to explode. Yesterday we had a water cooler meeting about how to print from a completely different EMR system. I’m not sure we will do it because we need to know patient location but the printing part is easy. Who knew a paperless organization prints so much.

We have about 500 printers on the network and less than 314 forms, my guess would peg us closer to 150 forms, but when I go through the forms directory I get 314.

Regardless, the number of forms is not the issue. I changed everything I needed programmatically as you will see below. All of the scripts in the post can be downloaded here.

First I needed to get a connection from one of our electronic medical records applications to the new Linux forms print server. I wrote a short script that shows me the arguments passed to the printer as well as the standard input. This allowed me to troubleshoot what was being passed from one server to another and also gave me insight into how the whole process worked.

Once I got the two servers communicating properly it was time to troubleshoot some configuration files that were copied from the AIX server to Linux. I ended up writing the next short script to change some application configuration files that were originally made to work on AIX. This changes the print command from the AIX qprt command to Linux lpd.

Now it was time to write a short script to convert the /etc/qconfig printer configuration file from AIX and add the printers on the Linux server. So I wanted to test adding and deleting a printer from cups on the command line. I thought the following command would work.

If all else fails read the man page, or in this case the manual on the web. It’s pretty cryptic but they tell you that the -m model has to be from the model directory. Where is the model directory?

So we want the model from /usr/share/cups/model. The following command works:

Now it’s time to convert from qconfig to cups. I used the following script to read in qconfig and create the printers. I had to add the sleep because the script got ahead of the lpadmin command. Once I added the sleep it worked and ta-da, 500 printers were created. I realize that printers are added and deleted from the AIX server daily so I had to make sure it would add and delete printers so that we will get them all on go-live day. It’s a rudimentary implementation but it works.

Notes on this script, it was easier than I expected, hence I a have hash instead of just working from the original array. It’s one of those things that I’m not going to go back and clean up for a simple script, sorry.