Chainring Circus

The other night we were doing a hardware upgrade on a cluster and testing. We were working with the command clusvcadm to relocate a service from one host in the cluster to another but the originating server kept getting power fenced. We assumed it was the command switches we were running so I went straight to the man page, my coworker went straight to google. Just for reference there is a 10 year difference in our ages, I grew up with man pages and it is a pet peeve of mine when either no man page exists or it is a terrible placeholder. I digress, through his search he came upon a webified man page while I was reading the man page. When I needled him about it his answer was, “But mine is nicely formatted and I can search the web page.” I was surprised, I can search the man page too, right in the pager and can even change man page viewers by changing the PAGER variable.

Three weeks ago I needed to bring up an https server on Ubuntu and spent 45 minutes googling around reading old, outdated or completely wrong howtos before finally going to help.ubuntu.com and 20 minutes later it was done.

The same thing happened over the past couple of weeks working with Xen and VirtualBox. I’ve toiled away looking at poorly written documentation and even mentioned it in my last Red Hat class. The instructor worked for Red Hat and took umbrage with my statement. He was amazed that I did not think Red Hat had great documentation, I was even more shocked that he considered their documentation more than rudimentary. Have a look for your self at the Red Hat documentation.

Just this week I was helping a friend who is the server and network administrator for a small school system configure the proper etherchannel load balancing for a server and he was frustrated at the Cisco documentation. I was astonished. It seemed that he was overwhelmed. He was stuck googling around trying find the “right” documentation rather than learning the layout of the Cisco documentation website.

The point of this post is that lately it seems I waste more time trying to find good information through searching on the web than trying to find the best source of information.

I spent a considerable amount of time over the last couple of days working with DRBD and Heartbeat.

Below are the links I used to get things running:
http://wiki.centos.org/HowTos/Ha-Drbd
http://www.howtoforge.com/vm_replication_failover_vmware_debian_etch_p3
http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Clusters_from_Scratch/s-intro-pacemaker.html
http://www.drbd.org/users-guide/s-heartbeat-r1.html
http://www.drbd.org/users-guide/s-heartbeat-config.html
http://www.drbd.org/users-guide/s-heartbeat-crm.html

Part of my problem was not understanding the difference between R1 and DRM style clusters and their accompanying daemons; heartbeat, pacemaker and the different protocol versions. Pacemaker is a more advanced cluster resource manager that can work with both Corosync and Heartbeat. Heartbeat uses an older protocol whereas pacemaker uses OpenAIS to be compatible with RedHat cluster services.

Regardless here are my notes for configuration, and just for completeness my notes are a mix of doing this first on VMWare and then on a Xen cluster so any inconsistencies are a result of doing this multiple times in different environments. Regardless the errors are mine and I would recommend reading the documentation linked above.

The basics behind the setup is that DRBD replicates data between two servers. DRBD is the network block device that mirrors the data. The heartbeat daemon keeps track of the shared IP, the daemons that are in HA and runs the init scripts appropriately.

DRBD Initialization

Format the disk:

Make sure that the names names are consistent throughout all of these configuration files. This may mean ensuring they are correct in DNS and /etc/hosts.

Locally configure name for this server:

DNS name for this server:

The /etc/drbd.conf file was designed to allow a verbatim copy on both nodes of the cluster.

Copy the configuration file:

Tried to start DRBD but got an error:

I did not initialize the meta data storage and this needs to be done before a DRBD resource can be brought online. The DRBD resource needs to be down or detached from its backing storage.

Check the status:

Make it primary:

Make a file system:

Testing the filesystem:

On the second server:

Heartbeat R1-style

Heartbeat in R1 configuration uses 3 files that must be configured if you are using the heartbeat protocol.
/etc/ha.d/ha.cf
/etc/ha.d/haresources
/etc/ha.d/authkeys

Before we take care of the ha.cf file we need to set up the ha_logd configuration file.

And make changes to the logd.cf file accordingly. Be sure to copy /etc/logd.cf to both servers. Also note that I had to completely stop and then restart the heartbeat daemon for my logging changes to take affect.

Cluster Management
To take over cluster management from a primary server:

Relinquishing cluster management to a secondary server:

The order of operations as set by the init scripts:

Notes for Xen users:

To allow live migration on Xen:

Split-brain
Playing around this morning I got the cluster into split-brain.

Here is how to fix it.
Run this first on the secondary node.

Run this on the primary node.

Every once in a while I get to write a neat piece of code that I can share. This is one of those times. I realize it is not large and by PerlMonk standards not very elegant. The problem therein lies with maintainability over the next few years. Regardless I like what I wrote and would like to share.

At the Circus we had a pretty good idea that we had some data leakage. Nothing like people taking off with everything needed to get home loans and rip off customers, just people not thinking about what they send through email. We didn’t know the extent of the problem or even if we had one. We just weren’t sure. Our C-level executives didn’t believe that employees would be so careless with customer data. We decided to find out.

I must say that the results were actually quite positive. We had a couple of people email work related data home so they could work at home over the weekend and a few emails regarding employment, but they were originated by the prospective employee.

Regardless, in order for us to find out I wrote a few scripts that hook into our email system. One that I am particularly proud of recurses through a directory of email messages and attachments scanning each file for relevant data.

Please note that by the time these scripts touch the data it has been scrubbed by the antivirus and other checks we have in place. I am only looking for keywords or regular expressions that would indicate customer related data loss.

Let me explain the directory structure. Under the email system is the directory /var/spool/filter that contains every email message that has been sent in the last 30 minutes. There is a cleanup process that erases all the files in that directory and that is actually where I wrote the hook, in the cleanup process. Here is a sample listing of the directory.

As you can see, each email header ends with a .hed extension and the message is in .txt format. The ETP.doc file is an attachment.

The subroutine I am most pleased with is the one that recurses through the directory structure. The slurp command returns a hash and if it is a subdirectory then it is a hash as well. I look for it with the following line of code.

That is how I find subdirectories to push onto the stack of recursive calls. As it traverses each directory it just looks at each file extension and makes a determination as to what to do with it.

I realize most system administrators are asking why I didn’t use the file command to make sure the script was acting appropriately for each file type but that does not work with the new Microsoft document types.

I thought it was a fun project and I enjoyed writing what I felt was an interesting piece of code.

I spent a considerable amount of time over the last couple of months testing different restore processes. This is my documentation for restoring Veritas/Symantec backups to a Linux server.

The general outline is this:
1. Create a LiveUSB drive to boot CentOS with a persistent overlay.
2. Install Symantec backupexec on the LiveUSB drive.
3. Recreate the drive layout on the new server.
4. Restore to the new server.

Create LiveUSB
CentOS makes a LiveCD toolset for CentOS. They also have directions for how to create a LiveUSB drive with persistent overlay. Please follow those links for more in depth directions.

You must install CentOS LiveUSB on an ext2/3/4 formatted USB drive in order for Symantec to work. If you leave the VFat partition Symantec will not work properly and you will get the error “An unknown error occurred within the NDMP subsystem.” Once I reformatted the USB drive as ext3 and installed a new LiveUSB with persistent overlay Symantec worked. My guess is it has to do with permission bits but that is only a guess.

I downloaded the LiveCD tools for Centos here.

Here is some of my history from that server:

LiveUSB Setup
I wanted to give it a persistent name and IP address for use in our data center. For some of this I was also shooting in the dark in order to get Symantec working, for thoroughness I include it here.

After I configured the hostname and network settings I rebooted to make sure that the persistent overlay worked. I also turned on sshd and set it to runlevel 3 in /etc/inittab because I did not want to mess with a gui, but that is your choice. When everything came up properly I installed Symantec and we did a test restore.

Install Symantec
I cover installing Symantec on Linux in another post here. You need to install an older package for compatibility:

The specific Symantec rpms I installed are listed below. I did try a newer package from Symantec but it did not allow us to restore erroring with a different message. I will also say that was when we were on a VFat partition. Once I got everything working on an ext3 partition I quit testing.

Recreate Drive Layout
For thoroughness I am going to cover creating the logical volumes that are default for CentOS and RHEL.

First I need to lay out the drive mappings. This is from the old server which I am cloning onto a similar server. In this section I am just going to show the output of a number of commands that confirm the file system layout of the server.

File layout on the old server
From the file /etc/fstab:

From the mount command:

From the fdisk command:

Working my way up from the bottom of the LVM stack with the physical volume, the volume group and finally the logical volume.

From pvdisplay:

From vgdisplay:

From lvdisplay:

On the old server the drive is broken into two partitions, sda1 and sda2:
sda1 /boot 100MB
sda2 Volume Group ~200GB

The volume group on the old server on the sda2 partition is broken into two logical volumes:
LogVol00 / ~200GB
LogVol01 swap ~2GB

It is important to remember that the drive mappings on the old server will not necessarily match the mappings on the new one. For instance on the old server the raid was on /dev/sda and on the new server the raid drive is mapped on /dev/sdb. That is only because I am booting from /dev/sda on the LiveUSB, under normal circumstance it will come back up as /dev/sda.

Working on the new server recreate the partitions

Turn off the the LVM in order to make changes, this is just a precautionary step if you have repartitioned your drive.

Create the LVM.

Create and activate the volume groups.

Finally, create the logical volumes. Even though I have 1.1T I decided to start using 800G, leaving myself room if I want to add another mount point.

Read in the new volume groups.

Format all of the partitions:

Here are some quick commands if you mess up and need to delete any of the LVM stack.

Restore
Mount the root under /mnt and then have Veritas restore to that mount point.

If you have made it this far then you must really need the information. Here are a couple of screenshots from our backup guru to help in the restore process.

“Preserve Tree”, by default it is selected.

Select “Restore over existing files”, “Restore all information for files and directories” and “Preserve tree”.

Install grub on the new machine
During the restore we restored all of the files and directories to /mnt, including /boot. In order to get everything working again we need to setup the boot directory and then grub. But Red Hat and CentOS 4.X uses legacy grub.

Copy all of /mnt/boot to the real /boot directory.

Restore /dev and /tmp
Depending upon your backup options you may need to restore the /dev directory and create a tmp directory. You need to set the sticky bit on /tmp.

Finally you need to set up your ethernet interfaces by editing the file,
/etc/sysconfig/networking/devices/ifcfg-eth0

At the Circus we have a password policy to change all passwords every 90 days. Today it was brought to my attention that one of the linux servers was not following that policy. I confirmed that was true and after a little digging I found that it was only accounts that had been migrated from AIX to linux. But we couldn’t force around 2000 users to all change their passwords at the same time because we would inundate the help desk.

This is the script that I wrote to fix the problem and distribute the password changes over a month. The result is that there are only 78 users per day that are forced to change their password each day over a 28 day period.

Now that RHEL6 is out I’ve begun playing with upgrading our older RHEL servers, starting with anything that is RHEL4 and then moving forward to RHEL5. The simplest one to start with was one of our network management servers. They are vital to our job but not customer facing and we can deal with some downtime on them.

My original plan was to install a base RHEL4 on the new server, which would give us a platform with the base system installed and the restore software hosted, then restore from backup over the top.

I took an old server we had that still had maintenance and pressed it into service. I wanted to take the opportunity to test our backups as well as the upgrade path from RHEL4 to RHEL6. So I installed RHEL4 on the new server and ran up2date to make sure it was the “latest” and greatest.

On the old server I ran up2date and then I queried the rpm database to see what packages were installed. The problem is that you cannot pipe rpm -qa output as input into an update script. Up2date wants “freetype-devel” as the package name, not the whole package name as listed in the output, “freetype-devel-2.1.9-17.el4_8.1″ note the version number. Notice also that some package names have multiple dashes while others have only one dash.

I could not easily use cut in a simple bash script so I wrote the following quick hack.

It has been interesting testing our backup software. It appears it will take some refinement for us to get the restore process worked out.

Aren’t statistics wonderful. I was looking through some referrer traffic and it appears that McKesson Star and DHCP are often googled and this blog comes up as #1 with that query. So I figured I had better write a post on how to set up McKesson Star and DHCP to all play well together.

Which leads to a funny story. When I first came to my present employer all PCs that accessed Star had static IP addresses. Well to be fair not all of them, but the default-lease-time was literally set for one year and IP addresses were used in the ports table. At the time we had ~1,500 PCs and 1,000 of them were static IP addresses. Woe unto you if you had a laptop and tried to access Star.

I guess the previous administrator was thinking he would only have to change an IP address in the ports table if the PC was turned off, or once annually _if_ it got a new address upon a renewal request. My day was filled with changing DNS entries and fixing that was high on my list of priorities.

We use ISC BIND and DHCP so let me give you an example of my DHCP configuration. I have another post on DHCP here.

/etc/tcpd.conf
McKesson wrote their own telnet daemon. The reason is because the view you get in Star as well as your default printer is set according to a DNS lookup done by their daemon. The McKesson telnet daemon options are set in /etc/tcpd.conf. Let’s discuss this next because how you define name lookups also makes a big difference. As a side note, our tcpd.conf did not change when we moved from AIX to Linux.

From /etc/dhcpd.conf:

What does all of this mean? Keep in mind that UNIX is case sensitive and so is Star. What this means is that defining a computer name in Star as well as on the PC, it is important to make sure that they all match. That is why it’s easier to use an IP address. Because the default file does not specify MAPNAME and therefore whether a PC technician uses HumpBack or ALLCAPS, or lowercase makes a difference in how a host name is defined in the Star tables.

GETNAME
The GETNAME option defines whether or not the server does a query for host.chainringcircus.org or just host. If you decide to do a SIMPLE lookup make sure you have all of the possible domains listed in /etc/resolv.conf.

We use simple because a host is defined as host in the Star table and returns the correct information from an nslookup command.

MAPNAME
If you don’t set MAPNAME you will have to make sure that the PC name, DNS name and Star table name all match case. We decided to stay with all lowercase PC names. This is very important so let me explain this again, differently. Go to a windows PC and look at it’s PC name.

Click:
My Computer
–> Properties
–> Computer Name

If it is DoctorPC521 then it will register in DNS as DoctorPC521. It will return from an nslookup as DoctorPC521 and so it had better be in the Star table as DoctorPC521 not DOCTORPC521 or it will not get the correct view and printer.

I hope this helps other administrators trying to figure out how to make McKesson Star and DHCP work well together.

I’ve been playing with more clustering as I prepare for a Red Hat class in August and figured I would write about ClusterIt. I was looking to run a few commands on about six servers and went looking for a simple solution. I believe ClusterIt provides an elegant solution for very little work.

Commands
Here is a list of commands and their description from their respective manpages.
dsh – Run a command on a cluster of machines as defined in the CLUSTER environmental variable.
dshbak – Takes input from the dsh command and formats it to look nicer for the user.
run – Run a command on a machine at random.
rseq – Run a command on a sequence of machines or cluster.
pcp – Copy a file to a number of machines.
pdf – Display free disk space across a number of machines, can be for a single filesystem or the entire machine.
prm – Delete a file, directory or list of files on a number of machines.
rvt – Remote terminal emulator.
clustersed – Quickly dissect cluster files, used to cut individual groups out of a cluster file.
dtop – Used to remotely monitor and display top information, this program segfaulted on my system.

There are also some more involved commands, the daemons for these must be set up on the remote machines.
barrier – Used to synchronize execution of commands on slower and faster machines. When a barrier is set, the process is not released until all of the nodes or processes have met the barrier condition.
barrierd – The daemon portion of barrier that accepts connections from the client program barrier.
jsh – Run scheduled commands on remote machines.
jsd – A simple command scheduling daemon for remote execution.

Installation
The first thing you need to do is make sure you have ssh password-less login set up. I went to our network management server and added a couple of the servers that needed to be able to run commands remotely.

In case you are doing this from scratch, here is the sequence of commands. Generate private/public keys on your management server A.

This will generate two files: ~/.ssh/id_dsa and ~/.ssh/id_dsa.pub. You now want to allow access from this server (A) to the remote server (B) by putting the contents of ~/.ssh/id_dsa.pub from A into ~/.ssh/authorized_keys2 on B.

Make sure permissions are correct and are not writable or readable except by the owner. Do this on both server A and B.

And to verify it works.

Now it’s time to install ClusterIt. I like to have a suite of programs installed in a common directory but don’t want to modify my MANPATH or worry about other nonsense. This is how I installed ClusterIt.

If you read the manpage for dsh or one of the other program in ClusterIt you can see a number of environmental variables and how to set up the ClusterIt environmental variables and files. A snippet of the manpage for dsh.

As a result I set up my .bashrc with the following options for ClusterIt.

Make sure you re-source your .bashrc.

And I have a simple /etc/clusterit/servers file:

Now to test.

Testing
And finally run some commands.

That last command you must have password-less login from the ClusterIt servers back to your management server.

In short, I returned my e-book to Narbik. I would recommend Micronicstraining to anyone. In fact I am now even more likely to go to Narbik’s class then I was before this incident.

The long version.
Later that day I called Micronicstraining to discuss my misgivings with them and actually spoke with Narbik. He was very helpful and understood my concerns saying that there would be no problem giving me licenses for more than one computer. With that I got off the phone placated to some extent. I tried to install LockLizard onto Wine and figured I would just deal with the inconvenience. But the installation onto Wine failed and I did not install LockLizard on Windows nor did I try to open the e-book. I didn’t even unrar the files.

That night I tossed and turned, woke up in the middle of the night and pondered my predicament. I figured I had nothing to loose by asking for my money back. That next morning I sent an email to Narbik explaining my dilemma. It is below.

Sir,

Regretfully I am writing to you to request a refund. I have not
activated my LockLizard license and am requesting that you have it
deactivated.

I would like to thank you for taking the time with me on the phone
yesterday. I had fewer misgivings concerning the number of computers
I would be allowed to study on after our conversation, however, I have
developed a study routine over the past 18 months and shoehorning
Windows into that process would not be beneficial at this time. I do
realize the lab PC runs Windows but I had already decided the last few
months of lab practice would be done in a Windows environment, not the
core of my studies.

I am truly disappointed. I downloaded the free workbook and have done
a number of labs from it. Because of that previous experience with
Micronics I did not expect the type of copy protection used in the
workbook as there is no mention of LockLizard on the Micronics
website. Over the past few months I have frequently visited the table
of contents for your workbook to map out my studies. My work
environment is based upon Linux, I do not have a Windows PC at home,
and I would be forced to change my study process in order to use the
workbook.

If you decide to change your copy protection to something more along
the lines of O’Reilly Media or Internetwork Expert please contact me,
I will be the first to purchase your workbook in a more portable
format. If you need to speak with me directly, my office phone number
is (xxx) xxx-xxxx and my cell phone number is (xxx) xxx-xxxx.

Sincerely,

Jud Bishop

I use the term hacking in the classic sense, not in the cracker sense.

We moved one of our enterprise electronic medical records (EMR) from AIX to Linux over the last few weeks. Go-live was last Thursday night, and I would like to take the time to discuss one of the more interesting hacks we did. It was a long project with some interesting puzzles but this was the most interesting to me.

We were told that you cannot integrate Star/MSE into active directory. As far as I was concerned that was throwing down the gauntlet of a challenge to make it work. We have had our fair share of problems with Samba and AD over the years so my boss was pushing to use Likewise rather than pure Samba. We have split infrastructure, most of the virtual servers use Likewise because my boss set them up, whereas all of the pure Linux servers use Samba because I set them up. It boiled down to my boss can hack around Likewise and I am more comfortable hacking Samba. I talked him into Samba so I had to make it work. My boss had hacked Likewise to do something similar so we discussed it and the resulting code is below.

For those who use Star/MSE you probably understand the login process, however, for those who don’t let me explain. Every user who gets a GUI interface on a Star server shares the same home directory under a restricted korn shell. We have about 1,500 users that all share one home directory but it doesn’t matter because the .profile just fires off a GUI program. In a typical setup all of the users are in the hbo group and in the password file their home points to /home/mse.

We configured winbind to use the system files first, then AD. This is so that we could have an orderly move from system authentication to AD authentication.

In AD we made two groups, hbo to map to the Linux hbo group and a nomse group. Then we forced every AD user into /home/mse directory upon login with the following configuration in /etc/samba/smb.conf.

The point of the nomse group is to be able to pick out the users who should not have the GUI fired off upon login. Even though the group numbers do not match and they are not group mapped with the net groupmap command it doesn’t matter. The trick here is that I am looking for group names in the .profile rather than gids. Below is a portion of the .profile, I would include more but I am not sure of the copyright and it is not pertinent to the discussion.