Chainring Circus

Long Time No Post

jud — Tue, 27 Dec 2011 00:15:41 +0000

I have been putting off this post because I didn’t really know what to say. So much has changed in the last six months I didn’t know how best to communicate what was going on in my life. Please understand that I am leaving out quite a bit of this story but I believe it needs to be told so that others will understand when they stumble across my blog.

Let’s start with some background. I have been working for the same employer for the last 8 years. It is a regional medical center and a great place to work. Before Fortune magazine stopped allowing non-profits to be on the “100 Best Places to Work” my employer made the list twice.

I started working at the hospital as a Programmer Analyst in early 2003 writing code that allowed hospital systems to communicate and was also the UNIX/Linux administrator. During this time I managed DNS/DHCP, most of the printing and portions of our email infrastructure on *NIX while also writing mostly TCL and Perl code.

When a Senior Network Engineer slot opened I applied and was promoted in early 2006. I earned my CCNA and began to learn quite a bit about networking. Shortly thereafter I decided I was ready to leave the hospital in order to learn more, but my wife loves working here, knows and trusts many of the doctors and made me promise that we would have our children at this hospital.

As a result of that promise to my wife I decided to work on my CCNP and started this blog, figuring both would help me get my next job. During that time I was selected for “50 for the Future” a two year management training program at the hospital and I finished my CCNP. I also became dedicated to my craft. As I learned more and more about networking, I began to understand how much I did not know and it became a joy to learn something new everyday. Not all of the lessons were fun at the time but I was learning so much I loved coming to work.

The next logical step was starting on my CCIE. It sounded like a great challenge and with a family now depending upon me, it was the next logical step for job security. It surely would allow me to get my next job and my promise to my wife was still not fulfilled. We had suffered through two miscarriages to get our first child, and our second child was born in April 2011.

With the birth of our second child my promise to my wife had been fulfilled and I was ready to leave. Literally the day after my second child was born I started applying for jobs. I was offered an interesting position in South Carolina working with a team that fit my goals. Some had their CCIEs and others were working toward their digits. I would fit right in and I knew it was the place for me, so I turned in my resignation. The wrinkle being that the hospital requests a 30 day notice for professional employees.

I had already spoken with my Director about leaving and he was gracious. I had spent $5,000 on my CCIE lab and the hospital had spent another $5,000, but my boss gave me the whole rack. Everything. I cannot describe how much that meant to me, I am truly grateful.

That day my Vice President asked me what it would take for me to stay and I told him I was chasing a dream. While I appreciated the offer and I knew the hospital was a great place to work, it was a goal I had set for myself and I felt I should leave in order to get the experience I needed.

Two weeks later I had both my AVP and VP stop by my desk within ten minutes of each other. My last day was drawing near and they wanted to discuss what it would take for me to stay. I declined a second time. I had made the decision to try to earn my CCIE and I did not want to let my family down. I believed that South Carolina offered me the best chance to achieve my goal.

My last day at the hospital came and went, but it was a tumultuous day. My AVP literally gave me a hug as I walked out the door for the last time and I cried in her arms.

I had a week off between jobs and I had lunch with friends to say goodbye, finished setting up the lab at home and packed. I made arrangements for long term housing in South Carolina and prepared to leave my family while we tried to sell our home.

Thursday evening of my week off I got a call on my cell phone. It was my VP and AVP calling to chat. The first question they asked was whether I was still in town or had I already left for South Carolina. Then they asked me to stay one last time. They had met with our CEO and Executive Vice President and wanted me to stay. I was floored. I had worked for nearly three years getting ready to leave. I had started down a road and I didn’t want to change direction.

But I did. My wife told me that I will never again be offered the chance to work at a company which knows neither what my job title will be nor what I will be doing, just that they want me to work for them. She was probably right.

I was offered the chance to make a difference working for a company that has been good to me and my family. It is a different challenge than what I had been working toward but an interesting one all the same. I am slowly settling into my new job as Technical Services Manager. Some of the changes I have made are for the better, some of the changes have not lasted and others have yet to be implemented, but I am still learning every day.

And so my goals are slowly evolving. It is hard to give up a goal that I have sacrificed so much to earn, without completing it. In the short term I will begin working toward my CCIP in the new year. Currently I do not have the time to complete the CCIE but I believe working toward my CCIP will keep my mind in networking and it will cover topics that I believe are needed if I decide to again try for my CCIE.

Thank you for reading. I plan to post more regularly in the coming year.

TestLab Script in AppleScript

jud — Sun, 14 Aug 2011 18:26:34 +0000

I got a new Mac Pro workstation at work and re-wrote some scripts to work on it. This morning I couldn’t find the script under the new file lay out, it was in /Applications so I decided I had better document the script so I don’t have to rewrite it if I can’t find it.

It uses the same script, tle, that I wrote a while ago, it just fires up iTerm instead of Gnome Terminal.

-- 2011-03-24
-- Jud Bishop

tell application "iTerm"
activate

-- If you don't have this you end up with two terminals
terminate the first session of the first terminal

set iterm to (make new terminal)

repeat with X from 1 to 6
set Y to "R" & X as string
tell iterm
make new session at the end of sessions
tell the last session
exec command "/usr/local/bin/tle " & Y & " testlab.chainringcircus.org"
set name to Y
end tell
end tell
end repeat

repeat with X from 1 to 4
set Y to "SW" & X as string
tell iterm
make new session at the end of sessions
tell the last session
exec command "/usr/local/bin/tle " & Y & " testlab.chainringcircus.org"
set name to Y
end tell
end tell
end repeat

repeat with X from 1 to 3
set Y to "BB" & X as string
tell iterm
make new session at the end of sessions
tell the last session
exec command "/usr/local/bin/tle " & Y & " testlab.chainringcircus.org"
set name to Y
end tell
end tell
end repeat

set the bounds of the first window to {0, 0, 1200, 900}

end tell

Does Google Hurt Efficiency?

jud — Sun, 31 Jul 2011 23:38:06 +0000

The other night we were doing a hardware upgrade on a cluster and testing. We were working with the command clusvcadm to relocate a service from one host in the cluster to another but the originating server kept getting power fenced. We assumed it was the command switches we were running so I went straight to the man page, my coworker went straight to google. Just for reference there is a 10 year difference in our ages, I grew up with man pages and it is a pet peeve of mine when either no man page exists or it is a terrible placeholder. I digress, through his search he came upon a webified man page while I was reading the man page. When I needled him about it his answer was, “But mine is nicely formatted and I can search the web page.” I was surprised, I can search the man page too, right in the pager and can even change man page viewers by changing the PAGER variable.

Three weeks ago I needed to bring up an https server on Ubuntu and spent 45 minutes googling around reading old, outdated or completely wrong howtos before finally going to help.ubuntu.com and 20 minutes later it was done.

The same thing happened over the past couple of weeks working with Xen and VirtualBox. I’ve toiled away looking at poorly written documentation and even mentioned it in my last Red Hat class. The instructor worked for Red Hat and took umbrage with my statement. He was amazed that I did not think Red Hat had great documentation, I was even more shocked that he considered their documentation more than rudimentary. Have a look for your self at the Red Hat documentation.

Just this week I was helping a friend who is the server and network administrator for a small school system configure the proper etherchannel load balancing for a server and he was frustrated at the Cisco documentation. I was astonished. It seemed that he was overwhelmed. He was stuck googling around trying find the “right” documentation rather than learning the layout of the Cisco documentation website.

The point of this post is that lately it seems I waste more time trying to find good information through searching on the web than trying to find the best source of information.

Service Provider Labs

jud — Thu, 09 Jun 2011 21:08:07 +0000

Running across the Hacking Cisco blog made me remember a similar site, CCIE18473.net. I actually spent about 30 minutes looking for that site and it was Tyson Scott from IPE that me helped find it.

I have added this site to my blogroll even though it is not a blog.

Routing Mnemonics

jud — Fri, 03 Jun 2011 13:28:50 +0000

I’ve been keeping track of some the mnemonics that I have come across or have figured out for myself. For instance in general, in layer 2 elections the lower priority usually wins, however, in layer 3 elections the higher priority usually wins.

Layer 2

LACP System Priority
2-bytes priority values followed by a 6-byte MAC address. Lowest system priority makes decisions about the etherchannel setup.

LACP Port Priority
LACP port priority is a 2-byte priority followed by a 2-byte port number. Lowest port priority is used to decide which ports are put in standby mode when not all ports can be put in etherchannel.

STP
Root bridge election, lowest bridge ID wins. Bridge ID consists of:
–2-byte bridge priority from 0-65,535 with a default of 32,768.
–6-byte MAC address
If the bridge priorities are equal, lowest MAC wins.

Root port, lowest root path cost.

Designated port — lowest root path cost or if equal use tie breakers:
1. Lowest root bridge ID
2. Lowest root path cost to root bridge
3. Lowest sender bridge ID
4. Lowest sender port ID

Frame Relay
DCE requires the clock rate, DCE, DTE, clock rate starts with a c and DCE is the one with a c in it.

Layer 3

HSRP
Active router election is based upon priority, highest priority wins. Default priority of 100 and a range of 0-255. Highest IP address on HSRP interface breaks ties.
Standby router is the second highest priority.

VRRP
Election of master is the router with the gateway IP address or if not a “real” IP address, the router with the highest priority. Priority ranges from 1 to 254 with 254 being highest, 100 is the default.

GLBP
Active virtual gateway (AVG) is elected by the highest priority value, tie breaker is the highest IP address in the group. Router priority is 1-255 with 255 being highest, 100 is the default.

OSPF DR/BDR Election
1. Highest priority wins.
2. Highest router ID breaks ties.
Priority range is 0-255 with 255 being highest, 1 is the default and 0 means the router will not participate in the election.

OSPF RID
1. router-id command wins.
2. If no router-id is set, the highest loopback address wins, even if it is not advertised and it is not advertised by default.
3. Highest physical address wins.

OSPF summary-address command or the range command.
The summary-address command is used on an ASBR and has an “S” in it, whereas the area range command is used on an ABR and does not have an “S” in it. Both commands are used to summarize routes.

OSPF ExStart
During ExStart of the OSPF packet exchange the neighbor with the highest RID will become the master and sets the DD sequence number.

DVMRP
An exception to the rule of Layer 2 lower takes priority and Layer 3 higher takes the priority. If two routers are the same distance from the source, the router with the numerically lower IP address becomes the designated forwarder for the network.

BGP best path mnemonic
We love oranges as oranges mean pure refreshment.

We — Weight (highest)
Love — LOCAL_PREF (highest)
Oranges — Originate (local)
AS — AS_PATH (shortest)
Oranges — Origin Code (IGP > EGP > Incomplete)
Mean — Med (lowest)
Pure — Paths (External > Internal)
Refreshment — RID (lowest)

Redistribution
RIP and any other protocol that has the letters R-I-P in it requires a seed metric, RIP, IGRP, EIGRP.

EIGRP MPLS VPN PE-CE SOO

jud — Tue, 31 May 2011 23:47:05 +0000

I couldn’t resist using all of those acronyms.
EIGRP – Enhanced Interior Gateway Routing Protocol
MPLS – Multiprotocol Label Switching
VPN – Virtual Private Networking
PE-CE – Provider Equipment – Customer Equipment
SOO – Site Of Origin

MPLS SOO
MPLS Fundamentals pp. 220-226

BGP->EIGRP and EIGRP->BGP

Advertisement of the SOO BGP extended community attribute is used to identify routes that have originated from a site so that they are not re-advertised back into the same site. Each SOO uniquely identifies the site and allows for the routes to be filtered. SOO filtering is configured at the interface level. It is commonly used when a site contains both VPN and back door links.

From the Cisco document:
The configuration of the SOO extended community allows MPLS VPN traffic to be filtered on a per-site basis. The SoO extended community is configured in an inbound BGP route map on the PE router and is applied to the interface with the ip vrf sitemap command. The SOO extended community can be applied to all exit points at the customer site for more specific filtering but must be configured on all interfaces of PE routers that provide VPN services to CE routers.

Hacking Cisco

jud — Tue, 24 May 2011 13:26:25 +0000

Someone on one of the lists posted about the blog Hacking Cisco so I went over to check it out. Wow. Jarek Rek is the author and he must be putting a ton of time into his studies.

Inspiring.

I have added his blog to the blogroll on the side.

DRBD and Heartbeat

jud — Tue, 10 May 2011 12:59:29 +0000

I spent a considerable amount of time over the last couple of days working with DRBD and Heartbeat.

Below are the links I used to get things running:
http://wiki.centos.org/HowTos/Ha-Drbd
http://www.howtoforge.com/vm_replication_failover_vmware_debian_etch_p3
http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Clusters_from_Scratch/s-intro-pacemaker.html
http://www.drbd.org/users-guide/s-heartbeat-r1.html
http://www.drbd.org/users-guide/s-heartbeat-config.html
http://www.drbd.org/users-guide/s-heartbeat-crm.html

Part of my problem was not understanding the difference between R1 and DRM style clusters and their accompanying daemons; heartbeat, pacemaker and the different protocol versions. Pacemaker is a more advanced cluster resource manager that can work with both Corosync and Heartbeat. Heartbeat uses an older protocol whereas pacemaker uses OpenAIS to be compatible with RedHat cluster services.

Regardless here are my notes for configuration, and just for completeness my notes are a mix of doing this first on VMWare and then on a Xen cluster so any inconsistencies are a result of doing this multiple times in different environments. Regardless the errors are mine and I would recommend reading the documentation linked above.

The basics behind the setup is that DRBD replicates data between two servers. DRBD is the network block device that mirrors the data. The heartbeat daemon keeps track of the shared IP, the daemons that are in HA and runs the init scripts appropriately.

DRBD Initialization

Format the disk:

fdisk /dev/xvdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

The number of cylinders for this disk is set to 10443.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p
Disk /dev/xvdb: 85.8 GB, 85899345920 bytes
255 heads, 63 sectors/track, 10443 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-10443, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-10443, default 10443):
Using default value 10443

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 83

Command (m for help): p
Disk /dev/xvdb: 85.8 GB, 85899345920 bytes
255 heads, 63 sectors/track, 10443 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/xvdb1 1 10443 83883366 83 Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Make sure that the names names are consistent throughout all of these configuration files. This may mean ensuring they are correct in DNS and /etc/hosts.

Locally configure name for this server:

uname -n
drbd01.chainringcircus.org

uname -n
drbd02.chainringcircus.org

DNS name for this server:

dig +short drbd01.chainringcircus.org
192.168.1.191
dig +short drbd02.chainringcircus.org
192.168.1.192

The /etc/drbd.conf file was designed to allow a verbatim copy on both nodes of the cluster.

cat /etc/drbd.conf
#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd83/drbd.conf
#

global {
usage-count no;
}

common {
protocol C;
handlers {
pri-on-incon-degr "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";
#pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
#pri-on-incon-degr This handler is called if the node is primary, degraded and the local
#copy of the data is inconsistent. It broadcasts an error, sleeps for 60 seconds and then halts.
}

startup {
wfc-timeout 10; # Wait for connection timeout. The init script blocks the boot process
# until the DRBD resources are connected. We wait for 10 seconds.
degr-wfc-timeout 30; # Wait for connection timeout if this node was a degraded cluster.
}

disk {
on-io-error detach;
} # or panic, ...

net {
cram-hmac-alg "sha1";
shared-secret "CHANGEME"; # Don't forget to choose a secret for auth
max-buffers 20000; # Play with this setting to achieve highest possible performance
unplug-watermark 12000; # Play with this setting to achieve highest possible performance
max-epoch-size 20000; # Should be the same as max-buffers
}
syncer {
rate 100M;
}
}

resource sites {
device /dev/drbd0;
disk /dev/sdb;
meta-disk internal; # Internal means that the last part of the backing device is used to store the metadata.
on drbd01.chainringcircus.org { #on hostname as seen in uname -n and the DNS lookup.
address 192.168.1.191:7788;
}
on drbd02.chainringcircus.org {
address 192.168.1.192:7788;
}
}

Copy the configuration file:

scp /etc/drbd.conf root@drbd02.chainringcircus.org:/etc/

Tried to start DRBD but got an error:

service drbd start
Starting DRBD resources: [
sites
no suitable meta data found :(
Command '/sbin/drbdmeta 0 v08 /dev/sdb internal check-resize' terminated with exit code 255
drbdadm check-resize sites: exited with code 255
d(sites) 0: Failure: (119) No valid meta-data signature found.

==> Use 'drbdadm create-md res' to initialize meta-data area. <==

[sites] cmd /sbin/drbdsetup 0 disk /dev/sdb /dev/sdb internal --set-defaults --create-device --on-io-error=detach failed - continuing!

s(sites) n(sites) ]..........
/etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
m:res cs ro ds p mounted fstype
0:sites WFConnection Secondary/Unknown Diskless/DUnknown C

/etc/init.d/drbd stop
Stopping all DRBD resources: .

I did not initialize the meta data storage and this needs to be done before a DRBD resource can be brought online. The DRBD resource needs to be down or detached from its backing storage.

drbdadm create-md sites
md_offset 1073737728
al_offset 1073704960
bm_offset 1073672192

Found some data

==> This might destroy existing data! <==

Do you want to proceed?
[need to type 'yes' to confirm] yes

Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.

service drbd start
Starting DRBD resources: [
sites
Found valid meta data in the expected location, 1073737728 bytes into /dev/sdb.
d(sites) s(sites) n(sites) ]..........

Check the status:

cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
0: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:1048508

Make it primary:

drbdadm -- --overwrite-data-of-peer primary sites
cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:67584 nr:0 dw:0 dr:67584 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:980924
[>...................] sync'ed: 6.7% (980924/1048508)K delay_probe: 10
finish: 0:01:27 speed: 11,264 (11,264) K/sec
[root@localhost etc]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:1019904 nr:0 dw:0 dr:1019904 al:0 bm:62 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:28604
[==================>.] sync'ed: 97.7% (28604/1048508)K delay_probe: 195
finish: 0:00:02 speed: 11,132 (10,404) K/sec
[root@localhost etc]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
ns:1048508 nr:0 dw:0 dr:1048508 al:0 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
You have new mail in /var/spool/mail/root

Make a file system:

mkfs.ext3 /dev/drbd0
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
131072 inodes, 262127 blocks
13106 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376

Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 24 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

Testing the filesystem:

mount /dev/drbd0 /sites

mount
/dev/sda2 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda5 on /home type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
.host:/ on /mnt/hgfs type vmhgfs (rw,ttl=1)
none on /proc/fs/vmblock/mountPoint type vmblock (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/drbd0 on /sites type ext3 (rw)

touch /sites/test.txt

ls /sites
lost+found test.txt

umount /sites

drbdadm secondary sites

On the second server:

drbdadm primary sites

mount /dev/drbd0 /sites/

mount
/dev/sda2 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda5 on /home type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
.host:/ on /mnt/hgfs type vmhgfs (rw,ttl=1)
none on /proc/fs/vmblock/mountPoint type vmblock (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/drbd0 on /sites type ext3 (rw)

ls /sites
lost+found test.txt

Heartbeat R1-style

Heartbeat in R1 configuration uses 3 files that must be configured if you are using the heartbeat protocol.
/etc/ha.d/ha.cf
/etc/ha.d/haresources
/etc/ha.d/authkeys

cat /etc/ha.d/authkeys
auth 1 # A numerical identifier between 1 and 15 inclusive
# must be unique within the file.
1 sha1 CHANGEME # Methods can be md5 sha1 or crc.
# The password is just a string.

chmod 600 /etc/ha.d/authkeys

Before we take care of the ha.cf file we need to set up the ha_logd configuration file.

cp /usr/share/doc/heartbeat-2.1.3/logd.cf /etc/

And make changes to the logd.cf file accordingly. Be sure to copy /etc/logd.cf to both servers. Also note that I had to completely stop and then restart the heartbeat daemon for my logging changes to take affect.

cat /etc/logd.cf
# File to write debug messages to
# Default: /var/log/ha-debug
debugfile /var/log/ha-debug.log

#
#
# File to write other messages to
# Default: /var/log/ha-log
logfile /var/log/ha.log

#
#
# Facility to use for syslog()/logger
# Default: daemon
#logfacility daemon

# Entity to be shown at beginning of a message
# for logging daemon
# Default: "logd"
entity logd

# Do we register to apphbd
# Default: no
#useapphbd no

# There are two processes running for logging daemon
# 1. parent process which reads messages from all client channels
# and writes them to the child process
#
# 2. the child process which reads messages from the parent process through IPC
# and writes them to syslog/disk

# set the send queue length from the parent process to the child process
#
#sendqlen 256

# set the recv queue length in child process
#
#recvqlen 256

cat /etc/ha.d/ha.cf
# The recommendation is to use logd.
use_logd yes
# Default option is 0, values are 0-255 with 1-3 being the most useful.
debug 0
# Timing according to the FAQ at www.linux-ha.org/wiki/FAQ
# warntime should be at least 2 * keepalive
# warntime should be 1/2 to 1/4 deadtime
# The interval between heartbeat packets.
keepalive 1
# How quickly Heartbeat should issue a "late heartbeat" warning. Warntime is
# important for tuning deadtime.
warntime 5
# How long to decide a cluster node is dead. Too low will flasely declare
# a death and too high will hinder takeover during a failure.
# Can be specified as a floating point number followed by a untis-specifier.
# If units are omitted it defaults to seconds.
# deadtime 1
# deadtime 100ms 100 milliseconds
# deadtime 1000us 1000 microseconds
deadtime 10
# 694 is the default but can be changed if multiple clusters are in use.
udpport 694
# Which interfaces send UDP broadcast traffic, more than one can be specified.
bcast eth0
# auto_failback can be "on" "off" or "legacy"
auto_failback off
# Set the nodes in the cluster.
node in1.eamc.org
node in2.eamc.org
# Make sure this IP address is pingable from the bcast network above.
ping 192.168.1.1
respawn hacluster /usr/lib/heartbeat/ipfail

cat /etc/ha.d/haresources
drbd01 192.168.1.190 drbddisk::sites Filesystem::/dev/drbd0::/sites::ext3 httpd
# Explanation:
# Primary Server name --> virtual IP address to be used --> DRBD resource as configurd in /etc/drbd.conf
# --> where to mount the DRBD resource and the filesystem type --> resource to start/stop in case of failover

Cluster Management
To take over cluster management from a primary server:

/usr/lib/heartbeat/hb_takeover

Relinquishing cluster management to a secondary server:

/usr/lib/hearbeat/hb_standby
/etc/init.d/heartbeat stop

The order of operations as set by the init scripts:

ls -al /etc/rc3.d/ | egrep "hear|drb"
lrwxrwxrwx 1 root root 14 Apr 1 11:40 S70drbd -> ../init.d/drbd
lrwxrwxrwx 1 root root 19 Jun 1 08:58 S75heartbeat -> ../init.d/heartbeat

Notes for Xen users:

# cat /etc/modprobe.d/drbd.conf
options debd disable_sendpage=1

To allow live migration on Xen:

net {
allow-two-primaries;
}

Split-brain
Playing around this morning I got the cluster into split-brain.

Jun 1 10:46:53 in1 kernel: block drbd0: Split-Brain detected but unresolved, dropping connection!
Jun 1 10:46:53 in1 kernel: block drbd0: helper command: /sbin/drbdadm split-brain minor-0

Here is how to fix it.
Run this first on the secondary node.

drbadm -- --discard-my-data connect sites

Run this on the primary node.

drbadm connect sites

CCIE R&S Written Blueprint

jud — Tue, 12 Apr 2011 18:49:35 +0000

Below is the CCIE written blueprint. My plan is to slowly fill in my notes with the corresponding sections. The problem arises when Cisco is vague as to what is actually required. For instance I have been studying MPLS and LDP recently. While MPLS is listed on the blueprint, you need to understand LDP as well, however, it is not on the blueprint.

I have decided go ahead and add sections I feel are necessary but my added sections will be in italics to set them apart from the official outline. They will also break from the Cisco numbering convention. For instance below, section 4.11 Implement LDP does not follow the Cisco numbering theme of 4.10, 4.20, 4.30 but instead uses 4.11. Italics combined with the odd numbering signal my personal changes to the outline.

Source:
Cisco Learning Network

1.00 Implement Layer 2 Technologies
1.10 Implement Spanning Tree Protocol (STP)
   (a) 802.1d
   (b) 802.1w
   (c) 801.1s
   (d) Loop guard
   (e) Root guard
   (f) Bridge protocol data unit (BPDU) guard
   (g) Storm control
   (h) Unicast flooding
   (i) Port roles, failure propagation, and loop guard operation
1.20 Implement VLAN and VLAN Trunking Protocol (VTP)
1.30 Implement trunk and trunk protocols, EtherChannel, and load-balance
1.40 Implement Ethernet technologies
   (a) Speed and duplex
   (b) Ethernet, Fast Ethernet, and Gigabit Ethernet
   (c) PPP over Ethernet (PPPoE)
1.50 Implement Switched Port Analyzer (SPAN), Remote Switched Port Analyzer (RSPAN),
and flow control
1.60 Implement Frame Relay
   (a) Local Management Interface (LMI)
   (b) Traffic shaping
   (c) Full mesh
   (d) Hub and spoke
   (e) Discard eligible (DE)
1.70 Implement High-Level Data Link Control (HDLC) and PPP

2.00 Implement IPv4
2.10 Implement IP version 4 (IPv4) addressing, subnetting,
and variable-length subnet masking (VLSM)
2.20 Implement IPv4 tunneling and Generic Routing Encapsulation (GRE)
2.30 Implement IPv4 RIP version 2 (RIPv2)
2.40 Implement IPv4 Open Shortest Path First (OSPF)
   (a) Standard OSPF areas
   (b) Stub area
   (c) Totally stubby area
   (d) Not-so-stubby-area (NSSA)
   (e) Totally NSSA
   (f) Link-state advertisement (LSA) types
   (g) Adjacency on a point-to-point and on a multi-access network
   (h) OSPF graceful restart
2.50 Implement IPv4 Enhanced Interior Gateway Routing Protocol (EIGRP)
   (a) Best path
   (b) Loop-free paths
   (c) EIGRP operations when alternate loop-free paths are available,
   and when they are not available
   (d) EIGRP queries
   (e) Manual summarization and autosummarization
   (f) EIGRP stubs
2.60 Implement IPv4 Border Gateway Protocol (BGP)
   (a) Next hop
   (b) Peering
   (c) Internal BorderGateway Protocol (IBGP) and
   External Border Gateway Protocol (EBGP)
2.70 Implement policy routing
2.80 Implement Performance Routing (PfR) and Cisco Optimized Edge Routing (OER)
2.90 Implement filtering, route redistribution, summarization,
synchronization, attributes, and other advanced features

3.00 Implement IPv6
3.10 Implement IP version 6 (IPv6) addressing and different addressing types
3.20 Implement IPv6 neighbor discovery
3.30 Implement basic IPv6 functionality protocols
3.40 Implement tunneling techniques
3.50 Implement OSPF version 3 (OSPFv3)
3.60 Implement EIGRP version 6 (EIGRPv6)
3.70 Implement filtering and route redistribution

4.00 Implement MPLS Layer 3 VPNs
4.10 Implement Multiprotocol Label Switching (MPLS)
4.11 Implement LDP
4.20 Implement Layer 3 virtual private networks (VPNs) on provider edge (PE),
provider (P), and customer edge (CE) routers
4.30 Implement virtual routing and forwarding (VRF) and Multi-VRF Customer Edge (VRF-Lite)

5.00 Implement IP Multicast
5.10 Implement Protocol Independent Multicast (PIM) sparse mode
5.20 Implement Multicast Source Discovery Protocol (MSDP)
5.30 Implement interdomain multicast routing
5.40 Implement PIM Auto-Rendezvous Point (Auto-RP),
unicast rendezvous point (RP), and bootstrap router (BSR)
5.50 Implementmulticast tools, features, and source-specific multicast
5.60 Implement IPv6 multicast, PIM, and related multicast protocols,
such as Multicast Listener Discovery (MLD)

6.00 Implement Network Security
6.01 Implement access lists
6.02 Implement Zone Based Firewall
6.03 Implement Unicast Reverse Path Forwarding (uRPF)
6.04 Implement IP Source Guard
6.05 Implement authentication, authorization, and accounting (AAA)
(configuring the AAA server is not required, only the client-side (IOS) is configured)
6.06 Implement Control Plane Policing (CoPP)
6.07 Implement Cisco IOS Firewall
6.08 Implement Cisco IOS Intrusion Prevention System (IPS)
6.09 Implement Secure Shell (SSH)
6.10 Implement 802.1x
6.11 Implement NAT
6.12 Implement routing protocol authentication
6.13 Implement device access control
6.14 Implement security features

7.00 Implement Network Services
7.10 Implement Hot Standby Router Protocol (HSRP)
7.20 Implement Gateway Load Balancing Protocol (GLBP)
7.30 Implement Virtual Router Redundancy Protocol (VRRP)
7.40 Implement Network Time Protocol (NTP)
7.50 Implement DHCP
7.60 Implement Web Cache Communication Protocol (WCCP)

8.00 Implement Quality of Service (QoS)
8.10 Implement Modular QoS CLI (MQC)
   (a) Network-Based Application Recognition (NBAR)
   (b) Class-based weighted fair queuing (CBWFQ), modified deficit round robin (MDRR),
   and low latency queuing (LLQ)
   (c) Classification
   (d) Policing
   (e) Shaping
   (f) Marking
   (g) Weighted random early detection (WRED) and random early detection (RED)
   (h) Compression
8.20 Implement Layer 2 QoS: weighted round robin (WRR), shaped round robin (SRR),
   and policies
8.30 Implement link fragmentation and interleaving (LFI) for Frame Relay
8.40 Implement generic traffic shaping
8.50 Implement Resource Reservation Protocol (RSVP)
8.60 Implement Cisco AutoQoS

9.00 Troubleshoot a Network
9.10 Troubleshoot complex Layer 2 network issues
9.20 Troubleshoot complex Layer 3 network issues
9.30 Troubleshoot a network in response to application problems
9.40 Troubleshoot network services
9.50 Troubleshoot network security

10.00 Optimize the Network
10.01 Implement syslog and local logging
10.02 Implement IP Service Level Agreement SLA
10.03 Implement NetFlow
10.04 Implement SPAN, RSPAN, and router IP traffic export (RITE)
10.05 Implement Simple Network Management Protocol (SNMP)
10.06 Implement Cisco IOS Embedded Event Manager (EEM)
10.07 Implement Remote Monitoring (RMON)
10.08 Implement FTP
10.09 Implement TFTP
10.10 Implement TFTP server on router
10.11 Implement Secure Copy Protocol (SCP)
10.12 Implement HTTP and HTTPS
10.13 Implement Telnet

11.00 Evaluate proposed changes to a Network
11.01 Evaluate interoperability of proposed technologies against deployed technologies
   (a) Changes to routing protocol parameters
   (b) Migrate parts of a network to IPv6
   (c) Routing Protocol migration
   (d) Adding multicast support
   (e) Migrate spanning tree protocol
   (f) Evaluate impact of new traffic on existing QoS design
11.02 Determine operational impact of proposed changes to an existing network
   (a) Downtime of network or portions of network
   (b) Performance degradation
   (c) Introducing security breaches
11.03 Suggest Alternative solutions when incompatible changes
are proposed to an existing network
   (a) Hardware/Software upgrades
   (b) Topology shifts
   (c) Reconfigurations

Data Loss Prevention

jud — Mon, 07 Mar 2011 13:48:45 +0000

Every once in a while I get to write a neat piece of code that I can share. This is one of those times. I realize it is not large and by PerlMonk standards not very elegant. The problem therein lies with maintainability over the next few years. Regardless I like what I wrote and would like to share.

At the Circus we had a pretty good idea that we had some data leakage. Nothing like people taking off with everything needed to get home loans and rip off customers, just people not thinking about what they send through email. We didn’t know the extent of the problem or even if we had one. We just weren’t sure. Our C-level executives didn’t believe that employees would be so careless with customer data. We decided to find out.

I must say that the results were actually quite positive. We had a couple of people email work related data home so they could work at home over the weekend and a few emails regarding employment, but they were originated by the prospective employee.

Regardless, in order for us to find out I wrote a few scripts that hook into our email system. One that I am particularly proud of recurses through a directory of email messages and attachments scanning each file for relevant data.

Please note that by the time these scripts touch the data it has been scrubbed by the antivirus and other checks we have in place. I am only looking for keywords or regular expressions that would indicate customer related data loss.

Let me explain the directory structure. Under the email system is the directory /var/spool/filter that contains every email message that has been sent in the last 30 minutes. There is a cleanup process that erases all the files in that directory and that is actually where I wrote the hook, in the cleanup process. Here is a sample listing of the directory.

#ls -1 /var/spool/filter/
msg-1299451572-29517-0
msg-1299451626-29523-0
msg-1299451695-29528-0
msg-1299452467-29565-0
msg-1299452491-29570-0
msg-1299453007-29593-0
msg-1299453086-29599-0

As you can see, each email header ends with a .hed extension and the message is in .txt format. The ETP.doc file is an attachment.

#ls -1 /var/spool/filter/msg-1299451626-29523-0/
ETP.doc
msg-29523-1.txt
msg-29523-2.dat.hed

The subroutine I am most pleased with is the one that recurses through the directory structure. The slurp command returns a hash and if it is a subdirectory then it is a hash as well. I look for it with the following line of code.

if (ref $structure->{$key} eq 'HASH')

That is how I find subdirectories to push onto the stack of recursive calls. As it traverses each directory it just looks at each file extension and makes a determination as to what to do with it.

I realize most system administrators are asking why I didn’t use the file command to make sure the script was acting appropriately for each file type but that does not work with the new Microsoft document types.

# file Test-Excel.xlsx
Test-Excel.xlsx: Zip archive data, at least v2.0 to extract

I thought it was a fun project and I enjoyed writing what I felt was an interesting piece of code.

#!/usr/bin/perl
# 2011-01-12 Jud Bishop
# This script goes looking for customer data being sent out through email and
# flags it for further review.
use strict;
use warnings;
use File::Find;
use File::Basename;
use File::Copy::Recursive qw(fcopy dircopy rcopy);
use File::Slurp::Tree;

#my $dir = "/home/jud/TestMessages";
#my $log = "/home/jud/TestMessages/violation";
#my $auditdir = "/home/jud/TestMessages/Trash/";
my $dir = "/var/spool/filter";
my $log = "/var/log/hipaa/violation";
my $auditdir = "/opt/smtpaudit/";
my $debug = 0;

###################
# MAIN
###################
my %tree;
my $tree = slurp_tree($dir);

open (LOG, '>>', $log) or die $!;

traverse_structure($dir, $tree);

close LOG or die $!;

##########
# This does the heavy lifting of the whole program. It recursively
# iterates through the directory structure and works on a file accordingly.
# Each directory is a hash key.
##########
sub traverse_structure {
if($debug){print "##traverse_structure\n";}
my ($base, $structure) = @_;
my $path;
my @violation;
my $secure;
foreach my $key ( keys %$structure) {
$path = $base . "/" . $key;
$secure = 0;
## If it's a HASH then it's a directory.
if (ref $structure->{$key} eq 'HASH'){
if($debug){print "key: $key\n"};
traverse_structure( $path, $structure->{$key} );
} else {
if($debug){print "file : $key\n"};
if($debug){print "base : $base\n"};
if($debug){print "path : $path\n"};
if($debug){print "secure: $secure\n"};
if($debug){print "violation: $#violation\n"};

## If the file is not being used...
if ($path =~ m/doc$/){
parse_doc($path, \@violation);
} elsif ($path =~ m/xlsx$|xls$/) {
parse_excel($path, \@violation);
} elsif ($path =~ m/txt$/) {
parse_message($path, \@violation);
} elsif ($path =~ m/pdf$/) {
parse_pdf($path, \@violation);
} elsif ($path =~ m/hed$/) {
parse_head($path, \@violation, \$secure);
}
}
}
# If it is a secure email than it is encrypted on
# the fly and not a violation.
if ( ($secure == 0) && ($#violation > 3) ){
push (@violation, "EMAIL: " . $base);
log_it(@violation);
copy_dir($base);
}
}

# For later review.
sub copy_dir {
my $path = shift;
if($debug){print "##copy_dir $path\n";}
my $file = fileparse($path);

if ($file =~ m/^msg/){
my $basename = basename($path);
my $newpath = $auditdir . $basename;

if($debug){print "dircopy $path $newpath\n";}
dircopy($path,$newpath);
}
}

# Log file that is easy to ready because an employee goes through
# this file and decides if it is a REAL violation.
sub log_it {
my @text = @_;
my $line;
if($debug){print "##log_it\n";}
print LOG "---------------------------------------------\n";
foreach $line (@text) {
print LOG "$line\n";
}
print LOG "---------------------------------------------\n";
}

sub parse_head {
my ($file, $violation_ref, $secure_ref) = @_;
my @body;
my $line;
if($debug){print "##parse_head $file\n";}

open(FILE,$file) || return 0;
@body = ;
close(FILE);

foreach $line (@body) {
if ($line =~ m/^From/){
push (@$violation_ref, $line);
} elsif ($line =~ m/^To/) {
push (@$violation_ref, $line);
} elsif ($line =~ m/^Subject/) {
push (@$violation_ref, $line);
if ($line =~ m/^secure/i )
{
$$secure_ref = 1;
}
}
}
}

sub parse_pdf {
my ($file, $violation_ref) = @_;
my @body;
my $new_file = $file . ".txt";
my $CMD;

if($debug){print "##parse_doc $dir $file\n";}
$CMD = "/usr/bin/pdftotext \"" . $file . "\" > \"" . $new_file . "\"";
if($debug){print "CMD: $CMD\n";}
system($CMD);
parse_text ($new_file, $violation_ref);
}

sub parse_doc {
my ($file, $violation_ref) = @_;
my @body;
my $new_file = $file . ".txt";
my $CMD;

if($debug){print "##parse_doc $dir $file\n";}
$CMD = "/usr/bin/antiword -st \"" . $file . "\" > \"" . $new_file . "\"";
if($debug){print "CMD: $CMD\n";}
system($CMD);
parse_text ($new_file, $violation_ref);
}

sub parse_excel {
my ($file, $violation_ref) = @_;
my @body;
my $new_file = $file . ".txt";
my $CMD;

if($debug){print "##parse_excel $file\n";}
$CMD = "/usr/local/bin/antiexcel \"" . $file . "\" > \"" . $new_file . "\"";
if($debug){print "CMD: $CMD\n";}
system($CMD);
parse_text ($new_file, $violation_ref);
}

sub parse_text {
my ($file, $violation_ref) = @_;
my @body;
if($debug){print "##parse_text $file\n";}

open(FILE,$file) || return 0;
@body = ;
close(FILE);

compare_text(\@body, $violation_ref);
}

sub parse_message {
my ($file, $violation_ref) = @_;
my @body;
if($debug){print "##parse_text $file\n";}

open(FILE,$file) || return 0;
@body = ;
close(FILE);

compare_text(\@body, $violation_ref);
}

# All of the earlier subroutines call this one.
# It takes the text and looks for keywords.
sub compare_text {
my ($text_ref, $violation_ref) = @_;
my @difference;
my @text_array;
my @elements;
my %count;
my %rules;
my $element;
if($debug){print "##compare_text\n";}

foreach $element (@$text_ref){
@elements = split(' ', $element);
push (@text_array, @elements);
}

# The parser was already created above.
my @rule = ("DOB", "D.O.B.", "d.o.b.", "dob", "death:", "release", "admit", "admission", "Age:", "SSN", "Social", "Security", "Account", "Acct", "claimant", "MRI", "myelogram", "credit", "card");

# Me being lazy.
foreach $element (@rule)
{
$rules{$element} = 1;
}

foreach $element (@text_array)
{
if (exists $rules{$element})
{
if($debug){print "$element\n";}
$element = "VIOLATION: " . $element;
push (@$violation_ref, $element);
}
# Social Security Number
elsif($element =~ /\d{3}-?\d{2}-?\d{4}/)
{
if($debug){print "$element\n";}
$element = "VIOLATION: " . $element;
push (@$violation_ref, $element);
}
# Credit Card Number or MRN
elsif($element =~ /\d{4}-?\d{4}-?\d{4}-?\d{4}/)
{
if($debug){print "$element\n";}
$element = "VIOLATION: " . $element;
push (@$violation_ref, $element);
}

}
}