Chainring Circus

I have been putting off this post because I didn’t really know what to say. So much has changed in the last six months I didn’t know how best to communicate what was going on in my life. Please understand that I am leaving out quite a bit of this story but I believe it needs to be told so that others will understand when they stumble across my blog.

Let’s start with some background. I have been working for the same employer for the last 8 years. It is a regional medical center and a great place to work. Before Fortune magazine stopped allowing non-profits to be on the “100 Best Places to Work” my employer made the list twice.

I started working at the hospital as a Programmer Analyst in early 2003 writing code that allowed hospital systems to communicate and was also the UNIX/Linux administrator. During this time I managed DNS/DHCP, most of the printing and portions of our email infrastructure on *NIX while also writing mostly TCL and PERL code.

When a Senior Network Engineer slot opened I applied and was promoted in early 2006. I earned my CCNA and began to learn quite a bit about networking. Shortly thereafter I decided I was ready to leave the hospital in order to learn more, but my wife loves working here, knows and trusts many of the doctors and made me promise that we would have our children at this hospital.

As a result of that promise to my wife I decided to work on my CCNP and started this blog, figuring both would help me get my next job. During that time I was selected for “50 for the Future” a two year management training program at the hospital and I finished my CCNP. I also became dedicated to my craft. As I learned more and more about networking, I began to understand how much I did not know and it became a joy to learn something new everyday. Not all of the lessons were fun at the time but I was learning so much I loved coming to work.

The next logical step was starting on my CCIE. It sounded like a great challenge and with a family now depending upon me, it was the next logical step for job security. It surely would allow me to get my next job and my promise to my wife was still not fulfilled. We had suffered through two miscarriages to get our first child, and our second child was born in April 2011.

With the birth of our second child my promise to my wife had been fulfilled and I was ready to leave. Literally the day after my second child was born I started applying for jobs. I was offered an interesting position in South Carolina working with a team that fit my goals. Some had their CCIEs and others were working toward their digits. I would fit right in and I knew it was the place for me, so I turned in my resignation. The wrinkle being that the hospital requests a 30 day notice for professional employees.

I had already spoken with my Director about leaving and he was gracious. I had spent $5,000 on my CCIE lab and the hospital had spent another $5,000, but my boss gave me the whole rack. Everything. I cannot describe how much that meant to me, I am truly grateful.

That day my Vice President asked me what it would take for me to stay and I told him I was chasing a dream. While I appreciated the offer and I knew the hospital was a great place to work, it was a goal I had set for myself and I felt I should leave in order to get the experience I needed.

Two weeks later I had both my AVP and VP stop by my desk within ten minutes of each other. My last day was drawing near and they wanted to discuss what it would take for me to stay. I declined a second time. I had made the decision to try to earn my CCIE and I did not want to let my family down. I believed that South Carolina offered me the best chance to achieve my goal.

My last day at the hospital came and went, but it was a tumultuous day. My AVP literally gave me a hug as I walked out the door for the last time and I cried in her arms.

I had a week off between jobs and I had lunch with friends to say goodbye, finished setting up the lab at home and packed. I made arrangements for long term housing in South Carolina and prepared to leave my family while we tried to sell our home.

Thursday evening of my week off I got a call on my cell phone. It was my VP and AVP calling to chat. The first question they asked was whether I was still in town or had I already left for South Carolina. Then they asked me to stay one last time. They had met with our CEO and Executive Vice President and wanted me to stay. I was floored. I had worked for nearly three years getting ready to leave. I had started down a road and I didn’t want to change direction.

But I did. My wife told me that I will never again be offered the chance to work at a company which knows neither what my job title will be nor what I will be doing, just that they want me to work for them. She was probably right.

I was offered the chance to make a difference working for a company that has been good to me and my family. It is a different challenge than what I had been working toward but an interesting one all the same. I am slowly settling into my new job as Technical Services Manager. Some of the changes I have made are for the better, some of the changes have not lasted and others have yet to be implemented, but I am still learning every day.

And so my goals are slowly evolving. It is hard to give up a goal that I have sacrificed so much to earn, without completing it. In the short term I will begin working toward my CCIP in the new year. Currently I do not have the time to complete the CCIE but I believe working toward my CCIP will keep my mind in networking and it will cover topics that I believe are needed if I decide to again try for my CCIE.

Thank you for reading. I plan to post more regularly in the coming year.

I got a new Mac Pro workstation at work and re-wrote some scripts to work on it. This morning I couldn’t find the script under the new file lay out, it was in /Applications so I decided I had better document the script so I don’t have to rewrite it if I can’t find it.

It uses the same script, tle, that I wrote a while ago, it just fires up iTerm instead of Gnome Terminal.

The other night we were doing a hardware upgrade on a cluster and testing. We were working with the command clusvcadm to relocate a service from one host in the cluster to another but the originating server kept getting power fenced. We assumed it was the command switches we were running so I went straight to the man page, my coworker went straight to google. Just for reference there is a 10 year difference in our ages, I grew up with man pages and it is a pet peeve of mine when either no man page exists or it is a terrible placeholder. I digress, through his search he came upon a webified man page while I was reading the man page. When I needled him about it his answer was, “But mine is nicely formatted and I can search the web page.” I was surprised, I can search the man page too, right in the pager and can even change man page viewers by changing the PAGER variable.

Three weeks ago I needed to bring up an https server on Ubuntu and spent 45 minutes googling around reading old, outdated or completely wrong howtos before finally going to help.ubuntu.com and 20 minutes later it was done.

The same thing happened over the past couple of weeks working with Xen and VirtualBox. I’ve toiled away looking at poorly written documentation and even mentioned it in my last Red Hat class. The instructor worked for Red Hat and took umbrage with my statement. He was amazed that I did not think Red Hat had great documentation, I was even more shocked that he considered their documentation more than rudimentary. Have a look for your self at the Red Hat documentation.

Just this week I was helping a friend who is the server and network administrator for a small school system configure the proper etherchannel load balancing for a server and he was frustrated at the Cisco documentation. I was astonished. It seemed that he was overwhelmed. He was stuck googling around trying find the “right” documentation rather than learning the layout of the Cisco documentation website.

The point of this post is that lately it seems I waste more time trying to find good information through searching on the web than trying to find the best source of information.

Running across the Hacking Cisco blog made me remember a similar site, CCIE18473.net. I actually spent about 30 minutes looking for that site and it was Tyson Scott from IPE that me helped find it.

I have added this site to my blogroll even though it is not a blog.

I’ve been keeping track of some the mnemonics that I have come across or have figured out for myself. For instance in general, in layer 2 elections the lower priority usually wins, however, in layer 3 elections the higher priority usually wins.

Layer 2

LACP System Priority
2-bytes priority values followed by a 6-byte MAC address. Lowest system priority makes decisions about the etherchannel setup.

LACP Port Priority
LACP port priority is a 2-byte priority followed by a 2-byte port number. Lowest port priority is used to decide which ports are put in standby mode when not all ports can be put in etherchannel.

STP
Root bridge election, lowest bridge ID wins. Bridge ID consists of:
–2-byte bridge priority from 0-65,535 with a default of 32,768.
–6-byte MAC address
If the bridge priorities are equal, lowest MAC wins.

Root port, lowest root path cost.

Designated port — lowest root path cost or if equal use tie breakers:
1. Lowest root bridge ID
2. Lowest root path cost to root bridge
3. Lowest sender bridge ID
4. Lowest sender port ID

Frame Relay
DCE requires the clock rate, DCE, DTE, clock rate starts with a c and DCE is the one with a c in it.

Layer 3

HSRP
Active router election is based upon priority, highest priority wins. Default priority of 100 and a range of 0-255. Highest IP address on HSRP interface breaks ties.
Standby router is the second highest priority.

VRRP
Election of master is the router with the gateway IP address or if not a “real” IP address, the router with the highest priority. Priority ranges from 1 to 254 with 254 being highest, 100 is the default.

GLBP
Active virtual gateway (AVG) is elected by the highest priority value, tie breaker is the highest IP address in the group. Router priority is 1-255 with 255 being highest, 100 is the default.

OSPF DR/BDR Election
1. Highest priority wins.
2. Highest router ID breaks ties.
Priority range is 0-255 with 255 being highest, 1 is the default and 0 means the router will not participate in the election.

OSPF RID
1. router-id command wins.
2. If no router-id is set, the highest loopback address wins, even if it is not advertised and it is not advertised by default.
3. Highest physical address wins.

OSPF summary-address command or the range command.
The summary-address command is used on an ASBR and has an “S” in it, whereas the area range command is used on an ABR and does not have an “S” in it. Both commands are used to summarize routes.

OSPF ExStart
During ExStart of the OSPF packet exchange the neighbor with the highest RID will become the master and sets the DD sequence number.

DVMRP
An exception to the rule of Layer 2 lower takes priority and Layer 3 higher takes the priority. If two routers are the same distance from the source, the router with the numerically lower IP address becomes the designated forwarder for the network.

BGP best path mnemonic
We love oranges as oranges mean pure refreshment.

We — Weight (highest)
Love — LOCAL_PREF (highest)
Oranges — Originate (local)
AS — AS_PATH (shortest)
Oranges — Origin Code (IGP > EGP > Incomplete)
Mean — Med (lowest)
Pure — Paths (External > Internal)
Refreshment — RID (lowest)

Redistribution
RIP and any other protocol that has the letters R-I-P in it requires a seed metric, RIP, IGRP, EIGRP.

I couldn’t resist using all of those acronyms.
EIGRP – Enhanced Interior Gateway Routing Protocol
MPLS – Multiprotocol Label Switching
VPN – Virtual Private Networking
PE-CE – Provider Equipment – Customer Equipment
SOO – Site Of Origin

MPLS SOO
MPLS Fundamentals pp. 220-226

BGP->EIGRP and EIGRP->BGP

Advertisement of the SOO BGP extended community attribute is used to identify routes that have originated from a site so that they are not re-advertised back into the same site. Each SOO uniquely identifies the site and allows for the routes to be filtered. SOO filtering is configured at the interface level. It is commonly used when a site contains both VPN and back door links.

From the Cisco document:
The configuration of the SOO extended community allows MPLS VPN traffic to be filtered on a per-site basis. The SoO extended community is configured in an inbound BGP route map on the PE router and is applied to the interface with the ip vrf sitemap command. The SOO extended community can be applied to all exit points at the customer site for more specific filtering but must be configured on all interfaces of PE routers that provide VPN services to CE routers.

Someone on one of the lists posted about the blog Hacking Cisco so I went over to check it out. Wow. Jarek Rek is the author and he must be putting a ton of time into his studies.

Inspiring.

I have added his blog to the blogroll on the side.

I spent a considerable amount of time over the last couple of days working with DRBD and Heartbeat.

Below are the links I used to get things running:
http://wiki.centos.org/HowTos/Ha-Drbd
http://www.howtoforge.com/vm_replication_failover_vmware_debian_etch_p3
http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Clusters_from_Scratch/s-intro-pacemaker.html
http://www.drbd.org/users-guide/s-heartbeat-r1.html
http://www.drbd.org/users-guide/s-heartbeat-config.html
http://www.drbd.org/users-guide/s-heartbeat-crm.html

Part of my problem was not understanding the difference between R1 and DRM style clusters and their accompanying daemons; heartbeat, pacemaker and the different protocol versions. Pacemaker is a more advanced cluster resource manager that can work with both Corosync and Heartbeat. Heartbeat uses an older protocol whereas pacemaker uses OpenAIS to be compatible with RedHat cluster services.

Regardless here are my notes for configuration, and just for completeness my notes are a mix of doing this first on VMWare and then on a Xen cluster so any inconsistencies are a result of doing this multiple times in different environments. Regardless the errors are mine and I would recommend reading the documentation linked above.

The basics behind the setup is that DRBD replicates data between two servers. DRBD is the network block device that mirrors the data. The heartbeat daemon keeps track of the shared IP, the daemons that are in HA and runs the init scripts appropriately.

DRBD Initialization

Format the disk:

Make sure that the names names are consistent throughout all of these configuration files. This may mean ensuring they are correct in DNS and /etc/hosts.

Locally configure name for this server:

DNS name for this server:

The /etc/drbd.conf file was designed to allow a verbatim copy on both nodes of the cluster.

Copy the configuration file:

Tried to start DRBD but got an error:

I did not initialize the meta data storage and this needs to be done before a DRBD resource can be brought online. The DRBD resource needs to be down or detached from its backing storage.

Check the status:

Make it primary:

Make a file system:

Testing the filesystem:

On the second server:

Heartbeat R1-style

Heartbeat in R1 configuration uses 3 files that must be configured if you are using the heartbeat protocol.
/etc/ha.d/ha.cf
/etc/ha.d/haresources
/etc/ha.d/authkeys

Before we take care of the ha.cf file we need to set up the ha_logd configuration file.

And make changes to the logd.cf file accordingly. Be sure to copy /etc/logd.cf to both servers. Also note that I had to completely stop and then restart the heartbeat daemon for my logging changes to take affect.

Cluster Management
To take over cluster management from a primary server:

Relinquishing cluster management to a secondary server:

The order of operations as set by the init scripts:

Notes for Xen users:

To allow live migration on Xen:

Split-brain
Playing around this morning I got the cluster into split-brain.

Here is how to fix it.
Run this first on the secondary node.

Run this on the primary node.

Below is the CCIE written blueprint. My plan is to slowly fill in my notes with the corresponding sections. The problem arises when Cisco is vague as to what is actually required. For instance I have been studying MPLS and LDP recently. While MPLS is listed on the blueprint, you need to understand LDP as well, however, it is not on the blueprint.

I have decided go ahead and add sections I feel are necessary but my added sections will be in italics to set them apart from the official outline. They will also break from the Cisco numbering convention. For instance below, section 4.11 Implement LDP does not follow the Cisco numbering theme of 4.10, 4.20, 4.30 but instead uses 4.11. Italics combined with the odd numbering signal my personal changes to the outline.

Source:
Cisco Learning Network

1.00 Implement Layer 2 Technologies
         1.10 Implement Spanning Tree Protocol (STP)                   
                  (a) 802.1d                   
                  (b) 802.1w                   
                  (c) 801.1s                   
                  (d) Loop guard                   
                  (e) Root guard                   
                  (f) Bridge protocol data unit (BPDU) guard
                  (g) Storm control                   
                  (h) Unicast flooding                   
                  (i) Port roles, failure propagation, and loop guard operation
         1.20 Implement VLAN and VLAN Trunking Protocol (VTP)
         1.30 Implement trunk and trunk protocols, EtherChannel, and load-balance
         1.40 Implement Ethernet technologies
                  (a) Speed and duplex
                  (b) Ethernet, Fast Ethernet, and Gigabit Ethernet
                  (c) PPP over Ethernet (PPPoE)
         1.50 Implement Switched Port Analyzer (SPAN), Remote Switched Port Analyzer (RSPAN),
          and flow control
         1.60 Implement Frame Relay
                  (a) Local Management Interface (LMI)
                  (b) Traffic shaping
                  (c) Full mesh
                  (d) Hub and spoke
                  (e) Discard eligible (DE)
         1.70 Implement High-Level Data Link Control (HDLC) and PPP

2.00 Implement IPv4
         2.10 Implement IP version 4 (IPv4) addressing, subnetting,
                    and variable-length subnet masking (VLSM)
         2.20 Implement IPv4 tunneling and Generic Routing Encapsulation (GRE)
         2.30 Implement IPv4 RIP version 2 (RIPv2)
         2.40 Implement IPv4 Open Shortest Path First (OSPF)
                  (a) Standard OSPF areas
                  (b) Stub area
                  (c) Totally stubby area
                  (d) Not-so-stubby-area (NSSA)
                  (e) Totally NSSA
                  (f) Link-state advertisement (LSA) types
                  (g) Adjacency on a point-to-point and on a multi-access network
                  (h) OSPF graceful restart
         2.50 Implement IPv4 Enhanced Interior Gateway Routing Protocol (EIGRP)
                  (a) Best path
                  (b) Loop-free paths
                  (c) EIGRP operations when alternate loop-free paths are available,
                  and when they are not available
                  (d) EIGRP queries
                  (e) Manual summarization and autosummarization
                  (f) EIGRP stubs
         2.60 Implement IPv4 Border Gateway Protocol (BGP)
                  (a) Next hop
                  (b) Peering
                  (c) Internal BorderGateway Protocol (IBGP) and
                  External Border Gateway Protocol (EBGP)
         2.70 Implement policy routing
         2.80 Implement Performance Routing (PfR) and Cisco Optimized Edge Routing (OER)
         2.90 Implement filtering, route redistribution, summarization,
         synchronization, attributes, and other advanced features

3.00 Implement IPv6
         3.10 Implement IP version 6 (IPv6) addressing and different addressing types
         3.20 Implement IPv6 neighbor discovery
         3.30 Implement basic IPv6 functionality protocols
         3.40 Implement tunneling techniques
         3.50 Implement OSPF version 3 (OSPFv3)
         3.60 Implement EIGRP version 6 (EIGRPv6)
         3.70 Implement filtering and route redistribution

4.00 Implement MPLS Layer 3 VPNs
         4.10 Implement Multiprotocol Label Switching (MPLS)
         4.11 Implement LDP
         4.20 Implement Layer 3 virtual private networks (VPNs) on provider edge (PE),
         provider (P), and customer edge (CE) routers
         4.30 Implement virtual routing and forwarding (VRF) and Multi-VRF Customer Edge (VRF-Lite)

5.00 Implement IP Multicast
         5.10 Implement Protocol Independent Multicast (PIM) sparse mode
         5.20 Implement Multicast Source Discovery Protocol (MSDP)
         5.30 Implement interdomain multicast routing
         5.40 Implement PIM Auto-Rendezvous Point (Auto-RP),
         unicast rendezvous point (RP), and bootstrap router (BSR)
         5.50 Implementmulticast tools, features, and source-specific multicast
         5.60 Implement IPv6 multicast, PIM, and related multicast protocols,
         such as Multicast Listener Discovery (MLD)

6.00 Implement Network Security
         6.01 Implement access lists
         6.02 Implement Zone Based Firewall
         6.03 Implement Unicast Reverse Path Forwarding (uRPF)
         6.04 Implement IP Source Guard
         6.05 Implement authentication, authorization, and accounting (AAA)
         (configuring the AAA server is not required, only the client-side (IOS) is configured)
         6.06 Implement Control Plane Policing (CoPP)
         6.07 Implement Cisco IOS Firewall
         6.08 Implement Cisco IOS Intrusion Prevention System (IPS)
         6.09 Implement Secure Shell (SSH)
         6.10 Implement 802.1x
         6.11 Implement NAT
         6.12 Implement routing protocol authentication
         6.13 Implement device access control
         6.14 Implement security features

7.00 Implement Network Services
         7.10 Implement Hot Standby Router Protocol (HSRP)
         7.20 Implement Gateway Load Balancing Protocol (GLBP)
         7.30 Implement Virtual Router Redundancy Protocol (VRRP)
         7.40 Implement Network Time Protocol (NTP)
         7.50 Implement DHCP
         7.60 Implement Web Cache Communication Protocol (WCCP)

8.00 Implement Quality of Service (QoS)
         8.10 Implement Modular QoS CLI (MQC)
                  (a) Network-Based Application Recognition (NBAR)
                  (b) Class-based weighted fair queuing (CBWFQ), modified deficit round robin (MDRR),
                  and low latency queuing (LLQ)
                  (c) Classification
                  (d) Policing
                  (e) Shaping
                  (f) Marking
                  (g) Weighted random early detection (WRED) and random early detection (RED)
                  (h) Compression
         8.20 Implement Layer 2 QoS: weighted round robin (WRR), shaped round robin (SRR),
                  and policies
         8.30 Implement link fragmentation and interleaving (LFI) for Frame Relay
         8.40 Implement generic traffic shaping
         8.50 Implement Resource Reservation Protocol (RSVP)
         8.60 Implement Cisco AutoQoS

9.00 Troubleshoot a Network
         9.10 Troubleshoot complex Layer 2 network issues
         9.20 Troubleshoot complex Layer 3 network issues
         9.30 Troubleshoot a network in response to application problems
         9.40 Troubleshoot network services
         9.50 Troubleshoot network security

10.00 Optimize the Network
         10.01 Implement syslog and local logging
         10.02 Implement IP Service Level Agreement SLA
         10.03 Implement NetFlow
         10.04 Implement SPAN, RSPAN, and router IP traffic export (RITE)
         10.05 Implement Simple Network Management Protocol (SNMP)
         10.06 Implement Cisco IOS Embedded Event Manager (EEM)
         10.07 Implement Remote Monitoring (RMON)
         10.08 Implement FTP
         10.09 Implement TFTP
         10.10 Implement TFTP server on router
         10.11 Implement Secure Copy Protocol (SCP)
         10.12 Implement HTTP and HTTPS
         10.13 Implement Telnet

11.00 Evaluate proposed changes to a Network
         11.01 Evaluate interoperability of proposed technologies against deployed technologies
                  (a) Changes to routing protocol parameters
                  (b) Migrate parts of a network to IPv6
                  (c) Routing Protocol migration
                  (d) Adding multicast support
                  (e) Migrate spanning tree protocol
                  (f) Evaluate impact of new traffic on existing QoS design
         11.02 Determine operational impact of proposed changes to an existing network
                  (a) Downtime of network or portions of network
                  (b) Performance degradation
                  (c) Introducing security breaches
         11.03 Suggest Alternative solutions when incompatible changes
         are proposed to an existing network
                  (a) Hardware/Software upgrades
                  (b) Topology shifts
                  (c) Reconfigurations

Every once in a while I get to write a neat piece of code that I can share. This is one of those times. I realize it is not large and by PerlMonk standards not very elegant. The problem therein lies with maintainability over the next few years. Regardless I like what I wrote and would like to share.

At the Circus we had a pretty good idea that we had some data leakage. Nothing like people taking off with everything needed to get home loans and rip off customers, just people not thinking about what they send through email. We didn’t know the extent of the problem or even if we had one. We just weren’t sure. Our C-level executives didn’t believe that employees would be so careless with customer data. We decided to find out.

I must say that the results were actually quite positive. We had a couple of people email work related data home so they could work at home over the weekend and a few emails regarding employment, but they were originated by the prospective employee.

Regardless, in order for us to find out I wrote a few scripts that hook into our email system. One that I am particularly proud of recurses through a directory of email messages and attachments scanning each file for relevant data.

Please note that by the time these scripts touch the data it has been scrubbed by the antivirus and other checks we have in place. I am only looking for keywords or regular expressions that would indicate customer related data loss.

Let me explain the directory structure. Under the email system is the directory /var/spool/filter that contains every email message that has been sent in the last 30 minutes. There is a cleanup process that erases all the files in that directory and that is actually where I wrote the hook, in the cleanup process. Here is a sample listing of the directory.

As you can see, each email header ends with a .hed extension and the message is in .txt format. The ETP.doc file is an attachment.

The subroutine I am most pleased with is the one that recurses through the directory structure. The slurp command returns a hash and if it is a subdirectory then it is a hash as well. I look for it with the following line of code.

That is how I find subdirectories to push onto the stack of recursive calls. As it traverses each directory it just looks at each file extension and makes a determination as to what to do with it.

I realize most system administrators are asking why I didn’t use the file command to make sure the script was acting appropriately for each file type but that does not work with the new Microsoft document types.

I thought it was a fun project and I enjoyed writing what I felt was an interesting piece of code.