Someone on one of the lists posted about the blog Hacking Cisco so I went over to check it out. Wow. Jarek Rek is the author and he must be putting a ton of time into his studies.
Inspiring.
I have added his blog to the blogroll on the side.
I spent a considerable amount of time over the last couple of days working with DRBD and Heartbeat.
Below are the links I used to get things running:
http://wiki.centos.org/HowTos/Ha-Drbd
http://www.howtoforge.com/vm_replication_failover_vmware_debian_etch_p3
http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Clusters_from_Scratch/s-intro-pacemaker.html
http://www.drbd.org/users-guide/s-heartbeat-r1.html
http://www.drbd.org/users-guide/s-heartbeat-config.html
http://www.drbd.org/users-guide/s-heartbeat-crm.html
Part of my problem was not understanding the difference between R1 and DRM style clusters and their accompanying daemons; heartbeat, pacemaker and the different protocol versions. Pacemaker is a more advanced cluster resource manager that can work with both Corosync and Heartbeat. Heartbeat uses an older protocol whereas pacemaker uses OpenAIS to be compatible with RedHat cluster services.
Regardless here are my notes for configuration, and just for completeness my notes are a mix of doing this first on VMWare and then on a Xen cluster so any inconsistencies are a result of doing this multiple times in different environments. Regardless the errors are mine and I would recommend reading the documentation linked above.
The basics behind the setup is that DRBD replicates data between two servers. DRBD is the network block device that mirrors the data. The heartbeat daemon keeps track of the shared IP, the daemons that are in HA and runs the init scripts appropriately.
DRBD Initialization
Format the disk:
Make sure that the names names are consistent throughout all of these configuration files. This may mean ensuring they are correct in DNS and /etc/hosts.
Locally configure name for this server:
DNS name for this server:
The /etc/drbd.conf file was designed to allow a verbatim copy on both nodes of the cluster.
Copy the configuration file:
Tried to start DRBD but got an error:
I did not initialize the meta data storage and this needs to be done before a DRBD resource can be brought online. The DRBD resource needs to be down or detached from its backing storage.
Check the status:
Make it primary:
Make a file system:
Testing the filesystem:
On the second server:
Heartbeat R1-style
Heartbeat in R1 configuration uses 3 files that must be configured if you are using the heartbeat protocol.
/etc/ha.d/ha.cf
/etc/ha.d/haresources
/etc/ha.d/authkeys
Before we take care of the ha.cf file we need to set up the ha_logd configuration file.
And make changes to the logd.cf file accordingly. Be sure to copy /etc/logd.cf to both servers. Also note that I had to completely stop and then restart the heartbeat daemon for my logging changes to take affect.
Cluster Management
To take over cluster management from a primary server:
Relinquishing cluster management to a secondary server:
The order of operations as set by the init scripts:
To allow live migration on Xen:
Split-brain
Playing around this morning I got the cluster into split-brain.
Here is how to fix it.
Run this first on the secondary node.
Run this on the primary node.
Below is the CCIE written blueprint. My plan is to slowly fill in my notes with the corresponding sections. The problem arises when Cisco is vague as to what is actually required. For instance I have been studying MPLS and LDP recently. While MPLS is listed on the blueprint, you need to understand LDP as well, however, it is not on the blueprint.
I have decided go ahead and add sections I feel are necessary but my added sections will be in italics to set them apart from the official outline. They will also break from the Cisco numbering convention. For instance below, section 4.11 Implement LDP does not follow the Cisco numbering theme of 4.10, 4.20, 4.30 but instead uses 4.11. Italics combined with the odd numbering signal my personal changes to the outline.
Source:
Cisco Learning Network
1.00 Implement Layer 2 Technologies
1.10 Implement Spanning Tree Protocol (STP)
(a) 802.1d
(b) 802.1w
(c) 801.1s
(d) Loop guard
(e) Root guard
(f) Bridge protocol data unit (BPDU) guard
(g) Storm control
(h) Unicast flooding
(i) Port roles, failure propagation, and loop guard operation
1.20 Implement VLAN and VLAN Trunking Protocol (VTP)
1.30 Implement trunk and trunk protocols, EtherChannel, and load-balance
1.40 Implement Ethernet technologies
(a) Speed and duplex
(b) Ethernet, Fast Ethernet, and Gigabit Ethernet
(c) PPP over Ethernet (PPPoE)
1.50 Implement Switched Port Analyzer (SPAN), Remote Switched Port Analyzer (RSPAN),
and flow control
1.60 Implement Frame Relay
(a) Local Management Interface (LMI)
(b) Traffic shaping
(c) Full mesh
(d) Hub and spoke
(e) Discard eligible (DE)
1.70 Implement High-Level Data Link Control (HDLC) and PPP
2.00 Implement IPv4
2.10 Implement IP version 4 (IPv4) addressing, subnetting,
and variable-length subnet masking (VLSM)
2.20 Implement IPv4 tunneling and Generic Routing Encapsulation (GRE)
2.30 Implement IPv4 RIP version 2 (RIPv2)
2.40 Implement IPv4 Open Shortest Path First (OSPF)
(a) Standard OSPF areas
(b) Stub area
(c) Totally stubby area
(d) Not-so-stubby-area (NSSA)
(e) Totally NSSA
(f) Link-state advertisement (LSA) types
(g) Adjacency on a point-to-point and on a multi-access network
(h) OSPF graceful restart
2.50 Implement IPv4 Enhanced Interior Gateway Routing Protocol (EIGRP)
(a) Best path
(b) Loop-free paths
(c) EIGRP operations when alternate loop-free paths are available,
and when they are not available
(d) EIGRP queries
(e) Manual summarization and autosummarization
(f) EIGRP stubs
2.60 Implement IPv4 Border Gateway Protocol (BGP)
(a) Next hop
(b) Peering
(c) Internal BorderGateway Protocol (IBGP) and
External Border Gateway Protocol (EBGP)
2.70 Implement policy routing
2.80 Implement Performance Routing (PfR) and Cisco Optimized Edge Routing (OER)
2.90 Implement filtering, route redistribution, summarization,
synchronization, attributes, and other advanced features
3.00 Implement IPv6
3.10 Implement IP version 6 (IPv6) addressing and different addressing types
3.20 Implement IPv6 neighbor discovery
3.30 Implement basic IPv6 functionality protocols
3.40 Implement tunneling techniques
3.50 Implement OSPF version 3 (OSPFv3)
3.60 Implement EIGRP version 6 (EIGRPv6)
3.70 Implement filtering and route redistribution
4.00 Implement MPLS Layer 3 VPNs
4.10 Implement Multiprotocol Label Switching (MPLS)
4.11 Implement LDP
4.20 Implement Layer 3 virtual private networks (VPNs) on provider edge (PE),
provider (P), and customer edge (CE) routers
4.30 Implement virtual routing and forwarding (VRF) and Multi-VRF Customer Edge (VRF-Lite)
5.00 Implement IP Multicast
5.10 Implement Protocol Independent Multicast (PIM) sparse mode
5.20 Implement Multicast Source Discovery Protocol (MSDP)
5.30 Implement interdomain multicast routing
5.40 Implement PIM Auto-Rendezvous Point (Auto-RP),
unicast rendezvous point (RP), and bootstrap router (BSR)
5.50 Implementmulticast tools, features, and source-specific multicast
5.60 Implement IPv6 multicast, PIM, and related multicast protocols,
such as Multicast Listener Discovery (MLD)
6.00 Implement Network Security
6.01 Implement access lists
6.02 Implement Zone Based Firewall
6.03 Implement Unicast Reverse Path Forwarding (uRPF)
6.04 Implement IP Source Guard
6.05 Implement authentication, authorization, and accounting (AAA)
(configuring the AAA server is not required, only the client-side (IOS) is configured)
6.06 Implement Control Plane Policing (CoPP)
6.07 Implement Cisco IOS Firewall
6.08 Implement Cisco IOS Intrusion Prevention System (IPS)
6.09 Implement Secure Shell (SSH)
6.10 Implement 802.1x
6.11 Implement NAT
6.12 Implement routing protocol authentication
6.13 Implement device access control
6.14 Implement security features
7.00 Implement Network Services
7.10 Implement Hot Standby Router Protocol (HSRP)
7.20 Implement Gateway Load Balancing Protocol (GLBP)
7.30 Implement Virtual Router Redundancy Protocol (VRRP)
7.40 Implement Network Time Protocol (NTP)
7.50 Implement DHCP
7.60 Implement Web Cache Communication Protocol (WCCP)
8.00 Implement Quality of Service (QoS)
8.10 Implement Modular QoS CLI (MQC)
(a) Network-Based Application Recognition (NBAR)
(b) Class-based weighted fair queuing (CBWFQ), modified deficit round robin (MDRR),
and low latency queuing (LLQ)
(c) Classification
(d) Policing
(e) Shaping
(f) Marking
(g) Weighted random early detection (WRED) and random early detection (RED)
(h) Compression
8.20 Implement Layer 2 QoS: weighted round robin (WRR), shaped round robin (SRR),
and policies
8.30 Implement link fragmentation and interleaving (LFI) for Frame Relay
8.40 Implement generic traffic shaping
8.50 Implement Resource Reservation Protocol (RSVP)
8.60 Implement Cisco AutoQoS
9.00 Troubleshoot a Network
9.10 Troubleshoot complex Layer 2 network issues
9.20 Troubleshoot complex Layer 3 network issues
9.30 Troubleshoot a network in response to application problems
9.40 Troubleshoot network services
9.50 Troubleshoot network security
10.00 Optimize the Network
10.01 Implement syslog and local logging
10.02 Implement IP Service Level Agreement SLA
10.03 Implement NetFlow
10.04 Implement SPAN, RSPAN, and router IP traffic export (RITE)
10.05 Implement Simple Network Management Protocol (SNMP)
10.06 Implement Cisco IOS Embedded Event Manager (EEM)
10.07 Implement Remote Monitoring (RMON)
10.08 Implement FTP
10.09 Implement TFTP
10.10 Implement TFTP server on router
10.11 Implement Secure Copy Protocol (SCP)
10.12 Implement HTTP and HTTPS
10.13 Implement Telnet
11.00 Evaluate proposed changes to a Network
11.01 Evaluate interoperability of proposed technologies against deployed technologies
(a) Changes to routing protocol parameters
(b) Migrate parts of a network to IPv6
(c) Routing Protocol migration
(d) Adding multicast support
(e) Migrate spanning tree protocol
(f) Evaluate impact of new traffic on existing QoS design
11.02 Determine operational impact of proposed changes to an existing network
(a) Downtime of network or portions of network
(b) Performance degradation
(c) Introducing security breaches
11.03 Suggest Alternative solutions when incompatible changes
are proposed to an existing network
(a) Hardware/Software upgrades
(b) Topology shifts
(c) Reconfigurations
Every once in a while I get to write a neat piece of code that I can share. This is one of those times. I realize it is not large and by PerlMonk standards not very elegant. The problem therein lies with maintainability over the next few years. Regardless I like what I wrote and would like to share.
At the Circus we had a pretty good idea that we had some data leakage. Nothing like people taking off with everything needed to get home loans and rip off customers, just people not thinking about what they send through email. We didn’t know the extent of the problem or even if we had one. We just weren’t sure. Our C-level executives didn’t believe that employees would be so careless with customer data. We decided to find out.
I must say that the results were actually quite positive. We had a couple of people email work related data home so they could work at home over the weekend and a few emails regarding employment, but they were originated by the prospective employee.
Regardless, in order for us to find out I wrote a few scripts that hook into our email system. One that I am particularly proud of recurses through a directory of email messages and attachments scanning each file for relevant data.
Please note that by the time these scripts touch the data it has been scrubbed by the antivirus and other checks we have in place. I am only looking for keywords or regular expressions that would indicate customer related data loss.
Let me explain the directory structure. Under the email system is the directory /var/spool/filter that contains every email message that has been sent in the last 30 minutes. There is a cleanup process that erases all the files in that directory and that is actually where I wrote the hook, in the cleanup process. Here is a sample listing of the directory.
As you can see, each email header ends with a .hed extension and the message is in .txt format. The ETP.doc file is an attachment.
The subroutine I am most pleased with is the one that recurses through the directory structure. The slurp command returns a hash and if it is a subdirectory then it is a hash as well. I look for it with the following line of code.
That is how I find subdirectories to push onto the stack of recursive calls. As it traverses each directory it just looks at each file extension and makes a determination as to what to do with it.
I realize most system administrators are asking why I didn’t use the file command to make sure the script was acting appropriately for each file type but that does not work with the new Microsoft document types.
I thought it was a fun project and I enjoyed writing what I felt was an interesting piece of code.
At the Circus we have a network management server that runs all of the normal services needed to manage a small network and so I rarely need to fire up the tftp server on my laptop. Today was one of those days I needed a quick tftp server and I spent too much time figuring it out. This is my attempt to remedy that shortcoming.
What is ironic is that after I googled around I found that I had “self documented” in the /private/tftpboot directory, unfortunately I expected the tftp directory for the tftp server to be in /tftpboot. I realize I can just put a symlink from /tftpboot to /private/tfpboot but I learned when working on AIX it is better to understand the file system layout of a UNIX vendor than it is to make it like another OS. It will bite you eventually.
Here is a listing of the /private/tftboot directory, you will notice the very last file is tftp.txt. That is where I told myself how to do this in the past. It also appears most of the IOS images were for the testlab.
This is what I had listed in the tftp.txt file. It tells how to start and stop a service in Mac OSX using launchctl.
For thoroughness I am including the tftp.plist file below. If I wanted the tftp daemon to start every time I turned on my laptop I would change Disabled to EnableTransactions.
I have added a CCIE category to the blog. Although I have been studying steadily I have only posted testlab scripts to date. I will most likely post very little CCIE specific content while I continue to study for the written until I am much closer to my written test day.
I have really struggled with my structure studying for the written. I am a hands on learner and cramming a bunch of reading without application makes it difficult for me to remember and understand the nuances of a technology. I would rather play with a protocol and learn about it through interaction than try to memorize a bunch of random facts for the written test. Recently I have been doing INE Workbook 1 labs as I feel they complement my reading well. They are not difficult and explore the intricacies of one protocol at a time. It is easy for me to do a lab and play around with the protocol to learn.
Reading another candidates blog I ran across his study plan which was taken from
this post. At the end of that blog entry is a list of core INE Workbook 1 labs you should do while preparing for the lab. They are below for convenience.
Bridging & Switching: 1.1-1.15
Frame-Relay: 2.1-2.10
IP Routing: 3.1-3.11
RIP: 4.1-4.6
EIGRP: 5.1-5.8
OSPF: 6.1-6.11, 6.21-6.31
BGP: 7.1-7.9, 7.16-7.26
IPv6: 9.1-9.5, 9.12-9.14, 9.17-9.20, 9.29-9.31
MPLS VPN: 14.1-14.7
That is 109 labs that INE recommends you complete before moving to more advanced labs which gives me a goal and structure. I plan to do these labs in the coming months to complement my reading. That also means I need only do 11 labs per month so I will cut back on labs on the weekends and do more reading and note taking. Actually I have already done ~25 of these labs so it is even fewer labs I need to do but I will do many of them multiple times so I’m not going to quibble with the numbers.
I have completely finished reading and taking notes for TCP/IP Vol I and am half way through Vol II. I will then read the Switching Exam Certification Guide and the QoS Exam Certification Guide again, followed by the CCIEv4 Exam Certification Guide. My goal is to pass the written next winter.
To put dates to my goals:
31 March — Finish Volume II
(Second child is due in April.)
30 June — Finish reading Switching Exam Certification Guide
31 August — Finish reading QoS Exam Certification Guide
31 October — Finish reading CCIEv4 Exam Certification Guide
Finally when I begin reading the CCIEv4 Exam Certification Guide I will begin to post more of my notes. What I found when studying for my CCNP was that immediately after I finished putting all of my notes on the web for a test was when I was the most prepared for theory based exams.
I spent a considerable amount of time over the last couple of months testing different restore processes. This is my documentation for restoring Veritas/Symantec backups to a Linux server.
The general outline is this:
1. Create a LiveUSB drive to boot CentOS with a persistent overlay.
2. Install Symantec backupexec on the LiveUSB drive.
3. Recreate the drive layout on the new server.
4. Restore to the new server.
Create LiveUSB
CentOS makes a LiveCD toolset for CentOS. They also have directions for how to create a LiveUSB drive with persistent overlay. Please follow those links for more in depth directions.
You must install CentOS LiveUSB on an ext2/3/4 formatted USB drive in order for Symantec to work. If you leave the VFat partition Symantec will not work properly and you will get the error “An unknown error occurred within the NDMP subsystem.” Once I reformatted the USB drive as ext3 and installed a new LiveUSB with persistent overlay Symantec worked. My guess is it has to do with permission bits but that is only a guess.
I downloaded the LiveCD tools for Centos here.
Here is some of my history from that server:
LiveUSB Setup
I wanted to give it a persistent name and IP address for use in our data center. For some of this I was also shooting in the dark in order to get Symantec working, for thoroughness I include it here.
After I configured the hostname and network settings I rebooted to make sure that the persistent overlay worked. I also turned on sshd and set it to runlevel 3 in /etc/inittab because I did not want to mess with a gui, but that is your choice. When everything came up properly I installed Symantec and we did a test restore.
Install Symantec
I cover installing Symantec on Linux in another post here. You need to install an older package for compatibility:
The specific Symantec rpms I installed are listed below. I did try a newer package from Symantec but it did not allow us to restore erroring with a different message. I will also say that was when we were on a VFat partition. Once I got everything working on an ext3 partition I quit testing.
Recreate Drive Layout
For thoroughness I am going to cover creating the logical volumes that are default for CentOS and RHEL.
First I need to lay out the drive mappings. This is from the old server which I am cloning onto a similar server. In this section I am just going to show the output of a number of commands that confirm the file system layout of the server.
File layout on the old server
From the file /etc/fstab:
From the mount command:
From the fdisk command:
Working my way up from the bottom of the LVM stack with the physical volume, the volume group and finally the logical volume.
From pvdisplay:
From vgdisplay:
From lvdisplay:
On the old server the drive is broken into two partitions, sda1 and sda2:
sda1 /boot 100MB
sda2 Volume Group ~200GB
The volume group on the old server on the sda2 partition is broken into two logical volumes:
LogVol00 / ~200GB
LogVol01 swap ~2GB
It is important to remember that the drive mappings on the old server will not necessarily match the mappings on the new one. For instance on the old server the raid was on /dev/sda and on the new server the raid drive is mapped on /dev/sdb. That is only because I am booting from /dev/sda on the LiveUSB, under normal circumstance it will come back up as /dev/sda.
Working on the new server recreate the partitions
Turn off the the LVM in order to make changes, this is just a precautionary step if you have repartitioned your drive.
Create the LVM.
Create and activate the volume groups.
Finally, create the logical volumes. Even though I have 1.1T I decided to start using 800G, leaving myself room if I want to add another mount point.
Read in the new volume groups.
Format all of the partitions:
Here are some quick commands if you mess up and need to delete any of the LVM stack.
Restore
Mount the root under /mnt and then have Veritas restore to that mount point.
If you have made it this far then you must really need the information. Here are a couple of screenshots from our backup guru to help in the restore process.
“Preserve Tree”, by default it is selected.
Select “Restore over existing files”, “Restore all information for files and directories” and “Preserve tree”.
Install grub on the new machine
During the restore we restored all of the files and directories to /mnt, including /boot. In order to get everything working again we need to setup the boot directory and then grub. But Red Hat and CentOS 4.X uses legacy grub.
Copy all of /mnt/boot to the real /boot directory.
Restore /dev and /tmp
Depending upon your backup options you may need to restore the /dev directory and create a tmp directory. You need to set the sticky bit on /tmp.
Finally you need to set up your ethernet interfaces by editing the file,
/etc/sysconfig/networking/devices/ifcfg-eth0
At the Circus we have a password policy to change all passwords every 90 days. Today it was brought to my attention that one of the linux servers was not following that policy. I confirmed that was true and after a little digging I found that it was only accounts that had been migrated from AIX to linux. But we couldn’t force around 2000 users to all change their passwords at the same time because we would inundate the help desk.
This is the script that I wrote to fix the problem and distribute the password changes over a month. The result is that there are only 78 users per day that are forced to change their password each day over a 28 day period.
I just finished reading an article in the Atlantic concerning military personnel and recruitment titled “Why Our Best Officers Are Leaving.” As a veteran and former officer who enjoyed my time in service I felt the urge to comment publicly.
My stint in the Air Force was the formative years of my professional development. The military formed many of the core beliefs I have today. My view on documentation and succession are much different than my most of my private sector peers. One maxim that my wife and I do not see eye to eye on, “Early is on time, on time is late, late you have a problem.” Another saying that was often repeated was, “Do your current job well and your next job will take care of itself.”
Today my hair is just as short as when I was in, if not shorter and my shoes are still spit shined. Only my uniform has changed; from blues or BDUs to khakis and polos or slacks and dress shirts.
But what I really wanted to comment on was my career development. My first encounter with the Air Force Personnel Center (AFPC) was even before I went active duty. When we got our assignments as college seniors mine was to Offutt AFB, Nebraska. As a cyclist I was not pleased to be heading to a station with a 2 month summer, with fall and spring similarly abbreviated. My commanding officer at the detachment asked me if I wanted him to make a call to AFPC on my behalf and see if I get a more amenable station. I declined stating that I didn’t want to start my time in the Air Force fighting the system.
I worked hard at Offutt. Not as many hours as I do now, but I learned a great deal. The one big project I handled was the leg work, research and negotiation to settle a $1M lawsuit against the base. Our Colonel had given three of us the project and I was the one that finished the job. The other two lieutenants just didn’t find it interesting.
An aside. I was also given the task to get a squadron t-shirt designed and approved, but I just couldn’t find the time. Someone else finally did it. Now I believe it would have been a good experience because you had to work through all the red tape, but I just didn’t find that appealing.
When it came time for me to change duty stations my commanding officer called me into his office and told me he had made some phone calls and found me a position at the Air Force Logistics Management Agency (AFLMA).
I don’t believe what I did at the AFLMA was outstanding, I ran a website for the Air Force the last couple of years I was in. The website had pretty high visibility and I gave presentations to nearly every full bird Colonel and met privately with every General in my career field. I traveled extensively during this time and gave presentations like I was a salesman.
Another aside. I got married on Saturday and left for Washington, DC Sunday to give a presentation Monday morning at the Pentagon.
When I declared my intention to leave the Air Force the AFPC representative for my career field took me to lunch. He offered to station me anywhere in the world. When I told him my wife was English/South African and we were considering moving to England he offered to double billet me in England. Next he offered me a nice opening in New Zealand where I would be in charge of my own office. I declined them both and ended up in graduate school.
I just figured every Lieutenant and Captain had the same experience I did. You work hard, show initiative and let your mentors steer you through the maze of jobs and promotions. Imagine my surprise when I found that is not the case in the private sector.
I’m sorry that all of these TestLab scripts are a recurring theme. Work purchased four 3560s and two 1841s for the lab so I have been updating all of my scripts. When I was working on the lab I kept having sessions hang so I wrote a quick script to clear all of the lines on the terminal server.