Archive

Archive for December, 2009

Cold Ride

December 29th, 2009 jud 3 comments

I have a friend in Chicago and another in Boulder who talk about how nice it must be to ride all winter in the south. It kills me. Check the low for each night, because that is the temperature I ride in every day. We ride at 6:00am every Tuesday-Thursday unless it’s raining too hard, it’s too cold or a combination of the two. But what is too cold?

Like a friend said, once you ride in 16 degree weather there aren’t many of excuses left. I’m not sure many of you know what 16 degrees feels like on a bike ride, a brain freeze when you go down hill. We usually ride mountain bikes when it’s that cold, because the wind chill is too great to ride the road.

This morning was just another training ride, except it had warmed up to 26F.
It was so cold I could feel the wind on my back this morning and figured I had just not tucked in my base layer well enough. It was only when I got home and the back of my jacket was frozen solid that I understood, my bottle had been leaking down my back and the water had frozen my jersey. When it’s in the 20s your bottle freezes shut, but if you put it in your back pocket it usually won’t. Not only had my bottle frozen shut and started to freeze solid, but the water that had leaked out had frozen too.

So to all my friends who still live up north, it’s not all roses in the south either.

Categories: Cycling, Musings Tags:

Riding Rickshaw

December 22nd, 2009 jud No comments

I live in Auburn Alabama and as you might have heard, it is a college town with a major college football team.  In the fall, socializing revolves around football.  I believe part of the reason is because there are no professional level sports in the state of Alabama.  Sure we have class 5A baseball or whatever they call the minor leagues, but in reality college football is king in the state of Alabama.  On game days this year I have been riding for TigerShaw.

In my line of work I rarely interact with the general public and don’t interact with end users very often either.  It’s nice at times because the world I work in is binary, black and white, on or off, there is no gray area.  Something works or it doesn’t.  The goal of my work is to abstract out the human element and when you don’t have that human interaction you feel rusty in social settings.

I sold suits when I was in high school and college undergrad.  When I say sold suits, I don’t mean $300 suits at Sears either, I sold suits at Bachrach back before they had a house brand. We sold Armani, Boss, Joseph Aboud and some high end Hart-Marx.  My biggest sales were in the $10,000 range and this was in the 1990′s.  I made good money for a kid.  What I learned during that job was how to work the public, how to be professional in your interactions, and how to sell.  Think about it, if an executive is going to buy a $2,000 to $3,000 suit from a kid, the kid had better be good.

The same holds true riding rickshaw.  I make good money riding rickshaw but I am professional in my dealings with clients and I work the public.  You have to be professional, work hard and sell yourself the same way to make $20 for dragging someone a mile and a half or $120 for selling a $2,000 suit.

Tailwinds.

Categories: Cycling, Musings Tags:

WLAN Management

December 15th, 2009 jud No comments

The five elements of a Cisco wireless network read like an add right of some glossy, it’s here because there might be a question on the test:
Client devices — laptops, phones and
Mobility platform — Composed of lightweight access points (LWAP), wireless lan controller (WLC) and the wireless control system (WCS)
Network unification — WLCs allow for integration into the wired network
Network management — WCS for control and monitoring
Unified advanced services — Catch-all for marketing

There are two types of Cisco wireless implementations, Autonomous and Lightweight. As a networker who has worked on both, LWAP is the way to go. Management is much easier, mobility groups are your friend and because LWAPP is tunnels wireless interactions across the network to the WCS/WLC it makes it much easier to deploy in a split layer environment, L2 and L3 around campus depending upon location and need.

Autonomous APs — Each AP has its own configuration and operates independently. Configuration can be controlled through CiscoWorks WLSE.

LWAPs — Configuration, monitoring and security is centrally controlled through the WLC/WCS. This is the more scalable solution.

Wireless Lan Solution Engine (WLSE)

    Provides basic management of autonomous APs:

  • Configuration
  • Fault and policy monitoring
  • Reporting
  • Firmware upgrades
  • Radio management

Wireless Control System (WCS)

    Provides advanced management of LWAPs:

  • Location services
  • “Self healing” — If you have a WCS you know what this means.
  • Failover between WLCs
  • Monitoring
  • Configuration
  • Firmware upgrades
Categories: CCNP ONT Tags:

The problem with 10-12 hour days

December 14th, 2009 jud No comments

No wonder Google has such “extravagant” benefits. When you spend _all_ day and most nights at work there is precious little time for incidentals. The barber doesn’t stay open late and neither does the dry cleaner. Thankfully the grocery is open at all hours, I got up at 5:10am this morning to go to the grocery. Went home to make lunch and dinner for the next five days, and froze it all. Then I bring them to work and eat them like frozen dinners.

Now I’m not complaining about making my own frozen dinners, it beats the alternatives, real frozen dinners or food from the cafeteria. What I’m really saying is that time management is a must-have skill when working long hours at a company not designed to accommodate it.

On my way home after another 11 hours to see my wife and daughter, they’ll put a smile on my face and remind me why I stay late to study.

Categories: Musings Tags:

WLAN Encryption and Authentication

December 14th, 2009 jud No comments
    Wired Equivalent Privacy (WEP)
    Wired Equivalent Privacy (WEP) was the first implementation of wireless encryption, it has several weaknesses and should not be used. Weaknesses include:

  • Weak encryption that has been broken
  • Vulnerable to dictionary attacks
  • Client trusts AP allowing for a rogue access point
  • Keys must be manually distributed

802.1x Extensible Authentication Protocols (EAP)
EAP was originally developed for wired port access control and was adopted for wireless access control. The pieces for an EAP implementation require a client wireless card and supplicant, authentication server and access point that are all EAP capable. A wireless client can only transmit EAP traffic until it is authenticated, the RADIUS server authenticates the client and the client authenticates the server in a challenge and response. EAP was originally defined in RFC 2284, which was made obsolete by RFC 3748 and defined for wireless LANs in RFC 4017.

    EAP Features:

  • RADIUS authentication server
  • Can use multiple encryption algorithms
  • Dynamic WEP keys
  • Encrypted passwords
  • Centralized control
    Cisco Lightweight Extensible Authentication Protocol (LEAP)

  • Fast and secure roaming
  • Single sign-on with various backends, AD, LDAP
  • Widely supported; MS, MAC and Linux clients
    Extensible Authentication Protocol – Flexible Authentication via Secure Tunneling (EAP-FAST)

  • EAP-FAST is nonproprietary, defined in RFC 4851
  • Mutual authentication of server and client by using the TLS handshake protocol
  • Immune to man in the middle attacks
  • Ability to use multiple password authentication backends
  • Computationally efficient
  • Does not require certificates
    EAP-FAST consists of three phases:

  1. Phase 0 — Client is dynamically provisioned with a Protected Access Credential (PAC) which can also be installed manually, so this phase is considered optional.
  2. Phase 1 — Server and client use PAC to authenticate each other and establish a secure tunnel.
  3. Phase 2 — Client sends credentials through tunnel for authentication.

Exensible Authentication Protocol – Transport Layer Security (EAP-TLS)
EAP-TLS was originally defined in RFC 2716 but was redefined by RFC 5216 in March of 2008, TLS enhancements were defined in RFC 4507. Uses public key infrastructure (PKI) meaning that both client and server need a certificate for authentication and the certificates must be issued by a certification authority (CA). Client is the supplicant, authenticator is the AP and the authentication server is the RADIUS server.

    EAP-TLS Authentication Process:

  1. Client associates to AP which restricts traffic to only EAP traffic.
  2. AP requests identity which it then passes to the RADIUS server.
  3. Client validates certificate and responds with EAP with it’s own certificate which starts cryptographic negotiations.
  4. After the RADIUS server validates the client certificate it responds with the cryptographic specifications for the session.

Protected EAP (PEAP)
PEAP only requires the authentication server to have a certificate.

    PEAP has two phases:

  • Phase 1 – The client authenticates the server using the CA to verify its certificate and an encrypted TLS tunnel is created with the client.
  • Phase 2 – Client is authenticated using industry supported authentication.
    PEAP has two authentication implementations:

  • Generic Token Card (GTC) (PEAP-GTC ) for client authentication using Novell or LDAP.
  • Microsoft Challenge Handshake Authentication Protocol (MS-CHAP) version 2 (PEAP-MSCHAPv2) for MS single sign-on.

WPA
WPA with TKIP can now be broken in a minute. It is not recommended for use, however it is still on the test. WPA with TKIP encryption was developed as an interim standard, created to maintain backward compatibility with hardware that had supported WEP. WPA performs authentication using either 802.1x/EAP or with preshared keys prior to the key management phase. WPA uses Temporal Key Integrity Protocol (TKIP) and Message Integrity Check (MIC) and per-packet keying (PPK) in an attempt to make it more secure.

802.11i or WPA2
802.11i is known more commonly as WPA2 and refers to the approved implementation of members of the Wi-Fi Alliance. It provides stronger encryption, AES rather than the weaker RC4 used by WEP and WPA. As a result it commonly required a hardware upgrade.

    Keys WPA2 facts from the ONT book:

  • Uses 802.1x for authentication
  • Uses similar method of key distribution and renewal as WPA
  • Supports Proactive Key Caching (PKC)
  • Has Intrusion Detection System (IDS)
    WPA/WPA2 provide two modes of operation:

  • Personal mode — Authentication is performed using PSK
  • Enterprise mode — 802.1x/EAP and AAA/Radius server is used for authentication

I will end with this quote, I wish I could find the reference but it is probably from the ONT book:

Some people mistakenly think that if the AP is configured not to broadcast its SSID, they have a secure wireless LAN; that is not true. When a legitimate wireless client with the correct SSID attempts to associate with its AP, the SSID is exchanged over the air unencrypted; that means that an illegitimate user can easily capture and use the SSID.

Additional resources:
NetCraftsmen Examining 802.1x and EAP
IEEE 802.1x Standard Document
TLS Deployment Guide
EAP, RFC 2284, RFC 3748
EAP Wireless, RFC 4017
EAP-FAST, RFC 4851
EAP-TLS, RFC 2716, RFC 5216
TLS Enhancements, RFC 4507

Categories: CCNP ONT Tags:

WLAN QoS

December 11th, 2009 jud No comments

Note
Where to begin? For each chapter from the ONT book I read that chapter, read the same topics in the QoS Exam Certification Guide, begin to take notes and then decide what feels lacking in my understanding. From that gut feeling I begin to branch out, starting with Cisco documentation, then trustworthy sources on the net.

WLAN QoS has morphed on me. ONT covers split-mac, WMM and implementation with coverage of 802.11e nearly non-existent. Not only that, but the portions of 802.11 covered in the book are narrow. For instance 802.11 defines two channel access mechanisms, distributed coordination function (DCF) which is covered, but point coordination function (PCF) is not even mentioned. As a result, I’m just going to fill in what makes sense to me as I believe having a basic understanding of WLAN QoS and wireless in general is the purpose of the wireless section anyway.

    Definitions:

  • 802.11e — QoS extensions for WLANs.
  • Wi-Fi Multimedia (WMM) — A subset of 802.11e with four access categories; voice, video, best effort and background. The WCS breaks them out as platinum, gold, silver and bronze.
  • Contention Window — CWmin and CWmax — Define the the range of the contention window, the back off timer in CSMA/CA. A client with a voice CWmin waits less time to send than a client in the best effort access category that has a larger CWmin.
  • Arbitrary Inter-Frame Space Number (AIFSN) — Controls the idle time, after which a client may transmit.
  • Transmission Opportunity (TXOP) — The time interval, start and maximum duration, a client holds the channel and is able to send data.

802.11 Basics
Wired clients use carrier sense multiple access with collision detection (CSMA/CD) to determine whether they can transmit, however, one of its cousins is token ring which is rarely used today. In the same manner wireless access has two channel access mechanisms, distributed coordination function (DCF) and point coordination function (PCF). Like token ring, PCF is rarely used. DCF uses carriers sense multiple access with collision avoidance (CSMA/CA) which on it’s own provides best effort delivery.

    Distributed Coordination Function (DCF) is composed of two main components:

  1. Interframe spaces — SIFS, PIFS and DIFS control access to the channel.
  2. Contention window — The random backoff timer.
    Interframe spacing is a portion of the time a client or AP waits before sending frames on the channel. It is independent of the backoff time or contention window.

  • Short Interframe Space (SIFS) — 802.11 management frames and those not expecting contention because they are port of a sequence of frames.
  • Point Coordination Function Interframe Space (PIFS) — Used by an AP to decide when to send, shorter than DIFS giving the AP priority over data, but longer than SIFS giving flows priority. SIFS < PIFS < DIFS
  • Distributed Coordination Function Interframe Space (DIFS) — When a client begins to send new data it waits DIFS, checking for a clear channel before sending.

WLAN QoS
802.11e and Wi-Fi multimedia (WMM) provide QoS for wireless networks. WMM was implemented by the Wif-Fi Alliance before 802.11e was ratified. I think of it as wireless QoS light as it implements a subset of 802.11e. 802.11e and WMM use Enhanced DCF (EDCF) to provide proportional back-off window sizes for each class.

802.11e

    802.11e defines enhancements to 802.11 Medium Access Control (MAC) to provide QoS features by using features from both PCF and DCF in what is called Hybrid Coordination Function (HCF).

  • 802.11e provides eight priority levels, 0 through 7.
  • Different acknowledgement rules provide greater efficiency in being able to send data.
  • Piggybacking allows data to be sent with poll requests and ACKs improving network performance.
  • Allows for contention free bursts, clients and APs are able to send several frames without contention.
    HCF has two modes of operation:

  • Enhanced Distribution Coordinate Access (EDCA)
  • HCF Controlled Channel Access (HCCA)
    Enhanced Distribution Coordinate Access (EDCA) is an extension to DCF that uses contention based access while providing prioritized access to the channel.
    EDCA has four key components:

  • CWmin with higher priority assigned a shorter CWmin.
  • CWmax
  • TXOP limit specifies the maximum duration a client can transmit, makes channel access more efficient.
  • Arbitration Inter-Frame Space (AIFS) specifies additional time between when a channel goes idle and the client starts to send. Each access class is assigned a different AIFS to further differentiate QoS.
    HCF Controlled Channel Access (HCCA) is polling based and is uses a coordinator to centrally manage access.

  • HCCA can poll clients during the contention period.
  • Supports scheduling of packets based on traffic flow requirements.
  • The coordinator has the highest priority access.

Wi-Fi Multimedia
802.11e priorities can be mapped to WMM access categories for backward compatibility:

WMM  802.11e
Voice (Platinum) 6 or 7
Video (Gold) 4 or 5
Best-Effort the Default (Silver) 0 or 3
Background (Bronze) 1 or 2

The image below is my recreation of a figure out of the Wi-Fi Alliance documentation below. It explains the interaction between AIFSN, CWmin/CWmax and it’s affect on WMM. In essence a voice client waits less time before trying to retransmit than a lower access category and will therefore have a better chance at sending data, it is not a strict priority system. Click here for a better view of the image.
WiFi-WMM Explanation

Split MAC Architecture
Centralizes wireless LAN configuration and control onto the wireless lan controller (WLC). Access points are lightweight and cannot act independently of a controller. The wireless LAN controller manages the access point configurations and firmware. The access points handle only real-time MAC functionality, leaving all the non-real-time MAC functionality to be processed by the wireless LAN controller.

    WLC manages:

  • Channel assignment
  • Association, disassociation and reassociation
  • 802.11e and WMM resource reservation
  • Transmit power optimization
  • Self-healing wireless coverage
  • Dynamic client load balancing
    LWAP manages:

  • Transmission of beacon frames
  • Probe transmission and response
  • 802.11e and WMM scheduling and queuing
  • Buffering frames for clients
  • Monitoring each radio channel for noise and interference

Light weight access point protocol (LWAPP)
Light weight access point protocol (LWAPP) is used to communicate between the WLC and the APs. Wireless LAN client data packets are encapsulated in LWAPP between the access point and the wireless LAN controller. When a wireless LAN client sends a packet, it is received by the access point, decrypted if necessary, encapsulated with an LWAPP header and forwarded to the controller. At the controller, the LWAPP header is removed and the frame switched from the controller onto a vitrual LAN (VLAN) in the switching infrastructure.

There are two types of LWAPP traffic, control messages and client data.

    LWAPP control messages:

  • Used to configure the LAP and manage its operation.
  • Authenticated and encrypted, the LAP is securely controlled by only the WLC.
  • Classified automatically with a DSCP of CS6.
  • Identified by UDP port 12223
    LWAPP data:

  • Packets to and from wireless clients associated with the LAP.
  • The data is encapsulated within LWAPP but is not encrypted.
  • The default classification for WLAN data traffic is best-effort.
  • Identified by UPD port 12222

Additional Sources:
WMM from Wi-Fi.org
WLAN Tuning
Cisco Mobility Design Guide
LWAPP Traffic Study
WLC Deployment Guide
Intel QoS Paper
European Wireless Conference Paper

Categories: CCNP ONT Tags:

LLQ Lab

December 7th, 2009 jud No comments

Basic Lab Diagram

Click here for a better image.

You can download the initial configuration files here.

The only difference between this lab and CBWFQ is making it LLQ. I would not do these one after the other, intersperse another lab, or do CBWFQ and then change it to LLQ.

To prepare for this lab only turn on one 800K link between R1 and R2, and the link between R4 and R1 for traffic generation. If you want traffic to make it around the lab then configure to your hearts content.

On R1:

  • Create access lists web, control and print.
  • Create class maps web, control and print.
  • Put http, ftp, pop3 and smtp in the web class map.
  • Put ntp, ssh, telnet and x11 in control.
  • Put 9100 in print..
  • Create the policy map cbwfq and give these bandwidth percentages; web: 30%, control: 20%, print: 10%.
  • Make the control group the priority queue.
  • Apply the configuration to S0/0.
  • Confirm the configuration and debug it.

Here are the protocols for which traffic is generated from our traffic generation configuration file:

for I in `grep dest-port r4-basic-tgn.cfg | cut -d\   -f 3`; do grep [[:space:]]$I/tcp /etc/services; done
telnet          23/tcp
http            80/tcp          
ftp             21/tcp
ntp             123/tcp
pop3            110/tcp      
smtp            25/tcp        
ssh             22/tcp          
x11             6000/tcp    
jetdirect       9100/tcp

Answer is below:

R1 Configuration for CBWFQ:

! CEF must be turned on.
ip cef
! 1.  Create the access-list.
ip access-list extended control
 permit tcp any any eq 123
 permit tcp any any eq telnet
 permit tcp any any eq 22
 permit tcp any any eq 6000
ip access-list extended print
 permit tcp any any eq 9100
ip access-list extended web
 permit tcp any any eq www
 permit tcp any any eq ftp
 permit tcp any any eq pop3
 permit tcp any any eq smtp
!
! 2.  Create the class-map.
class-map match-any control
 match access-group name control
class-map match-any web
 match access-group name web
class-map match-any print
 match access-group name print
!
! 3. Create the policy-map.
policy-map cbwfq
 class web
  bandwidth percent 30
 class control
! 3a.  Notice this is the only difference between LLQ and CBWFQ.
  priority percent 20
 class print
  bandwidth percent 10
 class class-default
  fair-queue
!
! 4.  Apply it to the interface.
interface Serial0/0
 bandwidth 800
 ip address 192.168.112.1 255.255.255.0
 service-policy output cbwfq

Confirm the configuration:

sh int s0/0
sh queueing
sh policy-map int s0/0
sh poicy-map int s0/0 output class control

Debug the configuration:

debug priority
Categories: CCNP ONT Tags:

CBWFQ Lab

December 7th, 2009 jud No comments

Basic Lab Diagram

Click here for a better image.

You can download the initial configuration files here.

To prepare for this lab only turn on one 800K link between R1 and R2, and the link between R4 and R1 for traffic generation. If you want traffic to make it around the lab then configure to your hearts content.

On R1:

  • Create access lists web, control and print.
  • Create class maps web, control and print.
  • Put http, ftp, pop3 and smtp in the web class map.
  • Put ntp, ssh, telnet and x11 in control.
  • Put 9100 in print..
  • Create the policy map cbwfq and give these bandwidth percentages; web: 30%, control: 20%, print: 10%.
  • Apply the configuration to S0/0.
  • Confirm the configuration and debug it.

Here are the protocols for which traffic is generated from our traffic generation configuration file:

for I in `grep dest-port r4-basic-tgn.cfg | cut -d\   -f 3`; do grep [[:space:]]$I/tcp /etc/services; done
telnet          23/tcp
http            80/tcp          
ftp             21/tcp
ntp             123/tcp
pop3            110/tcp      
smtp            25/tcp        
ssh             22/tcp          
x11             6000/tcp    
jetdirect       9100/tcp

Answer is below:

R1 Configuration for CBWFQ:

! CEF must be turned on.
ip cef
! 1.  Create the access-list.
ip access-list extended control
 permit tcp any any eq 123
 permit tcp any any eq telnet
 permit tcp any any eq 22
 permit tcp any any eq 6000
ip access-list extended print
 permit tcp any any eq 9100
ip access-list extended web
 permit tcp any any eq www
 permit tcp any any eq ftp
 permit tcp any any eq pop3
 permit tcp any any eq smtp
!
! 2.  Create the class-map.
class-map match-any control
 match access-group name control
class-map match-any web
 match access-group name web
class-map match-any print
 match access-group name print
!
! 3. Create the policy-map.
policy-map cbwfq
 class web
  bandwidth percent 30
 class control
  bandwidth percent 20
 class print
  bandwidth percent 10
 class class-default
  fair-queue
!
! 4.  Apply it to the interface.
interface Serial0/0
 bandwidth 800
 ip address 192.168.112.1 255.255.255.0
 service-policy output cbwfq

Confirm the configuration:

sh int s0/0
sh queueing
sh policy-map int s0/0
sh poicy-map int s0/0 output class control

Debug the configuration:

debug priority

http://www.cisco.com/en/US/docs/ios/12_0t/12_0t5/feature/guide/cbwfq.html

Categories: CCNP ONT Tags:

Priority Queuing Lab

December 4th, 2009 jud No comments

Basic Lab Diagram

Click here for a better image.

You can download the initial configuration files here.

To prepare for this lab only turn on one 800K link between R1 and R2, and the link between R4 and R1 for traffic generation. If you want traffic to make it around the lab then configure to your hearts content.

On R1:

  • Use priority list number 4.
  • Set smtp and pop3 with a high priority.
  • Set ssh and X11 with medium priority.
  • Set http, ftp and ntp as low priority.
  • Set telnet as normal priority.
  • Set the default priority as low.
  • Apply the configuration to S0/0.
  • Confirm the configuration and debug it.

Here are the protocols for which traffic is generated from our traffic generation configuration file:

for I in `grep dest-port r4-basic-tgn.cfg | cut -d\   -f 3`; do grep [[:space:]]$I/tcp /etc/services; done
telnet          23/tcp
http            80/tcp          
ftp             21/tcp
ntp             123/tcp
pop3            110/tcp      
smtp            25/tcp        
ssh             22/tcp          
x11             6000/tcp    
jetdirect       9100/tcp

Answer is below:

R1 Configuration for PQ:

priority-list 4 protocol ip high tcp smtp
priority-list 4 protocol ip high tcp pop3
priority-list 4 protocol ip medium tcp 22
priority-list 4 protocol ip medium tcp 6000
priority-list 5 protocol ip normal tcp 23
priority-list 5 default low

interface Serial0/0
 bandwidth 800
 ip address 192.168.112.1 255.255.255.0
 priority-group 4

Confirm the configuration:

sh int s0/0
sh queueing

Debug the configuration:

debug priority
Categories: CCNP ONT Tags:

Pingcheck

December 2nd, 2009 jud No comments

In my previous posts I have discussed other scripts to monitor or work with DNS. This pingcheck script is used to find used or unused IP addresses in the network, or to populate an arp table for a subnet. You can download the following code here as a normal shell script or as a .tar file.

This script also uses jot, one of the first scripts I posted from my script library.

#!/bin/bash
# 2005-06-06 Jud Bishop
# Released under GPLv2.
# pingcheck -- Finds open ip addresses or inversely active ip addresses in a range.
# Output includes activity as well as DNS resolution.

if [ -z "$1" ]
then
    echo "Usage:"
    echo "      pingcheck 190 254 192.168.1"
    echo "          190 is the starting ip address"
    echo "          254 is the last ip address"
    echo "          192.168.1 is the subnet"
    echo "Example:"
    echo "      pingcheck 190 254 192.168.1 "
else
    for I in `jot -s $1 -e $2`
    do
        NAME=`nslookup $3.$I | grep = |cut -d = -f 2`
        # These two lines are the same, just different ways to clean up the output.
        #ping -c 1 -w 1 $3.$I 2>/dev/null 1>/dev/null
        ping -c 1 -w 1 $3.$I >/dev/null  2>&1
        if [ $? -eq 0 ]
        then
            echo $3.$I exists $NAME;
        else
            echo $3.$I does not exist $NAME;
        fi
    done
fi
Categories: Code Tags: