Archive

Archive for the ‘Code’ Category

Two Variables, One Line

August 24th, 2010 jud No comments

This morning I could not for the life of me remember how to read two variables from one line in bash. As a result I am putting this simple script up here so that I have an easy place to reference.

The input file was a listing of printer IP addresses that are translated is in the file /tmp/printers.txt and looks like this.

cat /tmp/printers.txt
192.168.1.10  =  10.0.1.10
192.168.50.5  =  10.0.1.11
192.168.50.15  =  10.0.1.25

Here is the simple code to read both variables.

#!/bin/bash
# 2010-08-24 Jud Bishop
# Simple script to find names of local and remote printers
# that are translated.

while IFS== read remote local  
do
        name=`dig +short -x $local`
        echo -e "$name,$remote,$local"
done < /tmp/printers.txt

But it came it out in this format, not much of a problem but I prefer it more legible.

tcp5.chainringcircus.org.,192.168.1.10   ,   10.0.1.10
rmp7.chainringcircus.org.,192.168.50.5   ,   10.0.1.11
jlb3.chainringcircus.org.,192.168.50.15   ,   10.0.1.25

So I cleaned up the output. The first sed stanza deletes the third “.” in the output and the second sed stanza deletes the spaces.

./find-printers.sh | sed 's/\.//3
s/\ //g'
tcp5.chainringcircus.org,192.168.1.10,10.0.1.10
rmp7.chainringcircus.org,192.168.50.5,10.0.1.11
jlb3.chainringcircus.org,192.168.50.15,10.0.1.25
Categories: Code Tags:

More TestLab Scripts

June 25th, 2010 jud No comments

I wrote a couple more scripts this week. These two little gems update my entire lab by running one command from the testlab tftp server. I pass the main script a lab name and it proceeds to update every router and switch in the testlab to the new initial lab configuration. I have tested it on at least one 1841, 2611, 3640, 3550 and 3560. I probably spent more time writing the scripts than it would have taken me to upgrade the testlab by hand for each lab, but I guess I’ll never know. You can download the scripts here.

Let me explain how it is done. Every router and switch has a configuration file named flash:def. When the whole lab has flash:def loaded they can all get to the tftp server for the lab. First side note: it was named flash:default but default is a reserved word in expect so I had to change the name, def is short for default.

In the first round of interaction the script saves every running-config to flash:new and then does a configure replace flash:def. If you just wanted to reset your lab each time, you could stop the script there. You should make sure that the switches all get changed to a VTP mode transparent and set to a new VTP domain, oddly enough I used def as my VTP domain. By making all of the switches transparent if the new config loads any VLANs or changes the domain your VTP domain will be reset. Second side note: I was going to save the original running-config as flash:save, that would still be a trivial change to make and then you could go back and review the config if you needed, but some labs build on top of the next, so I didn’t.

After each router and switch has the def configuration file loaded the script does an ls of the correct lab directory on the tftp server and proceeds to tftp the new configuration file as flash:new, overwriting the previously saved running-config. The tftp server is hooked to the switch named Cat4 which is the last host upgraded by the script so all devices can get to the tftp server.

Finally the script goes back through each device doing a round of configure replace:new. The best part of this is that there is no reload. The 2611 can take nearly 10 minutes to reload so I did not want to have to reload any devices.

In summary this is the process:
1. Save every running-config as flash:new.
2. Load the default configuration flash:def that allows for tftp access for the entire lab.
3. Copy the initial configuration for the new lab to flash:new on the routers that need it, overwriting the saved flash:new.
4. Run configure replace flash:new on ever device, bringing up the newest lab.

Here is a listing of the files on R2.

R2#dir flash:
Directory of flash:/

    1  -rw-    32632600                    <no date>  c3640-a3js-mz.124-25b.bin
    2  -rw-        1132                    <no date>  r2-basic-ios.cfg
    3  -rw-        1200                    <no date>  r2-gre.cfg
    4  -rw-        6446  May 28 2002 02:23:57 +00:00  ipbasic.cfg
   13  -rw-         883                    <no date>  save
   15  -rw-        1061                    <no date>  def
   18  -rw-        1226                    <no date>  new

33030140 bytes total (372576 bytes free)
R2#

This is the tlu script. It is just the simple front end to the tlue script.

#!/bin/bash
# 2010-05-25 Jud Bishop
# tlu (testlab update)
# One of two scripts that updates the entire testlab to the current lab.
# This script calls tlue (testlab update expect) that does the heavy lifting.

if [ $# -ne 1 ]
then
        echo -e "Usage: ${0} LAB-X\n ${0} LAB-1\n"
        exit 1
fi

LAB=${1}

# First go through and set every router or switch to the default configuration.
# Calling tlue (testlab update expect) with the def command, the def
# configuration sets every router to clean confiuration that allows access to
# the tftp server.  
# R3 is the frame switch so it is not included.
config_replace ()
{
        LAB=${1}
        CONFIG=${2}
        for ROUTER in R1 R2 R4 R5 R6 R7 R8 R9 BB1 BB2 BB3 Cat1 Cat2 Cat3 Cat4
        do
                echo $ROUTER $LAB $CONFIG
                tlue $ROUTER $LAB $CONFIG
                sleep 10
        done
}

#
## Main
#
config_replace $LAB "def"

for CONFIG in `ls /tftpboot/VOL-1/${LAB}/INITIAL/`
do
        ROUTER=`basename $CONFIG .txt`
        echo $ROUTER
        tlue $ROUTER ${1} tftp
        sleep 10
done

config_replace $LAB "new"

The expect script is the one that does all of the work and took the most time to write.

#!/usr/bin/expect -f
# 2010-06-25 Jud Bishop
# tlue (testlab update expect)
# This script does the heavy lifting of updating the testlab.
# If you use this script you will have to change the IP address
# in the procedure copy_tftp.

set host "testlab.chainringcircus.org"
set pass "CHANGEME"
set ctrlz \032
set timeout 100
set spawn_telnet 0

# Sent in from the command line.
set router [lindex $argv 0]
set lab [lindex $argv 1]
set config [lindex $argv 2]

proc login {router} {
        puts "login"
        global host pass ctrlz spawn_telnet
        spawn telnet $host
        expect "Password:" {
                send "$pass\r"
        }

        expect "testlab>" {
                send "telnet $router\r"
        }
        sleep 2

        expect "Open" {
                send "\r"
        }
        sleep 2

        # Just in case we are in configure mode.
        send "$ctrlz"
        send "\r"
        set spawn_telnet $spawn_id
}

proc save_config {router} {
        puts "save_config"
        global spawn_telnet
        set spawn_id $spawn_telnet
        # Saving the current config in case I want it for some reason.
        expect "$router#" {
        send "copy run flash\:new\r"
        }

        # Destination filename [new]?
        expect {
                 "\[new\]" {send "\r"}
        }

        #Do you want to over write? [confirm]
        expect {
                -re ".*Do you want to over write.*" {send "y\r"}
                "$router#" {send "\r"}
        }
        sleep 2

        # No, do not erase flash:.
        # Erase flash: before copying? [confirm]
        expect  {
                 -re "Erase flash\:" {send "n"}
                "$router#" {send "\r"}
        }
        sleep 2
}

# Pass in the file to load from flash, for me it's either "new" or "def".
proc configure_replace {config router} {
        puts "configure_replace"
        global spawn_telnet
        set spawn_id $spawn_telnet
        # This gets the router/switch to a known configuration that can reach the ftp server.
        expect "$router#" {
                send "configure replace flash\:$config\r"
        }

        #configure replace flash:default
        #This will apply all necessary additions and deletions
        #to replace the current running configuration with the
        #contents of the specified configuration file, which is
        #assumed to be a complete configuration, not a partial
        #configuration. Enter Y if you are sure you want to proceed. ? [no]:
        expect -re "(.*)(no)(.*)" {
                send "y\r"
        }
        sleep 5
}

proc copy_tftp {lab router} {
        puts "copy_tftp"
        global spawn_telnet
        set spawn_id $spawn_telnet
        #R1#copy tftp://192.168.1.234/R1.txt flash:new
        expect "$router\#" {
                send "copy tftp://192.168.1.234/VOL-1/$lab/INITIAL/$router.txt flash:new\r"
        }

        #Destination filename [new]?
        expect {
                -re "Destination filename" {send "\r"}
        }

        #Erase flash: before copying? [confirm]n
        #OR
        #Do you want to over write? [confirm]y
        #expect -re {
        #       "(.*)(before copying)(.*)" {send "n\r"}
        #       "(.*)(over.write)(.*)" {send "y\r"}
        #}
        #Do you want to over write? [confirm]
        expect {
                -re ".*Do you want to over write.*" {send "y\r"}
                "$router#" {send "\r"}
        }
        sleep 2

        #If it was over write above, now it might ask for erase flash or
        #return the router prompt.
        #Erase flash: before copying? [confirm]
        expect {
                -re "Erase flash" {send "n\r"}
                "$router#" {send "\r"}
        }

        #Loading R1.txt from 192.168.1.234 (via FastEthernet0/0): !
        #[OK - 902 bytes]
        #Verifying checksum...  OK (0xFBC4)
        #902 bytes copied in 0.232 secs (3888 bytes/sec)
        #R1#

        expect "$router#" {
                send "\r"
        }
}

#
## Main
#
# Oh the irony.  If only I had known how many problems calling my "default"
# configuration flash:default would cause me.  As a result I changed it to def.
# However instead of a nice switch statement I'm using an if block, default is a
# reserved word in Tcl switch statements.  I did not take the time to go back and
# change it.
puts "router $router"
puts "lab $lab"
puts "config $config"

if {[ string compare $config tftp ] == 0}  {
        login $router
        copy_tftp $lab $router
} elseif {[ string compare $config new ] == 0}  {
        login $router
        configure_replace new $router
} elseif {[ string compare $config def ] == 0} {
        login $router
        save_config $router
        configure_replace def $router
} else {
        puts "tlue router lab \[def new tftp\]"
        puts "Example:"
        puts "tlue R1 LAB-1 def"
        puts "default saves the running-config to flash:new and runs configure replace:flash:default"
        puts "tftp loads the correct configuration for the lab from the tftp server as flash:new"
        puts "new runs configure replace flash:new, replacing the running config with new"
        exit
}

exit
Categories: Code, Routing Tags:

TestLab Scripts

June 18th, 2010 jud 2 comments

I wrote a couple of scripts for the testlab this week and figured I would share them. You can download them here. These could be modified for a GNS3 lab setup easily as well. Not wanting to reinvent the wheel I googled around and started hacking another script. It started off with something like this:

#!/bin/sh
# Usage: $0 [command]
pgrep -u "$USER" gnome-terminal | grep -qvx "$$"
if [ "$?" -eq 0 ]; then
  WID=`xdotool search --class "gnome-terminal" | head -1`
  xdotool windowactivate $WID
  #xdotool key ctrl+shift+t
  xdotool key ctrl+t

But gnome-terminal sets the environmental variable WINDOWID so I began by changing it to:

#!/bin/sh
# Usage: $0 [command]

xdotool windowactivate $WINDOWID
xdotool key ctrl+t

But then I read the gnome-terminal manpage to see what other environmental variables it set and decided all the xdotool commands were too much for what I needed. So I simplified into two main scripts, one to handle the gnome-terminal interactions and one to handle the router interactions.

#!/bin/bash
# 2010-06-14 Jud Bishop
# tlr
# This script opens a single gnome-terminal tab and log into a router in
# the testlab.

if [ $# -ne 1 ]
then
    echo -e "Usage: ${0} router_id\n tlr R1\n"
    exit 1
fi

gnome-terminal --tab -e "tle ${1}" -t "${1}"

The expect script to handle the interaction with the 2511-RJ getting logged into the router.

#!/usr/bin/expect
# 2010-06-14 Jud Bishop
# tle
# A short script to handle logging into a router in the lab.

set host "testlab.chainringcircus.org"
set pass "CHANGEME"

##############################
# Should not need any more changes.
set router [lindex $argv 0]

spawn telnet $host
expect "Password:"
send "$pass\r"
expect "testlab>"
send "telnet $router\r"
sleep 1
send "\r"
sleep 1
send "\r"
interact
exit

And finally the script that logs into every router in the lab, renaming the tab title to match the router name.

#!/bin/bash
# 2010-06-14 Jud Bishop
# tl
# This script fires up gnome-terminal with a bunch of tabs each executing
# the tle script and naming the tab with the router name.

gnome-terminal --tab -e "tle R1" -t "R1" \
--tab -e "tle R2" -t "R2" \
--tab -e "tle R3" -t "R3" \
--tab -e "tle R4" -t "R4" \
--tab -e "tle R5" -t "R5" \
--tab -e "tle R6" -t "R6" \
--tab -e "tle R7" -t "R7" \
--tab -e "tle R8" -t "R8" \
--tab -e "tle R9" -t "R9" \
--tab -e "tle Cat1" -t "Cat1" \
--tab -e "tle Cat2" -t "Cat2" \
--tab -e "tle Cat3" -t "Cat3" \
--tab -e "tle Cat4" -t "Cat4" \
--tab -e "tle BB1" -t "BB1" \
--tab -e "tle BB2" -t "BB2" \
--tab -e "tle BB3" -t "BB3"
Categories: Code, Routing Tags:

DocMgr with AD Authentication

April 23rd, 2010 jud No comments

Over the past couple of days I hacked an older version of DocMgr to authenticate users to Active Directory. I thought it might be interesting for some readers to see my thought process, hence this post.

All of the scripts in this post along with my test directory and my changes to DocMgr can be downloaded here.

First I started out with a simple php form for authentication. The input portion is in the download, this is the processing side. Some interesting notes about this script. At first I was using the full distinguished name (DN) of the user but that doesn’t scale. It was not until I downloaded adLDAP that I saw you could just use username@domain.org to authenticate. I don’t know why I never tried, but that stymied me.

The reason this script looks for groups the user belongs to is because originally I figured I would limit access by group. The problem is that DocMgr has it’s own permission system and you must be in the database in order to be able to login. The script at the bottom is how I dealt with that limitation.

The ldpad_set_option and ldap_get_option calls were for testing once I began hacking DocMgr. I added them in to test an error I was getting in ldap_search.

<?php
session_start();
header("Cache-control: private"); //IE 6 Fix

$username=$_POST['username'];
$password=$_POST['password'];
$ds;
$searchbase = 'OU=Users,DC=CIRCUS,DC=ORG';
$group = 'CN=DocMgr,OU=Users,DC=CIRCUS,DC=ORG';

echo "<html> <head>  <title>AD Test</title></head> <body>";
echo "<p align=\"left\">This is testAD</p>\n";
echo "Username: ". $username . "<br>\n";
echo "Password: ". $password . "<br>\n";

function ad_ldap_auth()
{
    global $username, $password, $ds;
    $ldapServer = 'server.circus.org';
    $ldapPort = '389';
    $ds = ldap_connect($ldapServer, $ldapPort)
        or die("Could not connect to ldap.");

    ldap_set_option($ds, LDAP_OPT_REFERRALS, 0);

    if ($ds)
    {
        $binddn = "$username@.circus.org";
        $ldapbind = ldap_bind($ds, $binddn, $password);

        if ($ldapbind)
        {
                echo "Bound<br>\n";                  
        }else{
                echo "<br>Not bound<br>\n";                  
        }
    }
    return $ldapbind;
}


$test = ad_ldap_auth();

if ($test)
{
    $_SESSION['name'] = $username;
    $_SESSION['expire'] = (time() + 3600);

    $filter = "sAMAccountName=$username";
    //$filter = "sAMAccountName=*";
    $attrib = array("memberOf");
    $attrsonly = 0;
    $sizelimit = 3;
    $timelimit = 10;

    ldap_get_option($ds, LDAP_OPT_SIZELIMIT, $optVal); // returns 0
    echo "size_limit =". $optVal ."<br>";
   
    ldap_set_option($ds, LDAP_OPT_SIZELIMIT, 5); // returns TRUE
    ldap_get_option($ds, LDAP_OPT_SIZELIMIT, $optVal); // returns 0
    echo "size_limit =". $optVal ."<br>";

    $sr = ldap_search($ds, $searchbase, $filter, $attrib);

    for ($entry=ldap_first_entry($ds,$sr); $entry!=false; $entry=ldap_next_entry($ds,$entry))
    {
        $values = ldap_get_values($ds, $entry, "memberOf");

        echo $values["count"] . " groups for this entry.<br />";

        for ($i=0; $i < $values["count"]; $i++)
        {
                echo $values[$i] . "<br />";
            if (strcmp($values[$i],$group) == 0)
            {
                echo "User is member.<br>";
            }
        }
    }
} else {
    echo "LDAP Failed.";
}

ldap_unbind($ds);

?>

After finishing my script it was time to move on to DocMgr. I turned on DEBUG in the config.php and began placing debug output throughout DogMgr so that I could see the flow of function calls as the program ran.

if (DEBUG)
   print("authroized == 1<br>");

I wrote down the flow of some key authentication functions and then began reading them to understand what each did and where I should start hacking. Then I went and broke the old ldap includes into open-ldap and ad-ldap includes and set up the ifdefs in the different files. Doing this I was able to still use my OpenLDAP authentication and could hack on ad-ldap.

I set up the ad-ldap-config.php file by comparing the output of ldapsearch in ldif format on both OpenLDAP and AD. We made some changes to the AD population script and moved forward using employeeNumber instead of uidNumber.

After I got the correct entries in AD and OpenLDAP figured out it was just a matter of hacking my test script into the correct functions and testing. There were a couple of errors that I needed to fix. The first was caused by searching by for sAMAccountName=* in the else portion of the ldap_account_search function of docmgr/include/ad_ldap.inc.php.

This was the first error and fix:

Warning: ldap_search(): Partial search results returned: Sizelimit exceeded in /var/www/docmgr/include/ad_ldap.inc.php on line 697

ldap_set_option($ds, LDAP_OPT_SIZELIMIT, 5);

This was the second error and fix:

Warning: ldap_search(): Search: Operations error in /var/www/docmgr/include/ad_ldap.inc.php

ldap_set_option($ldap, LDAP_OPT_REFERRALS, 0);

Once I finished making DocMgr authenticate to AD I wanted to make the integration into AD complete so that I did not have to manually add or remove users within the web interface. The following is the script that does that.

It is run every morning from cron.

# crontab -l
# m h  dom mon dow   command
# Add/remove DocMgr users according to AD group DocMgr.
16 4 * * * /usr/local/bin/ad-docmgr.pl

And the code for allowing access to DocMgr. I truncate the table and just add the users each time.

#!/usr/bin/perl

# 2010-04-13  Jud Bishop
# This script queries active directory for every member of the DocMgr group
# and adds them to the auth_accoutperm table, which gives them the ability
# to login to DocMgr after authenicationg to AD.

use strict;
use DBI;
use Net::LDAP;
use Net::LDAP::LDIF;
use Net::LDAP::Entry;

# Globals
# 1 == debug
# 0 == quiet
my $debug = 1;

sub connectLDAP {
    if($debug){print "connectLDAP\n";}

    my $ldap= Net::LDAP->new("server.circus.org", port =>"389", version =>"3" )
        or die $!;
   
    my $result = $ldap->bind( "CN=SpecialUser,DC=CIRCUS,DC=ORG",
                password => "PassWord" );
        die $result->error() if $result->code();

    return $ldap;
}

sub queryORG {

    my $mesg;
    my ($ldap, $dbh) = @_;

    if($debug){print "queryORG\n";}

    $mesg = $ldap->search(
        base => "OU=Users,DC=CIRCUS,DC=ORG",
        scope => "sub",
        filter => "(memberOf = CN=DocMgr,OU=Groups,DC=CIRCUS,DC=ORG)");
   
    if ( $mesg->count() > 0 )
    {
            my $max = $mesg->count;
                for ( my $j = 0 ; $j < $max ; $j++ )
                {
            my $entry = $mesg->entry ( $j );
            foreach my $t ( $entry->get_value( "employeeNumber" ))
                    {
                if($debug){print "employeeNumber $t\n";}
                insertQuery($dbh, $t);
            }
        }
    }
}

sub unbindLDAP {
    if($debug){print "unbindLDAP\n";}
    my $ldap = shift;
    $ldap->unbind();

}

sub connectPostgres {
    if($debug){print "connectPostgres\n";}

    my $dbh = DBI->connect('DBI:Pg:dbname=docmgr;host=127.0.0.1', 'DocMgrUserName', 'PassWord' , {
        PrintError => 0,
        RaiseError => 1 # Report errors via die(), needs less error handling because
                # DBI will check for us and die() automagically.
    });

    if ($dbh) {
        # From above we don't have to check, this is just for debugging.
        if($debug){print "connected\n";}
        return $dbh;
    }
}

# Truncate the table so we don't have to worry about who is there and who isn't.
sub deleteQuery {
    if($debug){print "deleteQuery\n";}

    my $dbh = shift;
    my $sth = $dbh->prepare("TRUNCATE auth_accountperm");
    $sth->execute;
}

sub insertQuery {
    if($debug){print "insertQuery\n";}

    my ($dbh, $employeeNumber) = @_;
   
    my $sth = $dbh->prepare("INSERT INTO auth_accountperm VALUES ($employeeNumber, 1, true)");

    $sth->execute;
    if($debug){print "INSERT INTO auth_accountperm VALUES ($employeeNumber, 1, true)\n"};
}

sub disconnectPostgres {
    if($debug){print "disconnectPostgres\n";}
   
    my $dbh = shift;
    $dbh->disconnect();
}

my $ldap = connectLDAP();

my $dbh = connectPostgres();

deleteQuery($dbh);

queryORG($ldap,$dbh);

disconnectPostgres($dbh);

unbindLDAP ($ldap);

# end main
Categories: Code, Linux Tags:

Linux DBI::ODBC AS400

April 19th, 2010 jud No comments

At the Circus we are removing the last vestige of our OpenLDAP implementation and moving it to Active Directory. As a result I’m going to document some of the odd scripts I have written as glue to help keep things running. Some of the scripts are too long to document and will just be kept in my script library, while others like the ones below, are generic enough that they might help someone else.

This is some documentation for how to query an AS400 from Linux. When I did this project I actually talked to one of the developers of the iSeries Access programs for help. He ended up sending me a snapshot rpm to get it all working.

Here are the configuration files.

#cat /etc/odbcinst.ini

[iSeries Access ODBC Driver]
Description     = iSeries Access for Linux ODBC Driver
Driver          = /opt/ibm/iSeriesAccess/lib/libcwbodbc.so
Setup           = /opt/ibm/iSeriesAccess/lib/libcwbodbcs.so
Threading       = 2
DontDLClose     = 1
UsageCount      = 1
#cat /etc/odbc.ini

[AS400]
Description     = iSeries Access ODBC Driver
Driver          = iSeries Access ODBC Driver
System          = as400.circus.org
UserID          = UserName
Password        = Password
Naming          = 0
DefaultLibraries= QGPL
Database        =
ConnectionType  = 0
CommitMode      = 2
ExtendedDynamic = 1
DefaultPkgLibrary = QGPL
DefaultPackage   = A/DEFAULT(IBM),2,0,1,0,512
AllowDataCompression = 1
LibraryView     = 0
AllowUnsupportedChar = 0
ForceTranslation= 0
Trace           = 1
DSN             = AS400

This is a simple script to list the drivers available on the system. This does not look like a script I wrote so I am reticent to take ownership.

#!/usr/bin/perl

use DBI;

my @drivers = DBI->available_drivers();

die "No drivers found. \n" unless @drivers;

# list the data sources
foreach my $driver ( @drivers )
{
    print "Driver: $driver\n";
    my @dataSources = DBI->data_sources( $driver );
    foreach my $dataSource ( @dataSources )
    {
        print "\tData Source: $dataSource\n";
    }
    print "\n";
}

exit

And the resulting output from the script above.

# perl drivers.pl
Driver: DBM
    Data Source: DBI:DBM:f_dir=PHPDemo
    Data Source: DBI:DBM:f_dir=SQL-LDAP
    Data Source: DBI:DBM:f_dir=.
    Data Source: DBI:DBM:f_dir=DBD-ODBC-1.09
    Data Source: DBI:DBM:f_dir=unixODBC-2.2.8
    Data Source: DBI:DBM:f_dir=DBI-1.42

Driver: ExampleP
    Data Source: dbi:ExampleP:dir=.

Driver: File
    Data Source: DBI:File:f_dir=PHPDemo
    Data Source: DBI:File:f_dir=SQL-LDAP
    Data Source: DBI:File:f_dir=.
    Data Source: DBI:File:f_dir=DBD-ODBC-1.09
    Data Source: DBI:File:f_dir=unixODBC-2.2.8
    Data Source: DBI:File:f_dir=DBI-1.42

Driver: ODBC
    Data Source: DBI:ODBC:AS400

This is a simple script to query an AS400. You will have to get with your AS400 administrator because the the query is actually an odd series of fields, at least it was for our organization. I just wrote this to learn how to query the AS400, it was never used in production.

#!/usr/bin/perl

# 2005-09-12 Jud Bishop
# Released under the New BSD License.

# This script uses the DBI::ODBC driver to interact with an AS 400
# database.  Make sure that you use a system DSN rather than a
# user defined one or you will have trouble later.

use strict;
use DBI;

my $UID="UserName";
my $PW="Password";
my $DSN="AS400";


print "connect\n";
my $dbh = DBI->connect ( "dbi:ODBC:$DSN", $UID, $PW, {
#   LongTruncOk => 1
    PrintError => 1
})
    or die "connection failed to database $DBI::errstr\n";

## Set up tracing
print "trace\n";
unlink 'trace.log' if -e 'trace.log';
DBI->trace( 2, 'trace.log' );

## Prepare the SQL statement for execution
print "prepare\n";
my @t = localtime(time);
my $sql_st = $dbh->prepare( " SELECT ALL X FROM Y WHERE I = J ORDER BY Z ");

print "execute\n";
$sql_st->execute()
    or die "Cannot execute SQL statement $DBI::errstr\n";

my @row;
while ( @row = $sql_st->fetchrow_array() )
{
    print "@row\n";
}
warn "Data fetch terminated by error $DBI::errstr\n"
    if $DBI::err;

$dbh->disconnect or warn "disconnection failed $DBI::errstr\n";

exit
Categories: Code, Linux Tags:

What I Learned Today

March 26th, 2010 jud No comments

One of my coworkers solved a problem today that I should have been able to solve. I got pulled into a project that uses a closed system with it’s own programming language that I had never seen nor programmed. My coworker was trying to figure out why his syntax was not working. So we cranked up the logging in Warn.log, Error.log and Fatal.log. We quickly figured out why it was failing, the program was looking for a file that did not exist and he was not catching that error, there is no catch statement.

So I began trying to write an IF/ELSE statement that would check for the error condition. I was given a .pdf of the manual for the system and just started trying different functions that made sense to me. I wrote a nice little one liner in the SYSTEM command that returned TRUE or FALSE if the FILE environment variable existed.

The problem was that we could not set the environment variable with a variable from the program. It has to be a string literal.

From the Trace.log:

      LET sys = EXPORT("FILE=" & $format)

      // Variable "Part01::sys" is a String
      // [  1] = ""

      LET sys = SYSTEM("if [ -f /usr/program/fmt/$FILE ]; then  echo "TRUE"; else echo "FALSE"; fi")

      // Variable "Part01::sys" is a String
      // [  1] = "FALSE"

      IF sys = "TRUE" THEN

        // Test result is FALSE - skipping...

      END IF

We tried a number of different variations on that theme, imported and exported variables to a subshell, tried different return values from the exit command. We got the system to turn circles but what we really wanted was squares.

Finally my coworker called and said, why don’t we just invert the logic. If it falls through to the end of the IF/ELSE statements it must be in the format we want. Let me take a step back and say this was his code, he should have been the one to figure it out. I believe I helped him figure out what his main problem was, which helped him find the solution. I’m not trying to minimize his work, nor am I trying to minimize my input into the solution.

The moral of the story is that I should have taken a higher level look at the problem to understand the solution. I am quick to believe that I can code my way out of any problem, regardless whether or not I have ever seen the language. What I really needed to do was step back and take a higher level view of the logic, rather than dive into syntax and functions.

Categories: Code, Linux, Musings Tags:

Xenix

March 21st, 2010 jud No comments

Today on Slashdot someone had a question about getting data off of an old Xenix server. A few years ago I did a consulting job for a customer who had an old Xenix server with no ethernet card that needed to get some data for the State Police. The server ran an old database that kept track of training records and they needed that information in order to “webify” the process.

I wish I could remember the ins and outs of the problem. I gave the customer a write up of how I did it and the problems I had getting the usr directory off of the system. Unfortunately I can’t find the report.

Regardless, it was an interesting project, here is the script from my library that eventually worked:

#!/bin/sh

for I in bin boot dos etc lib oa.files once shlib tmp u unit57 unit58 unit59 usr xenix
do
    echo "tar -cf - $I|uuencode -"
    tar -cf - $I|uuencode -
done

tar -cf - `find /usr -type f -print` >2/usr.err|uuencode -


for I in `cat usr.txt`
do
    tar -cf - $I |uuencode -
done
Categories: Code Tags:

DHCP NAT

March 8th, 2010 jud No comments

I have debated about whether or not to post this project. I wanted this to have a sqllite backend and run on the Cisco AXP but never had the time to go back and hack more on it. I think it could have won the AXP hacking contest, but I didn’t even know about the contest at the time.

I also want to say that this project is not for the inexperienced Linux admin. This project involves a solid understanding of DHCP, DNS and iptables. I have rolled a .tar file of all of the scripts and directories needed to make this happen, but I’ve said it before and I will say it again, you got it off the web, your mileage may vary. Here is the file.

History
The basic idea here is this, we have ~375 printers that were all translated using static NAT, but most of our infrastructure runs off DHCP. So when someone moves an office they unplug their VOIP phone and computer and take it to their new office, but the printer involves manual intervention, not good.

There is one thing to say about a lazy network administrator, it usually means problems like this get fixed one way or the other. I wanted a way to be able to move printers within the “enterprise” and have their NATed address follow them. This is my solution.

NAT

A picture is worth a thousand words so here is a simple diagram.

Diagram of WAN.

The 192.168.24.0/24 subnet is shared between the Circus and our Electronic Medical Records vendor, EMR for short. However, the Circus is partially switched and partially routed on campus so we have to NAT IP addresses on the shared subnet into IP addresses in the enterprise. So this diagram shows a translate server with eth0 at 192.168.24.2 and eth1 as 192.168.100.58, eth0:1 is a NAT’ed IP address for a printer somewhere in the enterprise.

Let me explain this again, this time with commands:
WAN address that is shared with another company, in our case a printer:
wan printer — 192.168.24.100 — IP address EMR uses to print.
translate eth0:1 — 192.168.24.100 — IP address EMR uses to print, actually a virtual interface on translate server.
lan printer — 192.168.10.100 — IP address of printer on campus.
translate eth0 — 192.168.24.2 — WAN leg of translate server.
translate eth1 — 192.169.100.2 — LAN leg of translate server.

The wanprinter ip address 192.168.24.100 is a virtual ip address on a linux
server, translate. It would be brought up from the command line like this:

# /sbin/ifconfig eth0:1 192.168.24.100 netmask 255.255.252.0 up

It is then natted from 192.168.24.100 to it’s local ip address:

# /sbin/iptables -t nat -D PREROUTING --destination 192.168.24.100 -j DNAT --to 192.168.10.100

The other rule that is added only once is to make it look like all traffic
is coming from the translate server:

# /sbin/iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to 192.168.100.2

rsyslog setup

You need to download rsyslog. This is because you need reliable remote logging and the normal syslog server (sysklogd) does not handle it. I am using librelp version 0.1.1 and rsyslog version 3.20.2. The problem that relp/rsyslog fixes is that standard syslog does not reliably log remotely. So what ends up happening is that dhcp log messages will be dropped, eventually it will get you because an address update message will be dropped.

The DHCP servers in our organization run Red Hat Enterprise Linux and the translate servers run Ubuntu LTS. Please check the README or INSTALL document from rsyslog for installation on your platform. The dhcp servers will be remote logging dhcp transactions to the translate servers so that a logwatch plugin can monitor address changes.

Just download these and read the doc/install.html.

wget http://download.rsyslog.com/librelp/librelp-0.1.1.tar.gz
./configure
make
make install
wget http://www.rsyslog.com/Downloads-req-getit-lid-141.phtml
./configure --enable-relp --enable-mail --enable-zlib --with-gnu-ld
make
make install

To compile on RedHat I had to use –with-gnu-ld after editing etc/ld.so.conf:

# cat /etc/ld.so.conf.d/librelp.conf
/usr/local/lib

I also had to add an environmental variable:

# export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

In order to get RHEL standard scripts to work you have to add a symlink to
/usr/local/sbin/rsyslogd from /usr/sbin/rsyslogd:

# ln -s /usr/local/sbin/rsyslogd /usr/sbin/rsyslogd

Make sure to edit the sysklogd init script or copy in a new one. Also make sure you get rsyslogd hacked into init. The easiest way is to run this command after putting my rewritten RH init script on the box. The rewritten script is in the tar file from above as daemon/rh-etc-init.d-rsyslogd.

# chkconfig syslogd off
cp rh-etc-init.d-rsyslogd /etc/init.d/rsyslogd
# chkconfig rsyslogd on
# chkconfig --list | grep sys

If you didn’t use my init script, edit /etc/cron.daily/sysklog script to make sure rsyslog is notified about rotation.

Also edit the logrotate configuration at /etc/logrotate.conf:

# cat /etc/logrotate.d/syslog
/var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron {
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Please see my dhcp-rsyslog.conf and translate.rsyslog.conf for the rsyslog
configurations for each server. If it’s not self explanatory copy the file daemon/dhcp-rsyslog.conf to the dhcp server /etc/rsyslog.conf. You will have to change the IP address of the translate server from 192.168.100.2 to whatever you are using.

DHCP Servers

Make these changes in NS1 and NS2, or if you have a setup similar to me,
just copy /etc/dhcpd/dhcpd.ad.master.conf from NS1 to NS2 and restart both DHCP servers.

In /etc/dhcpd/dhcpd.master.conf add these lines:

# Where to log stuff, this sets up remote logging for
# syslogscand and DHCP-Nat.
log-facility local3;

Restart dhcpd.

Translate Server

Our translate server is a Dell 1U 2600 or something like that. It presently
has ~375 virtual interfaces translating printers from the WAN subnet to our
internal addresses.

eth0 == External or WAN address
inet addr:172.22.25.193 Bcast:172.22.27.255 Mask:255.255.252.0

eth1 == Internal or LAN address
inet addr:172.22.100.52 Bcast:172.22.100.255 Mask:255.255.255.0

The internal IP address is only needed for the postrouting rule.

Install syslogscand on this server through CPAN and then copy DHCP.pm to the correct location. If all else fails just head DHCP.pm and see where it tells you to copy it. Then copy the file daemon/etc-syslogscand.conf to /etc/syslogscand.conf.

Finally move rc.local.firewall to a system directory and add it to rc.local
to fire off upon server start.

syslogscand Plugin Configuration

I wrote a syslogscand plugin, DHCP.pm, that watches the output of dhcp and any
time a mac address that it watches is given an IP address it queries through
dns lookups whether or not the mac address has gotten a new IP address. If it
has gotten a new address, the daemon updates the /etc/network/dhcp-nat file
as well as the /etc/network/interfaces file. It then updates iptables with the
new NAT translation.

Below is a dhcp-nat file. It is comma delimited with the first column being
the mac address of the printer, the second column is the internal LAN address,
the third column is the external WAN address and the forth column is the
hostname.

# cat /etc/network/dhcp-nat
00:0e:7f:3c:88:bc,172.22.81.110,172.22.25.252,printest252
00:1f:f3:5a:b2:6a,172.22.81.116,172.22.25.249,trek
00:19:b9:45:04:30,172.22.81.254,172.22.25.251,litespeed
00:11:0a:f6:3f:93,172.22.81.254,172.22.25.250,printest250

I have also added information to the /etc/network/interfaces file,
notice in the following stanza that I have added comments for the
mac and translate(d) address.

# head -6 /etc/network/interfaces
#mac 00:1f:f3:5a:b2:6a
#translate 172.22.81.116
auto eth0:8
iface eth0:8 inet static
address 172.22.25.249
netmask 255.255.252.0

Web Interface
There is also a web interface into this so that PC technicians can interact
with the system easily. In order to make that magic happen I added the
following line to /etc/sudoers:

www-data translate1 = NOPASSWD: /usr/bin/syslogscand, /sbin/iptables, /sbin/ifconfig

It allows the user www-data that apache runs under to restart syslogscand as
well as manipulate iptables and interfaces.

Copy all of the files in the cgi-bin directory to your local cgi-bin directory.

Copy all of the files in the html directory to the web server root.

You may have to tweak directory locations in the scripts.

Categories: Code, Linux Tags:

Converting AIX print queues to Linux

February 25th, 2010 jud No comments

I spent the last week working on a project to convert all of the forms printing at the Circus from an AIX server to a Linux server. Because the version we are stuck on is not officially supported by the vendor I had to do some reverse engineering to figure out how things work, this article describes some of the scripts I used to understand what the software was doing and how to make it work the way we needed.

The funny thing about any printer project I get into, usage seems to explode. Yesterday we had a water cooler meeting about how to print from a completely different EMR system. I’m not sure we will do it because we need to know patient location but the printing part is easy. Who knew a paperless organization prints so much.

We have about 500 printers on the network and less than 314 forms, my guess would peg us closer to 150 forms, but when I go through the forms directory I get 314.

# ls -1 | cut -d _ -f 1,2 | sort -u | wc
    314     314    3293

Regardless, the number of forms is not the issue. I changed everything I needed programmatically as you will see below. All of the scripts in the post can be downloaded here.

First I needed to get a connection from one of our electronic medical records applications to the new Linux forms print server. I wrote a short script that shows me the arguments passed to the printer as well as the standard input. This allowed me to troubleshoot what was being passed from one server to another and also gave me insight into how the whole process worked.

#!/usr/bin/perl -w
use strict;

# 2010-02-17  Jud Bishop
# Quick hack to see what I am being sent from the port.
# Released under the GNU GPLv2

my $file = "/tmp/print-test.txt";

sleep 5;
unlink $file;

open (NEWFILE, ">$file" ) or die("Error: can't open $file\n$!");

        print NEWFILE "Args:\n";
        for (my $i = 0; $i <= $#ARGV; $i++) {
                print NEWFILE "argv[$i] == $ARGV[$i]\n";
        }

        print NEWFILE "Data:\n";
        while (<STDIN>) {
                print NEWFILE $_;
        }

close NEWFILE or die("Error: can't close $file\n$!");

Once I got the two servers communicating properly it was time to troubleshoot some configuration files that were copied from the AIX server to Linux. I ended up writing the next short script to change some application configuration files that were originally made to work on AIX. This changes the print command from the AIX qprt command to Linux lpd.

# 2010-02-17  Jud Bishop
# This script changes all of the printer configuration channels
# from AIX specific qprt to Linux specific lp commands.
# Released uner the GNU GPLv2

for I in `ls -1`
do
    sed 's/lp -d/lp -d/g' $I > tmp
    if [ $? -eq 0 ]
    then
        mv -f tmp $I
    else
        echo $I "did not complete"
        exit
    fi
done

Now it was time to write a short script to convert the /etc/qconfig printer configuration file from AIX and add the printers on the Linux server. So I wanted to test adding and deleting a printer from cups on the command line. I thought the following command would work.

# lpadmin -p misp1 -E -v socket://misp1.circus.org -m laserjet
# lpadmin: Unable to copy PPD file!

If all else fails read the man page, or in this case the manual on the web. It’s pretty cryptic but they tell you that the -m model has to be from the model directory. Where is the model directory?

# find / -name model
...
/usr/share/cups/model
# ls -1 /usr/share/cups/model/
deskjet2.ppd.gz
epson9.ppd.gz  
laserjet.ppd.gz

So we want the model from /usr/share/cups/model. The following command works:

# lpadmin -p misp1 -E -v socket://misp1.circus.org -m laserjet.ppd.gz

Now it’s time to convert from qconfig to cups. I used the following script to read in qconfig and create the printers. I had to add the sleep because the script got ahead of the lpadmin command. Once I added the sleep it worked and ta-da, 500 printers were created. I realize that printers are added and deleted from the AIX server daily so I had to make sure it would add and delete printers so that we will get them all on go-live day. It’s a rudimentary implementation but it works.

Notes on this script, it was easier than I expected, hence I a have hash instead of just working from the original array. It’s one of those things that I’m not going to go back and clean up for a simple script, sorry.

#!/usr/bin/perl

# 2010-02-18  Jud Bishop
# This script reads in the qconfig file of an AIX server and
# converts it to a Linux based cups file.  
# I am expecting this to be a quick hack...
# Released under the GNU GPLv2

# /etc/qconfig is the AIX printer configuration file.
#labp4:
#        device = @hpjd007
#        up = TRUE
#        host = hpjd007.circus.org
#        s_statfilter = /usr/lib/lpd/bsdshort
#        l_statfilter = /usr/lib/lpd/bsdlong
#        rq = PORT1
#@hpjd007:
#        backend = /usr/lib/lpd/rembak -T 30
open (FILE,"<etc-qconfig") or die "Error: can't open file $! \n";
        @aix = <FILE>;
close FILE or die "Error: can't close file $! \n";

my %table;
my $j = 0;
my ($que, $host, $port, $trash);
for (my $i = 0; $i <= $#aix; $i++) {
        if ( $j == 0 ){
                ($que, $trash) = split (/:/, $aix[$i]);
        } elsif ( $j == 3) {
                chomp($aix[$i]);
                $aix[$i] =~ s/ //g;
                ($trash, $host) = split (/=/, $aix[$i]);
                if ( $host !~ /circus.org/ )
                {
                        $host = sprintf ("%s.circus.org", $host);
                }
        } elsif ( $j == 6) {
                chomp($aix[$i]);
                ($trash, $port) = split (/=/, $aix[$i]);
        }  
        if ( $j == 6 ) {
                $table{$que} = {'host'=>$host, 'port'=>$port};
        }
        $j++;
        if ( $j == 9 ) {
                $j = 0;
        }
}

# The command to add a printer.
# system  lpadmin -p printer -E -v socket://host.circus.org -m laserjet.ppd.gz
# The command to remove a printer, got to be able to back it out.
# lpadmin -x printer
foreach $key (keys(%table)) {
        print "$key $table{$key}->{host} $table{$key}->{port}\n";

        my $socket = sprintf ("socket://%s", $table{$key}->{host});

        # Swap these two commands if you need to delete.        
        #my @args = ("lpadmin", "-x", "$key");
        my @args = ("lpadmin", "-p", "$key", "-E", "-v", "$socket", "-m", "laserjet.ppd.gz" );

        system(@args) == 0
        or die "system @args failed: $?";
        sleep 1;
}
Categories: Code, Linux Tags:

Parsing Barracuda Log Files

February 17th, 2010 jud No comments

We have a Barracuda web filter here at the Circus. In general we are pleased with its performance, however, as our internet usage has climbed the days of history has declined. The inverse relationship is due to a “ring buffer” of 250,000 entries in the history log. There are times when we have a few hours of history and that doesn’t sit well when a manager wants to see the browsing history for a user or a pc. So we turned on syslog logging to a remote logging server and evaluated the log parsing packages that mentioned they parsed Barracuda logs. Let me give you a hint, there are not many.

My first hack was just to see what was going on and give a rudimentary understanding, it can be downloaded here.

!#~/bin/bash
grep $1 /var/log/barracuda.log | grep http_scan | awk '{print $28}' | sort | uniq -c | sort -n

Which gives output like this:

     63 autotrader.com
     72 google.com
     78 edmunds.com
     81 ad.doubleclick.net
     82 charter.com
    121 dealer.com
    138 alagasco.com
    194 synacor.com
    334 charter.net

The basis of that script gave me more understanding to write the following script which can be downloaded here.

#!/usr/bin/perl -w

use Getopt::Long;
use Number::Format;
use Tie::IxHash;


###############################################
# 2010-02-05 Jud Bishop
# This script parses /var/log/barracuda.log.
# Released under the GNU GPLv2.
###############################################
# Options that can be passed in:
# -u username
# -s source ip address
# -d destination url
# -c category
# -days number of days
# -pc the pcname
###############################################
# The format of a log file message.
# Used this document:
# http://www.barracudanetworks.ca/download/barracuda-web-filter-syslog.pdf
# And this one liner:
# tail -50 /var/log/barracuda.log | awk '{print $28}' && tail -1 /var/log/barracuda.log
#
# 0.  month
# 1.  day
# 2.  time
# 3.  barracuda_ip
# 4.  http_scan[process_id]
# 5.  md5sum
# 6.  number1
# 7.  source_ip
# 8.  destination_ip
# 9. content_type
# 10. source_ip
# 11. destination_address/URL
# 12. data_size
# 13. BYF
# 14. action (ALLOWED, BLOCKED, DETECTED)
# 15. reason (CLEAN, VIRUS, SPYWARE)
# 16. format_ver == 2 (Version of the policy engine output.)
# 17. match_flag (Whether an existing policy matched the traffic: 1=Yes, 0=No.)
# 18. tq_flag (Time qualified flag; 1=Yes 0=No.)
# 19. action_type (The documentation for this flag is incorrect.)
# 20. source_type (0=Any source, 1=group, 2=ipv4addr, 3=login, 4=auth_user, 5=min_score)
# 21. src_detail Detail related to matched source or "(-)" if not.
# 22. dest_detail If there is a destination match, what type (0=any, 1=particular category, 2=any category
#                 3=domain, 4=mimetype, 5=spyware, 6=uri_path_regex, 7=uri_regex, 8=application)
# 23. dest_detail Matched category or "(-)" if not matched.
# 24. spyware (If it is spyware, 0=allow, not spyware, 1=block, 2=infection.)
# 25. spyware_id (Name of the spyware if matched, if not "-".)
# 26. infection_weight Weight of the infection, mostly 0.
# 27. matched_part Part of the rule theat matched.
# 28. matched_category Comma delimited category name that matched traffic.
# 29. user_name Username, ([ANON], [ldap0:jud], [username:jud])

###############################################
# Variables you can change.
###############################################
my $debug = 0;
my $log_file = "/var/log/barracuda.log";
#my $log_file = "/var/log/barracuda.test";

###############################################
# Variables you should NOT change.
###############################################
my $arg_username = 0;
my $arg_pc = 0;
my $arg_source_ip = 0;
my $arg_dest_url = 0;
my $arg_category = 0;
my $arg_days = 0;
my $arg_help = 0;
my %table; # holds all the data for each session.
tie %table, "Tie::IxHash";
my %categories; # holds all of the categories this person went to.
my $data_sum = 0; # holds the total of all data for a user.
my $session_sum = 0; # hold the number of sessions for a user.
# The names of these variable are so that I don't have to keep looking
# above to figure out what name is what item in the array.
my $user = 29;
my $md5sum = 5;
my $month = 0;
my $day = 1;
my $time = 2;
my $source_ip = 7;
my $destination_ip = 8;
my $url = 11;
my $data_size = 12;
my $action = 14;
my $part = 27;
my $category = 28;

# Reads the options passed in.
sub get_options {
        GetOptions(
                'help|?|h!' => \$arg_help,
                'u=s' => \$arg_username,
                's=s' => \$arg_source_ip,
                'd=s' => \$arg_dest_url,
                'c=s' => \$arg_category,
                'days=i' => \$arg_days,
                'pc=s' => \$arg_pc);

        if ($debug)
        {
                print "username == $arg_username\n";
                print "source_ip == $arg_source_ip\n";
                print "pc == $arg_pc\n";
                print "dest_url == $arg_dest_url\n";
                print "category == $arg_category\n";
                print "days == $arg_days\n";
                print "help == $arg_help\n";
        }

        if ($arg_help)
        {
                print "usage: user-report.pl -days number [-u usernname] [-s source-ip] [-d destination-url] ";
                print "[-c category] [-pc pc_name] [--help|-?]\n";
                exit;
        }
}

# Parses the logs.
# Days equals log file days, makes it easy.
sub parse_logs {

        my ($search_field, $search_equals, $days) = @_;

        if ($debug){
                print "--------------------\n";
                print "parse_logs\n";
                print "search_field == $search_field\n";
                print "search_equals == $search_equals\n";
                print "days == $days\n";
        }

        # Loop through the log files based on number of days:
        # 0 == today
        # 1 == barracuda.log.1 one day past...
        # This is not formatted correctly because I added it as a retrofit.
        for (my $i = 0; $i <= $days; $i++)
        {
        if ($i == 0)
        {
                if ($debug) {print "open $log_file\n"};
                open (FILE, $log_file) or die "Error: can't open file\n $! \n";
        } else {
                if ($debug) {print "open $log_file.$days\n"};
                my $file = sprintf ("%s.%s", $log_file, $days);
                open (FILE, $file) or die "Error: can't open file\n $! \n";
        }
        while (<FILE>)
        {
                chomp;
                # Makes split work like awk, don't believe the man page.
                my (@log) = split /\s+/;
                # The next check is to catch the following type messages.
                # Feb  9 06:51:09 last message repeated 8 times
                if (defined ($log[$search_field]) and ($log[$search_field] eq $search_equals))
                {
                        if ($debug){
                                print "$log[$search_field] $search_equals\n";
                                for (my $i = 0; $i <= $#log; $i++)
                                {
                                        print "log[$i] == $log[$i]\n ";
                                }
                        }
                        # Each session gets a different md5sum, which is why it is the key in the table.
                        if( not exists $table{$log[$md5sum]}){
                                $table{$log[$md5sum]} = {'user'=>$log[$user], 'month'=>$log[$month], 'day'=>$log[$day], 'time'=>$log[$time], 'source_ip'=>$log[$source_ip], 'destination_ip'=>$log[$destination_ip], 'url'=>$log[$url], 'data_size'=>$log[$data_size], 'action'=>$log[$action], 'category'=>$log[$category], 'total_data'=>$log[$data_size], 'session_count'=>1 };

                                $data_sum += $log[$data_size];
                                $session_sum += 1;
                                if ( not exists $categories{$log[$category]} )
                                {
                                        $categories{$log[$category]} = 1;
                                } else {
                                        $categories{$log[$category]} += 1;
                                }

                                if($debug){
                                        print "does not exist\n";
                                        print "log user = $log[$user] \n";
                                        print "table user = $table{$log[$md5sum]}->{user} \n";
                                        print "sessions = $session_sum\n";
                                        print "bandwidth = $data_sum\n";
                                }
                        } else {
                                $table{$log[$md5sum]}->{total_data} += $log[$data_size];
                                $data_sum += $log[$data_size];
                                if($debug){
                                        print "exists \n";
                                        print "bandwidth = $data_sum\n";
                                }
                           }
                    }      
        }
        close FILE or die "Error: can't close file\n $! \n";
        }
}

sub print_report {

        if ($debug) {print "print_report\n";}
        if ($arg_username)
        {
                print "Useage report for: $arg_username\n";
        } elsif ($arg_pc) {
                print "Useage report for: $arg_pc\n";
        } elsif ($arg_source_ip) {
                print "Useage report for: $arg_source_ip\n";
        }
        print "Number of sessions: $session_sum\n";

        my $x = new Number::Format;
        $formatted = $x->format_bytes($data_sum);
        print "Total bandwidth consumed: $formatted\n\n";

        foreach my $category (sort (keys(%categories)))
        {
                printf "%s\n", uc($category);

                foreach $key (keys(%table))
                {
                        if ( $table{$key}->{category} eq $category)
                        {
                                my $url = substr($table{$key}->{url}, 0, 35);
                                print "$table{$key}->{month} $table{$key}->{day} $table{$key}->{time} $url $table{$key}->{action}\n";
                        }
                }
                print "\n";
        }
}

###############################################
# main
###############################################
my $search_field;
my $search_equals;

get_options();

if ($arg_username or $arg_pc) {
        $search_field = $user;
        if ($arg_username eq "ANON"){
                $search_equals = $arg_username;
        } else {
                $search_equals = sprintf("[ldap0:%s]", $arg_username);
        }
} elsif ($arg_source_ip) {
        $search_field = $source_ip;
        $search_equals = $arg_source_ip;
} elsif ($arg_dest_url) {
        $search_field = $url;
        $search_equals = $arg_dest_url;
} elsif ($arg_category) {
        $search_field = $category;
        $search_equals = $arg_category;
}
if ($debug) {print "search_field == $search_field\n"}

parse_logs($search_field, $search_equals, $arg_days);

print_report();

This script produces output like the following:

Usage report for: circus-user
Number of sessions: 401
Total bandwidth consumed: 19.39M

ADVERTISEMENT-POP-UPS,GAME-MEDIA,CUSTOM-2
Feb 17 15:44:17 http://games.mochiads.com/c/p/the-r ALLOWED

AUCTIONS
Feb 17 17:08:21 http://rover.ebay.com/ar/1/56033/1? ALLOWED

AUCTIONS,MOTOR-VEHICLES,CUSTOM-1
Feb 17 17:13:44 http://edmunds.autotrader.com/js/jq ALLOWED
Feb 17 17:13:45 http://edmunds.autotrader.com/inc/g ALLOWED
Feb 17 17:13:46 http://edmunds.autotrader.com/inc/j ALLOWED
Feb 17 17:13:46 http://edmunds.autotrader.com/inc/j ALLOWED
Feb 17 17:13:47 http://edmunds.autotrader.com/dwr/i ALLOWED
Feb 17 17:13:50 http://edmunds.autotrader.com/no_ca ALLOWED

BUSINESS
Feb 17 15:09:08 http://www.statcounter.com/counter/ ALLOWED
Feb 17 15:10:15 http://www.alagasco.com/fw/_css/fle ALLOWED
Feb 17 15:10:16 http://www.alagasco.com/scripts/jFa ALLOWED
Feb 17 15:10:17 http://www.alagasco.com/fw/_js/flex ALLOWED
Categories: Code, Linux Tags: