Archive

Archive for the ‘Code’ Category

DocMgr with AD Authentication

April 23rd, 2010 No comments

Over the past couple of days I hacked an older version of DocMgr to authenticate users to Active Directory. I thought it might be interesting for some readers to see my thought process, hence this post.

All of the scripts in this post along with my test directory and my changes to DocMgr can be downloaded here.

First I started out with a simple php form for authentication. The input portion is in the download, this is the processing side. Some interesting notes about this script. At first I was using the full distinguished name (DN) of the user but that doesn’t scale. It was not until I downloaded adLDAP that I saw you could just use username@domain.org to authenticate. I don’t know why I never tried, but that stymied me.

The reason this script looks for groups the user belongs to is because originally I figured I would limit access by group. The problem is that DocMgr has it’s own permission system and you must be in the database in order to be able to login. The script at the bottom is how I dealt with that limitation.

The ldpad_set_option and ldap_get_option calls were for testing once I began hacking DocMgr. I added them in to test an error I was getting in ldap_search.

<?php
session_start();
header("Cache-control: private"); //IE 6 Fix

$username=$_POST['username'];
$password=$_POST['password'];
$ds;
$searchbase = 'OU=Users,DC=CIRCUS,DC=ORG';
$group = 'CN=DocMgr,OU=Users,DC=CIRCUS,DC=ORG';

echo "<html> <head>  <title>AD Test</title></head> <body>";
echo "<p align=\"left\">This is testAD</p>\n";
echo "Username: ". $username . "<br>\n";
echo "Password: ". $password . "<br>\n";

function ad_ldap_auth()
{
    global $username, $password, $ds;
    $ldapServer = 'server.circus.org';
    $ldapPort = '389';
    $ds = ldap_connect($ldapServer, $ldapPort)
        or die("Could not connect to ldap.");

    ldap_set_option($ds, LDAP_OPT_REFERRALS, 0);

    if ($ds)
    {
        $binddn = "$username@.circus.org";
        $ldapbind = ldap_bind($ds, $binddn, $password);

        if ($ldapbind)
        {
                echo "Bound<br>\n";                  
        }else{
                echo "<br>Not bound<br>\n";                  
        }
    }
    return $ldapbind;
}


$test = ad_ldap_auth();

if ($test)
{
    $_SESSION['name'] = $username;
    $_SESSION['expire'] = (time() + 3600);

    $filter = "sAMAccountName=$username";
    //$filter = "sAMAccountName=*";
    $attrib = array("memberOf");
    $attrsonly = 0;
    $sizelimit = 3;
    $timelimit = 10;

    ldap_get_option($ds, LDAP_OPT_SIZELIMIT, $optVal); // returns 0
    echo "size_limit =". $optVal ."<br>";
   
    ldap_set_option($ds, LDAP_OPT_SIZELIMIT, 5); // returns TRUE
    ldap_get_option($ds, LDAP_OPT_SIZELIMIT, $optVal); // returns 0
    echo "size_limit =". $optVal ."<br>";

    $sr = ldap_search($ds, $searchbase, $filter, $attrib);

    for ($entry=ldap_first_entry($ds,$sr); $entry!=false; $entry=ldap_next_entry($ds,$entry))
    {
        $values = ldap_get_values($ds, $entry, "memberOf");

        echo $values["count"] . " groups for this entry.<br />";

        for ($i=0; $i < $values["count"]; $i++)
        {
                echo $values[$i] . "<br />";
            if (strcmp($values[$i],$group) == 0)
            {
                echo "User is member.<br>";
            }
        }
    }
} else {
    echo "LDAP Failed.";
}

ldap_unbind($ds);

?>

After finishing my script it was time to move on to DocMgr. I turned on DEBUG in the config.php and began placing debug output throughout DogMgr so that I could see the flow of function calls as the program ran.

if (DEBUG)
   print("authroized == 1<br>");

I wrote down the flow of some key authentication functions and then began reading them to understand what each did and where I should start hacking. Then I went and broke the old ldap includes into open-ldap and ad-ldap includes and set up the ifdefs in the different files. Doing this I was able to still use my OpenLDAP authentication and could hack on ad-ldap.

I set up the ad-ldap-config.php file by comparing the output of ldapsearch in ldif format on both OpenLDAP and AD. We made some changes to the AD population script and moved forward using employeeNumber instead of uidNumber.

After I got the correct entries in AD and OpenLDAP figured out it was just a matter of hacking my test script into the correct functions and testing. There were a couple of errors that I needed to fix. The first was caused by searching by for sAMAccountName=* in the else portion of the ldap_account_search function of docmgr/include/ad_ldap.inc.php.

This was the first error and fix:

Warning: ldap_search(): Partial search results returned: Sizelimit exceeded in /var/www/docmgr/include/ad_ldap.inc.php on line 697

ldap_set_option($ds, LDAP_OPT_SIZELIMIT, 5);

This was the second error and fix:

Warning: ldap_search(): Search: Operations error in /var/www/docmgr/include/ad_ldap.inc.php

ldap_set_option($ldap, LDAP_OPT_REFERRALS, 0);

Once I finished making DocMgr authenticate to AD I wanted to make the integration into AD complete so that I did not have to manually add or remove users within the web interface. The following is the script that does that.

It is run every morning from cron.

# crontab -l
# m h  dom mon dow   command
# Add/remove DocMgr users according to AD group DocMgr.
16 4 * * * /usr/local/bin/ad-docmgr.pl

And the code for allowing access to DocMgr. I truncate the table and just add the users each time.

#!/usr/bin/perl

# 2010-04-13  Jud Bishop
# This script queries active directory for every member of the DocMgr group
# and adds them to the auth_accoutperm table, which gives them the ability
# to login to DocMgr after authenicationg to AD.

use strict;
use DBI;
use Net::LDAP;
use Net::LDAP::LDIF;
use Net::LDAP::Entry;

# Globals
# 1 == debug
# 0 == quiet
my $debug = 1;

sub connectLDAP {
    if($debug){print "connectLDAP\n";}

    my $ldap= Net::LDAP->new("server.circus.org", port =>"389", version =>"3" )
        or die $!;
   
    my $result = $ldap->bind( "CN=SpecialUser,DC=CIRCUS,DC=ORG",
                password => "PassWord" );
        die $result->error() if $result->code();

    return $ldap;
}

sub queryORG {

    my $mesg;
    my ($ldap, $dbh) = @_;

    if($debug){print "queryORG\n";}

    $mesg = $ldap->search(
        base => "OU=Users,DC=CIRCUS,DC=ORG",
        scope => "sub",
        filter => "(memberOf = CN=DocMgr,OU=Groups,DC=CIRCUS,DC=ORG)");
   
    if ( $mesg->count() > 0 )
    {
            my $max = $mesg->count;
                for ( my $j = 0 ; $j < $max ; $j++ )
                {
            my $entry = $mesg->entry ( $j );
            foreach my $t ( $entry->get_value( "employeeNumber" ))
                    {
                if($debug){print "employeeNumber $t\n";}
                insertQuery($dbh, $t);
            }
        }
    }
}

sub unbindLDAP {
    if($debug){print "unbindLDAP\n";}
    my $ldap = shift;
    $ldap->unbind();

}

sub connectPostgres {
    if($debug){print "connectPostgres\n";}

    my $dbh = DBI->connect('DBI:Pg:dbname=docmgr;host=127.0.0.1', 'DocMgrUserName', 'PassWord' , {
        PrintError => 0,
        RaiseError => 1 # Report errors via die(), needs less error handling because
                # DBI will check for us and die() automagically.
    });

    if ($dbh) {
        # From above we don't have to check, this is just for debugging.
        if($debug){print "connected\n";}
        return $dbh;
    }
}

# Truncate the table so we don't have to worry about who is there and who isn't.
sub deleteQuery {
    if($debug){print "deleteQuery\n";}

    my $dbh = shift;
    my $sth = $dbh->prepare("TRUNCATE auth_accountperm");
    $sth->execute;
}

sub insertQuery {
    if($debug){print "insertQuery\n";}

    my ($dbh, $employeeNumber) = @_;
   
    my $sth = $dbh->prepare("INSERT INTO auth_accountperm VALUES ($employeeNumber, 1, true)");

    $sth->execute;
    if($debug){print "INSERT INTO auth_accountperm VALUES ($employeeNumber, 1, true)\n"};
}

sub disconnectPostgres {
    if($debug){print "disconnectPostgres\n";}
   
    my $dbh = shift;
    $dbh->disconnect();
}

my $ldap = connectLDAP();

my $dbh = connectPostgres();

deleteQuery($dbh);

queryORG($ldap,$dbh);

disconnectPostgres($dbh);

unbindLDAP ($ldap);

# end main
Categories: Code, Linux Tags:

Linux DBI::ODBC AS400

April 19th, 2010 No comments

At the Circus we are removing the last vestige of our OpenLDAP implementation and moving it to Active Directory. As a result I’m going to document some of the odd scripts I have written as glue to help keep things running. Some of the scripts are too long to document and will just be kept in my script library, while others like the ones below, are generic enough that they might help someone else.

This is some documentation for how to query an AS400 from Linux. When I did this project I actually talked to one of the developers of the iSeries Access programs for help. He ended up sending me a snapshot rpm to get it all working.

Here are the configuration files.

#cat /etc/odbcinst.ini

[iSeries Access ODBC Driver]
Description     = iSeries Access for Linux ODBC Driver
Driver          = /opt/ibm/iSeriesAccess/lib/libcwbodbc.so
Setup           = /opt/ibm/iSeriesAccess/lib/libcwbodbcs.so
Threading       = 2
DontDLClose     = 1
UsageCount      = 1
#cat /etc/odbc.ini

[AS400]
Description     = iSeries Access ODBC Driver
Driver          = iSeries Access ODBC Driver
System          = as400.circus.org
UserID          = UserName
Password        = Password
Naming          = 0
DefaultLibraries= QGPL
Database        =
ConnectionType  = 0
CommitMode      = 2
ExtendedDynamic = 1
DefaultPkgLibrary = QGPL
DefaultPackage   = A/DEFAULT(IBM),2,0,1,0,512
AllowDataCompression = 1
LibraryView     = 0
AllowUnsupportedChar = 0
ForceTranslation= 0
Trace           = 1
DSN             = AS400

This is a simple script to list the drivers available on the system. This does not look like a script I wrote so I am reticent to take ownership.

#!/usr/bin/perl

use DBI;

my @drivers = DBI->available_drivers();

die "No drivers found. \n" unless @drivers;

# list the data sources
foreach my $driver ( @drivers )
{
    print "Driver: $driver\n";
    my @dataSources = DBI->data_sources( $driver );
    foreach my $dataSource ( @dataSources )
    {
        print "\tData Source: $dataSource\n";
    }
    print "\n";
}

exit

And the resulting output from the script above.

# perl drivers.pl
Driver: DBM
    Data Source: DBI:DBM:f_dir=PHPDemo
    Data Source: DBI:DBM:f_dir=SQL-LDAP
    Data Source: DBI:DBM:f_dir=.
    Data Source: DBI:DBM:f_dir=DBD-ODBC-1.09
    Data Source: DBI:DBM:f_dir=unixODBC-2.2.8
    Data Source: DBI:DBM:f_dir=DBI-1.42

Driver: ExampleP
    Data Source: dbi:ExampleP:dir=.

Driver: File
    Data Source: DBI:File:f_dir=PHPDemo
    Data Source: DBI:File:f_dir=SQL-LDAP
    Data Source: DBI:File:f_dir=.
    Data Source: DBI:File:f_dir=DBD-ODBC-1.09
    Data Source: DBI:File:f_dir=unixODBC-2.2.8
    Data Source: DBI:File:f_dir=DBI-1.42

Driver: ODBC
    Data Source: DBI:ODBC:AS400

This is a simple script to query an AS400. You will have to get with your AS400 administrator because the the query is actually an odd series of fields, at least it was for our organization. I just wrote this to learn how to query the AS400, it was never used in production.

#!/usr/bin/perl

# 2005-09-12 Jud Bishop
# Released under the New BSD License.

# This script uses the DBI::ODBC driver to interact with an AS 400
# database.  Make sure that you use a system DSN rather than a
# user defined one or you will have trouble later.

use strict;
use DBI;

my $UID="UserName";
my $PW="Password";
my $DSN="AS400";


print "connect\n";
my $dbh = DBI->connect ( "dbi:ODBC:$DSN", $UID, $PW, {
#   LongTruncOk => 1
    PrintError => 1
})
    or die "connection failed to database $DBI::errstr\n";

## Set up tracing
print "trace\n";
unlink 'trace.log' if -e 'trace.log';
DBI->trace( 2, 'trace.log' );

## Prepare the SQL statement for execution
print "prepare\n";
my @t = localtime(time);
my $sql_st = $dbh->prepare( " SELECT ALL X FROM Y WHERE I = J ORDER BY Z ");

print "execute\n";
$sql_st->execute()
    or die "Cannot execute SQL statement $DBI::errstr\n";

my @row;
while ( @row = $sql_st->fetchrow_array() )
{
    print "@row\n";
}
warn "Data fetch terminated by error $DBI::errstr\n"
    if $DBI::err;

$dbh->disconnect or warn "disconnection failed $DBI::errstr\n";

exit
Categories: Code, Linux Tags:

What I Learned Today

March 26th, 2010 No comments

One of my coworkers solved a problem today that I should have been able to solve. I got pulled into a project that uses a closed system with it’s own programming language that I had never seen nor programmed. My coworker was trying to figure out why his syntax was not working. So we cranked up the logging in Warn.log, Error.log and Fatal.log. We quickly figured out why it was failing, the program was looking for a file that did not exist and he was not catching that error, there is no catch statement.

So I began trying to write an IF/ELSE statement that would check for the error condition. I was given a .pdf of the manual for the system and just started trying different functions that made sense to me. I wrote a nice little one liner in the SYSTEM command that returned TRUE or FALSE if the FILE environment variable existed.

The problem was that we could not set the environment variable with a variable from the program. It has to be a string literal.

From the Trace.log:

      LET sys = EXPORT("FILE=" & $format)

      // Variable "Part01::sys" is a String
      // [  1] = ""

      LET sys = SYSTEM("if [ -f /usr/program/fmt/$FILE ]; then  echo "TRUE"; else echo "FALSE"; fi")

      // Variable "Part01::sys" is a String
      // [  1] = "FALSE"

      IF sys = "TRUE" THEN

        // Test result is FALSE - skipping...

      END IF

We tried a number of different variations on that theme, imported and exported variables to a subshell, tried different return values from the exit command. We got the system to turn circles but what we really wanted was squares.

Finally my coworker called and said, why don’t we just invert the logic. If it falls through to the end of the IF/ELSE statements it must be in the format we want. Let me take a step back and say this was his code, he should have been the one to figure it out. I believe I helped him figure out what his main problem was, which helped him find the solution. I’m not trying to minimize his work, nor am I trying to minimize my input into the solution.

The moral of the story is that I should have taken a higher level look at the problem to understand the solution. I am quick to believe that I can code my way out of any problem, regardless whether or not I have ever seen the language. What I really needed to do was step back and take a higher level view of the logic, rather than dive into syntax and functions.

Categories: Code, Linux, Musings Tags:

Xenix

March 21st, 2010 No comments

Today on Slashdot someone had a question about getting data off of an old Xenix server. A few years ago I did a consulting job for a customer who had an old Xenix server with no ethernet card that needed to get some data for the State Police. The server ran an old database that kept track of training records and they needed that information in order to “webify” the process.

I wish I could remember the ins and outs of the problem. I gave the customer a write up of how I did it and the problems I had getting the usr directory off of the system. Unfortunately I can’t find the report.

Regardless, it was an interesting project, here is the script from my library that eventually worked:

#!/bin/sh

for I in bin boot dos etc lib oa.files once shlib tmp u unit57 unit58 unit59 usr xenix
do
    echo "tar -cf - $I|uuencode -"
    tar -cf - $I|uuencode -
done

tar -cf - `find /usr -type f -print` >2/usr.err|uuencode -


for I in `cat usr.txt`
do
    tar -cf - $I |uuencode -
done
Categories: Code Tags:

DHCP NAT

March 8th, 2010 No comments

I have debated about whether or not to post this project. I wanted this to have a sqllite backend and run on the Cisco AXP but never had the time to go back and hack more on it. I think it could have won the AXP hacking contest, but I didn’t even know about the contest at the time.

I also want to say that this project is not for the inexperienced Linux admin. This project involves a solid understanding of DHCP, DNS and iptables. I have rolled a .tar file of all of the scripts and directories needed to make this happen, but I’ve said it before and I will say it again, you got it off the web, your mileage may vary. Here is the file.

History
The basic idea here is this, we have ~375 printers that were all translated using static NAT, but most of our infrastructure runs off DHCP. So when someone moves an office they unplug their VOIP phone and computer and take it to their new office, but the printer involves manual intervention, not good.

There is one thing to say about a lazy network administrator, it usually means problems like this get fixed one way or the other. I wanted a way to be able to move printers within the “enterprise” and have their NATed address follow them. This is my solution.

NAT

A picture is worth a thousand words so here is a simple diagram.

Diagram of WAN.

The 192.168.24.0/24 subnet is shared between the Circus and our Electronic Medical Records vendor, EMR for short. However, the Circus is partially switched and partially routed on campus so we have to NAT IP addresses on the shared subnet into IP addresses in the enterprise. So this diagram shows a translate server with eth0 at 192.168.24.2 and eth1 as 192.168.100.58, eth0:1 is a NAT’ed IP address for a printer somewhere in the enterprise.

Let me explain this again, this time with commands:
WAN address that is shared with another company, in our case a printer:
wan printer — 192.168.24.100 — IP address EMR uses to print.
translate eth0:1 — 192.168.24.100 — IP address EMR uses to print, actually a virtual interface on translate server.
lan printer — 192.168.10.100 — IP address of printer on campus.
translate eth0 — 192.168.24.2 — WAN leg of translate server.
translate eth1 — 192.169.100.2 — LAN leg of translate server.

The wanprinter ip address 192.168.24.100 is a virtual ip address on a linux
server, translate. It would be brought up from the command line like this:

# /sbin/ifconfig eth0:1 192.168.24.100 netmask 255.255.252.0 up

It is then natted from 192.168.24.100 to it’s local ip address:

# /sbin/iptables -t nat -D PREROUTING --destination 192.168.24.100 -j DNAT --to 192.168.10.100

The other rule that is added only once is to make it look like all traffic
is coming from the translate server:

# /sbin/iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to 192.168.100.2

rsyslog setup

You need to download rsyslog. This is because you need reliable remote logging and the normal syslog server (sysklogd) does not handle it. I am using librelp version 0.1.1 and rsyslog version 3.20.2. The problem that relp/rsyslog fixes is that standard syslog does not reliably log remotely. So what ends up happening is that dhcp log messages will be dropped, eventually it will get you because an address update message will be dropped.

The DHCP servers in our organization run Red Hat Enterprise Linux and the translate servers run Ubuntu LTS. Please check the README or INSTALL document from rsyslog for installation on your platform. The dhcp servers will be remote logging dhcp transactions to the translate servers so that a logwatch plugin can monitor address changes.

Just download these and read the doc/install.html.

wget http://download.rsyslog.com/librelp/librelp-0.1.1.tar.gz
./configure
make
make install
wget http://www.rsyslog.com/Downloads-req-getit-lid-141.phtml
./configure --enable-relp --enable-mail --enable-zlib --with-gnu-ld
make
make install

To compile on RedHat I had to use –with-gnu-ld after editing etc/ld.so.conf:

# cat /etc/ld.so.conf.d/librelp.conf
/usr/local/lib

I also had to add an environmental variable:

# export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

In order to get RHEL standard scripts to work you have to add a symlink to
/usr/local/sbin/rsyslogd from /usr/sbin/rsyslogd:

# ln -s /usr/local/sbin/rsyslogd /usr/sbin/rsyslogd

Make sure to edit the sysklogd init script or copy in a new one. Also make sure you get rsyslogd hacked into init. The easiest way is to run this command after putting my rewritten RH init script on the box. The rewritten script is in the tar file from above as daemon/rh-etc-init.d-rsyslogd.

# chkconfig syslogd off
cp rh-etc-init.d-rsyslogd /etc/init.d/rsyslogd
# chkconfig rsyslogd on
# chkconfig --list | grep sys

If you didn’t use my init script, edit /etc/cron.daily/sysklog script to make sure rsyslog is notified about rotation.

Also edit the logrotate configuration at /etc/logrotate.conf:

# cat /etc/logrotate.d/syslog
/var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron {
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Please see my dhcp-rsyslog.conf and translate.rsyslog.conf for the rsyslog
configurations for each server. If it’s not self explanatory copy the file daemon/dhcp-rsyslog.conf to the dhcp server /etc/rsyslog.conf. You will have to change the IP address of the translate server from 192.168.100.2 to whatever you are using.

DHCP Servers

Make these changes in NS1 and NS2, or if you have a setup similar to me,
just copy /etc/dhcpd/dhcpd.ad.master.conf from NS1 to NS2 and restart both DHCP servers.

In /etc/dhcpd/dhcpd.master.conf add these lines:

# Where to log stuff, this sets up remote logging for
# syslogscand and DHCP-Nat.
log-facility local3;

Restart dhcpd.

Translate Server

Our translate server is a Dell 1U 2600 or something like that. It presently
has ~375 virtual interfaces translating printers from the WAN subnet to our
internal addresses.

eth0 == External or WAN address
inet addr:172.22.25.193 Bcast:172.22.27.255 Mask:255.255.252.0

eth1 == Internal or LAN address
inet addr:172.22.100.52 Bcast:172.22.100.255 Mask:255.255.255.0

The internal IP address is only needed for the postrouting rule.

Install syslogscand on this server through CPAN and then copy DHCP.pm to the correct location. If all else fails just head DHCP.pm and see where it tells you to copy it. Then copy the file daemon/etc-syslogscand.conf to /etc/syslogscand.conf.

Finally move rc.local.firewall to a system directory and add it to rc.local
to fire off upon server start.

syslogscand Plugin Configuration

I wrote a syslogscand plugin, DHCP.pm, that watches the output of dhcp and any
time a mac address that it watches is given an IP address it queries through
dns lookups whether or not the mac address has gotten a new IP address. If it
has gotten a new address, the daemon updates the /etc/network/dhcp-nat file
as well as the /etc/network/interfaces file. It then updates iptables with the
new NAT translation.

Below is a dhcp-nat file. It is comma delimited with the first column being
the mac address of the printer, the second column is the internal LAN address,
the third column is the external WAN address and the forth column is the
hostname.

# cat /etc/network/dhcp-nat
00:0e:7f:3c:88:bc,172.22.81.110,172.22.25.252,printest252
00:1f:f3:5a:b2:6a,172.22.81.116,172.22.25.249,trek
00:19:b9:45:04:30,172.22.81.254,172.22.25.251,litespeed
00:11:0a:f6:3f:93,172.22.81.254,172.22.25.250,printest250

I have also added information to the /etc/network/interfaces file,
notice in the following stanza that I have added comments for the
mac and translate(d) address.

# head -6 /etc/network/interfaces
#mac 00:1f:f3:5a:b2:6a
#translate 172.22.81.116
auto eth0:8
iface eth0:8 inet static
address 172.22.25.249
netmask 255.255.252.0

Web Interface
There is also a web interface into this so that PC technicians can interact
with the system easily. In order to make that magic happen I added the
following line to /etc/sudoers:

www-data translate1 = NOPASSWD: /usr/bin/syslogscand, /sbin/iptables, /sbin/ifconfig

It allows the user www-data that apache runs under to restart syslogscand as
well as manipulate iptables and interfaces.

Copy all of the files in the cgi-bin directory to your local cgi-bin directory.

Copy all of the files in the html directory to the web server root.

You may have to tweak directory locations in the scripts.

Categories: Code, Linux Tags:

Converting AIX print queues to Linux

February 25th, 2010 No comments

I spent the last week working on a project to convert all of the forms printing at the Circus from an AIX server to a Linux server. Because the version we are stuck on is not officially supported by the vendor I had to do some reverse engineering to figure out how things work, this article describes some of the scripts I used to understand what the software was doing and how to make it work the way we needed.

The funny thing about any printer project I get into, usage seems to explode. Yesterday we had a water cooler meeting about how to print from a completely different EMR system. I’m not sure we will do it because we need to know patient location but the printing part is easy. Who knew a paperless organization prints so much.

We have about 500 printers on the network and less than 314 forms, my guess would peg us closer to 150 forms, but when I go through the forms directory I get 314.

# ls -1 | cut -d _ -f 1,2 | sort -u | wc
    314     314    3293

Regardless, the number of forms is not the issue. I changed everything I needed programmatically as you will see below. All of the scripts in the post can be downloaded here.

First I needed to get a connection from one of our electronic medical records applications to the new Linux forms print server. I wrote a short script that shows me the arguments passed to the printer as well as the standard input. This allowed me to troubleshoot what was being passed from one server to another and also gave me insight into how the whole process worked.

#!/usr/bin/perl -w
use strict;

# 2010-02-17  Jud Bishop
# Quick hack to see what I am being sent from the port.
# Released under the GNU GPLv2

my $file = "/tmp/print-test.txt";

sleep 5;
unlink $file;

open (NEWFILE, ">$file" ) or die("Error: can't open $file\n$!");

        print NEWFILE "Args:\n";
        for (my $i = 0; $i <= $#ARGV; $i++) {
                print NEWFILE "argv[$i] == $ARGV[$i]\n";
        }

        print NEWFILE "Data:\n";
        while (<STDIN>) {
                print NEWFILE $_;
        }

close NEWFILE or die("Error: can't close $file\n$!");

Once I got the two servers communicating properly it was time to troubleshoot some configuration files that were copied from the AIX server to Linux. I ended up writing the next short script to change some application configuration files that were originally made to work on AIX. This changes the print command from the AIX qprt command to Linux lpd.

# 2010-02-17  Jud Bishop
# This script changes all of the printer configuration channels
# from AIX specific qprt to Linux specific lp commands.
# Released uner the GNU GPLv2

for I in `ls -1`
do
    sed 's/lp -d/lp -d/g' $I > tmp
    if [ $? -eq 0 ]
    then
        mv -f tmp $I
    else
        echo $I "did not complete"
        exit
    fi
done

Now it was time to write a short script to convert the /etc/qconfig printer configuration file from AIX and add the printers on the Linux server. So I wanted to test adding and deleting a printer from cups on the command line. I thought the following command would work.

# lpadmin -p misp1 -E -v socket://misp1.circus.org -m laserjet
# lpadmin: Unable to copy PPD file!

If all else fails read the man page, or in this case the manual on the web. It’s pretty cryptic but they tell you that the -m model has to be from the model directory. Where is the model directory?

# find / -name model
...
/usr/share/cups/model
# ls -1 /usr/share/cups/model/
deskjet2.ppd.gz
epson9.ppd.gz  
laserjet.ppd.gz

So we want the model from /usr/share/cups/model. The following command works:

# lpadmin -p misp1 -E -v socket://misp1.circus.org -m laserjet.ppd.gz

Now it’s time to convert from qconfig to cups. I used the following script to read in qconfig and create the printers. I had to add the sleep because the script got ahead of the lpadmin command. Once I added the sleep it worked and ta-da, 500 printers were created. I realize that printers are added and deleted from the AIX server daily so I had to make sure it would add and delete printers so that we will get them all on go-live day. It’s a rudimentary implementation but it works.

Notes on this script, it was easier than I expected, hence I a have hash instead of just working from the original array. It’s one of those things that I’m not going to go back and clean up for a simple script, sorry.

#!/usr/bin/perl

# 2010-02-18  Jud Bishop
# This script reads in the qconfig file of an AIX server and
# converts it to a Linux based cups file.  
# I am expecting this to be a quick hack...
# Released under the GNU GPLv2

# /etc/qconfig is the AIX printer configuration file.
#labp4:
#        device = @hpjd007
#        up = TRUE
#        host = hpjd007.circus.org
#        s_statfilter = /usr/lib/lpd/bsdshort
#        l_statfilter = /usr/lib/lpd/bsdlong
#        rq = PORT1
#@hpjd007:
#        backend = /usr/lib/lpd/rembak -T 30
open (FILE,"<etc-qconfig") or die "Error: can't open file $! \n";
        @aix = <FILE>;
close FILE or die "Error: can't close file $! \n";

my %table;
my $j = 0;
my ($que, $host, $port, $trash);
for (my $i = 0; $i <= $#aix; $i++) {
        if ( $j == 0 ){
                ($que, $trash) = split (/:/, $aix[$i]);
        } elsif ( $j == 3) {
                chomp($aix[$i]);
                $aix[$i] =~ s/ //g;
                ($trash, $host) = split (/=/, $aix[$i]);
                if ( $host !~ /circus.org/ )
                {
                        $host = sprintf ("%s.circus.org", $host);
                }
        } elsif ( $j == 6) {
                chomp($aix[$i]);
                ($trash, $port) = split (/=/, $aix[$i]);
        }  
        if ( $j == 6 ) {
                $table{$que} = {'host'=>$host, 'port'=>$port};
        }
        $j++;
        if ( $j == 9 ) {
                $j = 0;
        }
}

# The command to add a printer.
# system  lpadmin -p printer -E -v socket://host.circus.org -m laserjet.ppd.gz
# The command to remove a printer, got to be able to back it out.
# lpadmin -x printer
foreach $key (keys(%table)) {
        print "$key $table{$key}->{host} $table{$key}->{port}\n";

        my $socket = sprintf ("socket://%s", $table{$key}->{host});

        # Swap these two commands if you need to delete.        
        #my @args = ("lpadmin", "-x", "$key");
        my @args = ("lpadmin", "-p", "$key", "-E", "-v", "$socket", "-m", "laserjet.ppd.gz" );

        system(@args) == 0
        or die "system @args failed: $?";
        sleep 1;
}
Categories: Code, Linux Tags:

Parsing Barracuda Log Files

February 17th, 2010 2 comments

We have a Barracuda web filter here at the Circus. In general we are pleased with its performance, however, as our internet usage has climbed the days of history has declined. The inverse relationship is due to a “ring buffer” of 250,000 entries in the history log. There are times when we have a few hours of history and that doesn’t sit well when a manager wants to see the browsing history for a user or a pc. So we turned on syslog logging to a remote logging server and evaluated the log parsing packages that mentioned they parsed Barracuda logs. Let me give you a hint, there are not many.

My first hack was just to see what was going on and give a rudimentary understanding, it can be downloaded here.

!#~/bin/bash
grep $1 /var/log/barracuda.log | grep http_scan | awk '{print $28}' | sort | uniq -c | sort -n

Which gives output like this:

     63 autotrader.com
     72 google.com
     78 edmunds.com
     81 ad.doubleclick.net
     82 charter.com
    121 dealer.com
    138 alagasco.com
    194 synacor.com
    334 charter.net

The basis of that script gave me more understanding to write the following script which can be downloaded here.

#!/usr/bin/perl -w

use Getopt::Long;
use Number::Format;
use Tie::IxHash;


###############################################
# 2010-02-05 Jud Bishop
# This script parses /var/log/barracuda.log.
###############################################
# Options that can be passed in:
# -u username
# -s source ip address
# -d destination url
# -c category
# -days number of days
# -pc the pcname
#
###############################################
# The format of a log file message.
# Used this document:
# http://www.barracudanetworks.ca/download/barracuda-web-filter-syslog.pdf
# And this one liner:
# tail -50 /var/log/barracuda.log | awk '{print $28}' && tail -1 /var/log/barracuda.log
#
# 0.  month
# 1.  day
# 2.  time
# 3.  barracuda_ip
# 4.  http_scan[process_id]
# 5.  md5sum
# 6.  number1
# 7.  source_ip
# 8.  destination_ip
# 9. content_type
# 10. source_ip
# 11. destination_address/URL
# 12. data_size
# 13. BYF
# 14. action (ALLOWED, BLOCKED, DETECTED)
# 15. reason (CLEAN, VIRUS, SPYWARE)
# 16. format_ver == 2 (Version of the policy engine output.)
# 17. match_flag (Whether an existing policy matched the traffic: 1=Yes, 0=No.)
# 18. tq_flag (Time qualified flag; 1=Yes 0=No.)
# 19. action_type (The documentation for this flag is incorrect.)
# 20. source_type (0=Any source, 1=group, 2=ipv4addr, 3=login, 4=auth_user, 5=min_score)
# 21. src_detail Detail related to matched source or "(-)" if not.
# 22. dest_detail If there is a destination match, what type (0=any, 1=particular category, 2=any category
#                 3=domain, 4=mimetype, 5=spyware, 6=uri_path_regex, 7=uri_regex, 8=application)
# 23. dest_detail Matched category or "(-)" if not matched.
# 24. spyware (If it is spyware, 0=allow, not spyware, 1=block, 2=infection.)
# 25. spyware_id (Name of the spyware if matched, if not "-".)
# 26. infection_weight Weight of the infection, mostly 0.
# 27. matched_part Part of the rule theat matched.
# 28. matched_category Comma delimited category name that matched traffic.
# 29. user_name Username, ([ANON], [ldap0:jud], [username:jud])

###############################################
# Variables you can change.
###############################################
my $debug = 0;
my $log_file = "/var/log/barracuda.log";
#my $log_file = "/var/log/barracuda.test";

###############################################
# Variables you should NOT change.
###############################################
my $arg_username = 0;
my $arg_pc = 0;
my $arg_source_ip = 0;
my $arg_dest_url = 0;
my $arg_category = 0;
my $arg_days = 0;
my $arg_help = 0;
my %table; # holds all the data for each session.
tie %table, "Tie::IxHash";
my %categories; # holds all of the categories this person went to.
my $data_sum = 0; # holds the total of all data for a user.
my $session_sum = 0; # hold the number of sessions for a user.


# The names of these variable are so that I don't have to keep looking
# above to figure out what name is what item in the array.
my $user = 29;
my $md5sum = 5;
my $month = 0;
my $day = 1;
my $time = 2;
my $source_ip = 7;
my $destination_ip = 8;
my $url = 11;
my $data_size = 12;
my $action = 14;
my $part = 27;
my $category = 28;


# Reads the options passed in.
sub get_options {
    GetOptions(
        'help|?|h!' => \$arg_help,
        'u=s' => \$arg_username,
        's=s' => \$arg_source_ip,
        'd=s' => \$arg_dest_url,
        'c=s' => \$arg_category,
        'days=i' => \$arg_days,
        'pc=s' => \$arg_pc);
       
    if ($debug)
    {
        print "username == $arg_username\n";
        print "source_ip == $arg_source_ip\n";
        print "pc == $arg_pc\n";
        print "dest_url == $arg_dest_url\n";
        print "category == $arg_category\n";
        print "days == $arg_days\n";
        print "help == $arg_help\n";
    }

    if ($arg_help)
    {
        print "usage: user-report.pl -days number [-u usernname] [-s source-ip] [-d destination-url] ";
        print "[-c category] [-pc pc_name] [--help|-?]\n";
        exit;
    }
}

# Parses the logs.
# Days equals log file days, makes it easy.
sub parse_logs {

    my ($search_field, $search_equals, $days) = @_;

    if ($debug){
        print "--------------------\n";
        print "parse_logs\n";
        print "search_field == $search_field\n";
        print "search_equals == $search_equals\n";
        print "days == $days\n";
    }  

    # Loop through the log files based on number of days:
    # 0 == today
    # 1 == barracuda.log.1 one day past...
    # This is not formatted correctly because I added it as a retrofit.
    for (my $i = 0; $i <= $days; $i++)
    {
    if ($i == 0)
    {
        if ($debug) {print "open $log_file\n"};
        open (FILE, $log_file) or die "Error: can't open log file\n $! \n";
        print "open $log_file\n";
    } else {
        if ($debug) {print "open $log_file.$i\n"};
        my $file = sprintf ("%s.%s", $log_file, $i);
        print "open $file\n";
        open (FILE, $file) or die "Error: can't open log file\n $! \n";
    }
    while (<FILE>)
    {
        chomp;
        # Makes split work like awk, don't believe the man page.
        my (@log) = split /\s+/;

        # The next check is to catch the following type messages.
        # Feb  9 06:51:09 last message repeated 8 times
        # The following anomoly took a long time to track.
        # Mar  9 08:30:20 172.31.1.253 http_scan[6380] 1268145019 1 172.22.40.29 68.71.209.206 text/plain 172.22.40.29 http://funschool.kaboose.com/current/scripts/touch 401 BYF ALLOWED CLEAN
        if ($#log > 20 and ($log[$search_field] eq $search_equals))
        {
            if ($debug){
                print "$log[$search_field] $search_equals\n";
                for (my $i = 0; $i <= $#log; $i++)
                {
                    print "log[$i] == $log[$i]\n ";
                }
            }
            # Each session gets a different md5sum, which is why it is the key in the table.
            if( not exists $table{$log[$md5sum]}){
                $table{$log[$md5sum]} = {'user'=>$log[$user], 'month'=>$log[$month], 'day'=>$log[$day], 'time'=>$log[$time], 'source_ip'=>$log[$source_ip], 'destination_ip'=>$log[$destination_ip], 'url'=>$log[$url], 'data_size'=>$log[$data_size], 'action'=>$log[$action], 'category'=>$log[$category], 'total_data'=>$log[$data_size], 'session_count'=>1 };

                $data_sum += $log[$data_size];
                $session_sum += 1;     
                if ( not defined $categories{$log[$category]} )
                {
                    $categories{$log[$category]} = 1;
                } else {
                    $categories{$log[$category]} += 1;
                }
            } else {
                $table{$log[$md5sum]}->{total_data} += $log[$data_size];
                $data_sum += $log[$data_size];
                if($debug){
                    print "exists \n";
                    print "bandwidth = $data_sum\n";
                }
            }
        }
    }
    close FILE or die "Error: can't close file\n $! \n";
    }
}

sub print_report {

    if ($debug) {print "print_report\n";}
    if ($arg_username)
    {
        print "Useage report for: $arg_username\n";
    } elsif ($arg_pc) {
        print "Useage report for: $arg_pc\n";
    } elsif ($arg_source_ip) {
        print "Useage report for: $arg_source_ip\n";
    }
    print "Number of sessions: $session_sum\n";

    my $x = new Number::Format;
    $formatted = $x->format_bytes($data_sum);
    print "Total bandwidth consumed: $formatted\n\n";
    my $action = "Allowed";

    foreach my $category (sort (keys(%categories)))
    {
        printf "category %s\n", uc($category);

        foreach $key (keys(%table))
        {
            if ( $table{$key}->{category} eq $category)
            {
                my $url = substr($table{$key}->{url}, 0, 35);
                if ($table{$key}->{destination_ip} eq "172.27.72.27")
                {
                    $action = "Blocked";
                } else {
                    $action = "Allowed";
                }                  
                print "$table{$key}->{month} $table{$key}->{day} $table{$key}->{time} $url $action\n";
            }
        }
        print "\n";
    }
}

###############################################
# main
###############################################
my $search_field;
my $search_equals;

get_options();

if ($arg_username) {
    $search_field = $user;
    if ($arg_username eq "ANON"){
        $search_equals = $arg_username;
    } else {
        $search_equals = sprintf("[ldap0:%s]", $arg_username);
    }
} elsif ($arg_pc) {
    $search_field = $user;
    $search_equals = sprintf("[ldap0:%s]", $arg_pc);
} elsif ($arg_source_ip) {
    $search_field = $source_ip;
    $search_equals = $arg_source_ip;
} elsif ($arg_dest_url) {
    $search_field = $url;
    $search_equals = $arg_dest_url;
} elsif ($arg_category) {
    $search_field = $category;
    $search_equals = $arg_category;
}
if ($debug) {print "search_field == $search_field\n"}

parse_logs($search_field, $search_equals, $arg_days);

print_report();

This script produces output like the following:

Usage report for: circus-user
Number of sessions: 401
Total bandwidth consumed: 19.39M

ADVERTISEMENT-POP-UPS,GAME-MEDIA,CUSTOM-2
Feb 17 15:44:17 http://games.mochiads.com/c/p/the-r ALLOWED

AUCTIONS
Feb 17 17:08:21 http://rover.ebay.com/ar/1/56033/1? ALLOWED

AUCTIONS,MOTOR-VEHICLES,CUSTOM-1
Feb 17 17:13:44 http://edmunds.autotrader.com/js/jq ALLOWED
Feb 17 17:13:45 http://edmunds.autotrader.com/inc/g ALLOWED
Feb 17 17:13:46 http://edmunds.autotrader.com/inc/j ALLOWED
Feb 17 17:13:46 http://edmunds.autotrader.com/inc/j ALLOWED
Feb 17 17:13:47 http://edmunds.autotrader.com/dwr/i ALLOWED
Feb 17 17:13:50 http://edmunds.autotrader.com/no_ca ALLOWED

BUSINESS
Feb 17 15:09:08 http://www.statcounter.com/counter/ ALLOWED
Feb 17 15:10:15 http://www.alagasco.com/fw/_css/fle ALLOWED
Feb 17 15:10:16 http://www.alagasco.com/scripts/jFa ALLOWED
Feb 17 15:10:17 http://www.alagasco.com/fw/_js/flex ALLOWED
Categories: Code, Linux Tags:

Pingcheck

December 2nd, 2009 No comments

In my previous posts I have discussed other scripts to monitor or work with DNS. This pingcheck script is used to find used or unused IP addresses in the network, or to populate an arp table for a subnet. You can download the following code here as a normal shell script or as a .tar file.

This script also uses jot, one of the first scripts I posted from my script library.

#!/bin/bash
# 2005-06-06 Jud Bishop
# Released under GPLv2.
# pingcheck -- Finds open ip addresses or inversely active ip addresses in a range.
# Output includes activity as well as DNS resolution.

if [ -z "$1" ]
then
    echo "Usage:"
    echo "      pingcheck 190 254 192.168.1"
    echo "          190 is the starting ip address"
    echo "          254 is the last ip address"
    echo "          192.168.1 is the subnet"
    echo "Example:"
    echo "      pingcheck 190 254 192.168.1 "
else
    for I in `jot -s $1 -e $2`
    do
        NAME=`nslookup $3.$I | grep = |cut -d = -f 2`
        # These two lines are the same, just different ways to clean up the output.
        #ping -c 1 -w 1 $3.$I 2>/dev/null 1>/dev/null
        ping -c 1 -w 1 $3.$I >/dev/null  2>&1
        if [ $? -eq 0 ]
        then
            echo $3.$I exists $NAME;
        else
            echo $3.$I does not exist $NAME;
        fi
    done
fi
Categories: Code Tags:

Graphing HTTP response time with MRTG

October 19th, 2009 2 comments

If you read my SLA Labbing post ealier you know that we were having trouble graphing Cisco SLA output in MRTG. The problem was the the MIB was not returning information that made graphable sense to MRTG. Which is when I got involved to write a script that would help us out.

This is how you would download snmp data from your router:

# snmpwalk -v 2c -c public 192.168.12.1 1.3.6.1.4.1.9.9.42.1.3.4.1.11.1
SNMPv2-SMI::enterprises.9.9.42.1.3.4.1.11.1.104057532 = Counter32: 329

And to make it more MRTG friendly:

# snmpwalk -v 2c -c public 192.168.12.1 1.3.6.1.4.1.9.9.42.1.3.4.1.11.1 | cut -d \: -f 4 | sed -e 's/ //g'
357

Regardless, I abandoned this when our graphs were not that helpful and moved on to another format. This script and resulting graph show the ping and http download speed to the web server in question. I realize there is a considerable amount of application latency built in, and the graphs also confirm this. Remember, Cisco sla takes great pains to eliminate the upper layer latency.

You can download this script in .tar or .pl. I have removed the perldoc formatting from the script below so that it renders properly.

#!/usr/bin/perl
# 2009-10-13  Jud Bishop
# Please run perldoc on the script for more information.

use strict;
use Time::HiRes qw(gettimeofday);
use LWP::Simple;

my $server = "192.168.24.234";
my $page = "/Prod/site/default.aspx";

# Should not have to change anything below this.
my $download = "http://" . $server .  $page;

#Record time prior to request
my $start = gettimeofday();

# Test for successful download
if (head($download))
{
        my $t = (gettimeofday() - $start) * 100;
        printf ("%.4f \n", $t);
}
else {
        print "0\n";
}

system "ping -c 1 $server | grep rtt | cut -d \= -f 2 | cut -d \/ -f 1 | sed -e 's/ //g'";

print "Web Response\n";
print "Ping Response\n";

=head1 NAME

web-ping.pl - A script to download a web page and ping a server to compare response times.

=head1 SYNOPSIS

A script that outputs the time in ms to download a webpage and ping a server.

=head1 DESCRIPTION

This is for graphing both page download and ping response time for MRTG.
The external command must return 4 lines of output:


    Line 1 current state of the first variable, normally 'incoming bytes count' but it represents the web page load time.
    Line 2 current state of the second variable, normally 'outgoing bytes count' but it represents the ping time.
    Line 3 string (in any human readable format), telling the uptime of the target, not used.
    Line 4 string, telling the name of the target, not used.


Put this in your 192.168.1.1.cfg file.  You may need
to adjust the directories to match your configuration.


    WorkDir: /usr/local/www/data-dist/stats/CircusStats2
    Logformat: rrdtool
    PathAdd: /usr/local/bin/
    LibAdd: /usr/local/lib/perl5/site_perl/5.8.8/

    Target[CircusStats-http]: `/usr/local/www/data/stats/configs/web-ping.pl`
    Title[CircusStats-http]: Circus HTTP Response
    PageTop[CircusStats-http]: Circus Response
    LegendI[CircusStats-http]: HTTP Response
    LegendO[CircusStats-http]: Ping Response
    Ylegend[CircusStats-http]: Response in MS
    Legend1[CircusStats-http]: HTTP Response
    Legend2[CircusStats-http]: Ping Response
    ShortLegend[CircusStats-http]:   MS
    routers.cgi*Options[CircusStats-http]: fixunit nototal nopercent nomax
    routers.cgi*InCompact[CircusStats-http]: no
    routers.cgi*Graph[CircusStats-http]: Circus-Combined noi

=head1 COPYRIGHT

Copyright 2009-10-13  Jud Bishop
Released under the GPLv2.
=cut

This is the resulting output from the script and MRTG configuration.

MRTG Graph

Categories: Code, Routing Tags:

Delete files not in use

September 29th, 2009 No comments

Sometimes you need to clear out a directory of files but don’t want to delete any files in use by a program or user. This little script is the answer to those problems.

There are two interesting items in this script. The first is that this directory had grown so large that the rm -rf * command was not able to process it, which is the reason I dump the listing to a temporary file. The second item is the $? within the test command. I am testing the return of the last command that was executed. By Unix convention a 0 is returned upon successful completion, any other value implies failure. That line could also have been written:

if fuser /var/spool/filter/$I >/dev/null 2>&1
then
code...
fi

Regardless, here is the script.

#!/bin/bash
#  2007-08-28  Jud Bishop
#  Released under the GPLv2.
#  This little script checks to makes sure that a file is not in
#  use before it deletes the file or directory, which is why I
#  use an -rf, to delete the directory.  The reason I have to
#  write the directory listing to a file is because the directory
#  gets too large to be handled by the rm command.

ls /var/spool/filter >/tmp/purge-filter

for I in `cat /tmp/purge-filter`
do
     fuser /var/spool/filter/$I >/dev/null 2>&1
     if [ $? -ne 0 ]
     then
          #echo "Deleting /var/spool/filter/$I"
          rm -rf /var/spool/filter/$I
     fi
done
Categories: Code, Linux Tags: