Archive

Archive for February, 2010

Converting AIX print queues to Linux

February 25th, 2010 jud No comments

I spent the last week working on a project to convert all of the forms printing at the Circus from an AIX server to a Linux server. Because the version we are stuck on is not officially supported by the vendor I had to do some reverse engineering to figure out how things work, this article describes some of the scripts I used to understand what the software was doing and how to make it work the way we needed.

The funny thing about any printer project I get into, usage seems to explode. Yesterday we had a water cooler meeting about how to print from a completely different EMR system. I’m not sure we will do it because we need to know patient location but the printing part is easy. Who knew a paperless organization prints so much.

We have about 500 printers on the network and less than 314 forms, my guess would peg us closer to 150 forms, but when I go through the forms directory I get 314.

# ls -1 | cut -d _ -f 1,2 | sort -u | wc
    314     314    3293

Regardless, the number of forms is not the issue. I changed everything I needed programmatically as you will see below. All of the scripts in the post can be downloaded here.

First I needed to get a connection from one of our electronic medical records applications to the new Linux forms print server. I wrote a short script that shows me the arguments passed to the printer as well as the standard input. This allowed me to troubleshoot what was being passed from one server to another and also gave me insight into how the whole process worked.

#!/usr/bin/perl -w
use strict;

# 2010-02-17  Jud Bishop
# Quick hack to see what I am being sent from the port.
# Released under the GNU GPLv2

my $file = "/tmp/print-test.txt";

sleep 5;
unlink $file;

open (NEWFILE, ">$file" ) or die("Error: can't open $file\n$!");

        print NEWFILE "Args:\n";
        for (my $i = 0; $i <= $#ARGV; $i++) {
                print NEWFILE "argv[$i] == $ARGV[$i]\n";
        }

        print NEWFILE "Data:\n";
        while (<STDIN>) {
                print NEWFILE $_;
        }

close NEWFILE or die("Error: can't close $file\n$!");

Once I got the two servers communicating properly it was time to troubleshoot some configuration files that were copied from the AIX server to Linux. I ended up writing the next short script to change some application configuration files that were originally made to work on AIX. This changes the print command from the AIX qprt command to Linux lpd.

# 2010-02-17  Jud Bishop
# This script changes all of the printer configuration channels
# from AIX specific qprt to Linux specific lp commands.
# Released uner the GNU GPLv2

for I in `ls -1`
do
    sed 's/lp -d/lp -d/g' $I > tmp
    if [ $? -eq 0 ]
    then
        mv -f tmp $I
    else
        echo $I "did not complete"
        exit
    fi
done

Now it was time to write a short script to convert the /etc/qconfig printer configuration file from AIX and add the printers on the Linux server. So I wanted to test adding and deleting a printer from cups on the command line. I thought the following command would work.

# lpadmin -p misp1 -E -v socket://misp1.circus.org -m laserjet
# lpadmin: Unable to copy PPD file!

If all else fails read the man page, or in this case the manual on the web. It’s pretty cryptic but they tell you that the -m model has to be from the model directory. Where is the model directory?

# find / -name model
...
/usr/share/cups/model
# ls -1 /usr/share/cups/model/
deskjet2.ppd.gz
epson9.ppd.gz  
laserjet.ppd.gz

So we want the model from /usr/share/cups/model. The following command works:

# lpadmin -p misp1 -E -v socket://misp1.circus.org -m laserjet.ppd.gz

Now it’s time to convert from qconfig to cups. I used the following script to read in qconfig and create the printers. I had to add the sleep because the script got ahead of the lpadmin command. Once I added the sleep it worked and ta-da, 500 printers were created. I realize that printers are added and deleted from the AIX server daily so I had to make sure it would add and delete printers so that we will get them all on go-live day. It’s a rudimentary implementation but it works.

Notes on this script, it was easier than I expected, hence I a have hash instead of just working from the original array. It’s one of those things that I’m not going to go back and clean up for a simple script, sorry.

#!/usr/bin/perl

# 2010-02-18  Jud Bishop
# This script reads in the qconfig file of an AIX server and
# converts it to a Linux based cups file.  
# I am expecting this to be a quick hack...
# Released under the GNU GPLv2

# /etc/qconfig is the AIX printer configuration file.
#labp4:
#        device = @hpjd007
#        up = TRUE
#        host = hpjd007.circus.org
#        s_statfilter = /usr/lib/lpd/bsdshort
#        l_statfilter = /usr/lib/lpd/bsdlong
#        rq = PORT1
#@hpjd007:
#        backend = /usr/lib/lpd/rembak -T 30
open (FILE,"<etc-qconfig") or die "Error: can't open file $! \n";
        @aix = <FILE>;
close FILE or die "Error: can't close file $! \n";

my %table;
my $j = 0;
my ($que, $host, $port, $trash);
for (my $i = 0; $i <= $#aix; $i++) {
        if ( $j == 0 ){
                ($que, $trash) = split (/:/, $aix[$i]);
        } elsif ( $j == 3) {
                chomp($aix[$i]);
                $aix[$i] =~ s/ //g;
                ($trash, $host) = split (/=/, $aix[$i]);
                if ( $host !~ /circus.org/ )
                {
                        $host = sprintf ("%s.circus.org", $host);
                }
        } elsif ( $j == 6) {
                chomp($aix[$i]);
                ($trash, $port) = split (/=/, $aix[$i]);
        }  
        if ( $j == 6 ) {
                $table{$que} = {'host'=>$host, 'port'=>$port};
        }
        $j++;
        if ( $j == 9 ) {
                $j = 0;
        }
}

# The command to add a printer.
# system  lpadmin -p printer -E -v socket://host.circus.org -m laserjet.ppd.gz
# The command to remove a printer, got to be able to back it out.
# lpadmin -x printer
foreach $key (keys(%table)) {
        print "$key $table{$key}->{host} $table{$key}->{port}\n";

        my $socket = sprintf ("socket://%s", $table{$key}->{host});

        # Swap these two commands if you need to delete.        
        #my @args = ("lpadmin", "-x", "$key");
        my @args = ("lpadmin", "-p", "$key", "-E", "-v", "$socket", "-m", "laserjet.ppd.gz" );

        system(@args) == 0
        or die "system @args failed: $?";
        sleep 1;
}
Categories: Code, Linux Tags:

Symantec Linux Backup Client

February 19th, 2010 jud No comments

It’s frustrating when a vendor provides an install script that does not work. Especially when they take the time to print a bunch of fancy output across the screen about how it is working. For those out there this is how you install the Symantec backup client on Linux. This is using CentOS 5.4.

Even though the script says it completed successfully, the install failed because it needed some older libraries.

# yum provides libstdc++-libc6.2-2.so.3
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * addons: centos-distro.cavecreek.net
 * base: www.gtlib.gatech.edu
 * extras: mirrors.igsobe.com
 * updates: mirrors.adams.net
compat-libstdc++-296-2.96-138.i386 : Compatibility 2.96-RH standard C++ libraries
Repo        : base
Matched from:
Other       : libstdc++-libc6.2-2.so.3

compat-libstdc++-296-2.96-138.i386 : Compatibility 2.96-RH standard C++ libraries
Repo        : installed
Matched from:
Other       : Provides-match: libstdc++-libc6.2-2.so.3

# yum install compat-libstdc++-296-2.96-138.i386
# rpm -ivh VRTSralus-10.00.5629-0.i386.rpm
# rpm -ivh VRTSvxmsa-4.2.1-211.i386.rpm

# chkconfig VRTSralus.init on
service VRTSralus.init does not support chkconfig

Looks like we need to make it chkconfig compatible because it is a whole lot easier than creating a bunch of symlinks by hand. Add this to the top of /etc/init.d/VRTSralus.init

# vim /etc/init.d/VRTSralus.init
#
# VRTSralus.init      Start Symantec Backup
#
# chkconfig: 2345 08 95
# description:  Starts and stop backup.
#

And make sure it works.

# chkconfig VRTSralus.init on
# /etc/init.d/VRTSralus.init stop
# /etc/init.d/VRTSralus.init start
Categories: Linux Tags:

TSHOOT Exam Topology

February 19th, 2010 jud 4 comments

One of the Cisco guys has posted the topology for the new TSHOOT exam over at the Cisco Learning Network. Looks like I’ll be recabling the lab this weekend.

Categories: CCNP TSHOOT, Routing Tags:

Parsing Barracuda Log Files

February 17th, 2010 jud No comments

We have a Barracuda web filter here at the Circus. In general we are pleased with its performance, however, as our internet usage has climbed the days of history has declined. The inverse relationship is due to a “ring buffer” of 250,000 entries in the history log. There are times when we have a few hours of history and that doesn’t sit well when a manager wants to see the browsing history for a user or a pc. So we turned on syslog logging to a remote logging server and evaluated the log parsing packages that mentioned they parsed Barracuda logs. Let me give you a hint, there are not many.

My first hack was just to see what was going on and give a rudimentary understanding, it can be downloaded here.

!#~/bin/bash
grep $1 /var/log/barracuda.log | grep http_scan | awk '{print $28}' | sort | uniq -c | sort -n

Which gives output like this:

     63 autotrader.com
     72 google.com
     78 edmunds.com
     81 ad.doubleclick.net
     82 charter.com
    121 dealer.com
    138 alagasco.com
    194 synacor.com
    334 charter.net

The basis of that script gave me more understanding to write the following script which can be downloaded here.

#!/usr/bin/perl -w

use Getopt::Long;
use Number::Format;
use Tie::IxHash;


###############################################
# 2010-02-05 Jud Bishop
# This script parses /var/log/barracuda.log.
# Released under the GNU GPLv2.
###############################################
# Options that can be passed in:
# -u username
# -s source ip address
# -d destination url
# -c category
# -days number of days
# -pc the pcname
###############################################
# The format of a log file message.
# Used this document:
# http://www.barracudanetworks.ca/download/barracuda-web-filter-syslog.pdf
# And this one liner:
# tail -50 /var/log/barracuda.log | awk '{print $28}' && tail -1 /var/log/barracuda.log
#
# 0.  month
# 1.  day
# 2.  time
# 3.  barracuda_ip
# 4.  http_scan[process_id]
# 5.  md5sum
# 6.  number1
# 7.  source_ip
# 8.  destination_ip
# 9. content_type
# 10. source_ip
# 11. destination_address/URL
# 12. data_size
# 13. BYF
# 14. action (ALLOWED, BLOCKED, DETECTED)
# 15. reason (CLEAN, VIRUS, SPYWARE)
# 16. format_ver == 2 (Version of the policy engine output.)
# 17. match_flag (Whether an existing policy matched the traffic: 1=Yes, 0=No.)
# 18. tq_flag (Time qualified flag; 1=Yes 0=No.)
# 19. action_type (The documentation for this flag is incorrect.)
# 20. source_type (0=Any source, 1=group, 2=ipv4addr, 3=login, 4=auth_user, 5=min_score)
# 21. src_detail Detail related to matched source or "(-)" if not.
# 22. dest_detail If there is a destination match, what type (0=any, 1=particular category, 2=any category
#                 3=domain, 4=mimetype, 5=spyware, 6=uri_path_regex, 7=uri_regex, 8=application)
# 23. dest_detail Matched category or "(-)" if not matched.
# 24. spyware (If it is spyware, 0=allow, not spyware, 1=block, 2=infection.)
# 25. spyware_id (Name of the spyware if matched, if not "-".)
# 26. infection_weight Weight of the infection, mostly 0.
# 27. matched_part Part of the rule theat matched.
# 28. matched_category Comma delimited category name that matched traffic.
# 29. user_name Username, ([ANON], [ldap0:jud], [username:jud])

###############################################
# Variables you can change.
###############################################
my $debug = 0;
my $log_file = "/var/log/barracuda.log";
#my $log_file = "/var/log/barracuda.test";

###############################################
# Variables you should NOT change.
###############################################
my $arg_username = 0;
my $arg_pc = 0;
my $arg_source_ip = 0;
my $arg_dest_url = 0;
my $arg_category = 0;
my $arg_days = 0;
my $arg_help = 0;
my %table; # holds all the data for each session.
tie %table, "Tie::IxHash";
my %categories; # holds all of the categories this person went to.
my $data_sum = 0; # holds the total of all data for a user.
my $session_sum = 0; # hold the number of sessions for a user.
# The names of these variable are so that I don't have to keep looking
# above to figure out what name is what item in the array.
my $user = 29;
my $md5sum = 5;
my $month = 0;
my $day = 1;
my $time = 2;
my $source_ip = 7;
my $destination_ip = 8;
my $url = 11;
my $data_size = 12;
my $action = 14;
my $part = 27;
my $category = 28;

# Reads the options passed in.
sub get_options {
        GetOptions(
                'help|?|h!' => \$arg_help,
                'u=s' => \$arg_username,
                's=s' => \$arg_source_ip,
                'd=s' => \$arg_dest_url,
                'c=s' => \$arg_category,
                'days=i' => \$arg_days,
                'pc=s' => \$arg_pc);

        if ($debug)
        {
                print "username == $arg_username\n";
                print "source_ip == $arg_source_ip\n";
                print "pc == $arg_pc\n";
                print "dest_url == $arg_dest_url\n";
                print "category == $arg_category\n";
                print "days == $arg_days\n";
                print "help == $arg_help\n";
        }

        if ($arg_help)
        {
                print "usage: user-report.pl -days number [-u usernname] [-s source-ip] [-d destination-url] ";
                print "[-c category] [-pc pc_name] [--help|-?]\n";
                exit;
        }
}

# Parses the logs.
# Days equals log file days, makes it easy.
sub parse_logs {

        my ($search_field, $search_equals, $days) = @_;

        if ($debug){
                print "--------------------\n";
                print "parse_logs\n";
                print "search_field == $search_field\n";
                print "search_equals == $search_equals\n";
                print "days == $days\n";
        }

        # Loop through the log files based on number of days:
        # 0 == today
        # 1 == barracuda.log.1 one day past...
        # This is not formatted correctly because I added it as a retrofit.
        for (my $i = 0; $i <= $days; $i++)
        {
        if ($i == 0)
        {
                if ($debug) {print "open $log_file\n"};
                open (FILE, $log_file) or die "Error: can't open file\n $! \n";
        } else {
                if ($debug) {print "open $log_file.$days\n"};
                my $file = sprintf ("%s.%s", $log_file, $days);
                open (FILE, $file) or die "Error: can't open file\n $! \n";
        }
        while (<FILE>)
        {
                chomp;
                # Makes split work like awk, don't believe the man page.
                my (@log) = split /\s+/;
                # The next check is to catch the following type messages.
                # Feb  9 06:51:09 last message repeated 8 times
                if (defined ($log[$search_field]) and ($log[$search_field] eq $search_equals))
                {
                        if ($debug){
                                print "$log[$search_field] $search_equals\n";
                                for (my $i = 0; $i <= $#log; $i++)
                                {
                                        print "log[$i] == $log[$i]\n ";
                                }
                        }
                        # Each session gets a different md5sum, which is why it is the key in the table.
                        if( not exists $table{$log[$md5sum]}){
                                $table{$log[$md5sum]} = {'user'=>$log[$user], 'month'=>$log[$month], 'day'=>$log[$day], 'time'=>$log[$time], 'source_ip'=>$log[$source_ip], 'destination_ip'=>$log[$destination_ip], 'url'=>$log[$url], 'data_size'=>$log[$data_size], 'action'=>$log[$action], 'category'=>$log[$category], 'total_data'=>$log[$data_size], 'session_count'=>1 };

                                $data_sum += $log[$data_size];
                                $session_sum += 1;
                                if ( not exists $categories{$log[$category]} )
                                {
                                        $categories{$log[$category]} = 1;
                                } else {
                                        $categories{$log[$category]} += 1;
                                }

                                if($debug){
                                        print "does not exist\n";
                                        print "log user = $log[$user] \n";
                                        print "table user = $table{$log[$md5sum]}->{user} \n";
                                        print "sessions = $session_sum\n";
                                        print "bandwidth = $data_sum\n";
                                }
                        } else {
                                $table{$log[$md5sum]}->{total_data} += $log[$data_size];
                                $data_sum += $log[$data_size];
                                if($debug){
                                        print "exists \n";
                                        print "bandwidth = $data_sum\n";
                                }
                           }
                    }      
        }
        close FILE or die "Error: can't close file\n $! \n";
        }
}

sub print_report {

        if ($debug) {print "print_report\n";}
        if ($arg_username)
        {
                print "Useage report for: $arg_username\n";
        } elsif ($arg_pc) {
                print "Useage report for: $arg_pc\n";
        } elsif ($arg_source_ip) {
                print "Useage report for: $arg_source_ip\n";
        }
        print "Number of sessions: $session_sum\n";

        my $x = new Number::Format;
        $formatted = $x->format_bytes($data_sum);
        print "Total bandwidth consumed: $formatted\n\n";

        foreach my $category (sort (keys(%categories)))
        {
                printf "%s\n", uc($category);

                foreach $key (keys(%table))
                {
                        if ( $table{$key}->{category} eq $category)
                        {
                                my $url = substr($table{$key}->{url}, 0, 35);
                                print "$table{$key}->{month} $table{$key}->{day} $table{$key}->{time} $url $table{$key}->{action}\n";
                        }
                }
                print "\n";
        }
}

###############################################
# main
###############################################
my $search_field;
my $search_equals;

get_options();

if ($arg_username or $arg_pc) {
        $search_field = $user;
        if ($arg_username eq "ANON"){
                $search_equals = $arg_username;
        } else {
                $search_equals = sprintf("[ldap0:%s]", $arg_username);
        }
} elsif ($arg_source_ip) {
        $search_field = $source_ip;
        $search_equals = $arg_source_ip;
} elsif ($arg_dest_url) {
        $search_field = $url;
        $search_equals = $arg_dest_url;
} elsif ($arg_category) {
        $search_field = $category;
        $search_equals = $arg_category;
}
if ($debug) {print "search_field == $search_field\n"}

parse_logs($search_field, $search_equals, $arg_days);

print_report();

This script produces output like the following:

Usage report for: circus-user
Number of sessions: 401
Total bandwidth consumed: 19.39M

ADVERTISEMENT-POP-UPS,GAME-MEDIA,CUSTOM-2
Feb 17 15:44:17 http://games.mochiads.com/c/p/the-r ALLOWED

AUCTIONS
Feb 17 17:08:21 http://rover.ebay.com/ar/1/56033/1? ALLOWED

AUCTIONS,MOTOR-VEHICLES,CUSTOM-1
Feb 17 17:13:44 http://edmunds.autotrader.com/js/jq ALLOWED
Feb 17 17:13:45 http://edmunds.autotrader.com/inc/g ALLOWED
Feb 17 17:13:46 http://edmunds.autotrader.com/inc/j ALLOWED
Feb 17 17:13:46 http://edmunds.autotrader.com/inc/j ALLOWED
Feb 17 17:13:47 http://edmunds.autotrader.com/dwr/i ALLOWED
Feb 17 17:13:50 http://edmunds.autotrader.com/no_ca ALLOWED

BUSINESS
Feb 17 15:09:08 http://www.statcounter.com/counter/ ALLOWED
Feb 17 15:10:15 http://www.alagasco.com/fw/_css/fle ALLOWED
Feb 17 15:10:16 http://www.alagasco.com/scripts/jFa ALLOWED
Feb 17 15:10:17 http://www.alagasco.com/fw/_js/flex ALLOWED
Categories: Code, Linux Tags: