full of sound and fury

Courage, Bravery and Heroism

03 June 2015

I see people trying to compare one act of courage, bravery or heroism to another. These are all ways to describe someone acting despite their fear, uncertainty and doubt.

While there are many actions that will automatically be described in that way, they are not the only ones. Running into a burning building to save someone else or standing in harms way to protect others are two examples that are easily identified. But for some people just leaving their home is overcoming fear. Speaking in front of an assembled group of ones peers. Standing up to a bully. Taking a stand for something you believe in, whether that be a religious, moral or ethical issue. Depending on the time and place, these could all be very brave and courageous acts.

Just because something is not a challenge for you, or you do not agree with what someone else is doing does not mean that for them it is not an act of courage, bravery or heroism. Don't dismiss or judge harshly someone else just because you don't think what they are doing is brave or you don't agree with them.

One day it will be you that is overcoming fear, uncertainty and doubt in a way that others do not understand or approve of.


Watching long processes through CGI

21 March 2015

After slamming my head against a wall for a few hours Friday trying to implement: Watching long processes through CGI

I gave up and left the office. The thing that was really frustrating was that it appeared to be working, and then suddenly, it didn't. And when I even went back and ran Randall's code without any modification, it still didn't work. I was stumped.

Fast forward 12 hours, after some food, some sleep and more searching on google. Here's a clue: It is possible that the original article was written for apache 1.x, not apache 2.x. Anyway, I found a post on perlmonks.org where someone was having a similar problem, and they indicated that explicitly closing STDERR and STDIN, as well as STDOUT let the script work as intended.

So here is my diff against the listing in Randall's original article from 2002.

--- col39.listing.txt   2015-03-21 02:01:08.260866239 -0500
+++ /usr/lib/cgi-bin/session_test.cgi  2015-03-21 03:03:34.656454963 -0500
@@ -34,6 +34,8 @@
       print redirect(self_url());
     } elsif (defined $pid) {    # child does
       close STDOUT;             # so parent can go on
+      close STDERR;
+      close STDIN;
       unless (open F, "-|") {
         open STDERR, ">&=1";
         exec "/usr/sbin/traceroute", $host;

Hopefully this saves some other poor schmuck from leaving a bloody dent in their desktop.


New Namecheap SSL Cert for w9jz.org

29 January 2015

This is mainly here just so that I can remember the steps to turn a namecheap ssl cert into a pem.

Generate the csr:

openssl req -new -newkey rsa:2048 -nodes -keyout w9jz.org.key -out w9jz.org.csr

Once the zip file is recieved from namecheap, unzip it and cat the files together.

cat w9jz_org.crt COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt AddTrustExternalCARoot.crt > w9jz.org.pem

Copy the files into place:

sudo cp w9jz.org.pem /etc/ssl/certs/w9jz.org.pem
sudo cp w9jz.org.key /etc/ssl/private/w9jz.org.key

In /etc/nginx/sites-enabled/w9jz.org:

ssl on;
ssl_certificate         /etc/ssl/certs/w9jz.org.pem;
ssl_certificate_key     /etc/ssl/private/w9jz.org.key;
ssl_ciphers             HIGH:!ADH:!MD5;
ssl_prefer_server_ciphers on;
ssl_protocols           SSLv3 TLSv1;
ssl_session_cache       shared:SSL:10m;
ssl_session_timeout     5m;


fixing the raid

17 July 2014

I noticed a couple weeks ago that I kept getting errors in the logs for one of the disks in the multimedia array. This was upsetting, but since I had paid a little more for these disks to get the 5 year warranty, I wasn't too concerned. Then I did a warranty check on the disk. It was the disk that WD sent me to replace the last disk that failed.

Oh goody.

After futzing with the WD support website, they had changed it since the last time I had to RMA a disk, I finally got an advance RMA setup. The disk arrived last week, but I was unable, until today, to get around to replacing it.

So I powered down the server, pulled the bad disk ( installed 2011-04-20 according to the label I put on it ) and put the new one in it's place. Then I rebooted. The system complained about a degraded array, was I sure I wanted to continue, even though I hit 'y', it dropped to a minimal recovery shell. sigh

Reboot, let's try this again. Same thing.

dirty words

Reboot again, hit 'e' to edit the boot paramemeters in GRUB, add 'bootdegraded=true' and voila, it boots.

Now to it was as simple as:
Copy the partition map from the existing drive...

# sfdisk -d /dev/sde | sfdisk/dev/sda

Add the new disk into the array:

# mdadm --manage /dev/md127 --add /dev/sda1

And check on the progress:

 # cat /proc/mdstat 
 Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
 [raid4] [raid10] 
 md127 : active raid1 sda1[2] sde1[0]
       1953511936 blocks [2/1] [U_]
       [>....................]  recovery =  3.8% (75987264/1953511936) finish=218.8min speed=142988K/sec

unused devices: <none>

Not too shabby!

# df -kh /multimedia/
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/raid--lvm-movies  1.8T  841G  882G  49% /multimedia

Now I just need to get the old drive back into the shipping box and get it back to WD so they don't charge me $220 for the replacement disk.


PECL::Perl and Net::Telnet

30 January 2014

Well, I wasn't going to, but I decided to install the PECL::Perl extension for PHP afterall. So far it seems to work with the two modules that were proving to be difficult to find native PHP replacements for, Text::Diff and Net::Telnet.

Here is an example of using it with Net::Telnet:

<?php

$perl = new Perl();
$perl->eval('use Net::Telnet');

$t = $perl->eval(<<<'PERL_END'
new Net::Telnet(
    Prompt => '/\S+[#>][ ]?$/'
);
PERL_END
);

$t->open( '172.31.38.65' );
$t->login( $username, $password );

$lines = $t->array->cmd( 'show ver' );

print_r( $lines );
?>

I am sure that there are people that are going to hate this solution :)


PHP and networks

28 January 2014

Well, I did not install the perl PHP extension. I think my friend Benjamin summed it up the best when he responded to my post about it on facebook:

Surround your pullquote like this {" text to be quoted "}

So now I am back to looking for a good way to do diffs in PHP of files that are sometimes 75,000+ lines. Either my google-fu has improved and I have not seen some of these solutions before, or it has not and I blocked out the memory of having tested some of these solutions. We'll see which it is when I get into the office tomorrow.

Also, the other special kind of fun that I was trying to find a solution for was a replacement for the perl module Net::Netmask. Thanfully I found a github gist that had just what I was looking for!

Reproduced here are the ones that I adapted for what I needed, mainly so I can find them again later.

<?php

# REFERENCE - http://stackoverflow.com/questions/2942299/converting-cidr-address-to-subnet-mask-and-network-address

$ipNetmask = "192.168.1.12/16";
list($ip, $netmask) = split( "/", $ipNetmask );
$ip_elements_decimal = split( "[.]", $ip );
$netmask_result="";
for($i=1; $i <= $netmask; $i++) {
  $netmask_result .= "1";
}
for($i=$netmask+1; $i <= 32; $i++) {
    $netmask_result .= "0";
echo $netmask_result;
}
$netmask_ip_binary_array = str_split( $netmask_result, 8 );
$netmask_ip_decimal_array = array();
foreach( $netmask_ip_binary_array as $k => $v ){
    $netmask_ip_decimal_array[$k] = bindec( $v ); // "100" => 4
    $network_address_array[$k] = ( $netmask_ip_decimal_array[$k] & $ip_elements_decimal[$k] );
}
$network_address = join( ".", $network_address_array );

print_r($network_address);
?>
<?php
 
# REFERENCE http://stackoverflow.com/questions/594112/matching-an-ip-to-a-cidr-mask-in-php5
 
function cidr_match($ip, $range)
{
    list ($subnet, $bits) = split('/', $range);
    $ip = ip2long($ip);
    $subnet = ip2long($subnet);
    $mask = -1 << (32 - $bits);
    $subnet &= $mask; # nb: in case the supplied subnet wasn't correctly aligned
    return ($ip & $mask) == $subnet;
}
 
if (cidr_match("10.128.0.0", "10.128.1.0/16")) {
    echo 'match';
} else {
    echo 'no match';
}
 
echo cidr_match("10.128.0.0", "10.128.1.0/16");

?>
<?php
 
function mask2cidr($mask) {
 
$mask = split( "[.]", $mask );
 
$bits = 0;
 
foreach ($mask as $octect) {
    $bin = decbin($octect);
    $bin = str_replace ( "0" , "" , $bin);
    $bits = $bits + strlen($bin);
}
    return $bits;
}
 
echo mask2cidr("255.255.252.0");
 
?>


Here we go again

26 January 2014

I just can't seem to leave well enough alone.

Well, really, I kind of blame the current projects that I am working on for my employer. It's drawing me back into the world of webdesign. And while I wasn't unhappy with the octopress site that I was using, it felt a little too restrictive. Yeah, the guy that just wanted to be about to write on his blog decided to retool it. Again.

This time I am using vanilla jekyll but with bootstrap and jquery and fontawesome.

I'm not sure if my coworker is going to be happy that I found this stuff, and of course, now want to use them in the tools we are building...

Man, if only these tools would have been available ~16 years ago...


perl in PHP

24 January 2014

One of the projects that I am working on uses perl CGI scripts to do various things with a database and also getting information directly from routers and switches. One reason I did this in perl was because it was easier to port some of my existing commandline scripts to CGI.

The other reason was because I really did not want to use PHP to telnet to a router. I have seen the code that people use to do that, and it scares me.

So while poking around and researching what we would need to do to call perl scripts from php, in order to consolidate some of our development and get a little more speed... I ran across this link.

This has the potential to allow me to use perl for some of the things that I need, and php for the rest. I am pretty sure this is going to cause my coworker that is working on these projects with me, to go into a fit. :)

Next week I will test using Net::Telnet, Net::Netmask and Text::Diff from inside php... what's the worst that could happen?!

<?php
$perl = new Perl();
$perl->eval( 'use Net::Ping;' );
$ping = $perl->eval('Net::Ping->new');
foreach ( array( '172.27.1.3', '172.27.1.1' ) as $ip ) {
    if ( $ping->ping($ip) ) {
        echo "$ip is alive!\n";
    }
}
?>


Experimenting with nginx

18 January 2014

Seeing as how nginx is gaining popularity, partially due to it's performance on small virtual machines, aka virtual private servers, I decided to move my some of my personal sites to an nginx host from my current apache2 solution.

While I was in the process of doing this, I thought it would be handy to be able to create a new subdomain on the fly. There is already a wildcard configured in DNS, so now I just needed a wildcard setup in nginx.

This configuration checks for the existence of the directory named for the subdomain, if it doesn't exist, it redirects to www.domain.tld.

In theory, if all domains were in a standard base directory, I could have one configuration file that would route any website to the correct location. Maybe I'll experiment with that later.

And yes, I know that this kind of thing can be done with apache2 as well.

server {
    index index.html index.htm index.php;
    set $basepath "/home/username/";
    server_name ~^(?<sub>.+?)\.domain\.tld$;

    set $root $basepath/$sub.domain.tld;

    if ( !-d $root ) {
        rewrite ^ $scheme://www.domain.tld$uri redirect;
    }

    root $root;

    location / {
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.
        try_files $uri $uri/ /index.html;
    }
    # pass the PHP scripts to php-fpm
    #
    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;
    }

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    location ~ /\.ht {
    deny all;
    }
}


fail2ban + ansible

07 October 2013

I use fail2ban to slow down and stop brute force attempts against my servers. One drawback of the way it is currently implemented is that if they are really persistant, attackers keep coming back.

If they have managed to trip fail2ban multiple times, that means they have either attacked two different servers, or have come back to the same one multiple times. This means that I am quite happy to permanently ban them from access to my systems.

This requires some setup on the servers in question.

auto lo
iface lo inet loopback

pre-up iptables-restore < /etc/firewall-rules
post-up iptables-restore -n < /etc/iptables.blocklist


*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FIREWALL - [0:0]
:blocklist - [0:0]
-A INPUT -j blocklist
-A INPUT -j FIREWALL
-A FORWARD -j FIREWALL
-A FIREWALL -i lo -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 25 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 81 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 110 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 143 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 443 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 465 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 587 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 993 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 995 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 4650 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 4949 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 22 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p udp -m udp --sport 53 -j ACCEPT
-A FIREWALL -p tcp -m tcp --sport 53 -j ACCEPT
-A FIREWALL -p udp -m udp --dport 53 -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 53 -j ACCEPT
-A FIREWALL -p udp -m udp --dport 123 -j ACCEPT
-A FIREWALL -p udp -m udp --sport 123 -j ACCEPT
-A FIREWALL -p udp -m udp --sport 6277 -j ACCEPT
-A FIREWALL -p udp -m udp --sport 24441 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 0 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A FIREWALL -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j REJECT --reject-with icmp-port-unreachable
-A FIREWALL -p udp -m udp -j REJECT --reject-with icmp-port-unreachable
-A FIREWALL -p icmp -j DROP
COMMIT

The file /etc/iptables.blocklist starts out pretty baren.

*filter
:blocklist - [0:0]
COMMIT

It took me a little while to figure out what the bare minimum needed configuration was to be able to 'source' an iptables file.

I use two ways to update the /etc/iptables.blocklist file. One is to use a short shell script, which also has an option to kick off the ansible update as well.

The append.sh script typically takes two arguments, append.sh -i <IPADDRESS> will add the IP before the COMMIT line at the end. append.sh -p will 'push' the configuration using ansible. There is one other option append.sh -f <FILENAME> -i <IPADDRESS> to specify a different blocklist filename.

The other is to use a perl script that parses the subject lines for all of the email that I get from fail2ban looking for the ip address in the subject line.

#!/bin/dash

# Copyright (c) 2013, grephead.com, LLC
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#     * Redistributions of source code must retain the above copyright
#       notice, this list of conditions and the following disclaimer.
#     * Redistributions in binary form must reproduce the above copyright
#       notice, this list of conditions and the following disclaimer in the
#       documentation and/or other materials provided with the distribution.
#     * Neither the name of grephead.com, LLC nor the names of its
#       contributors may be used to endorse or promote products derived from
#       this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY grephead.com, LLC "AS IS" AND ANY EXPRESS OR
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL grephead.com, LLC BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

# append an IPv4 address to the blocklist iptable
# pushing the changes using ansible

# limited PATH
PATH=/bin:/usr/bin

# default filename
filename='iptables.blocklist'

while getopts i:f:p f
do
    case ${f} in 
        i)  ipaddress=${OPTARG} ;;
        f)  filename=${OPTARG} ;;
        p)  push=1 ;;
    esac 
done

# check to see if ${ipaddress} is zero length, not specified.
if [ -z ${ipaddress}  ];then
    echo 'Missing IP address, not updating file'
else 

    # check to make sure that the IP is not already blocked
    # if not, insert before COMMIT string

    if [ -f ${filename} ]; then
        if grep -q -- "-A blocklist -s ${ipaddress} -j DROP" ${filename}; then
            echo "${ipaddress} already blocked"
        else

sed -i.bak -e "/COMMIT/i \
-A blocklist -s ${ipaddress} -j DROP" iptables.blocklist

        fi
    else
        echo "${filename} does not exist"
    fi
fi

# if -p is specified, push updated blocklist using ansible playbook
if ( [ "${push}" = "1" ] && [ -f blocklist_playbook.yml ] )
then
    ansible-playbook blocklist_playbook.yml
fi

The perl script uses File::Slurp and Mail::Box modules.

#!/usr/bin/perl

# The MIT License (MIT)
# 
# Copyright (c) 2013 Matt Okeson-Harlow
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to
# deal in the Software without restriction, including without limitation the
# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
# sell copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# 
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.

use strict;
use warnings;
use Mail::Box::Manager;
use File::Slurp qw( :std :edit );

my ( %ip_lookup, %ip_count, @append );
my $count = 0;

## Directory and file locations
my $fail2ban_folder = qq{$ENV{HOME}/Maildir/.grephead.fail2ban};
my $trash_folder    = qq{$ENV{HOME}/Maildir/.Trash};
my $iptables        = qq{$ENV{HOME}/btsync/ansible/iptables.blocklist};

## mailbox handlers
my $mgr    = Mail::Box::Manager->new;
my $folder = $mgr->open( folder => $fail2ban_folder, access => q{rw} );
my $trash  = $mgr->open( folder => $trash_folder, access => q{rw} );

## Slurp iptables rules into an array
my @blocklist = read_file $iptables;

## process each message in the fail2ban folder, looking for 'banned $ip'
for my $email ( $folder->messages ) {
    my $subject = $email->subject;

    # [Fail2Ban] ssh: banned 122.226.109.178
    if ( $subject =~ m{banned \s+ (\d+[.]\d+[.]\d+[.]\d+)}xms ) {
        my $ip = $1;
        $ip_count{$ip}++;
        push @{ $ip_lookup{$ip} }, $count;
    }
    $count++;
}

## look for $num instances of an IP address, add it to the append
## array and move the related email to the trash.
for my $ip ( keys %ip_count ) {
    if ( $ip_count{$ip} > 1 and not grep /$ip/, @blocklist ) {
        print qq{$ip_count{$ip}\t$ip\n};
        my $iptables_line = qq{-A blocklist -s $ip -j DROP};
        push @append, $iptables_line;
        for my $index ( @{ $ip_lookup{$ip} } ) {
            print $index . qq{\t}
                . $folder->message($index)->subject . qq{\n};
            $mgr->moveMessage( $trash_folder, $folder->message($index) );
        }
    }
}

## find 'COMMIT' in iptables file, insert new rules before it
my $append_string = join "\n", @append;
edit_file sub {s/(COMMIT)/$append_string\n$1/g}, $iptables;

After the fail2ban_filter.pl script is run, the append.sh -p script needs to be run to push the changes using ansible or run the ansible-playbook blocklist_playbook.yml command.

[grephead]
vader.grephead.com
yoda.grephead.com
leia.grephead.com
# push out updated blocklist iptables rules and add them to the running
# firewall

# 2013/04/16 02:17:35 

---
- hosts: grephead
  sudo: yes
  connection: ssh

  tasks:
    - name: update iptables.blocklist
      action: copy src=iptables.blocklist dest=/etc/iptables.blocklist owner=root group=root mode=0644

    - name: merge into iptables
      action: shell /sbin/iptables-restore -n < /etc/iptables.blocklist

# vim: set ts=2:sw=2:sts=2:ft=yaml