Matthew Daly's Blog

On Friday of last week I unexpectedly got a text from Vodafone saying I was able to upgrade my phone early. I was pretty pleased about this as having been something of an Android early adopter, I was still using an early Android phone, namely my HTC Magic. While a fine phone when it was released, it was only the second Android phone to become available in the UK and was therefore a bit dated compared to newer devices. It has been upgraded to Froyo (albeit a cut-down custom build) but that did slow the phone down somewhat.

So as soon as I had the opportunity I had a good look around for a new one to replace it. Right from the start I had my eye on the HTC Desire Z. Much as I love touchscreen phones, it’s very often extremely handy to have a physical keyboard, and as I’ve found myself using ConnectBot to connect to my home server via SSH a lot, the keyboard-toting Desire Z immediately had an advantage over the touchscreen-only models. Ideally I didn’t want to change my plan, so I checked out the deals for HTC phones on the same plan, and the Desire Z happened to be the only one on the same plan, so it was a no-brainer.

I got the phone on Monday, and it is amazing. The keyboard is easy to use and works well, the phone is lightning fast, and the UI is spot-on – it has everything I love about Android on the Magic (like the great notification system) and more. In particular I love the RSS reader- it syncs with Google Reader, so if I have to wait for a train, I can at least read some feeds while I’m waiting.

One thing I’m hoping to get more use out of is SL4A. I had this on my Magic, but coding on a touchscreen phone is not easy! I’m hoping that with the Desire Z’s keyboard, this will be a lot more useful.

March 30th, 2011 by Matthew

While I was very pleased to get a proper Linux distro working on my Pogoplug, the Arch-based Plugbox Linux was never really my cup of tea. While it’s a fine distro, I always felt that Debian would have been a much better fit. Partly this is because Debian has established a strong history of being a solid, stable distro that would carry on working no matter what, whereas Arch is more bleeding-edge. Also, Debian has a colossal repository that included a lot of software I wanted that wasn’t in the Arch repositories and I couldn’t get to install or compile from source, such as procmail and Squirrelmail. Debian also has strong support for many different processor architectures, including armel. Finally, being an Ubuntu user on the desktop, Debian is a distro that feels much more familiar to me.

So I eventually gave up on running Plugbox Linux and took the opportunity of the release of Debian Squeeze to install it on my Pogoplug, thanks to this tutorial. With that done, I set about adding my favourite applications. Byobu is a really handy tool that makes GNU screen significantly more intuitive and useful, so that’s always one of the first things to go on, and one that I’d really missed in Plugbox. I’ve now gotten my mail server working again, with the addition of procmail as my mail filter and Squirrelmail to give me a web interface. I’ve also set up Leafnode on there as I’d really like to learn more about Usenet, and I’m beginning to get the hang of using slrn to read it.

It’s amazing how much running my own server has taught me about security. I was staggered to see the sheer number of attempts by script kiddies to connect via SSH to my Pogoplug, and it really made me start thinking about security in a way I’d never bothered beforehand. I’ve installed denyhosts to block atttempts to brute-force the password, and made sure I chose a good password. I’ve also set OpenSSH to listen on a different port, which should hopefully decrease the number of login attempts substantially (I presume most of these were just script kiddies scanning large blocks of IP addresses looking for hosts with port 22 open), and have disabled root login (as at right now my login is the only one that is allowed via SSH, so if anyone does bother to do a more thorough scan and try to connect to the port I’m running SSH on, they’ll need to guess my username AND password, and do so before denyhosts kicks them off – a pretty tall order).

The whole concept of “plug servers” is one I really like, and my experience with the Pogoplug has been extremely good – it’s an inexpensive and extremely hackable device that has been an absolute pleasure to use.

February 21st, 2011 by Matthew

For a while now I’ve wanted a home server of some description, the idea being that it was something I could use to run a web server for development purposes, and a mail server so I could have an offline backup for my Gmail account (considering how much I rely on it, it’s only prudent to plan for what might happen if Gmail went down), and whatever else I need. Also, I only have laptops at present so I liked the idea of having something I could leave on all the time and connect to remotely via SSH.

Around Christmas, I read a forum post by someone who’d bought a PogoPlug cheap from PC World and had hacked it into a web server using Plugbox Linux, an Arch-based Linux distro. Shortly afterwards, I went into a branch of Currys in Norwich, and they had one on sale (£20 off the RRP of £70), so I shelled out for it. I already had a load of USB flash drives lying around, and an 8GB one is big enough for what I had in mind. After all, I wasn’t going to be serving anything that demanding over it, so something small and low-powered should be fine.

This weekend I finally got round to getting it set up. The PogoPlug service is actually pretty good – if you’re unfamiliar with it, it’s basically a self-hosted version of Dropbox, where you buy the device, connect it to your router, attach up to 4 flash drives or hard drives via USB, then share the files stored on them easily across your home network or over the Internet. However, this wasn’t really what I wanted.

Installing Plugbox Linux wasn’t hard – I merely had to activate SSH from the PogoPlug’s control panel, connect and kill the hbwd process, then install a new bootloader to enable it to boot the new OS. Once that was done, it was a case of attaching a flash drive, ensuring it was correctly mounted and the filesystem was set up properly, then downloading the Plugbox Linux tarball and unpacking it on the flash drive, before rebooting into the new OS.

Once it was installed, it wasn’t too hard to get the hang of pacman. I’d prefer it to have been Debian-based as that’s what I’m most familiar with, but that’s just personal preference. After a little tinkering I now have Postfix and Dovecot working on there, as well as Apache (although it might make sense to switch to something lighter, such as lighttpd or Cherokee). I’ve given it a fully qualified domain name via a free subdomain at dyndns.org, and I can now access emails on there via IMAP. Outgoing email works fine too, so I can always set up a Perl script or two to notify me if anything goes wrong by sending an email to my Gmail account. I’ve set up fetchmail to pull emails from my Gmail account via POP3, so all my email is in the process of being backed up on there, and I can use my phone to access it via IMAP, or SSH in and read it with Mutt. Going forwards, I may install Squirrelmail as well to give me more options.

One thing I’m not too sure about – I couldn’t get incoming mails to work, and I’m unsure whether this is because it’s using a subdomain (the email address is basically matthew@mydomainname.dyndns.org) or Postfix is merely misconfigured. Is it possible to receive emails to a subdomain in this fashion?

Anyway, this is a really great little machine and it’s been lots of fun getting it set up. I have to say, though, I’m really disappointed with the range of home server and NAS products currently on the market. Most of the NAS systems offer very little in the way of functionality or customisability, and most of the home servers are a bit too big, powerful and expensive, and usually run Windows Home Server, which isn’t really my cup of tea.

What I’d like to see is a small home server with a couple of hard drive bays at most, and a Debian or Ubuntu-based OS with access to apt-get and tasksel, so it’s easy to install whatever you want from the repositories. Also, give it a web interface that’s simpler than Webmin and makes it quick and easy to set up common software, but offer an advanced option for those that want it. That would be a fantastic device for end users – if it made it easy to set up a UPnP server, a Firefly server, or a BitTorrent client, that would be really useful.

January 24th, 2011 by Matthew

Just to say I’ve changed the contact form I use on here. I always wanted to use one with a built-in CAPTCHA facility, as Disqus seems to have pretty much killed off the comment spam, but I was still getting it via the contact form. I’ve put off doing something about it till now, but it was getting out of hand so I’ve gone and found a new contact form. Let’s hope this kills the spam…

November 13th, 2010 by Matthew

Yesterday I realised that I had somehow managed to scatter Vim swap files all across the Dropbox folder I use to share Perl and Python scripts I’d written between several computers, and it would be a good idea to clear them up. I didn’t like the idea of using grep to search for them and manually deleting them, so I decided this was the ideal opportunity to write a Perl script to do it for me!

I came up with the following:

#!/usr/bin/perl -w

use strict;
use Cwd;

sub searchDir
{
    # Subroutine to scan a directory looking for Vim swap files
    # Get directory to read and current directory
    my $readdir = shift;
    my $startdir = cwd();

    # Change directory to the target one
    chdir($readdir) or die "Unable to open $readdir! $!\n";
    print "Scanning contents of directory $startdir\n";

    # Open the directory and grab the names of all the files and folders in it
    opendir(DIR, ".") or die "Unable to open current directory! $!\n";
    my @entries = readdir(DIR) or die "Unable to read directory! $!\n";
    closedir(DIR);

    # Loop through the files and folders in the directory
    foreach my $entry (@entries)
    {
        # Skip this one and the one above it in the filesystem hierarchy
        next if($entry eq ".");
        next if($entry eq "..");

        # If a file is a directory, call the searchDir subroutine recursively in order to scan it
        if(-d $entry)
        {
            searchDir($entry);
            next;
        }

        # Use a regular expression to check to see if the current file starts with a period, and ends with .swp - if it does, it's a Vim swap file
        if($entry =~ m/^\..*\.swp$/)
        {
            # Inform the user that a Vim swap file has been found and print out the path to it
            print "Found a Vim swap file!\n";
            my $swppath = cwd();
            print "It's the file $entry in $swppath.\n";
            my $fullpath = $swppath . "/" . $entry;
            print "The full path is $fullpath.\n";

            # Prompt the user to delete the file
            print "Do you wish to delete this file? (Y/N)\t";
            chomp(my $reply = <STDIN>);
            if($reply =~ m/y/i)
            {
                print "Deleting $fullpath...\n";
                unlink($fullpath);
            }
        }
    }

    chdir($startdir);
}

# Get directory to begin the search
print "Enter directory to start search: ";
chomp(my $beginSearch = <STDIN>);

# call searchDir to start the search
searchDir($beginSearch);

This isn’t perfect – it uses recursion to examine subdirectories, and when I ran it on my /home folder it somehow wound up in /sys on my Ubuntu machine and I ended up getting a deep recursion warning (a little research suggests this happens when it goes over 100 directories in). However, it seems to work fine for scanning individual folders in my /home directory, and that’s all I really wanted anyway.

I love how Perl makes writing this kind of simple script so easy. It’s a great language for that kind of systems administration task.

October 29th, 2010 by Matthew

In the time that I’ve been learning Perl, I’ve slowly grown to appreciate the strengths of the language more and more. There’s two things in particular that I like about Perl. Once that I really don’t think anyone is going to be surprised by is CPAN. It’s a fantastic resource – there are a huge quantity of Perl modules available for virtually any task under the sun, and they’re incredibly useful.

The other is just how good the documentation is – I’ve never considered myself to be someone who learns terribly well from Unix man pages, but perldoc seems to have very good documentation indeed, including that for CPAN modules. Also, it helps that if you don’t do well with the man page format, you have the option of running podwebserver and getting the documentation formatted as web pages.

To give an example, I’m particularly interested in all kinds of network programming, be it web development, IRC, Jabber or whatever, and I’d heard of the Net::IRC module so I decided to start using it to create a simple IRC bot (yes, I know I should really be using POE::Component::IRC instead!). Using the information gleaned from perldoc Net::IRC it was easy to get started writing a bot, and I’ve now come up with the following simple bot:

#!/usr/bin/perl -w

use strict;
use Net::IRC;

my $irc = new Net::IRC;
my $nick = "mattsbot";
my $server = "irc.freenode.net";
my $channel = "#botpark";
my $port = 6667;
my $ircname = "My wonderful bot";
my $owner = "mattbd"; 

sub on_connect
{
   my $self = shift;

   print "Joining $channel\n";
   $self->join($channel);
   $self->privmsg($channel,"Ready to go!");
}

sub on_disconnect
{
  my $self = shift;
  $self->join($channel);
  $self->privmsg($channel, "Sorry about that - dropped out for a sec.");
} 

sub on_join
{
  # Get the connection and event objects
  my ($conn, $event) = @_;

  # Get the nick that just joined
  my $newnick = $event->{nick};

  # Greet the new nick
  $conn->privmsg($channel, "Hello, $newnick! I'm a greeting bot!");
}

sub on_msg
{
  # Get the connection and event objects
  my ($conn, $event) = @_;

  # Get nick of messaging user
  my $messager = $event->{nick};

  # Respond negatively
  $conn->privmsg($messager, "Sorry, I'm just a bot. Please don't message me!");
}

sub on_public
{
  # Get the connection and event objects
  my ($conn, $event) = @_;

  # Get nick of messaging user
  my $messager = $event->{nick};

  # Get text of message
  my $text = $event->{args}[0];

  # Check to see if text contains name of bot - if so message the user negatively
  if($text =~ m/$nick/)
  {
    $conn->privmsg($channel, "Sorry, $messager,I'm just a simple bot!");
  }
}

my $conn = $irc->newconn(Nick =>$nick,Server=>$server,Port=>$port,Ircname=>$ircname);
$conn->add_global_handler('376', \&on_connect);
$conn->add_global_handler('disconnect', \&on_disconnect);
$conn->add_global_handler('msg', \&on_msg);
$conn->add_global_handler('join', \&on_join);
$conn->add_global_handler('msg', \&on_msg);
$conn->add_global_handler('public', \&on_public);
$irc->start();

Now, this bot isn’t exactly hugely capable – all it does is greet new joiners, and tell you to leave it alone if you try to talk to it, but it was pretty easy to code it, thanks to the documentation, and it’s a good base to build on. From here, it’s easy to extend the on_public and on_msg subroutines to deal with other messages – for instance, I could use a regular expression to look for “!respond” in the text of the message and if it’s found, respond with any appropriate text.

I’ve hard-coded the appropriate details into the script in this case to make it quicker and easier to test it, but it would be trivial to change it to either accept settings passed as arguments from the command line, or have it grab these from a separate text file.

My initial doubts about Perl are really wearing off. It’s a powerful language and one that, now I’ve picked up the basic syntax, I’m having little trouble getting work done with.

October 27th, 2010 by Matthew

Just a short post to say that today I did my CIW Perl Fundamentals exam, and passed it. Glad that’s out of the way!

October 19th, 2010 by Matthew

Here’s another simple Perl program I thought I’d share. It’s a very simple app for recording notes from the shell. As pasting the last one into WordPress didn’t work too well, it’s on my Dropbox account instead. I’ve used SQLite as the backend for the database (a no-brainer – I really like SQLite for little quick-and-dirty scripts like this) and it works pretty well.

At some point I may go back to this basic concept and expand on it a little – it would make sense to add a GUI to it, either by using a graphical toolkit such as Tk or turning it into a web app. I also like the idea of adapting it to use a non-relational database – something like CouchDB perhaps.

October 16th, 2010 by Matthew

After my initial struggles with Perl, I now think I’m really starting to get to grips with the language. I generally find it a pain when you have to learn by building small but basically useless scripts – I always do best when building something useful.

As one of the exercises for my studies I had to open a database connection to a Microsoft Access database, but I wanted to do the exercise in Ubuntu (I’ve always preferred using Unix-like operating systems for programming, and thanks to apt-get it’s a lot less grief installing additional libraries and modules as you need them) and couldn’t get Perl to connect to the database properly, so I resolved to export it to either MySQL or SQLite.

I was able to export it to MySQL in the end using mdbtools, but I wasn’t entirely happy with the end result. I resorted to re-exporting the data as a CSV file, then resolved to write a small Perl script to read the file, parse it using a regular expression to obtain the necessary information, then insert it into a new SQLite database.

Here’s what I came up with:

#!/usr/bin/perl -w

use strict;
use DBI;

my $db = "dbi:SQLite:backend.db";

if (!(-e "backend.db"))
{
   print "Database does not exist. Creating it...";

   # Create the database
   my $dbh = DBI->connect($db) or die "Error in connecting to database! $DBI::errstr";
   my $createdb = $dbh->do("CREATE TABLE CARS( ID INTEGER PRIMARY KEY, YEAR INTEGER, MAKE VARCHAR(30), MODEL VARCHAR(30), COLOR VARCHAR(30), PRICE INTEGER);");
   $dbh->disconnect();
}

open(READFILE, "cartable.csv");

while(<READFILE>)
{
   unless($_ =~ m/id,/)
   {
      m/\d+\,(\d{4})\,\"(\w+)\"\,\"(\w+)\"\,\"(\w+)\"\,\"(\d+)\"/;
      my $year = $1;
      my $make = $2;
      my $model = $3;
      my $color = $4;
      my $price = $5;

      my $dbh2 = DBI->connect($db) or die "Error in connecting to database! $DBI::errstr";
      my $insertdb = $dbh2->do("INSERT INTO CARS (YEAR, MAKE, MODEL, COLOR, PRICE) VALUES (\"$year\", \"$make\", \"$model\", \"$color\", \"$price\");");
      $dbh2->disconnect;
   }
}
close(READFILE);

print "Write completed!\n";
print "To demonstrate it works, we'll run a SELECT query against the database...\n";

# Read the database
my $readdb = DBI->connect($db);
my $dbselect = $readdb->prepare("SELECT * FROM CARS;");
$dbselect->execute;

# Print the results
print "ID\tYear\tMake\tModel\tColor\tPrice\n";
while(my @row = $dbselect->fetchrow_array)
{
   print "$row[0]\t$row[1]\t$row[2]\t$row[3]\t$row[4]\t$row[5]\n";
}

# Close the connection
$readdb->disconnect;

Apologies for the fact that the indentation doesn’t seem to have copied across from Vim very well (can anyone recommend a good WordPress plugin for displaying code, none of the ones I’ve tried seem to be any good?). It works well, and it’s also helped me grasp Perl’s database API better.

I think I’ve got a better idea now of what Python and Perl are best at and when to use each. Perl is a great language, but the fact that a lot of it is implicit makes it a little harder to pick up at first than Python – for instance, the default variable, which is quite a good idea, but takes a little getting used to. Its regex support is great, and I like the database API, but I would find it a lot harder to do any object-oriented programming in Perl than in Python (which I guess is why Moose exists). I’ve found Perl very useful for quick and dirty scripts and as a glue language, but for longer scripts Python seems the better choice.

October 15th, 2010 by Matthew

I own a lot of programming textbooks. I went through a long phase of buying ones about virtually any technology I was even remotely interested in, therefore I own loads of books about Perl, Ruby, Python, PHP and C, among others. Granted, with many of them I’ve done little more than flick through them (I find it’s hard to get round to learning things like that without some kind of plan, which was what made me eventually start doing a more formal course since it forced a plan on me), but I’ve seen quite a few.

But of course, not every textbook is equal. Some are great, truly seminal works that are raved about by well-known programmers. Examples include the Camel book (Programming Perl) and K & R (The C Programming Language). Others are rarely mentioned. But what makes a really good textbook? Here I’m going to list some of the attributes that I’ve found in my favourite and most effective programming texts, and that I think make for a good, effective and informative textbook that makes a good job of getting you up and running programming in a new language:

• Lots of working examples to enter – To learn to program, whether from scratch or in a brand new language, the best advice I’ve ever heard was that you need to read a lot of code, and write a lot of code. I find that, at least at the start, nothing helps me learn to code in a new language better than lots of examples for me to type in and run, in order to pick up the basic syntax and keywords of the language. After all, that’s how many people used to learn BASIC, by typing in listings from magazines, and it’s how you learn English as a child – you’re exposed to the language, and you copy it, then understanding comes later. One of the best examples of this is C for Dummies, All In One Desktop Reference, by Dan Godkin – it’s packed full of loads of great example programs to enter and run that demonstrate the basic concepts well in C.
• Maintains your interest by showing you how to do interesting things – Not many people are interested in learning a new programming language to do something tedious (that said, if someone already has to do something tedious, such as a task at work, teaching them how to write a program to do it for them may well be considerably more interesting for them than doing the task themselves, hence the popularity of scripting languages for automating dull tasks), so a good programming textbook needs to show the learner how to do something interesting. Games are an obvious example, but they can get a bit much – how many different versions of Hangman do people want to create? Simple web apps are also an option with many programming languages. If something needs to be more utilitarian, then if possible it should be genuinely useful for solving a problem (the programmer doesn’t necessarily need to have this problem, they just need to see how to create a program to fix it). Frivolous little scripts that do things like recite “99 Bottles of Beer” to demonstrate for loops have their place, but that place is near the start only – by the end a programmer wants to be able to write useful programs.
• Good exercises to stretch the reader - Many textbooks will have additional exercises for the reader at the end of each chapter that allow them to practice their skills and ensure they aren’t just copying a listing, but are genuinely capable of writing code from scratch in the language. These are effectively the “homework” assignments, and I’ve found that these can be far more important at teaching me how to use the language well for actual programming projects than the listings within the book.

These are my thoughts, but I’d be interested to read what other people think about this issue. What’s the best programming textbook you’ve ever used, and why do you like it? What do you think a good programming textbook should have?

August 22nd, 2010 by Matthew