Of all the testing we can do, there’s nothing quite like firing up your browser and navigating to the website in question, logging in and doing stuff. All of the integration tests, port tests, contrived “I’m alive” tests etc can give you an *indication* that your website code is all well and good. However what you really want to test is “can I jump to it, do x y z action, and complete those actions without it bombing out”. We have a two options, we hire an army of minions who will do that for us every 15minutes on the dot OR we could do it automatically. How do we do this ?

Well there are a number of ways. One option I’m going to walk through is using the magic genius of Safari Watir, RSpec tests, ruby, a few gems, cron and any Mac you can lay your hands on. We set this up a while ago monitoring some of our sites and it’s great for finding out when something in your webstack has fallen over. For a < $1k investment it's a no brainer for any web site.

Caveat - this ain't going to test your flash/flex content - this is just for HTML based sites. Read on...

1. Machine setup

First, take one mac mini. We need to install ruby, ruby gems, rails, rspec, safariwatir, and rb-applescript bridge on it. Follow the Hivelogic narrative here for instructions on installing a full rails stack. Then just simply run these commands to finish off the last remaining gems.

  sudo gem install rspec
  sudo gem install safariwatir --include-dependencies

2. First website check

So we’ve got our infrastructure setup. Now we need to write the scripts that actually run the test the site. These will fire up safari and tell it to navigate to various URL’s, then check the content on the pages returned. So in true behaviour driven development style we write a spec that in plain english walks through the steps. It fires up a browser instance courtesy of Watir::Safari.new and uses it to navigate around pages. First let’s create a file called check_google.rb like so:

require 'rubygems'
require 'safariwatir'
 
context "With www.google.com" do
  before(:all) do
    @browser = Watir::Safari.new
  end
 
  it "can navigate to home page" do
    @browser.goto("http://www.google.com/")
  end
 
  it "homepage should not have a 404" do
    @browser.contains_text("404").should == nil
  end
 
  #it "should throw an contrived error, when seeing if yahoo is on main page" do
  #  @browser.contains_text("www.yahoo.com").should == true
  #end
 
  after(:all) do
    @browser.close
  end
end

Now if we run

spec check_google.rb

we get output as follows:

 RowanH$ spec check_google.rb
..

Finished in 2.818356 seconds

2 examples, 0 failures

Fantastic, it’s what we expect google is alive and tickety-boo. Now to check the failure scenario, uncomment the yahoo test:

RowanH$ spec check_google.rb
..F

1)
'With www.google.com should throw an contrived error, when seeing if yahoo is on g main page' FAILED
expected: true,
     got: nil (using ==)
./test_mysite.rb:18:

Finished in 2.837444 seconds

Cool, we know what a failure looks like.

3. Turning your console output into a email message

We’re not going to be logging in and doing this all the time, so we want to be emailed of success/failures. We are going to use a custom formatter for RSpec, which can make our output look a lot more interesting and pop it into an email. As I was running these scripts on a machine with a full rails stack I used ActiveMailer for sending emails.. (you’ll see why in a minute) so let’s create a custom formatter that will format our output for mailing. Lets call it mail_formatter.rb and pop it into the same place as our existing scripts - as follows.

require 'rubygems'
require 'action_mailer'
 
#actual email thingy
ActionMailer::Base.server_settings[:address] = 'domain@myhost.com'
class Emailer < ActionMailer::Base
  def test_email(text, status)
    subject    "TEST Autotest Status - #{status}"
    from       "mytestbox@myhost.com"
    recipients "me@myhost.com"
    body       text
  end
end
 
class MailFormat < Spec::Runner::Formatter::BaseTextFormatter
  def start(spec_count)
    @mail_output = String.new
    @mail_output << "Test started \n"
    @mail_status = "Tests OK"
  end
 
  def add_behaviour(name)
    @mail_output << "\n #{name}\n"
  end
 
  def example_failed(example, counter, failure)
    @mail_output << " - #{example.description} FAILED \n"
    @mail_status = "#{counter} tests FAILED"
  end
 
  def example_passed(example)
    @mail_output << " - #{example.description} PASSED \n"
  end
 
  def dump_summary(duration, spec_count, failure_count, something_else)
    @mail_output << "\n"
    @mail_output << "Finished in #{duration} seconds "
    @mail_output << "#{spec_count} test#{'s' unless spec_count == 1}, #{failure_count} failure#{'s' unless failure_count == 1}"
    Emailer.deliver_test_email(@mail_output, @mail_status)
  end
end

Right so this time if we a new command like so, to require the mail_formatter, and output in mail format, we should by rights receive an email:

 RowanH$ spec check_google.rb -r mail_formatter.rb -f MailFormat

1)
'With www.google.com should throw an contrived error, when seeing if yahoo is on main page' FAILED
expected: true,
     got: nil (using ==)
./test_mysite.rb:18:

(we still get our console output if we get a failure)

5. Automating it

Okay so wrap that lot up in a shell script, and call it via cron (hey I’m not going to write *everything* out here), and you’ve now got a simple script that’s calling up your website and checking it.

6. Extending it

If you browse around for some of the safari-watir examples out there then you’ll see more of how to access specific fields and named form elements - this will let you actually login and do stuff on your site(s) your testing.

7. Extra Credit - Screenshots!!

Okay, we’ve got our system emailing us every test. But what would be really cool is if it actually sent us a screenshot everytime something turned to custard and failed so we could see what actually went wrong. Well… it’s suprisingly easy. We’re going to use OSX’s inbuilt screencapture utility and fire off a shell command to capture a png file everytime something goes wrong. Then pop those in as attachments to our status email.
Modify the code like so:

In the Emailer.test_mail method, add a parameter images, and pop this code at the bottom of the method:

   if ( images )
      for image_name in images
        attachment :content_type => "image/png", :filename => image_name ,
                        :body => File.read("/Users/rowanh/Desktop/failed_images/#{image_name}")
      end
    end

So this will allow our email method to read in a list of images and attach them to mail message. Next we want to update our formatter.

For the start method add the following:

    @failed_image_count = 0
    @failed_images = Array.new

Add a new method like so :

  def take_screenshot()
    @failed_image_count += 1
    screenshot_name = "failure_#{@failed_image_count}.png"
    system "screencapture /Users/rowanh/Desktop/failed_images/#{screenshot_name}"
    @failed_images << screenshot_name
  end

Next update example_failed so it takes_screenshot.

Finally in the deliver_test_email method, pop in the @failed_images array.

Conclusion

Bingo! you now have a hugely useful tester, that runs away, tests your website, tells you and will actually show you what went wrong. For not a whole lot of investment.

Off to setup a continuous integration machine that actually does this for each build……..

Credit:
http://blog.aslakhellesoy.com/2006/12/2/getting-screenshots-from-watir
http://redsquirrel.com/cgi-bin/dave/projects/watir