Migrating Unit Tests from Selenium to Watir Webdriver

Posted by Jordon Phillips

Until recently, HiretheWorld did all it’s front-end testing using Selenium. Our tests were quite old and they were pretty slow and unstable, they would fail fairly often for no reason, and would pass if the test was run again. I was tasked with improving our front-end testing process, the options were to try to optimize our selenium tests or find a new testing framework. Since both would require going through every test one by one, I decided to do some research of what different frameworks were out there.

Truth is, there aren’t that many – the big two are Selenium and Watir, with a bunch of other, much smaller, less well supported frameworks like windmill.

Watir

We eventually decided to use watir because it has a very clean and straight forward api, and Watir tests are written in ruby, like our old selenium based tests were, so it made the transition easier. We used watir webdriver, which uses selenium webdriver to drive the browser. The main difference between using webdriver through watir rather than selenium is that watir’s api is easier to use and has more features, making it much easier to write clean, efficient tests. Watir also has a ton of documentation and examples, this cheat sheet as well as the watir-webdriver site were great resources.

Diagram showing how our front-end testing setup fits together - test-unit plus Watir, driven by a test script - talking to web-driver, which drives the browser, which then outputs to the screen, or into a virtual framebuffer

Diagram showing how our front-end testing setup fits together - test-unit plus Watir, driven by a test script - talking to web-driver, which drives the browser, which then outputs to the screen, or into a virtual framebuffer

Setting Up Watir

Setting up watir is easy, you just need ruby, and a few gems. Some additional downloads can make the whole process better, such as using xfvb to do headless testing (no browser window) and using chromedriver to use google chrome as your test browser, which will speed things up. Also, a unit testing framework is needed to make batch testing easier; we use the test-unit framework.

  • Install Ruby
  • Install Ruby Gems
    • Ubuntu: sudo apt-get install rubygems
    • Windows: comes with installer
  • Install xvfb (if you want headless tests, linux only)
    • Ubuntu: sudo apt-get install xfvb
  • Install the following gems: watir-webdriver, test-unit, headless (if you want headless tests)
    • Ubuntu: sudo gem install watir-webdriver etc...
    • Windows: gem install watir-webdriver etc...
  • Download chromedriver if you want to use google chrome for testing (recommended)
  • More detailed instructions here: http://watir.com/installation/

Setting Up Tests with Test-Unit

First you need to setup your test unit class in a new ruby file

require "rubygems"
gem "test-unit"
require "test/unit"
require "watir-webdriver"
 
class TestExample < Test::Unit::TestCase

Next we create the setup and teardown functions – these are the functions that test-unit will run before and after each test:

# setup is run before every test
def setup
  $browser = 'chrome' if $browser.nil?
  $site = 'http://test.localhost' if $site.nil?
 
  if $headless
    require 'headless'
    $headless = Headless.new
    $headless.start
  end
  if $browser == 'chrome'
    $b = Watir::Browser.new :chrome
  elsif $browser == 'firefox'
    $b = Watir::Browser.new :ff
  elsif $browser == 'ie'
    $b = Watir::Browser.new :ie
  end
 
  $b.goto $site
end
 
# teardown is run after every test
def teardown
  $b.close
  if $headless
      $headless.destroy
  end
end

The setup code will pick a default browser and site if none is specified, then create the browser and go to the specified site. The teardown will close the browser; it can be useful to take a screenshot here so you can see where your test failed:

# take screenshot of end of test, useful for failures/errors
time = Time.new
$b.driver.save_screenshot(File.dirname(__FILE__) + '/screenshots/' + @method_name + '_' + time.strftime('%Y%m%d_%H%M%S') + '.png');

We also add a bit of code at the start of the file, after the requires, to handle arguments so that it’s easy to pick your browser and turn headless on/off using command line arguments:

require "rubygems"
gem "test-unit"
require "test/unit"
require "watir-webdriver"
 
# check arguments for browser or headless specification
ARGV.each { |arg|
    if arg.downcase.include? 'chrome'
        $browser = 'chrome'
    elsif arg.downcase.include? 'firefox'
        $browser = 'firefox'
    elsif arg.downcase.include? 'ff'
        $browser = 'firefox'
    elsif arg.downcase.include? 'ie'
        $browser = 'ie'
    elsif arg.downcase.include? 'headless'
        $headless = true
    end}
 
class TestExample < Test::Unit::TestCase

Writing Tests

For each test case you need a ruby function starting with the word test. Tests generally start with some manipulations of your site, followed by asserts to make sure everything behaved as expected. An example for our site is to fill in some of our forms, press the save button, then go back and see if everything is still there.

In test-unit you have three main assert functions, assert_true, assert_false and assert_equal. Your test will pass if the expression after the assert function give the expected outcome.

assert_equal 'Magic/More Magic', $b.text_field(:name, 'organization_name').value
assert_equal 'As mentioned above, we make magic and more magic.', $b.text_field(:name, 'question_38').value
assert_equal 'People who like magic and more magic, as opposed to less magic.', $b.text_field(:name,'question_39').value
assert_equal 'Im putting stuff into question 41', $b.text_field(:name,'question_41').value

Running Tests

You can run your tests just like running any ruby script, and you can pass parameters to which we can use to change things like the browser like above.

  • Run test:
    • ruby testexample.rb
  • Run test in headless firefox:
    • ruby testexample.rb headless firefox

Tips for writing good tests

The key to writing efficient tests is not to use explicit waits, our old tests were full of sleep functions to wait for pages to load or for ajax to load, this made them both slow and unstable. Instead, you should wait for an event, watir makes this very easy. I have a few other tips/tricks that I learned through the process of creating our tests as well as researching online.

Put everything you can in functions.

Putting everything you can in functions makes it easy to change repeated code, and when writing tests you will repeat things often. This will make it much easier to change your tests later while debugging or when you change your site.

def browse_to_new_project
  $b.goto $site + "/designtourney/projects/new"
end
 
def click_logo_design
  $b.link(:class, 'logo-design').click
end
 
def form_fill_first_page
  $b.text_field(:name, 'organization_name').set('Magic/More Magic')
  $b.text_field(:name, 'question_38').set('As mentioned above, we make magic and more magic.')
  $b.text_field(:name, 'question_39').set('People who like magic and more magic, as opposed to less magic.')
  $b.link(:id=> 'show-more').click
  $b.text_field(:name, 'question_41').set('Im putting stuff into question 41')
  $b.text_field(:name, 'question_45').set('Im putting stuff into question 45')
end

Waiting for Ajax

Watir will automatically wait for a page to load, but that won’t help you if your site uses a lot of ajax. A really handy function in watir for dealing with ajax loads is the wait_while_present function. It’s good practice to have an ajax loader animation displayed while your site loads an ajax call, using the wait_while_present function you can take advantage of this by waiting while the loader is present:

def wait_for_ajax
  $b.div(:id, 'ajax-loader').wait_while_present
end

Waiting for javascript animation

We use the fallr jquery plugin on HiretheWorld. It’s a slick looking animated modal box for prompts and alerts. These animations can cause trouble for automated tests. Using the wait functions in watir and a little javascript you can make this a little easier. I still had to use a short sleep here as waiting for a javascript function can be a little finicky, but it still helps make the test more stable.

$b.div(:id, 'fallr').wait_until_present
$b.wait_until{ $b.execute_script('return $(\'#fallr-wrapper\').is(\':animated\')') == false }
sleep 0.5
$b.link(:id, 'fallr-button-yes').click
$b.div(:id, 'fallr-overlay').wait_while_present

Dealing with timeouts

No matter how good you make your tests you will still occasionally get timeouts, you can make watir retry your timeouts using the ruby timeout class. I based this off of code I found here

def load_link(waittime)
  begin
    Timeout::timeout(waittime)  do
    yield
  end
  rescue Timeout::Error => e
    puts "Page load timed out: #{e}"
    retry
  end
end
 
def browse_to_new_project
load_link(30){ $b.goto $site + "/designtourney/projects/new" }
end
 
def click_logo_design
load_link(30){ $b.link(:class, 'logo-design').click }
end

Logging errors to a file

Another useful trick is to log your errors to a file. You can do this by overriding the test-unit run method. Do this at the top of your test script. I based this code off of this stackoverflow post

module Test
  module Unit
    class TestSuite
      alias :old_run :run
      def run(result, &progress_block)
        old_run(result, &progress_block)
        File.open('errors.log', 'w'){|f|
          result.faults.each{|err|
            case err
              when Test::Unit::Error, Test::Unit::Failure
                f << err.test_name
                f << "\n"
              #not in log file
              when Test::Unit::Pending, Test::Unit::Notification, Test::Unit::Omission
              end
          }
        }
      end
    end
  end
end

Retrying errors

Even with well written tests, you can still occasionally get random errors for no apparent reason. To help combat this, I created a script that will retry all the errors in the error log file from above. You can simply run this script after your tests if any errors happen, to see if they were repeatable errors.

# create string of all args
args = ""
ARGV.each { |arg| args+=" "+arg }
f = File.open("errors.log") or die "Unable to open file..."
 
# start with an empty array
errors=[]
f.each_line {|line|
  errors.push line
}
 
if errors.length > 0
  puts 'Attempting to resolve errors'
  try = 1
  while try <= 3
    puts "Try number: "+try.to_s
    errors.each_with_index{|name, i|
      test = /(.+?)\((.+?)\)/.match(name)
      if system "ruby \""+test[2]+".rb"+args+"\""
        errors[i] = false
      end
    }
    errors.delete(false)
    if errors.length == 0
      puts 'All errors resolved successfully!'
      break
    end
    try+=1
  end
  File.open('errors.log', 'w'){|f|
    errors.each{|error|
      f << error
      f << "\n"
    }
  }
  if errors.length != 0
    puts 'Errors unresolved'
  end
else
  puts 'There are no errors in errors.log'
end

Conclusion

Rewriting our tests turned out to be a great success for us. Not only are our tests a lot more stable now, but we also saw a 40% speed increase in running our entire testing suite! Below is the final code for this example:

require "rubygems"
gem "test-unit"
require "test/unit"
require "watir-webdriver"
 
# check arguments for browser or headless specification
ARGV.each { |arg|
    if arg.downcase.include? 'chrome'
        $browser = 'chrome'
    elsif arg.downcase.include? 'firefox'
        $browser = 'firefox'
    elsif arg.downcase.include? 'ff'
        $browser = 'firefox'
    elsif arg.downcase.include? 'ie'
        $browser = 'ie'
    elsif arg.downcase.include? 'headless'
        $headless = true
    end}
 
module Test
  module Unit
    class TestSuite
      alias :old_run :run
      def run(result, &progress_block)
        old_run(result, &progress_block)
        File.open('errors.log', 'w'){|f|
          result.faults.each{|err|
            case err
              when Test::Unit::Error, Test::Unit::Failure
                f << err.test_name
                f << "\n"
              #not in log file
              when Test::Unit::Pending, Test::Unit::Notification, Test::Unit::Omission
              end
          }
        }
      end
    end
  end
end
 
class TestExample < Test::Unit::TestCase
  # setup is run before every test
  def setup
    $browser = 'chrome' if $browser.nil?
    $site = 'http://test.localhost' if $site.nil?
 
    if $headless
      require 'headless'
      $headless = Headless.new
      $headless.start
    end
    if $browser == 'chrome'
      $b = Watir::Browser.new :chrome
    elsif $browser == 'firefox'
      $b = Watir::Browser.new :ff
    elsif $browser == 'ie'
      $b = Watir::Browser.new :ie
    end
 
    $timeout_length = 30
 
    load_link($timeout_length){ $b.goto $site }
  end
 
  # teardown is run after every test
  def teardown
    # take screenshot of end of test, useful for failures/errors
    time = Time.new
    $b.driver.save_screenshot(File.dirname(__FILE__) + '/screenshots/' + @method_name + '_' + time.strftime('%Y%m%d_%H%M%S') + '.png');
    $b.close
    if $headless
        $headless.destroy
    end
  end
 
  def browse_to_new_project
    load_link($timeout_length){ $b.goto $site + "/designtourney/projects/new" }
  end
 
  def click_logo_design
    load_link($timeout_length){ $b.link(:class, 'logo-design').click }
  end
 
  def form_fill_first_page
    $b.text_field(:name, 'organization_name').set('Magic/More Magic')
    $b.text_field(:name, 'question_38').set('As mentioned above, we make magic and more magic.')
    $b.text_field(:name, 'question_39').set('People who like magic and more magic, as opposed to less magic.')
    $b.link(:id=> 'show-more').click
    $b.text_field(:name, 'question_41').set('Im putting stuff into question 41')
    $b.text_field(:name, 'question_45').set('Im putting stuff into question 45')
  end
 
  def first_page_asserts type = 'regular'
    assert_equal 'Magic/More Magic', $b.text_field(:name, 'organization_name').value
    assert_equal 'As mentioned above, we make magic and more magic.', $b.text_field(:name, 'question_38').value
    assert_equal 'People who like magic and more magic, as opposed to less magic.', $b.text_field(:name,'question_39').value
    assert_equal 'Im putting stuff into question 41', $b.text_field(:name,'question_41').value
  end
 
  def wait_for_ajax
    $b.div(:id, 'ajax-loader').wait_while_present
  end
 
  def load_link(waittime)
    begin
      Timeout::timeout(waittime)  do
      yield
    end
    rescue Timeout::Error => e
      puts "Page load timed out: #{e}"
      retry
    end
  end
 
  def test_save_for_later
    browse_to_new_project
 
    click_logo_design
 
    form_fill_first_page
    $b.link(:class, 'save').click
    wait_for_ajax
 
    assert_true $b.div(:id, 'fallr').visible?
 
    browse_to_new_project
 
    $b.div(:id, 'fallr').wait_until_present
    $b.wait_until{ $b.execute_script('return $(\'#fallr-wrapper\').is(\':animated\')') == false }
    sleep 0.5
    $b.link(:id, 'fallr-button-yes').click
    $b.div(:id, 'fallr-overlay').wait_while_present
 
    # These assertions make sure the stuff for the first page is still all there
    first_page_asserts
  end
end

If you’ve got any questions about this, just leave a comment and we’ll try to help you out!

About Jordon Phillips

Jordon Phillips is a junior developer at HiretheWorld. A co-op student from SFU, he has a strong interest in anything relating to computer graphics or user interface and a growing knowledge of web development.
This entry was posted in Tech Blog and tagged , , . Bookmark the permalink.

2 Responses to Migrating Unit Tests from Selenium to Watir Webdriver

  1. Pingback: Watir Podcast #49 | Watir Podcast

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>