Evolution of an Automated Test

For awhile now I have been advocating that building an automated test is a four phase process. The phases are:

  1. Record
  2. Add checks
  3. Data drive
  4. Make it smart

In this post I’ll illustrate these steps in automating our One Minute Calculators using Selenium. Because this is tutorial in nature, not all aspects are automated; just enough to illustrate.

Record

The first step in a new script is to record it’s basic actions. Yes, you could do this by hand, but then you have to know what all your object id’s are etc. which is often not easy to get right. Or if your tool doesn’t give you the ability to record, hopefully you are able to use an existing script as a template. The goal of this set is get the basic structure in place as fast as possible. This is the script that Selenium-IDE created (which a bit of formatting cleanup):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head profile="http://selenium-ide.openqa.org/profiles/test-case">
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <link rel="selenium.base" href="" />
    <title>New Test</title>
  </head>
  <body>
    <table cellpadding="1" cellspacing="1" border="1">
      <thead>
        <tr>
          <td rowspan="1" colspan="3">New Test</td>
        </tr>
      </thead>
      <tbody>
        <tr>
        	<td>open</td>
        	<td>/one_minute/earthhour</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>link=more</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>instance_answers_bcartype_8090754_radio_1</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>instance_answers_bcartype_8090754_radio_2</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>instance_answers_bcartype_8090754_radio_3</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>instance_answers_bcartype_8090754_radio_4</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>instance_answers_bcartype_8090754_radio_5</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>instance_answers_bcartype_8090754_radio_6</td>
        	<td></td>
        </tr>
      </tbody>
    </table>
  </body>
</html>

All the script does is goto a url, click a button then select each of the 6 radio buttons in order, top to bottom. Unfortunately because of how the radio button id’s are constructed, this script cannot be run more than once without failing to find buttons.

A second part of the Record step is making sure you can run the recorded script more than once, so we have to pull out some XPath wizardry to find the labels minus the dynamic part. I found the pattern I used here but I am sure there are multiple ways to solve the same problem. This is what the script looks like now that it can be run multiple times and have every step execute properly.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head profile="http://selenium-ide.openqa.org/profiles/test-case">
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <link rel="selenium.base" href="" />
    <title>New Test</title>
  </head>
  <body>
    <table cellpadding="1" cellspacing="1" border="1">
      <thead>
        <tr>
          <td rowspan="1" colspan="3">New Test</td>
        </tr>
      </thead>
      <tbody>
        <tr>
        	<td>open</td>
        	<td>/one_minute/earthhour</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>link=more</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>//input[contains(@id, &quot;instance_answers_bcartype_&quot;) and contains(@id, &quot;_radio_1&quot;)]</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>//input[contains(@id, &quot;instance_answers_bcartype_&quot;) and contains(@id, &quot;_radio_2&quot;)]</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>//input[contains(@id, &quot;instance_answers_bcartype_&quot;) and contains(@id, &quot;_radio_3&quot;)]</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>//input[contains(@id, &quot;instance_answers_bcartype_&quot;) and contains(@id, &quot;_radio_4&quot;)]</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
          <td>//input[contains(@id, &quot;instance_answers_bcartype_&quot;) and contains(@id, &quot;_radio_5&quot;)]</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
          <td>//input[contains(@id, &quot;instance_answers_bcartype_&quot;) and contains(@id, &quot;_radio_6&quot;)]</td>
        	<td></td>
        </tr>
      </tbody>
    </table>
  </body>
</html>

Add Checks
A script that runs end-to-end without falling provides value in that all the steps were not blocked. However, thats about that end of its usefulness from a testing perspective. What we really want our tests to do is test which means they have to check for things as they go along. Every script’s thing will be different.

In this case I want to make sure that the number at the bottom (tonnes of carbon) at the column changes when a different radio button is checked. To do this I am getting the existing value, then clicking a radio button and checking that the value is different. I’m still inside the Selenium-IDE for this as it is a convenient environment to constantly tweak in.

Also, note that I’m not checking the actual result or that the number is moving in the correct direction or any number of other possible tests. And yes, this could hang if the calculation the two values is the same, but thats a pretty small risk and would still provide information.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head profile="http://selenium-ide.openqa.org/profiles/test-case">
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <link rel="selenium.base" href="" />
    <title>New Test</title>
  </head>
  <body>
    <table cellpadding="1" cellspacing="1" border="1">
      <thead>
        <tr>
          <td rowspan="1" colspan="3">New Test</td>
        </tr>
      </thead>
      <tbody>
        <tr>
        	<td>open</td>
        	<td>/one_minute/earthhour</td>
        	<td></td>
        </tr>
        <tr>
        	<td>store</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[0]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>link=more</td>
        	<td></td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>//input[contains(@id, &quot;instance_answers_bcartype_&quot;) and contains(@id, &quot;_radio_1&quot;)]</td>
        	<td></td>
        </tr>
        <tr>
        	<td>waitForTextNotPresent</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[0]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>verifyNotText</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[starts-with(@id, &quot;branch_&quot;)]</td>
        	<td>branch_result</td>
 
        </tr>
        <tr>
        	<td>store</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[0]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>//input[contains(@id, &quot;instance_answers_bcartype_&quot;) and contains(@id, &quot;_radio_2&quot;)]</td>
        	<td></td>
        </tr>
        <tr>
        	<td>waitForTextNotPresent</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[0]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>verifyNotText</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[starts-with(@id, &quot;branch_&quot;)]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>store</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[0]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>//input[contains(@id, &quot;instance_answers_bcartype_&quot;) and contains(@id, &quot;_radio_3&quot;)]</td>
        	<td></td>
        </tr>
        <tr>
        	<td>waitForTextNotPresent</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[0]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>verifyNotText</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[starts-with(@id, &quot;branch_&quot;)]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>store</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[0]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>//input[contains(@id, &quot;instance_answers_bcartype_&quot;) and contains(@id, &quot;_radio_4&quot;)]</td>
        	<td></td>
        </tr>
        <tr>
        	<td>waitForTextNotPresent</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[0]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>verifyNotText</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[starts-with(@id, &quot;branch_&quot;)]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>store</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[0]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>//input[contains(@id, &quot;instance_answers_bcartype_&quot;) and contains(@id, &quot;_radio_5&quot;)]</td>
        	<td></td>
        </tr>
        <tr>
        	<td>waitForTextNotPresent</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[0]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>verifyNotText</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[starts-with(@id, &quot;branch_&quot;)]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>store</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[0]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>click</td>
        	<td>//input[contains(@id, &quot;instance_answers_bcartype_&quot;) and contains(@id, &quot;_radio_6&quot;)]</td>
        	<td></td>
        </tr>
        <tr>
        	<td>waitForTextNotPresent</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[0]</td>
        	<td>branch_result</td>
        </tr>
        <tr>
        	<td>verifyNotText</td>
        	<td>//div[@class=&quot;calculation_result&quot;]/span[starts-with(@id, &quot;branch_&quot;)]</td>
        	<td>branch_result</td>
        </tr>
      </tbody>
    </table>
  </body>
</html>

We now have a script that not only interacts with a web application but does some behavior verification as well that was recorded, and modified through an IDE and required very little knowledge of scripting or programming aside from some XPath stuff. And you could stop here saying missing accomplished, but there is so much more you can do to this script to increase it’s value to the organization.

Data Drive
We are now beyond what the Selenium-IDE can handle. So the next thing to do is change the format it outputs into one of the Selenium-RC variants. In this case I chose Ruby as we’re a Ruby shop (primarily) and so I’m trying to use it whenever I can. Here is the Ruby version of the script looks like without any further modifications.

Actually, that is not quite accurate. The Selenium-IDE that you can install from the side and Selenium-RC Ruby implementations from subversion are widely different these days (essentially the Ruby client code is now a gem rather than a module you just ‘require’). I had to hack the script a bit to make it work with trunk, but mainly around session creation rather than the actual guts of the test.

require "test/unit"
require "rubygems"
gem "selenium-client", "=1.2.10"
require "selenium/client"
 
class NewTest < Test::Unit::TestCase
 
  def setup
    @verification_errors = []
 
    @browser = Selenium::Client::Driver.new 'localhost', 4444, '*firefox', 'http://localhost:3000', 10000
    @browser.start_new_browser_session
  end
 
  def teardown
    @browser.close_current_browser_session
  end
 
  def test_new
    @browser.open '/one_minute/earthhour'
    @browser.click 'link=more'
 
    # get the original branch_result
    branch_result = '//div[@class="calculation_result"]/span[0]'
 
    # click each radio button then wait for up to 60s for the branch total to update. Once it has updated then check that the
    # value is different
    @browser.click '//input[contains(@id, "instance_answers_bcartype_") and contains(@id, "_radio_1")]'    
    assert !60.times{ break unless (@browser.is_text_present('//div[@class="calculation_result"]/span[0]') rescue true); sleep 1 }
    begin
        assert_not_equal "branch_result", @browser.get_text('//div[@class="calculation_result"]/span[starts-with(@id, "branch_")]')
    rescue Test::Unit::AssertionFailedError
        @verification_errors << $!
    end
 
    branch_result = '//div[@class="calculation_result"]/span[0]'
    @browser.click '//input[contains(@id, "instance_answers_bcartype_") and contains(@id, "_radio_2")]'
    assert !60.times{ break unless (@browser.is_text_present('//div[@class="calculation_result"]/span[0]') rescue true); sleep 1 }
    begin
        assert_not_equal "branch_result", @browser.get_text('//div[@class="calculation_result"]/span[starts-with(@id, "branch_")]')
    rescue Test::Unit::AssertionFailedError
        @verification_errors << $!
    end
 
    branch_result = '//div[@class="calculation_result"]/span[0]'
    @browser.click '//input[contains(@id, "instance_answers_bcartype_") and contains(@id, "_radio_3")]'
    assert !60.times{ break unless (@browser.is_text_present('//div[@class="calculation_result"]/span[0]') rescue true); sleep 1 }
    begin
        assert_not_equal "branch_result", @browser.get_text('//div[@class="calculation_result"]/span[starts-with(@id, "branch_")]')
    rescue Test::Unit::AssertionFailedError
        @verification_errors << $!
    end
 
    branch_result = '//div[@class="calculation_result"]/span[0]'
    @browser.click '//input[contains(@id, "instance_answers_bcartype_") and contains(@id, "_radio_4")]'
    assert !60.times{ break unless (@browser.is_text_present('//div[@class="calculation_result"]/span[0]') rescue true); sleep 1 }
    begin
        assert_not_equal "branch_result", @browser.get_text('//div[@class="calculation_result"]/span[starts-with(@id, "branch_")]')
    rescue Test::Unit::AssertionFailedError
        @verification_errors << $!
    end
 
    branch_result = '//div[@class="calculation_result"]/span[0]'
    @browser.click '//input[contains(@id, "instance_answers_bcartype_") and contains(@id, "_radio_5")]'
    assert !60.times{ break unless (@browser.is_text_present('//div[@class="calculation_result"]/span[0]') rescue true); sleep 1 }
    begin
        assert_not_equal "branch_result", @browser.get_text('//div[@class="calculation_result"]/span[starts-with(@id, "branch_")]')
    rescue Test::Unit::AssertionFailedError
        @verification_errors << $!
    end
 
    branch_result = '//div[@class="calculation_result"]/span[0]'
    @browser.click '//input[contains(@id, "instance_answers_bcartype_") and contains(@id, "_radio_6")]'
    assert !60.times{ break unless (@browser.is_text_present('//div[@class="calculation_result"]/span[0]') rescue true); sleep 1 }
    begin
        assert_not_equal "branch_result", @browser.get_text('//div[@class="calculation_result"]/span[starts-with(@id, "branch_")]')
    rescue Test::Unit::AssertionFailedError
        @verification_errors << $!
    end
  end
end

We now have our script in Ruby and can start leveraging everything a real scripting language gives you; such as file system access.

The goal of this step is to extract out hard-coded test values and put those in a separate file. This lets use remove duplicate steps from the script and means the script gets modified less as the test file is what gets changed.

In this particular test I am making two things data driven: the calculator and the kilometers travelled. This in effect doubles the testing I can do with this script as it will check 2 different calculators. I also am starting to take defensive action against the Pesticide Paradox by changing the value of the distance travelled.

Since this is Ruby I am using a YAML file for this, but I have in the past successfully used CSV, Excel and XML for this as well.

require "test/unit"
require "rubygems"
gem "selenium-client", "=1.2.10"
require "selenium/client"
require 'yaml'
 
class NewTest < Test::Unit::TestCase
 
  def setup
    @verification_errors = []
 
    @browser = Selenium::Client::Driver.new 'localhost', 4444, '*firefox', 'http://localhost:3000', 10000
    @browser.start_new_browser_session
 
    @dd = YAML.load(File.read(File.join(File.dirname(__FILE__), 'omc_0001.yml')))
  end
 
  def teardown
    @browser.close_current_browser_session
  end
 
  def test_new
    @dd['calculators'].each do |url_part|
      @browser.open "/one_minute/#{url_part}"
 
      (0...6).each do |vt|
        # get the original branch_result
        branch_result = '//div[@class="calculation_result"]/span[0]'
 
        @browser.type '//input[starts-with(@id, "instance_answers[tbasiccarkm][")]', @dd['kilometers'][vt]
 
        # click each radio button then wait for up to 60s for the branch total to update. Once it has updated then check that the
        # value is different"
        @browser.click "//input[contains(@id, \"instance_answers_bcartype_\") and contains(@id, \"_radio_#{vt + 1}\")]"    
        assert !60.times{ break unless (@browser.is_text_present('//div[@class="calculation_result"]/span[0]') rescue true); sleep 1 }
        begin
            assert_not_equal "branch_result", @browser.get_text('//div[@class="calculation_result"]/span[starts-with(@id, "branch_")]')
        rescue Test::Unit::AssertionFailedError
            @verification_errors << $!
        end
      end
    end
  end
end

This is the .yml file that is being loaded.

calculators:
  - earthday
  - earthhour
 
kilometers:
  - 10
  - 4385
  - 48597
  - 43920
  - 387
  - 97874

Make it smart
If you have your scripts all data driven then you are in a pretty good place. If your input files are formatted correctly you might even get close to the holy grail of automation which is having your business analysts or even customers providing you your test data. What I find more powerful though is to go one (or many) step further and teach the script how to get it’s own test data. You just let it run forever then. This means that your actual test execution harness has to have some extra smarts to handle stopping tests gracefully and dynamic discovery of new tests, etc. but if you are scripting up things at this phase you are likely starting to deal with those problems anyways.

This first bit of smarts that I added to the script was to remove the data driving file and replaced it with Ruby module. The OmcHelper.omc_finder method will return a random url that uses the set of questions identified by 93. I also use a random value between 0 and 150000 as the amount driven annually. The selection of 150000 was pretty arbitrary but will likely be representative of most of the people using the calculator. I could likely figure out what the actual average is based upon historical data and do some clever math to come up with a better number, but in this case good enough really is good enough.

require "test/unit"
require "rubygems"
gem "selenium-client", "=1.2.10"
require "selenium/client"
require 'omc_helper'
 
class NewTest < Test::Unit::TestCase
 
  def setup
    @verification_errors = []
 
    @browser = Selenium::Client::Driver.new 'localhost', 4444, '*firefox', 'http://localhost:3000', 10000
    @browser.start_new_browser_session
  end
 
  def teardown
    @browser.close_current_browser_session
  end
 
  def test_new
    @browser.open "/one_minute/#{ OmcHelper.omc_finder('93') }"
 
    (0...6).each do |vt|
      # get the original branch_result
      branch_result = '//div[@class="calculation_result"]/span[0]'
 
      @browser.type '//input[starts-with(@id, "instance_answers[tbasiccarkm][")]', rand(150000)
 
      # click each radio button then wait for up to 60s for the branch total to update. Once it has updated then check that the
      # value is different"
      @browser.click "//input[contains(@id, \"instance_answers_bcartype_\") and contains(@id, \"_radio_#{vt + 1}\")]"    
      assert !60.times{ break unless (@browser.is_text_present('//div[@class="calculation_result"]/span[0]') rescue true); sleep 1 }
      begin
          assert_not_equal "branch_result", @browser.get_text('//div[@class="calculation_result"]/span[starts-with(@id, "branch_")]')
      rescue Test::Unit::AssertionFailedError
          @verification_errors << $!
      end
    end
  end
end

require 'mysql'
 
module OmcHelper
  class << self
    def omc_finder(qt_id)
      begin
        @dbh = Mysql.real_connect("localhost", "root", "", "development")
        res = @dbh.query("select something from some_table where magic_id = #{qt_id} order by rand() limit 1")
        res.fetch_row[0]
      rescue Mysql::Error => e
        puts "Error message: #{e.error}"
      ensure
        @dbh.close if @dbh
      end  
    end
  end
end

Now our script will automatically test any other calculators that have question set 93, though it might take a few runs since the calculator is randomly selected.

But like tattoos or plastic surgery, why stop there?

93 is a magic value which is not really scalable. So why not remove it and have the it select any calculator to test? Sure, that’s easy

require 'mysql'
 
module OmcHelper
  class << self
    def omc_finder(qt_id = nil)
      begin
        @dbh = Mysql.real_connect("localhost", "root", "", "development")
        if qt_id
          res = @dbh.query("select something from some_table where magic_id = #{qt_id} order by rand() limit 1")
        else
          res = @dbh.query("select something from some_table where magic_id is not null order by rand() limit 1")
        end
        res.fetch_row[0]
      rescue Mysql::Error => e
        puts "Error message: #{e.error}"
      ensure
        @dbh.close if @dbh
      end  
    end
  end
end

Now you can get a calculator based on a question set id or a completely random one. I’m not making use of this now though because we are stepping firmly into ‘can of worms’ territory since I would then have to teach the script:

  • How many columns are on the screen?
  • What are questions on the screen?
  • Where are their locations?
  • What type are they? Radio buttons? Text boxes? Sliders?
  • And how to interact with all of the above
  • Valid data in their contexts as some use miles and some use kilometers

And that is just off the top of my head. Which isn’t to say that it won’t happen, but not yet. One important lesson about automation to know is when things become fun rabbit holes instead of value producing. I know that some refactoring is due in the new year around these sorts of things which means my work would largely have to be redone so I should focus my energies elsewhere.

Hopefully people will find this useful. Like it or lump it, automated testing is here to stay and knowing how to create reusable, robust, intelligent scripts is and important tool to have in a tester’s bag of tricks.

Post a Comment

Your email is never published nor shared. Required fields are marked *