Element 34

BrowserMob scripts can be written to use VUs (Virtual Users) or RBUs (Real Browser Users); the difference being raw HTML over the wire vs. an Se driven browser. But there are situations where you want to mix the two and create a Hybrid script.

Hybrid scripts have value in…

• Accessing oracles
• Environment setup
• Step shortcutting
• Advanced synchronization

Hybrid scripts make use of the HttpClient API to sent GET and POST requests along the wire to external resources outside of the Se browser. This script shows just the HttpClient stuff for sending a POST and a GET inside a BrowserMob script. Were it flushed out there would be interaction with the Se browser as well.

var selenium = browserMob.openBrowser();
var httpclient = browserMob.getActiveHttpClient();
 
var tx = browserMob.beginTransaction();
 
browserMob.beginStep("get");
var getResponse = httpclient.get("http://search.twitter.com/search.json?q=monkey");
var info = getResponse.getInfo();
if (info.getStatusCode() == 200) {
  var responseAsJSON = getResponse.getBody();
  var responseAsObject = eval('(' + responseAsJSON + ')');
  browserMob.log(responseAsObject.max_id);
}
browserMob.endStep();
 
browserMob.beginStep("put");
var postResponse = httpclient.post("http://adam.goucher.ca/cgi-bin/simple-echo.pl?firstname=adam");
info = getResponse.getInfo();
if (info.getStatusCode() != 200) {
  browserMob.log("POST failed");
}
browserMob.endStep();
 
browserMob.endTransaction();

When doing load testing through something like BrowserMob, it is generally a better idea to construct your runs in such a manner to not need excessive use of this technique. But should you need it, it is available.

Another day, another language; such is the life of a consultant. This time it was Python and I was parameterizing tests for a client so I could log into their system without having to change anything other an an external file. Their CI server will have its own file, as will each of the team members writing / maintaining the Selenium scripts.

So building on what was covered in One, and only one config file and If you really must write your Selenium framework in Java, here is how I am currently suggesting how people incorporate configuration files into their scripts and frameworks.

First, you need to tell your framework (which in this case I mean ‘your own custom runner’) where exactly to find your config file.

import getopt
 
# this is a custom module / class you can see below
import config
 
try:
    opts, args = getopt.getopt(sys.argv[1:], "hc:", ["config="])
except:
    print("crap")
    sys.exit(2)
 
config_file = ""
for o, a in opts:
    if o == "-h":
        print("Usage:")
        print("sh.[bat|sh] script_pattern")
        print("    script_pattern can be either a script name (mco_rpt_wo_0001.py) or path part (mco_rpt)")
    if o in ("-c", "--config"):
        config_file = a
 
if config_file == "":
    logger.info("using config/flyingmonkey.cfg as config")
    config_file = "flyingmonkey.cfg"
 
# two ways to specify the config; either as a filename in the config dir or by path
cf = config.config()
if os.path.exists(os.path.abspath(config_file)):
    cf.configure(config_file)    
elif os.path.exists(os.path.join(os.path.dirname(__file__), "..",  "config", config_file)):
    cf.configure(os.path.join(os.path.dirname(__file__), "..",  "config", config_file))
else:
    print "config file (%s) does not exist" % config_file
    sys.exit(1)

Basically, what is happening here is that the runner can take as an optional parameter -c or –config which both take the name of a config file. And if the parameter isn’t provided then a default of flyingmonkey.cfg is used.

In my original way of doing this almost three years ago, I had it read the config from an XML file, but that doesn’t appear to be the Pythonic way of doing things. In the standard modules that Python comes with is the ConfigParser module which is a lot nicer — and since it is there by default is implicitly blessed as the way to do this sort of thing. Here is my new class for parsing this file.

import ConfigParser
 
class config(object):
    # singleton
    _instance = None
    def __new__(cls, *args, **kwargs):
        if not cls._instance:
            cls._instance = super(config, cls).__new__(cls, *args, **kwargs)
        return cls._instance
 
    def configure(self, config_file):
        # parse config
        self.config = ConfigParser.RawConfigParser()
        self.config.readfp(open(config_file))

A lot nicer.

And here is how it is used in a script.

import unittest,  logging
from selenium import selenium
import config, rabbit
 
class createAlert(unittest.TestCase):
    def setUp(self):
        self.log = logging.getLogger("monkey.createAlert")
        self.cf = config.config().config
        self.verificationErrors = []
        self.selenium = selenium("localhost", 4444, self.cf.get("General", "browser"), self.cf.get("General", "server"))
        self.selenium.start()
        rabbit.login(self.selenium, self.cf.get("Login", "username"), self.cf.get("Login", "password"))

As for the actual file that it reads from, for those who used to have to hack around in DOS and Windows 3.x, it is quite similar to the .ini format that everything used to be.

[General]
server: http://flying.monkey.com
environment: test
browser: *chrome
 
[Login]
username: adam@element34.ca
password: monkey

The observant reader will see that the way to reference a particular parameter is by “Section” and then “Key”.

I’m pretty happy with how easy, and nice, it is to do this now.

A lot of people just run headlong into an automation project and immediately start either recording or writing scripts. In my experience, that is a bit rash as there are some infrastructure bits that need taking care of first.

The very first thing is to figure out where the scripts will be stored in version control. Everything should be in version control for automation. (If anyone has a recommendation around solutions for automation consultants who don’t always have access to client repos but write code, ping me.) This should be an obvious first step, but it often isn’t. Sadly.

The next step is to figure out your logging strategy. The reason for this is two-fold. First, it gives you a means of debugging your script in a clean manner and second, it allows you to create a record of test execution details for latter reference.

I’m currently helping a client do a proof-of-concept script for their custom widget toolkit with Selenium 2 and C# and went down the rabbit hole of getting the logging (log4net) figured out yesterday with NUnit. Here is what was necessary from the myriad of posts that all contributed only part of the answer. Hopefully it will take you a lot less time to get wired up than it did me.

Assumption Alert – You have already installed NUnit and got a bare-bones ‘hello world’ test working

1. Download the latest log4net release
2. Add a log4net reference to the VS project
3. Add the log4net bits to the script

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

   using System; using System.Text; using System.Text.RegularExpressions; using System.Threading; using NUnit.Framework;   using OpenQA.Selenium; using OpenQA.Selenium.IE; using OpenQA.Selenium.Internal;   using log4net; using log4net.Config;   namespace Selenium2Tests { [TestFixture] public class NowWithLoggingTests { IWebDriver driver;   private static readonly ILog log = LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);   [SetUp] public void SetupTest() { verificationErrors = new StringBuilder();   driver = new InternetExplorerDriver(); driver.Navigate().GoToUrl("http://localhost:8080"); }   [TearDown] public void TeardownTest() { driver.Quit();   Assert.AreEqual("", verificationErrors.ToString()); }   [Test] public void TheNewTest() { Assert.AreEqual(1, 1); } } }

   Lines 11-12 and 21 are the important ones. The first two add the log4net functionality to the class and the third does some reflection trickiness to determine the class name in a safe-to-copy-and-paste-this-line-without-change manner. Also notice that there is not a BasicConfigurator.Configure() or similar call. We’re doing a bit of configuration magic to omit that.

4. Next we need to configure log4net. And of course there a tonne of different ways to do this, but this way I feel is the cleanest. In your project, add a new Application Configuration File and rename log4net.config. You can put whatever config you like in there; here is the one I am currently using.
   

   <?xml version="1.0" encoding="utf-8" ?>
   <log4net>
     <appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender" >
       <layout type="log4net.Layout.PatternLayout">
         <conversionPattern value="%date [%thread] %-5level %logger [%ndc] - %message%newline" />
       </layout>
     </appender>
     <appender name="RollingFile" type="log4net.Appender.RollingFileAppender">
       <file value="example.log" />
       <appendToFile value="true" />
       <maximumFileSize value="100KB" />
       <maxSizeRollBackups value="2" />
    
       <layout type="log4net.Layout.PatternLayout">
         <conversionPattern value="%level %thread %logger - %message%newline" />
       </layout>
     </appender>
     <root>
       <level value="DEBUG" />
       <appender-ref ref="ConsoleAppender" />
     </root>
   </log4net>

   As things stand right now though, log4net will never find this file as it expects it by default in an app.config or web.config. To tell log4net where it’s config can be found we add
   

   [assembly: log4net.Config.XmlConfigurator(ConfigFile = "log4net.config", Watch = true)]

   to our AssemblyInfo.cs file. You also need to remember to change the properties of this file to have:

   • Build Action: Content
   • Copy to Output Directory: Copy if newer

You have now integrated log4net into your project and you can send log messages to it using log.Debug, log.Info, log.Error, etc. and everything is rainbows and lollipops until you run the script with NUnit and don’t see anything being logged below the Error level. WTF?

5. NUnit will be default capture log4net messages as pass they along to the user if they are Error or higher — regardless of what log4net says. So we need another Application Configuration File, this time called, well, actually it is complicated. Depending on how the script is run, the file needs to be named one of five different ways. Here is a trick though, if you run your NUnit scripts from the NUnit GUI you can specify the config file you want — so go ahead and called it nunit.config. Just remember this bit of naming silliness when your logging “doesn’t work” sometime later.
   
   Here is my nunit.config — note the DefaultLogThreshold is now DEUBG.
   

   <?xml version="1.0" encoding="utf-8" ?>
   <configuration>
     <configSections>
       <sectionGroup name="NUnit">
         <section name="TestCaseBuilder"
         type="System.Configuration.NameValueSectionHandler"/>
         <section name="TestRunner"
         type="System.Configuration.NameValueSectionHandler"/>
       </sectionGroup>
     </configSections>
    
     <NUnit>
       <TestCaseBuilder>
         <add key="OldStyleTestCases" value="false" />
       </TestCaseBuilder>
       <TestRunner>
         <add key="ApartmentState" value="MTA" />
         <add key="ThreadPriority" value="Normal" />
       <add key="DefaultLogThreshold" value="DEBUG" />
     </TestRunner>
     </NUnit>
   </configuration>

   Don’t forget too that the properties of this file need to be set in the same manner as the log4net.config file above.

Now you are really integrated with log4net and NUnit with your messages actually showing up. If only it didn’t take me all day to get it working properly…

Part of my consulting practice is to act as ‘Auditor’ and provide a third-party review of company’s automation. Before even trying to run them, I first just browse through the code looking for Smells. It is hard to itemize things that come from experience and ‘gut’ feel, but this is a partial list of what I realized I am looking for.

• Excel – If automation code is reading test information or recording results in Microsoft Excel this is a smell that your automation is Windows specific. That is not always the case, but often is. It also is a smell that the people doing the automation code are not ‘trained’ programmers (which is OK, I’m not ‘trained’ either) as Excel is often the tool of Analysts and Classical Testers. There are three main problems with Excel. The first, and it is the killer, is that it is a binary format so any analysis between versions is impossible. If you need it to load in Excel then use a CSV as Excel will associate with it by default. No, you don’t need formatting in Test Data. This is the big reason I believe things like Robot Framework and Cucumber will succeed. The second problem is that it costs money. Yes, almost every desktop in your office has Office on it, but does your test farm as well? The last problem with Excel is that it is often used in place of something better. Why are you getting test data from Excel when it could come from the database directly? Why are you creating an extra layer of logging when the language of your automation has it already or the CI server has it?
• Exception Handling – The goal of automation is to find problems and problems manifest themselves in xUnit style automation as assertion failures. Let the code fail. Do not catch the exceptions. The frameworks know how to behave on those types of failures. Often people catch the exceptions so they can record the result in their custom logging, but that begs the question of why you have written another layer of logging — often using Excel. Yes, there are times when you will have exception handling in your script, but that should be for things like “I tried to read a record from the database and it wasn’t there implying this environment is not configured correctly” not “catch AssertionException”.
• Paths – The presences of a non-relative path on disk (windows or unix) is a smell that the automation is tied to a single platform at best, or a single specifically configured machine at worst. Automation should be self-contained with all necessary bits checked into version control. If I need to do more than ‘svn co’ and some minor editing of config files (my username to the database for instance) then we have a problem.
• DRY – Don’t Repeat Yourself is becoming my favourite smell. Copy and pasting is not a technique for increasing LoC amounts. Look really hard at why you are pasting that method into that class and not abstracting it to a separate class. This, like most things is heuristic though. You don’t want to introduce extra coupling of code by accident which can lead to more brittle code not more robust code.
• Configurability – How to I quickly and easily switch my automation from my local machine to something like Sauce Labs OnDemand? If the answer is anything other than ‘just change which config file is loaded’, then the automation framework needs some work. Ruby on Rails completely I think completely nailed this problem with their ‘environments’ and I have often implemented a ‘Selenium’ or ‘CI’ environment.
• Versioning – This smell is one that is most seen in traditional testing teams that have somehow assumed the mantle of automation creators and maintainers. Many universities still don’t teach their undergrads version control, but developers soon learn that in The Real World. Not so much in Test which often is the realm of Shared Drives (Q for QA commonly) and document management through email. If it has value it is checked in, if it has no value is it not created should be the mantra of anyone doing anything with automation. That is the first half of the smell. The other half is how the version control system is used. Are the commits atomic? Are the comments associated with them relevant and provide the necessary information? Are updates updates and not delete plus add?
• One Product – A final smell today is noticed through automation but is actually an organizational one and that is where the automation code is in relation to the production product code. If they are separate then the smell exists. I would guess that most organizations are not in the business of making a product and making automation. They are in the product business. The automation code needs to live with the product. This smell harkens back to the days when the Us vs. Them mentality of Dev vs. Test was seen as productive. There is one team, working towards on goal. Treat their artifacts the same.

That’s it for now. The next time I’m auditing I’ll try and keep track of my thoughts to see what other smells I can find.

Q: When is a button not a button?

A: When it is a YUI button

Selenium, at its core, deals with HTML things stored in the DOM. And does a pretty good job at it. But things start to get confusing when the people creating the page start to get fancy and use third-party widgets like those provided by YUI. This post is about how to fake your way through Se automation when you encounter YUI controls, specifically a YUI Button.

The big thing to understand is that YUI buttons are not buttons (to the browser), they are in fact just blobs of JS that happen to look like a button. When you ‘click’ on one, a callback is fired to execute another bit of JS to make it look like you click did something. Ahhh, AJAXy goodness.

But because there is nothing happening at the DOM level, we need to simulate both the ‘click’ and the subsequent action. For example, here is a helper function I wrote to deal with a ‘checkbox’ type YUI button for a client.

def enable_bubbles(se):
    if se.is_element_present("//span[@id='PinBalloonSelector']"):
        se.get_eval("window.ShowBalloonButton.set('checked', true)")
        se.get_eval("window.PersistBalloonPinMode()")
 
    se.wait_for_condition("selenium.isElementPresent(\"//div[@id='balloonContentDiv']/div\")", "60000")

Important things to note:

• YUI buttons are children of the window object so start your hunt there
• To ‘click’ the box on you set the attribute ‘checked’ to true. Setting to false ‘unclicks’ it
• After the click you need to call the JS function that would normally be triggered as we’re sneaking operations in under-the-hood
• Because all of this is happening in JS, you need to use the get_eval variant that your binding provides

That took me well over a day to figure out, so I figure someone else has likely run into issues where Se-IDE has recorded a button interaction one way but on playback or export it didn’t work because the button isn’t an HTML button, it is a YUI button.

So who wants to write a plugin for Se-IDE that deals with YUI controls properly? :)