Lessons learned from a cloud grid bug

Lessons Learned from a Cloud Grid Bug

Because my Selenium WebDriver with Java course covers as much of the Selenium WebDriver API as I can, I often have code usage in there that doesn’t see the light of day on many live projects. Therefore it can act as ‘edge cases’ for drivers and grid installations.

A few days ago the course code identified a problem on TestingBot, which they have now fixed.

All my @Test code now runs clean on the TestingBot environment.

This post describes the issue that TestingBot fixed, and some generic lessons learned that I draw from this situation.

Finding The Bug

I ran my @Test code from a Jenkins CI, configuring the grid setup using properties passed in as -D values.

  • firefox
  • version: 37
  • platform: WIN7

I isolated the problem to a specific action in my code by running the first listed failing JUnit tests in the Jenkins build.

I did this, but running the specific methods from IntelliJ and configuring my execution through Environment Variables via the “run \ edit configurations” functionality.

I found the test that triggered the hang:

@Test
public void submitFormWithDropDownFiveUsingKeyboardSpecialKeys(){

     driver = Driver.get("http://compendiumdev.co.uk/selenium/" +
            "basic_html_form.html");

    WebElement dropDownSelect;
    dropDownSelect = driver.findElement(By.cssSelector("select[name='dropdown']"));
    dropDownSelect.sendKeys(
            Keys.chord(
                Keys.HOME,
                Keys.ARROW_DOWN,
                Keys.ARROW_DOWN,
                Keys.ARROW_DOWN,
                Keys.ARROW_DOWN));

    waitForOption5DropDownSelected();

    clickSubmitButton();

    new WebDriverWait(driver,10).until(ExpectedConditions.titleIs("Processed Form Details"));

    assertDropdownValueIsCorrect();
}

I could see on the TestingBot “Test Steps” output that it had executed the findElement statement, but wasn’t executing the sendKeys, and it seemed to be freezing when executing the sendKeys.

findElement

    dropDownSelect.sendKeys(
            Keys.chord(
                Keys.HOME,
                Keys.ARROW_DOWN,
                Keys.ARROW_DOWN,
                Keys.ARROW_DOWN,
                Keys.ARROW_DOWN));

I emailed this issue off to TestingBot via the support contact form and received an email two days later letting me know the issue had been fixed.

… turns out it was a character encoding issue we had in our grid. The Keys.chord sends out Unicode characters, which somewhere in our grid was wrongly converted, causing the selenium node to hang. We use nginx in front of our grid, which in turn was stalling on the request, only timing out after 900 seconds.

So, all good.

Lessons Learned

And some generic lessons learned:

  • Our automated code often has workarounds in it due to environment
  • Maintaining a grid can be hard
  • Cloud Grid systems, by default, have lots of logging support
  • Your code config should support CI and local debugging
  • Investigate and raise bugs

Our automated code often has workarounds in it due to environment

If I was working on site where this code failed on my grid, I’d probably just change the code and find another way of selecting the items in the drop down. And to be fair to TestingBot, this isn’t the way that most people could select an item in a list, but my Selenium WebDriver with Java course is designed to teach the normal ways, and demonstrate alternative approaches because, when I automate, I often have to find workarounds to:

  • browser issues
  • Webdriver versioning issues
  • environment issues
  • system bugs
  • etc.

Maintaining a grid can be hard

To be honest. If this had happened on a local grid because of network configuration issues in our environment. I don’t think I’d have been able to track it down and fix it.

I’d have been able to create a workaround in my code, but what would happen next time there is an issue?

Maintaining a local grid can be hard, and takes time to keep it up to date, and up and running. That’s why I try to keep my grid configurations pretty simple, and use cloud based grid solutions when I need a grid with lots of versions and combinations.

Cloud Grid systems have lots of logging support

One thing I encounter on live sites is that people often write logging code very early into their testing ‘frameworks’ because they don’t have the kind of default logging support that you saw above on the output from TestingBot GUI.

The cloud grid systems (Saucelabs, BrowserStack, TestingBot) all show the WebDriver protocol messages received, and capture movies or screenshots of the code executing against the browser.

Consider in your own work environment whether you should write the logging code yourself, or take advantage of the logging already built into the cloud grid systems.

Your code config should support CI and local debugging

I described in a blog post some ways of configuring your driver abstractions.

This situation exemplifies why it is needed. I was able to run the @Test code via CI, and very quickly, run the individual @Test method in the IDE in debug mode. If you have to fiddle with config files or find this hard to do, then I recommend you revisit your code so you have the flexibility you need to run your test code as and when you need to.

Investigate and Raise Bugs

Do raise bugs when you find them. You don’t always get a good response like I did from TestingBot. i.e. they thanked me, and actually fixed the problem

But with a single fix, TestingBot became a viable platform for me to consider as competition to SauceLabs and BrowserStack. That’s a good result.

Posted in Grid | 2 Comments

Generic Selenium WebDriver Grid Configuration Handling to use BrowserStack, Saucelabs and TestingBot

Image taken from page 103 of ‘Der Beobachter. Allgemeine Anleitung zu Beobachtungen über Land und Leute.

In my Selenium WebDriver with Java course I have a Driver.java abstraction class.

I use an abstraction class so that in the @Test methods we make a call to Driver.get rather than individual firefox or chrome drivers, or RemoteDrivers

And the configuration of this is adjusted via the actual code, or a mix of environment variables and properties.

I was looking at some Mac Grid issues that I was helping Brian Long investigate. And I realised that my Generic Grid handling wasn’t as Generic as it needed to be to support my debugging.

I amended my code so that I could configure my Driver.java with either environment variables or properties:

  • Environment Variables are handy because they are system wide and you can configure them on the machine, but they are a bit of a pain to work with because you have to restart the console and IDE when you change them.
  • Properties are handy because you can pass them into mvn test jobs with -D parameters e.g. mvn test -Dparam=value and this makes it very easy to use them in CI

I primarily use IntelliJ and in the Run \ Edit Configuration section you can override environment variables, and pass in -D params so it is possible to work with these in the IDE.

I find it easier to override environment variables in the IDE because the GUI makes it easy to amend them in a list.

I find it easier to configure the CI jobs through properties by passing them in as -D to mvn test

Therefore I amended my Driver.java code to not care if its configuration comes from environment variables or properties

e.g.

String gridBrowser = getPropertyOrEnv("WEBDRIVER_GRID_BROWSER", "firefox");
String gridBrowserVersion = getPropertyOrEnv("WEBDRIVER_GRID_BROWSER_VERSION", "");
String gridBrowserPlatform = getPropertyOrEnv("WEBDRIVER_GRID_BROWSER_PLATFORM", "");

DesiredCapabilities gridCapabilities = new DesiredCapabilities();
gridCapabilities.setBrowserName(gridBrowser);
if(gridBrowserVersion.length()>0)
    gridCapabilities.setVersion(gridBrowserVersion);
if(gridBrowserPlatform.length()>0)
    gridCapabilities.setPlatform(Platform.fromString(gridBrowserPlatform));

And getPropertyOrEnv looks as follows


/**
 * Allow setting the controls via property or environment variable
 * property takes precedence, then environment variable, then default
 */
private static String getPropertyOrEnv(String name, String theDefault){

    String theValue = System.getProperty(name);
    if(theValue == null){

        System.out.println("Could not find Property " + name);
        theValue = System.getenv(name);

        if(theValue==null){

            System.out.println("Could not find Environment Variable " + name + " using default value " + theDefault);
            theValue = theDefault;

        }else{
            System.out.println("Using Environment Variable " + name + " with value " + theValue);
        }
    }else{
        System.out.println("Using Property " + name + " with value " + theValue);
    }

    return theValue;
}

This supported my local debugging, CI and console mvn test triggering.

But as with most tasks related to automating systems, I should expect to encounter additional maintenance and workarounds. And such was the case when I tried to execute the tests against BrowserStack in addition to Saucelabs and my local grid.

Basic Saucelabs configuration uses the capabilities:

  • browser
  • platform
  • browser_version

Basic BrowserStack configuration uses the capabilities

  • os
  • os_version
  • browser
  • browser_version

And of course, with BrowserStack we want it to take screenshots to track the @Test method execution so we need to set

  • browserstack.debug=true

Since I have a ‘generic’ grid config in my Driver.java I wanted to support the extra capabilities in a more generic way.

And I still wanted to allow them to be configured via environment variables or properties.

So, I run through all the environment variables and properties looking for any name prefixed with WEBDRIVER_GRID_CAP_X_

// Allow adding any capability defined as an environment variable
// extra environment capabilities start with "WEBDRIVER_GRID_CAP_X_"

// e.g. WEBDRIVER_GRID_CAP_X_os_version XP
// e.g. WEBDRIVER_GRID_CAP_X_browserstack.debug true
Map<String, String> anyExtraCapabilities = System.getenv();
addAnyValidExtraCapabilityTo(gridCapabilities, anyExtraCapabilities.keySet());

// Now check properties for extra capabilities
Properties anyExtraCapabilityProperties = System.getProperties();
addAnyValidExtraCapabilityTo(gridCapabilities, anyExtraCapabilityProperties.stringPropertyNames());

And the magic addAnyValidExtraCapabilityTo method, looks as follows:

private static void addAnyValidExtraCapabilityTo(DesiredCapabilities gridCapabilities, Set<String> possibleCapabilityKeys) {

    String extraCapabilityPrefix = "WEBDRIVER_GRID_CAP_X_";

    for(String capabilityName : possibleCapabilityKeys){

        if(capabilityName.startsWith(extraCapabilityPrefix)){

            String capabilityValue = getPropertyOrEnv(capabilityName, "");

            if(capabilityValue.length()>0){
                String capability = capabilityName.replaceFirst(extraCapabilityPrefix,"");
                System.out.println("To Set Capability " + capability + " with value " + capabilityValue);
                gridCapabilities.setCapability(capability, capabilityValue);
            }
        }
    }
}

Nothing earth shattering there, but it allowed me to add flexibility into the config without adding too much extra code into the Driver.java class.

If I was doing this in the real world. I would have created a new configuration reader type object, rather than adding the getPropertyOrEnv into the Driver.java code, but currently, the only config I use relates to the Driver.java code and I try to avoid too many radical code changes to the course if I can avoid it.

Also, in the real world, you tend not to need to add such a large amount of flexibility. You tend to stick to one grid provider, or refactor your code into more specific abstractions.

So don’t take this code as an exemplar of how to configure your remote driver, instead look upon it as a set of quick, but pragmatic changes to add more flexibility into the code base, and perhaps you’ll see something you can re-use here.

I’ve added additional lectures into my course to explain the config I used for Saucelabs, BrowserStack and TestingBot. And uploaded all the code changes extracted here to the code in the Selenium WebDriver with Java course.

Posted in Grid, Java, WebDriver | Leave a comment

Recent course source code changes for WebDriver 2.46.0

I just finished updating my course source code for WebDriver 2.46.0

I removed some of the code to github:

I had to make a few other minor changes which I summarise below.

Chrome and Firefox

Chrome and ChromeDriver 2.16 now seem to work as well as the Firefox driver. I used to have some issues with Frames and Window management but that has all gone away. Now I have a single set of tests for Chrome and Firefox with no ‘fails on Chrome’ and ‘fails on Firefox’ suites.

I did have to add some additional synchronisation when creating new windows in Firefox. Firefox used to block before returning control to the code when creating new windows, but now it doesn’t so I simple ‘wait’ for the element I need to be available before clicking on it. General synchronisation goodness that we all know and love.

Cookies

I did have to make changes for cookies though. I assume most people don’t mess much with cookies when using WebDriver, but it is handy functionality when you need it.

ChromeDriver still has a bug where it creates 2 cookies. And now, when ChromeDriver creates a cookie it prefixes the ‘domain’ with a ‘.’. This means you have to be careful working with cookies through ChromeDriver as you might have to work around the browser creating a duplicate cookie after you amend it.

Firefox updates have meant that I now have to append a ‘/’ to my cookie paths.

Minor Things

  • The Select support class gave me a string padded text, so I had to ‘trim’ the output. I suspect this was a browser compatibility thing with my app code.
  • On Mac, my window positioning code didn’t work because I set the Y value to 20, but Mac wanted more space for its top menu bar, so I changed the position to 40.

 

Posted in Courses, WebDriver | 1 Comment

OperaDriver is now officially deprecated in my code

Anyone remember Opera?

For a while it had the best developer tools and the best JavaScript debugger. It also had ‘good, not great’ WebDriver support, but an easy to add driver via maven.

That stopped when Opera went from version 12.17 and the Opera Driver didn’t work on most recent versions of Opera when they moved to webkit and then blink. But if you still wanted to, you could still use the driver and run tests with the older version of Opera.

But no longer.

The OperaDriver hasn’t been updated so it now compile clashes with Selenium WebDriver 2.46.0

You can exclude the conflicting apache.commons code if you want:

<dependency>
    <groupId>com.opera</groupId>
    <artifactId>operadriver</artifactId>
    <version>1.3</version>
    <exclusions>
        <!-- changes to Selenium 2.46.0 mean that the operadriver
            now conflicts with Selenium WebDriver. I excluded the
            apache.commons commons-exec from operadriver to allow
            the driver to retain compile time compatibility 
            with webdriver
        -->
        <exclusion>
            <groupId> org.apache.commons</groupId>
            <artifactId>commons-exec</artifactId>
        </exclusion>
    </exclusions>
</dependency>

That will get you compiled, but won’t help you execute.

In fact OperaDriver hasn’t executed since WebDriver version 2.34.0

<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-server</artifactId>
    <version>2.34.0</version>
</dependency>

The OperaDriver depends on code that moved in WebDriver 2.35.0.

Possibly just as well since Opera always had some issues:

  • Alert Handling
  • Did not refresh reliably
  • Window Management – did not maximise
  • JavaScript Executor and Asyn execution
  • File handling
  • Cookies – did not return the correct expiry date

I have now ‘officially’ removed OperaDriver from my course code base, and moved the basic Opera stuff, just as a legacy example, off to github.

https://github.com/eviltester/operaWebDriverExample

Posted in WebDriver | Leave a comment

WebDriverBackedSelenium example on github now

I had a small WebDriverBackedSelenium example in my Selenium 2 WebDriver API Course codebase.

Since the Selenium API has been deprecated by the main WebDriver project, although it can still be found in the current versions of Selenium WebDriver, I have moved the example to a separate github project.

https://github.com/eviltester/webDriverBackedExample

It isn’t much. It just shows how to get started with the API.

But I’m making the split now, so that:

  • students no longer see the deprecation messages when the compile the code source and think they have a problem
  • my main code won’t break badly on the future version of WebDriver which removes the Selenium API

 

Posted in Migrating to WebDriver, Selenium-RC, WebDriver | Leave a comment

Successful Test Automation for Managers

Illustration by Herb Lebowitz from the cover of “Automation” by Carl Dreher

What questions do Test Managers have regarding automation? What concerns and issues do non-technical managers face with test automation?

I presented at the Test Management Summit on the 29th April 2015 to try and find out.

I’ve been an automator and a test manager so I’ve experienced both sides of the coin, but I’ve always had the benefit of technical and programming knowledge and experience. So while I had an initial list of areas that I thought current managers working, or about to work, with automation would want to discuss, I wanted to see what would happen on the day.

At this point I’m working from memory without notes, because I was facilitating a discussion, so let’s see what I remember…

The Test Management Summit is a discussion based event, with the emphasis on the presenter moving to triggering and facilitating discussion rather than doing all the talking. As a result, when you look at the slides, the first half of the slides were designed for me to talk over, and the remainder held in reserve in case the discussion faltered.

In the event, I talked for 15 mins, and the discussion was lively and proceeded without requiring my reserve slides.

I made a distinction between use of tools, and automation. With automation having the characteristic of no manual intervention during its execution.

I emphasized the need to have an understanding of how the automation works. If you don’t understand how it works then you have added a dependency on your success on someone who does. This applies across the board:

  • If you use a commercial tool then you rely on the support team from the supplier
  • If you use open-source then you rely on the internet and the informal development team, but have the opportunity to involve yourself, if you have the skills
  • Similarly if you bring in frameworks or libraries into your automation, if you don’t have the skill sets to read, and possibly amend the code, then you limit the level that you can understand or fix them.

Regardless, we have to have some sort of way of deciding if the automation process supports your test approach, or not.

As to “Who can use automation?”. If you don’t have the skills to move beyond automation as ‘magic’ then you can only use it if an interface has been added to the automation which allows your skill level to engage with it. i.e. automation with one button to start it can be triggered by anyone, but if that is the level of your understanding then you won’t have the ability to intervene when something goes wrong.

This may, or may not, be managed as a risk in your organization.

Many people do not think through what they want from automation to identify if their approach to automation can meet their expectations, or to think through multiple approaches to achieving their aims. Too often we default to: “automate all the scripts”. I recommend thinking through your aims, and identifying options. This will also help you work out if your automation ‘works’ i.e. helps you meet your aims.

If you read the slides then you’ll see that I used the Carl Dreher book “Automation”  as one of the references for the talk.

Carl Dreher’s book is out of print, but I like it, as an historical overview of various forms of automation from mechanical devices through to cybernetics and computing. There are various quirks in language that I enjoyed, particularly the use of “Automationist” rather than “Automator” to describe someone who works on automation.

I also referenced “The Art of Leadership” by Captain S.W. Roskill. Another out of print book, but I can relate to the early sections in Roskill’s book where he describes two traits of military leaders that we don’t always expect from leaders in business:

  • An expectation that military leaders will train and teach as they gain experience
  • An expectation that military leaders need technical skills to ‘lead’ from the front and gain the respect of their troops

In order to balance this out for the business world, I think we make a distinction between ‘managers’ and ‘leaders’. Therefore managers may need to engage with automation in a different way than leaders who exhibit the traits above.

We discussed:

  • Should we convert manual scripts to automation scripts?
  • What would you do in the event of …
    • Single staff member having all the knowledge
    • Immediate “No – can’t be done” to requests for work
    • Staff who are ‘doing their job’ but are finished in half the day
  • How do you know your automation is working
  • How to recruit technical staff
  • What should a test manager do to improve their technical knowledge
  • What should a test programme manager do to improve their technical knowledge

Much of the discussion will remain private because it was provided by the participants themselves.

As add on points – during the discussion, I remember mentioning:

  • Use multiple layers of abstraction to support:
    • different “Who can use?” levels
    • use of automation in different ways, i.e. you can use the automation abstraction as a tool to support your testing, rather than only create non-manual intervention automation
  • Recruit with hands on exercises, i.e. pairing, discussion
  • Automating manual scripts does not lead to effective automation because
    • does not harness the ability for data driven automation (vary non-path invariant data)
    • does not harness the ability to use random data to explore assumptions inherent in equivalence class analysis
    • building on a point made by Adam Knight [http://www.a-sisyphean-task.com/] it does not encourage ‘alternative’ thinking about how automation can help test the system in ways that the script analysis did not identify e.g. API testing, adhoc automation to support defect investigation
  • Coaching without technical knowledge
    • Ask open questions of the team (i.e. non yes/no answers)
    • Ask “How can we…” questions
  • Encourage alternative investigations, but constrain them to time or scope

Some of the points deserve longer blog post on their own. And I will expand on some of these points at the Test Automation Day in June 2015

Hopefully the slide deck will have some points that encourage you to evaluate your own use of automation in your test process, and try something different, or try to investigate in more detail.

Posted in Conferences | Leave a comment

Automation Abstractions: Page Objects and Beyond – Conference Talk

I presented “Automation Abstractions: Page Objects and Beyond” as a conference talk at Star West 2014 and Oredev 2014

Essentially the same talk, but Oredev had a slightly smaller timeslot for the talk, so I summarised the verbage a little.

I have uploaded the slides to slideshare

And the source code is in my svn repo.

The talk was filmed and recorded by Oredev, unfortunately the audio doesn’t appear to have recorded properly and is very quiet in their released vimeo video..

However, I have subsequently amended their video and re-processed the audio so you can hear it, and released it on YouTube and you can watch it below.

Posted in Conferences, Practices | 9 Comments

Speedrun install for WebDriver 2.43.1 on Mac OS X 10 Mavericks

I added the notes and video for my ‘speedrun’ install of WebDriver 2.43.1 with Java, Maven and IntelliJ on the speedrun page.

I used VM Fusion to create a VM on my Mac with a clean install of OS X 10 Mavericks, then went through the process of installing on the Mac.

I updated the speedrun checklist to account for the Mac OS differences, and made notes on the install.

For some reason, when I installed on my ‘proper’ mac, I didn’t have to set the JAVA_HOME variable, but I did on the VM, so the video shows a full environment setup for JAVA_HOME, M2_HOME, M2 and updating PATH.

I found it interesting how Mac prompted me to the correct location a few times – ie. the install of JDK 1.8, so I did this earlier in the process than I did with Windows XP.

Also, since Mac OS X 10 Mavericks is slightly more ‘modern’ than Windows XP, I didn’t have to install an alternate browser to download the files, or an alternate text editor to amend the pom.xml file and update it to version 2.43.1 of WebDriver.

I edited the video down to 13 minutes, although the elapsed time (if I exclude the Wifi failure I had in the middle, was about 45 minutes).

You can find the speedrun video and installation notes on the speedrun page.

Posted in WebDriver | Leave a comment

First WebDriver Speedrun Install added

I have created a new page on the site for ‘speedruns’

seleniumsimplified.com/speedrun-installs

These are start to finish single run install of:

  • Java
  • Firefox
  • WebDriver
  • Maven
  • IDE
  • And running first test from command line and IDE

They have been edited to remove the boring bits, have annotations as simple explanation, but no verbal commentary.

I have an install checklist on the page, but you can also find that in our ‘start’ github project.

I’ll add more speedruns for different operating systems and versions over time.

For now, I have added Windows XP (using the modern.ie).

If you need more help getting started than the speedrun offers then watch our free get-started course or the free preview videos on our full course.

Posted in Training Courses | Leave a comment

Selenium WebDriver Page Object Abstractions and Beyond

I’ve been collating my thoughts on Page Objects and Abstraction Layers, particularly the implementation of them using Java.

I presented my initial thoughts at Let’s Test 2014 in a half day workshop. This could easily have been a full day workshop to allow for more experimentation with the different coding styles.

I released the slides to the tutorial on slideshare.

I confess to some nervousness about releasing the information since I don’t normally release the slides to tutorials and courses in case it cuts down on the value of running the tutorial or course again.

But, in this case, I don’t see enough information on the web about multiple Abstraction Layers and different ways of doing Page Objects, so I thought I’d throw my work out there, along with the supporting source code.

I’m presenting this material as a ‘talk’ rather than a workshop at StarWest 2014, and at Oredev 2014.

On a related note. I fronted a discussion at the Test Management Summit 2014 on Automation Abstractions. The slides for which were taken from the above presentation.

The aim for the slides and presentation is to demonstrate multiple approaches, so that people don’t just pick up the first ‘framework’ they see, or build ‘Page Objects’ without thinking through the modelling approaches open to them.

In the code and presentation I have examples of:

  • Page Objects
  • Page Objects that synchronize as slow loadable components
  • Navigation separate from Page Objects
  • Domain Objects
  • DOM WebElement abstractions
  • and more…

The code and slides demonstrate some of my biases.

The main point is that ‘none of these approaches are best’, and we make decisions when we build our automation. We should take responsibility for those decisions and experiment with what works best in our environment, with our coding skills, with our development standards, for our application.

I’ll expand this material in the future. But I hope it helps somewhat in its current form.

You might also find the slides for my BDD tutorial relevant since I discuss some of my thoughts on BDD and Domain Specific Languages as Abstraction Layers.

Posted in Conferences, Practices | 2 Comments