How to write a keyword driven test framework – an historical perspective

I don’t really like keyword driven frameworks. For a whole bunch of reasons that I won’t go into now, since you came here to learn how to write them.

Well, here’s the secret…

This is old tech. Older than old.

In fact, when I see it being promoted as ‘state of the art’ I’m amazed, and a little disappointed.

The first book I read that relates to this was “Write your own adventure programs for your Microcomputer”, back in 1983, and it teaches the basics of how a keyword driven framework works. I was using a ZX Spectrum 48K at the time. I loved these little Usborne books, they probably set me on the road to programming better than any other resource.

I don’t think I managed to get the listing in the book working, but I did manage to write my own crude adventure game parsers.

In fact, pretty much any tutorial on coding text adventure games will cover the basics.

And this is partly why Keyword driven frameworks exist.

Keyword Driven Frameworks exist because they are easy to write.

Not because they boost productivity, or are easy to maintain or anything like that. Simply because they are easy to write.

And as much as I denigrate them, if you write your own, then you may well learn lessons about parsing input that will help you improve your test automation abstraction layers in the future, so I shouldn’t be too hard on them.

And yet, I caution you to be cautious.

The basic principles you have to address for Keyword Driven Testing are:

  1. Decide on a vocabulary and phrases
  2. Decide on a parsing mechanism
  3. Decide how to map your custom code to the verbs you parsed

Decide on a vocabulary and phrases

 With an old text adventure you used a basic Verb/Noun parser. e.g.

  • Take Book,
  • Open Door

We might relate that to:

  • Open_Browser Firefox
  • Goto_URL http://www.eviltester.com
  • Assert_Title “Home | Evil Tester.com”

One of your first tasks is to decide on the Verbs you want to handle in your application and your domain.

 Decide on a parsing mechanism

Then you have to decide how you are going to parse them. csv? tab delimited? plain text?

When writing text adventures you have the disadvantage that you have to handle user input, dynamically.

What we did in the old days, with a verb noun parser was split the string looking for the first space. Everything before the space was the verb, everything after the space was the noun.

And that’s pretty much all you do with keyword driven testing. You take the line of input and split it based on whatever delimiter you choose.

This is why a lot of people use Excel format as input, you use a library that reads Excel, and treat each cell as an input. The first cell on the line is the verb, the rest are the nouns or parameters.

You can get more complicated, and indeed, most home grown keyword driven test frameworks seem very crude given the type of parsing we eventually learned how to do when coding adventure games.

But if you want to do this, and you want to get started now, then keep it simple.

You might want to split your input verbs and nouns with tabs, then you can accommodate spaces in words and parameters. Up to you.

Decide how to map your custom code to the verbs you parsed

Assuming you have parsed the input and stored the verbs and nouns in some variables and some sort of collection, then you have to decide what to do with that data and map it on to custom methods.

There are lots of ways of doing this, and your level of programming experience will pretty much dictate the one you choose.

In the game above, they tokenised the input and associated each word with a number, to make it easy to decide what to do. This was also done because of memory constraints. Its actually a pretty good way of working, but most buy inderal 10mg people writing keyword driven frameworks don’t do this, they tend to do something like the following pseudo code:

If uppercase(keyword) = "OPEN_BROWSER" 
     then open_browser(params_array);
If uppercase(keyword) = "ASSERT_TITLE" 
     then assert_title(params_array);

And that’s fine, create a function for each keyword, then call it when you find it.

Job Done

And what do you know. You’ve done it. That’s it.

When writing an adventure game you have to maintain state, create maps, handle lots of objects, track location, handle random events, etc. etc.

But with a keyword driven test framework you just map a keyword to some code you’ve written and voila.

It’s ugly. It’s crude. It’s basic. And it’s still what many people do.

If you want to get ‘fancy’ then you could read a ‘how to write your own interpreter’ or compiler book. And there are loads of those. I’m not up to date on them because I studied this stuff years ago, although one of my favourites [Aho, Sethi and Ullman]’s “Compilers: Principles, Techniques and Tools” is still well regarded

Although I prefer the Dragons cover, which is also much more fitting with the text adventure theme.

I still have my copy of “The Dragon” book, it seems to be the only compiler/interpreter book I kept. (In addition I also have “Introduction to Automata Theory Languages and Computation” by Hopcroft and Ullman, which isn’t quite as simple to pigeon hole )

Why not use a simple and more CV friendly approach?

However, we have learned a lot now about Domain Specific Languages, and writing maintainable abstraction layers. So instead of writing a simple text adventure parser, why not use something like Cucumber.

Cucumber essentially implements the parsing for you, and the execution, and the reporting, and the running, and all you do are implement the methods which map on to the commands, e.g. the ‘open_browser’ method above.

Yes your code and scripts will remain pretty horrible because you’ll have abstracted at a verb keyword level rather than a logical action level, but that’s exactly what you were going to do with a keyword driven approach anyway.

At least this way, you get a head start with a ‘modern’ tool so you don’t have to re-invent the wheel.

And you can gradually refactor your test abstractions to something readable and maintainable if you start to see issues with a keyword driven approach.

This also has the benefit that it is in vogue, and therefore easy for other people to understand as an addition to your CV. No-one really cares if you can do verb/noun parsing anymore, but if you can use Cucumber, they will look more closely at your CV.

Warning: Dragons within

But… here’s The Thing.

I loved writing parsers. I did. I wrote text adventure game parsers, AI parsers, Natural Language parsers, Interpreters and Compilers.

I loved it, they were great fun.

I learned a lot.

And that is part of the danger.

The people writing the framework are going to have a blast.

The people using it, are going to create long unmaintainable scripts that are ugly, confusing and brittle, and all in the name of Keyword driven testing.

So I warn you.

Your techy testers, when just starting out, are going to recommend Keyword Driven approaches because:

  1. The ‘industry’ still loves to promote them
  2. The frameworks are fun and challenging to build from a technical perspective
  3. Your testers don’t need to learn anything technical, just a ‘small’ vocabulary of verbs (or so the promotional material goes)

I caution you. Here be Dragons.

PS: If enough people contact me then I’ll write a short keyword driven framework example in Java, just to show how easy it can be, and how crude and ugly I can make my code.

PPS: I’ve long since given away my copy of “Write your own adventure programs for your Microcomputer” but I found it again on the wonderful World Of Spectrum  as a pdf

This entry was posted in Practices, Selenium Simplified Blog and tagged . Bookmark the permalink.

7 Responses to How to write a keyword driven test framework – an historical perspective

  1. Lavanya says:

    Hi,

    Iam facing a problem,While running my application in firefox its working fine,In IE iam getting flickering issue.Can anybody please help me out of this.

    Regards,
    Lavanya

    • Alan says:

      Lavanya, you would need a more specific description to receive any help. Can you describe “flickering issue” in more detail?

  2. Savita says:

    Hi Alan,

    I enjoyed reading keyword driven framework post. I agree with you that keyword driven framework is easy to implement.
    I do not agree with people who says that Keyword driven framework should be implemented because its easy to maintain 🙂

    Savita

  3. Kent Wood says:

    Great article Alan, as always. I’ve enjoyed working with the Robot Framework and it’s Selenium2Library, but quickly found that I was extending the existing keywords with Python/Selenium libraries of my own in order to really make a go of things. I’m working in Java/Junit again and am finding that I prefer working directly with the code. On the other hand again, the reporting for RF was phenomenal as was the ability to give each test one or more tags and then specify which tag sets to run very easily at the command line.

    I realize now also that in a recent project I did for a small company I actually wrote a keyword parser but didn’t realize it as such! Rather than read the commands from a file, I simply made them available as Java methods, closely tailored to work nicely with the program under test which was rather finicky and required quite a bit of javascript and such to navigate. The end result though, was a Java testcase template which the testers were able to easily copy and then use the keyword/methods I provided to script their way quickly through a testcase which would have been a real bear for them to do from scratch. It might be fun to go back sometime and try to make it read the commands from a simple file instead.

    Thanks always for the guidance and inspiration!

  4. Pingback: Challenge yourself, week 1. Test Automation Framework

  5. Great article, we reached similar conclusion of not going with extreme keyword driven approach by parsing excel. But we are going with Bot pattern which is a good for keyword based abstraction and the script is still a Java program so users can use it them on case by case basis rather than rigid text keywords.

Leave a Reply

Your email address will not be published. Required fields are marked *