Your Index into
Software Test Automation

Preface

Originally written in December 1997 by Thomas Arnold, this article still holds helpful information for those embarking on an automation project for the first time using Visual Test 6.x with minimal programming experience. (See also, "Visual Test 6 Bible" available from Amazon.com).

"Building an Automation Framework with Rational Visual Test"
by Thomas Arnold

When I first sat down to work with Visual Test – then known as Microsoft Test 1.0 (Beta) – I had no idea where to begin. Supposedly the tool would allow me to automate portions of my testing effort. Significant portions? I didn’t know. All I knew was that I was managing a six-person testing team that was ramping up its testing approach, on a short time line, and the last thing I wanted to deal with was a tool that I knew nothing about.

I threw it on the pile of other stuff on my desk and figured I’d get to it later, as-in six months down the road when the product had shipped. The layers on my desk made me think of how geologists study the layers of sedimentary rock to find clues to the past. When I got to the point of digging down through these layers I’d come across this tool again and – perhaps – place an order for the non-Beta version.

It turned out it wasn’t a bullet I could dodge, however. I was asked by one of the muckity-mucks to give a demonstration to some of the other, higher-up, muckity-mucks about how this tool worked and how we could apply it to our existing testing efforts. I had two days to learn it and have something worthwhile to show.

Defining "Framework"

When I began learning Visual Test I started by picking a simple feature of a Windows application that I knew to be stable. I picked the Open dialog box in the Microsoft® Notepad application. I picked it because I knew that when I began to write source code – automation script – for a small piece of the product the rest of the skeleton or framework would slowly form and fall in place. Like any industry, the software industry has it’s own set of jargon to help communicate thoughts and ideas quickly. Framework is one of those words that gives you a general sense of the meaning but leaves you wondering if you really understand what’s behind the definition of that word.

My definition of framework is simple and practical. In this article, it is a rudimentary definition: a framework is nothing more than a bunch of pieces that, when combined, make it easier for anyone creating automation scripts to freely add to those scripts without a lot of busy work. These pieces can be anything from coding guidelines to common utilities to external file relationships.

A framework is built over time. Just knowing that the framework will grow and evolve makes the job easier since you don’t have to worry about doing it exactly right the first time. And, with a flexible tool like Visual Test, there is no one "right" way to create a test automation approach.

It Begins

The first version of this script is simple. I’m taking the approach that I want to simulate a user’s actions by dropping down the File menu, selecting the Open... menu item to display the Open dialog box, and typing in the name of a file to be opened into the Notepad application. The first draft of this script is shown in Listing 1.

This simple, first script does exactly what we ask it to. The run statement causes an instance of Microsoft Notepad to be executed and displayed. The nowait parameter instructs Visual Test to continue executing instead of waiting for Notepad to terminate before continuing on.

The next three commands – wMenuSelect, wEditSetText, wButtonClick – are Visual Test statements for working with an application’s user interface. WMenuSelect will cause the File menu to drop down and the Open... menu item to be selected. WEditSetText will set the text of an edit control that, in this case, has a label next to it with the text File name:. The text bootlog.txt is what is placed into that edit control. And lastly, the wButtonClick command will cause a mouse-event to be generated just as it would if a user clicked upon the dialog box’s button labeled Open. (In case you’re wondering, Visual Test’s language is not case-sensitive. You can use capitalized and lowercased letters as you see fit).

There are some problems with this script that need to be addressed. The first is that the script will run an instance of Microsoft Notepad even if it is already up and running. The result is one additional copy of Notepad loaded into memory. This isn’t harmful, but it can get confusing and downright messy.

The other problem is that the script doesn’t clean up after itself. When it finishes its test – in this example: opening a text file – it doesn’t shut down the instance of Microsoft Notepad that it invoked. If this script is run three times there will be three additional copies of Notepad loaded in memory and sitting on the desktop.

Set-up and Clean-up

The next version of the script first checks to see if an instance of Notepad already exists. If it’s already up and running why run another copy? It’s arguable that there may be a reason why Notepad would be up and running and such a special case can be programmed into the script. To keep this simple we’re going to operate on the assumption that it is preferable to have only one copy of the program being tested running at a given time. With that in mind, our previous test script now evolves into the script shown in Listing 2.

Listing 2 – Start-up and clean-up code is added to the previous script. '*** Start-up code if wFndWnd("- Notepad", FW_PART OR FW_ALL OR FW_FOCUS) = 0 then   print "Notepad window not found."   print "Attempting to run Notepad."   if run("notepad.exe") <> 0 then     print "Unable to run Notepad."     print "Tests stopped."     stop   endif endif '*** Test wMenuSelect("File\Open") wEditSetText("File name:","bootlog.txt") wButtonClick("Open") '*** Clean-up code if wFndWnd("- Notepad", FW_PART OR FW_ALL OR FW_FOCUS) then   wMenuSelect("File\Exit") endif print "Test completed on "+DATETIME$ END

The script now checks to see if Notepad is running and, if it is, activates the window and gives it the focus. If it isn’t already running then it runs it. If an error occurs when attempting to run Notepad then our script reports the problem and terminates. Notice also that the run statement has now been replaced with a function of the same name. The run function returns a 0 if it succeeds in finding and executing Notepad. If it fails, it returns a number that corresponds to why it failed (e.g. low memory, file not found).

When the script completes, the computer is set back to the state it was in prior to execution. In this case it’s simple: the script shuts down Notepad.

Coding Guidelines

One of the most important parts of a programming effort – and therefore an important part in constructing a framework – is following agreed-upon guidelines. These guidelines range from where the source code is stored, to how the variables are named, to comments.

Many software development companies already have a coding standards document that a quality assurance group can borrow and modify for their efforts. Because coding guidelines are going to be enforced and used by anyone on the team writing source code, it is important that feedback and input be solicited during the definition of such standards. Guidelines range all over the board depending on an individual’s background. I’ve seen programmers come close to blows over an argument as simple as where the open and close brackets should be placed in a code-block under and IF statement in the C/C++ programming language. Work these issues out up front and make sure everyone agrees to follow those guidelines. (Refer to Chapter 8 of Visual Test 6 Bible, ISBN # 0-7645-3255-3).

One example of a coding standard is using constants for any value that is used in more than one place, as shown in Listing 3. Updates to source code like this should continue throughout the project causing the script to be more manageable and flexible as it continues to grow. This is a key part of programming alone, not to mention a part of building a framework.

Listing 3 – The script continues to evolve. Script writers should take every opportunity to clean up the code to keep it manageable and flexible. CONST CAP_MAINWIN$ = "- Notepad" CONST FW_MYCRITERIA& = FW_PART OR FW_ALL OR FW_FOCUS CONST FN_TESTAPP$ = "notepad.exe" CONST MNU_FILE_OPEN$ = "File\Open..." CONST MNU_FILE_EXIT$ = "File\Exit" CONST LBL_FILENAME$ = "File name:" CONST BTN_OPEN$ = "Open" '*** Start-up code if wFndWnd(CAP_MAINWIN, FW_MYCRITERIA) = 0 then   print FN_TESTAPP+" window not found."   print "Attempting to run "+FN_TESTAPP   if run(FN_TESTAPP) = 0 then     print "Unable to run "+FN_TESTAPP     print "Tests stopped."     stop   endif endif '*** Test wMenuSelect(MNU_FILE_OPEN) wEditSetText(LBL_FILENAME,"bootlog.txt") wButtonClick(BTN_OPEN) '*** Clean-up code if wFndWnd(CAP_MAINWIN, FW_MYCRITERIA) then   wMenuSelect(MNU_FILE_EXIT) endif print "Test completed on "+DATETIME$ END

Many coding examples are obvious, especially to anyone who has taken one or more programming classes. But many times test engineers, while technical in their areas of testing, don’t necessarily have programming backgrounds. This is fine because programming is not a requirement nor in most cases a necessity to verify the quality of a product, especially in black box testing. It is necessary to have a programming background, however, when using an automation tool like Visual Test.

Such guidelines as using constants, although obvious to programmers, requires examination and discussion. Some of the best programmers in the software industry write incredibly complicated code that does wonderful things, but is a pain to maintain because simple programming habits aren’t always followed. (Refer to Writing Solid Code by Steve Maguire, ISBN #1556155514).

And On It Goes

Refinement of the source code continues. Our initial 5-line script has evolved into the 36-line script shown in Listing 4. As with any source code that can be used multiple times, that code must be encapsulated into a subroutine or function. This is yet another obvious step for programmers and also an intermediate step to moving the code into a separate file.

Listing 4 – Potentially sharable code has been moved into subroutines as an intermediate step to eventually moving into a separate include file. '*** Constant decls CONST CAP_MAINWIN$ = "- Notepad" CONST FW_MYCRITERIA& = FW_PART OR FW_ALL OR FW_FOCUS CONST FN_TESTAPP$ = "notepad.exe" CONST MNU_FILE_OPEN$ = "File\Open..." CONST MNU_FILE_EXIT$ = "File\Exit" CONST LBL_FILENAME$ = "File name:" CONST BTN_OPEN$ = "Open" '*** Program begins StartTest   wMenuSelect(MNU_FILE_OPEN)   wEditSetText(LBL_FILENAME,"bootlog.txt")   wButtonClick(BTN_OPEN) StopTest END '*** Subroutines '*** Start-up code sub StartTest   if wFndWnd(CAP_MAINWIN, FW_MYCRITERIA) = 0 then     print FN_TESTAPP+" window not found."     print "Attempting to run "+FN_TESTAPP     if run(FN_TESTAPP) = 0 then       print "Unable to run "+FN_TESTAPP       print "Tests stopped."       stop     endif   endif end sub 'StartTest '*** Clean-up code sub StopTest   if wFndWnd(CAP_MAINWIN, FW_MYCRITERIA) then     wMenuSelect(MNU_FILE_EXIT)   endif   print "Test completed on "+DATETIME$ end sub 'StopTest

The initialization routines introduced in Listing 2 have now been moved into separate, encapsulated blocks of code. Because these routines are helpful to other scripts that will eventually be written for other areas of the product under test, and potentially by other teammates, they need to be shared somehow so that others have access to those routines.

This is done in Visual Test – and every other language – by moving the shareable, reusable code into a separate file. In the case of Visual Test it’s a .INC file (similar to a .h file in C) which is then included using compile-time commands that insert the source code into the block of code that references it. By moving the sharable routines – in this case StartTest and StopTest – and also moving the constants that are applicable to other test files that will be created (.MST files in Visual Test) we arrive at the final version of the script shown in Listing 5.

Listing 5 – The constants have been moved into notemain.inc and the sharable routines have been moved into noteutil.inc. '$include "notemain.inc" '$include "noteutil.inc" '*** Program begins StartTest   '*** Test      wMenuSelect(MNU_FILE_OPEN)   wEditSetText(LBL_FILENAME,"bootlog.txt")   wButtonClick(BTN_OPEN) StopTest END

In this final version we have accomplished many things. The first is that with the sharable code moved into a separate file, others have access to those common routines. In short, they don’t have to re-invent the wheel when they want to have routines that, in this case, set up the test (StartTest) and then clean up after the test is completed (StopTest). Although simplistic in this example, the utility file will continue to grow with routines that handle navigation, logging, initialization, tracking, benchmarking, and so on.

The second is that the code that represents the test case is now cleaner, more manageable, and much easier to read. When a new teammate comes on board they don’t need to know what noteutil.inc looks like, they only need to know what routines it has and how they work. The coding has been simplified so that those new to programming in Visual Test can follow by example and allow the more experienced programmers to add the complex routines. This is one of the key reasons for having a framework, whether it’s a framework in Microsoft Visual C++ or a framework created under Visual Test by a testing project’s own group of test engineers.

Flexibility is the third benefit. Should a menu item or button caption’s text change, as is the case with a product under development, one need only go to the file that holds the declarations that refer to those captions. Make the change in one place and the potentially hundreds of scripts that are using the framework inherit those changes.

Global updates is yet another benefit in the long list of reasons to create a structured framework approach to your automation scripts. As someone builds a better mouse trap, perhaps by updating and increasing the speed of a search or sorting routine used by many of the test files, those improvements propagate through the entire automation structure.

Final Thoughts

This article was written to help those new to automation understand that it is important to move forward when attempting to set up an automation approach. Planning is helpful and key, but when you’re new to a tool the easiest approach is to pick a simple aspect of a product and start banging out code. The simple act of picking a small part of the product to automate creates the initial skeleton or framework that will continue to be added to throughout a test automation project. This was purposely presented in a simplistic manner to point out that there is no magic to creating automated scripts. Following this simple example will help create flexible, sharable and structured automation code base so that those who inherit your code, or who come onto the testing team late in the game, can be effective in their automation efforts in a shorter period of time.

Copyright 2002, Thomas R. Arnold II. All rights reserved.