WatiN Performance Tips

We use WatiN for integration and user acceptance testing (UATs). These tests are the slowest part of our build process. We are obsessed with finding ways to make these test run faster. Here is a list of what we do to make our UATs run as fast as possible.


Use one IE instance

Creating new IE instances is one of the most expensive operations in using WatiN. We use the following in our TestFixtureSetUp.

protected static IE ie;[TestFixtureSetUp]public virtual void TestFixtureSetUp(){    if(ie == null)    {        ie = new IE();        OpenBrowserToHomePage();    }}

Avoid using ie.Element

All Element methods are slow. Always be as specific as possible (even when accessing the body element).Don’t

ie.Element(id).Click();

Do

ie.Link(id).Click();

Avoid driving the web site to the testing point

In the beginning, we started all of our tests at the site entry point. We then clicked links until we made it to the section of the site we wanted to test. This is a very expensive pattern. We realized this and started our tests at the test point.Don’t

OpenBrowserToHomePage();ClickLink("Properties for Sale");  // http://localhost/sales

Do

OpenBrowserTo(region, "sales");  // http://localhost/sales

More tips

Do you have any more tips? I would love to hear about them.

Advertisements

Integrating JsUnit with NUnit using WatiN

We have 500+ JsUnit tests on my current project. These tests are a very valuable asset to the team. Their true value is not realized unless they are integrated into our automated build process.

[Test]public void RunJSUnitTests(){    string testDirectory = ConfigurationManager.AppSettings["JsTestDir"];    ITestFileReader reader =        new ExcludeTestFileReader("jsunit",        new SuffixTestFileReader("tests.htm",        new TestFileReader(testDirectory)));    Assert.IsTrue(new JsUnitTestRunner(reader).Run());}

This NUnit test runs all of our JsUnit tests.ITestFileReader is the most crucial part of the above code. ITestFileReader is the type responsible for feeding JsUnit test files to the JsUnitTestRunner. There are currently three types of ITestFileReaders:TestFileReader

Returns each file in the directory, including subdirectories, with each call to GetNextTestFile. Expects the name of a directory as its parameter.

SuffixTestFileReader

Decorator for a ITestFileReader that ensures all test files end with a specific postfix. Expects a sufÅfix string and a ITestFileReader.

ExcludeTestFileReader

Decorator for a ITestFileReader that ensure only files that do not match the exclude string are returned from the GetNextTestFile.

When new JsUnitTestRunner(reader).Run() is executed a new JsUnit suite is generated and saved to all_tests.html. WatiN is then used to execute the suite in JsUnit’s test runner.Add the following to your App.config so JsUnitTestRunner can find your JsUnit installation. InstallDir is the directory where you have unzipped JsUnit.

<configSections>    <section name="JSUnit" type="System.Configuration.NameValueSectionHandler" /></configSections><JSUnit>    <add key="InstallDir" value="c:\Projects\jsunit\" /></JSUnit>

If you want more information, please ask and/or look through the unit tests used to create the JsUnitTestRunner. All comments and suggestions are welcome. All of the source code can be found in my svn repository.https://esterline.devguard.com/svn/code/WatiN.Utils/

Accessing Body from WatiN

We have added many WatiN tests over the last few weeks. Performance is becoming more important every day. Several of our tests must ensure that a class is present in a body tag.

protected void AssertSiteModeEquals(string mode){    StringAssert.EndsWith(mode, ie.Element("body").ClassName);}

This block of code took several seconds to execute. Accessing ie.Element is a very slow operation in WatiN. Below is a much faster way of performing the same assert.

protected void AssertSiteModeEquals(string mode){    StringAssert.EndsWith(mode, ie.HtmlDocument.body.className);}

Story Telling with WatiN

We use WatiN for integration and user acceptance testing (UATs).Example

[Test]public void PageForwardButtonRendersNextPage(){    ie.GoTo("http://localhost/offices/lists.rails");    ie.Link("fwid").Click();    Assert.AreEqual(2, int.Parse(ie.Select("pgs").SelectedItem))}

This version is not bad, but I have to think when I read it. This test has at least two problems.

  1. HTML specifics have leaked into the test
  2. Element ids (fwid, pgs) are not clear

This all adds up to a potentially brittle and unclear test. Below is the same test, but with our Domain Specific Language (DSL) applied.

[Test]public void PageForwardButtonRendersNextPage(){    OpenBrowserTo("offices/list.rails");    ClickPageForwardButton();    AssertCurrentPageNumberEquals(2);}

As you can see, the test is very readable (almost like a story). Not only does the DSL help keep our tests clear, but it also isolates us from HTML changes.I would encourage everyone to try this technique on your tests. It will make your tests more clear and reliable. Heck, your customer might even be able to read it ;)As always your comments are welcome.

WatiN, Watir and Selenium Reviewed

WatiN, Watir and Selenium are three of the most popular open source web application testing frameworks. We tried all three frameworks during our current project.Why do we (or anyone else) need a web application testing framework (WAT)?A WAT provides two purposes for us.

  1. User Acceptance Testing (UAT). UATs provide a defined ending point on an iteration. UATs also provide us a regression testing mechanism.
  2. Programmer Tests. Programmer tests are used to describe bugs from a users perspective and to provide integration testing.

What requirements do we have for a WAT?

  1. Integrate with NUnit.
  2. Work with IE.
  3. Fast.
  4. Reliable with a heavily AJAXed site.

Watir, Selenium and WatiN were each given a chance. Each framework was used several weeks before moving on to the next framework. I will provide the details of our experience with each.WatirWatir was our first choice. Over the course of several weeks, we wrote 50+ tests with Watir.

  1. Integration with NUnit is not Watir’s strongest point. Watir can be integrated using this technique described by Scott Hanselman. We did use the Hanselman technique to integrate Watir into our NUnit tests and for the most part it worked as advertised.
  2. Watir only works with IE (FireWatir).
  3. Watir had acceptable speed.
  4. Reliability became our biggest problem with Watir. Tests would hang or fail randomly. The problem became worse the more tests we wrote. We speculated that the reliability issues were related to Watir’s AJAX support (or lack there of). We eventually stopped running our Watir tests regularly. Having repeatable/reliable tests is very important, so we decided to try a new framework.

Please see my update for more information.SeleniumSelenium was our next choice. It was a big departure Watir, but we had an experienced guide for our new journey. Over the course of several weeks, we wrote 20+ tests with Selenium.

  1. Integration with NUnit is great. You can actually write C# code to drive Selenium.
  2. Selenium works with just about any browser, so working with IE was not a prob`lem.
  3. Selenium was very slow for us.
  4. Selenium was not as good at testing an AJAX site as I had hoped. Selenium does not seem to know when the browser is done working. Our typical Selenium tests was full of waits (wait for text, wait for elements). This caused our slow performance as indicated by number three.

This is an example of one of our Selenium tests. As you can see, we needed 7 waits/sleeps in order to make the test work.

public void MinAndMaxPriceRestoredWhenOpenedAfterUsingBackButton(){    OpenBrowserTo("welcome/index.rails");    bot.Click("priceDT");    WaitForText("Price Range");    WaitForText("515 N. County Road");    bot.Select("MaxDropDownList", "$5,000,000");    WaitForText("Prewar 8 Off Fifth Avenue");    bot.Select("MinDropDownList", "$2,000,000");    WaitForText("of 86");    bot.Click("link=Prewar 8 Off Fifth Avenue");    WaitForText("Rarely available triple mint restoration");    bot.GoBack();    Thread.Sleep(20000);    bot.Click("priceDT");    WaitForText("Price Range");    Assert.AreEqual("$5,000,000", bot.GetSelectedLabel("MaxDropDownList"));    Assert.AreEqual("$2,000,000", bot.GetSelectedLabel("MinDropDownList"));}

WatiNWatiN is our latest choice. Given our experience with Watir, we hoped it would be an easy transition into WatiN. It was. We currently have 70+ tests written.

  1. Integration with NUnit is great.
  2. WaitN, just as Watir, only works with IE.
  3. WaitN has been very responsive for us. I think this is due to the great AJAX support built into WatiN.
  4. Great AJAX support (best of the three). WatiN just works for most of our AJAX.

This is an example of one of our WatiN tests (converted from the Selenium test above). This test required no waits.

public void MinAndMaxPriceRestoredWhenOpenAfterUsingBackButton(){    OpenBrowserTo("welcome/index.rails");    ClickLink("Price");    SelectMaxPrice("$5,000,000");    SelectMinPrice("$2,000,000");    ClickLink("Prewar 8 Off Fifth Avenue")    GoBack();    ClickLink("Price");    Assert.AreEqual("$5,000,000", SelectedMaxPrice());    Assert.AreEqual("$2,000,000", SelectedMinPrice());}

ConclusionOut of the three frameworks we evaluated, WatiN is the clear winner. WatiN has everything and more that we were looking for in a WAT.As always comments are always welcome.