WatiN, Watir and Selenium Reviewed

WatiN, Watir and Selenium are three of the most popular open source web application testing frameworks. We tried all three frameworks during our current project.Why do we (or anyone else) need a web application testing framework (WAT)?A WAT provides two purposes for us.

  1. User Acceptance Testing (UAT). UATs provide a defined ending point on an iteration. UATs also provide us a regression testing mechanism.
  2. Programmer Tests. Programmer tests are used to describe bugs from a users perspective and to provide integration testing.

What requirements do we have for a WAT?

  1. Integrate with NUnit.
  2. Work with IE.
  3. Fast.
  4. Reliable with a heavily AJAXed site.

Watir, Selenium and WatiN were each given a chance. Each framework was used several weeks before moving on to the next framework. I will provide the details of our experience with each.WatirWatir was our first choice. Over the course of several weeks, we wrote 50+ tests with Watir.

  1. Integration with NUnit is not Watir’s strongest point. Watir can be integrated using this technique described by Scott Hanselman. We did use the Hanselman technique to integrate Watir into our NUnit tests and for the most part it worked as advertised.
  2. Watir only works with IE (FireWatir).
  3. Watir had acceptable speed.
  4. Reliability became our biggest problem with Watir. Tests would hang or fail randomly. The problem became worse the more tests we wrote. We speculated that the reliability issues were related to Watir’s AJAX support (or lack there of). We eventually stopped running our Watir tests regularly. Having repeatable/reliable tests is very important, so we decided to try a new framework.

Please see my update for more information.SeleniumSelenium was our next choice. It was a big departure Watir, but we had an experienced guide for our new journey. Over the course of several weeks, we wrote 20+ tests with Selenium.

  1. Integration with NUnit is great. You can actually write C# code to drive Selenium.
  2. Selenium works with just about any browser, so working with IE was not a prob`lem.
  3. Selenium was very slow for us.
  4. Selenium was not as good at testing an AJAX site as I had hoped. Selenium does not seem to know when the browser is done working. Our typical Selenium tests was full of waits (wait for text, wait for elements). This caused our slow performance as indicated by number three.

This is an example of one of our Selenium tests. As you can see, we needed 7 waits/sleeps in order to make the test work.

public void MinAndMaxPriceRestoredWhenOpenedAfterUsingBackButton(){    OpenBrowserTo("welcome/index.rails");    bot.Click("priceDT");    WaitForText("Price Range");    WaitForText("515 N. County Road");    bot.Select("MaxDropDownList", "$5,000,000");    WaitForText("Prewar 8 Off Fifth Avenue");    bot.Select("MinDropDownList", "$2,000,000");    WaitForText("of 86");    bot.Click("link=Prewar 8 Off Fifth Avenue");    WaitForText("Rarely available triple mint restoration");    bot.GoBack();    Thread.Sleep(20000);    bot.Click("priceDT");    WaitForText("Price Range");    Assert.AreEqual("$5,000,000", bot.GetSelectedLabel("MaxDropDownList"));    Assert.AreEqual("$2,000,000", bot.GetSelectedLabel("MinDropDownList"));}

WatiNWatiN is our latest choice. Given our experience with Watir, we hoped it would be an easy transition into WatiN. It was. We currently have 70+ tests written.

  1. Integration with NUnit is great.
  2. WaitN, just as Watir, only works with IE.
  3. WaitN has been very responsive for us. I think this is due to the great AJAX support built into WatiN.
  4. Great AJAX support (best of the three). WatiN just works for most of our AJAX.

This is an example of one of our WatiN tests (converted from the Selenium test above). This test required no waits.

public void MinAndMaxPriceRestoredWhenOpenAfterUsingBackButton(){    OpenBrowserTo("welcome/index.rails");    ClickLink("Price");    SelectMaxPrice("$5,000,000");    SelectMinPrice("$2,000,000");    ClickLink("Prewar 8 Off Fifth Avenue")    GoBack();    ClickLink("Price");    Assert.AreEqual("$5,000,000", SelectedMaxPrice());    Assert.AreEqual("$2,000,000", SelectedMinPrice());}

ConclusionOut of the three frameworks we evaluated, WatiN is the clear winner. WatiN has everything and more that we were looking for in a WAT.As always comments are always welcome.

Advertisements

4 Responses

  1. The way Selenium waits is a trade-off in the bigger battle of browser compatibility testing. You see, only IE has a nice and easy “documentReadyState” that can be polled in either “traditional” request/response apps or Ajax apps to know when it’s safe to continue. Other popular browsers (notably Firefox), does not have a documentReadyState function, so Selenium has to go to great links to know when Firefox is truly done loading a webpage (or Ajax callback). DocumentReadyState is an IE innovation (like XMLHttpRequest) that we’d like to see make its way to all other browsers. It would make Selenium’s job a lot easier!

    Anyway, I’m glad Selenium was in the running, even though it didn’t win in the end for you. Regarding the slowness of Selenium, when you use commands like ” Thread.Sleep(20000);”… of course, your tests are going to run slow. 😉 On a serious note, though, one way to combat the slowness is to break up your tests and run them in parallel on many machines (or in many threads on one machine). It’s not easy to setup, but it is a proven technique to reduce total test times. In the past, I’ve used IronPython to parse the .Net test suites, split them up, run them in threads, and then aggregate the results back to Nant and CC.Net. Ruby could probably do the job just as well. As more and machines are going multi-core, it’s an important option to consider using.

    Cheers,
    jason huggins

  2. s/great links/great *lengths*/

    Sorry for the unintentional pun. 🙂

  3. Thanks for the reply. We will have to give the multiple machines a try.

    You are correct 20 seconds is a long time to sleep. I am not sure why the sleep was that long in this test. I converted it over to WatiN, but I did not write the original Selenium test. The more important parts were all the other waits as they occur in every one of the other Selenium tests.

    I am sure we will give Selenium another shot at some point in the future. One of my peers really likes it and wanted us to keep using it. We liked everything about it but the waits.

  4. We would appreciate any leads you can give about where you found reliability problems with Watir. Our apologies for your troubles.

Comments are closed.

%d bloggers like this: