| ion | HP (Mercury) QuickTest Professional | Rational Functional Tester | AutomatedQA - TestComplete | WATIR | Selenium | WET |
| Objects Recognition | Windows Handle based recognition and operation. Smart object recognition technology. This enables an easy and robust recognition of objects. | ScriptAssure Technology. This also enables an easy and robust recognition of objects. | Object Recognition as powerful as QTP's (if not more). Supports web-based and desktop applications as well as applications running on portable devices such as PDAs, Pocket PCs, and smartphones. Object name mapping is flexible and customizable (in TestComplete7). | Fairly good object recognition. However, when accessing dynamic menus, i had to use some workarounds. For example the following code is used to click a dynamic menu element. The menu is basically a table that pops when the mouse hovers on its title: | Good object recognition. Objects can be identified by name, ID or located by using XPath. | Supported |
|
| ie.image(:src, 'path_to_the_menu_tile_image/loanapp_mnu.gif').fire_event("onmouseover") |
| tempTable1 = ie.table(:id, 'Table_ID') |
| (tempTable1[8][1]).click() |
|
| This code simulates the user action of hovering on the menu title and clicking on the specified menu element. |
| Browsers Supported | IE, Firefox | IE, Firefox | IE7, IE8, up to Firefox 3.0.12 | IE, FF, Recently on Safari | IE (only partly because selenium IDE do not support IE), FF, Safari, Chrome and Opera | IE 6.0:version 6.0.2800.1106 |
| Recording and Playback | Stable recording and playback. Records in VBScript. | Stable recording and playback | Stable recording and playback. Can record "Keyword-based script", but normally records object-based operations | Watir Recorder ++ is available, But it's very simple recorder. Playback is stable. | Advanced Recorder: Selenium IDE | Supported |
| Operating System | Windows | Windows, Linux and Mac | Windows (including 64-bit) | Windows (IE, FF), Linux (FF), Mac (Safari, FF) | Windows, Linux, Mac | Supported Operating Systems: Windows 98/ME/2000 SP3/XP SP2/Server 2003 |
| Test Results Reports | XML based execution log. Every single step is logged. Viewed as collapsible/expandable tree. Could be manually modified to view as a web-page report (using XSL transformation script). From QTP 10 could be exported into other document formats. | Detailed reporting supported | Detailed execution log, displayed via built-in viewer. Can be exported as HTML and XML. Log shows details of checkpoint failures and image snapshots. | Watir has no reporting facilities. So we pumped our testing results dynamically to database while the script is running. And eventually, a nice grid should be developed to display the results in a user friendly way. | Reporting is supported through the test runner and various logs, screen shots can also be captured. | Results are displayed in a easily readable HTML format |
| Support for Terminal Application | Supported through an extension (TE-addin). | Supported through an extension | N/A | N/A | N/A | Not supported since it’s Mainly performed for testing web applications. |
| Programming Language | VBScript. Can also include external compiled modules. Can access objects and invoke methods using COM. | Java programming language and VB .Net | VBScript, DelphiScript, JScript, and JScript variants (e.g., C++Script, C#Script). Recommend using JScript (dialect of ECMAScript) | Ruby Scripting. An easy programming language. | Native language is 'Selenese', the scripts can be exported as C++, C#, Java, Ruby, Python and HTML. | WET scripts are written using Ruby |
| Version Control and sharing of scripts. | Scripts can be shared among automation developers, but HP Quality Center is necessary to achieve this. Can use own versioning control tool, but many things are binary (object repository) so hard to merge and diff. | Scripts can be shared among developers. We have two options: IBM Clearcase or the free CVS technology. We used CVS in previous SLSS project and it proved reliable. | Scripts can be shared among developers and run on standalone playback application called TestExecute. Integrates with VSS, ClearCase, Team Foundation Server via SCC Provider | Not in built, but can easily use something like TortoiseSVN which is free an easy to use. | Scripts can be saved in the native language or exported in a supported langauge for inclusion into test frameworks such as JUnit or NUnit. Version control can be done using any source control tool such as SVN. CVS, etc. | Not Supported |
| Speed of Execution | Fast, customizable in many ways. | Medium | Medium-Fast | Generallly fast. Watir speed varied a lot according to various network topologies. The speed is changed by a factor of 10 slower or faster. So you have to be careful and make sure that Watir server and the web application under test are neighboring. You can arrange that with your network admin. Of course this is the rule for all automation tools. | Speed of execution via the Selenium IDE can be varied, the setting is from 'fast' to 'slow'. If scripts have been exported to a supported language execution can be adjusted programmatically. | Medium , however the time needed for recording may vary from application to another |
| Datapools for Data dynamicity | "DataTable" Object - Set of worksheets. "Environment" Object - XML tree. Can easily create any required data model, or access Excel workbooks and XML trees directly through COM | Supported | Supports Database tables, Files, Images, XMLs, Excel via Storages. DB, CSV, and Excel can be accessed using TestComplete's DDT object | Supported through a third party driver. | Supported | Supported , data table supported using either Excel spreadsheet or XML |
| DB Access | Supp orted, through ADO | Supported | Supported via DDT or COM | Supported through a third party driver. | Supported | Supported through a third party driver. |
| Issues and Concerns | - Problems with Terminal Application Execution. | Speed is not optimized and demanding high hardware requirements. | AutomatedQA support via newsgroup is old and not standard. They really need to create a different web-based support system. Newsgroup does not work. | Very basic recorder that requires a lot of editing which is time consuming. | components are difficult to set-up. | - Hardly to deal with JavaScript’s error |
| Crashes occurs sporadically. |
| Recommend to use on IE: IE developer toolbar, Web metrics RIA script recorder for analyse frames, and index etc.. |
| - Recording may take too much time |
| - Very expensive to version control the scripts. |
| Sometimes, automation execution will be working great one on machine, but not the other. Same when playing back on different operating systems. |
| Online documents are not overly helpful as they assume knowledge and omit key details. Some files required for the set-up of RC and Grid are not part of the Selenium tools (e.g. ANT, JDK), this also means required set-up files contain errors in context of Selenium. | - Objects implemented using JavaScript are not recognized by the tool. |
|
|
|
|
| - Keyword driven not supported. |
|
| TestComplete has a slew of hard_to_reproduce issues that make it difficult for them to fix. |
| Selenium does not natively support features required by testers such as iteration or data driven testing and there is difficulty testing Flash that has to be overcome by changing Actionscripts. | - Automatic Exception Handling |
|
|
|
|
| - Interactive test debugging not supported |
|
| If an application crashes and Dr. Watson window appears, there is no uniform way to closing the Dr. Watson window to continue the automation execution. Currently, you have to create a third-party script/tool to detect Dr. Watson windows and excuse them. |
| To extend Selenium functionality it is common to combine a Selenium based automation framework with a language such as Ruby. This provides ways to implement iteration, etc. mentioned above and access Gems (in the case of Ruby). |
|
|
|
|
|
|
|
|
| TestComplete 7 introduces performance enhancements to the Object Browser (map of the application objects), but we find the enhancement breaks parts of our automation framework. |
|
|
|
|
|
|
|
|
|
|
| Your mileage may vary. |
|
|
|
| Price | The price could vary according to license. | Priced without Tax USD: | Named User Licenses: | FREE | FREE, Open Source | FREE |
| BM Rational Functional Tester Authorized User License and SW Subscription & Support 12 Months (D53NFLL) 5,821.00 | Enterprise with one year maintenance support $1999 |
| Standard with one year maintenance support $999 |
| IBM Rational Functional Tester Floating User License + SW Subscription & Support 12 Months (D530BLL) 11,235.00 |
|
| Floating User Licenses: |
| IBM Rational Functional Tester Authorized User Initial Fixed Term License + SW Subscription & Support 12 Months (D54SHLL) 3,146.00 | Enterprise with one year maintenance support $4499 |
| Standard with one year maintenance support $2999 |
| IBM Rational Functional Tester Extension Floating User License + SW Subscription & Support 12 Months (D59RMLL) 2,846.00 |
|
| URL |
|
| http://www.automatedqa.com/products/testcomplete/ | http://watir.com/ | http://seleniumhq.org |
|
