Tuesday, July 13, 2010
IEEE 829 standard for software testing documentation
IEEE 829 standard for software testing documentation
One of the challenges facing software testers has been the availability of an agreed set of document standards and templates for testing. The IEEE 829 provides an internationally recognised set of standards for test planning documentation.
IEEE 829 has been developed specifically with software testing in mind and is applicable to each stage of the testing life cycle including system and acceptance testing.
Types of Document
The IEEE 829 standard covers 8 document types.
Test specification
• Test Plan: Covers how the testing will be managed, scheduled and executed.
• Test Design Specification: defines logically what needs to be tested by examining the requirements or features. Then these requirements can be converted into test conditions.
• Test Case Specification: Converts the test conditions into test cases by adding real data, pre-conditions and expected results
• Test Procedure: Describes in practical terms how the tests are run.
• Test Item Transmittal Report: Specify the items released for testing.
Test execution
• Test Log: Is an audit trail that records the details of tests in chronologically.
• Test Incident Report: Record details of any unexpected events and behaviours that need to be investigated.
Test reporting
• Test Summary Report: Summarise and evaluate tests.
Documentation for test specification
The test preparation is by far the most important part of any software testing project. During this stage you must create your tests and define the requirements of your test environment.
IEEE 829 - Test Plan
The Test Plan describes how you will deliver the testing.
1. Test Plan Identifier
2. References
3. Introduction
4. Test Items
5. Software Risk Issues
6. Features to be tested
7. Features not to be tested
8. Approach
9. Item Pass/Fail Criteria
10. Suspension Criteria and Resumption Requirements
11. Test Deliverables
12. Remaining Test Tasks
13. Environmental Needs
14. Staffing and Training Needs
15. Responsibilities
16. Schedule
17. Planning Risks and Contingencies
18. Approvals
19. Glossary
IEEE 829 - Test Design Specification
The test design specification identifies the test conditions from the requirements and functional design documentation.
Let’s use a Banking project example where the following testing requirements have been defined. In this case Bank A is rolling out new “hole in the wall” machines all over the country and the project will include testing the functionality of the ATMs to demonstrate the ability to:
• Complete a valid withdrawal of funds.
• Print a report summary of recent transactions.
• Change passwords.
The test design does not record the values to be entered for a test, but describes the requirements for defining those values. This is done at a logical level and should not be cluttered with individual examples and data.The design specification provides a link between test requirements and test cases.
IEEE 829 - Test Case Specification
The test cases can be produced when the test design is completed. A test case may cover one or more test conditions derived from the test design. A test case should include:
• The precise data that is required to execute the test. This is a combination of input data, and application and system data that is needed.
• The expected results and outputs
• Pre-conditions that will determine the starting point for each test. A feature (requirement) from the test design may be tested in more than one test case, and a test case may test more than one feature. The aim is for a set of test cases to test each feature (requirement) in the test design at least once. Taking the ATM project example the first 2 requirements could be tested using one test case:
• Test case 1 could be defined to include the completion of a cash withdrawal from the ATM and then a printout request to show this withdrawal has been correctly executed and the right amount has been debited.
IEEE 829 - Test Procedure Specification The test procedures are developed from both the test design and the test case specification. The procedure document describes how the tester will physically run the test, the physical set-up required, and the procedure steps that need to be followed. The standard defines ten procedure steps that may be applied when running a test.
IEEE 829 - Test Item Transmittal Report
This document is a handover document and provides details of the previous stage of testing.
Similar to a release note this provides a list of what is being delivered and shows any changes and new items contained. It includes the person responsible for each item, its physical location and its status.
Documentation for Test execution
The schedule of what test cases are run and when, is defined in the Test Plan. The test results are recorded in the test log, and in test incident reports.
IEEE 829 - Test Log
The Test Log records the details of what Test Cases have been run, the order of their running, and the results of the test. The results are either the test passed, meaning that the actual and expected results were identical, or it failed and that there was a discrepancy. If there is a discrepancy than one or more Test Incident Reports are raised or updated, and their identities recorded on the Test Log.
The Test Log is important as it allows progress of the testing to be checked, as well as providing valuable information for finding out what caused an incident. If an incident is a coding fault, the fault may have occurred not in the Test Case that failed but in one that was run previously. Thus the sequence of the tests enables the fault to be found.
IEEE 829 -Test Incident Report
This report documents any event that requires subsequent investigation. An incident should be raised when there is an unexpected result or any unexpected behaviour during testing. At this point it may not always be clear whether there is a bug or fault in the software, since incidents can occur as a result of configuration errors, faults in the software, faults in the requirements and in-correct expected results recorded in the test case.
The report consists of all details of the incident such as actual and expected results, when it failed, and any supporting evidence that will help in its resolution. The report will also include, if possible, an assessment of the impact upon testing of an incident.
Documentation for test reporting
Eventually testing will be completed according to the criteria specified in the Test Plan. This is when the success or failure of the system is decided based on the results. The Test Summary records this information.
IEEE 829 - test summary
The summary provides the results of the designated testing activities and an evaluation of these results.
Moreover the summary provides an overall view of the testing and the quality of the software.
Use of the standard
The 829 standard is a good start point to create a sensible structure for your testing documents. However, it should be customised or changed to reflect the needs of your project.
One of the challenges facing software testers has been the availability of an agreed set of document standards and templates for testing. The IEEE 829 provides an internationally recognised set of standards for test planning documentation.
IEEE 829 has been developed specifically with software testing in mind and is applicable to each stage of the testing life cycle including system and acceptance testing.
Types of Document
The IEEE 829 standard covers 8 document types.
Test specification
• Test Plan: Covers how the testing will be managed, scheduled and executed.
• Test Design Specification: defines logically what needs to be tested by examining the requirements or features. Then these requirements can be converted into test conditions.
• Test Case Specification: Converts the test conditions into test cases by adding real data, pre-conditions and expected results
• Test Procedure: Describes in practical terms how the tests are run.
• Test Item Transmittal Report: Specify the items released for testing.
Test execution
• Test Log: Is an audit trail that records the details of tests in chronologically.
• Test Incident Report: Record details of any unexpected events and behaviours that need to be investigated.
Test reporting
• Test Summary Report: Summarise and evaluate tests.
Documentation for test specification
The test preparation is by far the most important part of any software testing project. During this stage you must create your tests and define the requirements of your test environment.
IEEE 829 - Test Plan
The Test Plan describes how you will deliver the testing.
1. Test Plan Identifier
2. References
3. Introduction
4. Test Items
5. Software Risk Issues
6. Features to be tested
7. Features not to be tested
8. Approach
9. Item Pass/Fail Criteria
10. Suspension Criteria and Resumption Requirements
11. Test Deliverables
12. Remaining Test Tasks
13. Environmental Needs
14. Staffing and Training Needs
15. Responsibilities
16. Schedule
17. Planning Risks and Contingencies
18. Approvals
19. Glossary
IEEE 829 - Test Design Specification
The test design specification identifies the test conditions from the requirements and functional design documentation.
Let’s use a Banking project example where the following testing requirements have been defined. In this case Bank A is rolling out new “hole in the wall” machines all over the country and the project will include testing the functionality of the ATMs to demonstrate the ability to:
• Complete a valid withdrawal of funds.
• Print a report summary of recent transactions.
• Change passwords.
The test design does not record the values to be entered for a test, but describes the requirements for defining those values. This is done at a logical level and should not be cluttered with individual examples and data.The design specification provides a link between test requirements and test cases.
IEEE 829 - Test Case Specification
The test cases can be produced when the test design is completed. A test case may cover one or more test conditions derived from the test design. A test case should include:
• The precise data that is required to execute the test. This is a combination of input data, and application and system data that is needed.
• The expected results and outputs
• Pre-conditions that will determine the starting point for each test. A feature (requirement) from the test design may be tested in more than one test case, and a test case may test more than one feature. The aim is for a set of test cases to test each feature (requirement) in the test design at least once. Taking the ATM project example the first 2 requirements could be tested using one test case:
• Test case 1 could be defined to include the completion of a cash withdrawal from the ATM and then a printout request to show this withdrawal has been correctly executed and the right amount has been debited.
IEEE 829 - Test Procedure Specification The test procedures are developed from both the test design and the test case specification. The procedure document describes how the tester will physically run the test, the physical set-up required, and the procedure steps that need to be followed. The standard defines ten procedure steps that may be applied when running a test.
IEEE 829 - Test Item Transmittal Report
This document is a handover document and provides details of the previous stage of testing.
Similar to a release note this provides a list of what is being delivered and shows any changes and new items contained. It includes the person responsible for each item, its physical location and its status.
Documentation for Test execution
The schedule of what test cases are run and when, is defined in the Test Plan. The test results are recorded in the test log, and in test incident reports.
IEEE 829 - Test Log
The Test Log records the details of what Test Cases have been run, the order of their running, and the results of the test. The results are either the test passed, meaning that the actual and expected results were identical, or it failed and that there was a discrepancy. If there is a discrepancy than one or more Test Incident Reports are raised or updated, and their identities recorded on the Test Log.
The Test Log is important as it allows progress of the testing to be checked, as well as providing valuable information for finding out what caused an incident. If an incident is a coding fault, the fault may have occurred not in the Test Case that failed but in one that was run previously. Thus the sequence of the tests enables the fault to be found.
IEEE 829 -Test Incident Report
This report documents any event that requires subsequent investigation. An incident should be raised when there is an unexpected result or any unexpected behaviour during testing. At this point it may not always be clear whether there is a bug or fault in the software, since incidents can occur as a result of configuration errors, faults in the software, faults in the requirements and in-correct expected results recorded in the test case.
The report consists of all details of the incident such as actual and expected results, when it failed, and any supporting evidence that will help in its resolution. The report will also include, if possible, an assessment of the impact upon testing of an incident.
Documentation for test reporting
Eventually testing will be completed according to the criteria specified in the Test Plan. This is when the success or failure of the system is decided based on the results. The Test Summary records this information.
IEEE 829 - test summary
The summary provides the results of the designated testing activities and an evaluation of these results.
Moreover the summary provides an overall view of the testing and the quality of the software.
Use of the standard
The 829 standard is a good start point to create a sensible structure for your testing documents. However, it should be customised or changed to reflect the needs of your project.
Test Automation Tools Comparison Matrix
| ion | HP (Mercury) QuickTest Professional | Rational Functional Tester | AutomatedQA - TestComplete | WATIR | Selenium | WET |
| Objects Recognition | Windows Handle based recognition and operation. Smart object recognition technology. This enables an easy and robust recognition of objects. | ScriptAssure Technology. This also enables an easy and robust recognition of objects. | Object Recognition as powerful as QTP's (if not more). Supports web-based and desktop applications as well as applications running on portable devices such as PDAs, Pocket PCs, and smartphones. Object name mapping is flexible and customizable (in TestComplete7). | Fairly good object recognition. However, when accessing dynamic menus, i had to use some workarounds. For example the following code is used to click a dynamic menu element. The menu is basically a table that pops when the mouse hovers on its title: | Good object recognition. Objects can be identified by name, ID or located by using XPath. | Supported |
| ie.image(:src, 'path_to_the_menu_tile_image/loanapp_mnu.gif').fire_event("onmouseover") | ||||||
| tempTable1 = ie.table(:id, 'Table_ID') | ||||||
| (tempTable1[8][1]).click() | ||||||
| This code simulates the user action of hovering on the menu title and clicking on the specified menu element. | ||||||
| Browsers Supported | IE, Firefox | IE, Firefox | IE7, IE8, up to Firefox 3.0.12 | IE, FF, Recently on Safari | IE (only partly because selenium IDE do not support IE), FF, Safari, Chrome and Opera | IE 6.0:version 6.0.2800.1106 |
| Recording and Playback | Stable recording and playback. Records in VBScript. | Stable recording and playback | Stable recording and playback. Can record "Keyword-based script", but normally records object-based operations | Watir Recorder ++ is available, But it's very simple recorder. Playback is stable. | Advanced Recorder: Selenium IDE | Supported |
| Operating System | Windows | Windows, Linux and Mac | Windows (including 64-bit) | Windows (IE, FF), Linux (FF), Mac (Safari, FF) | Windows, Linux, Mac | Supported Operating Systems: Windows 98/ME/2000 SP3/XP SP2/Server 2003 |
| Test Results Reports | XML based execution log. Every single step is logged. Viewed as collapsible/expandable tree. Could be manually modified to view as a web-page report (using XSL transformation script). From QTP 10 could be exported into other document formats. | Detailed reporting supported | Detailed execution log, displayed via built-in viewer. Can be exported as HTML and XML. Log shows details of checkpoint failures and image snapshots. | Watir has no reporting facilities. So we pumped our testing results dynamically to database while the script is running. And eventually, a nice grid should be developed to display the results in a user friendly way. | Reporting is supported through the test runner and various logs, screen shots can also be captured. | Results are displayed in a easily readable HTML format |
| Support for Terminal Application | Supported through an extension (TE-addin). | Supported through an extension | N/A | N/A | N/A | Not supported since it’s Mainly performed for testing web applications. |
| Programming Language | VBScript. Can also include external compiled modules. Can access objects and invoke methods using COM. | Java programming language and VB .Net | VBScript, DelphiScript, JScript, and JScript variants (e.g., C++Script, C#Script). Recommend using JScript (dialect of ECMAScript) | Ruby Scripting. An easy programming language. | Native language is 'Selenese', the scripts can be exported as C++, C#, Java, Ruby, Python and HTML. | WET scripts are written using Ruby |
| Version Control and sharing of scripts. | Scripts can be shared among automation developers, but HP Quality Center is necessary to achieve this. Can use own versioning control tool, but many things are binary (object repository) so hard to merge and diff. | Scripts can be shared among developers. We have two options: IBM Clearcase or the free CVS technology. We used CVS in previous SLSS project and it proved reliable. | Scripts can be shared among developers and run on standalone playback application called TestExecute. Integrates with VSS, ClearCase, Team Foundation Server via SCC Provider | Not in built, but can easily use something like TortoiseSVN which is free an easy to use. | Scripts can be saved in the native language or exported in a supported langauge for inclusion into test frameworks such as JUnit or NUnit. Version control can be done using any source control tool such as SVN. CVS, etc. | Not Supported |
| Speed of Execution | Fast, customizable in many ways. | Medium | Medium-Fast | Generallly fast. Watir speed varied a lot according to various network topologies. The speed is changed by a factor of 10 slower or faster. So you have to be careful and make sure that Watir server and the web application under test are neighboring. You can arrange that with your network admin. Of course this is the rule for all automation tools. | Speed of execution via the Selenium IDE can be varied, the setting is from 'fast' to 'slow'. If scripts have been exported to a supported language execution can be adjusted programmatically. | Medium , however the time needed for recording may vary from application to another |
| Datapools for Data dynamicity | "DataTable" Object - Set of worksheets. "Environment" Object - XML tree. Can easily create any required data model, or access Excel workbooks and XML trees directly through COM | Supported | Supports Database tables, Files, Images, XMLs, Excel via Storages. DB, CSV, and Excel can be accessed using TestComplete's DDT object | Supported through a third party driver. | Supported | Supported , data table supported using either Excel spreadsheet or XML |
| DB Access | Supp orted, through ADO | Supported | Supported via DDT or COM | Supported through a third party driver. | Supported | Supported through a third party driver. |
| Issues and Concerns | - Problems with Terminal Application Execution. | Speed is not optimized and demanding high hardware requirements. | AutomatedQA support via newsgroup is old and not standard. They really need to create a different web-based support system. Newsgroup does not work. | Very basic recorder that requires a lot of editing which is time consuming. | components are difficult to set-up. | - Hardly to deal with JavaScript’s error |
| Crashes occurs sporadically. | Recommend to use on IE: IE developer toolbar, Web metrics RIA script recorder for analyse frames, and index etc.. | - Recording may take too much time | ||||
| - Very expensive to version control the scripts. | Sometimes, automation execution will be working great one on machine, but not the other. Same when playing back on different operating systems. | Online documents are not overly helpful as they assume knowledge and omit key details. Some files required for the set-up of RC and Grid are not part of the Selenium tools (e.g. ANT, JDK), this also means required set-up files contain errors in context of Selenium. | - Objects implemented using JavaScript are not recognized by the tool. | |||
| - Keyword driven not supported. | ||||||
| TestComplete has a slew of hard_to_reproduce issues that make it difficult for them to fix. | Selenium does not natively support features required by testers such as iteration or data driven testing and there is difficulty testing Flash that has to be overcome by changing Actionscripts. | - Automatic Exception Handling | ||||
| - Interactive test debugging not supported | ||||||
| If an application crashes and Dr. Watson window appears, there is no uniform way to closing the Dr. Watson window to continue the automation execution. Currently, you have to create a third-party script/tool to detect Dr. Watson windows and excuse them. | To extend Selenium functionality it is common to combine a Selenium based automation framework with a language such as Ruby. This provides ways to implement iteration, etc. mentioned above and access Gems (in the case of Ruby). | |||||
| TestComplete 7 introduces performance enhancements to the Object Browser (map of the application objects), but we find the enhancement breaks parts of our automation framework. | ||||||
| Your mileage may vary. | ||||||
| Price | The price could vary according to license. | Priced without Tax USD: | Named User Licenses: | FREE | FREE, Open Source | FREE |
| BM Rational Functional Tester Authorized User License and SW Subscription & Support 12 Months (D53NFLL) 5,821.00 | Enterprise with one year maintenance support $1999 | |||||
| Standard with one year maintenance support $999 | ||||||
| IBM Rational Functional Tester Floating User License + SW Subscription & Support 12 Months (D530BLL) 11,235.00 | ||||||
| Floating User Licenses: | ||||||
| IBM Rational Functional Tester Authorized User Initial Fixed Term License + SW Subscription & Support 12 Months (D54SHLL) 3,146.00 | Enterprise with one year maintenance support $4499 | |||||
| Standard with one year maintenance support $2999 | ||||||
| IBM Rational Functional Tester Extension Floating User License + SW Subscription & Support 12 Months (D59RMLL) 2,846.00 | ||||||
| URL | http://www.automatedqa.com/products/testcomplete/ | http://watir.com/ | http://seleniumhq.org |
Software Test Estimation - 9 General Tips on How to Estimate Testing Time Accurately
Software Test Estimation - 9 General Tips on How to Estimate Testing Time Accurately
Posted In | Testing Life cycle, Test strategy
For success of any project test estimation and proper execution is equally important as the development cycle. Sticking to the estimation is very important to build good reputation with the client.
Experience play major role in estimating “software testing efforts”. Working on varied projects helps to prepare an accurate estimation for the testing cycle. Obviously one cannot just blindly put some number of days for any testing task. Test estimation should be realistic and accurate.
In this article I am trying to put some points in a very simple manner, which are helpful to prepare good test estimations. I am not going to discuss the standard methods for test estimations like testing metrics, instead I am putting some tips on - How to estimate testing efforts for any testing task, which I learned from my experience.
Factors Affecting Software Test Estimation, and General Tips to Estimate Accurately:
1) Think of Some Buffer Time
The estimation should include some buffer. But do not add a buffer, which is not realistic. Having a buffer in the estimation enables to cope for any delays that may occur. Having a buffer also helps to ensure maximum test coverage.
2) Consider the Bug Cycle
The test estimation also includes the bug cycle. The actual test cycle may take more days than estimated. To avoid this, we should consider the fact that test cycle depends on the stability of the build. If the build is not stable, then developers may need more time to fix and obviously the testing cycle gets extended automatically.
3) Availability of All the Resources for Estimated Period
The test estimation should consider all the leaves planned by the team members (typically long leaves) in the next few weeks or next few months. This will ensure that the estimations are realistic. The estimation should consider some fixed number of resources for test cycle. If the number of resources reduces then the estimation should be re-visited and updated accordingly.
4) Can We Do Parallel Testing?
Do you have some previous versions of same product so that you can compare the output? If yes, then this can make your testing task bit easier. You should think the estimation based on your product version.
5) Estimations Can Go Wrong - So re-visit the estimations frequently in initial stages before you commit it.
In early stages, we should frequently re-visit the test estimations and make modification if needed. We should not extend the estimation once we freeze it, unless there are major changes in requirement.
6) Think of Your Past Experience to Make Judgments!
Experiences from past projects play a vital role while preparing the time estimates. We can try to avoid all the difficulties or issues that were faced in past projects. We can analyze how the previous estimates were and how much they helped to deliver product on time.
7) Consider the Scope of Project
Know what is the end objective of the project and list of all final deliverables. Factors to be considered for small and large projects differ a lot. Large project, typically include setting up test bed, generating test data, test scripts etc. Hence the estimations should be based on all these factors. Whereas in small projects, typically the test cycle include test cases writing, execution and regression.
8 ) Are You Going to Perform Load Testing?
If you need to put considerable time on performance testing then estimate accordingly. Estimations for projects, which involve load testing, should be considered differently.
9) Do You Know Your Team?
If you know strengths and weaknesses of individuals working in your team then you can estimate testing tasks more precisely. While estimating one should consider the fact that all resources may not yield same productivity level. Some people can execute faster compared to others. Though this is not a major factor but it adds up to the total delay in deliverables.
And finally tip number 10.
Over To You!
This test estimation tip is purposefully left blank so that you can comment your best estimation techniques in below comment section.
Posted In | Testing Life cycle, Test strategy
For success of any project test estimation and proper execution is equally important as the development cycle. Sticking to the estimation is very important to build good reputation with the client.
Experience play major role in estimating “software testing efforts”. Working on varied projects helps to prepare an accurate estimation for the testing cycle. Obviously one cannot just blindly put some number of days for any testing task. Test estimation should be realistic and accurate.
In this article I am trying to put some points in a very simple manner, which are helpful to prepare good test estimations. I am not going to discuss the standard methods for test estimations like testing metrics, instead I am putting some tips on - How to estimate testing efforts for any testing task, which I learned from my experience.
Factors Affecting Software Test Estimation, and General Tips to Estimate Accurately:
1) Think of Some Buffer Time
The estimation should include some buffer. But do not add a buffer, which is not realistic. Having a buffer in the estimation enables to cope for any delays that may occur. Having a buffer also helps to ensure maximum test coverage.
2) Consider the Bug Cycle
The test estimation also includes the bug cycle. The actual test cycle may take more days than estimated. To avoid this, we should consider the fact that test cycle depends on the stability of the build. If the build is not stable, then developers may need more time to fix and obviously the testing cycle gets extended automatically.
3) Availability of All the Resources for Estimated Period
The test estimation should consider all the leaves planned by the team members (typically long leaves) in the next few weeks or next few months. This will ensure that the estimations are realistic. The estimation should consider some fixed number of resources for test cycle. If the number of resources reduces then the estimation should be re-visited and updated accordingly.
4) Can We Do Parallel Testing?
Do you have some previous versions of same product so that you can compare the output? If yes, then this can make your testing task bit easier. You should think the estimation based on your product version.
5) Estimations Can Go Wrong - So re-visit the estimations frequently in initial stages before you commit it.
In early stages, we should frequently re-visit the test estimations and make modification if needed. We should not extend the estimation once we freeze it, unless there are major changes in requirement.
6) Think of Your Past Experience to Make Judgments!
Experiences from past projects play a vital role while preparing the time estimates. We can try to avoid all the difficulties or issues that were faced in past projects. We can analyze how the previous estimates were and how much they helped to deliver product on time.
7) Consider the Scope of Project
Know what is the end objective of the project and list of all final deliverables. Factors to be considered for small and large projects differ a lot. Large project, typically include setting up test bed, generating test data, test scripts etc. Hence the estimations should be based on all these factors. Whereas in small projects, typically the test cycle include test cases writing, execution and regression.
8 ) Are You Going to Perform Load Testing?
If you need to put considerable time on performance testing then estimate accordingly. Estimations for projects, which involve load testing, should be considered differently.
9) Do You Know Your Team?
If you know strengths and weaknesses of individuals working in your team then you can estimate testing tasks more precisely. While estimating one should consider the fact that all resources may not yield same productivity level. Some people can execute faster compared to others. Though this is not a major factor but it adds up to the total delay in deliverables.
And finally tip number 10.
Over To You!
This test estimation tip is purposefully left blank so that you can comment your best estimation techniques in below comment section.
Subscribe to:
Posts (Atom)