IEEE 829 standard for software testing documentation
One of the challenges facing software testers has been the availability of an agreed set of document standards and templates for testing. The IEEE 829 provides an internationally recognised set of standards for test planning documentation.
IEEE 829 has been developed specifically with software testing in mind and is applicable to each stage of the testing life cycle including system and acceptance testing.
Types of Document
The IEEE 829 standard covers 8 document types.
Test specification
• Test Plan: Covers how the testing will be managed, scheduled and executed.
• Test Design Specification: defines logically what needs to be tested by examining the requirements or features. Then these requirements can be converted into test conditions.
• Test Case Specification: Converts the test conditions into test cases by adding real data, pre-conditions and expected results
• Test Procedure: Describes in practical terms how the tests are run.
• Test Item Transmittal Report: Specify the items released for testing.
Test execution
• Test Log: Is an audit trail that records the details of tests in chronologically.
• Test Incident Report: Record details of any unexpected events and behaviours that need to be investigated.
Test reporting
• Test Summary Report: Summarise and evaluate tests.
Documentation for test specification
The test preparation is by far the most important part of any software testing project. During this stage you must create your tests and define the requirements of your test environment.
IEEE 829 - Test Plan
The Test Plan describes how you will deliver the testing.
1. Test Plan Identifier
2. References
3. Introduction
4. Test Items
5. Software Risk Issues
6. Features to be tested
7. Features not to be tested
8. Approach
9. Item Pass/Fail Criteria
10. Suspension Criteria and Resumption Requirements
11. Test Deliverables
12. Remaining Test Tasks
13. Environmental Needs
14. Staffing and Training Needs
15. Responsibilities
16. Schedule
17. Planning Risks and Contingencies
18. Approvals
19. Glossary
IEEE 829 - Test Design Specification
The test design specification identifies the test conditions from the requirements and functional design documentation.
Let’s use a Banking project example where the following testing requirements have been defined. In this case Bank A is rolling out new “hole in the wall” machines all over the country and the project will include testing the functionality of the ATMs to demonstrate the ability to:
• Complete a valid withdrawal of funds.
• Print a report summary of recent transactions.
• Change passwords.
The test design does not record the values to be entered for a test, but describes the requirements for defining those values. This is done at a logical level and should not be cluttered with individual examples and data.The design specification provides a link between test requirements and test cases.
IEEE 829 - Test Case Specification
The test cases can be produced when the test design is completed. A test case may cover one or more test conditions derived from the test design. A test case should include:
• The precise data that is required to execute the test. This is a combination of input data, and application and system data that is needed.
• The expected results and outputs
• Pre-conditions that will determine the starting point for each test. A feature (requirement) from the test design may be tested in more than one test case, and a test case may test more than one feature. The aim is for a set of test cases to test each feature (requirement) in the test design at least once. Taking the ATM project example the first 2 requirements could be tested using one test case:
• Test case 1 could be defined to include the completion of a cash withdrawal from the ATM and then a printout request to show this withdrawal has been correctly executed and the right amount has been debited.
IEEE 829 - Test Procedure Specification The test procedures are developed from both the test design and the test case specification. The procedure document describes how the tester will physically run the test, the physical set-up required, and the procedure steps that need to be followed. The standard defines ten procedure steps that may be applied when running a test.
IEEE 829 - Test Item Transmittal Report
This document is a handover document and provides details of the previous stage of testing.
Similar to a release note this provides a list of what is being delivered and shows any changes and new items contained. It includes the person responsible for each item, its physical location and its status.
Documentation for Test execution
The schedule of what test cases are run and when, is defined in the Test Plan. The test results are recorded in the test log, and in test incident reports.
IEEE 829 - Test Log
The Test Log records the details of what Test Cases have been run, the order of their running, and the results of the test. The results are either the test passed, meaning that the actual and expected results were identical, or it failed and that there was a discrepancy. If there is a discrepancy than one or more Test Incident Reports are raised or updated, and their identities recorded on the Test Log.
The Test Log is important as it allows progress of the testing to be checked, as well as providing valuable information for finding out what caused an incident. If an incident is a coding fault, the fault may have occurred not in the Test Case that failed but in one that was run previously. Thus the sequence of the tests enables the fault to be found.
IEEE 829 -Test Incident Report
This report documents any event that requires subsequent investigation. An incident should be raised when there is an unexpected result or any unexpected behaviour during testing. At this point it may not always be clear whether there is a bug or fault in the software, since incidents can occur as a result of configuration errors, faults in the software, faults in the requirements and in-correct expected results recorded in the test case.
The report consists of all details of the incident such as actual and expected results, when it failed, and any supporting evidence that will help in its resolution. The report will also include, if possible, an assessment of the impact upon testing of an incident.
Documentation for test reporting
Eventually testing will be completed according to the criteria specified in the Test Plan. This is when the success or failure of the system is decided based on the results. The Test Summary records this information.
IEEE 829 - test summary
The summary provides the results of the designated testing activities and an evaluation of these results.
Moreover the summary provides an overall view of the testing and the quality of the software.
Use of the standard
The 829 standard is a good start point to create a sensible structure for your testing documents. However, it should be customised or changed to reflect the needs of your project.
