Test process

 

The fundamental test process consists of the following main activities:

Test planning

Test planning is the activity of defining the objectives of testing & the specification of test activities in order to meet the objectives and mission

Test planning has the following major tasks, given approximately in order, which help us build a test plan:

• Determine the scope and risks and identify the objectives of testing
• Determine the test approach  
• Implement the test policy and/or the test strategy
• Determine the required test resources
• Schedule test analysis and design tasks, test implementation, execution and evaluation
• Determine the exit criteria
  

Test monitoring and control

Test monitoring involves the on-going comparison of actual progress against planned progress using any test monitoring metrics defined in the test plan. 

Test control involves taking actions necessary to meet the objectives of the test plan (which may be updated over time). 

Test monitoring and control are supported by the evaluation of exit criteria, which are referred to as the definition of done in some software development life-cycle models. 

Test monitoring and control has the following major tasks:

• Measure and analyze the results of reviews and testing
• Monitor and document progress, test coverage and exit criteria
• Provide information on testing
• Initiate corrective actions 
• Make decisions
  

Test analysis

During test analysis, the test basis is analyzed to identify testable features and define associated test
conditions.  

Test analysis includes the following major activities:

• Analyzing the test basis appropriate to the test level being considered, for example:

o Requirement specifications, such as business requirements, user stories, use cases
o Design and implementation information, such as system or software architecture diagrams or documents, design specifications, call flow graphs, modelling diagrams (e.g., UML or entity-relationship diagrams), interface specifications
o The implementation of the component or system itself, including code, database metadata and queries, and interfaces
o Risk analysis reports, which may consider functional, non-functional, and structural aspects of the component or system

• Evaluating the test basis and test items to identify defects of various types, such as ambiguities, omissions, inconsistencies, inaccuracies, contradictions, superfluous statements  
• Identifying features and sets of features to be tested 
• Defining and prioritizing test conditions for each feature based on analysis of the test basis, and
  considering functional, non-functional, and structural characteristics, other business and technical
  factors, and levels of risks 
• Capturing bi-directional traceability between each element of the test basis and the associated
  test conditions
  

The application of black-box, white-box, and experience-based test techniques can be useful in the
process of test analysis to reduce the likelihood of omitting important test conditions and to define more precise and accurate test conditions.

In some cases, test analysis produces test conditions which are to be used as test objectives in test
charters. When these test objectives are traceable to the test basis, coverage achieved during such experience-based testing can be measured.

The identification of defects during test analysis is an important potential benefit, especially where no
other review process is being used and/or the test process is closely connected with the review process.
Such test analysis activities not only verify whether the requirements are consistent, properly expressed,
and complete, but also validate whether the requirements properly capture customer, user, and other
stakeholder needs.  

Test design 

During test design, the test conditions are elaborated into high-level test cases, sets of high-level test
cases, and other testware. So, test analysis answers the question “what to test?” while test design answers the question “how to test?”

Test design includes the following major activities:

• Designing and prioritizing test cases and sets of test cases
• Identifying necessary test data to support test conditions and test cases
• Designing the test environment and identifying any required infrastructure and tools
• Capturing bi-directional traceability between the test basis, test conditions, and test cases

Test implementation

During test implementation, the testware necessary for test execution is created and/or completed,
including sequencing the test cases into test procedures. So, test design answers the question “how to
test?” while test implementation answers the question “do we now have everything in place to run the
tests?”

Test implementation includes the following major activities:

• Developing and prioritizing test procedures, and, potentially, creating automated test scripts
• Creating test suites from the test procedures and (if any) automated test scripts
• Arranging the test suites within a test execution schedule in a way that results in efficient test execution  
• Building the test environment and verifying that everything needed has been set up correctly
• Preparing test data and ensuring it is properly loaded in the test environment
• Verifying and updating bi-directional traceability between the test basis, test conditions, test cases, test procedures, and test suites

Test execution

Test execution includes the following major activities:

• Recording the IDs and versions of the test item(s) or test object, test tool(s), and testware
• Executing tests either manually or by using test execution tools
• Comparing actual results with expected results
• Analyzing anomalies to establish their likely causes (e.g., failures may occur due to defects in the code, but false positives also may occur)
• Reporting defects based on the failures observed
• Logging the outcome of test execution (e.g., pass, fail, blocked) 
• Repeating test activities either as a result of action taken for an anomaly, or as part of the planned testing (e.g., execution of a corrected test, confirmation testing, and/or regression testing) 
• Verifying and updating bi-directional traceability between the test basis, test conditions, test cases, test procedures, and test results.
  

Test completion 

Test completion activities collect data from completed test activities to consolidate experience, testware, and any other relevant information. Test completion activities occur at project milestones such as when a software system is released, a test project is completed (or cancelled), an Agile project iteration is finished, a test level is completed, or a maintenance release has been completed.

Test completion includes the following major activities:

• Checking whether all defect reports are closed, entering change requests or product backlog items for any defects that remain unresolved at the end of test execution
• Creating a test summary report to be communicated to stakeholders
• Finalizing and archiving the test environment, the test data, the test infrastructure, and other testware for later reuse
• Handing over the testware to the maintenance teams, other project teams, and/or other stakeholders who could benefit from its use
• Analyzing lessons learned from the completed test activities to determine changes needed for future iterations, releases, and projects
• Using the information gathered to improve test process maturity