Risk-based Testing

Risk is used to focus the effort required during testing. It is used to decide where and when to start testing and to identify areas that need more attention. Testing is used to reduce the probability of an adverse event occurring, or to reduce the impact of an adverse event. Testing is used as a risk mitigation activity, to provide information about identified risks, as well as providing information on residual (unresolved) risks.

A risk-based approach to testing provides proactive opportunities to reduce the levels of product risk. It

involves product risk analysis, which includes the identification of product risks and the assessment of each risk’s likelihood and impact. The resulting product risk information is used to guide test planning, the specification, preparation and execution of test cases, and test monitoring and control. Analyzing product risks early contributes to the success of a project.

In a risk-based approach, the results of product risk analysis are used to:

• Determine the test techniques to be employed
• Determine the particular levels and types of testing to be performed (e.g., security testing, accessibility testing)
• Determine the extent of testing to be carried out
• Prioritize testing in an attempt to find the critical defects as early as possible
• Determine whether any activities in addition to testing could be employed to reduce risk (e.g., providing training to inexperienced designers)

Risk-based testing draws on the collective knowledge and insight of the project stakeholders to carry out product risk analysis. To ensure that the likelihood of a product failure is minimized, risk management activities provide a disciplined approach to:

• Analyze (and re-evaluate on a regular basis) what can go wrong (risks)
• Determine which risks are important to deal with
• Implement actions to mitigate those risks
• Make contingency plans to deal with the risks should they become actual events

In addition, testing may identify new risks, help to determine what risks should be mitigated, and lower uncertainty about risks.

Read More

Designing Acceptance Tests

Test Techniques for Acceptance Testing 

In a requirements-based approach to acceptance testing, the tester derives test cases from the acceptance criteria related to each requirement or user story using black-box techniques such as equivalence partitioning or boundary value analysis

Acceptance testing may be augmented with other test techniques or approaches:

• Business process-based testing, possibly combined with decision table testing, validates business processes and rules
• Experience-based testing leverages the tester’s experience, knowledge and intuition  
  
• Risk-based testing is based on risk types and levels. Prioritization and thoroughness of testing depends on previously identified product risks.
• Model-based testing uses graphical (or textual) models to obtain acceptance tests
  

Acceptance criteria should be verified by acceptance tests and traceability between the requirements / user story and related test cases should be managed.

Exploratory testing is an experience-based test technique that is not based on detailed predefined test procedures. The testers are domain experts. They are familiar with user needs, requirements and business processes, but they are not necessarily familiar with the product under test.

During an exploratory testing session, the tester accomplishes the following:

• Learns how to work with the product
• Designs the tests
• Performs the tests
• Interprets the results

It is a good practice in exploratory testing to use a test charter. The test charter is prepared prior to the testing session (possibly jointly by the business analyst and the tester) and is used by the person in charge of the exploratory session (either a business analyst, tester or another stakeholder). It includes information about the purpose, target, and scope of the exploratory session, the test setup, the duration of the session, and possibly some tactics to be used during the session (such as the type of user that shall be simulated during the exploratory session). Time-boxed sessions help to control the time and effort dedicated to the exploratory session. It is also good practice to perform exploratory testing in pairs or as team work.

In Agile development, exploratory test sessions can be conducted during an iteration by the product owner and/or the testers for acceptance testing of user stories assigned to the iteration.

Exploratory testing should be used to complement other more formal techniques in acceptance testing. For example, it may be used to provide rapid feedback on new features before methodical testing is applied.

Using the Gherkin Language to Write Test Cases

In ATDD and BDD, acceptance tests are often formulated in a structured language, referred to as the Gherkin language. Using the Gherkin language, test cases are phrased declaratively using a standardized pattern:

• Given [a situation]
• When [an action on the system]
• Then [the expected result]

The pattern allows business analysts, testers and developers to write test cases in a way that is easily shared with stakeholders and may be translated into automated tests.

The “Given” block aims to put the test object in a state before performing test actions in the “When” block. The "Then" block specifies the consequences that can be observed from the actions defined in the "When" block. Test cases written in Gherkin do not refer to user interface elements but rather to user actions on the system. They are structured natural language test cases that can be understood by all relevant stakeholders.

In addition, the structure “Given – When – Then” can be parsed in an automated way.

This allows automated test script creation using a keyword-driven testing approach.

Initially, Gherkin was specific to some software tools supporting BDD, but it is now synonymous with the “Given – When – Then” acceptance test design pattern.

Read More