Designing Acceptance Tests

Test Techniques for Acceptance Testing 

In a requirements-based approach to acceptance testing, the tester derives test cases from the acceptance criteria related to each requirement or user story using black-box techniques such as equivalence partitioning or boundary value analysis

Acceptance testing may be augmented with other test techniques or approaches:

• Business process-based testing, possibly combined with decision table testing, validates business processes and rules
• Experience-based testing leverages the tester’s experience, knowledge and intuition  
  
• Risk-based testing is based on risk types and levels. Prioritization and thoroughness of testing depends on previously identified product risks.
• Model-based testing uses graphical (or textual) models to obtain acceptance tests
  

Acceptance criteria should be verified by acceptance tests and traceability between the requirements / user story and related test cases should be managed.

Exploratory testing is an experience-based test technique that is not based on detailed predefined test procedures. The testers are domain experts. They are familiar with user needs, requirements and business processes, but they are not necessarily familiar with the product under test.

During an exploratory testing session, the tester accomplishes the following:

• Learns how to work with the product
• Designs the tests
• Performs the tests
• Interprets the results

It is a good practice in exploratory testing to use a test charter. The test charter is prepared prior to the testing session (possibly jointly by the business analyst and the tester) and is used by the person in charge of the exploratory session (either a business analyst, tester or another stakeholder). It includes information about the purpose, target, and scope of the exploratory session, the test setup, the duration of the session, and possibly some tactics to be used during the session (such as the type of user that shall be simulated during the exploratory session). Time-boxed sessions help to control the time and effort dedicated to the exploratory session. It is also good practice to perform exploratory testing in pairs or as team work.

In Agile development, exploratory test sessions can be conducted during an iteration by the product owner and/or the testers for acceptance testing of user stories assigned to the iteration.

Exploratory testing should be used to complement other more formal techniques in acceptance testing. For example, it may be used to provide rapid feedback on new features before methodical testing is applied.

Using the Gherkin Language to Write Test Cases

In ATDD and BDD, acceptance tests are often formulated in a structured language, referred to as the Gherkin language. Using the Gherkin language, test cases are phrased declaratively using a standardized pattern:

• Given [a situation]
• When [an action on the system]
• Then [the expected result]

The pattern allows business analysts, testers and developers to write test cases in a way that is easily shared with stakeholders and may be translated into automated tests.

The “Given” block aims to put the test object in a state before performing test actions in the “When” block. The "Then" block specifies the consequences that can be observed from the actions defined in the "When" block. Test cases written in Gherkin do not refer to user interface elements but rather to user actions on the system. They are structured natural language test cases that can be understood by all relevant stakeholders.

In addition, the structure “Given – When – Then” can be parsed in an automated way.

This allows automated test script creation using a keyword-driven testing approach.

Initially, Gherkin was specific to some software tools supporting BDD, but it is now synonymous with the “Given – When – Then” acceptance test design pattern.

Read More

Business Analysis and Acceptance Testing

Requirements / User Stories, Acceptance Criteria and Acceptance Tests 

During requirements elicitation, business analysts and testers (possibly together with developers) should begin to create specific acceptance criteria and develop acceptance tests as a joint effort. This ensures that there is a mutual understanding of what “acceptable” means from the business, development, and testing perspectives, right from the beginning of the project.

Acceptance criteria relate directly to a specific requirement or user story. They are either part of the detailed description or an attribute of the related requirement. If user stories are used, acceptance criteria are part of the user story’s definition and extend the story.

In all cases, acceptance criteria are measurable criteria, formulated as a statement (or a set of statements), which can be either true or false. They are used to check whether a requirement or user story has been implemented as expected. Acceptance criteria represent the test conditions which determine “what” to test. They do not contain the detailed test procedures.

In Agile development, the INVEST criteria define a set of criteria, or checklist, to assess the quality of a product backlog item. Product Backlog Item (PBI) may be used in a Scrum backlog, Kanban board or XP project, commonly written in user story format, but not required to be. 

Letter	Meaning	Description
I	Independent	The PBI should be self-contained, in a way that there is no inherent dependency on another PBI.
N	Negotiable	PBIs are not explicit contracts and should leave space for discussion.
V	Valuable	A PBI must deliver value to the stakeholders.
E	Estimable	You must always be able to estimate the size of a PBI.
S	Small	PBIs should not be so big as to become impossible to plan/task/prioritize within a level of accuracy.
T	Testable	The PBI or its related description must provide the necessary information to make test development possible.

The following good practices should be considered when writing acceptance criteria  

• Well-written acceptance criteria are precise, measurable and concise. Each criterion must be written in a way that enables the tester to measure whether or not the test object complies with the acceptance criterion.
• Well-written acceptance criteria do not include technical solution details. They concentrate on the question "What shall be achieved?" rather than on the question "How shall it achieved?".
• Acceptance criteria should address non-functional requirements (quality characteristics) as well as functional requirements.

Acceptance test cases are derived from acceptance criteria. These tests specify how the verification of the acceptance criteria should be performed.

Read More

How Acceptance Testing Can Drive the Development Process: ATDD and BDD

The wide acceptance of Agile software development practices has influenced how acceptance testing relates to requirements elicitation and other business analysis activities. In sequential lifecycle models, acceptance test analysis, design, and implementation are activities to be handled by the testers after the requirements are finalized. With the Agile lifecycle model, acceptance criteria and acceptance test cases

are created during requirements analysis, requirements refinement sessions, and product backlog refinement. This allows the implementation of the "Early Testing" principle by using the design of test cases as part of the requirements definition activities.

In the following two approaches, acceptance test analysis and design are formally part of the requirements engineering process:

• In Acceptance Test-Driven Development (ATDD), acceptance tests are produced collaboratively during requirements analysis by business analysts, product owners, testers and developers.
• Behavior-Driven Development (BDD) uses a domain-specific scripting language, Gherkin, that is based on natural language statements. The requirements are defined in a ‘Given – When –Then’ format. These requirements become the acceptance test cases and also serve as the basis for test automation.
  

Both of these approaches engage the entire Agile team and help to focus the development efforts on the business goals. The approaches also treat the acceptance test cases as living documentation of the product because they can be read and understood by business analysts and other stakeholders. Acceptance test cases represent scenarios of usage of the product.

The two approaches are similar and the two terms are sometimes used interchangeably.

In practice, BDD is associated with the use of Gherkin to support writing acceptance tests, while ATDD relies on different forms of textual or graphic acceptance test design. For example, the graphical representation of application workflows may be used to implement a visual ATDD approach.

Read More

Continuous Integration

A CI system allows the automated compilation, integration, testing and deployment of the entire code base in a project. 

Delivery of a product increment requires reliable, working, integrated software at the end of every
sprint. Continuous integration addresses this challenge by merging all changes made to the software
and integrating all changed components regularly, at least once a day. Configuration management,
compilation, software build, deployment, and testing are wrapped into a single, automated, repeatable
process. 

Following the developers’ coding, debugging, and check-in of code into a shared source code repository, a continuous integration process consists of the following automated activities: 

• Static code analysis: executing static code analysis and reporting results
• Compile: compiling and linking the code, generating the executable files
• Unit test: executing the unit tests, checking code coverage and reporting test results
• Deploy: installing the build into a test environment
• Integration test: executing the integration tests and reporting results
• Report (dashboard): posting the status of all these activities to a publicly visible location or emailing status to the team
  

Continuous integration can provide the following benefits:

• Allows earlier detection and easier root cause analysis of integration problems and conflicting changes
• Gives the development team regular feedback on whether the code is working
• Keeps the version of the software being tested within a day of the version being developed
• Reduces regression risk associated with developer code refactoring due to rapid re-testing of the code base after each small set of changes
• Provides confidence that each day’s development work is based on a solid foundation
• Makes progress toward the completion of the product increment visible, encouraging developers and testers
• Eliminates the schedule risks associated with big-bang integration
• Provides constant availability of executable software throughout the sprint for testing, demonstration, or education purposes
• Reduces repetitive manual testing activities
• Provides quick feedback on decisions made to improve quality and tests

However, continuous integration is not without its risks and challenges:

• Continuous integration tools have to be introduced and maintained
• The continuous integration process must be defined and established
• Test automation requires additional resources and can be complex to establish. Data sets have to be created, people trained to use the tools and time must be spent to create, for example, a keyword test framework. Also, the test results analysis can be time-consuming if there are too many test scripts. 
• Well-specified test cases are required to achieve automated testing advantages. This will increase the time needed for testing tasks.
• Thorough test coverage is essential to achieve automated testing advantages
• Teams sometimes over-rely on unit tests and perform too little system and acceptance testing. The test strategy must plan how upper level tests, such as system or end-to-end tests, will be executed

Continuous integration requires the use of tools, including tools for testing, tools for automating the
build process, and tools for version control.

Read More

Traditional vs Agile Testing

Traditional Testing

With Waterfall model, testing tends to happen towards the end of the project life cycle so defects are detected close to the live implementation date. 

The V-model was developed to address some of the problems experienced using the traditional waterfall approach. The V-model provides guidance that testing needs to begin as early as possible in the life cycle, which is one of the fundamental principles of structured testing. There are a variety of activities that need to be performed before the end of the coding phase. These activities should be carried out in parallel with development activities, and testers need to work with developers and business analysts so they can perform these activities and tasks and produce a set of test deliverables. By starting test design early, defects are often found in the test basis documents.

Agile Testing

• It promotes the generation of business stories to define the functionality.
• It demands an on-site customer for continual feedback and to define and carry out functional acceptance testing.
• It promotes pair programming and shared code ownership amongst the developers.
• It states that component test scripts shall be written before the code is written and that those tests should be automated.
• It states that integration and testing of the code shall happen several times a day.
• It states that we always implement the simplest solution to meet today's problems

Agile testing with its focus on business value and delivering the quality customers require, is different from traditional testing which focuses on conformance to requirements. 

Read More

Software Development Lifecycle

A software development lifecycle model describes the types of activity performed at each stage in a software development project, and how the activities relate to one another logically and chronologically.
 

There are numerous development life cycles that have been developed in order to achieve different required objectives. Each of these methodologies has its place in modern software development and the most appropriate development process should be applied to each project. The models specify the various stages of the process and the order in which they are carried out.

 

Sequential development models

A sequential development model describes the software development process as a linear, sequential flow of activities. This means that any phase in the development process should begin when the previous phase is complete. In theory, there is no overlap of phases, but in practice, it is beneficial to have early feedback from the following phase.

Sequential development models deliver software that contains the complete set of features, but typically
require months or years for delivery to stakeholders and users. 

Waterfall model

In the Waterfall model, the development activities (e.g., requirements analysis, design, coding, testing) are completed one after another. In this model, test activities only occur after all other development activities have been completed.

With this model it has been difficult to get feedback passed backwards up the waterfall and there are difficulties if we need to carry out numerous iterations for a particular phase.  

Defects were being found too late in the life cycle, as testing was not involved until the end of the project. Testing also added lead time due to its late involvement. 

V-model

The V-model is a model that illustrates how testing activities (verification and validation) can be integrated into each phase of the life cycle. Within the V-model, validation testing takes place especially during the early stages, e.g. reviewing the user requirements, and late in the life cycle, e.g. during user acceptance testing.

A common type of V-model uses four test levels. The four test levels used, each with their own objectives, are component testing, integration testing, system testing, acceptance testing.

 

Iterative and incremental development models

There are iterative or incremental life cycles where, instead of one large development time line from beginning to end, we cycle through a number of smaller self-contained life cycle phases for the same project.

A common feature of iterative approaches is that the delivery is divided into increments or builds with each increment adding new functionality. The initial increment will contain the infrastructure required to support the initial build functionality. 

The increment produced by an iteration may be tested at several levels as part of its development. Subsequent increments will need testing for the new functionality, regression testing of the existing functionality, and integration testing of both new and existing parts. Regression testing is increasingly important on all iterations after the first one. This means that more testing will be required at each subsequent delivery phase which must be allowed for in the project plans. 

This life cycle can give early market presence with critical functionality, can be simpler to manage because the workload is divided into smaller pieces, and can reduce initial investment although it may cost more in the long run. Also early market presence will mean validation testing is carried out at each increment, thereby giving early feedback on the business value and fitness for-use of the product.

Examples of iterative or incremental development models are prototyping, Rapid Application Development (RAD), Rational Unified Process (RUP) and agile development.

Read More