Acceptance Testing for Non-Functional Requirements

Non-functional Quality Characteristics 

Non-functional Characteristic	Sub-characteristics
Performance efficiency	Time-behavior  Resource utilization  Capacity
Compatibility	Co-existence  Interoperability
Usability	Appropriateness recognizability  Learnability  Operability  User error protection  User interface aesthetics  Accessibility
Reliability	Maturity Availability Fault tolerance Recoverability
Security	Confidentiality Integrity Non-repudiation Accountability Authenticity
Maintainability	Modularity Reusability Analyzability Modifiability Testability
Portability	Adaptability Installability Replaceability

Usability and User Experience

User eXperience (UX) expands the term usability to include aesthetic and emotional factors such as an appealing, desirable design, aspects of confidence building, or satisfaction to use (e.g., pleasure, comfort). The context of using the system has a strong influence on the user experience as it may totally differ based on a number of factors such as location (e.g., the user is sitting behind a desk, driving a car or hiking), weather (e.g., sun, rain, cold), health conditions of the user (e.g., fatigue, age), environment (e.g., stressful, noisy).

UX requirements analysis is based upon the following four pillars:

• User analysis: Users are categorized in terms such as physical and intellectual characteristics, technical skills, business knowledge, socio-economic, and cultural background. Business analysts can also use models.
• Task analysis: Functionality is identified and formalized (e.g., through use cases and scenarios). User behavior and expectations are analyzed to design an optimized system or product.
• Context analysis: The context in which the system or product will be used is analyzed. External conditions (e.g. light, temperature, movement, humidity or dust), physical conditions (e.g., sitting, standing, lying, moving, hands-free) or “psychological” conditions (e.g. stress level, motivation, or the difference between private and professional usage) are considered to give directions to the subsequent design steps. Devices, platforms and form-factors (device-specific display) are also considered as part of the context.
• Competition analysis: Unless creating a disruptive design is the goal, business analysts should analyze the competitors and take inspiration from the successful implementation of their solutions to retain or attract users and customers. Another source of inspiration can come from successful solutions found in similar or even different sectors.

Due to common human limitations and biases (e.g., cognitive or perceptive biases, visual impairment, inexperience) some users might face more specific and sometimes severe difficulties in using software or products that are part of the business solution. Business analysts and testers should assess if products or services are accessible to all users by considering these limitations when designing acceptance criteria and test cases.

Usability Testing

There are different approaches to testing usability in acceptance testing:

• Checklist-based evaluations: Users evaluate the system or product under test according to checklists to evaluate, compare and qualify their experience.
• Expert reviews: Usability experts evaluate the usability of the system or product according to pre-defined criteria or checklists based upon usability heuristics to identify strong and weak points of an interface.
• Walkthrough and think-aloud techniques: Users explore the product or systems and describe their actions and impressions out loud while doing so. They may be given specific tasks to accomplish to identify how they interact with the product and to learn about expectations or difficulties.
• Biometrics-based evaluations: User behavior is monitored with specific biometric devices (e.g., eye-movement recording, mouse-eye-movement recording) to understand how the user interacts with a page or a system, what attracts their attention, or what is more or less visible.
• Log files analysis: Retrospective analysis is conducted to review how the users interacted with the system to discover areas for possible improvement and to verify if actual use correlates with the intended profile/use.

Performance Efficiency

Performance efficiency (or simply “performance”) is an essential part of providing a “good experience” for users when they use their applications on a variety of fixed and mobile platforms. Performance tests must be considered at all levels of testing.

During acceptance testing, performance tests are particularly addressed during Operational Acceptance Testing (OAT), usually by the operating teams. However, business analysts and testers should also be involved when developing acceptance criteria and related test cases. Acceptance criteria for performance efficiency requirements should provide objective measures, thus avoiding subjective performance evaluation during acceptance test execution.

High-level Performance Acceptance Tests

Performance testing aims to determine a system’s responsiveness and stability under certain conditions. In a typical performance test, concurrent users or transactions are simulated with specific tools to generate a given workload which mimics, as closely as possible, actual conditions with real users and realistic interactions. The response times of key elements of the system under test (e.g., web server, application server, database) are then measured by a tool and compared to pre-defined performance requirements.

This can be also done for the use of memory, system input/output, CPU busy times, and access to security devices, depending on what component is (expected to be) the bottle neck or is targeted.

Based upon the analysis of results, specific elements in the architecture (hardware and software) may be modified (such as providing additional server capacity). The cycle of testing, analysis, and improvement may be repeated until the performance target is reached.

Different types of testing can be performed, depending on what needs to be measured. These include load, stress, and endurance / stability tests. Workload can be simulated by using different models: steady state, increasing, scenario-based or artificial.

Acceptance Criteria for Performance Acceptance Tests

Performance acceptance criteria can be expressed from different perspectives as shown in the following:

• From a user perspective, the perceived response time reflects the user’s real experience with the system. For example, users may abandon a web site if the response time is more than 10 seconds.
• From a business perspective, the number of concurrent users, the types of scenarios or transactions performed, and the expected response times are factors to be considered. Higher numbers of concurrent users performing resourceintensive transactions will result in longer response times. Other factors might also influence the response time based on location, time or time zone.
• From a technical perspective, available system resources (e.g., network bandwidth, CPU usage, RAM capacity) and system architecture, (e.g., server load balancing, use of data caching) are factors which influence performance efficiency. For example, web-based systems with limited network bandwidth will tend to have lower performance efficiency, especially when subjected to high loads caused by large numbers of users conducting tasks that generate significant network traffic.

The development of acceptance criteria and acceptance tests for performance requirements must address these three different perspectives (user, business and technical).

Security

Information security management and general security requirements should be part of an overall security policy for an organization. Business Analysts and testers should use the security policy for recommendations and guidelines, and as a basis for managing security risks on their projects.

Security requirements should be considered at all stages of business analysis, requirements engineering and related acceptance testing including the following:

• Information security should be part of risk management and non-functional requirements elicitation and analysis. The value of information in the system under test or in a given business process should be assessed, followed by an evaluation and prioritization of security risks.
• Measurable acceptance criteria should be defined for information security requirements. They may cover a large variety of aspects such as authentication, authorization and accounting procedures, sanitization of input data, use of cryptography, and data privacy constraints.
• High-level information security test cases should be defined according to the security requirements and the acceptance criteria. These test cases define the context of the test, the main steps and the expected results.
• Some security acceptance tests can be run by the acceptance tester and others by more specialized security testers, depending on the level of technical complexity of the test.

Read More

Designing Acceptance Tests

Test Techniques for Acceptance Testing 

In a requirements-based approach to acceptance testing, the tester derives test cases from the acceptance criteria related to each requirement or user story using black-box techniques such as equivalence partitioning or boundary value analysis

Acceptance testing may be augmented with other test techniques or approaches:

• Business process-based testing, possibly combined with decision table testing, validates business processes and rules
• Experience-based testing leverages the tester’s experience, knowledge and intuition  
  
• Risk-based testing is based on risk types and levels. Prioritization and thoroughness of testing depends on previously identified product risks.
• Model-based testing uses graphical (or textual) models to obtain acceptance tests
  

Acceptance criteria should be verified by acceptance tests and traceability between the requirements / user story and related test cases should be managed.

Exploratory testing is an experience-based test technique that is not based on detailed predefined test procedures. The testers are domain experts. They are familiar with user needs, requirements and business processes, but they are not necessarily familiar with the product under test.

During an exploratory testing session, the tester accomplishes the following:

• Learns how to work with the product
• Designs the tests
• Performs the tests
• Interprets the results

It is a good practice in exploratory testing to use a test charter. The test charter is prepared prior to the testing session (possibly jointly by the business analyst and the tester) and is used by the person in charge of the exploratory session (either a business analyst, tester or another stakeholder). It includes information about the purpose, target, and scope of the exploratory session, the test setup, the duration of the session, and possibly some tactics to be used during the session (such as the type of user that shall be simulated during the exploratory session). Time-boxed sessions help to control the time and effort dedicated to the exploratory session. It is also good practice to perform exploratory testing in pairs or as team work.

In Agile development, exploratory test sessions can be conducted during an iteration by the product owner and/or the testers for acceptance testing of user stories assigned to the iteration.

Exploratory testing should be used to complement other more formal techniques in acceptance testing. For example, it may be used to provide rapid feedback on new features before methodical testing is applied.

Using the Gherkin Language to Write Test Cases

In ATDD and BDD, acceptance tests are often formulated in a structured language, referred to as the Gherkin language. Using the Gherkin language, test cases are phrased declaratively using a standardized pattern:

• Given [a situation]
• When [an action on the system]
• Then [the expected result]

The pattern allows business analysts, testers and developers to write test cases in a way that is easily shared with stakeholders and may be translated into automated tests.

The “Given” block aims to put the test object in a state before performing test actions in the “When” block. The "Then" block specifies the consequences that can be observed from the actions defined in the "When" block. Test cases written in Gherkin do not refer to user interface elements but rather to user actions on the system. They are structured natural language test cases that can be understood by all relevant stakeholders.

In addition, the structure “Given – When – Then” can be parsed in an automated way.

This allows automated test script creation using a keyword-driven testing approach.

Initially, Gherkin was specific to some software tools supporting BDD, but it is now synonymous with the “Given – When – Then” acceptance test design pattern.

Read More

Business Analysis and Acceptance Testing

Requirements / User Stories, Acceptance Criteria and Acceptance Tests 

During requirements elicitation, business analysts and testers (possibly together with developers) should begin to create specific acceptance criteria and develop acceptance tests as a joint effort. This ensures that there is a mutual understanding of what “acceptable” means from the business, development, and testing perspectives, right from the beginning of the project.

Acceptance criteria relate directly to a specific requirement or user story. They are either part of the detailed description or an attribute of the related requirement. If user stories are used, acceptance criteria are part of the user story’s definition and extend the story.

In all cases, acceptance criteria are measurable criteria, formulated as a statement (or a set of statements), which can be either true or false. They are used to check whether a requirement or user story has been implemented as expected. Acceptance criteria represent the test conditions which determine “what” to test. They do not contain the detailed test procedures.

In Agile development, the INVEST criteria define a set of criteria, or checklist, to assess the quality of a product backlog item. Product Backlog Item (PBI) may be used in a Scrum backlog, Kanban board or XP project, commonly written in user story format, but not required to be. 

Letter	Meaning	Description
I	Independent	The PBI should be self-contained, in a way that there is no inherent dependency on another PBI.
N	Negotiable	PBIs are not explicit contracts and should leave space for discussion.
V	Valuable	A PBI must deliver value to the stakeholders.
E	Estimable	You must always be able to estimate the size of a PBI.
S	Small	PBIs should not be so big as to become impossible to plan/task/prioritize within a level of accuracy.
T	Testable	The PBI or its related description must provide the necessary information to make test development possible.

The following good practices should be considered when writing acceptance criteria  

• Well-written acceptance criteria are precise, measurable and concise. Each criterion must be written in a way that enables the tester to measure whether or not the test object complies with the acceptance criterion.
• Well-written acceptance criteria do not include technical solution details. They concentrate on the question "What shall be achieved?" rather than on the question "How shall it achieved?".
• Acceptance criteria should address non-functional requirements (quality characteristics) as well as functional requirements.

Acceptance test cases are derived from acceptance criteria. These tests specify how the verification of the acceptance criteria should be performed.

Read More

How Acceptance Testing Can Drive the Development Process: ATDD and BDD

The wide acceptance of Agile software development practices has influenced how acceptance testing relates to requirements elicitation and other business analysis activities. In sequential lifecycle models, acceptance test analysis, design, and implementation are activities to be handled by the testers after the requirements are finalized. With the Agile lifecycle model, acceptance criteria and acceptance test cases

are created during requirements analysis, requirements refinement sessions, and product backlog refinement. This allows the implementation of the "Early Testing" principle by using the design of test cases as part of the requirements definition activities.

In the following two approaches, acceptance test analysis and design are formally part of the requirements engineering process:

• In Acceptance Test-Driven Development (ATDD), acceptance tests are produced collaboratively during requirements analysis by business analysts, product owners, testers and developers.
• Behavior-Driven Development (BDD) uses a domain-specific scripting language, Gherkin, that is based on natural language statements. The requirements are defined in a ‘Given – When –Then’ format. These requirements become the acceptance test cases and also serve as the basis for test automation.
  

Both of these approaches engage the entire Agile team and help to focus the development efforts on the business goals. The approaches also treat the acceptance test cases as living documentation of the product because they can be read and understood by business analysts and other stakeholders. Acceptance test cases represent scenarios of usage of the product.

The two approaches are similar and the two terms are sometimes used interchangeably.

In practice, BDD is associated with the use of Gherkin to support writing acceptance tests, while ATDD relies on different forms of textual or graphic acceptance test design. For example, the graphical representation of application workflows may be used to implement a visual ATDD approach.

Read More

Test levels

Test levels are groups of test activities that are organized and managed together. Each test level is an instance of the test process, performed in relation to software at a given level of development, from individual units or components to complete systems or, where applicable, systems of systems. Test levels are related to other activities within the software development lifecycle. 

 The test levels are:

• Component testing: searches for defects in and verifies the functioning of software components (e.g. modules, programs, objects, classes etc.) that are separately testable;
• Integration testing: tests interfaces between components, interactions to different parts of a system such as an operating system, file system and hard ware or interfaces between systems;
• System testing: concerned with the behavior of the whole system/product as defined by the scope of a development project or product. The main focus of system testing is verification against specified requirements;
• Acceptance testing: validation testing with respect to user needs, requirements, and business processes conducted to determine whether or not to accept the system.

Test levels are characterized by the following attributes:

• Specific objectives
• Test basis, referenced to derive test cases
• Test object (i.e., what is being tested)
• Typical defects and failures
• Specific approaches and responsibilities

For every test level, a suitable test environment is required. In acceptance testing, for example, a production-like test environment is ideal, while in component testing the developers typically use their own development environment.

Read More