Overview

This article provides an overview of Risk-Based Testing (RBT).

RBT is built on top of Risk Management and thus the same concepts and principles apply.


If you want to learn how to perform RBT whenever using Xray, please have a look at Performing Risk-Based Testing (RBT) with Xray


Risk-Based Testing (RBT)

RBT (Risk-Based Testing) is a testing strategy that follows the principles of Risk Management but in a testing context.

In software development projects, or in any development project in general, there are a set of common objectives among these are:

These goals can be affected by risks coming from different sources (internal or external,) due to all sorts of factors. In case of negative risks, the risk level will depend on things such as the complexity of the change, number and kind of impacted users, frequency of usage, likelihood of failure, etc.

Risks can affect the overall quality, including the customer perceived quality.

Quality is a broad word embracing "fit" of the product to customer needs and many quality attributes (i.e. "quality properties" as per ISO/IEC 25010:2011), which risks may affect. An intrinsic part of quality is ensuring that a product has few bugs (or at least a reduced probability of bugs on important/risky areas).

If testing is performed as a means to find/avoid bugs, RBT can be used as a "mitigation", or even as an "avoidance" strategy.

On the other hand, testing is also about understanding, discovering and providing valuable feedback that can help build "better" features and better products. In that sense, RBT may be used to perform thorough testing or doing it lighter/roughly depending on the risk. 

Thus, RBT can be seen as a strategy that uses risks as means for adequate testing (depth and priority) in order to maximize testing goals. It will affect all testing activities (e.g test planning, design, execution).

Therefore, RBT assists in making testing related decisions based on the assessment of risks.

Purpose

The main purpose of RBT is to use risk management principles for adequate testing.

First of all, it provides a framework for clear communication and discussion of risks, by defining terms and a common language; it also makes risks visible and actionable.

RBT covers customer needs and also the needs of the development team, using risks as the input to support testing activities.

Customers are mostly concerned about business features, timing, visible quality and costs.

The development team has similar concerns; however, it sees quality in a broader scope as it has to maintain and possibly evolve with the product being developed. Timing and costs need to be managed effectively to be on a budget and avoid delays; however quality must not be hurt and if decisions need to be taken to meet timing/costs criteria, RBT can provide the means to ensure that focus will be given to the features/issues that matter most to customers.

Both customers and the development team want to avoid important defects, thus RBT focuses the testing efforts on what matters most - where we can get most value from.


Overall, RBT's purpose and benefits can be summed up as:

RBT at different levels

Risks may be evaluated at different levels:

Many teams perform RBT identifying and evaluating risks at "requirement"/Story level: thoroughly testing is performed on risky "requirements", while less risky "requirements" are tested in a more pragmatic way.

However, sometimes "requirements" may not exist at all, defined explicitly as such. This may be common in scenarios where teams inherit legacy projects, that have few or no "requirements" but that have Tests that validate the intended behaviour.



Project level

"Requirement" / User Story level

Test level

Pros

  • some risks may simply be project wide
  • most common approach
  • allows deciding whether a story should be addressed and how based on its risk
  • shapes the test's definition on the Test Design phase taking into account the risk
  • allows distinct prioritization of Test cases for the same requirement
  • allows RBT in projects without requirements (e.g. legacy projects)

Cons

  • misses a more granular way of assessing risks and addressing them
  • not differentiates between Tests for the same story, although some additional criteria (e.g. ”priority”) could be used for that
  • not applicable to projects without requirements
  • a lot of effort to evaluate Risk for tons of test cases
  • Tests are not specified taking into account requirement risk; applies only to existing test cases and not to new ones (at the acceptance criteria would probably be a better approach)

RBT Process

RBT is much more than selecting what Tests to execute and their order; RBT affects test planning, authoring and execution.

You may implement RBT in several different ways, formalizing the process a bit more or not.


The overall process involves the following macro-activities.

  1. What and how "things" can affect our project?
    1. Risk Identification
      1. identification of risks to product (e.g. software/hardware) features; this is normally performed by the team and discussed together
      2. risks are identified before implementation starts and are reviewed throughout the development life cycle; new risks can be identified during implementation
    2. Risk Analysis
      1. risks are discussed together in the team (may involve customer)
      2. risks impact and probability are calculated, and thus the related risk level
  2. How will we handle it in the most effective way (i.e. what can return the most value)?
    1. Risk Evaluation and Treatment
      1. Test Planning
        1. using the input from Risk Analysis, the test manager can...
          1. define test strategy (e.g. level of testing to perform, techniques to use, environments to choose)
          2. estimate testing effort
          3. define/estimate schedule (target dates, number of testing iterations)
      2. Test Design
        1. specification of the Tests taking the identified risks as input; tests are designed to mitigate the risk (i.e. diminish their probability)
        2. make use of more extensive data with data-driven testing and automated testing/checking, if needed
      3. Test Execution
        1. perform testing by descending order of risk level (i.e. execute Tests related to higher risks first); experienced testers, with a high degree of domain knowledge, may provide more valuable feedback 
        2. thoroughly test risky items, using scripted and exploratory approaches
    2. Risk Monitoring and Reviewing
      1. look at items where you assessed the risk and evaluate if you need to take additional measures/treatments ("Are those items having failed tests? Were bugs found? Are those bugs relevant? Did we find any new risk while testing?")

RBT in Waterfall and variants

In projects following waterfall, or one of the waterfall variants, in their SDLC, RBT can fit as followed: (high-level, simplified overview)

  1. Risk Identification & Risk Assessment
    1. BAs together with the team discuss risks in the ”Requirement Analysis” phase 
  2. Risk Treatment
    1. performed by testers, during the “Testing” phase
      1. Test Planning takes the input of Risk Assessment to define the testing strategy and effort; the testing scope is agreed
      2. if adopting the scripted testing approach, testers start by exhaustively detailing test cases for the riskiest items, during Test Design
      3. testers will perform more in-depth testing on the risky items during Test Execution
      4. if important bugs are found on the risky items, the software can go back to "Implementation" phase as-soon-as-possible
  3. Risk Monitoring and Reviewing
    1. throughout the project development life cycle; special focus is given at testing closure, 


Please note that even though you may be doing Risk Identification and Assessment on the first phase (i.e. "Requirement Analysis,") risk factors are present in all development phases as each one is prone to risks.

As your development project goes through different stages sequentially, problems found at a later stage of the pipeline can lead to rolling back to a previous phase; thus, it can go from Test Execution back to Test Planning... or from the "Testing" high-level phase back to "Implementation"... or even from "Implementation" back to "Design" and "Requirement Analysis.

RBT in Agile

In projects adopting Agile principles and values software is delivered incrementally and frequently, and highly focused on customer needs and feedback. In Agile, changes are welcome and not really tied to "long phases." Both Agile Software Development and RBT have the customer and their most important needs/concerns at the center. 

A possible implementation of RBT in Scrum based projects could be as follows. 

  1. Risk Identification and Assessment
    1. Release planning
      1. Highest-risk tasks must be placed in the earliest sprints
    2. Backlog refinement/grooming
      1. the team can identify and discuss risks together on each PBI (product backlog item)
    3. Sprint planning (by the whole agile team)
      1. Testers can start creating a risk-based analysis by using the stories given for that iteration, identifying the high-risk stories that can be tested on that iteration, and prioritizing the user stories depending on their criticalness to the business
      2. Risk’s impact and likelihood can be evaluated using “Risk Poker” (like planning poker)
  2. Risk Treatment
    1. During sprint lifetime, by performing more extensive testing on high-risk stories (including for regression testing purposes)
  3. Risk Monitoring and Reviewing
    1. During sprint lifetime

"Traditional", RBT and Hybrid RBT approaches

While in "traditional" testing, tests are executed in "ad-hoc" way (without any special criteria/order), in RBT risk is used to choose what tests are to be executed and their order of execution.

RBT also conditions how testing is done (and how test cases are specified): focus is given to tests associated, directly or indirectly, with and higher risk.

In Hybrid RBT, non-risk-based Tests (i.e. tests specified without considering risks) are executed by the risk descending order.

References