...

RBT is built on top of Risk Management and thus the same concepts and principles apply.

But what is Risk, Risk Management and Risk-Based Testing after all?

Lets start by the basic concepts.

Concepts and Terminology

Risk

...

This risk criteria includes the definition of multiple levels for the probability and impact variables. 

Assess the risks by calculating the likelihood and impact each requirement could have on the project taking the defined criteria's like cost, schedule, resources, scope, technical performance safety, reliability, complexity, etc. into consideration.

Examples: cost, schedule, resources, scope, technical performance safety, reliability, complexity, 

ers, together 
with the risk analyst, inform values to metrics like 
complexity, cost, size, quality, and others in order 
to find the Risk Exposure (RE) value for each 
functionality

Examples:

• ???

Categories

Risk categories are a way of classifying and grouping risks together.

Examples:

• business
• technical
• operational
• project management
• external
• compliance
• ...

"Types" of Risks

Inherent Risk

Existent risk, and implicitly risk level, before any treatment actions are taken.

Residual Risk

Remaining risk, and implicitly residual risk level, after risk treatment actions have been taken.

Exposures

Exposure

Although not part of ISO 31000, "exposure" is commonly used to refer a categorization (e.g. "low", "medium", "high", "severe") of the risk based on its level. Sometimes it's also use a synonym for the calculated risk level value, so please be aware of it.

Residual Exposure

Residual exposure corresponds to the exposure after treatment, based on the remaining residual risk. Sometimes it's also use a synonym for the calculated residual risk level value, so please be aware of it.

Risk Register

The Risk Register is used to store and document risks along with the risk treatment responses. It is an essential tool for assisting on the risk management activities that can give visibility of current risks, help on the process of documenting and reviewing them.

Risk Management

Risk Management corresponds to the "coordinated activities to direct and control an organization with regard to risk".

All organisations and projects are subject to risks, of many different categories, that may or not happen and, if so, may impact on the defined objectives.

The purpose of Risk Management is to have a systematic approach to address risks effectively having the defined objectives in mind, by taking advantage of them in case they're opportunities that benefit our objectives or by minimizing threats that may impact negatively on what we foresee to achieve.

Risk Management Process

There are a set of activities that are part of the Risk Management Process: Establishing the Context, Risk Assessment, Risk Treatment, Monitoring & Reviewing . Risk Assessment is in turn composed of Identification, Analysis and Treatment.

The overall process starts by establishing the context, so users have a common understanding on the objectives, on the internal/external "constraints" and agree on the definition of risk criteria.

After that, users will identify and describe the risks, analyze them as means to obtain its characteristics (e.g. risk level) and decide what actions (i.e. treatments), if any, to take depending on the latter.

Monitoring and reviewing of risks is an ongoing activity that provides a feedback loop to the whole Risk Management Process, to depict changes on context, risk criteria, risks themselves and related treatments.

0. Estabilishing the Context

The foundation of the Risk Management Process is establishing the context.

The context provides information about the objectives that need to be pursued and also the internal and external environments or contexts that be involved.

Part of setting the context is the definition of risk criteria itself, based on the internal and external contexts and on the objectives to be pursue.

1. Risk Assessment

Risk assessment corresponds to the overall process of performing risk identification, analysis and evaluation (ISO 31000).

1.1. Risk Identification

Risk identification is the process of finding, recognizing and describing risks (ISO 31000). 

1.2. Risk Analysis

Per ISO 31000, it corresponds to: "Process to comprehend the nature of risk and to determine the level of risk. Risk analysis provides the basis for risk evaluation and decisions about risk treatment. Risk analysis includes risk estimation."

In other words, and oversimplifying it a bit, the purpose of this activity is to assess the event probability/likelihood and impact for the identified risks. This may involve answering a set of well-defined weightned questions to find out their value.

Probability and Impact can be defined as integers with a limited range (e.g. 0-4). By having a very limited range for the values, it makes the process more manageable and easier to perform.

Categories

Risk categories are a way of classifying and grouping risks together.

Examples:

• business
• technical
• operational
• project management
• external
• compliance
• ...

"Types" of Risks

Inherent Risk

Existent risk, and implicitly risk level, before any treatment actions are taken.

Residual Risk

Remaining risk, and implicitly residual risk level, after risk treatment actions have been taken.

Exposures

Exposure

Although not part of ISO 31000, "exposure" is commonly used to refer a categorization (e.g. "low", "medium", "high", "severe") of the risk based on its level. Sometimes it's also use a synonym for the calculated risk level value, so please be aware of it.

Residual Exposure

Residual exposure corresponds to the exposure after treatment, based on the remaining residual risk. Sometimes it's also use a synonym for the calculated residual risk level value, so please be aware of it.

Risk Register

The Risk Register is used to store and document risks along with the risk treatment responses. It is an essential tool for assisting on the risk management activities that can give visibility of current risks, help on the process of documenting and reviewing them.

Risk Management

Risk Management corresponds to the "coordinated activities to direct and control an organization with regard to risk".

All organisations and projects are subject to risks, of many different categories, that may or not happen and, if so, may impact on the defined objectives.

The purpose of Risk Management is to have a systematic approach to address risks effectively having the defined objectives in mind, by taking advantage of them in case they're opportunities that benefit our objectives or by minimizing threats that may impact negatively on what we foresee to achieve.

Risk Management Process

There are a set of activities that are part of the Risk Management Process: Establishing the Context, Risk Assessment, Risk Treatment, Monitoring & Reviewing . Risk Assessment is in turn composed of Identification, Analysis and Treatment.

The overall process starts by establishing the context, so users have a common understanding on the objectives, on the internal/external "constraints" and agree on the definition of risk criteria.

After that, users will identify and describe the risks, analyze them as means to obtain its characteristics (e.g. risk level) and decide what actions (i.e. treatments), if any, to take depending on the latter.

Monitoring and reviewing of risks is an ongoing activity that provides a feedback loop to the whole Risk Management Process, to depict changes on context, risk criteria, risks themselves and related treatments.

0. Estabilishing the Context

The foundation of the Risk Management Process is establishing the context.

The context provides information about the objectives that need to be pursued and also the internal and external environments or contexts that be involved.

Part of setting the context is the definition of risk criteria itself, based on the internal and external contexts and on the objectives to be pursue.

1. Risk Assessment

Risk assessment corresponds to the overall process of performing risk identification, analysis and evaluation (ISO 31000).

1.1. Risk Identification

Risk identification is the process of finding, recognizing and describing risks (ISO 31000). 

1.2. Risk Analysis

Per ISO 31000, it corresponds to: "Process to comprehend the nature of risk and to determine the level of risk. Risk analysis provides the basis for risk evaluation and decisions about risk treatment. Risk analysis includes risk estimation."

In other words, and oversimplifying it a bit, the purpose of this activity is to assess the event probability/likelihood and impact for the identified risks. This may involve answering a set of well-defined weightned questions to find out their value.

Probability and Impact can be defined as integers with a limited range (e.g. 0-4). By having a very limited range for the values, it makes the process more manageable and easier to perform.

After having agreed on risk's probability and impact, The same principle can be applied to the calculated risk level After having agreed on risk's probability and impact, The same principle can be applied to the calculated risk level which we may categorize by truncating it to something similar to, for example:

...

Briefly, some possible "treatments" are:

...

• at the issue level with some custom fields
• or at the project level, using a specific issue type

Risk-Based Testing (RBT)

...

Risks can affect the overall quality, including the customer perceived quality.

Quality is a broad word embracing "fit" of the product to customer needs and many quality attributesquality attributes (i.e. "quality properties" as per ISO/IEC 25010:2011), which risks may affect. Intrinsic part of quality is ensuring that a product has few bugs (or at least reduce the probability of bugs on important/risky areas).

...

Development team has similar concerns; however, it sees quality in a broader scope as it has to maintain and possibly evolve the product being developed. Timing and costs need to be managed effectively to be on budget and avoid delays; however quality must not be hurt and if decisions need to be taken to meet timing/costs criteria, RBT can provide the means to ensure that focus will be given to the features/issues that matter most to customers.

Both customers and development team want to avoid important defects, thus RBT focus testing effort in what matters most, on where we can get most value from.

Overall, RBT purpose and benefits can be sumed up as:

be given to the features/issues that matter most to customers.

Both customers and development team want to avoid important defects, thus RBT focus testing effort in what matters most, on where we can get most value from.

Overall, RBT purpose and benefits can be sumed up as:

• Focus more on customer
  • test more thoroughly what customers are more concerned with
    
    • deliver what is most important to business
• Reduce probability and impact of negative risks
  
  • by focusing testing on higher, negative risks, probability of missing important defects lowers and therefore the probability assigned to the risk lowers as its corresponding risk level; on the other hand, as testing also provides feedback to the team including its developers, impact of a certain risk may be also decreased as the feature being worked out may be done differently/better
• Increase confidence
• Focus more on customer
  • test more thoroughly what customers are more concerned with
    
    • deliver what is most important to business
• Reduce probability and impact of negative risks
  
  • by focusing testing on higher, negative risks, probability of missing important defects lowers and therefore the probability assigned to the risk lowers as its corresponding risk level; on the other hand, as testing also provides feedback to the team including its developers, impact of a certain risk may be also decreased as the feature being worked out may be done differently/better
• Increase risk level of positive risks
  • If an opportunity is identified, RBT can be used to provide thoroughly testing and take advantage of it. If a feature is being implemented and if the team finds that making it slighter differently, perhaps by generalizing it further, for examplem or making it more visible, it can increase the probability of the positive risk happen, as end users may start using it for additional scenarios and thus increase the product added value. Using testing as a constructive feedback loop, knowing the opportunities and their relative relevance, can increase the likelihood, impact and the overall risk level for opportunities
• Find important defects sooner
  • RBT can help on finding more important defects first, by focused testing on higher risks (paper)by focused testing on higher risks (paper)
  • by focused testing on the higher risks (or risky areas/features), RBT ensures that important items are exhaustively tested and important defects are found sooner; thus, product and it's most important items (e.g. features) can be released with a high degree of confidence, while ensuring they meet expected quality goals
• Optimize QA effort and cost
  • RBT can answer questions such as...
  • • What should we test? Everything? But we don't have time
...
• When can we stop testing? We have to make the release and move on...
• Can we reduce the testing effort somehow? How and at what "cost"?
• If we have more time for testing, what is the best way to take advantage of it?
• • • ...
    • When can we stop testing? We have to make the release and move on...
    • Can we reduce the testing effort somehow? How and at what "cost"?
    • If we have more time for testing, what is the best way to take advantage of it?
  • RBT provides the means to define the testing scope, focusing on what is relevant, by identifying what tests to execute and their execution priority, given time and resource constraints; the overall amount of test cases may be reduced as not everything needs to be tested or be tested in the same depth 
  • RBT provides a way (based on risks) to choose tests for regression testing
  • RBT provides some clues for selecting candidates for automation
• Increase risk level of positive risks
  • If an opportunity is identified, RBT can be used to provide thoroughly testing and take advantage of it. If a feature is being implemented and if the team finds that making it slighter differently, perhaps by generalizing it further, for examplem or making it more visible, it can increase the probability of the positive risk happen, as end users may start using it for additional scenarios and thus increase the product added value. Using testing as a constructive feedback loop, knowing the opportunities and their relative relevance, can increase the likelihood, impact and the overall risk level for opportunities
• RBT provides the means to define the testing scope, focusing on what is relevant, by identifying what tests to execute and their execution priority, given time and resource constraints 
• RBT provides a way (based on risks) to choose tests for regression testing
RBT provides some clues for selecting candidates for automation

• Make a release go/no go decision based on risks
  • sometimes you may need to ship the product sooner, for time or budget constraints
    • RBT can give you visibility of risks, so you can take a go/no go decision "knowing the risk"
  • RBT implicitly explains why certain tests where executed over other ones, and thus eases communication with other stakeholders and avoids discussion on why some tests where not executed at all
  Increased confidence
  • by focused testing on the higher risks (or risky areas/features), RBT ensures that important items are exhaustively tested and import defects are found sooner; thus, product and it's most important items (e.g. features) can be released with a high degree of confidence, while ensuring they meet expected quality goals

RBT at different levels

Risks may be evaluated at different levels:

• at project level

  Expand
  title Examples...
  missing cross-functional knowledge on team outsourcing IP

• at "requirement" (e.g. Story) level

  Expand
  title Examples...
  open-source library support performance degradation in concurrent scenarios

• directly at Test level

  Expand
  title Examples...
  validation of specific scenarios/use cases whose relevance is different from other ones, in some  context (e.g. on a specific release)

Many teams perform RBT identifying and evaluating risks at "requirement"/Story level: thoroughly testing is performed on risky "requirements", while less riskier "requirements" are tested in a more pragmatic way.

However, sometimes "requirements" may not exist at all, defined explicitly as such. This may be common in scenarios where teams inherit legacy projects, that have few or no "requirements" but that have Tests that validate the intended behaviour.

...

Project level

...

"Requirement" / User Story level

...

Test level

...

Pros

...

• some risks may simply be project wide

...

• most common approach
• allows to decide whether a story should be addressed and how based on its risk
• shapes the tests definition on the Test Design phase taking into account the risk

...

• allows distinct priorization of Test cases for the same requirement
• allows RBT in projects without requirements (e.g. legacy projects)

...

Cons

...

• misses a more granular way of assessing risks and addressing them

...

• not differentiates between Tests for the same story, although some additional criteria (e.g. ”priority”) could be used for that
• not applicable to projects without requirements

...

• a lot of effort to evaluate Risk for tons of test cases
• Tests are not specified taking into account requirement risk; applies only to existing test cases and not to new ones (at the acceptance criteria would probably be a better approach)

as such. This may be common in scenarios where teams inherit legacy projects, that have few or no "requirements" but that have Tests that validate the intended behaviour.

RBT Process

RBT is much more besides selecting what Tests to execute and their order; RBT affects test planning, authoring and execution.

You may implement RBT in several different ways, formalizing the process a bit more or not.

The overall process involves the following macro-activities.

1. What and how "things" can affect our project?
   1. Risk Identification
      1. identification of risks to product (e.g. software/hardware) features; this is normally performed by the team and discussed together
      2. risks are identified before implementation starts and are reviewed throughout the development life cycle; new risks can be identified during implementation
   2. Risk Analysis
      1. risks are discussed together in the team (may involve customer)
      2. risks impact and probability are calculated, and thus the related risk level
2. How will we handle it, in the most effective way (i.e. the one that can return most value)?
   1. Risk Evaluation and Treatment
      1. Test Planning
         
         1. using the input from Risk Analysis, the test manager can...
            1. define test strategy (e.g. level of testing to perform, techniques to use, environments to choose)
            2. estimate testing effort
            3. define/estimate schedule (target dates, number of testing iterations)
      2. Test Design
         1. specification of the Tests taking the identified risks as input; tests are designed to mitigate the risk (i.e. diminish their probability)
         2. make use of more extensive data with data-driven testing and automated testing/checking, if needed
      3. Test Execution
         1. perform testing by descending order of risk level (i.e. execute Tests related with higher risks first); experienced testers, with high degree of domain knowledge may provide more valuable feedback 
         2. thoroughly test risky items, using scripted and exploratory approaches
   2. Risk Monitoring and Reviewing
      1. look at items where you assessed the risk and evaluate if you need to take additional measures/treatments ("Are those items having failed tests? Were bugs founds? Are those bugs relevant? Did we find any new risk while testing?")

RBT in Waterfall

In projects following waterfall, or one of the waterfall variants, in their SDLC, RBT can fit as follows (high-level overview).

1. Risk Identification & Risk Assessment
   1. BAs together with the team discuss risks on the ”Requirement Analysis” phase 
2. Risk Treatment
   1. performed by testers, during the “Testing” phase
      1. Test Planning takes the inputs of Risk Assessment to define the testing strategy and effort
      2. if adopting the scripted testing approach, tester start by detailing exaustively test cases for the most risky item, during Test Design
      3. testers will perform more in-depth testing on the risky items during Test Execution
      4. if important bugs are found on the risky items, the software can go back to "Implementation" phase as-soon-as-possible
3. Risk Monitoring and Reviewing
   1. throughout the project lifecycle

Please note that even though you may be doing Risk Identification and Assessment on the first phase (i.e. "Requirement Analysis"), risk factors are present in all development phases as each one is prone to risks.

As your development project goes through different stages sequentially, problems found at a later stage of the pipeline can can lead to rolling back to a previous phase; thus, it can go from Test Execution back to Test Planning... or from the "Testing" high-level phase back to "Implementation"... or even from "Implementation" back to "Design" and "Requirement Analysis.

RBT in Agile

In projects adopting Agile principles and values software is delivered incrementaly and frequently, being

RBT Process

RBT is much more besides selecting what Tests to execute and their order; RBT affects test planning, authoring and execution.

1. Risk Identification
2. Risk Analysis
3. Risk Response
4. Test Scoping
5. Test Process definition

...

The overall process involves the following macro-activities.

1. What and how "things" can affect our project?
   1. Risk Identification
      1. identification of risks to product (e.g. software/hardware) features; this is normally performed by the team and discussed together
      2. risks are identified before implementation starts and are reviewed throughout the development life cycle; new risks can be identified during implementation
   2. Risk Analysis
      1. risks are discussed together in the team (may involve customer)
      2. risks impact and probability are calculated, and thus the related risk level
2. How will he handle it, in the most effective way (i.e. the one that can return most value)?
   1. Risk Evaluation and Treatment
      1. Test Planning
         
         1. using the input from Risk Analysis, the test manager can  ..
         2. define test strategy (e.g. level of testing to perform, techniques to use, environments to choose)
         3. estimate testing effort
         4. define/estimate schedule (target dates, number of testing iterations)
      2. Test Design
         1. specification of the Tests taking the identified risks as input
         2. use more extensive data for data-driven testing if needed, using automated testing/checking
         3. tests are designed to mitigate the risk (i.e. diminish their probability)
      3. Test Execution
         1. perform testing by descending order of risk level (i.e. execute Tests related with higher risks first); experienced testers, with high degree of domain knowledge may provide more valuable feedback 
         2. thoroughly test risky items, using scripted and exploratory approaches
   2. Risk Monitoring and Reviewing
      1. look at items where you assessed the risk and evaluate if you need to take additional measures/treatments ("Are those items having failed tests? Were bugs founds? Are those bugs relevant? Did we found any new risk while testing?")

RBT in Waterfall

In projects following waterfall methodology in their SDLC, RBT can fit as follows.

<WATERFALL DIAGRAM>

1. Risk Identification & Risk Assessment
   1. BAs together with the team on the ”Requirements” (analysis) phase
2. Risk Treatment
   1. by testers, during the “Testing/Validation” phase
3. Risk Monitoring and Reviewing
   1. throughout the project lifecycle

Please note that risk factors are present in all development phases as each one is prone to risks.

RBT in Agile

Projects adopting Agile principles and values releases, software is delivered incrementaly and frequently, highly focused on customer needs and feedback. In Agile changes are welcome and not really tied to "long phases". Both Agile Software Development and RBT have the customer and its ( most important needs/concerns ) at the center. 

A possible implementation of RBT in Scrum based projects could be as follows. 

1. Risk Identification and Assessment
   1. Release planning (by "business stakeholders" - PO?)
      1. Highest-risk tasks must be placed in the earliest sprints
   2. Sprint planning (by the whole agile team)
      1. Testers can start creating a risk-based analysis by using the stories given for that iteration, identifying the high-risk stories that can be tested on that iteration, and prioritizing the user stories depending on their criticality to their criticalness to the business
      2. Risk’s impact and likelihood can be evaluated using “Risk Poker” (like planning poker)
2. Risk Treatment (by "testers")
   1. During sprint lifetime, by  by performing more extensive testing on high-risk stories (including for regression testing purposes)
3. Risk Monitoring and Reviewing (by whom?)
   1. During sprint lifetime,(( how??)

"Traditional", RBT and Hybrid RBT approaches

While in "traditional" testing tests are executed in "ad-hoc" way (without any special criteria/order), in RBT risk is used to choose what tests to execute and their order of execution.

RBT also conditionates how testing is done (and how test cases are specified): focus is given to tests associated, directly or indirectly, with and higher risk.

In hybrid Hybrid RBT, non risk-based Tests (i.e. tests specified without considering risks) are executed by the risk descending order.

...

• Risk management - Principles and guidelines: ISO 31000:2009(E)

• Souza, Ellen Polliana & Gusmao, Cristine & Venâncio, Júlio. (2010). Risk-Based Testing: A Case Study. 1032 - 1037. 10.1109/ITNG.2010.203. 

• RISK FACTORS IN SOFTWARE DEVELOPMENT PHASES, Haneen Hijazi, Msc Hashemite University, Jordan Shihadeh Alqrainy, PhD; Hasan Muaidi, PhD; Thair Khdour, PhD; Albalqa Applied University, Jordan

• What Techniques Can Be Used for GUI Risk-based Testing?, Behzad Nazarbakhsh and Dietmar Pfahl ,Institute of Computer Science, University of Tartu
• James Bach on Risk-Based Testing: How to conduct heuristic risk analysis, James Bach, 1999
• ISO/IEC 25010:2011 and old ISO 9126:1999 (brief, non-official overview)