This article provides an overview of Risk Management and Risk-Based Testing (RBT).
RBT is built on top of Risk Management and thus the same concepts and principles apply.
In general, we can define Risk as the “effect of uncertainty on objectives” (ISO 31000:2018) that can be positive (opportunity) or negative (threat).
We can also look at it as an uncertain event with a positive or negative effect on the measurable success criteria of a project.
Therefore, and as a way to overcome a common misconception, risks are not always negative; however, many times we may be more focused on these ones.
Examples:
Level of Risk
The level of risk, sometimes just simply called "risk value" or "risk", can be defined as a combination of the probability/likelihood and the impact/consequence of an event on the objectives.
| Risk = Probability (of event) * Impact | |||
| Also know as: | (Level of Risk / Risk Score / Risk Exposure) | (Likelihood/Frequency) | (Consequence/Damage/Revenue/Business Criticality) |
Probability / Likelihood
Probability of an event occurring.
It can be defined as a percentage, a number interval (e.g. 0-4) or as a ordered list of values (e.g. "low", "medium", "high", "very high").
Probability can be evaluated using multiple weightned criteria/factors (e.g. software maturity, software complexity, type of change, number of changed components, related defect rate, etc).
Impact / Consequence
Outcome of an event affecting objectives. In other words, it can also be defined as the overall loss or revenue that could occur IF the risk occurs.
An impact...
Thus, if the probability of an event is 0, that risk won't happen. Likewise, a risk with no impact also leads to a risk score of 0.
Having an high-impact risk by itself may be something not to worry much about unless the probability of the associated event is "high" enough.
Impact be defined as a number interval (e.g. [0-4]) or as a ordered list of values (e.g. ["low", "medium", "high", "very high"]).
The risk source is the "element which alone or in combination has the intrinsic potential to give rise to risk" (ISO 31000).
Examples:
Risk criteria corresponds to the "terms of reference against which the significance of a risk is evaluated" (ISO 31000).
"Criteria can be imposed by, or derived from, legal and regulatory requirements and other requirements to which the organization subscribe."
This risk criteria includes the definition of multiple levels for the probability and impact variables.
Assess the risks by calculating the likelihood and impact each requirement could have on the project taking the defined criteria's like cost, schedule, resources, scope, technical performance safety, reliability, complexity, etc. into consideration.
Examples: cost, schedule, resources, scope, technical performance safety, reliability, complexity,
ers, together
with the risk analyst, inform values to metrics like
complexity, cost, size, quality, and others in order
to find the Risk Exposure (RE) value for each
functionality
Examples:
Risk categories are a way of classifying and grouping risks together.
Examples:
Existent risk, and implicitly risk level, before any treatment actions are taken.
Remaining risk, and implicitly residual risk level, after risk treatment actions have been taken.
Although not part of ISO 31000, "exposure" is commonly used to refer a categorization (e.g. "low", "medium", "high", "severe") of the risk based on its level. Sometimes it's also use a synonym for the calculated risk level value, so please be aware of it.
Residual exposure corresponds to the exposure after treatment, based on the remaining residual risk. Sometimes it's also use a synonym for the calculated residual risk level value, so please be aware of it.
The Risk Register is used to store and document risks along with the risk treatment responses. It is an essential tool for assisting on the risk management activities that can give visibility of current risks, help on the process of documenting and reviewing them.
Risk Management corresponds to the "coordinated activities to direct and control an organization with regard to risk".
All organisations and projects are subject to risks, of many different categories, that may or not happen and, if so, may impact on the defined objectives.
The purpose of Risk Management is to have a systematic approach to address risks effectively having the defined objectives in mind, by taking advantage of them in case they're opportunities that benefit our objectives or by minimizing threats that may impact negatively on what we foresee to achieve.
There are a set of activities that are part of the Risk Management Process: Establishing the Context, Risk Assessment, Risk Treatment, Monitoring & Reviewing . Risk Assessment is in turn composed of Identification, Analysis and Treatment.
The overall process starts by establishing the context, so users have a common understanding on the objectives, on the internal/external "constraints" and agree on the definition of risk criteria.
After that, users will identify and describe the risks, analyze them as means to obtain its characteristics (e.g. risk level) and decide what actions (i.e. treatments), if any, to take depending on the latter.
Monitoring and reviewing of risks is an ongoing activity that provides a feedback loop to the whole Risk Management Process, to depict changes on context, risk criteria, risks themselves and related treatments.
The foundation of the Risk Management Process is establishing the context.
The context provides information about the objectives that need to be pursued and also the internal and external environments or contexts that be involved.
Part of setting the context is the definition of risk criteria itself, based on the internal and external contexts and on the objectives to be pursue.
Risk assessment corresponds to the overall process of performing risk identification, analysis and evaluation (ISO 31000).
Risk identification is the process of finding, recognizing and describing risks (ISO 31000).
Per ISO 31000, it corresponds to: "Process to comprehend the nature of risk and to determine the level of risk. Risk analysis provides the basis for risk evaluation and decisions about risk treatment. Risk analysis includes risk estimation."
In other words, and oversimplifying it a bit, the purpose of this activity is to assess the event probability/likelihood and impact for the identified risks. This may involve answering a set of well-defined weightned questions to find out their value.
Probability and Impact can be defined as integers with a limited range (e.g. 0-4). By having a very limited range for the values, it makes the process more manageable and easier to perform.
After having agreed on risk's probability and impact, The same principle can be applied to the calculated risk level which we may categorize by truncating it to something similar to, for example:
If you segment the calculated value or not, and how, is a bit up to you.
This is an example of a risk matrix with the calculated risk levels. You may also use colours to quickly depict the higher risks.
Probability (likelihood) | 4 (very high) | 0 | 4 | 8 | 12 | 16 |
|---|---|---|---|---|---|---|
4 (high) | 0 | 3 | 6 | 9 | 12 | |
2 (medium) | 0 | 2 | 4 | 6 | 8 | |
1 (low) | 0 | 1 | 2 | 3 | 4 | |
0 (none) | 0 | 0 | 0 | 0 | 0 | |
0 (none) | 1 (low) | 2 (medium) | 3 (high) | 4 | ||
Impact (consequence) | ||||||
Per ISO 31000, it corresponds to the "process of comparing the results of risk analysis with risk criteria to determine whether the risk and/or its magnitude is acceptable or tolerable. Risk evaluation assists in the decision about risk treatment".
Having the inputs from Risk Analysis, risks need to be evaluated and compared against certain criteria (i.e. risk criteria). This will lead to some sort of relative risk ranking that can be used to decide what decisions to take, which risks need treatments to apply and their priorities.
In the end, users need to decide whether risks are acceptable or tolerable, of if they want to modify them instead by treatment measures. As an example, you may decide to apply treatments only on risks categorized as "high", "very high" or "severe".
Risk Treatment comes as natural consequence of Risk Evaluation and it corresponds to the "process to modify risk" (ISO 31000).
A risk can be "modified" either by removing its source or because the probability or the impact changed.
Whenever acting on a risk, users can apply one or more of the several possible response strategies (i.e. treatments), depending on the nature/consequence of the risk (i.e. positive or negative risk).
Briefly, some possible "treatments" are:
"Communication and consultation with external and internal stakeholders should take place during all stages of the risk management process"
Communication should happen continuously in order to leverage other activities.
First of all, internal and external stakeholders should be consulted, as means to establish the context (or reviewing it later on). However, all of them are invited to provide feedback on the remaining activities as projects/systems and risks evolve in a dynamic.
These are some of the questions we need to answer.
"Risk monitoring and reviewing" tries to answer the previous questions; it can be seen as an active surveillance over the whole process as means to ensure that we're effectively on control.
There are several apps for Risk Management in Jira.
However, they follow different approaches for handling risk.
Some of them perform Risk Management...
RBT (Risk-Based Testing) is a testing strategy that follows the principles of Risk Management but in testing context.
In software development projects, or in any development project in general, there are a set of common objectives among other ones:
These goals can be affected by risks coming from different sources (internal or external), due to all sorts of factors. In case of negative risks, the risk level will depend on things such as the complexity of the change, number and kind of impacted users, frequency of usage, likelihood of failure, etc.
Risks can affect the overall quality, including the customer perceived quality. Quality is a broad word embracing "fit" of the product to customer needs and many quality attributes. Intrinsic part of quality is ensuring that a product has few bugs (or at least reduce the probability of bugs on important/risky areas).
If testing is performed as means to find/avoid bugs, RBT can be used as a "mitigation", or even as a "avoidance", strategy.
On the other hand, testing is also about understanding, discovering and providing valuable feedback that can help build "better" features and better products. In that sense, RBT may be used to perform thoroughly testing or doing it more lighter/roughly depending on the risk.
Thus, RBT can be seen as a strategy that uses risks as means to adequate testing (depth and priority) in order to maximize testing goals. It will affect all testing activities (e.g test planning, design, execution).
Therefore, RBT assists on taking testing related decisions based on assessment of risks.
The main purpose of RBT is to use risk management principles to adequate testing.
First of all, it provides a framework for clear communication and discussion of risks, by defining terms and a common language; it also makes risks visible and actionable.
RBT covers customer needs and also development team needs, using risks as the input to support testing activities.
Customer are mostly concerning about business features, timing, visible quality and costs.
Development team has similar concerns; however, it sees quality in a broader scope as it has to maintain and possibly evolve the product being developed. Timing and costs need to be managed effectively to be on budget and avoid delays; however quality must not be hurt and if decisions need to be taken to meet timing/costs criteria, RBT can provide the means to ensure that focus will be given to the features/issues that matter most to customers.
Both customers and development team want to avoid important defects, thus RBT focus testing effort in what matters most, on where we can get most value from.
Overall, RBT purpose and benefits can be sumed up as:
Risks may be evaluated at different levels:
at project level
|
at "requirement" (e.g. Story) level
|
directly at Test level
|
Many teams perform RBT identifying and evaluating risks at "requirement"/Story level: thoroughly testing is performed on risky "requirements", while less riskier "requirements" are tested in a more pragmatic way.
However, sometimes "requirements" may not exist at all, defined explicitly as such. This may be common in scenarios where teams inherit legacy projects, that have few or no "requirements" but that have Tests that validate the intended behaviour.
Project level | "Requirement" / User Story level | Test level | |
|---|---|---|---|
Pros |
|
|
|
Cons |
|
|
|
RBT is much more besides selecting what Tests to execute and their order; RBT affects test planning, authoring and execution.
The overall process involves the following macro-activities.
In projects following waterfall methodology in their SDLC, RBT can fit as follows.
<WATERFALL DIAGRAM>
Please note that risk factors are present in all development phases as each one is prone to risks.
Projects adopting Agile principles and values releases, software is delivered incrementaly and frequently, highly focused on customer needs and feedback. In Agile changes are welcome and not really tied to "long phases". Both Agile Software Development and RBT have the customer and its (most important needs/concerns) at the center.
A possible implementation of RBT in Scrum based projects could be as follows.
While in "traditional" testing tests are executed in "ad-hoc" way, in RBT risk is used to choose what tests to execute and their order of execution.
RBT also conditionates how testing is done (and how test cases are specified): focus is given to tests associated, directly or indirectly, with and higher risk.
In hybrid RBT, non risk-based Tests (i.e. tests specified without considering risks) are executed by the risk descending order.
RISK FACTORS IN SOFTWARE DEVELOPMENT PHASES, Haneen Hijazi, Msc Hashemite University, Jordan Shihadeh Alqrainy, PhD; Hasan Muaidi, PhD; Thair Khdour, PhD; Albalqa Applied University, Jordan