Versions Compared

Old Version 2

changes.mady.by.user Sergio Freire

Saved on

compared with

New Version Current

changes.mady.by.user Unknown User (fnsi)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This article provides background information on Risk-Based Testing (RBT) and its foundations on Risk Management concepts, which will be briefly presented and clarified.

...

The first step is to decide where to set the risk (i.e. on requirements, Tests or at project level); that will affect how you will implement the related fields and how you will use Xray during RBT.

...

After you have decided where to set the risk and how, and configured properly, you may perform RBT and use Xray to assist you.  

...

Info
titleLearn more

If you're new to Risk Management and RBT, please check the Risk Management and Risk-Based Testing article.

Anchor
wheretosetrisk
wheretosetrisk

Where to set the risk

As long as you are able to define a sort-able value for the risk as a custom field in either cover-able issues (e.g. "requirements"/stories) or Test themselves, then you will be able to select and rank Tests for execution, based on risk level. If you have identified global risks at project level as issues, and those risks can be mitigated somehow by testing, then you can also use RBT with them.

...

The following screenshots show risk being defined at different levels, while also using different tools to implement them. 

    

Adding Tests to a Test

...

Execution/Test

...

Plan/Test Set, sorted by risk level

Risk defined at Test level

...

Overall Test Results gadget

Image RemovedImage Added  Image RemovedImage Added

Tests List gadget

Image RemovedImage Added  Image RemovedImage Added

Tests report (legacy report on project tab)

The legacy Tests report, available on the project tab, can be configured to show additional sortable columns. Thus, you may add the risk level related field (or others) and sort by it.


Image RemovedImage Added

Note: this report is still a bit limited, so no advanced/flexible filtering mechanisms are available.

...

Note: all changes are tracked, directly on the respective issues; in the case of Test Runs, changes also get tracked on the related Activity section.Test Runs, changes also get tracked on the related Activity section.

Use different Test Plans based on Risk level

Xray gives you the ability to create one or more Test Plans if you wish to manage and track the testing progress for different tests independently.

Therefore you may create, for example, one Test Plan containing all the Tests related with with the highest risk level items and distinct ones for the other risk levels; on each Test Plan you may further prioritize Tests.

If you have multiple Test Plans, and since they're issues, you may priorize them in the current Sprint or Kanban boards. You can also "label" these Test Plans somehow to distinguish their relative relevance; you may use Jira's native Priority field for this as it comes by default and fulfills this purpose.

Anchor
setupandconfig
setupandconfig

Setup and Configuration

In order to start with RBT, you have to configure the risk management related fields. You may use Jira built-in capabilities or use a risk management or even a customization app instead.

...

If you're adopting risk at Test level, you may create several saved filters (e.g. "tests_risk_low", "tests_risk_medium", "tests_risk_high", "tests_risk_very_high", "tests_risk_severe") as this can later be quite useful, namely for using within gadgets or reports.

Image RemovedImage Added


Risk at requirement or at project level

...

Create a set of filters, each one related with a different risk level. Using a nomenclature for the saved filters can help you out. Example: "RR_risks_low", "RR_risks_medium", "RR_risks_high", "RR_risks_extreme".

Image RemovedImage Added

To make our life easier, we may also use saved filters to group Tests by the indirectly associated risk revel, using JQL and the requirementTests() function. 

Image RemovedImage Added

You won't need these pre-created filters to pick tests sorted by risk level whenever adding Tests to a Test Execution/Test Plan as you may write JQL query inline; however, they will be required if you aim to filter out Tests in gadgets and in reports by the related risk level.

...

After choosing the type of fields you'll need, adding/creating them is straightforward; this can be done right from the issue screen using Admin > Add field. 

Image RemovedImage Added    Image RemovedImage Added


It can also be done from Jira's administration, under Issues > Custom fields > Add custom field.

Image RemovedImage Added

Configuration examples

...

If you use a "Select List" then you need to make sure their option values are sorted in the right order and that their values are also alphabetically sorted in the same way.

Image RemovedImage Added

In the previous example, 0 is redundant with having an empty value; thus, you may either set the default to be 0 (instead of "none") or remove the "0" option.

...

You may create a custom field, either "Number" or "Select List (single choice)" based, to store and manage your risk. A "Select List" may provide additional benefits.

Image RemovedImage Added  Image RemovedImage Added


No matter your choice, the risk level field won't be automatically calculated based on the probability/impact fields; you'll need to use a app or custom development for that as shown in the next sections.

...

To configure Risk Management for Jira, go to Add-ons > Risk Management > Required > Custom Fields.

Image RemovedImage Added

In the previous screen you can choose to reuse existing fields for abstracting the probability and the impact (they should be "Select List" based).

It can also create these fields for you as well, along with the risk level. Choose "Add Custom Fields" and the name for the impact (i.e. consequence), probability and risk level (i.e. risk index) fields; please check "RiskIndexNumberField" flag.


Image RemovedImage Added


Info
titlePlease note
If you already had values for probability and impact/consequence, indexing will not recalculate the risk level, as the app uses a listener to update it upon changes.

...

On issues having the impact/consequence and probability related fields, risk level ("Risk Score" on the following example) will be updated upon update on any of these fields.

Image RemovedImage Added


Within the Risk Matrix, issues can easily be found/searched as they're mapped to a cell and to a specific color, depending on their probability, impact/consequence and calculated risk level.

Image RemovedImage Added


In Xray, whenever adding Tests to a Test Execution for example, the risk level field (e.g "Risk Score") can later be used to sort Tests ascending/descending; it can also be used as a filter.

Image RemovedImage Added


Using Risk Register app

...

  1. configure Risk Register (fields, screens, issue type)
    1. configure global settings of the app (Add-ons > Risk Register > Settings), including the custom fields used by Risk Register, which are always created (you may change their name afterwards). If you have existing custom fields with similar names, new ones will be created (no existing fields will or can be reused).   
      You need also to define which issue type will be used to abstract the risk; Risk Register can create one for you (e.g. "Risk").
      Image RemovedImage RemovedImage Added  Image Added
    2. enable risk management for your project, either from the project settings "Risk register" section or from Jira administration > Add-ons > Risk Register > Risk registers.
      Image RemovedImage RemovedImage Added Image Added
  2. in Xray, configure Risk as a "requirement issue type" (i.e. a coverable issue type that you can cover with test cases)
    1. Some of the risks that you have identified during risk assessment may be covered with tests. Therefore, they can be handled similar to "requirements" in Xray.  Just go to your Jira administration >  Add-ons > Xray > Issue Type Mapping and drag the Risk issue type to the "Requirement Issue Types" column and save it. The project containing Risk issues must be enabled as a requirements project, either from  Jira administration >  Add-ons > Xray > Requirement Projects or from the project settings page, within the Summary section, using the action "Enable Xray Requirement Coverage."

...

    1. Image Added


Using it

Some of the identified project level risks may be mitigated with tests while other ones may not. 

For those that can, you may start by defining the "Risk assessment" related fields, including probability and impact, in the Risk issues (i.e. the ones whose issue type is configured to be handled as risk).

Image RemovedImage Added


Later on, we'll need to find these Tests based on the "exposure" of the associated/covered risk. To make our life easier, we may use saved filters to group Tests by the indirectly associated risk exposure.

Create a set of filters, each one related with a different "Exposure" (i.e risk level). Using a nomenclature for the saved filters can help you out. Example: "RR_risks_low", "RR_risks_medium", "RR_risks_high", "RR_risks_extreme".

Image RemovedImage Added

To obtain the related Tests, you can use JQL and the requirementTests() function.

Thus, lets say you want to obtain all Tests related with Risks with an "High" exposure in order to add them to an existing Test Execution; you can do as follows.

Image RemovedImage Added

The returned Tests may have different relevance; therefore, if you want, you may use a complementary field (e.g. "Priority") to sort them out.

...

For the calculation of risk you may use a Calculated Single-select" field (recommended) or a "Calculated Number" field from JMCF.

Image RemovedImage Added


Note
After creating the calculated field, you'll need to perform re-indexing if your project already has values for probability and impact.

...

Using a single-select field will ease the search of issues later on.


Image AddedImage Removed


Code Block
languagegroovy
titleGroovy Formula for Risk (Level) using a Calculated Single-select Field
collapsetrue
impact = issue.get("Impact")
probability = issue.get("Probability")

def risk
def level
if ((impact) && (probability)) {
    risk = Double.parseDouble(impact) * Double.parseDouble(probability)
} else {
    risk = null
}

if ((risk>0) && (risk<4)) {
 level = "L1: Low"
} else if ((risk>=4) && (risk<8)) {
 level = "L2: Medium"
} else if ((risk>=8) && (risk<12)) {
  level = "L3: High"
} else if ((risk>=12) && (risk<16)) {
  level = "L4: Very High"
} else if (risk==16) {
  level = "L5: Severe"
} else {
  level = null
}
 
return level

...

After creating the previous fields, you may use it, for example, in the dialog for adding Tests to an existing Test Execution; you can also easily filter issues by risk level.

Image AddedImage Removed

Calculating risk and storing it in a Calculated Number field

In this approach, risk is calculated and stored in a Calculated Number field; we just need to define the groovy script that will be used to calculate it.

Image RemovedImage Added

Code Block
languagegroovy
titleGroovy Formula for Risk (Level) using a Calculated Number Field
collapsetrue
impact = issue.get("Impact")
probability = issue.get("Probability")

def risk
if ((impact) && (probability)) {
    risk = Double.parseDouble(impact) * Double.parseDouble(probability)
} else {
    risk = null
}

return risk

...

  1. create a standard custom field (e.g. "Risk Level") of type "Select List (single choice)," having the options corresponding to the different risk level bands correctly ordered (e.g. "L1: Low", "L2: Medium", "L3: High", "L4: Very High", "L5: Severe"). Make sure the option values maintain their order if they're sorted alphabetically (this will help you later on);
    Image RemovedImage Added

  2. using ScriptRunner, go to you Jira administration and create accustom Script Field (Add-ons > ScriptRunner > Script Fields > Add New Item > Custom Script Field). 

    Image Added

...


  1. Image Added

...


  1.  



Code Block
languagegroovy
titleRisk (Level) calculated using a Script Listener
collapsetrue
import com.atlassian.jira.issue.fields.CustomField
import com.atlassian.jira.issue.CustomFieldManager
import com.atlassian.jira.issue.MutableIssue
import com.atlassian.jira.issue.Issue
import com.atlassian.jira.component.ComponentAccessor
import com.atlassian.jira.event.issue.IssueEvent
import com.atlassian.jira.issue.util.DefaultIssueChangeHolder
import com.atlassian.jira.issue.ModifiedValue
import com.atlassian.jira.issue.history.ChangeItemBean
import com.atlassian.jira.issue.IssueManager
import com.atlassian.jira.issue.customfields.option.Option
import com.atlassian.jira.event.type.EventDispatchOption
import com.atlassian.jira.issue.index.IssueIndexingService
import com.atlassian.jira.util.ImportUtils


import org.apache.log4j.Logger
import org.apache.log4j.Level
  
def log = Logger.getLogger("com.onresolve.jira.groovy")
log.setLevel(Level.DEBUG)

MutableIssue issue = event.issue
def issueType = issue.issueTypeObject.name

// Only process Test or Story based issues (ADAPT IT tou your needs)
if (issueType != "Test" && issueType != "Story"){
    return
}

String cfImpactName = "Impact"
String cfProbabilityName = "Probability"
String cfRiskLevelName = "Risk Level"

def changeManager = ComponentAccessor.getChangeHistoryManager();
def changeLog = event.getChangeLog();

def impactChanged = false, probabilityChanged = false;

if ( changeLog != null ) {
	def changeId = changeLog.getLong( "id" );
	if ( changeId != null ) {
		def change = changeManager.getChangeHistoryById( changeId );
		for (ChangeItemBean bean : change.getChangeItemBeans() ) {
        	if ( bean.getField().equals(cfImpactName) 
                   && ChangeItemBean.CUSTOM_FIELD.equals( bean.getFieldType() )) {
                	log.debug(cfImpactName + " changed");
					impactChanged = true;
            }
            if ( bean.getField().equals(cfProbabilityName) 
                   && ChangeItemBean.CUSTOM_FIELD.equals( bean.getFieldType() )) {
                	log.debug(cfProbabilityName + " changed");
					probabilityChanged = true;
            }
        }
    }
}

if (impactChanged || probabilityChanged){
    def issueManager = ComponentAccessor.issueManager
    def customFieldManager = ComponentAccessor.customFieldManager
    def cfImpact = customFieldManager.getCustomFieldObjectByName(cfImpactName)
    def cfProbability = customFieldManager.getCustomFieldObjectByName(cfProbabilityName)
    
    def option  
    def impact
    def probability

    option = (Option)issue.getCustomFieldValue(cfImpact)
    if (option != null) impact = option.getValue()
     option = (Option)issue.getCustomFieldValue(cfProbability)
    if (option != null) probability = option.getValue()  

    def risk
    def level

    if ((impact) && (probability)) {
        risk = Double.parseDouble(impact) * Double.parseDouble(probability)
    } else {
        risk = null
    }

    if ((risk>0) && (risk<4)) {
     level = "L1: Low"
    } else if ((risk>=4) && (risk<8)) {
     level = "L2: Medium"
    } else if ((risk>=8) && (risk<12)) {
      level = "L3: High"
    } else if ((risk>=12) && (risk<16)) {
      level = "L4: Very High"
    } else if (risk==16) {
      level = "L5: Severe"
    } else {
      level = null
    } 

    def cfRiskLevel = customFieldManager.getCustomFieldObjectByName(cfRiskLevelName)
    def fieldConfig = cfRiskLevel.getRelevantConfig(issue)
    def value = ComponentAccessor.optionsManager.getOptions(fieldConfig)?.find { it.toString() == level }

    issue.setCustomFieldValue(cfRiskLevel, value)
    def currentUser = ComponentAccessor.getJiraAuthenticationContext().getLoggedInUser()
    issueManager.updateIssue(currentUser, issue, EventDispatchOption.DO_NOT_DISPATCH, false);

    def issueIndexingService = ComponentAccessor.getComponent(IssueIndexingService)
    boolean wasIndexing = ImportUtils.isIndexIssues()
    ImportUtils.setIndexIssues(true)
    issueIndexingService.reIndex(issueManager.getIssueObject(issue.id))
    ImportUtils.setIndexIssues(wasIndexing)
    log.debug(value)
}

...

Upon creation, you'll be warned about configuring the context and screen, for choosing to which issue types it's applicable, and to the screens where you want it visible.

Image RemovedImage Added

Using it

As an example, you may use the previous field as a way to sort and filter Tests in the "Add Tests" dialog, whenever adding them from within an existing Test Execution.

Image RemovedImage Added

Calculating risk at requirement or project level and storing it in a Single Select field at Test level using a Script Listener 

...

As changes are made to probability or to impact on the parent issue (e.g. requirement, project risk,) a listener will handle the issue updated & created events and will update a "Select List (single choice)" based custom field defined for Test issues. That field needs to be created and configured beforehand will all the possible options.


Image RemovedImage Added




Info
titlePlease note

This approach and the code of following script assumes that a Test covers just one issue. If, for example, a given requirement is covered by a Test that also covers another requirement, then whenever editing the impact in one of the requirements will update/overwrite the calculated risk level on the custom field on the Test issue.

Thus, you may have to think if you want to follow this approach as it has some constraints.

...

In this example, we're calculating the risk as a number using a custom Script Field (Add-ons > ScriptRunner > Script Fields > Add New Item > Custom Script Field). 

Image Removed

Image Removed


Image Added    Image Added


Code Block
languagegroovy
titleRisk (Level) calculated as a number
collapsetrue
import com.atlassian.jira.issue.Issue
import com.atlassian.jira.issue.MutableIssue
import com.atlassian.jira.issue.IssueManager
import com.atlassian.jira.component.ComponentAccessor
import org.apache.log4j.Logger
import org.apache.log4j.Level
  
def log = Logger.getLogger("com.onresolve.jira.groovy")
log.setLevel(Level.DEBUG)

def issueManager = ComponentAccessor.issueManager

def impact =  getCustomFieldValue("Impact").toString()
def probability =  getCustomFieldValue("Probability").toString()

def risk

if ((impact!="null") && (probability!="null")) {
    risk = Double.parseDouble(impact) * Double.parseDouble(probability)
} else {
    risk = null
}

return risk

...

Info
titlePlease note

Even if risk value related custom field is created using the Template "Number Field", as shown earlier, sorting won't work as expected as it will be done by text comparison (i.e. alphabetically).

Image RemovedImage Added

You will need to to guarantee that Jira uses the "Number Searcher" by editing the custom field details. If you change this, don't forget to re-index.

Image RemovedImage Added


Calculating risk as text using a Script Field

...