Overview

Software quality gates are predefined checkpoints or criteria that must be met at various stages of the software development lifecycle to ensure that the software meets specific quality standards. Quality gates are used to assess the software product's quality, reliability, and readiness before proceeding to the next phase of development or deployment.

Jira (and Xray) can also be used to define Quality Gates (QGs) as in Jira we can define custom workflows or implement rules and conditions to define those QGs. Jira’s ability to integrate with other tools such as CI/CD tools also makes it a good candidate for QGs.

To have valuable Quality Gates in place you must consider some key strategies, such as defining a clear criteria, the ability to integrate with automated tools, the possibility to establish gatekeepers and make sure to have a practice of continuous refinement.

Prerequisites

Examples

In the below section we present possible implementations of Quality Gates using Jira and Xray data, in each example you will have information on how to define it and a final section on how you can follow the adherence using built-in reports from Xray.

We will showcase different possibilities of quality gates implementation focusing on data present in Jira by default and a combination of data present in Jira and Xray. There are a lot of possible quality gates that you can implement depending on what you want to follow, these are just two examples to showcase the possibilities using Jira and Xray.

Quality Gate 1 : All issues are covered by tests

Description

User Stories can only be closed if covered by tests.

Metric

The number of Tests covering the User Story. 

QG

Number of Tests covering the User Story > 0

Implementation

Given that your team works in sprints and has the following sprint board, we want to make sure that all User Stories that are Resolved must have Tests that are covering it.

In order to ensure that all User Stories have Tests covering them we add a Workflow Validator that will prevent the transition of the User Story to Resolved if there are no Tests associated with it.

Access “Project Settings -> Workflows” and edit the workflow by clicking the pencil icon below “Actions”

There we choose the transition that we want to add the Validator to by pressing on it, once we do it a pop up will appear where we can choose the Validators option.

We add a new Validator by adding a name to it and define a Jira Expression in the Script Validator window. Make sure you add a meaningful error message in the field: "Validation failed message", so that when the transition is not permitted users can understand why.

The Jira Expression used is:

issue.issueType.name.match('(Story)$') != null && issue.links.filter(link => link.type.name == 'Test').length > 0 

The above validation verifies if there is at least one linked issue of the type ‘Test’, if that condition is not true the issue will not be allowed to transition to the next stage.

Note: We can use an issue to "validate the validator" by using the Test Against -> Test option available. This allows you to confirm that the validator will work as expected.

Quality Gate 2 : All issues closed have no open bugs

Description

User Stories can only be closed if there are no open Bugs linked to it.

Metric

The number of opened Bugs linked to the User Story. 

QG

The number of opened Bugs linked to the User Story == 0

Implementation

Another example of a Quality Gate is to prevent closing Stories with open bugs linked to them.

To add a validator that will prevent a Story with open bugs to be closed we must access “Project Settings -> Workflows” and edit the workflow by clicking the pencil icon below “Actions”

There we choose the transition that we want to add the Validator to by pressing on it, once we do it a pop up will appear where we can choose the Validators option.

We add a new Validator by adding a name to it and define a Jira Expression in the Script Validator window. We can also add a meaningful error message so that when the transition is not permitted users can understand why.

The Jira Expression used for this Validator is:

if (issue.issueType.name.match('^(Story)$')!=null)
{ 
    if (issue.links.length > 0)
  	{
        issue.links.filter(link => (link.inwardIssue.issueType.name=='Bug') && (link.inwardIssue.status.name.match('Open|Reopened|Todo'))).length == 0
    }
  	else
  	{
		return true
  	}
}
else
{
	return true
}

This Jira Expression checks if a Story has links of the type Bug and if they are in the status Open, if so the transition will not be allowed. Notice that the message shown is the one defined above.

Note: We can use an issue to "validate the validator" by using the Test Against -> Test option available. This allows you to confirm that the validator will work as expected.

Track QGs with Jira and Xray

Jira Cloud with Xray has a series of possible reports available out of the box, those reports are also available in the form of dashboard gadgets that allow users to define custom dashboards.

From those, two reports allow you to assess if the Requirements are or are not being covered by tests:

• Overall Test Coverage (report that shows the Test coverage filtered by priority or another chosen filter)
• Requirement List (report that lists all requirements with their status)

Once added to a dashboard we can check if we have uncovered requirements and the status of test execution if tests cover them.

Note: You can customize these reports further by using JQL filters or using different options.

Tips

• Use the Test capabilities of the workflow validator to experiment if your rule is ok.
• Ensure you define an informative error message because it is the only information the user will see if a transition fails.
• Use <JIRA_ENDPOINT>rest/api/2/issue/<ISSUE_KEY>?expand=names to verify what fields are exposed and their ids.