Overview

In Agile Methodologies, Acceptance Criteria (AC) are used to define when a given user story can be "accepted" by some relevant stakeholder.

The criteria that a component or system must satisfy in order to be accepted by a user, customer, or other authorized entity. - ISTQB Glossary

Acceptance Criteria:

clarify some aspects of the underlying story that are relevant to someone;
can be checked and validated by testing; they should be clear enough so that clear accept/don't accept decisions can be taken.

But the following question arises: How can we know whether a certain criterion has been checked or validated by testing?

Before diving into it, we need to be aware of typical patterns teams follow for handling AC.

"Accepting" the story doesn't mean it's bug-free or that acceptance criteria should describe all possible usage scenarios about the story; it's only a way to clarify some usage rules that are important for some stakeholders and that the user story "can work".

A common misconception is on "done" vs "acceptance criteria". Acceptance criteria are user story specific, while the definition of done applies to all product increments/user stories.

Therefore the fact that acceptance criteria have been met, doesn't mean it can be considered as "done" and thus ready to be shipped.

Jira and Acceptance Criteria

Jira doesn't enforce a way of handling AC.

Usually, teams describe AC as a list in the Description field of the related Story.

Xray Cloud Documentation > Implementing Acceptance Criteria with Xray and Jira > image2021-9-27_11-9-2.png

AC can be enumerated using bullets: either numbered bullets or standard ones but with some prefix.

This is an unstructured approach that inhibits full traceability between story<=>AC<=>tests<=>test runs<=>defects.

Some teams may decide to use specific custom fields for this purpose or even to use specific Jira apps/plugins to clearly manage AC. However, these usually don't have unique IDs that can be referred elsewhere.

Xray and Acceptance Criteria

Xray does not provide nor enforce a way of handling AC. But, you can take advantage of some built-in capabilities to have more explicit visibility of it.

While most teams create Tests and associate them to the related requirements/stories (e.g., Story<=>Test(s)), this may not be sufficient if you want to analyze each criterion individually.

If you aim to understand which Tests address a specific acceptance criterion and how it is based on the related testing results, then we need to turn each acceptance criterion into a manageable, identifiable, and coverable entity.

That will allow us to have more detailed traceability, such as Epic<>Story/ies<>Acceptance Criteria<>Test(s), and track the coverage status up to the Acceptance Criteria level.

Having explicit visibility of Acceptance Criteria and tracking their status with Xray

AC implemented as "sub-requirements" in Xray

Xray supports the concept of requirements and sub-requirements.

In this approach, a given story/requirement is decomposed into multiple sub-requirements, which in turn can be covered by tests.

The sub-requirement, in this case, is our acceptance criterion that will be implemented as an issue.

We can define the relation between the Story and related acceptance criteria based on a sub-task or on a specific issue link type, accordingly to what we have defined on Xray coverage-related settings.

Configuring it

Create an issue type named "Acceptance Criterion" or equivalent. Go to Jira Settings > Issues > Issue Types.
Add that issue type to the issue type scheme used by your Jira project
Define in Xray settings, the sub-requirement (e.g. the "Acceptance Criterion" issue) to be handled as a coverable issue (i.e. by adding it to the "Requirement Issue Types" column)
Define in Xray settings, the relation between the parent requirement (e.g. the "Story" issue) and the sub-requirement (e.g. the "Acceptance Criterion" issue)
1. Note: if you want to define this relation based on a custom issue link type (e.g., you could create a "Satisfy: satisfies/is satisfied by"), then you need to create it first

In Xray, sub-requirements can be identified and related to parent requirements using:

a specific, configured, issue link (e.g., you could create a "Satisfy: satisfies/is satisfied by")
sub-tasks

Using sub-tasks may allow further control over the progress of the parent issue (e.g. the Story), such as giving the ability to restrict closing it or not. However, the usage of "sub-tasks" namely for this purpose, is something that has to be evaluated in your specific context.

To use sub-tasks:

Please make sure they are enabled. Go to Jira Settings > Issues > Issue types > Sub-tasks.
Create the sub-requirement/acceptance criterion as a sub-task issue type. Go to Jira Settings > Issues > Issue Types.
In Xray settings, make sure the Issue-Sub-tasks relationship is defined for test coverage hierarchy

Using it

Create the Story as usual. The story may be part of an Epic or not.

Xray Cloud Documentation > Implementing Acceptance Criteria with Xray and Jira > image2021-9-10_16-25-24.png

Then create the Acceptance Criterion issues. The exact details depend on how you decided to implement the AC and relate them to the Story issues:

If using a specific issue link type to relate the AC to the Story, then:
- Create the Acceptance Criterion issue and link the previous issue to the Story using the configured issue link type:

Xray Cloud Documentation > Implementing Acceptance Criteria with Xray and Jira > image2021-9-10_17-1-34.png

If using sub-tasks as a way to relate the AC to the Story, then proceed as follows:

Xray Cloud Documentation > Implementing Acceptance Criteria with Xray and Jira > image2021-9-10_16-28-59.png

From the Acceptance Criterion issue create Tests and execute them as usual, reporting the final status (e.g., PASSED/FAILED)

Xray Cloud Documentation > Implementing Acceptance Criteria with Xray and Jira > image2021-9-10_16-32-22.png

You can then check coverage directly on the Story, which is based on all related testing results (i.e., results from Tests covering all the related Acceptance Criterion issues).

Xray Cloud Documentation > Implementing Acceptance Criteria with Xray and Jira > image2021-9-10_16-39-28.png

You can also go to each Acceptance Criterion issue to understand if it's being satisfied or not, based on the coverage information, which in turn is based on the related testing results. In this example, one of the AC (CALC-1305) is NOK.

Xray Cloud Documentation > Implementing Acceptance Criteria with Xray and Jira > image2021-9-10_16-35-59.png

In the Traceability Report, it's possible to see full traceability between Epic<=>Stories<=>Acceptance Criteria<=>Tests<=>Test Runs<=>Defects. The following screenshot clearly shows this hierarchy:

Xray Cloud Documentation > Implementing Acceptance Criteria with Xray and Jira > image2021-9-10_16-47-21.png

Supported features

	Supported?
create one or more AC per Story/requirement	x
uniquely identify each AC	x
ability to create linkable references to AC (.e.g ability to mention AC from somewhere else)	x
AC as issues	x
see AC on the Story/requirement issue screen along with their status	x
link Tests to each AC	x
"ignore" AC without Tests for the calculation of coverage	x
full traceability story/requirement<=>AC<=>tests<=>test runs<=>defects	x

FAQ

Do we have to implement AC as issues?
- No. Only if you wish to track coverage status (i.e., their acceptance information) on a per acceptance criterion basis.
Can a Test be considered a form of AC?
- A test is a concrete example of a certain usage of the requirement/story, so to an extent it can be seen as a sort of acceptance criteria. However, usually tests are one level below AC and contain fine details.
How can we implement Acceptance Criteria whenever adopting Gherkin (e.g. "Cucumber") to support our BDD endeavour?
- Recent versions of Cucumber have support for the Rule entity, which is an acceptance criterion. The Rule sits between the Feature and the Scenarios. Each Scenario represents an example (and thus Scenario have been renamed to Example) of a given Rule. However, even if you're not adopting a recent version of Gherkin supporting the Rule keyword, your Scenario/Scenario Outline can represent AC; it all depends on how you use them. Xray doesn't yet support the Rule keyword.
Are there other ways of implementing AC in Jira? How? Can they be used together with Xray?
- You can use your own way to implement AC, such as having a dedicated custom field on the Story. You can also decide to use a Jira app/plugin (e.g., Checklist for Jira, Smart Checklist for Jira) for that purpose. All those are valid approaches but keep in mind that if you want to track the coverage status of each acceptance criterion defined that way, as these are not entities that Xray has visibility of, then you won't be able to do so.