Page History
Xray provides useful and detailed information about the Requirement Requirements Coverage in every project trough a project tab panel named "via the Requirement Coverage Charts" tab panel and also the Requirement Status Custom Field custom field.
| Table of Contents |
|---|
Requirement Coverage Status
The Requirement issue coverage status is calculated based on all Tests that are associated with the Requirement and all the Sub-Requirements Status. The following list details the coverage status of a requirement:
- OK - All the Tests associated with the Requirement are PASSED
- NOK - At least one Test associated with the Requirement is FAILED
- NOT RUN - At least one Test associated with the Requirement is TODO or ABORTED and there are no Tests with status FAILED
- UNKNOWN - At least one Test associated with the Requirement is UNKNOWN and there are no Tests with status FAILED
- UNCOVERED - The Requirement has no Tests associated with it
The Status of a Test for a particular Requirement is calculated based in the Requirement Coverage Strategy.
Requirement conjunction
| REQ ∧ SUB-REQ | OK | NOK | NOT RUN | UNKNOWN | UNCOVERED |
|---|---|---|---|---|---|
| OK | OK | NOK | NOT RUN | UNKNOWN | OK |
| NOK | NOK | NOK | NOK | NOK | NOK |
| NOT RUN | NOT RUN | NOK | NOT RUN | UNKNOWN | NOT RUN |
| UNKNOWN | UNKNOWN | NOK | UNKNOWN | UNKNOWN | UNKNOWN |
| UNCOVERED | OK | NOK | NOT RUN | UNKNOWN | UNCOVERED |
| Info | ||
|---|---|---|
| ||
Even if you have sub-requirements, when you have tests that are directly linked to the parent requirement, Xray considers Xray assumes that you are validating the requirement directly. Thus, thus it's irrelevant if the sub-requirements are uncovered by tests. |
Ignore Requirements with statuses
Requirement issues with certain statuses can be excluded from the Requirement Coverage charts. These statuses can be configured in the Xray configuration.
Ignore Requirements with labels
Requirement issues with certain labels can be excluded from the Requirement Coverage charts. These labels can be configured in the Xray configuration.
Ignore Tests with statuses
Test issues with certain statuses can be excluded from the Requirement Coverage calculation. Test issues with ignored statuses will not be considered when calculating in the calculation of the Requirement Status, even if they are associated with Requirement issues. These statuses can be configured in the Xray configuration.
Requirement Test Coverage Section
The Requirement Status and the Test Coverage are presented in the Requirement issue view.
...
Analysis by version
You can choose a specific analysis version to calculate the Requirement Status. This can be useful when testing multiple versions of your requirements at the same time or if you need to see the requirement coverage for previous versions. The analysis version can be " No Version". In this case, , in which case the requirement status will be calculated based in on the latest execution for each Test, independently regardless of the execution version.
Analysis by Test Plan
You can also choose a specific Test Plan to calculate the Requirement Status. This will calculate the requirement status based on the executions for that Test planPlan. This can be useful when you have multiple test plans and need to see the requirement coverage for a specific Test Plan.
Requirement Coverage Charts
The "Requirement Coverage Charts" This is divided in two reports:
- The Overall Coverage Chart that provides an overall view of the current Requirement coverage of a project for a particular Version.
- The Daily Historical Requirement Coverage that contains a comprehensive stacked bar chart for seeing to see the evolution of the requirement coverage over time for a particular Version.
...
The requirement coverage reports provide allow you to analyse the test status of requirement issues in the current project and have na , provide an overview of how the many requirements are OK, NOK, Not Run and Uncovered.These reports will also give you , and an overview of the requirement status by version or test plan.
Analysis by Version
You can choose a specific analysis version to generate the report. This will calculate the requirement status based on executions for that version. This can be useful when testing multiple versions of your requirements at the same time or if you need to see the requirement coverage for previous versions. The analysis version can be "None". In this case, None, in which case the requirement status will be calculated based in the latest execution for each Test, independently regardless of the execution version.
Analysis by Test Plan
You can also choose a specific Test Plan to generate the report. This will calculate the requirement status based on the executions for that Test planPlan. This can be useful when you have multiple test plans and need to see the requirement coverage for a specific Test Plan.
Analysis by Environment
Besides being able to analyse the coverage status by Version and Test Plan, Xray also allows you to specify a Test Environment to calculate the status for of requirement issues. You can specify an Environment when using both Version or Test Plan analysis. In case where If you want to aggregate the status for all environments, just choose the "All Environments" option. If you choose a specific environment, then Xray will only consider the Test Execution issues that are scoped by within this environment when calculating the Requirement Status.
Grouping Results
You can also choose a grouping field to group requirements within these charts. You can, for instance, generate a report based on requirement Priority, Resolution, or any other field of type "Select List" that you have configured for your requirement issues.
Requirement Filters
In the Requirement Coverage charts, you can filter requirements to show in the charts either using . Use the basic filter fields provided:
- Fix Version (you can also choose whether to include the previous versions -- equivalent in JQL to fixVersion <= X. The default value for this option can be changed in the Xray configuration.)
- Workflow Status
- Resolution
- Assignee
- Key or Summary
- Component
...
There is also an options menu where you can choose the visualisation visualization type for the charts:
- hierarchical - only the parent requirement issues will be presented in the charts. Sub-requirements can still be visualised visualized in the overall details table.
- flatten - the requirement issues will not consider the parent - /child relationrelationship. All parent and child requirements will be considered for the chart.
Requirement Status Custom Field
The Requirement Status custom field is created when Xray is installed and associated with the JIRA Jira Default Issue Screen. This custom field is only shown for a particular issue if it's its issue type is defined as a Requirement in the Xray configuration.
The Requirement Status displays a color bar reflecting the current Status of the Requirement coverage. When hovering you hover over this bar, a detail bar appears showing all the tests for the Requirement and their current status. The number of Tests in each status is also displayed and can be clicked to show the Test issues in the issue search page.
Because the Status status of a Requirement can depend of on the Version, there are 3 possible strategies you can choose that affect the versions displayed by this fieldfrom the following strategies. These strategies can be changed in the Xray configuration.
No Version
When this option is enabled, the status of each Test associated with the Requirement is calculated based in the latest TestRun, independently on Test Run, regardless of the version it was executed.
Earliest Unreleased Version
When this option is enabled, only the earliest unreleased Fix Version is displayed in the Custom Field.
Unreleased Versions
When this option is enabled, all unreleased Fix Versions are displayed in the Custom Field.
Assigned Versions
When this option is enabled, only explicitly assigned Fix Versions are displayed in the Custom Field.
Requirement Coverage Strategy
The requirement coverage strategy defines how the Requirement Status must be calculated. The strategy to use must be configured in the Xray configuration.
Use versioned Test Executions for Requirement Coverage
When the option for "Use versioned Test Executions for Requirement Coverage" this option is enabled in Xray configuration, the Status of a Requirement for a specific Fix Version V is calculated based on the latest Test Run with Fix Version V of each associated Test (check how the latest Test Run is chosen). With this strategy, a Requirement can have different Statuses statuses, one for each Fix Version in the project.
Use versioned Test Sets for Requirement Coverage
When the option for "Use versioned Test Sets for Requirement Coverage" this option is enabled in the Xray configuration, the requirement coverage will be calculated based in on the Test Sets.
The main goal of this option is to use Test Sets as containers for tests of different versions related with to the same requirement. This allows for regression Test Sets in subsequent versions.
...
Once Test T is considered for coverage in version V, it's Status is its status is calculated based on the latest Test Run with the specified Fix Version V .(check how the latest Test Run is chosen).
Troubleshooting
"I have an Xray Project to Test my Requirements project. I've created a Test and I have already executed the Test, but I always get requirements status Covered but Not Run."
...
| Warning |
|---|
You might be using environments on Test Executions that are affecting the aggregated status for the Tests. If you have a couple of Test Executions, one with a specific environment (ex: e.g., Android) and another without any environment, then Xray will also consider the empty environment for environment when calculating the aggregated status (this is, of course, if all Test Executions are in within the same scope of Version or Test Plan). |
| Warning |
|---|
You have not configured your versions accordingly. Remember that requirement coverage through are requirements coverage is calculated with Test Execution and/or Test Set for a Requirement Fix version. |
...
| Tip |
|---|
You must have the same versions version name between projects. You must guarantee you have set The Test Execution fix version with must have the same value of value as the Requirement Issue fix version. |