Page History
| Anchor | ||||
|---|---|---|---|---|
|
| Table of Contents |
|---|
| Info | ||
|---|---|---|
| ||
Please read Understanding the calculation of coverage status and the status of Tests for an in-depth explanation of the calculation of these statuses. |
Analysis Scopes
Tests and coverable issues can be analyzed from different perspectives/scopes.
The same Test or Story can be analyzed, for example, on version 1.0 and also on version 2.0. This will take into account the respective executions made for those versions.
Therefore, a Story may be OK on version 1.0, but may be NOK on version 2.0 due to regression.
Tests and coverable issues can be analyzed from these scopes (or dimensions):
- Latest
- Version
- Test Plan
They can also be analyzed based with some additional criteria that will affect the calculated values, such as:
- Test Environment
- Final statuses precedence over non-final ones
Latest
If you don't care about versions, or you're not using versions at all and just want to see the calculated statuses based on the latest runs, analysis by "Latest" can be used for that purpose.
This is useful to have a quick view of the latest results or calculated status for the Test or coverable issue.
Note that when analyzing by latest results, it also considers the latest results made in the different environments (i.e., when "All Environments" is selected as Test Environment).
Version
The "Version" scope allows users to analyze Tests and coverable issues from a version perspective. It helps address such questions as
- "How is the requirement on version X?"
- "How are these Tests performing on version Y?"
When analyzing by version, only the Test Executions made for the given version are considered (i.e., the Test Execution's Fix Version field).
As an example, a user story aimed for version 3.0 may be analyzed from the point of view of the executions made for version 3.0 or for the ones made afterwards on version 4.0.
Test Plan
This gives you the ability to evaluate the Test status or coverage status based on some planned testing (i.e., on the Tests and related executions made in the scope of the selected Test Plan).
When analyzing by Test Plan, only the Test Executions linked to that Test Plan are considered.
This allows you to evaluate if a given coverable issue is covered by the Tests of some Test Plan or not. If so, then you can see how is it going based on the executions performed from the related planned Test Executions.
Test Environment
Analysis by Test Environment gives the ability to analyze the Test status or the coverage status of an issue for some Test Environment.
It addresses such questions as "How is this requirement doing on environment X?"
When analyzing by Test Environment, only Test Executions made for the given Test Environment are considered.
Final Statuses Precedence
Final statuses precedence is used to perform the analysis based on "finished work" (non-intermediate Test Runs).
The flag "Final statuses have precedence over non-final statuses" gives the additional ability to consider just those Test Runs whose status is configured as being a final status.
This helps address questions such as:
- "What is the current status of this requirement (or test)?" (if final statuses precedence is unchecked)
- "What is the status of this requirement (or test), considering just the finished work?" (if final statuses precedence is checked)
Analyzing Xray entities
Xray entities can be analyzed in the issue view screen as well as in specific reports containing those entities.
Coverable issues
The status of coverable issues can be evaluated directly in the issue view screen as well as on some reports, including the Tests Coverage report.
Issue screen
Within the issue screen, the coverage status can be evaluated for the specified scope within the Test Coverage section. The calculated coverage status is shown on the right side.
Reports
Test Coverage
Coverable issues can be analyzed for a given scope using the Test Coverage report. More info about this report in Test Coverage Report. The chosen scope will be store in the user preferences, and will affect all issues in the same project.
Tests
The status of Tests can be evaluated directly in the Test issue view screen as well as on some reports, including the Tests List report and, indirectly, the Test Sets List report.
Tests can also be evaluated in the coverable issue screen, within the "Test Coverage" section.
Issue screen
Within the Test issue screen, the Test status can be evaluated for the specified scope. The calculated status is cached for performance reasons. Since data between Jira and Xray may not be in sync, it's possible to enforce a recalculation to display an up-to-date value. The chosen scope will be store in the user preferences, and will affect all issues in the same project.
Reports
Tests List
The Tests can be analyzed for a given scope using the Tests List report.
Test Sets
The status of Test Sets (i.e., of the Tests contained within a Test Set) can be analyzed using the Test Sets List report.
Reports
Tests Sets List
The Test Sets, and implicitly the Tests within them, can be analyzed for a given scope using the Test Sets List report.
Xray provides useful and detailed information about the Requirements Coverage in every project via the Requirement Coverage Charts tab panel and also the Requirement Status custom field.
| Table of Contents |
|---|
Requirement Coverage Status
The Requirement issue coverage status is calculated based on all Tests that are associated with the Requirement and all the Sub-Requirements Status.
- OK - All the Tests associated with the Requirement PASSED
- NOK - At least one Test associated with the Requirement FAILED
- NOT RUN - At least one Test associated with the Requirement is TODO or ABORTED and there are no Tests with status FAILED
- UNKNOWN - At least one Test associated with the Requirement is UNKNOWN and there are no Tests with status FAILED
- UNCOVERED - The Requirement has no Tests associated with it
The Status of a Test for a particular Requirement is calculated based in the Requirement Coverage Strategy.
Requirement conjunction
...
| Info | ||
|---|---|---|
| ||
Even if you have sub-requirements, when you have tests that are directly linked to the parent requirement, Xray assumes that you are validating the requirement directly. Thus, it's irrelevant if the sub-requirements are uncovered by tests. |
Ignore Requirements with statuses
Requirement issues with certain statuses can be excluded from the Requirement Coverage charts. These statuses can be configured in the Xray configuration.
Ignore Requirements with labels
Requirement issues with certain labels can be excluded from the Requirement Coverage charts. These labels can be configured in the Xray configuration.
Ignore Tests with statuses
Test issues with certain statuses can be excluded from the Requirement Coverage calculation. Test issues with ignored statuses will not be considered in the calculation of the Requirement Status, even if they are associated with Requirement issues. These statuses can be configured in the Xray configuration.
Requirement Test Coverage Section
The Requirement Status and the Test Coverage are presented in the Requirement issue view.
Analysis by version
You can choose a specific analysis version to calculate the Requirement Status. This can be useful when testing multiple versions of your requirements at the same time or if you need to see the requirement coverage for previous versions. The analysis version can be No Version, in which case the requirement status will be calculated based on the latest execution for each Test, regardless of the execution version.
Analysis by Test Plan
You can also choose a specific Test Plan to calculate the Requirement Status. This will calculate the requirement status based on the executions for that Test Plan. This can be useful when you have multiple test plans and need to see the requirement coverage for a specific Test Plan.
Requirement Coverage Charts
This is divided in two reports:
- The Test Coverage Report - TBD provides an overall view of the current Requirement coverage of a project for a particular Version.
- The Daily Historical Requirement Coverage contains a comprehensive stacked bar chart to see the evolution of the requirement coverage over time for a particular Version.
The requirement coverage reports allow you to analyse the test status of requirement issues in the current project, provide an overview of how many requirements are OK, NOK, Not Run and Uncovered, and an overview of the requirement status by version or test plan.
Analysis by Version
You can choose a specific analysis version to generate the report. This will calculate the requirement status based on executions for that version. This can be useful when testing multiple versions of your requirements at the same time or if you need to see the requirement coverage for previous versions. The analysis version can be None, in which case the requirement status will be calculated based in the latest execution for each Test, regardless of the execution version.
Analysis by Test Plan
You can also choose a specific Test Plan to generate the report. This will calculate the requirement status based on the executions for that Test Plan. This can be useful when you have multiple test plans and need to see the requirement coverage for a specific Test Plan.
Analysis by Environment
Xray also allows you to specify a Test Environment to calculate the status of requirement issues. You can specify an Environment when using both Version or Test Plan analysis. If you want to aggregate the status for all environments, just choose the All Environments option. If you choose a specific environment, then Xray will only consider the Test Execution issues that are within this environment when calculating the Requirement Status.
Grouping Results
You can also choose a grouping field to group requirements within these charts. You can, for instance, generate a report based on requirement Priority, Resolution, or any other field of type "Select List" that you have configured for your requirement issues.
Requirement Filters
In the Requirement Coverage charts, you can filter requirements to show in the charts. Use the basic filter fields:
- Fix Version (you can also choose whether to include the previous versions -- equivalent in JQL to fixVersion <= X. The default value for this option can be changed in the Xray configuration.)
- Workflow Status
- Resolution
- Assignee
- Key or Summary
- Component
or use a previously "Saved Filter" with Requirement issues.
Each time a user accesses the "Requirement Coverage" project page, the chart will be generated with the user's last chosen options. This includes the report and all the filters for the requirement issues.
There is also an options menu where you can choose the visualization type for the charts:
...
Troubleshooting
"I have an Xray Project to Test my Requirements project. I've created a Test and I have already executed the Test, but I always get requirements Requirements status Covered but Not Run."
| Warning |
|---|
You might be using environments on Test Executions that are affecting the aggregated status for the Tests. If you have a couple of Test Executions, one with a specific environment (e.g., Android) and another without any environment, then Xray will also consider the empty environment when calculating the aggregated status (this is, of course, if all Test Executions are within the same Version or Test Plan). |
| Warning |
|---|
You probably have not configured your versions accordinglyproperly. Remember that requirements coverage is calculated with Test Execution and/or Test Set for a Requirement Fix version. |
| Tip |
|---|
You must have the same version name between projects. The Test Execution fix version must have the same value as the Requirement Issue fix version. |
...
coverage status of some issue for some version is calculated based on the Fix Version associated with the Test Execution issues. |