Page History
...
Thus, semantics of what a "Test Environment" represents depends on your specific context.
Some tests may be tailored for or only run in some environments.
| Info | ||
|---|---|---|
| ||
In Xray, Test Environments are focused on the execution aspect, providing the means to schedule tests and analyze their results in different environments. Thus, they're explicitly associated with Test Execution issues. |
...
Test Environments in Xray at a glance
The current status of a Test is generally calculated by looking at the Test's last execution (more detail here). However, this does not work well if you execute the same Test in two different test environments (e.g., devices) and you want the two results to be consolidated.
...
Note: If you don't use Test Environments, then only the latest result matters for assessing the current status of the Test.
...
Test
...
.
How it works
Let's say we have a Test being executed two times. We will start by creating Test Executions TE1 and TE2.
...
Below are example scenarios and shows how the overall, consolidated status of the Test is calculated in each case.
If you have a mix of Test Executions (i.e., with and without Test Environment as in scenario D), it will be treated the same way as scenarios A, B, and C.
| Scenario | Test Environment(s) of TE 1 | Test Environment(s) of TE 2 | Test run status in TE1 | Test run status in TE2 | Calculated value for the overall, consolidated status of the Test (i.e. for the "All Environments") | Other |
|---|---|---|---|---|---|---|
A | Android | iOS | PASS | PASS | PASS | The test will be considered to be PASS in both Android and iOS environments. |
B | iOS | iOS | PASS | FAIL | FAIL | The test will be considered to be FAIL in iOS. |
C | iOS | iOS | FAIL | PASS | PASS | The test will be considered to be FAIL in iOS. |
D | iOS | - | FAIL | PASS | FAIL | The test will be considered to be FAIL in iOS and PASS for the empty environment. |
E | - | - | PASS | FAIL | FAIL | The test will be considered to be FAIL for the empty environment. |
F | - | - | FAIL | PASS | PASS | The test will be considered to be PASS for the empty environment. |
| Info | ||
|---|---|---|
| ||
The empty Test Environment is treated similarly to any other environment having a well-defined name. |
How to use
When Whenever creating a Test Execution, you must set the Test Environment in which the execution will be executed. You can use this field as a simple label: just add the environment or reuse a previously created one.
Please see Tips and Recommendations ahead.
Test Execution for “Android” Test Environment
...
The Test Environments column is shown in your Test Runs table so you can distinguish each execution of the Test between the different environments.
The same Test test has been executed in both Test Environments (a Test Execution per Test Environment).
The Considering the previous screenshot, the Requirement Status will show NOK because the Test has failed for one of the environments.
...
Internally, this field will store the status of the test for all possible scopes, which besides other things includes the information about the status in all different environments.
| Info |
|---|
If you start using Test Environments in your Test Executions, then it's not only your test status calculation that will change (i.e. the one stored in the TestRunStatus custom field). All custom fields that depend on it (e.g., Requirement status, Test Sets status) will change. Consequently, the requirement coverage calculation and all associated charts/gadgets are also affected. |