"Test Environment" is a concept introduced with Xray v2.
You may want to have a look at a short video which explains the concept of Test Environments and how you can use them.
Overview
A Test "status" (i.e. TestRunStatus) is generally calculated by looking at the test's last execution (more detail here). But this does not work well if you execute the same test in two different test environments (e.g. devices) and you want the status to be like a "merge" of the two results.
Within a Test Execution you may specify the Test Environment(s) where the tests will be executed in the respective attribute. A Test Environment is similar to a label, but Xray provides some special logic to deal with it.
If you use Test Environments, you may reuse the same Test for multiple test environments and create Test Executions for each one with the same Test. If you don't use Test Environments and you want to track tests for multiple environments (e.g. devices) then the only way you have is to create multiple tests, one per each test environment.
Let's say that you have executions for two Test Environments: "Android" and "iOS". The test is considered as PASS only if the latest executions for Android and iOS are both PASS; it will be FAIL if any of them has failed.
Note: If you don't use Test Environments, then the TestRunStatus is calculated as usual.
If you start using Test Environments in your Test Executions then it's not only your test status calculation that will be changed. All custom fields that depend on it (e.g. Requirement status, Test Set sstatus) will be affected accordingly.
Therefore, the requirement coverage calculation and the related charts/gadgets are also affected.
How it works
Let's consider that we have some Test being executed multiple times (two times). So, we will start by creating the respective Test Executions for that: TE1 and TE2.
TE2 will be executed after TE1, so TE1 is older.
Let's see bellow some example scenarios and how calculated value for TestRunStatus.
You may notice that in scenario D, if you have a mix of Test Executions with and without Test Environments; in that case it wil be treated similar to scenarios A, B and C.
| Scenario | Test Environment(s) of TE 1 | Test Environment(s) of TE 2 | Test run status in TE1 | Test run status in TE2 | Calculated value of "TestRunStatus" custom field |
|---|---|---|---|---|---|
A | Android | iOS | PASS | PASS | PASS |
B | iOS | iOS | PASS | FAIL | FAIL |
C | iOS | iOS | FAIL | PASS | FAIL |
D | iOS | - | FAIL | PASS | FAIL |
E | - | - | PASS | FAIL | FAIL |
F | - | - | FAIL | PASS | PASS |
Example
When creating a Test Execution, you must set the Test Environment in which the execution will be executed on. You can use this field as a simple label: just add the environment or reuse a previously created one.
Test Execution for “Android” Test Environment
Test Execution for “iOS” Test Environment
You can see the Test Environments column in your Test Runs table so you can distinguish each execution of the Test between the different environments.
The Test has been executed in both Test Environments (a Test Execution per Test Environment).
Now, let’s check our Requirement Status: will see that it’s NOK because the Test has failed for one of the environments.
Tips
- Use Test Environments if you really need to
- If you do use, then preferably assign just one Test Environment per Test Execution
- You may filter Test Environments in your Test Executions panel to see how the executions are doing, per environment
- Avoid cloning Tests if you want to run them on different environments, that's the reason behind Test Environments so maybe it's best to use them instead