Page History
...
| Info |
|---|
"Test Environment" is a concept introduced with introduced in Xray v2. You may want to have a look at a Here is a short video which explains the concept of Test Environments and how you can use them. |
Overview
A Test "status" The status of a Test (i.e., TestRunStatus) is generally calculated by looking at the testTest's last execution (more detail here). But However, this does not work well if you execute the same test Test in two different test environments (e.g., devices) and you want the status two results to be like a "merge" of the two results.consolidated.
Within a Test Execution, you may specify the Test Environment(s) where the tests will be executed in the respective attribute. A Test Environment is similar to a label, but Xray provides some has special logic to deal with it.
If you use Test Environments, you may reuse the same Test for multiple test environments and create Test Executions for each one with the same Test. If you don't use Test Environments and you want to track tests for multiple environments (e.g., devices), then the only way you have to do it is to create multiple tests, one per each test environment.
Let's say that you have executions for two Test Environments: "Android" and "iOS". The test is considered as PASS only if the latest executions for Android and iOS are iOS are both PASS; otherwise, it will be FAIL if any of them has failed.
| Gliffy Diagram | ||||
|---|---|---|---|---|
|
...
| Info |
|---|
If you start using Test Environments in your Test Executions, then it's not only your test status calculation that will be changedchange. All custom fields that depend on it (e.g., Requirement status, Test Set sstatus) will be affected accordingly.ThereforeSets status) will change. Consequently, the requirement coverage calculation and the related and all associated charts/gadgets are also affected. |
How it works
Let's consider that say we have some a Test being executed multiple times ( two times). So, we We will start by creating the respective Test Executions for that: TE1 and TE2.
TE2 will be executed after TE1, so TE1 is considered older.
Let's see bellow some Below are example scenarios and shows how calculated value for TestRunStatus.the TestRunStatus is calculated in each.
If You may notice that in scenario D, if you have a mix of Test Executions (i.e., with and without Test Environments; in that case Environment as in scenario D), it wil be treated similar to the same way as scenarios A, B and C.
| Scenario | Test Environment(s) of TE 1 | Test Environment(s) of TE 2 | Test run status in TE1 | Test run status in TE2 | Calculated value of "TestRunStatus" custom field |
|---|---|---|---|---|---|
A | Android | iOS | PASS | PASS | PASS |
B | iOS | iOS | PASS | FAIL | FAIL |
C | iOS | iOS | FAIL | PASS | FAIL |
D | iOS | - | FAIL | PASS | FAIL |
E | - | - | PASS | FAIL | FAIL |
F | - | - | FAIL | PASS | PASS |
...
When creating a Test Execution, you must set the Test Environment in which the execution will be executed on. You can use this field as a simple label: just add the environment or reuse a previously created one.
...
Test Execution for “iOS” Test Environment
You can see the The Test Environments column is shown in your Test Runs table so you can distinguish each execution of the Test between the different environments.
The same Test has been executed in both Test Environments (a Test Execution per Test Environment).
Now, let’s check our The Requirement Status : will see that it’s show NOK because the Test has failed for one of the environments.
...
Tips
- Use Test Environments only if you really need towant to run the same Test on different environments.
- If you do use it, then preferably assign just one Test Environment per Test Execution.
- You may filter using Test Environments in your Test Executions panel to see how the executions are doing, per environmentAvoid cloning Tests if you want to run them on different environments, that's the reason behind Test Environments so maybe it's best to use them instead.