---
title: "Test Environments"
canonical: "https://docs.getxray.app/space/XRAYCLOUD/44565129/Test%20Environments"
format: markdown
---
Table of Contents
> Macro (toc)
# Introduction
A Test Environment contains all necessary elements, including the system under test, for testing to be performed on it.
Depending on your context, a Test environment may represent:
- A Testing stage (e.g. "development", "staging", "preproduction", "production").
- A device model or device operating system (e.g. "Android", "iOS").
- An operating system (e.g. "Windows", "macOS", "Linux").
- A browser (e.g. "Edge", "Chrome", "Firefox").
The semantics of what a Test Environment represents depends on your specific context.
> ℹ️ In Xray, Test Environments are focused on the **execution** aspect, providing the means to schedule [Tests](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565156) and analyze their results in different environments.
> ℹ️
> ℹ️ They're explicitly associated with [Test Execution](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565117) Issues.
# Benefits of Using Test Environments
- Avoid duplication of Tests, whenever you have to run the same Test on different environments.
- Ability to track the latest status of Tests on different environments.
- Ability to track coverage on each environment.
- Ability to track overall [coverage](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565185), considering the coverage/results in each environment.
- Ability to perform [reporting](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44566291), including traceability, per each environment or globally (i.e. considering all results on all different environments).
# Test Environments in Xray at a Glance
The current [status of a Test](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44566467) is generally calculated by looking at the Test's last execution. However, this does not work well if you execute the same Test in two different Test Environments (e.g., devices) and you want the two results to be consolidated.
Within a Test Execution, you may specify the **Test Environment(s)** where the Tests will be executed in the respective attribute. A Test Environment is similar to a label, but Xray has special logic to deal with it.
If you use Test Environments, you may reuse the same Test for multiple Test Environments and create Test Executions for each one. If you don't use Test Environments and you want to track Tests for multiple environments (e.g., devices), then the only way to do it is to create multiple Tests, one per Test Environment.
Let's say that you have executions for two Test Environments: "Android" and "iOS". The Test is considered as PASSED only if the **latest** executions for Android and iOS are **both** PASSED; otherwise, it will be FAILED.
> Macro (gliffy)
Figure 1 - Tests
> ℹ️ If you don't use Test Environments, then only the latest result matters for assessing the current status of the Test.
## How it Works
Let's say we have a Test being executed two times. We will start by creating Test Executions TE1 and TE2.
TE2 will be executed after TE1, so TE1 is considered older.
The table below has examples of scenarios and it shows how the overall, **consolidated** **status** of the Test is calculated in each case.
If you have a mix of Test Executions (i.e., with and without Test Environment as in scenario D), it will be treated the same way as scenarios A, B, and C.
| **Scenario** | **Test Environment(s) of TE 1** | **Test Environment(s) of TE 2** | **Test run status in TE1** | **Test run status in TE2** | **The calculated value for the overall, consolidated status of the Test (i.e. for the "All Environments")** | **Other** |
| --- | --- | --- | --- | --- | --- | --- |
| **A** | Android | iOS | PASSED | PASSED | **PASSED** | The Test will be considered to be PASSED in both Android and iOS environments |
| **B** | iOS | iOS | PASSED | FAILED | **FAILED** | The Test will be considered to be FAILED in iOS |
| **C** | iOS | iOS | FAILED | PASSED | **PASSED** | The Test will be considered to be PASSED in iOS |
| **D** | iOS | - | FAILED | PASSED | **FAILED** | The Test will be considered to be FAILED in iOS and PASSED for the empty environment |
| **E** | - | - | PASSED | FAILED | **FAILED** | The Test will be considered to be FAILED for the empty environment |
| **F** | - | - | FAILED | PASSED | **PASSED** | The Test will be considered to be PASSED for the empty environment |
> ℹ️ The *empty* Test Environment is treated similarly to any other environment having a well-defined name.
# Operations
Whenever creating a Test Execution, you must set the Test Environment in which the execution will be executed. You can use this field as a simple label: just add the environment or reuse a previously created one.
Please see some important [Tips and Recommendations](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565129/Test+Environments#TestEnvironments-TipsandRecommendations) below.
## Creating a Test Execution
You can [create a Test Execution Issue](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565117) from the [Test Plan](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565153) and the Test Issue using actions provided by Xray. In this context, the Test Environment field (Figure 1; Figure 2 - 1) will be provided so that you can immediately specify a Test Environment.
## Tracking the Results on Different Environments
The Test Environments column is shown in your Test Runs table (Figure 3) so you can distinguish each execution of the Test between the different environments (Figure 3 - 1).
This information can be seen in the Test Issue screen (Figure 3) or in other places that show a list of Test Runs (e.g. Test Plan Issue screen).
> ℹ️ The same Test has been executed in both Test Environments (a Test Execution per Test Environment).
### Analyzing the Impact of the Results on Different Environments
Results obtained for Test Environments will impact coverage.
Considering Figure 3, the Test Status for the Test Issue will show** ****FAILED**** **because the Test has failed for one of the environments. On the Test Issue screen, a web panel named is provided by Xray that allows users to calculate the status of the Test considering different scopes.
If the *All Environments* option is selected for the Test Environment field (Figure 4 - 1), Xray will consider the latest Test Execution for each environment which, in this case, will result in** ****FAILED**** **(Figure 4 - 2).
If you want to analyze the Test status and show the latest results on a specific environment, click *Latest* (Figure 4 - 3). As seen below, this will produce different results because different results were obtained in different environments (Figure 5 - 1; Figure 6 - 1).
Please check [Coverage Analysis](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565185) to learn more about coverage analysis possibilities.
#### Requirement Issues
Just like we can calculate the status for different Test Environment scopes on the Test Issue, we can perform the same analysis on Requirement Issues. If you go into a Requirement Issue covered by Test cases, a web panel named Test Coverage will be visible on the Issue screen, allowing users to analyze the coverage status. This analysis also includes the Test Environment field. Hence, you can also see the status of the Requirement Issues based on the Test Environments in which the Tests were executed.
#### Traceability/Overall Coverage Report
It is possible to analyze Testing thoroughly considering Test Environments; this analysis can be done using the [Requirement Traceability Report](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565208) or the [Overall Coverage Report](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565335), among others.
The exact behavior upon choosing a specific Test Environment depends on the report itself but, either explicitly or implicitly, Test Runs will be filtered by the selected Test Environment and reports will reflect it.
##### Traceability Report being used to analyze the results of a Story for *All Environments* (Figure 7 - 1; 2)
##### Traceability Report being used to analyze the results of a Story for the *Chrome* Test Environment (Figure 8 - 1; 2)
##### Traceability Report being used to analyze the results of a Story for the *Firefox* Test Environment (Figure 9 - 1; 2)
##### Test Coverage Report being used to analyze the results of a Story for *All Environments* (Figure 10 - 1)
##### Test Coverage Report being used to analyze the results of a Story for the *Chrome* Test Environment (Figure 11 - 1)
##### Test Coverage Report being used to analyze the results of a Story for the *Firefox* Test Environment (Figure 12 - 1)
## Using Multiple Environments at the Same Time
Sometimes, you may have multiple categorizations for an environment; in theory, you can think of it as something multi-dimensional.
Consider a very basic example: whenever performing web/UI-based testing, you will be using a browser and an operating system, and you may want to analyze the results from a browser perspective or from an operating system perspective.
> ℹ️ The recommended way to deal with environments having multiple dimensions is to treat each dimension (e.g. browser name, operating system name) individually. In other words, add the values of each dimension to the Test Environments field separately (Figure 13 - 1). Once you're finished, click the tick icon (Figure 13 - 2).
> ℹ️
> ℹ️ 
Whenever you assign *macOS* and *Firefox* to the Test Environments of a Test Execution, it's equivalent to saying that your Test Run is scheduled for/was run in the *macOS* and also in the *Firefox* environment.
This approach will **limit the number of environments to the total number of possible values for each dimension**, as opposed to having ******... environments.
The **drawback** of this solution is that you won't be able to analyze the results for an environment tagged as *macOS* and *Firefox* at the same time, for example; you can just analyze results from a specific dimension.
> ℹ️ One way to deal with these kinds of environments would be to flatten them and treat them as usual, i.e. you could name the environment *Windows10_Chrome* or *macOS_Firefox*, but:
> ℹ️
> ℹ️ - You could have a ton of composed environments which wouldn’t be manageable.
> ℹ️ - You couldn’t analyze coverage just from the perspective of one of those variables (e.g. *macOS* or *Firefox*); you would be restricted to analyzing it from the perspective of the composed environment.
### Example
Test executed in the context of Test Execution assigned to several environments at the same time (Figure 14):
- *Windows10, Chrome* (FAILED; Figure 14 - 2).
- *Windows10, Firefox* (PASSED; Figure 14 - 1).
- *macOS, Firefox* (PASSED; Figure 14 - 3).
The calculated status of the Test, per Test environment, will be the following:
| **Environment** | **Status** | **Why?** |
| --- | --- | --- |
| **Windows10** | **PASSED** | Due to the last result obtained in the *Windows10* environment on STORE-792 |
| **macOS** | **PASSED** | Due to the last result obtained in the *macOS* environment on STORE-790 |
| **Firefox** | **PASSED** | Due to the last result obtained in the *Firefox* environment on STORE-792 |
| **Chrome** | **FAILED** | Due to the last result obtained in the *Chrome* environment on STORE-791 |
| **All Environments**
**(if analyzing the status of the Test without identifying a specific environment)** | **FAILED** | as the last result for one of the environments (*Chrome*) was tagged as **FAILED** (i.e. on STORE-791) |
# Tips and Recommendations
## Do's
- Use Test Environments only if you want to run the **same** Test case in different environments and track results per each environment.
- Evaluate if you really need to assign multiple environments at the same time; using just one is preferable if you can afford that simplicity.
## Don'ts
- Don't create dozens or hundreds of Test Environments as it will harden their usage and add some performance overhead.
- Don’t use composed environment names, such as *__* as it will pollute the environment namespace and harden management.
- Don't try to do data-driven Testing using Test Environments; they're not tailored for that.
Support/Troubleshooting
If you have questions or technical issues, please [contact the Support team via the Customer Portal (Jira service management)](https://jira.getxray.app/servicedesk/customer/portal/2/user/login?destination=portal%2F2%2Fcreate%2F28) or [send us a message using the in-app chat](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44577312).