Page History
Info |
---|
"Test Environment" is a concept introduced in Xray v2. Here is a short video that explains Test Environments and how you can use them. |
Table of Contents |
---|
What is a test environment
Generically speaking, a test environment is an environment that contains all necessary elements, including the SUT, so you can perform testing on it.
Depending on your context, a test environment may represent:
- a testing stage (e.g. "development", "staging", "preproduction", "production")
- a device model or device operating system (e.g. "Android", "iOS")
- an operating system (e.g. "Windows", "macOS", "Linux")
- browser (e.g. "Edge", "Chrome", "Firefox")
Some tests may be tailored for or only run in some environments.
Whenever performing testing on different environments, results
Test Environments in Xray
In Xray, Test Environments are focused on the execution aspect, providing the means to schedule tests and analyze their results in different environments.
Thus, they're explicitly associated with Test Execution issues.
Overview
The status of a Test (i.e., TestRunStatus) is generally calculated by looking at the Test's last execution (more detail here). However, this does not work well if you execute the same Test in two different test environments (e.g., devices) and you want the two results to be consolidated.
...
Using multiple environments at the same time
...
Sometimes, you may have multiple semantics for environments (e.g. browser, operating system, testing stage); in theory, you can think as it being something multidimensional
- You could name the environment such as “windows_edge” or “mac_chrome” but…
- You could have a ton of composed environments which wouldn’t be manageable at all
- You couldn’t analyze coverage just from the perspective of one of those variables (e.g. “mac” or “chrome”); you would be restricted to analyze it from the perspective of the composed environment
...
The calculated status of the test, per Test environment, will be the following.
Status | Why? | |
---|---|---|
windows | PASS | due to the last result obtained in "windows" environment on CALC-5262 |
mac | PASS | due to the last result obtained in the "mac" environment on CALC-5262 |
edge | FAIL | due to the last result obtained in "chrome" environment on CALC-5261 |
chrome | PASS | due to the last result obtained in "edge" environment on CALC-5263 |
"All Environments" (if analyzing the status of the test without identifying a specific environment) | FAIL | as the last result for one of the environments ("chrome") was FAIL (i.e. on CALC-5261) |
Tips and Recommendations
Do's
...