What is a Test Environment

Generically speaking, a test environment is an environment that contains all necessary elements, including the SUT, so you can perform testing on it.

Depending on your context, a test environment may represent:

• a testing stage (e.g. "development", "staging", "preproduction", "production")
• a device model or device operating system (e.g. "Android", "iOS")
• an operating system (e.g. "Windows", "macOS", "Linux")
• browser (e.g. "Edge", "Chrome", "Firefox")

Thus, semantics of what a "Test Environment" represents depends on your specific context.

Some tests may be tailored for or only run in some environments. 

Benefits of using Test Environments

• avoid duplication of Tests, whenever you have to run the same test on different environments
• ability to track the latest status of tests on different environments
• ability to track coverage on each environment
• ability to track overall coverage, considering the coverage/results on each environment
• ability to perform reporting, including traceability, per each environment or globally (i.e. considering all results on all different environments)

Test Environments in Xray at a glance

The status of a Test is generally calculated by looking at the Test's last execution (more detail here). However, this does not work well if you execute the same Test in two different test environments (e.g., devices) and you want the two results to be consolidated. 

Within a Test Execution, you may specify the Test Environment(s) where the tests will be executed in the respective attribute. A Test Environment is similar to a label, but Xray has special logic to deal with it.

If you use Test Environments, you may reuse the same Test for multiple test environments and create Test Executions for each one. If you don't use Test Environments and you want to track tests for multiple environments (e.g., devices), then the only way to do it is to create multiple tests, one per each test environment.

Let's say that you have executions for two Test Environments: "Android" and "iOS". The test is considered as PASS only if the latest executions for Android and iOS are both PASS; otherwise, it will be FAIL.

Note: If you don't use Test Environments, then only the latest result matters for assessing the current status of the Test.

How it works

Let's say we have a Test being executed two times. We will start by creating Test Executions TE1 and TE2.

TE2 will be executed after TE1, so TE1 is considered older.

Below are example scenarios and shows how the status of the Test is calculated in each.

If you have a mix of Test Executions (i.e., with and without Test Environment as in scenario D), it will be treated the same way as scenarios A, B, and C.

How to use

When creating a Test Execution, you must set the Test Environment in which the execution will be executed. You can use this field as a simple label: just add the environment or reuse a previously created one.

Test Execution for “Android” Test Environment

 

Test Execution for “iOS” Test Environment

The Test Environments column is shown in your Test Runs table so you can distinguish each execution of the Test between the different environments.

The same Test has been executed in both Test Environments (a Test Execution per Test Environment).

The Requirement Status will show NOK because the Test has failed for one of the environments.

 

Using multiple environments at the same time

Sometimes, you may have multiple semantics for environments (e.g. browser, operating system, testing stage); in theory, you can think as it being something multidimensional

• You could name the environment such as “windows_edge” or “mac_chrome” but…
• You could have a ton of composed environments which wouldn’t be manageable at all
• You couldn’t analyze coverage just from the perspective of one of those variables (e.g. “mac” or “chrome”); you would be restricted to analyze it from the perspective of the composed environment

• Just assign each variable value (e.g. name of operation system, name of browser) as an individual environment

How to

Assign each environment (e.g. name of operation system, name of browser vendor) as you do for a single environment; in other words, just add the multiple environment names as multiple, distinct labels.

Whenever creating a Test Execution:

Whenever updating an existing Test Execution:

Example

Test executed in the context of Test Execution assigned to several environments at the same time

1. windows, chrome (fail)
2. windows, edge (pass)
3. mac, edge (pass)

The calculated status of the test, per Test environment, will be the following.

Tips and Recommendations

Do's

• Use Test Environments only if you want to run the same Test case in different environments and track results per each environment. 
• Simplify the names of Test Environments (i.e. lowercase it, shorten it)
  • Example: macOS => mac
• Evaluate if you really need to assign multiple environments at the same time; using just one is preferable if you can afford that simplicity

Don'ts

• Don't create dozens or hundreds of Test Environments as it will harden their usage and add some performance overhead
• Don’t make composed environment names, such as ”<os>_<browser>_<stage>” as it will pollute the environment namespace and harden management
• Don't try to do data-driven testing using Test Environments; they're not tailored for that

• You may filter using Test Environments in your Test Executions panel to see how the executions are doing, per environment.

Test Environments and the TestRunStatus custom field

The "TestRunStatus" custom field is associated with Test issues and can be used to provide information about the latest status of your test; more info here.

This custom field calculates the status of the test for "all environments" (i.e. the consolidated status), giving you a high-level view; it cannot be configured to show the status for a specific environment.

Internally, this field will store the status of the test for all possible scopes, which besides other things includes the information about the status in all different environments.