Page History
...
Creating a Test Execution
Test Execution for “Android” “android” Test Environment
Test Execution for “iOS” “ios” Test Environment
Tracking the results on different environments
...
This information can be seen in the Test issue screen (see next screenshot) or in other places that show a list of Test Runs (e.g. Test Plan issue screen).
The same test has been executed in both Test Environments (a Test Execution per Test Environment).
...
Results obtained for Test Environments will impact coverage.
Considering the previous screenshot, the "Requirement Status" custom field for the Test issue will show NOK because the Test has failed for one of the environments. This information is independent of the environment picker below within the "Test Coverage" section, which in turn is used to calculate the coverage on request for the selected scope, showing it on the right side along with the corresponding test results.
If you want to analyze the coverage for the requirement (e.g. "story") and show the latest results on that environment, just use the picker on the "Test Coverage" section. As seen ahead, this will produce different results because different results were obtained in different environments.
Please check Coverage Analysis to learn more about coverage analysis possibilities.
...
It is also possible to analyze testing results on a specific Test Environment.This thoroughly considering Test Environments; this analysis can be done using the Traceability Report or the Overall Coverage Report, among others.
...
| Info | ||
|---|---|---|
| ||
One way to deal with these kinds of environments would be to flatten them and treat them as usual, i.e. you could name the environment such as “windows_edge” or “mac_chrome” but…
|
How to use
Assign each environment (e.g. name of operation system, name of browser vendor) as you do for a single environment; in other words, just add the multiple environment names as multiple, distinct labels.
...
| Status | Why? | |
|---|---|---|
| windows | PASS | due to the last result obtained in "windows" environment on CALC-5262 |
| mac | PASS | due to the last result obtained in the "mac" environment on CALC-5262 |
| edge | FAIL | due to the last result obtained in "chrome" environment on CALC-5261 |
| chrome | PASS | due to the last result obtained in "edge" environment on CALC-5263 |
"All Environments" (if analyzing the status of the test without identifying a specific environment) | FAIL | as the last result for one of the environments ("chrome") was FAIL (i.e. on CALC-5261) |
Advanced
Test Environments and the TestRunStatus custom field
The "TestRunStatus" custom field is associated with Test issues and can be used to provide information about the latest status of your test; more info here.
This custom field calculates the status of the test for "all environments" (i.e. the consolidated status), giving you a high-level view; it cannot be configured to show the status for a specific environment.
Internally, this field will store the status of the test for all possible scopes, which besides other things includes the information about the status in all different environments.
| Info |
|---|
If you start using Test Environments in your Test Executions, then it's not only your test status calculation that will change (i.e. the one stored in the TestRunStatus custom field). All custom fields that depend on it (e.g., Requirement status, Test Sets status) will change. Consequently, the requirement coverage calculation and all associated charts/gadgets are also affected. |
Tips and Recommendations
Do's
...
| Info | ||
|---|---|---|
| ||
Besides other usage issues, if you have a large number of environments (>>10), it will impact the calculations that need to be done and the size of the Lucene index. Please try to have a limited, restricted and well-defined list of Test Environments. |
Test Environments and the TestRunStatus custom field
The "TestRunStatus" custom field is associated with Test issues and can be used to provide information about the latest status of your test; more info here.
This custom field calculates the status of the test for "all environments" (i.e. the consolidated status), giving you a high-level view; it cannot be configured to show the status for a specific environment.
Internally, this field will store the status of the test for all possible scopes, which besides other things includes the information about the status in all different environments.
| Info |
|---|
If you start using Test Environments in your Test Executions, then it's not only your test status calculation that will change (i.e. the one stored in the TestRunStatus custom field). All custom fields that depend on it (e.g., Requirement status, Test Sets status) will change. Consequently, the requirement coverage calculation and all associated charts/gadgets are also affected. |