Tests

In very simple terms, a Test is a sequence of steps coupled with conditions or variables, test inputs and an expected result. It is intended to establish the quality, performance or reliability of a component of a system, i.e., test target. Usually, every requirement or objective the test target is expected to achieve needs at least one Test. The success of the Test is determined by comparing the expected and actual results. A Test is an issue type installed by Xray that can be executed multiple times.

Learn more about Tests here.

Preconditions specify the conditions that need to be fulfilled before executing a test.

A Precondition is like defining the step "0" for your tests. This is convenient when you have many use cases that start by doing exactly the same thing. For example, if you are going to do something on a system, you have to do several things  (e.g., restore the DB, authenticate) before going through your Test steps. Each of those things may be done and used independently, i.e., some Tests may require just restoring the DB, another may require just the authentication, others may require both or more.

Learn more here.

A Test Set is a collection of Tests. It often contains detailed instructions or goals as well as information on the configuration to be used during testing.

Test Sets are simple, flat lists of Tests that you can use as a basis for creating Test Executions or Test Plans. Since the Tests are grouped in some logical way, such as grouping all Tests related with regression testing or security, or all Tests for a specific component or some high-level feature/business case, a Test can be part of different Test Sets. Test Sets can also be used as a dynamic way to cover requirements.

Learn more here.

The purpose of a Test Plan is to enable you to have a better visibility and management over your Tests. It provides a table that shows all the Tests added to the Test Plan and their respective Test Executions that are also associated with the Test Plan. If tracking a particular version, the Test Plan will present the latest status for each Test, independent of the number of testing iterations (i.e.,Test Executions) you make with them. You can group a bunch of Test Executions and easily give feedback on the current/latest status of the Tests involved in those Test Executions.

From the Test Plan, it's possible to create one or more Test Executions, for all or just a subset of the Tests contained within the Test Plan (e.g., only those Tests that are still failling). You are also able to assign one or more Test Plans to a given release and track progress of those tests within each Test Plan.

Learn more here.

The status of a Test is generally calculated by looking at the Test's last execution. However, this does not work well if you execute the same Test in two different test environments (e.g., devices) and you want the two results to be consolidated. 

Within a Test Execution, you may specify the Test Environment(s) where the tests will be executed. Test Environments can be created by project managers and made accessible for users to specify in Test Execution issues.

If you use Test Environments, you may execute the same Test for multiple test environments by creating Test Executions for each one. If you don't use Test Environments and you want to track tests for multiple environments (e.g., devices), then the only way to do it is to create multiple tests, one per each test environment.

Let's say that you have executions for two Test Environments: "Android" and "iOS". The test is considered as PASS only if the latest executions for Android and iOS are both PASS; otherwise, it will be FAIL. Xray allows you to calculate the status of Tests and Requirements considering all Environments or a specific Environment.

Learn more here.

Cucumber is a framework for BDD that enables clear understanding of features being implemented and their validation scenarios / acceptance criteria.

With Xray, you can make and manage the specification of Cucumber Tests (i.e., Scenarios / Scenario Outlines) as well as Cucumber Preconditions (i.e., Backgrounds).

The specification can be extracted from Xray and then it can be run in the CI tool against the code that implements each Gherkin phrase. Finally, the results can be submitted back to Xray either by using the UI or the REST API.

Learn more here.

Xray provides a REST API that enables you to import results from Cucumber automated tests as well as Manual and Generic Tests by using specific endpoints.

Learn more here.

Xray provides project- and global-level settings. Project settings can be configured by project administrators. Some settings can be inherited from global settings; some are only global and some only belong to the project level.

Learn more here.