Xray is a flexible Test Management app for Jira. This means that you can use it in very different ways, adapting it to your needs and leveraging the potential you have within your team.
As detailed in the Xray Terms and Concepts, Xray uses different entities as abstractions of the different testing phases; each entity addresses a different purpose. Thus, you use Tests to write the specification of your test cases. Organization, on the other hand, is handled either by Test Sets or by the Test Repository. The planning of the testing, in some version or context of a sprint, for example, is handled by the Test Plan.
Summary of Xray entities
The following diagram briefly illustrates the purpose of each entity.
Tests are written to validate requirements, or any other coverable issue. Optionally, they can be complemented with Preconditions, as means to specify the initial conditions to be assured before Tests can be run. Tests and Preconditions address test designing.
Concerning organization (i.e. the way that Tests are organized in the context of some project), Xray gives two approaches: one is list-based, abstracted as Test Sets issues; and the other is hierarchical, abstracted as the Test Repository (which is not an issue type).
Tests that are to be validated at a certain time for a certain target of the SUT (e.g., version, sprint) can be tracked in Test Plans.
Tests can be scheduled for execution by creating assignable “tasks” called Test Executions. They contain the list of Tests along with the recorded result (i.e., for each Test, the Test Execution contains a Test Run). They can be created in an ad hoc way or in a planned way (i.e., linked to some Test Plan(s)). The results of planned Test Executions will be visible and consolidated on the linked Test Plan(s).
Test Set vs. Test Plan
While Test Sets address organization, Test Plans address planning. Thus, they have different purposes.
A Test Set is basically an ordered, flat list of Tests, grouped by any logical criteria (e.g., by business case, component, security, performance). You may have as many Test Sets as you want, and a Test may be included in multiple Test Sets. Test Sets are great if you want to have full control over certain groups of Tests. You can easily select them whenever you want (e.g., when creating Test Plans or Test Executions) instead of having to search individual Tests by hand. Test Sets can also be used to dynamically cover requirements.
A Test Set does not provide any direct information related with test results.
A Test Plan, on the other hand, tracks the results of some Tests in some version/sprint of the SUT. Thus, its purpose is to identify the Tests you want to run, to schedule executions for them and then group the respective results.
Thus, a Test Plan can be seen as a higher-level entity above Test Executions. It groups multiple Test Executions and presents a consolidated overview (i.e., the latest status) for the Tests tracked within it.
For a given version, or a given sprint, you may create one or more Test Plans to track the test results related to different tests (e.g., you may create a Test Plan for regression testing and another one for non-regression testing).
In summary...
Each Xray entity (e.g., Test, Precondition, Test Set, Test Execution, Test Plan) has a purpose. Typically, these entities are related to different testing phases.
We clarified the distinction between a Test Set and a Test Plan. While both contain lists of Tests, the Test Plan has planning/execution semantics while the Test Set is a simple list of grouped test cases.
Xray gives you the ability to use all or just a subset of these entities so you can better meet your team’s needs. You may decide to start simple and then enhance your process along the way by using additional entities. Either way, Xray is as flexible as you want it to be.