Xray offers integration with Structure, enabling the ability to manage Test Sets, Test Executions and Test Plans.
Features and Limitations
The current integration allows you to:
- see the Tests associated with a Test Set, Test Execution, Test Plan or Pre-conditions in the Structure tree
- add/remove Tests to Test Sets, Test Executions or Test Plans by dragging Tests or Test Sets to other Test Sets, Test Executions Test Plans or Pre-conditions
- manage Test Executions underneath the related Test Plans
- see and manage the association between Tests and requirements (supported indirectly by Structure)
- see Sub-Test Executions below the requirement issues (supported indirectly by Structure).
It's not possible to:
- associate Tests contained in folders to Test Sets, Test Executions and Test Plans
- manage folders inside Test Sets, Test Executions, Test Plans.
Using Structure vs Test Repository/Test Plan Board
Since Xray v3.0, you can use the Test Repository to organize Tests hierarchically in folders at the project level; you may also organize Tests, in an independent way, at Test Plan level, for each Test Plan.
The question arises: which approach should you use? Organize entities using Structure add-on or using Xray built-in capabilities?
Let's see the different and summarize the characteristics of each approach, focused on the related entities.
| Structure | Xray's Test Repository | Xray's Test Plan Board |
|---|
| Pros | - can manage different entities (e.g. stories, Tests, Test Executions, etc)
- can be used to have an high-level visual overview, specially for teams not used to Jira, starting at the version, then seeing all entities associated to, organized in some way
- not bound to a specific project, thus each "structure" can have issues from different projects
- can be used alternatively or complementarily to Test Sets
- can manage entities from several projects in the same structure (this can be a pro/con, depending on the perspective)
- can have the same issue multiple times within a given structure (this can be a pro/con, depending on the perspective)
- can be used complimentarily with Test Plan Board (i.e. manage the project/version related entities in a structure and manage the organization of the Test Runs inside each Test Plan Board)
| - focused just on organizing Test cases of each project
- not bound to execution aspects
- used as means to create or add Tests to existing entities (e.g. create Test Executions, create Test Plans)
- can't have the same issue multiple times within the Test Repository (this can be a pro/con, depending on the perspective; we see it as an advantage, since it simplifies its management)
| - focused on organizing the Test cases of each Test Plan
- gives the ability to track down the results on a per folder basis, within the Board
- easily rank Tests visually and take that ranking into account for new scheduled Test Executions
- gives the ability to analyze the results per Test Environment
- can contain Tests from several projects
- can't have the same issue multiple times within the Board (this can be a pro/con, depending on the perspective; we see it as an advantage, since it simplifies its management)
|
|---|
| Cons | - can't create folders within Xray entities (e.g. you can just see/manage the Tests associated with a Test Execution)
- independent organization, i.e. does not integrate neither with the Test Repository nor the Test Plan Board concepts and their organizations
- unable to evaluate results for certain Test Environments
- unable to see/manage Test Runs nor execute them immediately
- unable to easily create other entities (e.g. Test Executions) for the Tests within some folder
| - does not provide an high-level overview of the project or a project version and all the related entities associated with that version
- unable to manage entities from several projects in the same Test Repository (this can be a pro/con, depending on the perspective)
| - does not provide an high-level overview of the project or a project version and all the related things
|
|---|
Installation
In order to obtain the new extenders that Xray provides, you need to:
- Install Structure plugin version 3.4.x or above;
- Install Xray version 2.1 or above.
Note: If you already had Xray before you installed Structure, you need to restart your Jira server.
For more information about the Structure plugin, refer to its documentation here.
Adding Xray Extender to the Structure:
1. Set your structure to Automation
2. Choose the correct extender
3. Choose the options to enable, including Xray specific actions, and click "Apply" on the dialog
You should see a new entry in the structure, similar to this one:
Settings
Whenever adding the Xray Extender or whenever editing an already existing one (by double clicking on it), you may have several Xray Actions that are in fact options that you can enable:
- Manage Test Executions under Test Plans: allows you to see related Test Executions underneath Test Plans, at first level, and manage their association by adding/removing Test Executions to/from the Test Plan
- Synchronize Tests between Test Execution and related Test Plan(s): whenever adding Tests from Test Executions, it will automatically add Tests to the related Test Plans; on the other hand, whenever removing Tests from Test Executions, if those Tests are not part of other Test Executions linked to the same Test Plan(s) (i.e. if the Tests are not "already being used"), they will be removed from the Test Plan(s)
- Manage Tests under Test Plans: allows you to see related Tests underneath Test Plans, at first level, and manage their association by adding/removing Tests to/from the Test Plan
Recommended Setups
The way you use Structure will vary depending on your team needs.
Within this section you may see some possible setup scenarios.
Scenario 1
The following example focus on these needs:
- quickly showing, on a per version, the planned testing (i.e. Test Plans and related Test Executions)
- track the overall progress of Test Plans and Test Executions and also the coverage status of "requirements"
- on each "requirement" ...
- see Tests directly covering it
- track related Sub-Test Executions
Setup Automation as follows:
Setup Xray Extender settings:
- Manage Test Executions under Test Plans: enabled
- Synchronize Tests between Test Execution and related Test Plan(s): enabled
- Manage Tests under Test Plans: disabled
Actions
Xray provides specific actions that will interact and integrate with changes made on Structure. These actions must be invoked explicitly.
When moving a Test to different issue types, the Test will not be removed from source issue. Example: When moving Test A from Test Set B into Test Execution C, the Test will be added to Test Execution C, but not removed from Test Set B. |
Whenever doing action using Structure, you may be prompted to perform one of multiple available actions. Please choose the proper action or else you may not obtain what you were expecting for. For example, if you add a Test Execution underneath a Test Plan, or if you add some Tests to a Test Plan, you may be prompted with a dialog similar to the following one. If you are aiming to perform Xray related actions, please choose the one containing the Xray icon. If you choose "Move within ...", then the item will be moved visually within the structure but no interaction will be made with Xray. 
|
Pre-Condition
With the Extender added, you can now do the following actions:
- Manage Tests of the Pre-Condition:
- Drag a Pre-Condition to a structure and have its associated Tests dynamically appear;
- Drag a Test to a Pre-Condition and have it automatically associated to the Pre-Condition. When you drag a Test to a Pre-Condition, a message will appear as displayed in the image below. You should always choose the Xray "Add" action since it's the option that will use the Xray Extender;
- Drag a Test Set to a Pre-Condition and have all the Tests in the Test Set automatically associated to the Pre-Condition. You will get the same message as above;
- Remove a Test from a Pre-Condition and have the association automatically removed from the Pre-Condition. When you click on the X and you have a Test selected, the Test will be removed from the Pre-Condition;
Test Set
With the Extender added, you can now do the following actions:
- Drag a Test Set to a structure and have its Tests dynamically appear.
- Drag a Test to a Test Set and have it automatically associated to the Test Set.
When you drag a Test to a Test Set, a message will appear. You should always choose the "Add" Xray action since it's the option that will use the Xray Extender.
- Drag a Test Set to a Test Set and have all the Tests in the Test Set automatically associated to the other Test Set.
When you drag a Test Set to another Test Set, a message will appear. You should always choose the "Add" Xray action since it's the option that will use the Xray Extender..
- Remove a Test from a Test Set and have the association automatically removed from the Test Set.
When you click on the X and you have a Test selected, the Test will be removed from the Test Set. If you remove multiple Tests, an additional message may be shown that you should accept.
Test Execution
With the Extender added, you can now do the following actions:
- Manage Tests of the Test Execution:
- Drag a Test Execution to a structure and have its Tests dynamically appear. You may also choose to make the related Test Executions appear; these behaviours depend on the options chosen in the Extender;
- Drag a Test to a Test Execution and have it automatically associated to the Test Execution.
When you drag a Test to a Test Execution, a message will appear. You should always choose the Xray "Add" action since it's the option that will use the Xray Extender;
- Drag a Test Set to a Test Execution and have all the Tests in the Test Set automatically associated to the other Test Execution. You will get the same message as above;
- Remove a Test from a Test Execution and have the association automatically removed from the Test Execution. When you click on the X and you have a Test selected, the Test will be removed from the Test Execution.
- Indirectly manage the Tests of the related Test Plans (you must enable a a specific option in the Extender of this):
Drag a Test/Test Set to a Test Execution and have the Test, or related Tests, automatically associated to all Test Plans linked to that Test Execution;
Remove a Test from a Test Execution and have the association automatically removed also from the Test Plan, if that Test is not part of any other Test Execution linked to the same Test Plan.
Sub-Test Execution
You may perform similar actions to the Test Execution. Please see above.
Test Plan
With the Extender added, you can now do the following actions:
- Manage Tests of the Test Plan:
- Drag a Test Plan to a structure and have its Tests dynamically appear;
- Drag a Test to a Test Plan and have it automatically associated to the Test Plan. When you drag a Test to a Test Plan, a message will appear. You should always choose the Xray "Add" action since it's the option that will use the Xray Extender;
- Drag a Test Set to a Test Plan and have all the Tests in the Test Set automatically associated to the other Test Plan. You will get the same message as above;
- Remove a Test from a Test Plan and have the association automatically removed from the Test Plan. When you click on the X and you have a Test selected, the Test will be removed from the Test Plan.
- Manage the Test Executions under the Test Plan (you need to enable a specific option in the Extender for this):
- Drag a Test Execution to a Test Plan and have it automatically associated to the Test Plan;
- Remove a Test Execution from a Test Plan and have the association automatically removed from the Test Plan.
Other features
Association between Tests and requirements
Since the association between Tests and requirements is done using the "Tests" link, you can take advantage of the built-in Structure extender for dealing with linked issues and therefore, manage this association.
For this, you must have "Automation" enabled. Choose "Extend with..." and then "Linked issues...".
Then, choose the "Tests" link type.
As soon as you do this, you're able to see the association between Tests and requirements, as shown below. You can see the story being validated by the manual Test.
You can also link your Test with other requirements just by dragging them under the Test and choose the action that will create the proper issue link between the Test and the requirement.
If you want, you can also manage the association from the opposite end, by looking from the requirement's perspective. This will allow you to see and manage the Tests that validate a given requirement. For this, you just need to create an extender for the "Tested by" link type. |
Association between Sub-Test Executions and requirements
Sub-Test Executions are created as sub-tasks for requirement issues. You can take advantage of the built-in Structure extender for dealing with sub-tasks and therefore, see this association.
For this, you must have "Automation" enabled. Choose "Extend with..." and then "Sub-tasks...", and then the Sub-Test Execution issue type.
Afterwards, you're able to see all the Sub-Test Executions created in the context of a requirement issue that is part of the structure.
Show requirement and test statuses
You can include information about the status of your "requirements" and its current state, along with the latest status for the corresponding Tests, in the issues being shown in the structure.
For this, you need to add the "Requirement Status" and the "TestRunStatus" columns, which correspond to the "Requirement Status" custom field of requirements and the "TestRunStatus" custom field that is available for Test issues, respectively.
Please make sure that those custom fields are configured in Xray (under "Custom Field Preferences" admin settings) to show the respective information in the same way (i.e., for the same version). Refer to this configuration page, the FAQ or Using custom fields if you have questions about these fields.
Show progress information of Test Executions and Test Plans
You can include information about the progress (i.e. the "overall execution status") of (Sub) Test Executions or Test Plans. For this, you need to include the proper custom fields: "Test Execution Satus" and "Test Plan Status", respectively. More information on these fields can be found here.
Note: you need to have the latest version of Structure installed.
Show status overview of Tests contained within Test Sets
You can include information about the status of the Tests belonging to a Test Set, by including the "Test Set Status" custom field as a column. Although Test Sets are not related with execution aspects, sometimes it may be quite useful to have an idea how certain groups of Tests (i.e. the ones that you have in some Test Set) are.
Please make sure this custom field is configured properly in Xray (under "Custom Field Preferences" admin settings), to show the status of Tests for the version you want to.
More information on this field can be found here.
Note: you need to have the latest version of Structure installed.
Learn more
For specific questions concerning Structure, please use Structure app support channel.
Please refer to the following resources for more information:
