Xray offers integration with Structure, enabling the ability to manage Test Sets, Test Executions and Test Plans.

Features and Limitations

The current integration allows you to:

  • see the Tests associated with a Test Set, Test Execution, Test Plan or Pre-conditions in the Structure Tree
  • add/remove Tests to Test Sets, Test Executions or Test Plans by dragging Tests or Test Sets to other Test Sets, Test Executions Test Plans or Pre-conditions
  • see and manage the association between Tests and requirements (supported indirectly by Structure)
  • see Sub-Test Executions below the requirement issues (supported indirectly by Structure).

It's not possible to:
  • associate Tests contained in folders to Test Sets, Test Executions and Test Plans
  • manage folders inside Test Sets, Test Executions, Test Plans.


Things to be aware

Default Structure Action - Try not to use the Structure default action as it might cause confusion and errors.


Using Structure vs Test Repository/Test Plan Board

Since Xray v3.0, you can use the Test Repository to organize Tests hierarchicaly in folders at the project level; you may also organize Tests, in an independent way, at Test Plan level, for each Test Plan.

The question arises: which approach should you use? Organize entities using Structure add-on or using Xray built-in capabilities?

Let's see the different and summarize the characteristics of each approach, focused on the related entities.



StructureXray's Test RepositoryXray's Test Plan Board
Pros
  • can manage different entities (e.g. stories, Tests, Test Executions, etc)
  • can be used to have an high-level visual overview, specialy for teams not used to Jira, starting at the version, then seeing all entities associated to, organized in some way
  • not bound to a specific project, thus each "structure" can have issues from different projects
  • can be used alternatively or complementarily to Test Sets
  • can manage entities from several projects in the same structure (this can be a pro/con, depending on the perspective)
  • can have the same issue multiple times within a given structure (this can be a pro/con, depending on the perspective)
  • can be used complimentarily with Test Plan Board (i.e. manage the project/version related entities in a structure and manage the organization of the Test Runs inside each Test Plan Board)

  • focused just on organizing Test cases of each project
  • not bound to execution aspects
  • used as means to create or add Tests to existing entities (e.g. create Test Executions, create Test Plans)
  • can't have the same issue multiple times within the Test Repository (this can be a pro/con, depending on the perspective; we see it as an advantage, since it simplifies its management)
  • focused on organizing the Test cases of each Test Plan
  • gives the ability to track down the results on a per folder basis, within the Board
  • easily rank Tests visually and take that ranking into account for new scheduled Test Executions
  • gives the ability to analyze the results per Test Environment
  • can contain Tests from several projects
  • can't have the same issue multiple times within the Board (this can be a pro/con, depending on the perspective; we see it as an advantage, since it simplifies its management)
Cons
  • can't create folders within Xray entities (e.g. you can just see/manage the Tests associated with a Test Execution)
  • independent organization, i.e. does not integrate neither with the Test Repository nor the Test Plan Board concepts and their organizations
  • unable to evaluate results for certain Test Environments
  • unable to see/manage Test Runs nor execute them immediately
  • unable to easily create other entities (e.g. Test Executions) for the Tests within some folder
  • does not provide an high-level overview of the project or a project version and all the related entities associated with that version
  • unable to manage entities from several projects in the same Test Repository (this can be a pro/con, depending on the perspective)
  • does not provide an high-level overview of the project or a project version and all the related things


Installation

In order to obtain the new extenders that Xray provides, you need to:

  1. Install Structure plugin version 3.4.x or above;
  2. Install Xray version 2.1 or above.

Note: If you already had Xray before you installed Structure, you need to restart your Jira server.

For more information about the Structure plugin, refer to its documentation here.

Adding Xray Extender to the Structure:

1. Set your structure to Automation

2. Choose the correct extender

3. Click "Apply" on the dialog

You should see something like this, assuming your structure was empty before

Test Set

With the Extender added, you can now do the following actions:

  • Drag a Test Set to a structure and have its Tests dynamically appear.
  • Drag a Test to a Test Set and have it automatically associated to the Test Set.
    When you drag a Test to a Test Set, a message will appear. You should always choose the "Add Tests to Test Set" since it's the option that will use the Test Set Extender.


  • Drag a Test Set to a Test Set and have all the Tests in the Test Set automatically associated to the other Test Set. 
    When you drag a Test Set to another Test Set, you might get an extra message telling you that you might be doing an action that will affect more than 1 item. Just click Continue and you click the "Add Tests to Test Set"  button.

  • Remove a Test from a Test Set and have the association automatically removed from the Test Set.
    When you click on the X and you have a Test selected, the Test will be removed from the Test Set.

Test Execution

With the Extender added, you can now do the following actions:

  • Drag a Test Execution to a structure and have its Tests dynamically appear.
  • Drag a Test to a Test Execution and have it automatically associated to the Test Execution.
    When you drag a Test to a Test Execution, a message will appear. You should always choose the "Add Tests to Test Execution" since it's the option that will use the Test Execution Extender.
  • Drag a Test Set to a Test Execution and have all the Tests in the Test Set automatically associated to the other Test Execution. You will get the same message as above.
  • Remove a Test from a Test Execution and have the association automatically removed from the Test Execution. When you click on the X and you have a Test selected, the Test will be removed from the Test Execution.

Sub-Test Execution

With the Extender added, you can now do the following actions:

  • Drag a Sub-Test Execution to a structure and have its Tests dynamically appear.
  • Drag a Test to a Sub-Test Execution and have it automatically associated to the Sub-Test Execution. When you drag a Test to a Sub-Test Execution, a message will appear. You should always choose the "Add Tests" since it's the option that will use the Test Execution Extender.

  • Drag a Test Set to a Sub-Test Execution and have all the Tests in the Test Set automatically associated to the Sub-Test Execution. You will get the same message as above
  • Remove a Test from a Sub-Test Execution and have the association automatically removed from the Sub-Test Execution. When you click on the X and you have a Test selected, the Test will be removed from the Sub-Test Execution.

Test Plan

With the Extender added, you can now do the following actions:

  • Drag a Test Plan to a structure and have its Tests dynamically appear.
  • Drag a Test to a Test Plan and have it automatically associated to the Test Plan. When you drag a Test to a Test Plan, a message will appear. You should always choose the "Add Tests to Test Plan" since it's the option that will use the Test Plan Extender.
  • Drag a Test Set to a Test Plan and have all the Tests in the Test Set automatically associated to the other Test Plan. You will get the same message as above.
  • Remove a Test from a Test Plan and have the association automatically removed from the Test Plan. When you click on the X and you have a Test selected, the Test will be removed from the Test Plan.

Pre-Conditions

With the Extender added, you can now do the following actions:

  • Drag a Pre-Condition to a structure and have its associated Tests dynamically appear.
  • Drag a Test to a Pre-Condition and have it automatically associated to the Pre-Condition. When you drag a Test to a Pre-Condition, a message will appear as displayed in the image below. You should always choose the "Add Tests" to the Pre-Condition since it's the option that will use the Pre-Condition Extender.

  • Drag a Test Set to a Pre-Condition and have all the Tests in the Test Set automatically associated to the Pre-Condition. You will get the same message as above.
  • Remove a Test from a Pre-Condition and have the association automatically removed from the Pre-Condition. When you click on the X and you have a Test selected, the Test will be removed from the Pre-Condition.


Note

When moving a Test to different issue types, the Test will not be removed from source issue.

Example: When moving Test A from Test Set B into Test Execution C, the Test will be added to Test Execution C, but not removed from Test Set B.

Other features

Association between Tests and requirements

Since the association between Tests and requirements is done using the "Tests" link, you can take advantage of the built-in Structure extender for dealing with linked issues and therefore, manage this association.

For this, you must have "Automation" enabled. Choose "Extend with..." and then "Linked issues...".

Then, choose the "Tests" link type.


As soon as you do this, you're able to see the association between Tests and requirements, as shown below. You can see the story being validated by the manual Test.


You can also link your Test with other requirements just by dragging them under the Test and choose the action that will create the proper issue link between the Test and the requirement.





Note

If you want, you can also manage the association from the opposite end, by looking from the requirement's perspective.

This will allow you to see and manage the Tests that validate a given requirement.

For this, you just need to create an extender for the "Tested by" link type.


Association between Sub-Test Executions and requirements

Sub-Test Executions are created as sub-tasks for requirement issues. You can take advantage of the built-in Structure extender for dealing with sub-tasks and therefore, see this association.

For this, you must have "Automation" enabled. Choose "Extend with..." and then "Sub-tasks...", and then the Sub-Test Execution issue type.


Afterwards, you're able to see all the Sub-Test Executions created in the context of a requirement issue that is part of the structure.


Show requirement and test statuses

You can include information about the status of your "requirements" and its current state, along with the latest status for the corresponding Tests, in the issues being shown in the structure.

For this, you need to add the "Requirement Status" and the "TestRunStatus" columns, which correspond to the "Requirement Status" custom field of requirements and the "TestRunStatus" custom field that is available for Test issues, respectively.

Please make sure that those custom fields are configured in Xray (under "Custom Field Preferences" admin settings) to show the respective information in the same way (i.e., for the same version). Refer to this configuration page, the FAQ or Using custom fields if you have questions about these fields.


Show progress information of Test Executions and Test Plans

You can include information about the progress (i.e. the "overall execution status") of (Sub) Test Executions or Test Plans. For this, you need to include the proper custom fields: "Test Execution Satus" and "Test Plan Status", respectively. More information on these fields can be found here.

Note: you need to have the latest version of Structure installed.


Show status overview of Tests contained within Test Sets

You can include information about the status of the Tests belonging to a Test Set, by including the "Test Set Status" custom field as a column. Although Test Sets are not related with execution aspects, sometimes it may be quite useful to have an idea how certain groups of Tests (i.e. the ones that you have in some Test Set) are.

Please make sure this custom field is configured properly in Xray (under "Custom Field Preferences" admin settings), to show the status of Tests for the version you want to.

More information on this field can be found here.

Note: you need to have the latest version of Structure installed.


  • No labels