---
title: "Tips for Organizing & Planning Tests"
canonical: "https://docs.getxray.app/space/XRAYCLOUD/659259394/Tips%20for%20Organizing%20%26%20Planning%20Tests"
format: markdown
---
> Macro (rw-ui-expands-macro)
>
> > Macro (rw-expand)
>
> > Macro (toc)
# **Introduction**
Properly [organizing](https://docs.getxray.app/space/XRAYCLOUD/44566566/Organizing+Tests) and [planning](https://docs.getxray.app/space/XRAYCLOUD/44565044/Planning+Tests) Tests are core activities of effective Test management that often go hand in hand. They enable more structured execution, reporting, and maintenance.
Xray offers versatile means and dedicated entities to handle those activities - as a refresher, the following diagram briefly illustrates the purpose of different Xray entities, and we are focusing on stages 2 and 3 in this article.
> ℹ️ In theory, only Test Cases and Test Executions are mandatory for the Xray process, so you could ignore the rest (and e.g. just use Jira Labels to bulk select tests for Test Execution directly).
> ℹ️
> ℹ️ In practice, other Xray entities help facilitate more scalable and efficient processes, so it’s a good idea to familiarize yourself with their use case nuances.
As a brief preview, this guide will follow these steps:
- Prepare the Tests by following a consistent terminology and process.
- Organize the Tests - in folders within the respective [Test Repository](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565166) or in Test Sets - by what you consider relevant from a semantic perspective of those Tests.
- Plan and track the Tests and their execution for a release using one or more Test Plans (potentially with ranking and Plan Board folder structure).
# **Prerequisites**
To “fully” enable both Organizing and Planning stages, we need to address two questions in advance - “what relevant information could we use to categorize Tests?” and “how do we add that info to the Tests and/or related Work items?”. We have to think about the Test’s **purpose (i.e., the “why”)**, **how** we actually achieve it, **where**, and **what** quality aspect we’re looking at.
The answers will allow us to establish consistent rules for logical, systematic grouping. Here is a non-exhaustive list of things we could consider the:
- Level of Test (e.g., unit, integration, API, e2e).
- Type of Test (e.g., functional, non-functional) and specific quality attribute being tested (e.g., correctness, performance, security).
- Type of Test scenario (positive, negative).
- Priority and risk (of the Test directly or the associated Requirements).
- Application area/component/feature, etc.;
> ℹ️ Some grouping criteria (e.g. Priority) could be useful for both Organizing and Planning, while others (e.g. Component) tend to be more stage-specific.
Then we figure out how to account for those aspects in the Jira context. Start by enabling **discoverability**, adding proper semantics to your Tests, and using the most adequate fields. Some of the common examples include:
- ***Labels ***(system Jira field)*:*
- If you start having many testers and your project grows, having a well-defined list of labels may help.
- Avoid grouping too many different attributes into the Labels field (e.g. 4+ labels for each Test, such as “integration” + “positive” + “security” + “featureX”). That could make filtering a bit messier.
- ***Priority ***(system Jira field).
- ***Component ***(system Jira field).
- **Fix Version **(aka Release; system Jira field)
- ***Test Type ****(*from the perspective of “manual”, “automated”, “exploratory”;* *system Jira field).
- **Test Category ***(*for other dimensions, e.g. functional/security/performance/etc.; (custom Jira field)*.*
- **Risk Score **(custom Jira field).
> ℹ️ Don’t rely on just adding the previous semantics into the *Summary* or *Description* fields of Test Work items, as that makes them harder to parse & filter; use dedicated fields instead.
> ℹ️
> ℹ️ Make sure that the fields you select are easily available via JQL and filters in Xray Test Set/Plan/Execution. Single-choice lists are the most compatible. Only then can we effectively use this preparation in the subsequent stages.
- Check “Search” tab options on step 3 in these - [https://docs.getxray.app/space/XRAYCLOUD/44565117/Test+Execution#Existing-Tests](https://docs.getxray.app/space/XRAYCLOUD/44565117/Test+Execution#Existing-Tests), [https://docs.getxray.app/space/XRAYCLOUD/44565153/Test+Plan#Adding-Tests-to-a-Test-Plan](https://docs.getxray.app/space/XRAYCLOUD/44565153/Test+Plan#Adding-Tests-to-a-Test-Plan) , [https://docs.getxray.app/space/XRAYCLOUD/44565163/Test+Set#Using-an-Existing-Test](https://docs.getxray.app/space/XRAYCLOUD/44565163/Test+Set#Using-an-Existing-Test) (may also want to review the filter and “Group By” options in [Reports](https://docs.getxray.app/space/XRAYCLOUD/338526916/Reporting+Center)).
- Validate basic [JQL](https://support.atlassian.com/jira-software-cloud/docs/jql-fields/) statements/[functions](https://support.atlassian.com/jira-software-cloud/docs/jql-functions/) as well as [https://docs.getxray.app/space/XRAYCLOUD/44565225/Enhanced+Querying+with+JQL](https://docs.getxray.app/space/XRAYCLOUD/44565225/Enhanced+Querying+with+JQL).
> ℹ️ We also share a few [practical examples](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/656474150) for users who do not have Xray JQL Functions enabled. Feel free to adapt these to your internal process to fit your team better.
# **Organizing**
We define this stage as a more static, longer-term arrangement of Tests into groups for easier finding and maintenance. And typically, this arrangement has no immediate impact on or relevance to execution.
Xray provides two ways to organize Tests:
- Folders within a **hierarchical** structure via the [Test Repository](https://docs.getxray.app/space/XRAYCLOUD/44565166/Test+Repository) at the project level (similar to the way you would organize the files on your computer);
- ordered, **flat** lists via the Test Set work item.
> ℹ️ For the avoidance of doubt, Test Repository always exists in projects with Xray enabled. Whether you create a folder structure or keep everything in “Orphans” is your decision.
Let’s start with the common points. You can quickly:
- Add a group of Tests to a Test Repository folder or a Test Set using the criteria we discussed in Prerequisites (via the Filters or JQL functionality).
- Create a Test Plan or a Test Execution for **all** tests in a Test Repository folder (using right-click context menu) or a Test Set (using the “Add Tests” menu on the target Plan or Execution).
The differences are:
- The same Test can be part of **different** Test Sets but can only belong to **one** Test Repository folder (i.e. we cannot add the same Test to multiple folders at the same time, so we have to think in a hierarchical way, "What is the most important? How do I look for the Tests to schedule executions?").
- Test Set is a **Jira work item**, so it can be linked directly to cover requirements, and you can apply Labels, Workflow, JQL, etc. to it. You cannot do any of that with the Repository folders.
On the other hand, in general, **Test Repository scales better** (because it doesn’t create additional Jira items) and provides arguably easier filtering, finding, and management of tests because all of them are in one entity vs separate Test Set work items. One Test Set as a “master” with 1000s of tests may struggle performance-wise.
Each approach has advantages and drawbacks, but they can also work in a complementary way. With proper guidance, you can leverage both in more advanced usage scenarios, such as using Test Sets to cover complex requirements while using Test Repository to organize the Tests overall.
## Test Repository Example
Let’s take a look at the Test Repository examples (Figures 2 and 3):
Typically, you create top-level folders for the aspect that you value the most whenever searching for Tests, such as their goal (e.g., "performance", "security", "smoke Tests"). Once you establish that “hierarchy anchor”, you then decompose into the next logical layer. For example, if your Tests are the "components" of the system, you can create a top-level "components" folder and sub-folders for each sub-component.
It is common to have a mix of Prerequisites influencing your folder structure. Another example could be:
- System.
- Components:
- Component A.
- component N.
- Features:
- Subscription.
- Payments.
- Recommendation system.
- Admin.
- APIs.
- Performance.
- Security.
- Smoke Tests.
## Test Set Example
Now onto the Test Set example (Figure 4):
Figure 4 - Test Set layout
In this Test Set, we have grouped Cucumber tests that are relevant for PCI DSS compliance and that belong to the Account component. We can easily reflect that using the Jira fields (Figure 4 - 1). Test Type is visible in the table, but could also be added as a field/note to the Test Set level.
Especially if we use multiple Prerequisite levels at the same time (e.g. Cucumber + Generic, Medium + High priority), it can be a good idea to customize the Columns view (Figure 4 - 2), adding the relevant semantics to the view, and leverage the Filters menu (Figure 4 - 3) to limit the table view only to a relevant subset.
> ℹ️ - Keep in mind that Test Repository is a project-level organization mechanism (i.e., there is only one shared folder organization for the Test cases of that project). Thus, any structural changes you make affect all project users, so you would need the coordinated approach.
> ℹ️ - For the avoidance of doubt, you can create Test Repository folder structure or/and Test Sets before or after you have Tests created - the order doesn’t really matter.
## Good Practices
Before we discuss Planning, let’s cover a few recommended good practices that could apply to organizing via either Test Repository or Test Sets:
- Don’t try to over-organize; even though we could organize Tests on multiple different levels based on all sorts of criteria, it’s impossible to organize Tests from all possible angles and, even if it was, it would cause too much clutter.
- Don't organize the Tests based on temporary attributes; avoid mixing execution or planning-related aspects into the Organizing stage, as the same Tests may apply to v1 and v2 of the System Under Test (SUT), for example.
- Avoid duplicating Tests (e.g., to enable having the “same” Tests in multiple folders) as that will negatively impact the maintenance of Tests and add confusion.
- Deprecate Tests whenever appropriate, using a workflow status, for example, to be able to quickly exclude those Tests (learn more about [using workflows for Testing purposes](https://docs.getxray.app/space/XRAYCLOUD/44567832/Using+Jira+workflows+for+testing+purposes)).
And a special mention - **regression Testing**. If we create a “regression” folder in the Test Repository, would that work?
Before answering it, let’s think what regression Testing is all about - checking for regressions on features that previously worked fine (i.e., testing already existing features). That means we usually run Tests that cover features from previous versions; we don’t run all of them - usually we pick a set of them for time constraint reasons.
Therefore, these are not “new Tests” made explicitly for the purpose of regression Testing; they are existing Tests covering existing Requirements. As a consequence of this - we wouldn’t be able to have these tests in both the “regression” folder and some other folder at the same time.
What about a Test Set that groups all tests with e.g. a “regression” label? While you could have the same test in that Test Set and another, feature-based one, it would still be suboptimal because the selection of tests would likely change too much from one regression execution to the next.
Therefore, it is often more efficient to manage the regression Testing effort as part of a dedicated Test Plan, or even a folder inside the Test Plan Board. With that, let’s talk more about the next stage.
# **Planning**
We refer to “Planning” as the activity where you decide your **Testing strategy, **e.g., which Requirements you want to validate, how you want to validate them, Test prioritization, whether it will be manual or automated Tests, the resource allocation, and **the execution plan** (who and when will execute the Tests). Compared to Organizing, Planning is more short-term and execution-focused.
Tests that are to be validated at a certain time for a certain target of the SUT (e.g., version, sprint) can be tracked in Xray **Test Plans**. For a given target, you may create one or more Test Plans to track the results related to different Tests (e.g., you may create a Test Plan for regression Testing and another one for non-regression Testing).
You may ask - “I already have my Tests grouped, why do I need another Work item? - Test Plan - if I can just create Test Executions from a folder or a Test Set?”. Two key reasons:
1. **Dynamic nature and time horizon**. You may have folders or Test Sets that contain the Tests you would **generally** consider to be a part of a certain execution effort, like regression. However, for each **actual** run in a given sprint/release/etc., you may select slightly different mixes of those Tests. You wouldn’t want to edit the overall grouping nor create temporary folders/sets, as we discussed. Test Plans give you flexibility as an intermediate step.
The lifecycle of a Test Plan is typically equal in length to one large execution iteration, depending on your methodology, for example:
- Agile Scrum - Test Plan per sprint.
- Agile Kanban - Test Plan per release (or its part, based on the board progress).
- Waterfall - Test Plan per current SUT version.
- Iterative Waterfall - Test Plan per intermediate release.
Whereas Test Set or a Repository folder “survives” until major project structure changes.
> ℹ️ The Test Executions made in the context of the “intermediate” Test Plans can also be associated with a “master” Test Plan, so their results would also get reflected in a broader sense.
Furthermore, you would likely want to see the progression in the Test Plan workflow status over the course of one release as the execution results come in. Having a dedicated, execution-focused entity allows you to e.g. easily mark it as “Done” in Sprint 1, clone it (which resets the execution statuses and unlinks Test Executions but keeps the assigned tests), then “restart” the process in Sprint 2 (with or without adjustments to the roster of tests).
This facilitates a more efficient and consistent process with better visibility for each iteration of your testing effort.
2. **Immediate aggregate visibility into multiple execution results.** When you open a Test Repository folder or a Test Set, you don’t know whether the related Tests are passing, failing, or not being executed at all. When you open a Test Execution, you only know the results for that one group of Tests.
However, inside a Test Plan you can aggregate the results across multiple Test Executions, without having to open “full-blown” Reports. One of the classic examples is executing the same-ish suite of tests in different environments - you create a Test Execution work item per environment, then link all of them to one Test Plan (that has the full list of Tests) and **quickly get consolidated insights**. The Test Plan will present the latest status of each Test, independently of the number of Testing iterations (i.e., Test Executions) you perform.
## Example
Let’s take a look at a quick example (Figures 5 and 6):
Figure 5 - Test Plan - Tests tab
Figure 6 - Test Plan - Test Executions tab
We want to plan an execution cycle to validate the PCI DSS compliance for “v2” of our mortgage quoting app, regardless of the component, so we pull Tests from both Account and RatingEngine, for the total suite of 27 (Figure 5).
We then create three Test Executions - 2 for the different environments we need in Account testing (using Xray’s [Test Environment](https://docs.getxray.app/space/XRAYCLOUD/44565129/Test+Environments) field) and 1 for RatingEngine.
The Tests tab of our Test Plan gives us the latest execution status for **all** Tests we need for our compliance goal, regardless of the component or environment. We can not only Filter the view but also switch the status calculation to a specific environment (Figure 5 - 1).
The Test Executions tab (Figure 6) shows us the progress of each associated Work item separately, so we can easily spot issues for a given component or environment without needing the “full power” of Xray reporting.
To reiterate, you are not limited to creating **one** Test Plan for your release or any other iteration of work. You may create **multiple** Test Plans to track different Test groups. You may create different Test Plans for:
- Tests related to new features.
- Regression Testing (i.e., for Tests related to features implemented in previous versions).
- Security-related Tests.
- Performance-related Tests.
- Compliance-related Tests.
- Other non-functional Tests.
> ℹ️ **[Static](https://docs.getxray.app/space/XRAYCLOUD/44565153/Test+Plan)**** vs ****[Dynamic](https://docs.getxray.app/space/XRAYCLOUD/44565153/Test+Plan#Configuring-a-Test-Plan-with-a-Saved-Filter-(-Xray-Enterprise-Only-))**** Test Plans **(Dynamic Test Plans** **are only available in **[Xray Enterprise](https://marketplace.atlassian.com/apps/1229688/xray-enterprise-test-management-for-jira)****)**.
> ℹ️
> ℹ️ The role in the planning stage is the same, as are other usage considerations we’ve discussed above. Dynamic Test Plans improve your ability to manage the dynamic nature of the test list - once you specify the JQL filter (e.g. issuetype = 'Test' and labels = 'regression'), any test that matches it is pulled into the Test Plan automatically. Any test that no longer fits the filter is removed automatically.
> ℹ️
> ℹ️ So you save time on not having to manually edit the Test Plan.
And a special mention - [Test Plan Board](https://docs.getxray.app/space/XRAYCLOUD/44565285/Test+Plans+in+the+Testing+Board). By default, the Plan content is a flat list of Tests, but you have an option to go beyond that. You may organize and track the Test progress using folders, where these folders are more focused on the things that matter the most to you from the context of planning and tracking the progress (e.g. by tester, by component, by flow, or just by the Plan-specific ranking order of tests). Each Test Plan has an associated Test Plan Board, and a Test can only belong to one folder within the Board.
> ⚠️ For the avoidance of doubt, you could create Test Plans even before you have Tests created/organized - the order doesn’t really matter from the perspective of operations allowed by Xray and Jira.
# Conclusion
Each Xray entity (e.g., Test, Precondition, Test Set, Test Execution, Test Plan) has a purpose. In this article, we focused on the distinction between Test Repository, Test Set, and Test Plan. While all contain groups of Tests, there are important differences that we have summarized below:
| | **Test Repository** | **Test Set** | **Test Plan** |
| --- | --- | --- | --- |
| Purpose | Organizing | Organizing | Planning |
| Nature | Xray-specific, project-level folder structure | Jira Work Item | Jira Work Item |
| Content | - Hierarchical groups of Tests.
- Each Test can only belong to one folder. | - Flat list of Tests, one logical group per Set.
- Each Test can belong to multiple Sets (no need to clone tests). | - Flat list of Tests (unless you use Test Plan Board), one or more logical groups per Plan.
- Each Test can belong to multiple Plans (no need to clone Tests). |
| Available actions | - Add Tests (new or existing).
- Filters to manage Tests.
- Context menu to create Sets, Plans, Executions from a folder (for all tests in it) | - Jira features (JQL, Workflow, etc.)
- Add Tests (new or existing).
- Filters to manage Tests.
- All tests from a Set can be added to a Plan or Execution (from the UI of those target entities). | - Jira features (JQL, Workflow, etc.)-
- Add Tests (new or existing).
- Filters to manage tests
- Create Test executions (for all or **a subset of Tests**).
- Aggregate multiple Test executions. |
Xray gives you the ability to use all or just a subset of these entities, so you can better meet your team’s needs. You may decide to start simple and then enhance your process along the way by using additional entities. Either way, Xray is as flexible as you want it to be.
> Macro (rw-ui-expands-macro)
>
> > Macro (rw-expand)
>
> If you have questions or technical issues, please [contact the Support team via the Customer Portal (Jira service management)](https://jira.getxray.app/servicedesk/customer/portal/2/user/login?destination=portal%2F2%2Fcreate%2F28) or [send us a message using the in-app chat](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44577312).