Page History
| Table of Contents |
|---|
Executive Summary
Xray is used by many Agile teams or , as well as by teams doing the who are undergoing an Agile transformation , and are moving out from of traditional waterfall scenarios.
Being Agile also One of the primary points of being Agile means having a tool does not dictacte how you should work but that gives you instead freedom to constantly adapt it to your team needs. This that adapts to your team's needs. This document sums up the different possibilities you have in working with Xray in an agile context and provides some guidance , although you should in getting started. Of course, we always suggest you evaluate what best fits best to your context.your team's needs.
In order to have a successful Agile process, you need to have
...
these processes/mindsets in place:
- immediate information/no handoffs: Xray provides immediate information visibility right within your work item; thus. This way, you don't have to ask someone to know the about results, the testing progress, or what is the coverage status of a an Epic/Story. Thus, for For example, you can see how a user story is from a QA standpoint, right from the Story issue screen; likewise, you can see QA feedback in your existing Scrum Boards
- collaboration: everyone is invited to collaborate, no matter their role; therefore, anyone can interact and collaborate on user stories, Test test cases, on test planning, on test execution; as . This is because Xray works inside Jira and uses different issue types as an abstraction of its own entities, meaning you can collaborate inside Jira, the Jira wayway
In short, after successful installation & setup, we advise you to:
- have Test Plan(s) per sprint
- configure your existing Agile Boards (i.e. Scrum Board of active sprint) to show Test Plans and (Sub) Test Executions assigned to your Sprints and to show the coverage status on the covered issues
But However, as there is much more to know besides these points, please read the following sections to have a thoroughly a thorough understanding of how can to take advantage of Xray in an Agile context.
Xray in Scrum based teams
If you're following Agile, particularly the Scrum framework, you'll have sprints of one, two, or more weeks (a sprint has a short, well-defined time period).
...
| Info | ||
|---|---|---|
| ||
As Xray is a flexible tool, and every team has its own demands, we highly recommend to start simple and explore some of possibilities offered by the tool. Following the Agile principles, your processes should be reviewed often by the team and adapted accordingly with team needs. You should also not dictate a long and very strict processprocesses, as you want to promote "Individuals individuals and interactions over processes and tools". Therefore, you need to find the right balance. |
Process
- review the user stories with your team (remember: testers are part of the team)
- create Tests for each user story
- organize the Tests
- create one or more Test Plans (see above) and add the relevant Tests
- schedule multiple Test Executions, from the Test Plan or directly from each Story as Sub-Test Executions
- track overall, consolidated progress at the Test Plan level
- use reports to assess additional information, such as...
- Overall Coverage Report, for evaluating the current coverage status of the coverable entities (e.g. user stories)
- Traceability Report, to quickly filter out relevant entities based on their coverage, check the runs of related Tests and reported defects
- Test Executions Report and/or Test Plans Report, as means to see the results obtained on different environments, on different revisions of the SUT, and to check the reported defects and if they're still to be resolved or not
- use Jira dashboards to share information about your project, including testing progress; use some specific gadgets for the later
What belongs to a release?
Tests, Pre-Conditions and Test Sets are reusable entities; thus, in typical scenarios, they do not belong to the scope of a release, or of a sprint.
In the other hand, Test Executions, Sub-Test Executions and Test Plans represent work targeted for a given release and/or a given sprint. Thus, they can be tracked as any other issue that needs to be handled in the scope of your release/sprint. If you have these issues assigned to some release through the FixVersion/s field, then they will appear in the summary information of the Releases project page.
| Info | ||
|---|---|---|
| ||
As of Xray v3.4, Tests created from the "requirement"/story screen will inherit the FixVersion/s from the parent issue. The version assigned to the Test case has no special meaning, unless you want to handle it as specification work that you need to address in the context of that release. If that's not the case, you can simply ignore. Xray may change this behaviour in the future. |
Coverage: Epics and Stories
Xray understands the "hierarchical" relation between Epic<=>Story. In fact, it's just one of the possible scenarios for handling parent and "sub-requirements".
...
| Info | ||
|---|---|---|
| ||
Please make sure you have the Epic<=>Story relation enabled under Xray "Issue Type Mappings" administration section. |
Note that you can create and/or link a Test explicitly with an Epic, if you wish to cover it directly.
Creating Test cases
Tests can be created right from the Epic/Story screen. That way, they will be automatically linked with the respective issue.
Otherwise, if you create Tests using the Create button from the top navigation bar, or from a Test Set, or from a Test Repository folder action, then you have to specify the covered issue.
You can do that at the moment of creation...
or later on, by using the More > Link action available from the Test issue screen (or from the Story/Epic issue screen, but using the "tested by" issue link in that case).
Organizing the Test cases
Organization is essential to easily find the Tests later on, whenever you need to perform regression testing, for example.
...
Thus, you'll need a structured way to organize your Test cases to make your work more efficient.
Possible organization scenarios
Xray provides several different options for organizing Test cases.
...
| Info | ||
|---|---|---|
| ||
Each organization approach provides its own benefits and its own limitations. Check out some comparison information providing better insights on the possibilities and limitations of some of these approaches |
Regression testing
The list of Tests you wish to run for regression testing may be quite dynamic and change over time, or it can be quite static; it depends on your internal process.
...
On the other hand, you can have an implicit way of defining the Tests you wish to run during regression testing. How? Well, you may start by creating a first Test Plan for regression testing, adding all the Tests that you consider relevant. On upcoming releases, you may clone (using "issue clone" operation) the previous Test Plan and add/remove the Tests that you want.
Defining the scope of your testing using Test Plans
During the lifetime of a sprint, you will perform testing as a way to ensure you have a shippable product. Therefore, you'll need at least one Test Plan as means to identify what you want to test and also to track its results.
...
| Expand | ||
|---|---|---|
| ||
|
Basic: one Test Plan per Sprint
The simplest approach is to have just one Test Plan per sprint; you can set this explicitly by setting the Sprint field on the Test Plan issue.
...
Test Executions created from the Test Plan will also, by default, inherit the Test Plan assigned version (i.e. FixVersion/s).
Test Plans for RT and NRT
Sometimes, you may prefer to distinguish regression testing (RT) from non-regression testing (NRT), for several reasons.
...
You can analyze your coverage (i.e. how your "requirements"/user stories are) from the release perspective or from each Test Plan perspective.
Multiple Test Plans
As a manager, you can define as many Test Plans as you wish.
...
The following diagram shows an example of possible different criteria for creating Test Plans.
Using Sub-Test Executions
Xray provides the ability to easily create Test Executions containing all the Tests that cover some "requirement"/Story as a sub-task of it.
This may provide a quick way for you to schedule the execution of certain Tests; you can also link that Sub Test Execution to an existing Test Plan.
...
| Info | ||
|---|---|---|
| ||
To be clear, you don't have to use Sub-Test Executions. However, as mentioned above they may provide some interesting benefits for some specific use cases. However, if you decide to also use Sub-Test Executions, please always link them to the proper Test Plan so you can track their results at an higher level: at the Test Plan level. That way, you will be able to always track the progress from the Test Plan point-of-view. |
Tracking progress in Agile/Scrum based boards
As shown in Agile Enhancements page, you can configure your Agile Scrum board to show immediate feedback about:
...
Including in the board Test Executions coming from automated testing (i.e. during CI), may not be recommended as it will overload the board and not provide any real added value. Instead, you may include only Test Plan(s) as means to group and show the consolidated results coming from automation.
Xray in Kanban based teams
You may have a look at the previous instructions for Scrum teams as most of it can also be applied in Kanban teams.
...
Thus, you can have a Test Plan to track the Tests related with the features being addressed on the Kanban board. You may complement it with additional Test Plans, namely, for tracking the regression testing. It may be a good approach to have the Test Plan with regression tests, or smoke tests, as an independent Test Plan, so it is more apparent if regression is occurring or not.
Questions
- We're not really adopting Scrum perfectly... can we use these approaches in our case?
- Yes, you may follow these same principles and adapt them to your methodology.
- We have a hardening/clean-up sprint where we wish to test thoroughly, including deep regression testing. How can this be handled?
- Hardening sprints are normally a sign that Agile is not fully being embraced. Having said that, you can have a Test Plan to track the testing performed in the scope of that sprint, adding all Tests related with user stories and other issues "implemented" on previous sprints, and thus consider the results performed in the scope of this Test Plans as being the relevant ones
- Are Test Plans, Test Executions or Sub-Test Executions child issues of an Agile board? In other words, do they belong to some Agile board?
- No. All those issues, first of all, belong to some project. They can optionally be assigned to a release (through the FixVersion/s field) and to a sprint (through the Sprint custom field).
- A different thing is showing whatever issues you want in your existing boards; as those entities are implemented as issues, you can configure boards to show them. In fact, the same issues can be included (i.e. shown) in different boards.