Page History
Table of Contents |
---|
Executive Summary
Xray is used by many Agile teams, as well as by teams who are undergoing an Agile transformation and are moving out of traditional waterfall scenarios.
...
However, as there is much more to know besides these points, please read the following sections to have a thorough understanding of how to take advantage of Xray in an Agile context.
Xray in Scrum based teams
If you're following Agile, particularly the Scrum framework, you'll have sprints of one, two, or more weeks (a sprint has a short, well-defined time period).
...
Info | ||
---|---|---|
| ||
As Xray is a flexible tool, and every team has its own demands, we highly recommend you start simple and explore some of the possibilities offered by the tool. Following the Agile principles, your processes should be reviewed often by the team and adapted accordingly with the team's needs. You should also not dictate long and very strict processes, as you want to promote "individuals and interactions over processes and tools". Therefore, you need to find the right balance. |
Process
- review the user stories with your team (remember: testers are part of the team)
- create Tests for each user story
- organize the Tests
- create one or more Test Plans (see above) and add the relevant Tests
- schedule multiple Test Executions, from the Test Plan or directly from each Story as Sub-Test Executions
- track overall, consolidated progress at the Test Plan level
- use reports to assess additional information, such as...
- Overall Coverage Report, for evaluating the current coverage status of the coverable entities (e.g. user stories)
- Traceability Report, to quickly filter out relevant entities based on their coverage, check the runs of related Tests and reported defects
- Test Executions Report and/or Test Plans Report, as a means to see the results obtained on different environments, on different revisions of the SUT, and to check the reported defects if they're still to be resolved or not
- use Jira dashboards to share information about your project, including testing progress; you can use specific gadgets for this later
What belongs to a release?
Tests, Pre-Conditions and Test Sets are reusable entities; thus, in typical scenarios, they do not belong to the scope of a release or of a sprint.
...
Info | ||
---|---|---|
| ||
As of Xray v3.4, Tests created from the "requirement"/story screen will inherit the FixVersion/s from the parent issue. The version assigned to the Test case has no special meaning unless you want to handle it as specification work that you need to address in the context of that release. If that's not the case, you can simply ignore. (Xray may change this behaviour in the future.) |
Coverage: Epics and Stories
Xray understands the "hierarchical" relation between Epic<=>Story. In fact, it's just one of the possible scenarios for handling parent and "sub-requirements".
...
Note that you can create and/or link a Test explicitly with an Epic, if you wish to cover it directly. (do you mean exclusively?)
Creating Test cases
Tests can be created right from the Epic/Story screen. That way, they will be automatically linked with the respective issue.
...
or later on, by using the More > Link action available from the Test issue screen (or from the Story/Epic issue screen, but using the "tested by" issue link in that case).
Organizing the Test cases
Organization is essential to easily finding Tests later on. For example, whenever you need to perform regression testing.
...
Thus, you'll need a structured way to organize your Test cases to make your work more efficient.
Possible organization scenarios
Xray provides several different options for organizing Test cases.
...
Info | ||
---|---|---|
| ||
Each organization approach provides its own benefits and its own limitations. Check out some comparison information providing better insights on the possibilities and limitations of some of these approaches |
Regression testing
The list of Tests you wish to run for regression testing may be quite dynamic and change over time, or it can be quite static; it all depends on your internal process.
...
On the other hand, you can have an implicit way of defining the Tests you wish to run during regression testing. How? Well, you may start by creating a first Test Plan for regression testing, adding all the Tests that you consider relevant. On upcoming releases, you may clone (using "issue clone" operation) the previous Test Plan and add/remove the Tests that you want.
Defining the scope of your testing using Test Plans
During the lifetime of a sprint, you will perform testing as a way to ensure you have a shippable product. Therefore, you'll need at least one Test Plan as a means to identify what you want to test and also to track its results.
...
Expand | ||
---|---|---|
| ||
|
Basic: one Test Plan per Sprint
The simplest approach is to have just one Test Plan per sprint; you can set this explicitly by setting the Sprint field on the Test Plan issue.
...
Test Executions created from the Test Plan will also, by default, inherit the Test Plan assigned version (i.e. FixVersion/s).
Test Plans for RT and NRT
Sometimes, you may prefer to distinguish regression testing (RT) from non-regression testing (NRT), for several reasons.
...
You can analyze your coverage (i.e. how your "requirements"/user stories are) from the release perspective or from each Test Plan perspective.
Multiple Test Plans
As a manager, you can define as many Test Plans as you wish.
...
The following diagram shows an example of possible different criteria for creating Test Plans.
Using Sub-Test Executions
Xray provides the ability to easily create Test Executions containing all the Tests that cover some "requirement"/Story as a sub-task of it.
...
Info | ||
---|---|---|
| ||
To be clear, you don't have to use Sub-Test Executions. However, as mentioned above they may provide some interesting benefits for some specific use cases. (mention what the benefits would be) However, if you decide to also use Sub-Test Executions, please always link them to the proper Test Plan so you can track their results at a higher level: at the Test Plan level. That way, you will be able to always track the progress from the Test Plan point-of-view. |
Tracking progress in Agile/Scrum based boards
As shown in the Agile Enhancements page, you can configure your Agile Scrum board to show immediate feedback about:
...
It's not recommended to Include in the board Test Executions coming from automated testing (i.e. during CI), as it will overload the board and not provide any real added value. Instead, you may include only Test Plan(s) as a means to group and show the consolidated results coming from automation.
Xray in Kanban based teams
You may have a look at the previous instructions for Scrum teams as most of it can also be applied to Kanban teams.
...
Thus, you can have a Test Plan to track the Tests related to the features being addressed on the Kanban board. You may complement it with additional Test Plans, namely, for tracking the regression testing. It may be a good approach to have the Test Plan with regression tests, or smoke tests, as an independent Test Plan, so it is more apparent if regression is occurring or not.
Questions
- We're not really adopting Scrum perfectly... can we use these approaches in our case?
- Yes, you may follow these same principles and adapt them to your methodology.
- We have a hardening/clean-up sprint where we wish to test thoroughly, including deep regression testing. How can this be handled?
- Hardening sprints are normally a sign that Agile is not fully being embraced. Having said that, you can have a Test Plan to track the testing performed in the scope of that sprint, adding all Tests related with user stories and other issues "implemented" on previous sprints, and thus consider the results performed in the scope of this Test Plans as being the relevant ones
- Are Test Plans, Test Executions or Sub-Test Executions child issues of an Agile board? In other words, do they belong to some Agile board?
- No. All those issues belong to some project. They can optionally be assigned to a release (through the FixVersion/s field) and to a sprint (through the Sprint custom field).
- A different thing is showing whatever issues you want in your existing boards; as those entities are implemented as issues, you can configure boards to show them. In fact, the same issues can be included (i.e. shown) in different boards.