Testing Process

Is it possible to use Time Tracking with Xray? Is it possible to log time associated with the running of Test Runs?

Yes. Please refer to the Time Tracking documentation.

Upgrading

What should I do to update/upgrade my current version of Xray?

Upgrades are managed automatically by our team and the process is transparent to you.

Requirements

I cannot find the "Requirement" issue type. Why? 

There is no "Requirement" issue type installed by Xray. In fact, Xray does not install any requirement-related issue types. If you had the opportunity to attend one of our webinars, particularly one similar to the Xray walkthrough, you will learn that with Xray, you are able to define which issue types Xray should consider as requirements so you can cover them with Tests. Besides the standard "Story" and "Epic" Jira issue types, Jira allows you to create your own issue types. You just need to configure Xray properly, as detailed here, to let Xray internally treat those issue types as requirements.

I'm unable to see the "Test Coverage" panel in my issues. Why? 

You need to ensure that your issue type is configured to be handled as a Covered Issue Type (i.e., a testable entity) This can be done in Project Settings: Test Coverage.

Also check out the Quick Setup for an overview of the initial setup steps you need to follow.

Tests

Can I use a Test for testing different system versions?

Yes. A Test is like a test template. You can use it in multiple Test Sets or Test Executions, which may be assigned with different versions.

How do I associate a Test with a project version?

The association of a Test with a project version is not direct. Your Tests are written in order to validate some requirement, that itself belongs to a specific version. Although a Test may be associated with a requirement of a specific version, you are able to run Tests for the versions you like. In terms of requirement coverage, you have different behaviors to choose from in the administration settings (see Requirement Coverage Strategy). You may also to have a look at Tips for implementing Test Versioning.

Do you support versioning of Tests?

Please refer to Tips for implementing Test Versioning.

How do I define which Tests should be used for regression testing?

Please take a look at Tips for organizing tests.

Can I link Tests to other Tests?

Yes. You can create normal links between Test issues, as you would do for any standard Jira issue. Xray will not process those relationships in any special way; it will ignore them unless they are of "tests" or "tested by" relationships.

Test Sets

Can I clone a Test Set? Will that clone the Tests that belong to the Test Set?

Not yet.

Test Executions

How do I create a Test Execution for all failed tests (or other test status) of a previous execution?

You can do it easily within the Test Plan screen via the "Create Test Execution" button. You may also create a new Test Execution for some of the tests listed in the table below the Filters. Select one or more tests (you may use a filter to help you out) and then choose the action available in the button next to the bulk selection (e.g., "(xx) Selected").

Can I run a Test Execution multiple times?

Not yet.

Do I have to use Test Environments?

No, but if you execute the same tests in different environments, you certainly will find it as the most useful and accurate way of testing a requirement.

Do I have to execute tests in all Test Environments?

No. It’s up to you to organize your Tests per environment.

Can I take screenshots and attach them when executing tests?

Xray does not provide this feature out-of-the-box. However, you may use a simple tool dedicated to taking screenshots (e.g., LightShot) and easily attach the screenshot to the evidences during the execution of the test.

Test Plans

A “Test Plan” seems like a group of Test Executions or a group of Tests.

A bit of both. A Test Plan contains a list of Tests. For calculating the status of each test, it takes into account test runs from Test Executions explicitly linked to the Test Plan, through the field “Test Plan”.

Can I clone “Test Plans”?

Not yet.

Can I have multiple Test Plans for a version?

Yes, if you wish to track different groups of Tests (e.g., regression tests versus tests for new requirements).

Do I have to use "Test Plans"?

You don’t have to, but there are great advantages if you start using them.

Agile Integration

Can I add a Sub-Test Execution to multiple coverable issues (e.g., requirements)?

A sub-task can only have one parent issue, so it won’t be possible to have multiple requirement issues with the same Sub-Test Execution. However, you can add Tests from another Requirement issue to a Sub-Test Execution. You’ll be able to execute them, but you are going to have the same behavior as the Test Execution, while using Agile Boards. So, if you intend to track the executions in the Agile Board, we recommend that you have a Sub-Test Execution for each coverable issue.

Do I have to associate all Tests that are testing a coverable issue (e.g., requirement) with a Sub-Test Execution?

The Tests already linked with the coverable issue will be automatically added to the Sub-Test Execution. You can remove them if you want. If you add a new Test to the Requirement issue after creating the Sub-Test Execution, then you’ll need to add them manually.

Configuration

Is it possible to use custom workflows for Test-related issues? Does Xray install or enforce any workflow?

Xray does not install any "default" workflow for the issues it provides. You may define whatever worfklow you think fits best your scenario/organization. Xray is workflow-aware though, and provides some configurations specifically for that purpose (see Project Settings: Miscellaneous).   

Licenses

Where can I find the pricing for Xray and licensing information?

More information can be found in the Xray section of the Atlassian's Marketplace.

What happens when my license expires?

If you have a paid or trial license and it expires, you won't be able to use the Xray app.

Do I need a license?

Yes. Even for trying Xray, you need a trial license. More information can be found in the Xray section in Atlassian's Marketplace.

Can I renew my trial license?

Yes, up to 5 times. More information can be found in the Xray section in Atlassian's Marketplace.

Installation

What happens if I uninstall Xray?

If, for some reason, you need to uninstall Xray, you can do it safely. No relevant data is removed from Jira. This means that Xray issue types, custom fields, Xray settings and all recorded information related with testing is kept.

Note: If you uninstall and install Xray once again, nothing special should happen. In other words, it should be idempotent. 

Integrations

Can I trigger/start Jenkins/Bamboo builds from Xray?

Currently, this is not possible.

Security, Storage, Privacy and Compliance

As this product is cloud based, what are the important documents on documents on security, storage, privacy, compliance?

Please see our EULA and Security Statement. Xblend has also successfully completed Atlassian's security self-assessment program.

Xray Cloud integrates with Atlassian Cloud infrastructure; therefore, we'd recommend having a look at a Atlassian documentation on security, reliability, privacy and compliance.

Contact

Can I send you guys an email?

We really prefer that you contact us through our Service Desk, so the right person can properly address your question asap.

I would like a demo. Is that possible?

Sure. Please leave us a message in our Service Desk and we'll get back to you right away. We also have webinars (both live and recorded) that you can attend.