Upgrading
What should I do to update/upgrade my current version of Xray?
If you have Xray installed in a standard JIRA server, you can just update it as detailed in FAQ (please check the JIRA compatibility matrix first).
If you are using JIRA Data Center, you should restart all nodes after installing the new version of Xray.
Requirements
I cannot find the "Requirement" issue type. Why?
There is no "Requirement" issue type installed by Xray. In fact, Xray does not install any requirement related issue types. If you had the opportunity to attend one of our webinars, namely the Xray walkthrough, you may see that with Xray you are able to define which issue types Xray should considered as requirements so you can cover them with Tests. Besides the standard "Story" and "Epic" JIRA issue types, JIRA allows you to create your own issue types. You just need to configure Xray properly, as detailed here, to let Xray internaly treat those issue types as being requirements.
I'm unable to see the the "Test Coverage" panel and the "Requirement Status" in my requirements. Why?
You need to assure two things:
- that your project is defined as being a requirements project;
- that your issue type is configured to be a requirement issue type.
Both can be achieved in Xray's administration (see FAQ for instructions), although the first one may be done also in the project settings page.
The "Requirement Status" custom field does not show the "expected" value. Why?
The "Requirement Status" is a calculated field, so it shows the information for some version depending on a Xray setting available in Custom Field Preferences. Currently, it is not possible to specify explicitly the version that you want to calculate the coverage on. Please read Using custom fields for a deeper explanation.
Tests
Can I use a Test for testing different system versions?
Yes. A Test is like a test template. You can use it in multiple Test Sets or Test Executions, which may be assigned with different versions.
How do I associate a Test with a project version?
The association of a Test with a project version is not direct. Your tests are written in order to validate some requirement, that itself belongs to a specific version. Although a Test may be associated with a requirement of a specific version, you are able to run tests for the versions you would like to. In terms of requirement coverage, you have different behaviors to choose from in the administration settings (see Requirement Coverage Strategy). You may also to have a look at Test Versioning.
Do you support versioning of Tests?
Please have a look at Test Versioning.
How do I define which tests should be used for regression testing?
Please take a look at FAQ.
Can I links Tests to another Tests?
Yes. You can create normal links between Test issues, as you would do for any standard JIRA issue. Xray will not process those relations in any special way; it will ignore them unless they are of "tests" or "tested by" relations.
The "TestRunStatus" custom field does not show the "expected" value. Why?
The "TestRunStatus" is a calculated field, so it shows the information for some version depending on a Xray setting available in Custom Field Preferences. Currently, it is not possible to specify explicitly the version that you want to calculate the coverage on. Please read Using custom fields for a deeper explanation.
Test Sets
Can I clone a Test Set? Will that clone also the Tests that belong to the Test Set?
You can clone a Test Set and the cloned Test Set will have the same Test Sets that were in the original Test Set. The tests themselves are not cloned. If you really want to clone a bulk list of tests, then you must clone them manually (you may find add-ons for this purpose), and then associate them to the new Test Set. In fact, whenever you clone either Test Sets, Test Executions or Test Plans, the tests that they contain are not cloned; just the internal association.
Test Executions
How do I create a Test Execution for all failed tests (or other test status) of a previous execution?
You can do it easily within the Test Plan screen, by pressing the "Create Test Execution" button. You may also create a new Test Execution for some of the tests listed in the table bellow the Filters, by pressing the bulk icon 
and then pick the relevant tests (you may use filter to help you out), and then choose the Create action in the actions icon 
, available on the right side of the table.
Can I run a Test Execution multiple times?
Kind of. A specific Test Execution represents a execution task of a list of tests, for a specific version/revision, done in a specific time. With Xray, the normal way executing a set of tests multiple times is by creating a Test Execution for each time you want to execute the tests. All you need to do is clone an existing Test Execution; the newly created Test Execution will contain the same list of tests, in clean state since you haven't executed them yet.
Do I have to use Test Environments?
No. But if you execute the same tests on different environments you certainly will find it as the most useful and accurate way of testing a requirement.
Do I have to execute tests in all Test Environments?
No. It’s up to you to organize your Tests per environments, this is the reason we use the Test Environment Custom Field as a label.
Can I take screenshots and attach them whenever executing tests?
Xray does not provide by itself this feature out-of-the-box. However, you may use a simple tool dedicated to take screenshots (e.g. LightShot) and easily attach the screenshot to the evidences during the execution of the test.
Test Plans
A “Test Plan” seems like a group of Test Executions or a group of Tests…
Yes, a bit of both in fact. A Test Plan contains a list of Tests. For calculating the status of each test, it takes into account test runs from Test Executions explicitly linked to the Test Plan, through the field “Test Plan”
Can I clone “Test Plans”?
Yes. The list of related Tests is copied; Tests are not cloned.
Can I have multiple Test Plans for a version?
Yes, if you wish to track different groups of Tests (e.g. regression tests vs tests for new requirements).
Do I have to use "Test Plans"?
You don’t but soon you will find great advantages if you start using them.
Agile Integration
Can I add a Sub Test Execution to multiple Requirement issues?
A sub-task can only have one parent issue, so it won’t be possible to have multiple requirement issues with the same Sub Test Execution. However, you can add Tests from another Requirement issue to a Sub Test Execution. You’ll be able to execute them but you are going to have the same behavior as the Test Execution, while using Agile Boards. So, if you intend to track the executions in the Agile Board, we recommend you to have a Sub Test Execution for each Requirement issue.
Do I have to associate all Tests that are testing a Requirement with a Sub Test Execution?
The Tests already linked with the Requirement issue will be automatically added to the Sub Test Execution. You can remove them if you intend to. If you add a new Test to the Requirement issue after creating the Sub Test Execution, then you’ll need to add them manually.
Configuration
Is it possible to use custom workflows for the Test related issues? Does Xray install or enforces any workflow?
Xray does not install any "default" workflow for the issues it provides. You may define whatever worfklow you think fits best your scenario/organization. Xray is workflow aware though, and provides some configurations specificaly for that purpose (see Global Preferences page),
Licenses
How do I install a license provided by Xpand IT?
Once you have a license provided by Xpand IT, you should perform the following steps:
- On your JIRA Administration, select the option "Manage Add-Ons"
- Go to "Xray Configuration", section "License Management"
- Click on the button "Click here to install a license directly from Xpand IT"
- Put your license on the text box. Remember, this shoud be a license provided by Xpand IT and not via Atlassian Marketplace
- Click on add
- After the executing the previous steps, your license should be installed.
How can I replace my marketplace license with a license provided by Xpand IT?
If you need to replace your marketplace license with a license provided by Xpand IT, you should perform the following steps:
- On your JIRA Administration, select the option "Manage Add-Ons"
- Click on the Add-On "Xray". Detail about the add-on will be shown
- Edit the license key, and remove all the text, then click on "Update". Your marketplace license has been removed.
- Go to "Xray Configuration", section "License Management"
- Click on the button "Click here to install a license directly from Xpand IT"
- Put your license on the text box. Remember, this shoud be a license provided by Xpand IT and not via Atlassian Marketplace
- Click on add
- After the executing the previous steps, your license should be installed.
Installation
What JIRA versions do you support?
Please see our Installation page.
Do you support JIRA Data Center (i.e. high availability)?
Yes.
Rollback to a previous version of the add-on
If you need to rollback you installation to a previous version of the add-on, perform the following steps:
- Uninstall your current version of the add-on: Navigate to your JIRA administration, section Manage Add-ons, select the "Xray" add-on and click on "Uninstall".
- Download the version that you want from https://marketplace.atlassian.com/plugins/com.xpandit.plugins.xray/versions (Click over the version and choose "download this version")
- After executing the previous step, also in the Manage Add-Ons page, select "Upload Add-on", browse to the file downloaded in the previous step.
- The add-on should now be installed with the previous version.
How to enable debug logging
If you need to enable logging for the add-on, you should perform the following steps:
- Go to the JIRA Administration, section System
- Selection option "Logging & Profiling"
- In section "Default Loggers", click on "Configure logging level for another package"
- Set the Package name to com.xpandit and Logging Level to DEBUG and click on Add.
Once you performed the previous steps, you should reproduce your problem and retrieve the atlassian-jira.log file.
Import/Export
Import JSON execution results from cucumber returning error
If you get a message like : “Error importing execution results to database: User XXX does not have permission to execute test execution with key TEST-XX”
- Check that User that upload the result file have the RESOLUTION permission
- Check that Test Execution is not in a Workflow status marked as non-executable in Xray "Global Preferences"
Is it possible to import Tests and link them to other issues (e.g. requirements)?
Yes. You can add one or more columns in your CSV and then map them to "Link Tests"'. Please see the documentation of FAQ.
Can I migrate my Tests from one JIRA server to another one?
Yes, with some constraints. Please see the proper section within the documentation of FAQ.
How can I migrate my legacy executions to JIRA?
The built-in Test Case Importer is only able to import Test specifications and not results. There is no immediate way to achive what you need. However, there are some options.
One way eventualy would be by using the REST API: FAQ. This will require some development effort from your side and will also require that you have the Tests already created in JIRA.
Another option could be using the built-in JIRA CSV importer (not Xray's Test Case Importer), in order to create the Test Executions with the Tests. However, you would still need to use the REST API in order to submit the results for the Tests contained in those Test Executions.
Integrations
Can I trigger/start Jenkins/Bamboo builds from Xray?
Currently this is not possible.
JQL functions
When searching issues, custom fields (e.g. TestRunStatus) do not return the correct values
Columns presented in the results do not take into account the context of the JQL query. The TestRunStatus custom field will always present the same result independently of the JQL query.
For instance, imagine that tou want to see the tests of a given Test Execution, using testExecutionTests(...), and their results. If a Test A is executed by TE 1 and TE2 and you're filtering by TE2; you will get the "global" status calculated, and this can be the value in TE1 if TE1 is the latest test run.
Contact
Can I send you guys an email?
We really prefer that you contact us through our Service Desk, so we can properly address your question on time and by the right person.
I would like a demo, would it be possible?
Sure. Please leave us a message in our Service Desk and we'll come back to you. We are also publishing webinars that you can assist.