...

1.  Xray and Jira configuration
   • Xray server/DC or Xray Cloud ? (which implictly will define if Jir server/DC or Jira Cloud are being used) 
   • Server/DC specific settings 
     • Use Jira username/password or Personal Access Token?
     • Jira username
     • Jira password
     • Personal Access Token
   • Cloud specific settings
     • Jira username
     • Jira API Token
2. On each test on the tool, associate it to an existing Xray Test issue
   1. specify Test issue key

Create/update test(s) in Xray

In Xray Tests are usually associated with Requirements (either being a User Story, Task, Epic, or other issues your team uses), this is usually how teams work and will allow Xray to calculate the Coverage.

Regarding the flow of interacting with external tools the most common case is the one where the Test is already defined in Xray and the execution is done in another tool (like a test automation tool), having the need to import those execution results back to Xray (as described above). 

Ideally when you create a new test in the test automation tool you should create a Test in Xray that will be his counterpart (this is to prevent having the need to manually create those), in order to link the execution results to the same Tests. However this may imply the need to update the Test (in Xray) each time it is changed in the eternal tool, so please take this into consideration when choosing to implement this integration.

At high-level, the flow can be depicted in the following diagram.

Image Removed

In Xray we can create three types of Tests:

• Manual, are user-assisted tests, listing an assortment of guidelines the user must follow, and the results that such behavior is expected to accomplish. They use steps with action, data and expected result to define a manual Test.
• Generic, is an agnostic Test that does not contain any structure. The only field that is related to a Generic Test is its Definition. For an automated test, the Definition field can have an ID or name that maps to the external automated Test case. An example might be a JUnit test created automatically by Xray when the JUnit XML report was imported with test results. Usually the ones used for automation as we only have one field where an unique identifier is expected that will be used by Xray to be able to connect back the executions.
• Cucumber, Cucumber is a tool that executes plain-text functional descriptions as automated tests. The language that Cucumber understands is called Gherkin, which is a business-readable, domain-specific language that describes the software’s behavior without detailing how that behavior is implemented. Gherkin is a line-oriented language that uses indentation to define the structure. Line endings terminate statements (e.g., steps). Either spaces or tabs may be used for indentation (but spaces are more portable).

For the following examples we will provide one example of each type.

Before implementing the integration, review the following questions to help guide your implementation approach.

...

Questions

...

Possible answers

...

Tips

...

Do I want to link the Test to a Requirement?

...

No, we only want to see the execution results in Xray and do not use the coverage report or will link it by hand afterwards.

Yes, I will provide the requirement in the request to create the issue to be linked since the start.

...

Coverage is a great asset that will bring visibility to all the elements of the team about the overall status of the release/project.

...

Do I need to know the type of test whenever creating a new Test?

...

Yes, because the format for creating tests is different from test type to test type, due to their kind (i.e. whether they are composed of steps or not, or if they are gherkin)

...

Can I differentiate these Tests from the “normal” ones created in Xray?

...

Yes, you can add an extra label that can be used in filters or add a link to the corresponding Test detail of your tool.

...

Adding an endpoint in the Test for the users to be able to jump from one tool to another easily is very useful.

...

...

Xray Server/DC

...

In order to find the custom field ids you must go to the administration menu and click in "Issues"

Image Removed

Once there, click on the entry "Custom Fields" in the left menu (under "FIELDS")

Image Removed

The Custom fields page should appear, in here you can filter the custom fields to find those you want by typing, foe example, "test" in the search box

Image Removed

Finally to find out the ids of the custom field you want you must press the "Actions" gear and choose one of those entries (with exception of the Delete one for obvious reasons), in our case we are clicking on "Configure".

When you are overing that entry you can already see in the bottom left of the page the endpoint with the custom field id in it.

Image Removed

Check the URL of the configuration page as you can see in the screenshot (repeat to find all custom field ids that you need).

Image Removed

Provide links on external tools to Xray entities to facilitate navigation

Sometimes, some external tools may refer to Xray entities, such as Tests, Test Runs, or others. It may be useful to provide a direct link from the external tool to Xray, to be able to see the target entity on Xray side.

The link to the UI showing the details of the target entity may not returned on the API calls; however, it's possible to easily build it following some rules.

Building the URL for the target link depends on the entity we aim to navigate to; the following table sums this information.

Create/update test(s) in Xray

In Xray Tests are usually associated with Requirements (either being a User Story, Task, Epic, or other issues your team uses), this is usually how teams work and will allow Xray to calculate the Coverage.

Regarding the flow of interacting with external tools the most common case is the one where the Test is already defined in Xray and the execution is done in another tool (like a test automation tool), having the need to import those execution results back to Xray (as described above). 

Ideally when you create a new test in the test automation tool you should create a Test in Xray that will be his counterpart (this is to prevent having the need to manually create those), in order to link the execution results to the same Tests. However this may imply the need to update the Test (in Xray) each time it is changed in the eternal tool, so please take this into consideration when choosing to implement this integration.

At high-level, the flow can be depicted in the following diagram.

Image Added

In Xray we can create three types of Tests:

• Manual, are user-assisted tests, listing an assortment of guidelines the user must follow, and the results that such behavior is expected to accomplish. They use steps with action, data and expected result to define a manual Test.
• Generic, is an agnostic Test that does not contain any structure. The only field that is related to a Generic Test is its Definition. For an automated test, the Definition field can have an ID or name that maps to the external automated Test case. An example might be a JUnit test created automatically by Xray when the JUnit XML report was imported with test results. Usually the ones used for automation as we only have one field where an unique identifier is expected that will be used by Xray to be able to connect back the executions.
• Cucumber, Cucumber is a tool that executes plain-text functional descriptions as automated tests. The language that Cucumber understands is called Gherkin, which is a business-readable, domain-specific language that describes the software’s behavior without detailing how that behavior is implemented. Gherkin is a line-oriented language that uses indentation to define the structure. Line endings terminate statements (e.g., steps). Either spaces or tabs may be used for indentation (but spaces are more portable).

For the following examples we will provide one example of each type.

Before implementing the integration, review the following questions to help guide your implementation approach.

Examples

Scenario1: creating Test(s) in Xray

In this case we're assuming:

• Requirements(s) already exist in Xray (they could have been created by a user or by some automation process)
• An automated test equivalent exist in the external tool, for each Test

And we want to...

• Create the “same” test(s) in Xray 

The following request shows how to create Tests (of different types) in Server/DC and Cloud.

Xray server/DC

As the API call is made on Jira server/DC REST API, we need to use its authentication mechanisms (e.g. basic authentication or Personal Access Tokens).

curl -H "Content-Type: application/json" -X PUT -u jira_username:jira_password --data @data.json https://jiraserver.example.com/rest/api/2/issue

curl -H "Accept: application/json" -X PUT -H "Authorization: Bearer MDkzNjIyMTEwNjQzOqb+ApTGm3af+z3eGdcyDm7xHv6R" --data @data.json https://jiraserver.example.com/rest/api/2/issue

What will change between each type of Test is the payload, so we are providing one example for each type.

Manual Test

{
   "fields": {
       "project":
       {
          "key": "XT"
       },
       "summary": "T4",
       "description": "T4: example of manual test. Details of the [external tool|http://www.example.com]",
       "labels": ["External_Tool"],
       "issuetype": {

Examples

Scenario1: creating Test(s) in Xray

In this case we're assuming:

• Requirements(s) already exist in Xray (they could have been created by a user or by some automation process)
• An automated test equivalent exist in the external tool, for each Test

And we want to...

• Create the “same” test(s) in Xray 

The following request shows how to create Tests (of different types) in Server/DC and Cloud.

Xray server/DC

As the API call is made on Jira server/DC REST API, we need to use its authentication mechanisms (e.g. basic authentication or Personal Access Tokens).

curl -H "Content-Type: application/json" -X PUT -u jira_username:jira_password --data @data.json https://jiraserver.example.com/rest/api/2/issue

curl -H "Accept: application/json" -X PUT -H "Authorization: Bearer MDkzNjIyMTEwNjQzOqb+ApTGm3af+z3eGdcyDm7xHv6R" --data @data.json https://jiraserver.example.com/rest/api/2/issue

What will change between each type of Test is the payload, so we are providing one example for each type.

Manual Test

{
   "fields": {
       "project":
       {
          "key": "XT"
       },
       "summary": "T4",
       "description": "T4: example of manual test. Details of the [external tool|http://www.example.com]",
       "labels": ["External_Tool"],
       "issuetype": {
          "name": "Test"
       },
       "customfield_12100": "Calculator",
       "customfield_10000": { "value": "Manual" },
       "customfield_10004": {
            "steps": [
				{
                    "index": 1,
                    "fieldsname": {"Test"
       },
       "customfield_12100": "Calculator",
       "customfield_10000": { "Actionvalue": "Manual"Step 1"},
            "customfield_10004": {
            "Datasteps": "input Data 1",[
				{
                    "index": 1,
                    "fields": {
                        "Action": "Step 1",
                        "Data": "input Data 1",
                        "Expected Result": "Excepted result 1"
                    }
                },
                 {
                    "index": 2,
                    "fields": {
                        "Action": "Step 2",
                        "Data": "input Data 2",
                        "Expected Result": "Excepted result 2"
                    }
                },
                 {
                    "index": 3,
                    "fields": {
                        "Action": "Step 3",
                        "Data": "input Data 3",
                        "Expected Result": "Excepted result 3"
                    }
                },
                 {
                    "index": 4,
                    "fields": {
                        "Action": "Step 4",
                        "Data": "input Data 4",
                        "Expected Result": "Excepted result 4"
                    }
                }
            ]
        }
   },
        "update":{
            "issuelinks":[
                {
                    "add":{
                    "type":{
                        "name":"Tests"
                    },
                    "outwardIssue":{
                        "key":"XT-227"
                    }
                    }
                }
            ]
        }
}

...

JIRA_API_BASE_URL=<your_jira_cloud_url>/rest/api/2
JIRA_USERNAME=<jira_username>
JIRA_API_TOKEN=<jira_api_token>
TEST_ISSUE_KEY=<test_issue_key>

curl -H "Content-Type: application/json" -X PUT -u $JIRA_USERNAME:$JIRA_API_TOKEN   --data @"payload_cloud.json" $JIRA_API_BASE_URL/issue/$TEST_ISSUE_KEY

{
    "update" : {
        "description" : [{ "set": "[EXTOOL_1.2|http://www.example.com]"}]
    }
}

After making the request, the Test issue will  have the our field (i.e. "description") with a link to the test on the external tool.

Image Removed

...

Xray Server/DC

• Update Test Steps

Xray Cloud

• Update Test Type
• Update Unstructured Test Definition
• Update Gherkin Test Definition
• Update Test Step
• Update Test Folder

Tips

Use wiki markup to not only to format text but also to provide additional information or access to it

The Description field support Jira wiki markup, so you can use formatting to insert. a link for instance.

There can be some subtle differences between Jira server/DC and Jira Cloud, so we advise you to try it out on both deployment types.

Use labels

When creating a new Test you have the ability to associate a label to it.

Implementation Checklist

For this purpose, please make sure your tool provides these:

1.  Xray and Jira configuration
   • Xray server/DC or Xray Cloud ? (which implicitly will define if Jira server/DC or Jira Cloud are being used) 
   • Server/DC specific settings 
     • Use Jira username/password or Personal Access Token?
     • Jira username
     • Jira password
     • Personal Access Token
   • Cloud specific settings
     • Jira username
     • Jira API Token
2. On each test on the tool, associate it to an existing Xray Test issue
   1. specify Test issue key
3. Discover the custom fields ids of the entities being used

...

/issue/$TEST_ISSUE_KEY

{
    "update" : {
        "description" : [{ "set": "[EXTOOL_1.2|http://www.example.com]"}]
    }
}

After making the request, the Test issue will  have the our field (i.e. "description") with a link to the test on the external tool.

Image Added

Tips

Use wiki markup to not only to format text but also to provide additional information or access to it

The Description field support Jira wiki markup, so you can use formatting to insert. a link for instance.

There can be some subtle differences between Jira server/DC and Jira Cloud, so we advise you to try it out on both deployment types.

Use labels

When creating a new Test you have the ability to associate a label to it.

Implementation Checklist

For this purpose, please make sure your tool provides these:

1.  Xray and Jira configuration
   • Xray server/DC or Xray Cloud ? (which implicitly will define if Jira server/DC or Jira Cloud are being used) 
   • Server/DC specific settings 
     • Use Jira username/password or Personal Access Token?
     • Jira username
     • Jira password
     • Personal Access Token
   • Cloud specific settings
     • Jira username
     • Jira API Token
2. On each test on the tool, associate it to an existing Xray Test issue
   1. specify Test issue key
3. Discover the custom fields ids of the entities being used

Xray internals

How is Xray data stored

Xray server/DC

Xray data is stored on Jira's database, mostly in Xray specific tables (e.g., for test runs).

As Xray reuses Jira concepts (e.g., issue types,  links, custom fields), some of its information is stored on standard Jira entities/tables (e.g., Xray issues standard Jira metadata - summary, description, fixVersion, component, issue links used for coverage purpose).

Xray also provides a bunch of custom fields (CFs) to present or manage some information; some of these are calculated, some are stored in the database (on Jira tables for CFs).

Therefore, a Jira database's dump contains all relevant information from Jira and Xray.

Some of this information is exposed and managed using the Jira's REST API or through Xray's REST API.

Xray Cloud

Due to the way Atlassian cloud architecture works,  apps such as Xray manage their data in their own way (e.g., in the app cloud infrastructure or in a cloud DB provider). Jira related information (e.g., issues, links) is managed by Atlassian and stored in Atlassian's cloud imfrastructure.

Some of this information is exposed and managed using different APIs:

• Jira's REST API: for Jira managed information (e.g., issues, standard fields, CFs, links, attachments, comments) 
• Xray's REST API: mostly to import test results and other more "operational" tasks
• Xray's GraphQL API: for most of Xray data and their relations; a quite flexible API

Depending on the use case, one or a combination of the previous may be required. Note that Jira's REST API uses the authentication mechanisms of Jira while Xray's APIs use their own authentication mechanism based on an API key pair (client_id+client_secret).

It's possible to generate a dump of the Xray managed data; note that this dump doesn't contain the information managed by Jira/Atlassian as that follows a different mechanism.

Custom fields

Generically speaking, custom fields (CFs) are like custom attributes. They exist as a means to add additional, well-defined information, to certain entities.

...

But whenever "custom fields" come to mind, we may be referring to different kinds of CFs (as Xray also provides some kind of custom fields), and this may cause some misunderstanding. These are the types of fields we may be referring to:

• Jira's traditional "custom fields": In Jira terminology, custom fields (CFs) are a way to collect information that isn't available on the default fields. In other words, Jira CFs are a way to add additional metadata to issues.; we can think on them as issue level "attributes", having a id, name, type, and value. Jira CFs are defined at administration level (server/DC, cloud), and may be included on some screens, for certain issue types. These fields can easily be included on Jira standard gadgets, listings, etc. A field has a visual representation and a text representation. Some fields may be searchable and included on JQL queries. Some fields store data, while others are calcula. Some calculated. It's usual for Jira apps/plugins to create specific Jira CFs to present or manage some of their data as seen ahead.

• Xray managed Jira CFs:  Xray data managed as a Jira CF (server/DC, cloud)server/DC, cloud); some of these CFs are read-only, while other are read/write. These are also Jira CFs in strict sense. Xray server/DC has many of these; Xray cloud has very few, due to the way apps integrate with Jira cloud. On Xray server/DC some of these fields are stored in Jira's database, others are stored on lucene.

• Xray's test Test step custom fields: Xray proprietary way of defining named attributes for steps on "manual" scripted tests; definable at project level. These are not listable/includable on Jira issues/gadgets nor exportable using standard Jira mechanisms; they are also not visible to Jira's REST API.
• Xray's Test Run custom fields: Xray proprietary way of defining named attributes for Test Runs; definable at project level. These are not listable/includable on Jira issues/gadgets nor exportable using standard Jira mechanisms; they are also not visible to Jira's REST API.

Tips

• try to use the proper APIs as much as possible and avoid using CFs to obtain or modify data

References

• Xray Cloud
  • Xray managed custom fields
  • Test step "custom fields"
  • Test Run "custom fields"
• Xray server/DC
  • Xray managed custom fields
  • Test step "custom fields"
  • Test Run "custom fields" Run CFs

Recommendations, tips,  and concerns

...