In very simple terms, a Test is a sequence of steps coupled with conditions or variables, test inputs and an expected result. It is intended to establish the quality, performance or reliability of a piece of  system, i.e., test target. Usually, every requirement or objective the test target is expected to achieve needs at least one Test. The success of the Test is determined by comparing expected and actual results.

Test Types

Tests in Xray can be of two types: automated or manual.

• Automated
• Manual

Automated

Automated Tests are executed through an external tool which controls the execution of tests and the comparison of actual outcomes to predicted outcomes. They can automate some repetitive tasks in a formalized testing process already in place, or add additional testing that would be difficult do perform manually. The following automated Test types are available:

• Cucumber
• Generic

In Jira's administration, you may add other values for the "Test Type" custom field (see list below). For example, you may want to differentiate automated tests that may exist simultaneously in a project.

However, internally they will and must be handled as one of the two default Test Types for automated tests: "Cucumber" or "Generic".

Note: "Test Type" custom field values not shown above are not supported.

Cucumber

Cucumber is a tool that executes plain-text functional descriptions as automated tests. The language that Cucumber understands is called Gherkin, which is a business-readable, domain-specific language that describes the software’s behavior without detailing how that behavior is implemented. It is line-oriented language and uses indentation to define structure. Line endings terminate statements (e.g., steps). Either spaces or tabs may be used for indentation (but spaces are more portable). Most lines start with a keyword.

For more information on the Cucumber Basics, please consult the Wiki and Tutorials at: http://cucumber.io

Fields

Cucumber Tests are divided in two Scenario Types:

•  Scenario,  a set of instructions exclusively composed by constant values, which, when executed by a tool, will only give a static result (e.g., 2 + 3 = 5)
•  Scenario Outline,  a generic set of instructions containing variables, which, when executed by a tool, will prompt varied results according to the specified input.

- The behavior to be reproduced by the Cucumber tool in entered in the Steps field.

Generic

A Generic Test is an automated agnostic Test that is executed by an external tool. Execution results will then be imported back to Jira. An example might be a JUnit test that is used for quality assurance.

Fields

The only Field that is related to a Generic Test is its Definition. Here, the automated external test may be defined or referenced through an open text field. 

The Generic Test Definition field supports the Jira markup wiki language.

Manual

Manual Tests are user-assisted tests, listing an assortment of guidelines the user must follow, and the results such behavior is expected to accomplish. If the actual test result matches the expected test result, the test is considered as passed; otherwise, it is marked as failed.

Fields

Manual Tests are segmented in Manual Steps. Each Manual Step is composed of 4 fields (Step, Data, Expected Result and Attachments) with Step being the only mandatory one.

• Step - The action to be reproduced by the tester
• Data - Any data the related step requests (e.g., login credentials) to be used by the tester 
• Expected Result - The behavior the step should accomplish 
• Attachments - External files that provide additional information related with the corresponding Manual Step. Attachments can only be added or removed in the Test Issue view page.

The Step, Data and Expected Result fields support the Jira markup wiki language.

Actions

• Create

• Edit

• Clone

• Export to Cucumber

Create

In order to create a new Test Issue, 

Step 1: Click the Create Issue at the top of the screen to open the Create Issue dialog box / page.

Step 2: Select the  Project  and on  Issue Type,  select Test.

Step 3: Under the General tab, type a Summary for the test and fill all appropriate fields — at least, the mandatory ones marked with an asterisk.

Step 4: On the Test Details tab, select the Test Type and complete all appropriate fields;

Step 4a: If you are writing a Manual Test:
- Select the Test Type as Manual. 
- After filling out the Manual Steps fields, press the Add button to jump to the next Manual Step. 
- Use the Pencil button to edit a previously entered Manual Step, and press the Update button when done. 
- Use the Bin button to delete a previous entered Manual Step.
- To change a Manual Step position in the Test, drag the left area of the row and drop it to the desired place.

Step 4b: If you are writing a Cucumber Test:
- Select the Test Type as Cucumber. 
- On Cucumber Type, select either Scenario or Scenario Outline as test type. 
- On Cucumber Scenario, enter the Cucumber Scenario clauses in Gherkin.

Step 4c: If you are writing a Generic Test:
- Select the Test Type as Generic.
- On Generic Test Definition, enter the definition for the automated generic Test.

Optional:
- Click on the Test Sets tab to associate the current issue with Test Sets.
- Click on the Pre-Conditions tab to associate the current issue with Pre-Conditions.
- Click on the Test Plans tab to associate the current issue with Test Plans. 
- Click on the Link Issues tab to link with other issues and specify the nature of the association.
- To create a series of similar tests – within the same  Project  – check the  Create another box at the bottom of the dialog. 

Step 5: When you are satisfied with the content of your test, click the Create button.

Step 5a: If you checked the Create another box, a new Create Issue dialog appears. This issue is automatically pre-populated with your previous test details, while leaving the Summary field blank.

Edit

Editing an existing Test can be done in one of two ways:

• Using the Edit Issue dialog box
• Using the Inline Edit for JIRA Plugin

Using the Edit Issue dialog box

The Edit Issue dialog box is similar to the Create Issue dialog box, and can be accessed through the following steps:

Step 1: Click the Edit button (at the top-left of the 'view issue' page) to open the Edit Issue dialog box.

Step 2: Modify your Test's details in the appropriate fields of the Edit Issue dialog box General tab.

Step 3: Click on the Test Details tab to edit the Test Type and any complementary fields.

Step 3a: If you are editing a Manual Test:
- To input a new Manual Steps field, use the empty areas at the bottom and click the Add button.
- Use the Pencil button to edit a previously entered Manual Step, and click the Update button when done.
- Use the Bin button to delete a previously entered Manual Step.
- To change a Manual Step position in the Test, drag the left area on the that row and drop it to the desired place.

Step 3b: If you are editing a Cucumber Test: 
- To change test type on Cucumber Type, select either Scenario or Scenario Outline.
- In the Steps field, you can edit a previously entered or create a new Cucumber Scenario clauses in Gherkin.

Step 3c: If you are editing a Generic Test:
-In the Steps field, you can edit a previously entered or create a new Definition for the automated generic Test.

Optional:
- Click on the Test Sets tab to edit Test Set associations.
- Click on the Pre-Conditions tab to edit Pre-Condition associations.
- Click on the Link Issues tab to edit Links with other issues.

Step 4: Click the Update button to save your changes.

Inline Editing Xray Tests

Xray Test issue sections support inline editing similar to Jira native and custom fields inline edit. This prevents full page reloads when modifying fields.

Inline Edit Fields

To inline edit a field, just hover over the field view area and click it.

The following Test fields support inline edit:

• Type: The Test issue type
• Definition: The automated generic test definition (only visible for Generic tests)
• Cucumber Type: The Cucumber Scenario type (only visible for Cucumber tests)

To inline edit automated Test issues:

Step 1b: If you are inline editing a Cucumber Test: 
- You can hover over the actual Test Type, click it, and change it to another type that is available on the select box.
- You can hover over the actual Cucumber Type, click it, and change it to another type that is available on the select box.
- You can hover over the Steps field, click it, and edit its contents.

Step 1c: If you are inline editing a Generic Test:
- You can hover over the actual Test Type, click it, and change it to another type available on the select box.
- You can hover over the Definition field, click it, and edit its contents.

Notes: On Jira versions greater or equal to 7.3.0, the Definition field supports Jira's Rich Text Editor. To enable the Rich Text Editor, you have to set the renderer of the definition custom field to  Wiki Style Renderer.  It is also possible to use Rich Text Editor on Jira versions greater than 7.2.0 by enabling the experimental Rich Text Editor in Jira configurations, but this feature is not totally supported by Xray.

Manual Test Steps Table

In addition to inline editable fields, Xray provides a Restful table for inline editing manual Test steps direcly from the Test view issue screen.

From this table you can:

Add Steps: After entering the Test step details (Step, Data and Expected Result fields) using the bottom table row text boxes, click on the Plus icon for creating a new step. The new step will be added in the last position in the steps table. 

Remove Steps: The last column for each step contains a Bin button for removing steps. After clicking the button, you will be prompted with a confirmation message to delete the step. This operation cannot be undone.

Edit Steps: When hovering over Step, Data and Expected Result fields, they get highlighted and if you click them, the step row visualization will change to edit mode. From here, you can edit the fields and click on the Update button that appears in the last column to save changes or you can cancel the changes by clicking on the Cancel button. Either way, the row will be back in View mode.

Move Steps: You can move a Test step Up or Down in the Test steps table in one of two ways:

• • Click on the drag area in the first column of each step and drag the step to the desired position.
  • Click on the actions menu and choosing the "Move To.." option. This method allows you to directly move a step to a specific position.

Add Attachments: While in View or Edit modes, you can add attachments to a step using the Plus sign appearing in the Attachments column. After clicking the button, you will be prompted with a dialog to choose the attachments to add.

Remove Attachments: While in View or Edit modes, you can remove attachments of Test steps using the Bin button that appears when hovering over existing attachments. After clicking the button, you will be prompted with a confirmation message to remove the attachment or not.

Copy Step Fields: You can copy the Step, Data and Expected Result fields to the clipboard by hovering over these fields and choosing the Copy icon next to each highlighted field. If the copy is successful, a green border will appear momentarily around the Copy icon.

Expand Fields: If the text boxes are too small to show all the information, you can expand the Step, Data and Expected Result fields by clicking the Expand icon next to each highlighted field. An inline dialog, with a bigger area, will be opened with the contents of the field and a wiki notation toolbar to assist you in writing your information. This operation is available in both View and Edit modes for manual Test steps. 

Clone Steps: To clone an existing Test step, click on the actions menu and choose the "Clone" option. A confirmation message will appear. Once you confirm, a new step will be created in the last position, containing all information of the cloned step (Step, Data, Expected Result and Attachments).

Changing the Type of a Test

In the Global Preferences, Xray allows you to configure if a test definition is deleted or not.

If enabled, the previous Info details will be deleted.

Otherwise, the Info details will not be deleted, and it'll be possible to undo the Test Type change operation and keep the previous information.

Additionally, in both cases, because each pre-condition must be of the same Type of the Test, any associated Pre-Conditions will be lost.

Clone

Cloning a Test issue creates a new Test with all the information of the cloned Test, except its executions. Therefore, the cloned issue will not be associated with any Test Execution issue.

Step 1: Select More > Clone. The Clone Issue screen will appear.

Step 2: You can edit the clone Test's Summary if you wish.

  - If the test contains links to other issue(s), you can select whether or not to include the links in the new clone test.

  - If the test contains sub-tasks, you can select whether or not to create the sub-tasks in the new clone test.

  - If the test contains attachments, you can select whether or not to include the attachments in the new clone test.

Step 3: Click Create.

Configure Test Run Columns

More info in Test Runs.

Configure Test Set Columns

Configure Test Plan Columns

Create Pre-Condition

To create a Pre-Condition directly associated to the current Test from the Test issue view screen:

Step 1: Open the Test issue view screen you wish to create a Pre-Condition with.

Step 2: Click the Create Pre-Condition button (under the Pre-Condition section on the view page) to open the Create Issue dialog box.

Step 3: Type a Summary for the Pre-Condition and complete all appropriate fields — at least, the required ones marked by an asterisk.This issue is automatically pre-populated with your previous Test issue Fix Version/s.

Step 4: On the Pre-Condition Details tab, complete all appropriate fields. This issue is automatically pre-populated with your previous Test type.

Step 5: Optional. Click on the Tests tab to edit Test associations to associate more Tests. This issue is automatically pre-populated with your previous Test key.

Step 6: Click Create.

Associate Pre-Conditions

To associate Tests with a Pre-Condition from the Test issue view screen:

Step 1: Open the Test issue view screen you wish to associate a Pre-Condition with.

Step 2: Click the Associate Pre-Condition button (under the Pre-Condition section on the view page) to open the Associate Test with Pre-Condition dialog box.

Step 3: Select or search the Pre-Condition issues to be associated with the Test:

Select Pre-Condition Issues -  In order to select the Pre-Conditions to be associated, you can:

- Input the desired Pre-Condition Issue Key on the Pre-Conditions field.
- Click on the Down Arrow on the Pre-Conditions field and select the Test Set from its History Search list.
- Click on the + button which will prompt an Issue Selector Pop-Up. 
- Click on the Search tab to use the Find Jira issues browser.

Search  Pre-Condition  Issues -  In order to search for Pre-Conditions to be associated you can, under the Search tab:

- Browse for the desired Pre-Condition Project or terms it Contains.
- Click on the More button to add more browse fields such as Label, Fix Version, Saved Filter and Workflow Status.
- Click on the Magnifier button after filling the browse fields to get a list of matching entries. 
- Click on the Advanced button to perform a JQL search.

Step 4: Click Add.

Remove Pre-Conditions

To remove a Pre-Condition association from the Test issue view screen:

Step 1: Open the Test you wish to remove the Pre-Condition association.

Step 2: On the Pre-Conditions table, hover over the desired Pre-Condition row and click on the Trash button that appears in the end of the row. You will be prompted with a confirmation dialog.

Reorder Pre-Conditions

Export to Cucumber

Cucumber Tests in Jira can be exported to feature files ready to be executed in Cucumber. The generated feature files follow the rules defined in Export Cucumber Features and take into account the context of the Jira Test issue.

A single Cucumber feature file should be generated and prompted for download, unless the target Test is associated with multiple Requirements, which, in that case, will prompt a FeatureBundle zip file containing a single Cucumber feature file for every Requirement associated with the target Test.

To export a Cucumber Test issue to feature files, 

Step 1: Open the Cucumber Test issue view page. 

Step 2: Select More > Export to Cucumber. The file will be downloaded to the browser.

Export Test to XML

Xray Test issues can be exported to XML. This export differs from the Jira default action for exporting to XML. Rather than exporting all fields, the Xray export action will only export to XML the most relevant fields for Tests, such as the Test definition fields, Pre-Conditions and some issue-tracking information.

To export a Test issue to XML, 

Step 1: Open the Test issue view page. 

Step 2: Select More > Export Test to XML. The resulting XML file will be downloaded to the browser.

Export Test Runs to CSV

Xray Test Runs can be exported to CSV. Xray export action will export to CSV the most relevant fields for Test Runs such as:

• • Test Key
  • Test Execution Key
  • Executed By
  • Assignee
  • Start
  • Finish
  • Defects issue keys (list separated by ",")
  • Attachment JIRA links (list separated by ",")
  • Comment
  • Status

To export a Test Run to CSV, 

Step 1: Open the Test issue view page. 

Step 2: Select More > Export Test Runs to CSV. The resulting CSV file will be downloaded to the browser.

Export Test as Text

Automated Generic Test issues can be exported to a plain text file. Only the Test Definition field will be included in the plain text file. This action can be useful when executing a Generic Test outside of Jira.

To export an Automated Generic Test definition to plain text, 

Step 1: Open the Automated Generic Test issue view page. 

Step 2: Select More > Export Test as Text. The resulting plain text file will be downloaded to the browser.

From the Test issue view screen

Import Steps

Test Steps can be imported from external sources such as:

• • Test
  • CSV
  • JSON
  • Clipboard

You can configure which columns you want to import, including if you want to override the steps already present in the Test.

From Test

The import from tests dialog allows you to choose which steps you wants to clone from a given test and a checkbox to choose if you want to override the steps already present in the Test.

From CSV

The import steps from the CSV dialog allows you to choose the file that you wants to import the steps from, the delimiter that is used in the CSV, a dropdown to specify which column layout the steps present in the file have, a checkbox to specify if the file has header or not, and a checkbox to allow you to choose if you wants to override the steps already present in the Test.

From JSON

The import from JSON dialog allows you to specify which file you wants to import the steps from and provides a checkbox if you want to override the steps already present in the Test.

The JSON file should be an array of step objects. The objects should have the field step and can optionally have the fields data and result. Below is an example of a valid file:

[
   {
      "step":"Step 1",
      "data":"Data for Step 1",
      "result":"Result for Steps 1"
   },
   {
      "step":"Step 3",
      "data":"Data for Step 3"
   },
   {
      "step":"Step 4",
      "result":"Result for Step 4"
   },
   {
      "step":"Step 2"
   }
]

From Clipboard

The import from clipboard allows you to copy the steps from Excel or CSV and paste them in the dialog. You can choose if you want to use the default Excel delimiter or use a custom one, if you want to copy steps from CSV. You can choose which column layout the steps in the file have and provides a checkbox if you want to override the steps already present in the Test.

After the steps are pasted in the dialog, it will display the previous version of the steps in the same dialog.

Export Steps

Manual Test Steps can be exported to CSV by clicking the “Export Steps” button in the Test Details section of the Test.

The following dialog will pop up.

The CSV delimiter can be changed and optionally, the attachment links can also be included.

The result will be a CSV file containing the Step, Data, Expected Result columns and, if the option is selected, the Attachments column that contains the attachments links.