A Test is a scenario made up of a sequence of steps and conditions or variables, where test inputs are provided and the test target is run using those inputs, to see how it performs. Certain working conditions are also present, to verify how they are handled by the test target. Every requirement or objective the test target is expected to achieve, needs at least one test. An expected result is outlined and the actual result is compared to it in order to assess the test success.

Test Types

Tests in Xray can be of two types: automated or manual.

• Automated
• Manual

Automated

Automated Tests are executed through an external tool which controls the execution of tests and the comparison of actual outcomes to predicted outcomes. They can automate some repetitive tasks in a formalized testing process already in place, or add additional testing that would be difficult do perform manually. The following automated Test types are available:

• Cucumber
• Generic

In JIRA's administration you may add other values (see list bellow) for the "Test Type" custom field, for example if you want to differentiate automated tests that may exist simultaneously in a project.

However, internally they will and must be handled as one of the two default Test Types for automated tests: "Cucumber" or "Generic".

Note: other values for the "Test Type" custom field are not supported.

Cucumber

Cucumber is a tool that executes plain-text functional descriptions as automated tests. The language that Cucumber understands is called Gherkin, which is a Business Readable, Domain Specific Language that describes software’s behavior without detailing how that behavior is implemented. It is a line-oriented language that uses indentation to define structure. Line endings terminate statements (e.g., steps). Either spaces or tabs may be used for indentation (but spaces are more portable). Most lines start with a keyword.

Reference cucumber site

For more information on the Cucumber Basics, please consult the Wiki and Tutorials at: http://cucumber.io

Fields

Cucumber Tests are divided in two Scenario Types:

•  Scenario,  a set of instructions exclusively composed by constant values, which, when executed by a tool will only give a static result. (e.g.: 2 + 3 = 5)
•  Scenario Outline,  a generic set of instructions containing variables, which, when executed by a tool will prompt varied results according to the specified input.

- On the  Steps  Field, is inputted the behavior to be reproduced by the Cucumber tool.

Generic

A Generic Test is an automated agnostic Test that is executed by an external tool. Execution results will then be imported back to JIRA. An example might be a JUnit test that is used for quality assurance.

Fields

The only Field that is related to a Generic Test is it's Definition. Here the automated external test may be defined or referenced through an open text field. 

The Generic Test Definition field supports the JIRA markup wiki language.

Manual

Manual Tests are user assisted tests, listing an assortment of guidelines the user must follow, and the results such behavior is expected to accomplish. After performing the test as instructed, the actual test result is compared with the expected test result, and if the they positively match, the test is considered as passed, and if they don’t, the test is considered as failed.

Fields

Manual Tests are segmented in Manual Steps. Each Manual Step is composed with 4 fields - Step, Data, Expected Result and Attachments - being the Step field the only one that’s mandatory;

• In the Step Field, is inputted the action to be reproduced by the tester;
• In the Data field is inputted any data the related step requests (p.e. login credentials) to be used by the tester;
• In the Expected Result is inputted the behaviour the step should accomplish;
• The Attachments are external files that provide additional information related with the corresponding Manual Step. Attachments can only be added or removed in the Test Issue view page.

The Step, Data and Expected Result fields support the JIRA markup wiki language.

Actions

• Create

• Edit

• Clone

• Export to Cucumber

Create

In order to create a new Test Issue the user must:

Step 1: Click the Create Issue at the top of the screen to open the Create Issue dialog box / page.

Step 2: Select the  Project  and on  Issue Type  select Test.

Step 3: Under the General tab, type a Summary for the test and fill any appropriate fields — mandatory fields are marked with an asterisk.

Step 4: On the Test Details tab select the Test Type and complete any appropriate fields;

Step 4a: If you are writing a Manual Test:
- Select the Test Type as Manual; 
- After filling the Manual Steps Fields, press the Add button to jump to the next Manual Step;
- Use the Pencil button to edit a previous inputted Manual Step, and in the end press the Update button;
- Use the Bin button to delete a previous inputted Manual Step;
- To change a Manual Step position in the Test, grab the area on the left of it's row and move it to the desired place;

Step 4b: If you are writing a Cucumber Test:
- Select the Test Type as Cucumber; 
- On Cucumber Type select test type between Scenario or Scenario Outline;
- On Cucumber Scenario enter the Cucumber Scenario clauses in Gherkin;

Step 4c: If you are writing a Generic Test:
- Select the Test Type as Generic;
- On Generic Test Definition enter the definition for the automated generic Test;

Optional:
- Click on the Test Sets tab to associate the current issue with Test Sets;
- Click on the Pre-Conditions tab to associate the current issue with Pre-Conditions;
- Click on the Test Plans tab to associate the current issue with Test Plans;
- Click on the Link Issues tab to Link with other issues and specify the nature of the association;
- To create a series of similar tests – in the same  Project  – select the  Create another  check box at the bottom of the dialog;

Step 5: When you are satisfied with the content of your test, click the  Create  button.

Step 5a: If you selected the  Create another  check box (above), a new  Create Issue  dialog appears. This issue is automatically pre-populated with your previous test details, while leaving the  Summary  field blank.

Edit

Editing an existing Test can be done using either one of the following procedures:

• Using the Edit Issue dialog box
• Using the Inline Edit for JIRA Plugin

Using the Edit Issue dialog box

The Edit Issue dialog box is similar to the Create Issue dialog box, and can be accessed through the following steps:

Step 1: Click the Edit button (at the top-left of the 'view issue' page) to open the Edit Issue dialog box.

Step 2: Modify your Test's details in the appropriate fields of the Edit Issue dialog box General tab.

Step 3: Click on the Test Details tab to edit the Test Type and any complementary fields;

Step 3a: If you are editing a Manual Test:
- To input a new Manual Steps field, use the empty at the bottom and press the Add button;
- Use the Pencil button to edit a previous inputted Manual Step, and in the end press the Update button;
- Use the Bin button to delete a previous inputted Manual Step;
- To change a Manual Step position in the Test, grab the area on the left of it's row and move it to the desired place;

Step 3b: If you are editing a Cucumber Test: 
- To change test type on Cucumber Type select between Scenario or Scenario Outline;
- In the Steps field you can edit previous inputed, or create new, Cucumber Scenario clauses in Gherkin;

Step 3c: If you are editing a Generic Test:
-In the Steps field you can edit previous inputed, or create new, Definitions for the automated generic Test;

Optional:
- Click on the Test Sets tab to edit Test Set associations;
- Click on the Pre-Conditions tab to edit Pre-Condition associations;
- Click on theLink Issues tab to edit Links with other issues;

Step 4: Click the Update button to save your changes.

Inline Editing Xray Tests

Xray Test issue sections support inline editing similar to JIRA native and custom fields inline edit. This prevents full page reloads when modifying fields.

Inline Edit Fields

To inline edit a field just hover the field view area and click it.

The following Test fields support inline edit:

• Type: the Test issue type
• Definition: The automated generic test definition (only visible for Generic tests)
• Cucumber Type: The Cucumber Scenario type (only visible for Cucumber tests)

To inline edit automated Test issues:

Step 1b: If you are Inline editing a Cucumber Test: 
- You can hover on the actual Test Type, click it, and change it for another available on the select box;
- You can hover on the actual Cucumber Type, click it, and change it for another available on the select box;
- You can hover on the Steps field, click it, and edit the its contents;

Step 1c: If you are Inline editing a Generic Test:
- You can hover on the actual Test Type, click it, and change ir for another available on the select box;
- You can hover on the Definition field, click it, and edit the its contents;

Notes: On JIRA versions greater or equal to 7.3.0 the definition field supports Jira's Rich Text Editor. To enable the Rich Text Editor you have to set the renderer of the definition custom field to  Wiki Style Renderer.  Its also possible to use Rich Text Editor on JIRA versions greater than 7.2.0 by enabling the experimental Rich Text Editor in JIRA configurations but this feature its not totally supported by Xray.

Manual Test Steps Table

In addition to inline editable fields, Xray provides a Restful table for inline editing manual Test steps direcly from the Test view issue screen.

From this table you can:

Add Steps: After entering the Test step details (Step, Data and Expected Result fields) using the bottom table row text boxes, click on Plus icon for creating a new step. The new step will be added in the last position in the steps table. 

Remove Steps: The last column for each step contains a Bin button for removing steps. After clicking the button you will be prompted with a confirmation message to delete the step. This operation cannot be undone.

Edit Steps: When hovering over Step, Data and Expected Result fields they highlight and if you click them the step row visualisation will change to edit mode. From here you can edit the fields and click on the Update button that appears in the last column to save changes or you can cancel the changes by clicking on the cancel button. Either way, the row will be back in View mode.

Move Steps: You can move a Test step Up or Down in the Test steps table from two different ways:

• • Clicking on the drag area in the first column of each step and drag the step to the desired position.
  • Clicking on the actions menu and choosing the "Move To.." option. This method allows you to directly move a step to a specific position.

Add Attachments: While in View or Edit modes, you can add attachments to a step using the Plus sign appearing in the Attachments column. After clicking the button you will be prompted with a dialog to choose the attachments to add.

Remove Attachments: While in View or Edit modes, you can remove attachments of Test steps using Bin button that appears when hovering existing attachments. After clicking the button you will be prompted with a confirmation message to remove or not the attachment.

Copy Step Fields: You can copy the Step, Data and Expected Result fields to clipboard by hovering these fields and choosing the Copy icon next to each highlighted field. If the copy is successful, a green border will appear momentarily around the Copy icon.

Expand Fields: If the text boxes are too small to show all the information, you can expand the Step, Data and Expected Result fields by clicking the Expand icon next to each highlighted field. An inline dialog, with a bigger area, will be opened with the contents of the field and a wiki notation toolbar to assist you in writing your information. This operation is available in both View and Edit modes for manual Test steps. 

Clone Steps: You can clone existing test steps. To clone a Test step click on the actions menu and choose the "Clone" option. A confirmation message will be prompted. You you confirm, a new step will be created in the last position, containing all information of the cloned step (step, data, expected result and attachments).

Changing the Type of a Test

Xray provides the ability in the Global Preferences to configure if the test definition is deleted or not.

If this option is enabled, the previous Info details will be deleted.

Otherwise the Info details will not be deleted, and it'll be possible to undo the Test Type change operation keeping the previous information.

Additionally, in both cases, because each pre-condition must be of the same Type of the Test, any associated Pre-Conditions will be lost.

Clone

Cloning a Test issue creates a new Test with all the information of the cloned Test except it's executions. Therefore, the cloned issue will not be associated with any Test Execution issue.

Step 1: Select More > Clone. The Clone Issue screen will appear.

Step 2: You can edit the clone Test's  Summary  if you wish.

  - If the test contains links to other issue(s), you can select whether or not to include the links in the new clone test.

  - If the test contains sub-tasks, you can select whether or not to create the sub-tasks in the new clone test.

  - If the test contains attachments, you can select whether or not to include the attachments in the new clone test.

Step 3: Click Create.

Configure Test Run Columns

Configure Test Set Columns

Configure Test Plan Columns

Create Pre-Condition

To create a Pre-Condition directly associated to the current Test from the Test issue view screen:

Step 1: Open the Test issue view screen you wish to create a Pre-Condition with.

Step 2: Click the Create Pre-Condition button (under the Pre-Condition section on the view page) to open the Create Issue dialog box.

Step 3: Type a Summary for the Pre-Condition and complete any appropriate fields — at least required ones which are marked by an asterisk.This issue is automatically pre-populated with your previous Test issue Fix Versions/s.

Step 4: On the Pre-Condition Details tab complete any appropriate fields. This issue is automatically pre-populated with your previous Test type.

Step 5: Optional. Click on the Tests tab to edit Test associations to associate more Tests. This issue is automatically pre-populated with your previous Test key.

Step 6: Click Create.

Associate Pre-Conditions

To associate Tests with a Pre-Condition from the Test issue view screen:

Step 1: Open the Test issue view screen you wish to associate a Pre-Condition with.

Step 2: Click the Associate Pre-Condition button (under the Pre-Condition section on the view page) to open the Associate Test with Pre-Condition dialog box.

Step 3: Select or search the Pre-Condition issues to be associated with the Test:

Select Pre-Condition Issues -  In order to select the Pre-Conditions to be associated you can:

- Input the desired Pre-Condition Issue Key on the Pre-Condition s field;
- Click on the Down Arrow on the Pre-Conditions field and select the Test Set from its History Search list;
- Click on the + button wich will prompt an Issue Selector Pop-Up; 
- Click on the Search tab to use the Find JIRA issues browser;

Search  Pre-Condition  Issues -  In order to search for Pre-Conditions to be associated you can, under the Search tab:

- Browse for the desired Pre-Condition Project or terms it Contains;
- Click on the More button to add more browse fields such as Label, Fix Version, Saved Filter and Workflow Status;
- Click on the Magnifier button after filling the browse fields, to get a list of matching entries; 
- Click on the Advanced button to perform a JQL search;

Step 4: Click Add.

Remove Pre-Conditions

To remove a Pre-Condition association from the Test issue view screen:

Step 1: Open the Test you wish to remove the association with a Pre-Condition.

Step 2: On the Pre-Conditions table hover the desired Pre-Condition row and click on the Trash button that appears in the end of the row. You will be prompted with a confirmation dialog.

Reorder Pre-Conditions

Export to Cucumber

Cucumber Tests in JIRA can be exported to feature files ready to be executed in Cucumber. The generated feature files follow the rules defined in Export Cucumber Features and take into account the context of the JIRA Test issue.

A single Cucumber feature file should be generated and prompt to be downloaded, unless the target Test is associated with multiple Requirements, which, in that case, will prompt a FeatureBundle zip file containing a single Cucumber feature file for every Requirement associated with the target Test.

To export a Cucumber Test issue to feature files, the following steps must be taken:

Step 1: Open the Cucumber Test issue view page. 

Step 2: Select More > Export to Cucumber. The file will be downloaded to the browser.

Export Test to XML

Xray Test issues can be exported to XML. This export differs from the JIRA default action for exporting to XML. Rather that exporting all fields, the Xray export action will only export to XML the most relevant fields for Tests such as the Test definition fields, Pre-Conditions and some issue tracking information.

To export a Test issue to XML, the following steps must be taken:

Step 1: Open the Test issue view page. 

Step 2: Select More > Export Test to XML. The resulting XML file will be downloaded to the browser.

Export Test Runs to CSV

Xray Test Runs can be exported to CSV. Xray export action will export to CSV the most relevant fields for Test Runs such as:

• • Test Key
  • Test Execution Key
  • Executed By
  • Assignee
  • Start
  • Finish
  • Defects issue keys (list separated by ",")
  • Attachment JIRA links (list separated by ",")
  • Comment
  • Status

To export a Test Run to CSV, the following steps must be taken:

Step 1: Open the Test issue view page. 

Step 2: Select More > Export Test Runs to CSV. The resulting CSV file will be downloaded to the browser.

Export Test as Text

Automated Generic Test issues can be exported to a plain text file. Only the Test Definition field will be included in the plain text file. This action can be useful to execute Generic Test externally from JIRA.

To export an Automated Generic Test definition to plain text, the following steps must be taken:

Step 1: Open the Automated Generic Test issue view page. 

Step 2: Select More > Export Test as Text. The resulting plain text file will be downloaded to the browser.

From the Test issue view screen

Import Steps

Test Steps can be imported from external sources such as:

• • Test
  • CSV
  • JSON
  • Clipboard

The user can configure which columns he wants to import along with if he wants to override the steps already present in the Test.

From Test

The import from tests dialog allows the user to choose which steps he wants to clone from a given test and a checkbox to allow to user to choose if he wants to override the steps already present on the test.

From CSV

The import steps from CSV dialog allows the user to choose the file that he wants to import the steps from, the delimiter that is used in the csv, a dropdown to specify which columns layout the steps present in the file have, a checkbox to specify if the file has header or not and a checkbox to allow to user to choose if he wants to override the steps already present on the test.

From JSON

The import from json dialog allows the user to specify which file he wants to import the steps from and provides the user with a checkbox that allows him to choose if he wants to override the steps already present on the test.

The json file should be an array of step objects this objects should have the field step and can optionally have the fields data and result. Bellow you can see an example of a valid json file:

[
   {
      "step":"Step 1",
      "data":"Data for Step 1",
      "result":"Result for Steps 1"
   },
   {
      "step":"Step 3",
      "data":"Data for Step 3"
   },
   {
      "step":"Step 4",
      "result":"Result for Step 4"
   },
   {
      "step":"Step 2"
   }
]

From Clipboard

The import from clipboard allows the user to copy the steps from Excel or CSV and paste them in the dialog. The dialog allows the user to choose if he wants to use the default excel delimiter or use a custom one if the user wants to copy steps from csv, to choose which column layout the steps in the file have and also provides the user with a checkbox that allows him to choose if he wants to override the steps already present on the test.

After the steps are pasted in the dialog, it will displayed a previous of the steps in the same dialog.

Export Steps

Manual Test Steps can be exported to CSV by clicking the “Export Steps” button, in the Test Details section of the Test.

The following dialog will pop up.

The CSV delimiter can be changed and optionally, the attachment links can also be included.

The result will be a CSV file containing the Step, Data and Expected Result columns and, if this options has been selected, the Attachments column that contains the attachments links.