Page History
...
You can either execute your tests manually or automatically. Even if the test it is an automated test, you can manually set the test status for that execution.You can also create custom status statuses.
Tests are always executed always within a Test Execution context, even if it's an ad - hoc execution. You can execute tests inline (i.e., define the test run status), directly in the Test Execution issue screen, or you can use the Execution Screen which is detailed extensively bellow in the next sectionbelow.
A Test Execution is validated if all the Tests that verify the specific environment target by it are passed. In order to execute, monitor the progress and report the results, each Test has its own Execution page details. Each Test associated with a Test Execution is called a Test Run. A Test Run is like an instance of the Test that contains a copy of the test definition (e.g., description, steps) and the result of that test in the context of that Test Execution .
When a Test definition is modified, the respective Test Runs aren't automatically updated. This happens because the execution of a Test depends directly of it's on its definition. If the Test definition is changed, the executions might not be consistent with the definition anymore. Each time a Test Run is created, the Test definition is copied to it. This guarantees that when the Test definition changes, the Test Runs are not affected and will always contain the original (i.e., when it was added to the Test Execution) Test definition.
A given user is You are allowed to execute a Test if:
- is has you have permission to "Browse" the Test Execution issue
- it has you have permission to "Resolve" the Test Execution issue
- the Test Execution issue is not in a status configured to disallow executions
Even though the user you might not have permission to execute a Test Run , and if the execution has already been started, the user you can see the execution details page in readonly read-only mode. An informative message will be shown to the user on at the top of the execution page if the current Test Run is in readonly read-only mode.
Execution Screen
When you execute a user executes a Test, the Xray Execution Screen is shown. This screen is accessible from the Test Execution issue view page and the Test issue view page (Test Runs section), and this is where the execution results are displayed and registered. This page contains information about the Test definition and the Test Execution issue context to allow the tester you to execute the Test without exiting this page.
The execution screen is composed by the following sections:
| Table of Contents |
|---|
Navigation bar
Always available, the navigation bar contains breadcrumbs to allow easy navigation between the Test issue hierarchy. It also contains also a toolbar with the Return to Test Execution button , that allows you to jump to the current Test Execution issue page, as well as the Next and Previous buttons, to quickly navigate to the execution screen of the other Tests included on the same Test Execution, if applicable.
Export
...
Automated Test Definition
If you are executing an automatic automated Test, you can export it's its definition right from the execution screen.
...
If you're executing a Cucumber Test, an option to export the Test as a feature file will appear.
Generic
If you're executing a Generic Test, then an option to export the Test as test text will appear.
Execution Status
The Execution Status describes the current Test progress. The possible (native) status for a Test Run are TODO, EXECUTING, FAIL, ABORTED and PASS. You can also create custom status statuses in the Xray Manage Test Statuses Execute Tests administration page.
You can change the status of the execution directly through this field. It supports inline editing and once activated, it will display a set of statuses that you can set the Test Run. if If the option for setting the Test Run status manually , without having to execute all Test Steps (or Examples, as in the case of Cucumber Tests) , is enabled in the Xray administration page, then you can set the Test Run Status to any status, independently independent of the current aggregate status of the Test Steps (or Examples). If this option is disabled, then the status that you can set through this field will be limited to the available transitions based on the current Steps or Examples aggregated statuses.
The status section contains other relevant fields such as:
- Assignee, that shows - the User assigned to perform the current test execution;
- Executed By, that shows - the last User that changed the status of the current test run;
- Versions, that shows - the target release version tested by the current test execution;
- Revision, that shows the - the source code and documentation version used in the current test execution;
- Started On, that shows the date of when - the date and time the execution of the current test started;
- Finished On, that shows the date of when - the date and time the execution of the current test finished;
You can change the Test Run assignee directly on the field. It supports inline editing and once activated, you can filter by user and then select the one you wish to assign to the Test Run.
If the dates are configured to be edited, you can edit the Started On and Finished On dates date by clicking in the calendar icon and choosing the date and time. You can check your configurations by going to JIRA ADMINISTRATION Jira Administration > Add-ons > Global Preferences.
Note: The JIRA Jira jira.date.time.picker.java.format and jira.date.time.picker.javascript.format should properties in Jira Administration advanced settings should belong to the same pair of properties in order to for the datetime picker work properly. See the following link for more information
...
This section provides the ability to manually set the status for Requirement issues that are tested by the current Test issue. By default, the requirement status is calculated based on the latest Test Run of each Test associated with a given requirement. Even though the Test Run status is failedis FAILED, not all requirement issues associated with the Test issue might have failed. Some of these requirements can, for instance, be passedbe PASSED, if the tester chooses to explicitly set the requirement status. This functionality makes it possible for a single Test issue to test multiple requirement issues with different concerns and functionalities. These requirement statuses that are explicitly setted set in a Test Run , will then be considered when calculating the Requirement Status and Requirement Coverage.
| Info |
|---|
This section is only visible if the "Separation of Concerns" option is disabled in Xray Requirement Requirements Coverage administration settings page. |
All affected requirement statuses will be updated whenever a step (or example for cucumber in the case of Cucumber automated Tests) status is changed. The status for any affected requirement can always be changed by clicking on the status and choosing the desired option.
...
The Execution Defects field lists the overall defects related with the current execution. It is possible to create a new Defect issue, create a new Sub-task or Add existing Defects by clicking on the respective option.
Execution Evidences
The Execution Evidences field lists the overall attachments related with the current execution and clicking on the Add Evidences, opens the Attach Files dialog box.
Comment
The Comment field is inline editable to input overall appreciations about the current execution without reloading the page
The field supports the JIRA Jira markup wiki language.
Test Description
This section includes the current Test issue description. The Test issue description often contains useful information for testers to execute the Test.
The field supports the JIRA Jira markup wiki language.
Test Issue Links
This sections section shows all links associated to with the Test.
Test Issue Attachments
This sections section shows all attachments associated to with the Test.
Pre-Condition
The Pre-Condition field shows the Pre-Conditions issues associated with the current Test, if present.
Pre-Condition Attachments
The Pre-Condition field also shows the Pre-Conditions attachment in a inline dialog.
Manual Test Step Results Table
A Manual Test execution screen contains the Test Steps field, listing the Steps, Data, Expected Result and Attachments to be used as reference while testing.
By clicking in on each step, the following fields became become editable:
- the Comment field , where can be consulted and inputed any appreciation which contains any notes about the corresponding step;
- the Defects field , where can be consulted which contains any defect associated with the corresponding step. It is also possible to input new defects by:
- Clicking on the Create Defect to create a new Defect;
- Clicking on the Create Sub-Task to create a new Sub-Task;
- Clicking on the Add Defects to add an already existing defect; - the Evidences field , where can be consulted which contains any attachment associated with the corresponding step. It is also possible to input new attachments by clicking on the + button, that which opens the Attach Files dialogue box.;
- the Status, where which contains the current step testing status can be consulted and inputed. The Step Status can also . This can be changed by clicking on any of the status in front of your Step Status.
Note: Changing the Status on the each Step also affects will affect the Execution Status field on the Execution Screen.
Automated Test Details
A Automated An Automated Test execution screen contains the Automated Test Details field, listing which list the Steps to be used as reference by the external tool while testing.
Examples
A scenario outline ofan automated Cucumber test with a will normally contain an examples table containing several inputs and outputs for the given scenario.
...
A scenario outline test is PASS only if all examples are passed.
A scenario outline test is FAIL if at least one of the examples is failed.
A scenario outline test is TODO if at least one of the examples is did not run and all the other examples are passed or todo.
Results
Xray for JIRA Jira supports the import of multiple execution results for the same Test Run. Multiple execution for the same Test Run specify different results for different context/environments. Multiple executions are only supported for Automated Tests. Manual Tests do not have this concept.
When a Test Run contains multiple executions imported from external sources , like Calabash and Xamarin Test Cloud, these executions are presented in a "Results" section , on the execution page.
This section contains a Table with all executions for the same Test Run. The table specifies has the following columns:
- Context: An execution context name (in the case of Xamarin Test Cloud results, the mobile device and operating system version)
- Error Message: A possible error message if the result has failed.
- Duration: The complete duration for of an execution result.
- Status: The execution results status.
Attaching Screenshots
To attach a screenshot, paste the image from your clipboard using the relevant keyboard shortcut. (For Max Mac OSX, use CMD+V, ; for Windows, use Ctrl+V.)
Enter a file name for the screenshot you are attaching (The file name defaults to "screenshot-" concatenated with the next number available.) and choose where you want to place the evidence (Overall Execution or Test Steps).
Click Upload to attach the captured image to your Test Run.
...
The Activity field lists all the activity activities related with the current execution.
Updating the definition of the Test after execution started
When a Test definition has changed, a new dialog box will appear on the top of the Execution page , informing the user of the obsolete current execution, and prompting to a informing you that the current execution is obsolete, and will prompt you with Reset or Merge buttons that, when pressed, will load the updated definition as well as delete all the execution data or merge the new Test definition into current state of execution.
Reset Execution
You can choose to reset the Execution or keep it for a historic record. If you reset the execution, all information previously saved (status, evidences, defects...) will be erased.
Merge
...
Execution
You can choose to merge the new Test definition into the current status of execution. This will keep steps that weren't changed along with the associated results/comments/evidences. It will also load new steps (moved, erased or created), pre-conditions or attachments definitions. This will always force a recalculation of Test Run Status. For instance, if the current Test Run Status is FAIL because of a step and this step is removed from Test, if you merge, then the Test Run Status will change to whatever the value combination of the others steps returns.