In order to install TestRail Lite you just have to search the app and install it through the Apps entry of Jira.

TestRail Lite will be available for all your projects and you can choose to enable/disable it from the configuration page, on a per project basis, and apply the default settings regarding the suite and and the case template to be used by default.

Other then the option to active TestRail Lite in each project you can also define which is the default case template to be used or the default suite (it comes by default with a Master suite).

On the Global Setup area you can define the default timezone ti be used and if you allow TestRail Lite user to delete test runs.

All these new objects (cases, suites, test runs) are internal to TestRail Lite, meaning that you cannot search, use in pre defined workflows or define a custom field.

It also means that they will not appear in your board or reports, they only will surface in the context of the Jira issues. 

In order to install Xray you just have to search the app and install it through the Apps entry of Jira.

Please check this page for further details on the installation process.

Once installed, we then need to configure each project individually to enable certain features of Xray, such as the ability to create Tests for certain issues covered by those Tests (see Quick Setup).

The team needs to choose whether they want to have Tests alongside the same project where they have their requirements/stories or have it separated. Having a all-in-one scenario, where testing entities are stored in the same project where Story, Task and Bug issues are, is a good practice; however, Xray allows different approaches for project organization to suit what is best for your team (more info on Project Organization).

Aiming to provide flexibility to the teams, Xray is adaptable to match your way of work giving you configuration options to shape the tool to your team needs or you just can use the default (that will serve most cases).

In Xray all the main entities are in fact Jira issues, this leverages all the advantages of being a Jira issue, such as permission control, workflow implementation, custom fields, screens customization, assignment, changes tracking, collaboration and much more...

TestRail Lite

Once TestRail Lite is installed into your Jira Cloud projects, you can create test cases inside your Jira issues.

A new panel opens up and allow you to define information to some fields and define the template that should be used to create the Test Case:

The template choice will influence the information to be filled in the right pane, the available templates are described below: 

• Test Case (Text)
  • This template lets you define optional preconditions, steps, and expected results for a general test case in a flexible way.
• Test Case (Steps)
  • With this template you can determine specific steps your team should take when testing this case, and what the expected results should be. You will find a number next to each step.
• Exploratory session
  • Use this template if you need to test in a more exploratory way for your project defined by a certain goal or mission instead of steps and expected results.

You also have the possibility to define a Precondition for the Test Case, it is available as text field within the Test Case.

Xray 

A full test specification may be done using just a Test issue, where you specify the steps and the expected results. Sometimes, you may prefer to abstract the initial conditions instead of writing them directly as the first Test steps, enabling you to use them in different tests. In this case, you may create one or more Precondition issues and associate them to a Test.

Although tests may be edited anytime, the immutability of tests is guaranteed whenever tests are scheduled to be executed in some test Execution.

You have several possibilities to create a Test, as all the Xray issues are Jira issues, you can use any create option available in Jira to create issues.

The following example will show one of those options, in this case the creation of an issue within the User Story in the Xray area:

Once you have added a new Test you can edit it to add detailed information, such as the Test Type to use:

The Test Type definition will define what are the information needed to each type:

• Manual (that will match the Test Case (Steps/Text) from TestRail Lite)
  • Xray provides a section to view, create and edit the Test Steps. These steps can be configured to have different fields; check this page Test Step Fields configuration page in the project settings.
• Generic (Usually used for automation but is also used to abstract an exploratory test charter and gather evidences for an exploratory session)

  • A Generic Test is an automated agnostic Test that is executed by an external tool. Execution results will then be imported back to Jira. An example might be a JUnit test that is used for quality assurance.

• Cucumber (Not matched in TestRail Lite)
  • Cucumber is a tool that executes plain-text functional descriptions as automated tests. The language that Cucumber understands is called Gherkin, which is a business-readable, domain-specific language that describes the software’s behavior without detailing how that behavior is implemented. It is a line-oriented language and uses indentation to define structure. Line endings terminate statements (e.g., steps). Either spaces or tabs may be used for indentation (but spaces are more portable). Most lines start with a keyword.

These are the predefined Test Types made available in Xray but you have the possibility to create your own as Xray enables you to create custom Test Types that best match your needs.

You can also define a PreCondition and associate it to a Test as any Jira issue by linking it to the Test, as Preconditions are a Jira issue they can be reused and shared amongst tests.

Xray has an Exploratory App tailored for exploratory testing that integrates with Jira and Xray.

The Xray Exploratory App does not perform exploratory testing for you; however, it can surely help you stay focused while testing and ease documentation/note-taking so you can review all you did and discuss it later on with your team.

TestRail Lite

TestRail Lite lets you organize your test cases by grouping them in Suites and Sections. This helps you find and reuse cases so that you can add them to test runs easily. By default TestRail Lite comes with a default suite created called “Master”.

In this screen you can add Suites and Sections (with Sub-Sections) and associate the Test Cases to each. This panel also enable you to bulk edit Test Cases.

Xray

Organizing your tests is a crucial part of your daily testing activities. Since your project may have hundreds or thousands of tests, it is imperative that you organize them efficiently so you can find relevant tests quickly.

While many organize using labels, components, priority, etc., they are usually insufficient and would not scale according to your needs.

Xray provides two different but very effective ways of organizing Tests:

1. using Test Sets issues
2. using the Test Repository
   
   

The better match in Xray for TestRail Lite' Suites and Sections is the Test Repository.

The Test Repository provides a tree-like organization at the project level. Users can hierarchically organize Tests within folders and sub-folders. This folder concept is common in some tools and resembles user's own files/documention organization in their computer's operating system.

As an alternative, Xray has a specific issue type for grouping tests in flat lists: the Test Set. This is a simple way to create different groups of Tests. You may have as many Test Sets as you wish and a Test may be included in multiple Test Sets. Test Sets are great if you want to have full control over certain groups of Tests. You can easily select them whenever you want (e.g., when creating Test Plans or Test Executions). Test Sets can also be used as a dynamic way to cover requirements.

TestRail Lite

TestRail Lite does not have any entity to assist on the planning of Test Cases. You may achieve it with Test Suites and Sections even though they're not specificaly tailored for that.

Xray

Although you may plan tests in different ways, we recommend that you use the Test Plan issue specifically for this purpose.

In your Test Plan, you can put the Tests you want to track for a given version and the Test Plan will show you the consolidated results for those Tests. In other words, it will present the latest status of each Test, independent of the number of testing iterations (i.e., Test Execution) you make with them.

You may create a Test Plan per sprint, or for the whole release, for example; you decide.

TestRail Lite

After you have added a few test cases, you can start executing tests with TestRail Lite by adding a Test Run. 

Open a Jira Issue and click on the arrow next to ‘Add test case’. Select ‘Add test run’ from the dropdown menu and a new window will pop up.

After adding test runs in your Jira Issues, you can find them below the TEST RUNS tab.

Select one of the test runs from the list to run the test cases.

Once you are in the details of the Test Run you can pass/fail/block/retest the Test automatically if you use the drop down in the top menu.

If you want to add comments, screenshots or bugs to the Test Run you must choose Run Test option from the top menu or bottom right buttons.

If you choose to RUN TEST you will taken to an execution panel where you can add comments, screenshots, link bugs or issues, see the details of the Test Case and define its overall status.

What will change amongst the different templates is the right panel that will show different details depending on the test template used.

Xray

Xray provides the Test Execution issue (that will match the TestRail Lite Test Run), which acts as an assignable task for executing a group of tests (either manual or automated) and collects the corresponding test results for a specific project version and environment.

A Sub-Test Execution is similar to a Test Execution, but created as a sub-task of a requirement.

Normally, you will create Test Executions from Test Plans, or create ad hoc Test Executions (directly from a Test).

Depending on the strategy used to perform the validation of your product/project you will be able to create Test Executions from different origins:

• Entering in the Test details and checking the Test Runs section
  • 
• If using Test Plan as a strategy, you will be able to check the Test Execution status:
  • In the Test Plan view
    • 
  • In the Test Plan details within the Test section
    • 

Once you have your Test Executions created, you're ready to execute them. Each time you schedule a given Test for execution, a Test Run will be created and Xray will keep the information related to that specific execution of that specific Test. 

On the detail view of the Test Execution you will able to start the execution using the Play icon present next to each Test:

That option will take you to the execution panel where you will have all information needed to perform the execution (Description, link to requirements and bugs, Preconditions)

and for each Step you will able to add a comment, create a defects, associate an evidence and define the status of that step

These possibility to add details per steps will enabled a more specific report for a defects being able to pin point where it failed and what were the steps performed so far.

TestRail Lite

TestRail Lite only has available the list of Test Runs that you can export as an HTML or PDF document.

You can reach this report through the left menu entry:

Once you access one of the Test Runs you will be taken to the following page where you can print the report:

The other way is through the requirement where you have defined the Test into:

In this last option you will not have the option to print the report.

Xray

Xray essentially uses Jira issues, so you can always use JQL to obtain relevant results. Xray also provides some custom fields and some entity properties that may be queryable using JQL.

The custom fields may be exposed directly in the standard Jira search result table or in tools, such as Jira's Filter Results.

In summary, your reporting may be accomplished by:

• using standard JQL queries 
• consulting the specific panels of Tests/Test Sets/Test Plans/Test Executions
• using the Built-in Reports
• building out dashboards using gadgets

You will able to access the Test Executions results from the following sources:

• Entering in the Test details and checking the Test Runs section
  • 
• If using Test Plan as a strategy, you will be able to check the Test Execution status:
  • In the Test Plan view
    • 
  • In the Test Plan details within the Test Execution section
    • 
• Accessing the report Test Executions List (and with full details Test Runs List)
  • 
• The more important one is on the requirement itself where we will have the visibility of the coverage (calculating the status with a combination of all status of all Tests associated to it), bringing the visibility of the coverage to the requirement level.
  • 

Let's take a look on two major Reports available through Xray:

• Traceability
  • This report enables you to follow the life of a requirement in both forwards and backwards direction (i.e., from its origin all the way through tests, test runs and defects). It facilitates analysis of the overall requirements coverage status.
  • 
• Coverage

  • In order to analyze the current status of your release, or even other releases from the past, you need to evaluate the status of requirements, taking into account the respective results.

    You might want to analyze just the requirements implemented in a given version or you may want to analyze requirements from earlier versions based on your regression testing.

    The overall requirements coverage report provides the means to analyze your requirements statuses in real-time.

  •