Overview

Exploratory testing is an approach for testing where the testers plays a key role in uncovering problems that otherwise could stay unnoticed.

While exploring, testers perform test ideas by interacting with the target system, from where they observe and get insights for further test ideas.

Therefore, there's no predefined script for the tests. What we do is dynamic, based on real-time feedback we get from the system, together with our knowledge and context that are key to identifying problems and gaps in our systems.

Learn more here.

Process at a glance

In exploratory testing, usually we start with a goal, or a so-called charter. The charter is a high-level description of what we aim to test and the why, for the purpose of the time we decide to allocate to it.

Sometimes, our testing sessions, where we perform the actual testing, may be well-delimited (e.g., 30min sessions), or may not have an exact time frame.

During our session, we take notes and other evidence (e.g., screenshots,  videos, logs) that we can then share and debrief with the team afterwards. We may even report defects or create new ideas in the backlog.

Exploratory testing with Xray

First, we need to use some entity as an abstraction of an exploratory test. In fact, exploratory tests are... tests. Therefore, we will use Test issues for this purpose.

By default, in Xray we have different types of Tests, including "Manual" (i.e. scripted, composed of well-known predefined steps), Cucumber (i.e. Gherkin), and Generic (i.e., unstructured tests).

For defining our test charter, we'll  use unscructured tests; we can either use Tests of "Generic" test type or even, probably better, create a specific test type and name it, for example, "Exploratory".

In sum, an exploratory test in Xray will be a Test. A testing session, that represents the testing performed for a given charter, will be abstracted as a Test Run for the exploratory Test.

Using Xray built-in capabilities

To perform exploratory testing, as seen earlier, we just need to use Tests (either of Generic type or of a specific custom type such as Exploratory).

The process would be similar to how we manage "manual"/scripted test cases:

1. define the Test
2. link it to some story or requirement, generally speaking
3. organize it in a Test Repository folder or add it to some Test Set kind of list
4. add it to some Test Plan
5. "execute" it 

The steps 2-4, even though optional, are common.

Let's focus our attention on defining the Test and then on the "execution" (i.e., the actual testing session).

Defining an exploratory Test and its charter with Xray

To define an exploratory test, the process is similar to the creation of typical Test in Xray, with some subtle nuances.

1. Create an exploratory Test issue
   1. for the Summary, use a name that summarizes the test charter
   2. choose the proper test type, as mentioned earlier
   3. fill out the "Generic Test Definition" field, with the test charter
   4.   

Performing the test session (i.e., "executing" the test)

Before performing our test session, we need to schedule an execution for our Test. The way to do this in Xray, is to schedule a Test Execution for our Test (or to include our Test in an existing Test Execution).

Test Executions can be created from multiple places (e.g., from Test Repository, Test Plan, etc), including directly from the Test issue screen.

  

During the test session, on the Test Run execution screen, we can:

• report the overall status of our testing session, using the Execution Status field. For example, as PASS if all seems good enough, or as FAIL if there are major issues; you can also use custom test run statuses if you want
• leave notes, using Comment section. As it supports wiki markup, we can use it to organize our text notes; even though there's no structure, we recommend having a brief summary and having the text notes organized by categories
• report defects, on the Execution Defects section
• attach evidence, on the Execution Evidence section. We can attach logs, or other files that may be relevant, including screenshots; we can use our tool of choice for taking screenshots. You can use copy&paste; whenever "pasting" an image the following dialog will appear
  • 

Using Xray together with Xray Exploratory App

The Xray Exploratory App (XEA) is a side companion desktop app that is used to assist testers on exploratory testing related tasks.

To be able to use Xray Exploratory App, please see its Getting Started documentation that details the installation and configuration process.

Concerning the integration with Xray, there are two main flows to have in mind:

1. define the test charter in Xray, by creating an exploratory Test, and then use XEA to assist on note-taking and report results back to Jira, where the team can analyze them and track its impacts
2. use XEA to define the test charter, assist on note and evidence taking during testing, and then report results back to Jira, where the team can analyze them and track its impacts

The subtle difference is which tool we'll use to create/define the exploratory Tests. Teams may use the first approach, where Xray is the master, if they wish to follow similar processes as they do for manual scripted test cases, and have some sort of management/planning.

Starting an exploratory testing session for an existent Test in Xray

In this scenario, we're considering that an exploratory test and its charter have already been defined and exists in Xray.

The exploratory test may have been created in Xray (see above instructions on how to use Xray for defining exploratory tests) or by Xray Exploratory App on a previous session (please see information ahead on how to use Xray Exploratory App to create Tests).

Having the Test, how can we start our testing session and use XEA to assist us?

We can either start the session right from Xray, or directly from XEA.

Starting the session from Xray

To start a session from Xray, we need first to schedule a execution for our test by creating a Test Execution with it, or adding it to an existing Test Execution and then use the option to execute it using Xray Exploratory App  from the test run details screen.

     

Then start your session.

First, you need to select the screen/application you wish to record (in case you aim to take screenshots or record videos).

We can then perform exploratory testing and use Xray Exploratory App to assist us during the process. In the end, results can be uploaded to Xray to the original Test Execution created in Xray, being associated with the corresponding Test Run of the exploratory Test.  

Please see Xray Exploratory App's documentation for more information on executing exploratory testing sessions and all the available options.

Starting the session directly from XEA

Assuming Xray Exploratory App is installed and the connection to the Jira/Xray instance have been configured, we just need to start Xray Exploratory App and choose "New Session for Existent Test".

Then we need to select the Jira project where we have our Test, and then identify it by its issue key. Then we just need to load it and start our testing session.

  

We can then perform exploratory testing and use Xray Exploratory App to assist us during the process. In the end, results can be uploaded to Xray to a newly created Test Execution having the Test we selected along with its result (i.e., the corresponding Test Run).  

Please see Xray Exploratory App's documentation for more information on executing exploratory testing sessions and all the available options.

Create a new exploratory test using XEA and report results to Jira and Xray

Assuming Xray Exploratory App is installed and the connection to the Jira/Xray instance have been configured, we just need to start Xray Exploratory App and choose "New Session".

Then we need to select the Jira project where we aim to create our Test on, along with some additional information:

• Summary: for a brief summary of our test charter
• Test Type: make sure to select the proper you will be using as an abstraction of exploratory tests (e.g., "Exploratory", "Generic")
• Charter: the test charter

  

We can then perform exploratory testing and use Xray Exploratory App to assist us during the process. In the end, results can be uploaded to Xray to a newly created Test Execution having the newly created Test along with its result (i.e., the corresponding Test Run).  

Please see Xray Exploratory App's documentation for more information on executing exploratory testing sessions and all the available options.