Overview

While performing the test sessions, testers collect and organize information so they can share it with the stakeholders or with the internal team.

In this article, we'll cover the essentials of collecting data that can be turned into actionable information, so that we can improve the product or system we're building.

Findings and evidence

To share findings & evidence properly, we need to think about our goal: why are we doing it?

• To ease the replication of a certain situation or bug
• To make RCA (Root-Cause Analysis) easier, and get to the source of the problem

Perhaps we're collecting information for ourselves, or maybe we're thinking about reporting defects so the developers can have a look at them. Maybe we have an idea to share with the PO or with the team for further discussion. Maybe something really works well, and the team needs to know that. Knowing to whom we're collecting information is relevant as we may need tailored information for them.

Let's see a brief summary of information that we can collect during exploratory testing and how it can be organized.

Text notes

Notes in the form of text are the most common way of storing our findings. These notes can be of different natures.

Testers use labels, or categories to distinguish text notes, to make them easier to find afterward.

Next are some examples of possible labels/categories that are frequently used, along with their meaning:

• Concern
  • Something that represents a potential, negative risk that can somehow affect value, now, or in the future.
• Problem
  • A clear problem that impacts value to some stakeholders (internal or external).
• Question
  • Something that is not clear; may or not affect value.
• Suggestion
  • A recommendation/suggestion to change some existing aspect/feature.
• Idea
  • Some ideas to make the product better and thus improve it, usually by recommending new features. Eventually, you may merge "Suggestion" with "Idea."
• Praise/Positive
  • Some positive aspects we have found and that we want to share with the stakeholders, usually the internal team.
• Comment
  
  • A general note/comment.

Evidence and audio recordings

The evidence consists mainly of files that we either extract from the application (e.g., logs, other artifacts) or that are strongly connected to the system (e.g., screenshots, videos). 

Audio recordings, usually taken more as a verbalized text note, can also be included in this section as they will be stored as a file that we can upload as evidence.

Evidence is like facts that are of great help later on, for example, for bug analysis and fixing.

In sum, we can have the following:

• screenshots
• video/screen recordings
• audio recordings
• logs
  • application logs
  • JS console logs
  • crash dumps
  • ...

Tips for collecting and organizing information

Text notes

Assess the different categories you want to have to mark information

Start with a few categories (e.g., Problem, Question, Idea, Praise) and adapt to your needs.

XEA supports taking text notes using these note types: Comment, Problem, Suggestion, Idea, Question, Concern, and Positive.

Organize text notes using mindmaps

• some users prefer a more visual and hierarchical way of organizing findings; mindmaps may work well, either using tools such as Miro, Mindmup, Xmind, or even the free Freemind tool
• mindmaps allow a totally flexible yet non-standard way to organize way notes; this may create some challenges in navigating through different test session results and even searching for what matters in a given session

In the end, add the mindmap or a link to it, as evidence or as a note in your test session.

Focus on the problem and not on the individual

Whenever taking text notes, avoid mentioning someone as the cause of the problem. Focus instead on technical facts/evidence. It's ok to mention some decisions that may have been taken in the past as it can be connected to the problem and may also point to other related impacts.

Screenshots

Use visual annotations to highlight relevant areas to make explanations clearer

Screenshots are quite valuable as they make information visible, which would be hard to put into words. However, we often want to refer to a certain part of the application/site. By drawing around what we want, we can make it clearer for review later on.

In the Xray Exploratory App, it is possible to draw squares, circles, or just freestyle drawings on top of the screenshot. 

Combine screenshots with categorized notes

Take a screenshot and add some text, marking it as a Problem, Idea, or other. This will provide more context to the screenshot you're taking.

In the app, this is done at the same time you take the screenshot, right after the screenshot, at the bottom.

Record only what matters

To provide greater focus and avoid some noise, record just the browser, or the application's UI. Record the whole screen if it makes more sense for your use case (e.g., the interaction of a desktop application with other applications).

In the Xray Exploratory App, the selection of what to record (i.e., the desktop, a specific application) whenever performing video recordings, or screenshots, is done at the start of the testing session. It can also be performed after resuming a paused session.

Video/screen recordings

Record only what matters

To provide greater focus and avoid some noise, record just the browser, or the application's UI. Record the whole screen if it makes more sense for your use case (e.g., the interaction of a desktop application with other applications).

In the app, the selection of what to record (i.e., the desktop, a specific application) whenever performing video recordings, or screenshots, is done at the start of the testing session. It can also be performed after resuming a paused session.

Audio recordings

Use audio notes 

If you prefer to verbalize while testing, then record audio notes and review them later.

In the app, you can easily take audio nodes.

Combine screen recordings with audio notes

Combine screen recording with audio notes, so you can stay even more focused on testing. 

In the app's global settings, you can enable "Audio on screen capture."

Evidence

Upload evidence and add categorized notes to provide better context

Upload some files as evidence and add some text, marking it as a Problem, Idea, or other.  This will provide more context to the screenshot you're taking.

This is done at the same time you upload evidence, at the bottom.

If using the browser, record the traffic to a .har file

Record the browser network traffic and upload it to XEA. In Chrome, this can be found in "Developer Tools > Network"; make sure to select "Preserve Log" option.

Later on, the .har file can be uploaded from the same place to be analyzed in further detail.

Add the .har file as standard evidence to your test session. Switch to the "Timeline" view and then drag the file to XEA.

General

Sequence matters, so record it

Problems often occur due to having the target system/application in a certain state, which in turn is the consequence of a sequence of steps. To replicate problems/situations, we need to remember the sequence of actions performed.

The Xray Exploratory App produces a timeline with the notes and evidence you take. This timeline will be available as a . PDF report. It's also possible to perform a desktop/application screen recording to make it more trustworthy and easier.

References

• PQIP
  
  • Three Digestible Diagrams to Describe Exploratory Testing