How to relate and track corresponding results

Sometimes, you may start with a "manual" test (i.e. test composed of steps executed manually) and later on implement an automated variant of it, using Selenium on top of an automation framework (e.g. TestNG, Junit).

You should ask yourself if:

  1. The tests do exactly the same (if they’re semantically the same)
  2. And if so, you wish to track results independently (i.e. manual related results from automation results).

Case 1: reusing the same Test entity (i.e. issue) for both manual and automated tests

If the manual and automated variants are semantically equal and you don’t need to track automation and manual test results independently, you can reuse the same Test issues.

In this case, and since the “manual” Test would be already pre-created, from the automation standpoint, you would need to use a format that provides the ability to report against an existing Test issue, given its issue key. For that, you can either use TestNG, NUnit or Robot framework XML reports. You can also use Xray JSON format, which is unique that allows you to report the status for each step, while the other ones can’t (they will either PASS or FAIL the whole Test Run).


Note: The current Junit implementation (as of 2019), does not allow you to put labels on the Tests; thus, there is no way to enforce that a given automated test method is mapped against an existing Test in Xray.

Case 2: using separate Test entities (i.e. issues) for manual and automated tests

If manual and automated variant are semantically different or if you wish to track automation and manual test results independently, you need to have different Test issues.

This will give you some added benefits:

  • Clearly distinguish from manual and automated test artefacts and respective results
  • Clearly distinguish whenever automation was introduced and its results
  • Ability to deprecate manual scripted tests without losing history
  • Ability to keep executing manual scripted tests as such and manage them independently

Xray does not enforce a specific way for relating the manual and automated test variant. However, you can, for example, create a link (using whatever issue link you want, such as "automates") between these Test issues.

In this scenario, you would:

  1. Create the “manual” scripted Test and executed it as usual
  2. If using JUnit, Robot, NUnit or TestNG…
    1. Create the automated test code (outside Jira) and submit the results using REST API or one of available CI plugins
      1. “Generic”, unstructured, Test issues would be automatically provisioned, if needed (e. if they don’t already exist).
        1. Xray tries to find an existing Generic Test having the same Definition field (e.g.  classname+method); if it succeeds, it maps it to that existing Test otherwise it will create one Test
  3. If using Cucumber (which doesn’t support auto-provisioning of Test issues)…
    1. You would need to follow one of the flows depicted here
  4. Manually create the link between the two Tests (if you use links as a means to relate them)

At any moment in time, you could deprecate the manual scripted test