What are Xray's features for handling RCA?

Root Cause Analysis (RCA) involves answering questions like these:

• What is the root cause of a problem?
• When was a problem introduced?
• Why was the problem introduced?
• Where was the problem introduced at?
• How can it be prevented to happen in the future?

When?

• From a time perspective:
  • You may analyze the coverage status of the Test on each version, then you can find out the Test Runs for the first version having problems and look at the revision associated with the related Test Execution. 
  • In which phase of the development cycle?

Where?

Since Bug is linked to:

• Test
  • If automated, we can evaluate the linked code

• Requirement
  • We may evaluate the code that changed on the requirement

• Test Execution
  • We may evaluate on which environment, version and revision of the SUT the problem arises

Why?

There are many reasons; here are just a few of them:

• Unclear, incomplete or inadequate Requirements
  • You may look at the related requirements and check if they were clear and were reviewed.

• Lack of testing at different levels,  including unit and integration testing
  • Integration tests can be seen in the Test Coverage
  • Unit Tests can be assessed on the related commits
• Are the requirements covered by unit and integration tests?

• Communication gap and reviews
  • Did communication and collaboration happened, on requirements and also on the specification of tests?
  • Were requirements reviewed?
  • Was a proper test plan defined and has it been reviewed?
  • Were tests themselves reviewed?

 In Xray, most of the entities including requirements, Tests, Pre-Conditions, Test Sets, Test Executions and Test Plans are implemented like issues and thus you may apply workflows on them as means to enforce revision and approval rules. Since they are issues, users can easily collaborate and comment on them, track changes, assign them to users and watch them, etc.

• Problems with Test Environment
  • In Xray, in the respective Test Execution(s), you may see which Test Environment was used
  • Was a proper Test Environment used?
  • Was the Test Environment in a healthy state?

• Weak Test Coverage
  • In Xray, you may evaluate the Test Coverage on the requirement issue screen or on the Overall Requirement Coverage report.
  • Few test cases and not the best-tailored test cases for a given requirement

• Lack of regression testing
  • You may evaluate if regression tests are being performed
  • Could that bug be caught by a specific (automated) regression test?

• Was the automated tests code reviewed?
  • In Xray, a Test is an issue thus you may commit code associated with it; you may look at that code from the Test issue screen and check if it has been reviewed

How (to prevent it)?

• Assess coverage
• Enforce revision of requirements specification, Tests and Test Plans
• Link automation code to tests and make it reviewable
• Ensure regression testing is happening

Related articles

Best practices on writing great Test cases