Overview

This document provides somes tips to improve performance of your Jira instance while using Xray.

As Xray is built on top of Jira, the first thing you neeed to make sure is that your Jira instance is properly configured and tunned beforehand. Thus, we recommend looking at Jira specifics section first.

After ensuring your Jira instance is fine-tuned, then you may proceed to Common Tips as they apply both to Server and Data Center based deployments. In the end of the document, you have some specific Data Center Tips that complement the previous section.

Background

Xray mainly uses Jira issue types for implementing Test Management related entities, such as:

The only relevant exception are Test Runs. A Test Run is an internal entity managed by Xray itself, which is an instance of a Test containing also its result related data.

Everytime a Test is executed within some Test Execution context, a Test Run is created. Thus, Test Runs are a potential variable affecting performance, amongst other ones.


The amount of projects or amount of issues in Jira by themselves may not affect at all Xray's performance. From Xray perspective, performance will depend on usage patterns in terms of testing, including the amount of Test Runs and on-demand reporting.

Common Tips

This document sums up tips to keep the performance of Xray and JIRA as optimized as possible.

The tips are grouped by area or topic; therefore, they do not follow any specific order. However, you should start by looking at the Process section first as it 

Within each area/topic, tips will be presented by descending order of risk; each tip is preceeded by a icon that corresponds to the risk level.  


Legend:

Jira specifics

 Xray is built on top of Atlassian's Jira, therefore is mostly dependant on the architecture and technologies followed by Jira itself.

Atlassian provides some peformance tips related with scaling, that thus also apply to Xray.

Large organizations or organizations with huge amounts of data and/or having many users should consider Jira Data Center, which increases performance and improves throughput on higher concurrency usage scenarios; it also provides high-availability for critical scenarios.


  1. moderate the usage of custom fields

    1. Managing custom fields in JIRA effectively

    2. Optimizing custom fields
  2. ??

Process

  1.  "Unified" development process: define a process that can be applied to all teams in the way how to manage the STLC. Every team is different and has its own needs, therefore your process should not be too strict but it should provide some guidance on how development life cycle should be addressed, covering requirement management, bug management and test management. Having teams working completely in different ways, hardens communication and leads to unproper and unoptimized tool usage. If you have a well-defined process that can be used organization wide, better; this is the key to ensure an optimal usage and have the best performance.
  2.  Are you adopting Agile and Scrum? Check out Using Xray in an Agile context for tips on how you can take advantage of Xray in such scenarios;. Agile software development page provides high-level overview of Agile and Agile Testing and besides background information on them, it will also provide some useful tips so your team can be more Agile and avoid doing things that are unnecessary.

Specification

  1.  Avoid having many (>>10) sub-requirements per requirement (e.g. Stories per Epic) as it can impact the calculation of their statuses
    1. normally it is a signal that the requirement needs to be further decomposed. Besides hardening analysis and its management, it will also require additional resources during computation of its status upon changes in any of the related sub-requirements, that in turn is affected by the status of the related Tests.
  2.  Requirements being covered by many (>>20) Tests
    1. normally it is a signal that the requirement needs to be further decomposed. Besides hardening analysis and its management, it will also require additional resources during computation of its status, that in turn is affected by the status of the related Tests.

Organizing

  1.  If you have thousands of Tests, using the Test Repository approach may provide additional benefits over using "lists" (i.e. Test Set issues), as it makes management of test cases easier while avoiding the creation of issues.

Planning

  1.  Instead of creating one Test Plan for your release, you may create multiple Test Plans to track different Tests (e.g. manual vs automated); this may be useful if you want to have clear visibility of how certain groups of Tests are progressing and if their execution lifecycle is different from other ones. It will also make your Test Plans considerable lighter.
  2.  If adopting Scrum, create Test Plans per Sprint, to track the testing being done in the scope of your Sprint; manage them as artifacts of your Sprint and add them to your Scrum boards so everyone sees their progress. Per each Sprint you may have more than one Test Plan; check out some possible usage patterns here. 

Execution

  1.  don't create dozens or hundreds of Test Environments; don't try to do data-driven testing using test environments

Entities

Tests

  1.  Don't add many custom fields to Tests as it will add some additional overhead to Jira.
  2.  We recommend up to 5 linked Tests per each requirement; ideally a Test should be focused on the validation of one requirement.
  3.  Promote reusability and avoid cloning Test cases, if they’re the same. A Test can be reused multiple times and can be used to cover more than one requirement (if really needed), no matter in which project it is located.

Pre-Conditions

  1.  Although the impact is neglectable, try to use Pre-Conditions as means to have manageable end reusable initial conditions that you can link to multiple Tests. That will avoid creating additional steps in all those test cases.

Test Sets

  1.  Although there isn’t an hard limit, we recommend having no more than 2000 Tests in a given Test Set mostly to ease their management. This limit may be easily superseded depending on Jira instance deployment configuration.

Test Repository

  1.  Creating a Test Plan from within some Test Repository folder that has many Tests and sub-folders can take some time, if you choose to replicate the folder structure into the destination Test Plan Board. However, this is temporary.
  2.  Avoid many folders shown at the same time as it will impact browser performance at some time. You can do this by limiting of direct child folders you create at a given parent folder and by using the "Expand all" moderately. The amount of folders you have does not affect by itself your Jira backend performance though.

Test Executions

  1.  Reuse existing Test Executions (and related Test Runs), if possible, to make calculations faster througout the application and thus make reports and some panels, for example, also faster
    1. if you don't need historical data of interim builds, you may reuse an existing Test Execution and update the corresponding Test Runs. During CI, depending on the endpoint being used, you may specify a Test Execution issue key. In such case, Xray will overwrite the existing Test Runs information, or add new ones if the Test is not yet part of the Test Execution.
  2.  Clean-up old, unneeded executions related data, to make calculations faster througout the application and thus make reports and some panels, for example, also faster

    1. If your organization performs a high number of Test Executions (consequently creating also a high number of Test Runs) we recommend deleting old Test Executions issues from time to time. This recommendation applies to organizations that import many automated executions daily using the REST API.

      The clean-up process must be scheduled for a low Jira usage period because when Test Executions are deleted, Xray will re-calculate TestRunStatus and Requirement Status. This might temporarily slow down the Jira instance depending on the amount of issues affected.

    2. Deleting Test Runs can affect the calculated and consolidated status of your Tests and of your requirements for all scopes (e.g. versions or Test Plans + Test Environments); please use carefully
  3.  Although there isn’t an hard limit, we recommend having no more than 2000 Tests in a given Test Execution mostly to ease their management. This limit may be easily superseded depending on Jira instance deployment configuration.

Test Plans

  1.  Xray provides a setting "Max number of Tests per Test Plan" where you may define a soft limit for the number of Tests within Test Plans. Although you may adjust this value, we recommend having no more than 2000 Tests in a given Test Plan:
    1. to ease their management;
    2. and to make it more performant and lighter.
    This limit may be easily superseded depending on Jira instance deployment configuration.
    Note that a Test Plan aggregates and consolidates the results of the related Test Executions and Tests, thus the overall number of Test Runs you'll have can add some overhead to the calculation of the consolidated results.

Test Plan Board

  1.  Using the Test Plan Board as means to do operations over certain Tests of the Test Plan (e.g. schedule Test Executions for them) and to track results can be more efficient than using the Test Plan issue screen. Although the previous two are not equivalent, the Board provides essentialy the same operations while being lighter most of the times as it not shows all the information you can see in the Test Plan issue screen.
  2.  Avoid many folders shown at the same time as it will impact browser performance at some time. You can do this by limiting of direct child folders you create at a given parent folder and by using the "Expand all" moderately. The amount of folders you have does not affect by itself your Jira backend performance though.

Integrations

Automation & Continuous Integration

  1.  Upload only relevant test results (e.g. don’t upload unit test results); choose properly what testing results you want to track within Xray
  2.  Choose properly the upload frequency
    1. aggregate relevant results in some job run periodically (e.g. hourly, daily) and submit those
  3.  Don’t mix the CI tool with TM tool

REST API

  1.  Review the "Max results per request" setting in the the Miscellaneous administration settings as it controls the pagination on the REST API calls. The default value should be ok.
  2.  export results - custom fields
  3. https://confluence.xpand-it.com/display/XRAY/Export+Execution+Results+-+REST
  4. diamantino vai ver o q mandou pra ubs
  5.  Limit API calls (Jira and Xray related ones) using a reverse proxy
    1. Evaluate what REST API calls are being used, discuss their real need with users
      1. Make sure that pagination is being used on the REST API calls
    2. Restrict access to REST API calls
      1. Limit access to well-known hosts/applications

Reporting

JQL

Xray provides dozens of JQL functions but you have to use them carefuly to make sure your instance is not affected. Please do train your users on JQL before "allowing" to use them throughout Jira.

  1.   Using unoptimized JQL queries can degrade performance substantially
    1. most times this happens because users don't understand how JQL works first of all; JQL is not like SQL (see Understanding JQL Performance). Thus, filtering issues by project by adding the "project = <xxx>" clause is not the same as specifying the project as argument to the subsequent JQL function.
      1. Example: 

        1. Use

          issue in requirements('OK','CALC')

          ...instead of ...

          project = 'CALC' and issue in requirements('OK')

  2.  Some JQL functions, such as the ones dealing with requirement coverage, may be more intensive than other ones, since Xray may have to, for example, load all the related Test Runs in order to obtain relevant data. The following JQL 
    1. PIOR testPlanTests() - 
    2. requirements(), with dates
    3. testExecutionTests() (diamantino tem duvidas?)  - whenever filtering by tests/requirements in a given status; it will depend on the amount of Tests you have on the Test Execution
    4. parentRequirements() (diamantino tem duvidas?)  - depending on the amount of requirements and sub-requirements you have, it can take a while to complete and require some resources

Diamantino Campos , Any other Jql function?


Xray calculated custom fields

  1.  Xray provides some specific custom fields that calculate their values on the fly. This means that you should have that in mind, specially if you're including them in tables/issue listings/gadgets.

Test Execution Defects

test set status

tests count



Reports and Gadgets

  1.  One way of doing reporting is by using gadgets. Gadgets are great to share information between team members and even between different teams; however, if not use carefully, they can degrade JIRA performance if all users have the same report on their dashboard as they will probably generate multiple requests once users access the dashboard. Thus, use carefully most intensive gadgets such as the "Historical Daily Requirement Coverage" and the "Tests Evolution" gadget and others that do aggregations (e.g. "Test Runs Summary" gadget). Gadgets that just "list" entities should not affect performance significatively.
  2.  Limit the target issues for the reports/gadget CONF, e.g. generate the Overall Requirement Coverage (report/gadget) just for issues that you really need and not all Jira requirements or projects.

Dashboards

  1. Use dashboards to have the high-level overview and the things that matter
  2. Avoid creating highly complex dashboards or dashboards
  3. Try to normalize the dashboards and make them like a standard organization wid
    1. facilitates communication
    2. avoids “wrong” usage
  4. Choose properly the filters you use for each gadget, in order to restrict the amount of issues that will be processed
  5. refresh times

Custom Reports

  1.  If possible, use eazyBI for making custom table/chart based reports with drill-down capabilities; it is highly flexible and works in its own database, thus not impacting Jira unless the database is hosted in the same instance of Jira server/database
  2.  If using the Xporter app to build custom PDF, Word or Excel based reports, then please see Xporter specific recommendations.

Administration and Customization

Settings

Xray comes with default settings that are ok for most organizations. Nevertheless, have a look at the different settings; it’s an opportunity to know the product better and for you to think on ways to fine tune it later.

  1.  Review the settings "Max Test Runs for bulk operations", "Max number of requirements per report or gadget results", "Max number of Tests per Test Plan", "Max results per request" in the the Miscellaneous administration settings. The default values should be ok.

Your own custom fields

  1.  Beware with calculated custom fields implemented using some customization app (e.g. Jira Misc Custom Fields, ScriptRunner, etc) as they will be calculated each time they're accessed and they can impact things such as
  2.  Adding a lot of custom fields on Jira will add some overhead on performance; therefore, Atlassian itself recommends a responsible usage of custom fields

Workflows on Xray entities

  1.  Don't use complex workflows for Xray entities as it will harden their usage

Reindex???

?? avaliar questao de tirar searcher ao Test Execution Defects



bulk cloning test plans ou test executions using addons

Data Center Tips

  1.  Review Atlassian's best practices for Data Center and the Node sizing overview for Atlassian Data Center article
  2.  Configure your load balancer properly to use dedicated nodes for REST API calls, which can reduce impact on other application nodes
  3. Integration with other apps
    1.  See eazyBI's Data Center related recommendations here

References