Overview

In this article, we will describe how our tools help at different stages of establishing & improving the compliant Quality Management System (QMS), using the PDCA cycle (Plan, Do, Check, Act) as the guideline. 

Plan

This stage is about strategic design of the QMS structure and its processes. We will focus on a few popular universal standards, namely ISO 9001, 27001, and 31000.

ISO 9001

This standard sets out the criteria for a QMS and applies to the organizations engaged in design, development, production, and servicing of goods (i.e., to most software development organizations).

ISO 9001 is based on 7 quality management principles:

In this article (this section and beyond), we will dive deeper into principles 3-6.

Engagement of people

This principle has two important components: empowerment/competence and collaboration.

Regarding competence, it is essential to have documentation covering the basics and nuances of the processes, tools, possible configurations, extensions/integrations, etc. Xray has plenty of self-paced resources (DC/Server, Cloud) to facilitate training, improve knowledge, and ensure seamless adoption. Besides the extensive user guide documentation available online, there are also many courses on Xray Academy to understand Xray essentials and more advanced topics, such as test automation and optimized test design.

To facilitate collaboration and foster clarity, you can invite every team member to participate in quality-related tasks, removing the friction that exists whenever different team roles use siloed tools:

• Artifacts can be shared in a variety of formats.
  
• Comments and evidence can be added to all Xray issue-based entities (e.g., Tests) and also to Test Runs. 

Process approach and Improvement

The effectiveness of the quality management approach depends on how thoroughly it is integrated into all facets of your organization. Having the Atlassian ecosystem as the single source of truth significantly simplifies this process aspect.

Furthermore, Jira and Xray enable multiple customizations to adapt to the evolving organizational needs: 

• Different deployment versions (Server/DC and Cloud) with seamless connectivity between Jira and Xray
• Flexible ways for organizing your project-related items (all-in-one vs. testing entities separately)
• Different levels of objectives with traceability and quality gates
• Improved visibility into interrelated tasks with proper assignments and notifications
• Establish KPIs and quantifiably measure them
• add custom fields to any entity (e.g. probability, severity, performance thresholds)
• implement additional customizations (e.g., ScriptRunner, Automation for Jira)
• integrate with other Jira apps to widen the feature set provided by our tools

Evidence-based decision making

This principle is primarily enabled by detailed reports and formats that promote easier visibility and awareness. With Jira and Xray, you can export data in compliance-focused, human-readable formats and automate data snapshots.

All core entities, including Test Runs (i.e., results), can be exported to PDF, Word, or Excel using fully customizable documents in terms of layout. This is a way to obtain a formal, readable copy of the relevant test data, whether related to test specification or execution.

Two options exist to achieve this: using the built-in Document Generator capabilities or the more complete and flexible Xporter App.

With Xporter, it is possible to automate the creation of these documents and, for example, generate them upon a workflow transition, attach them to an existing Confluence page, or send them via email.

To consolidate the information from Jira and Xray, you can also use Jira Snapshots:

“Easy to make reports, diagnosis is quick and straightforward. The FDA submission requires specification reports and traceability reports. Jira Snapshots compiles these reports from the Jira and Xray data, avoiding burdening the team.”

Caris Life Sciences Success Case

ISO 27001

This standard establishes the requirements for an information security management system. ISO 27001 focuses primarily on maintaining the confidentiality, integrity, and availability of information:

Confidentiality and Information integrity

To support confidentiality and information integrity principles, Jira and Xray allow you to: 

• Implement access control and permissions for any entity (i.e. Project, Story, Test Plan, Test Execution, etc.)
  
  

Source

• Use e-signatures to track explicit approval
• Xray issue-based entities can all be digitally signed, using one of many available Jira apps for that purpose. This is a unique advantage as it provides full control over the core testing activities.

Atlassian is ISO 27001 certified. Xray holds a SOC 2 Type 2 certification.

Availability of data

With Xray, you can ensure data persistence, maintain history visibility, and track changes without tampering.

• Data must be stored to enable auditing and facilitate diagnosis, and changes, whenever applicable, need to be clearly identified. All data is persisted in Xray and can be easily tracked using a historical timeline. You can even monitor individual test step modifications - if a step is changed, the record gets into the Xray Test History tab on the Test issue screen (under Activity).

• Historical results (i.e., Test Run details) cannot be modified. The same applies to past changes made on Jira issue-based entities.
• If a Test changes, the recorded results don't. A warning appears in case the user can rerun the test. If they are not allowed (because the Test Execution issue is resolved), the execution cannot even be restarted. 

ISO 31000

This standard family sets the guidelines for engaging in Enterprise Risk Management (ERM). It guides you in identifying, assessing, treating, and communicating risks.

ISO 31000 cannot be used for certification purposes directly. However, organizations can compare risk management practices with an internationally recognized benchmark, providing sound principles for effective management and corporate governance.

Risk definition & mitigation

ISO 31000 defines risk as the "effect of uncertainty on objectives". There are three key concepts: 

• Potential event
• The probability of that event occurring
• The resulting severity of the outcome, should the event occur

In line with the improvement principle of ISO 9001, ISO 31000 emphasizes the iterative nature of risk management, drawing on new knowledge and analysis for revising actions and controls at each stage of the process.

It would be near impossible to successfully implement and sustain the ISO 31000 risk management standard if an organization's process heavily depends on paper-based communication and record keeping.

In Jira and Xray, you can:

• manage risks (e.g., create, edit, delete) and track their status
• classify risks by impact, probability, and other user-defined fields

We will dive deeper into this topic when discussing the online banking example later.

Do, Check, Act

The Do stage of PDCA is more about the tactical implementation of the QMS. Check and Act stages detail how the quantified results are processed. The goals are to determine the effectiveness and efficiency of each process toward its objectives, to communicate these findings to the stakeholders, and to develop new best practices based on the audit.

This section will focus on the popular standards specific to the financial services sector.

For organizations in that industry, there needs to be a balance between managing compliance and investing in innovative, customer-centric capabilities (e.g., instant payments). Further, as the speed of transition to a digital environment increases, so do regulatory and compliance requirements.

According to Verizon's 2020 Data Breach Investigations Report, out of the 3,950 confirmed breaches, the financial and insurance sector had the most cases, and personal data was accessed in nearly 60% of all breaches.

Before diving into the details, let's summarize the compliance aspects unique to the financial services sector:

• Compliance requirements include 3 categories: financial, anti-money laundering (AML), and data protection.
• One of the major recent trends has been the introduction of banking in the cloud. It demands higher levels of cybersecurity and more safeguards for the customers' personal information.
• The level of regulation and the non-compliance expenses are some of the highest across all industries. Banks worldwide pay billions in regulatory fines and penalties every year.

With that background in mind, for this article, let's assume we are introducing the online banking system at a global financial services provider. We will first summarize the key compliance requirements from GDPR, PCI DSS, and ISO 20022, then demonstrate how they could be accomplished in Xray and Jira Cloud.

GDPR

The European General Data Protection Regulation (EU-GDPR) is a security framework designed by the European Union to protect its citizens from compromised personal data. All businesses processing data linked to EU citizens, either manually or through automated mechanisms, must comply with the GDPR, regardless of the business's physical location. 

GDPR lays out seven protection principles:

1. Lawfulness, fairness, and transparency 
2. Purpose limitation (data must be processed for the legitimate purposes specified explicitly to the data subject)
3. Data minimization (businesses should collect and process only as much data as necessary for the purposes specified)
4. Accuracy
5. Storage limitation (businesses may only store personally identifying data for as long as necessary for the specified purpose)
6. Integrity and confidentiality
7. Accountability

We have covered the features supporting principles 6 and 7 (i.e., anti-tampering, change tracking, access control, and permissions, etc.) in the ISO 27001 section. We will focus on principles 3 and 5 for the online banking example below. 

Both Xray and Jira are also committed to complying with GDPR, for instance:

• encrypting data in transit and at rest;
• providing data residency program; 
• providing optionality within different product or account settings;
• using robust security controls.

PCI DSS

Payment Card Industry Data Security Standards (PCI DSS) is a set of standards for protecting the personal details of credit cardholders and reducing credit card fraud. Organizations that process customer credit card information must comply with PCI DSS, including merchants and payment solution providers. 

PCI DSS has six goals and 12 security requirements for ensuring compliance. The goals are:

1. Building and maintaining a secure network
2. Protecting cardholder data
3. Maintaining a vulnerability management program
4. Implementing strong access control steps
5. Routine monitoring and testing networks
6. Maintaining an up-to-date information security policy

For this article, we will focus on goals 4 and 5 when we discuss the online banking example later.

ISO 20022

ISO 20022 can help overcome such barriers as syntaxes and semantics when exchanging financial information. It is a global, open standard that describes a modeling methodology to capture – in a syntax-independent way – financial business areas, business transactions, and associated message flows. 

By 2025, ISO 20022 is expected to be the common language of the global financial industry (co-existence period with the MT standard until then).

Thanks to its structured and richer data elements, ISO 20022 enables participants to: 

• increase automation in transaction processing, 
• reduce costly manual interventions, 
• improve visibility into cash flows and positions as well as their customers' choices. 

More complete and accurate data leads to more effective and efficient screening, compliance, and anti-money laundering (AML) processes.

For this article, we will need to ensure our online banking system can send messages in both MX and MT formats for various transaction scenarios.

Developing the Online Banking System with Jira and Xray 

It's best to consider risk and compliance early in development and ensure the software product is created with the regulations in mind. That allows you to avoid last-minute chaos and wasting time on numerous corrections in the future. Considering all three standards for our online banking system, we can establish the project in Jira with the following characteristics/in the following way.

Requirements

The access controls and the data integrity measures mentioned in the ISO 27001 section facilitate the fulfillment of critical GDPR and PCI DSS requirements at this stage. 

Define what a requirement is and its relevant metrics

Xray provides multiple levels of requirements and different ways to define the hierarchical relationship (e.g., using issue links, sub-tasks).

Define what a requirement is and its relevant metrics

Xray provides multiple levels of requirements and different ways to define the hierarchical relationship (e.g., using issue links, sub-tasks).

You can represent the compliance requirements in a way that reflects scope, dependencies, progress, etc. For our online banking example:

This hierarchical view is achieved with the Structure add-on for Jira Cloud

We can capture and track key pieces of information in structured and unstructured (description, attachments, mockup links, etc.). Specifically, we can track the relationship to the standard via Labels (or dedicated custom fields) and add high-level risk assessment as the custom field:

We could then classify the requirement entities by the standard membership or any other user-defined field.

Proper risk management

Handling risks is an intrinsic part of good testing and essential to organizations working in highly regulated environments. Xray supports Risk-Based Testing (RBT) and allows you to define risks at different levels: project, requirement, or test. 

We have already set up the aggregate risk score values at the requirement/story level. We can choose to maintain the same level of detail at the test level or break the risk assessment down based on ISO 31000 guidelines (i.e., the custom fields Severity and Probability).

You can not only add tests to the execution based on risk factors in the JQL but also prioritize the execution order accordingly.

If you decide to keep the same risk metrics between requirements and tests, you can set up the rules for inheriting using Jira Automation, so you do not need to set the values for each entity manually.

Implement workflows to have explicit control over the process  

Ensure issues get done by assigning them to users and tracking them using workflows. To elaborate on the list we mentioned in the Process section of ISO 9001, with Jira and Xray it is possible to:

• implement an approval mechanism, commonly having one approver
• Implement quality gates based on field status (see the nuances below)
• make items "read-only" when transitioned to a certain workflow status by setting a Jira property ("jira.issue.editable")
• restrict usage of Tests in a certain workflow status (see how)
• disallow executions for Test Executions in a specific status (see how)

To be clear, Xray does not have an explicit feature for "quality gates." However, Xray has mechanisms to implement quality-related checks, mainly at the workflow status. 

Examples of leveraging ScriptRunner include:

• validating a requirement/story based on their coverage status, before allowing to make a transition
• reopening/transitioning Tests linked to a requirement
• whenever you change the specification of a requirement, you most probably will need to review the Tests that you have already specified
• inhibiting transition of the Test Plan workflow status, unless all tests or a certain % are passing
• can be accomplished with the combination of ScripRunner and Xray GraphQL API

Coverage and traceability

Guarantee requirements coverage

Epic and Story issues, common in Agile environments, can be covered by creating tests and tracking coverage directly from Jira. From the user story issue screen, you can immediately create a Test. It is also easy to go back to the user story using the issue link created automatically between the Test and the user story.

Within the Test Coverage Report, you can have a birds-eye view of the quality status of your requirements based on the tests that cover them and their results. Coverage information is updated in real-time and is multidimensional.

You can group stories by specific metrics, such as the standard or risk level, to ease your analysis. That way, you can have a better understanding of whether a higher-priority issue is ready to be released or not.

Enable audit visibility with full traceability

PCI DSS Requirement 10 " ensures the presentation of an audit trail for all credit card-related processes." Xray helps address such compliance demands by providing full traceability between requirements, tests, their runs, and reported defects, including the relationship and status of each entity. 

Xray implements its test management capabilities using Jira issues. This means that Jira manages all the activity logs and history. We can even look at the code of the requirements to facilitate analysis, as Jira can connect to development tools. Traceability can be evaluated for a specific context (version, environment, etc.), which helps with more precise auditing and diagnosing the impact. 

Another useful artifact, right in Jira, is the Traceability Report. We can analyze our Epics, child Story issues, show the Tests that cover them, along with the reported results and, hopefully, not many defects that may have been found. If the user story is part of an Epic, the corresponding panel from the Traceability Report will automatically show all the Tests that validate the child stories and provide a quick overview of the Epic status, making it easier to decide whether it can be released.

You can also handle reports focused on custom metrics. For example, one of the metrics that help address PCI DSS compliance is the percentage of known vulnerabilities patched or mitigated. You can start by customizing a template (from scratch or the template store) to fit your compliance requirements, with all the data to be embedded, logos, and other information (e.g., legal disclaimers). 

Then, Xporter can use JQL to get two lists of defect-type issues by status, perform the count on each, and then show the division result.

Testing techniques

For the testing side of the requirements from all 3 standards, given the complexity of modern enterprises, all types of testing are necessary, with efficient coordination between them. 

For example, to satisfy PCI DSS, you should perform the following:

• Requirement 6.6: annual web application testing
• Requirement 11.2: quarterly vulnerability scanning
• Requirement 11.3: annual penetration testing

Performance is also important to remember, not just for PCI DSS, but for ISO 20022 as well. For example, you would need to load test APIs to ensure they can handle expected volumes in both MX and MT formats.

No matter which approaches teams choose or the tools they prefer for test automation and CI/CD goals, it is essential to enable all of them in a frictionless way. And our suite of tools is flexible enough to do that.

Xray supports the full arsenal of techniques and offers test case versioning:

• Manual tests (including parameterized ones aligned with the dataset);
• Automated tests (Cucumber, including data-driven outline, and others);
• Exploratory testing using a desktop Xray Exploratory App (XEA).

Storing all those different testing specifications and execution results in a centralized manner with Xray and Jira significantly simplifies the preparation for compliance audits.

So far, we have talked about mitigating risk by evaluating the criticality of each requirement and guaranteeing its coverage with tests. But you can also "fortify" another angle - the coverage of important data interactions within your systems. 

That can be achieved with the optimized scenario generation facilitated by Test Case Designer (TCD; part of Xray Enterprise). Its combinatorial, model-based methodology is based on the research results about the causes of defects in production (e.g. such studies as "Estimating t-Way Fault Profile Evolution During Testing" and "Practical Combinatorial Testing", presented by the National Institute of Standards and Technology in 2017 and 2010 respectively). It is a suggested technique in ISO/IEC/IEEE 29119:4 and allows risk-based fine tuning at the parameter level.

For example, testing access control or MX/MT messaging with TCD, given the number of possible scenarios, could provide significant speed and quality benefits.

Mind Map and Coverage Matrix from a sample TCD model for ISO 20022 testing

Execution

Once the test suite is established, you can launch the execution directly from Jira (for both manual and automated types). Regardless of the testing approach or execution type, Xray can provide visibility of testing results, including evidence, in one place for faster feedback loops. 

Whenever manually executing scripted test cases, we can report results at the step level and attach screenshots as evidence, accelerating the collaboration & defect triaging process.

XEA can integrate with Xray to bring the best of both worlds:

• Have exploratory testing evidence tracked in Jira.
• Be accessible by the team.
• Reflect on the related requirements.

In Agile and DevOps teams, enabling the adoption of any test automation tool/library and CI/CD tool is especially important. With the new feature - Remote Jobs Triggering - you can launch the pipeline action directly from the Test Execution issue. Then, detailed results from the automation framework can be imported to Xray and tracked consistently with other tests. You can take it a step further and link test code to requirements.

This accelerates compliance with, e.g., PCI DSS rule 5 about recurring testing and monitoring and with the requirements about continuously performing vulnerability scans.

Lastly, in Xray, several different issue types can be defined as "defects." That makes them easier to distinguish and manage - they can be classified (similar to other entities) by priority, severity, impact, and user-defined fields. They can also be covered explicitly with tests.