Page History
...
- In order to enable auditing and facilitate diagnosis, data must be stored, and changes, whenever applicable, need to be clearly identified. In Xray, all data is persisted and can be easily tracked using a historical timeline. You can even monitor individual test step modifications - if a step is changed, the record gets into the Xray Test History tab on the Test issue screen (under Activity).
- Historical results (i.e. Test Run details) cannot be modified. The same applies to past changes made on Jira issue-based entities.
- If a Test changes, the recorded results don't. A warning appears in case the user is still allowed to rerun the test. If they are not allowed (because the Test Execution issue is resolved) then the execution cannot even be restarted.
| Info |
|---|
SOC 2 is similar in its goals & controls and includes five Trust Services Criteria: Security, Availability, Processing Integrity, Confidentiality and Privacy. The features described above also align with the sample SOC2 checklists. |
...
Xray provides multiple levels of requirements and different ways to define the hierarchical relationship (e.g., using issue links, sub-tasks).
So, you can reflect the ISO 26262 and ASPICE model for documenting the system (both hardware and software) requirements in a way that clearly reflects the scope, dependencies, status, etc. For this example, let’s assume we are working with an Automated Driving System, here is how a few requirements for it could be arranged:
The hierarchical view is achieved with the Structure add-on for Jira
...
| Info |
|---|
ASIL = Impact x (Probability x Controllability). Although the definition of risk in ISO 31000 is slightly different, these 3 key concepts are still largely applicable for that standard as well. |
We could then classify the requirement entities by ASIL directly, or just impact, or any other user-defined field.
...
We have already set up the ASIL and related values to match ISO 26262 at the requirement/story level. We can choose to maintain the same level of detail at the test level or do a more customized risk assessment - e.g., combining ASIL with complexity for an aggregate Risk Score metric:
You can not only add tests to the execution based on risk values in filters, but also prioritize the execution order accordingly.
...
Epic and Story issues, common in Agile environments, can be covered by creating tests and tracking coverage directly from Jira. From the user story issue screen you can immediately create a Test. It is also easy to go back to the user story, using the issue link that was created automatically between the Test and the user story.
Within the Test Coverage Report, you can have a birds-eye view of quality status of your requirements, based on the tests that cover them and their results. To ease your analysis, you can group stories by specific metrics; for example, by risk level. That way you can have a better understanding of whether a higher priority issue is ready to be released or not.
Coverage information is updated in real-time and is multidimensional. In other words, we can analyze requirements from different perspectives: risk level, priority, version, environment, etc.
Enable audit visibility with full traceability
...
Another useful report, right in Jira, is the Traceability Report. We can analyze our Epics, child Story issues, show the Tests that cover them, along with the reported results and, hopefully not many, defects that may have been found. If the user story is part of an Epic, the corresponding panel from the Traceability Report will automatically show all the Tests that validate the child stories and will provide a quick overview of the Epic status, making it easier to decide whether it can be released or not.
You can also handle reports focused on custom metrics. For example, one of the items for ISO 26262 compliance is estimating random failure rates over the lifetime of the device (Failure Rate= R/T, where R is the number of failures and T is total time). You can start by customizing a template (from scratch or from the template store) to fit your compliance requirements, with all the data to be embedded, logos, and other information (e.g., legal disclaimers). Then Xporter can use a JQL function to get a list of defect-type issues and perform the count on it (or you can consider this snippet option which is a bit trickier).
...
The ASPICE framework consists of a process reference model, a process assessment model, and a process capability model. The process reference model defines a set of processes that are relevant to automotive software development, while the process assessment model provides a method for evaluating the maturity of these processes within an organization. The process capability model provides a way to measure and compare the capability of organizations in the automotive supply chain.
Given the overlap mentioned, we are going to focus on the aspects we didn’t cover in the ISO 26262 section, specifically the process reference model and its validation/testing aspects.
...
XEA can integrate with Xray to bring the best of both worlds: have exploratory testing evidence tracked in Jira, be accessible by the team, and reflect on the related requirements.
In Agile and DevOps teams, it is also especially important to enable adoption of any test automation tool/library and CI/CD tool. With the new feature - Remote Jobs Triggering - you can launch the pipeline action directly from the Test Execution issue. Then, detailed results from the automation framework can be imported to Xray and tracked consistently with other tests. You can take it a step further and link test code to requirements.
Lastly, in Xray it is possible to define several different issue types to be handled as "defects." That makes them easier to distinguish and manage - they can be classified (similar to other entities) by priority, severity, impact, and user-defined fields. They can also be covered explicitly with tests.
| Info |
|---|
A few other relevant automotive standards that we can help with are:
|