|
|
Locust is a load testing tool that uses Scala to write tests.
Using an expressive DSL and having scenarios defined in code makes its code suitable to be kept in a version control system.
Gatling also has an Enterprise version, that was formerly known as Gatling FrontLine, it is a management interface for Gatling, that includes advanced metrics and advanced features for integration and automation.
For this example, we will use Gatling to define a series of Performance tests using the Maven plugin available. We will use the assertions to define KPIs that will fail or succeed the tests. You will need:
|
To start, define a simple load test in Gatling that will target a demo site (travel agency) supplied by BlazeMeter that you can find here.
The test will exercise 3 different endpoints:
To start using Gatling please follow the documentation.
In the documentation you will find that there are several ways to use the tool, in our case we are using the Maven Plugin available, since we will use this code in a CI/CD tool further ahead.
Before jumping into the code you can find below the pom.xml file content for the project after it has been configured with the Maven plugin.
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.xpandit.xray.tutorials</groupId> <artifactId>gatling-perf</artifactId> <version>3.6.1</version> <properties> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <gatling.version>${project.version}</gatling.version> <gatling-maven-plugin.version>3.1.2</gatling-maven-plugin.version> <maven-jar-plugin.version>3.2.0</maven-jar-plugin.version> <scala-maven-plugin.version>4.4.1</scala-maven-plugin.version> </properties> <dependencies> <dependency> <groupId>io.gatling.highcharts</groupId> <artifactId>gatling-charts-highcharts</artifactId> <version>${gatling.version}</version> <scope>test</scope> </dependency> </dependencies> <build> <testSourceDirectory>src/test/scala</testSourceDirectory> <sourceDirectory>main/scala</sourceDirectory> <plugins> <plugin> <artifactId>maven-jar-plugin</artifactId> <version>${maven-jar-plugin.version}</version> </plugin> <plugin> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <version>${scala-maven-plugin.version}</version> <executions> <execution> <goals> <goal>testCompile</goal> </goals> <configuration> <jvmArgs> <jvmArg>-Xss100M</jvmArg> </jvmArgs> <args> <arg>-target:jvm-1.8</arg> <arg>-deprecation</arg> <arg>-feature</arg> <arg>-unchecked</arg> <arg>-language:implicitConversions</arg> <arg>-language:postfixOps</arg> </args> </configuration> </execution> </executions> </plugin> <plugin> <groupId>io.gatling</groupId> <artifactId>gatling-maven-plugin</artifactId> <version>${gatling-maven-plugin.version}</version> </plugin> </plugins> </build> </project> |
The tests, as we have defined above, will target 3 different endpoints. We started by extending the Simulation class of Gatling signalling that this class will hold our simulation.
... class MySimulation extends Simulation { ... |
Next we have created 3 objects that are mirroring the operations we want to exercise:
For each we have defined the endpoint we want to access, the parameters needed to perform the operation and a waiting time at the end to simulate a real user, as you can see below:
...object Login { val login = exec(http("Access Reserve").post("http://blazedemo.com/reserve.php") .formParam("""fromPort""", """Paris""") .formParam("""toPort""", """Buenos+Aires""")) .pause(2, 3) } object Reserve { val reserve = exec(http("Access Reserve").post("http://blazedemo.com/reserve.php") .formParam("""fromPort""", """Paris""") .formParam("""toPort""","""Buenos+Aires""")) .pause(2, 3) } object Purchase { val purchase = exec(http("Access Purchase").post("http://blazedemo.com/purchase.php") .formParam("""fromPort""", """Paris""") .formParam("""toPort""", """Buenos+Aires""") .formParam("""airline""", """Virgin+America""") .formParam("""flight""", """43""") .formParam("""price""", """472.56""")) .pause(2, 3) }... |
Another thing we will need is the protocol definition for the simulation, in our case we are using HTTP protocol and have defined it with some default values, notice nevertheless the baseUrl pointing to the endpoint of the application.
... val httpProtocol = http .baseUrl("http://blazedemo.com") .acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") .doNotTrackHeader("1") .acceptLanguageHeader("en-US,en;q=0.5") .acceptEncodingHeader("gzip, deflate") .userAgentHeader("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20100101 Firefox/16.0") ... |
Finally we need to define the user scenarios, in our case it will be a one-to-one correspondence, we have created one user per operation thus creating 3 different scenarios:
This will help define the different definitions of profile injections of users we want to simulate.
... val loginUsers = scenario("LoginUsers").exec(Login.login) val reserveUsers = scenario("ReserveUsers").exec(Reserve.reserve) val purchaseUsers = scenario("PurchaseUsers").exec(Purchase.purchase) ... |
Notice that this is only one of the possibilities to define a load test, Gatling has different ways to support your performance testing, for more information please check the documentation.
After having all of that defined we need to instruct Gatling on how to use that information to execute the load test, for that Gatling have the SetUp function.
In our case we have defined it like below:
setUp( loginUsers.inject(atOnceUsers(10)), reserveUsers.inject(rampUsers(2).during(10.seconds)), purchaseUsers.inject(rampUsers(1).during(10.seconds)) ).protocols(httpProtocol) |
In more detail:
To execute the tests you can use several ways, in our case we are using the command line.
mvn gatling:test |
The command line output will look like this:
This will be enough to execute performance tests, however a manual validation of results must always be done in the end to assess if the performance is enough or not, and looking at Json files is not always easy.
We need the ability to:
In order to do that we will use the assertions available in Gatling and build a converter function that will convert the assertions.json produced by Gatling into an Xray Json Test Result file ready to be imported to Xray.
In order to use performance tests in a pipeline we need tests to be able to fail the build if the result is not the expected. For this we need to have the ability to automatically assess if the performance tests were successful (within the parameters we have defined) or not.
Gatling has out of the box the ability to define assertions, in our case we want to define the following ones globally:
To achieve this we have added the following assertions in the setUp :
setUp( loginUsers.inject(atOnceUsers(10)), reserveUsers.inject(rampUsers(2).during(10.seconds)), purchaseUsers.inject(rampUsers(1).during(10.seconds)) ).assertions( global.responseTime.percentile(90).lt(5000), global.failedRequests.count.lte(0), global.requestsPerSec.lt(500) ).protocols(httpProtocol) |
Once we execute the test again we will notice that now we have information about the assertions and those results can be acted upon:
Now we are executing Tests to validate the performance of our application and we are capable of defining KPIs to validate each performance indicator in a build (enable us to add these Tests to CI/CD tools given that the execution time is not long), so what we need is to be able to ship these results to Xray to bring visibility over these types of Tests also.
Gatling produces HTML files with a detailed report of the tests and also some Json files with the details of the assertions and the stats of the respective tests (valuable to perform a post analysis but hard to convert into proper pass or fail result).
We need to produce a result that will hold all the information produced and will bring value to our project. To do so, we are going to create an Xray Json report to hold these results.
In Xray we have defined one Test and one Test Plan:
This will serve to centralize all the results of the performance executions in each sprint and bring visibility to the team of those results providing an overall view of the status of the project.
Gatling generates several files but for this example we will convert the assertions.json (in the ./target/gatling/mysimulation-DATE/js, you will find two files: assertions.json and stats.json).
The assertions file will have the details regarding the assertions we have defined in the Test, so we are going to convert that into a valid Xray Json file with a small script developed in python.
import json, argparse from base64 import b64encode class convert2XrayJson: def injectFile(self, fileName): with open(str(fileName), 'rb') as open_file: byte_content = open_file.read() return b64encode(byte_content).decode('utf-8') def appendToXrayResult(self, data, testkey, metric, name, value, comment, status, projectkey, testplankey, evidencefile): done = False if len(data['tests']) > 0: for tests in data['tests']: for key, value in tests.items(): if key == 'testKey' and value == testkey: tests['results'].append({ 'name': metric + ' for ' + name, 'log': comment, 'status': 'PASSED' if status else 'FAILED' }) done = True if not done: info = { 'info': { 'summary': ' Perf test', 'description': 'Perf test', 'project': projectkey, 'testPlanKey': testplankey, }, } data['tests'].append({ 'testKey': testkey, 'comment': metric, 'status': 'PASSED' if status else 'FAILED', 'results': [ { 'name': metric + ' for ' + name, 'log': comment, 'status': 'PASSED' if status else 'FAILED' } ], 'evidences': [ { 'data': self.injectFile(evidencefile), 'filename': evidencefile.rsplit('/', 1)[-1], 'contentType': 'application/json' } ] }) data.update(info) ## _________________________________________________ parser = argparse.ArgumentParser(description='Helper to convert Gatling assertions output to Xray Json') parser.add_argument('--gatlingFile', dest='gatlingfile', type=str, help='Path of the Gatling assertion file') parser.add_argument('--outputFile', dest='outputfile', type=str, help='Name of the Xray Json output file') parser.add_argument('--testKey', dest='testkey', type=str, help='Key of the test to associate in Xray') parser.add_argument('--testPlan', dest='testplan', type=str, help='Test Plan key to associate in Xray') parser.add_argument('--jiraProject', dest='jiraproject', type=str, help='Jira project key') parser.add_argument('--evidenceFile', dest='evidencefile', type=str, help='File to add as an evidence') args = parser.parse_args() gatlingfile = args.gatlingfile outputfile = args.outputfile testkey = args.testkey testplan = args.testplan jiraproj = args.jiraproject evidencefile = args.evidencefile data = {} data['tests'] = [] cXray = convert2XrayJson() with open(gatlingfile) as json_file: filedata = json.load(json_file) for p in filedata['assertions']: cXray.appendToXrayResult(data, testkey, p['target'], p['path'], testplan, p['message'], p['result'], jiraproj, testplan, evidencefile) with open(outputfile, 'w') as outfile: json.dump(data, outfile) |
The usage is straight forward and can is explained with the helper function available if you use:
python convert2XrayJson.py -h usage: convert2XrayJson.py [-h] [--gatlingFile GATLINGFILE] [--outputFile OUTPUTFILE] [--testKey TESTKEY] [--testPlan TESTPLAN] [--jiraProject JIRAPROJECT] [--evidenceFile EVIDENCEFILE] Helper to convert Gatling assertions output to Xray Json optional arguments: -h, --help show this help message and exit --gatlingFile GATLINGFILE Path of the Gatling assertion file --outputFile OUTPUTFILE Name of the Xray Json output file --testKey TESTKEY Key of the test to associate in Xray --testPlan TESTPLAN Test Plan key to associate in Xray --jiraProject JIRAPROJECT Jira project key --evidenceFile EVIDENCEFILE File to add as an evidence |
One example of the execution of the tool is:
python convert2XrayJson.py --gatlingFile /target/gatling/mysimulation-20211007103948126/js/assertions.json --outputFile xrayJson.json --testKey 'XT-246' --testPlan 'XT-245' --jiraProject XT --evidenceFile /target/gatling/mysimulation-20211007103948126/js/stats.json |
The Xray Json file generated is:
{"info": {"project": "XT", "testPlanKey": "XT-245", "description": "Perf test", "summary": " Perf test"}, "tests": [{"comment": "90th percentile of response time", "status": "PASSED", "results": [{"status": "PASSED", "name": "90th percentile of response time for Global", "log": "Global: 90th percentile of response time is less than 5000.0"}, {"status": "PASSED", "name": "count of failed events for Global", "log": "Global: count of failed events is less than or equal to 0.0"}, {"status": "PASSED", "name": "mean requests per second for Global", "log": "Global: mean requests per second is less than 500.0"}], "testKey": "XT-246", "evidences": [{"contentType": "application/json", "data": "ewogICJ0eXBlIjogIkdST1VQIiwKIm5hbWUiOiAiR2xvYmFsIEluZm9ybWF0aW9uIiwKInBhdGgiOiAiIiwKInBhdGhGb3JtYXR0ZWQiOiAiZ3JvdXBfbWlzc2luZy1uYW1lLWIwNmQxIiwKInN0YXRzIjogewogICAgIm5hbWUiOiAiR2xvYmFsIEluZm9ybWF0aW9uIiwKICAgICJudW1iZXJPZlJlcXVlc3RzIjogewogICAgICAgICJ0b3RhbCI6IDI2LAogICAgICAgICJvayI6IDI2LAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAibWluUmVzcG9uc2VUaW1lIjogewogICAgICAgICJ0b3RhbCI6IDE3OSwKICAgICAgICAib2siOiAxNzksCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJtYXhSZXNwb25zZVRpbWUiOiB7CiAgICAgICAgInRvdGFsIjogMzIzNywKICAgICAgICAib2siOiAzMjM3LAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAibWVhblJlc3BvbnNlVGltZSI6IHsKICAgICAgICAidG90YWwiOiAxNzMxLAogICAgICAgICJvayI6IDE3MzEsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJzdGFuZGFyZERldmlhdGlvbiI6IHsKICAgICAgICAidG90YWwiOiAxMzM0LAogICAgICAgICJvayI6IDEzMzQsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJwZXJjZW50aWxlczEiOiB7CiAgICAgICAgInRvdGFsIjogNjM4LAogICAgICAgICJvayI6IDYzOCwKICAgICAgICAia28iOiAwCiAgICB9LAogICAgInBlcmNlbnRpbGVzMiI6IHsKICAgICAgICAidG90YWwiOiAzMTU3LAogICAgICAgICJvayI6IDMxNTcsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJwZXJjZW50aWxlczMiOiB7CiAgICAgICAgInRvdGFsIjogMzIxOCwKICAgICAgICAib2siOiAzMjE4LAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAicGVyY2VudGlsZXM0IjogewogICAgICAgICJ0b3RhbCI6IDMyMzcsCiAgICAgICAgIm9rIjogMzIzNywKICAgICAgICAia28iOiAwCiAgICB9LAogICAgImdyb3VwMSI6IHsKICAgICJuYW1lIjogInQgPCA4MDAgbXMiLAogICAgImNvdW50IjogMTQsCiAgICAicGVyY2VudGFnZSI6IDU0Cn0sCiAgICAiZ3JvdXAyIjogewogICAgIm5hbWUiOiAiODAwIG1zIDwgdCA8IDEyMDAgbXMiLAogICAgImNvdW50IjogMCwKICAgICJwZXJjZW50YWdlIjogMAp9LAogICAgImdyb3VwMyI6IHsKICAgICJuYW1lIjogInQgPiAxMjAwIG1zIiwKICAgICJjb3VudCI6IDEyLAogICAgInBlcmNlbnRhZ2UiOiA0Ngp9LAogICAgImdyb3VwNCI6IHsKICAgICJuYW1lIjogImZhaWxlZCIsCiAgICAiY291bnQiOiAwLAogICAgInBlcmNlbnRhZ2UiOiAwCn0sCiAgICAibWVhbk51bWJlck9mUmVxdWVzdHNQZXJTZWNvbmQiOiB7CiAgICAgICAgInRvdGFsIjogMy4yNSwKICAgICAgICAib2siOiAzLjI1LAogICAgICAgICJrbyI6IDAKICAgIH0KfSwKImNvbnRlbnRzIjogewoicmVxX2FjY2Vzcy1yZXNlcnZlLTQ2MzU0IjogewogICAgICAgICJ0eXBlIjogIlJFUVVFU1QiLAogICAgICAgICJuYW1lIjogIkFjY2VzcyBSZXNlcnZlIiwKInBhdGgiOiAiQWNjZXNzIFJlc2VydmUiLAoicGF0aEZvcm1hdHRlZCI6ICJyZXFfYWNjZXNzLXJlc2VydmUtNDYzNTQiLAoic3RhdHMiOiB7CiAgICAibmFtZSI6ICJBY2Nlc3MgUmVzZXJ2ZSIsCiAgICAibnVtYmVyT2ZSZXF1ZXN0cyI6IHsKICAgICAgICAidG90YWwiOiAxMiwKICAgICAgICAib2siOiAxMiwKICAgICAgICAia28iOiAwCiAgICB9LAogICAgIm1pblJlc3BvbnNlVGltZSI6IHsKICAgICAgICAidG90YWwiOiAxNzksCiAgICAgICAgIm9rIjogMTc5LAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAibWF4UmVzcG9uc2VUaW1lIjogewogICAgICAgICJ0b3RhbCI6IDMyMzcsCiAgICAgICAgIm9rIjogMzIzNywKICAgICAgICAia28iOiAwCiAgICB9LAogICAgIm1lYW5SZXNwb25zZVRpbWUiOiB7CiAgICAgICAgInRvdGFsIjogMjkxOSwKICAgICAgICAib2siOiAyOTE5LAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAic3RhbmRhcmREZXZpYXRpb24iOiB7CiAgICAgICAgInRvdGFsIjogODI3LAogICAgICAgICJvayI6IDgyNywKICAgICAgICAia28iOiAwCiAgICB9LAogICAgInBlcmNlbnRpbGVzMSI6IHsKICAgICAgICAidG90YWwiOiAzMTU2LAogICAgICAgICJvayI6IDMxNTYsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJwZXJjZW50aWxlczIiOiB7CiAgICAgICAgInRvdGFsIjogMzE2MSwKICAgICAgICAib2siOiAzMTYxLAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAicGVyY2VudGlsZXMzIjogewogICAgICAgICJ0b3RhbCI6IDMyMzcsCiAgICAgICAgIm9rIjogMzIzNywKICAgICAgICAia28iOiAwCiAgICB9LAogICAgInBlcmNlbnRpbGVzNCI6IHsKICAgICAgICAidG90YWwiOiAzMjM3LAogICAgICAgICJvayI6IDMyMzcsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJncm91cDEiOiB7CiAgICAibmFtZSI6ICJ0IDwgODAwIG1zIiwKICAgICJjb3VudCI6IDEsCiAgICAicGVyY2VudGFnZSI6IDgKfSwKICAgICJncm91cDIiOiB7CiAgICAibmFtZSI6ICI4MDAgbXMgPCB0IDwgMTIwMCBtcyIsCiAgICAiY291bnQiOiAwLAogICAgInBlcmNlbnRhZ2UiOiAwCn0sCiAgICAiZ3JvdXAzIjogewogICAgIm5hbWUiOiAidCA+IDEyMDAgbXMiLAogICAgImNvdW50IjogMTEsCiAgICAicGVyY2VudGFnZSI6IDkyCn0sCiAgICAiZ3JvdXA0IjogewogICAgIm5hbWUiOiAiZmFpbGVkIiwKICAgICJjb3VudCI6IDAsCiAgICAicGVyY2VudGFnZSI6IDAKfSwKICAgICJtZWFuTnVtYmVyT2ZSZXF1ZXN0c1BlclNlY29uZCI6IHsKICAgICAgICAidG90YWwiOiAxLjUsCiAgICAgICAgIm9rIjogMS41LAogICAgICAgICJrbyI6IDAKICAgIH0KfQogICAgfSwicmVxX2FjY2Vzcy1wdXJjaGFzZS0zZGE0YiI6IHsKICAgICAgICAidHlwZSI6ICJSRVFVRVNUIiwKICAgICAgICAibmFtZSI6ICJBY2Nlc3MgUHVyY2hhc2UiLAoicGF0aCI6ICJBY2Nlc3MgUHVyY2hhc2UiLAoicGF0aEZvcm1hdHRlZCI6ICJyZXFfYWNjZXNzLXB1cmNoYXNlLTNkYTRiIiwKInN0YXRzIjogewogICAgIm5hbWUiOiAiQWNjZXNzIFB1cmNoYXNlIiwKICAgICJudW1iZXJPZlJlcXVlc3RzIjogewogICAgICAgICJ0b3RhbCI6IDEsCiAgICAgICAgIm9rIjogMSwKICAgICAgICAia28iOiAwCiAgICB9LAogICAgIm1pblJlc3BvbnNlVGltZSI6IHsKICAgICAgICAidG90YWwiOiAzMTU4LAogICAgICAgICJvayI6IDMxNTgsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJtYXhSZXNwb25zZVRpbWUiOiB7CiAgICAgICAgInRvdGFsIjogMzE1OCwKICAgICAgICAib2siOiAzMTU4LAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAibWVhblJlc3BvbnNlVGltZSI6IHsKICAgICAgICAidG90YWwiOiAzMTU4LAogICAgICAgICJvayI6IDMxNTgsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJzdGFuZGFyZERldmlhdGlvbiI6IHsKICAgICAgICAidG90YWwiOiAwLAogICAgICAgICJvayI6IDAsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJwZXJjZW50aWxlczEiOiB7CiAgICAgICAgInRvdGFsIjogMzE1OCwKICAgICAgICAib2siOiAzMTU4LAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAicGVyY2VudGlsZXMyIjogewogICAgICAgICJ0b3RhbCI6IDMxNTgsCiAgICAgICAgIm9rIjogMzE1OCwKICAgICAgICAia28iOiAwCiAgICB9LAogICAgInBlcmNlbnRpbGVzMyI6IHsKICAgICAgICAidG90YWwiOiAzMTU4LAogICAgICAgICJvayI6IDMxNTgsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJwZXJjZW50aWxlczQiOiB7CiAgICAgICAgInRvdGFsIjogMzE1OCwKICAgICAgICAib2siOiAzMTU4LAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAiZ3JvdXAxIjogewogICAgIm5hbWUiOiAidCA8IDgwMCBtcyIsCiAgICAiY291bnQiOiAwLAogICAgInBlcmNlbnRhZ2UiOiAwCn0sCiAgICAiZ3JvdXAyIjogewogICAgIm5hbWUiOiAiODAwIG1zIDwgdCA8IDEyMDAgbXMiLAogICAgImNvdW50IjogMCwKICAgICJwZXJjZW50YWdlIjogMAp9LAogICAgImdyb3VwMyI6IHsKICAgICJuYW1lIjogInQgPiAxMjAwIG1zIiwKICAgICJjb3VudCI6IDEsCiAgICAicGVyY2VudGFnZSI6IDEwMAp9LAogICAgImdyb3VwNCI6IHsKICAgICJuYW1lIjogImZhaWxlZCIsCiAgICAiY291bnQiOiAwLAogICAgInBlcmNlbnRhZ2UiOiAwCn0sCiAgICAibWVhbk51bWJlck9mUmVxdWVzdHNQZXJTZWNvbmQiOiB7CiAgICAgICAgInRvdGFsIjogMC4xMjUsCiAgICAgICAgIm9rIjogMC4xMjUsCiAgICAgICAgImtvIjogMAogICAgfQp9CiAgICB9LCJyZXFfYWNjZXNzLXJlc2VydmUtLTBjODMzIjogewogICAgICAgICJ0eXBlIjogIlJFUVVFU1QiLAogICAgICAgICJuYW1lIjogIkFjY2VzcyBSZXNlcnZlIFJlZGlyZWN0IDEiLAoicGF0aCI6ICJBY2Nlc3MgUmVzZXJ2ZSBSZWRpcmVjdCAxIiwKInBhdGhGb3JtYXR0ZWQiOiAicmVxX2FjY2Vzcy1yZXNlcnZlLS0wYzgzMyIsCiJzdGF0cyI6IHsKICAgICJuYW1lIjogIkFjY2VzcyBSZXNlcnZlIFJlZGlyZWN0IDEiLAogICAgIm51bWJlck9mUmVxdWVzdHMiOiB7CiAgICAgICAgInRvdGFsIjogMTIsCiAgICAgICAgIm9rIjogMTIsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJtaW5SZXNwb25zZVRpbWUiOiB7CiAgICAgICAgInRvdGFsIjogMzUyLAogICAgICAgICJvayI6IDM1MiwKICAgICAgICAia28iOiAwCiAgICB9LAogICAgIm1heFJlc3BvbnNlVGltZSI6IHsKICAgICAgICAidG90YWwiOiA2NDgsCiAgICAgICAgIm9rIjogNjQ4LAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAibWVhblJlc3BvbnNlVGltZSI6IHsKICAgICAgICAidG90YWwiOiA1MjcsCiAgICAgICAgIm9rIjogNTI3LAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAic3RhbmRhcmREZXZpYXRpb24iOiB7CiAgICAgICAgInRvdGFsIjogMTAxLAogICAgICAgICJvayI6IDEwMSwKICAgICAgICAia28iOiAwCiAgICB9LAogICAgInBlcmNlbnRpbGVzMSI6IHsKICAgICAgICAidG90YWwiOiA1NzAsCiAgICAgICAgIm9rIjogNTcwLAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAicGVyY2VudGlsZXMyIjogewogICAgICAgICJ0b3RhbCI6IDU5NiwKICAgICAgICAib2siOiA1OTYsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJwZXJjZW50aWxlczMiOiB7CiAgICAgICAgInRvdGFsIjogNjM2LAogICAgICAgICJvayI6IDYzNiwKICAgICAgICAia28iOiAwCiAgICB9LAogICAgInBlcmNlbnRpbGVzNCI6IHsKICAgICAgICAidG90YWwiOiA2NDYsCiAgICAgICAgIm9rIjogNjQ2LAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAiZ3JvdXAxIjogewogICAgIm5hbWUiOiAidCA8IDgwMCBtcyIsCiAgICAiY291bnQiOiAxMiwKICAgICJwZXJjZW50YWdlIjogMTAwCn0sCiAgICAiZ3JvdXAyIjogewogICAgIm5hbWUiOiAiODAwIG1zIDwgdCA8IDEyMDAgbXMiLAogICAgImNvdW50IjogMCwKICAgICJwZXJjZW50YWdlIjogMAp9LAogICAgImdyb3VwMyI6IHsKICAgICJuYW1lIjogInQgPiAxMjAwIG1zIiwKICAgICJjb3VudCI6IDAsCiAgICAicGVyY2VudGFnZSI6IDAKfSwKICAgICJncm91cDQiOiB7CiAgICAibmFtZSI6ICJmYWlsZWQiLAogICAgImNvdW50IjogMCwKICAgICJwZXJjZW50YWdlIjogMAp9LAogICAgIm1lYW5OdW1iZXJPZlJlcXVlc3RzUGVyU2Vjb25kIjogewogICAgICAgICJ0b3RhbCI6IDEuNSwKICAgICAgICAib2siOiAxLjUsCiAgICAgICAgImtvIjogMAogICAgfQp9CiAgICB9LCJyZXFfYWNjZXNzLXB1cmNoYXNlLWMxNGJmIjogewogICAgICAgICJ0eXBlIjogIlJFUVVFU1QiLAogICAgICAgICJuYW1lIjogIkFjY2VzcyBQdXJjaGFzZSBSZWRpcmVjdCAxIiwKInBhdGgiOiAiQWNjZXNzIFB1cmNoYXNlIFJlZGlyZWN0IDEiLAoicGF0aEZvcm1hdHRlZCI6ICJyZXFfYWNjZXNzLXB1cmNoYXNlLWMxNGJmIiwKInN0YXRzIjogewogICAgIm5hbWUiOiAiQWNjZXNzIFB1cmNoYXNlIFJlZGlyZWN0IDEiLAogICAgIm51bWJlck9mUmVxdWVzdHMiOiB7CiAgICAgICAgInRvdGFsIjogMSwKICAgICAgICAib2siOiAxLAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAibWluUmVzcG9uc2VUaW1lIjogewogICAgICAgICJ0b3RhbCI6IDQ4MCwKICAgICAgICAib2siOiA0ODAsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJtYXhSZXNwb25zZVRpbWUiOiB7CiAgICAgICAgInRvdGFsIjogNDgwLAogICAgICAgICJvayI6IDQ4MCwKICAgICAgICAia28iOiAwCiAgICB9LAogICAgIm1lYW5SZXNwb25zZVRpbWUiOiB7CiAgICAgICAgInRvdGFsIjogNDgwLAogICAgICAgICJvayI6IDQ4MCwKICAgICAgICAia28iOiAwCiAgICB9LAogICAgInN0YW5kYXJkRGV2aWF0aW9uIjogewogICAgICAgICJ0b3RhbCI6IDAsCiAgICAgICAgIm9rIjogMCwKICAgICAgICAia28iOiAwCiAgICB9LAogICAgInBlcmNlbnRpbGVzMSI6IHsKICAgICAgICAidG90YWwiOiA0ODAsCiAgICAgICAgIm9rIjogNDgwLAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAicGVyY2VudGlsZXMyIjogewogICAgICAgICJ0b3RhbCI6IDQ4MCwKICAgICAgICAib2siOiA0ODAsCiAgICAgICAgImtvIjogMAogICAgfSwKICAgICJwZXJjZW50aWxlczMiOiB7CiAgICAgICAgInRvdGFsIjogNDgwLAogICAgICAgICJvayI6IDQ4MCwKICAgICAgICAia28iOiAwCiAgICB9LAogICAgInBlcmNlbnRpbGVzNCI6IHsKICAgICAgICAidG90YWwiOiA0ODAsCiAgICAgICAgIm9rIjogNDgwLAogICAgICAgICJrbyI6IDAKICAgIH0sCiAgICAiZ3JvdXAxIjogewogICAgIm5hbWUiOiAidCA8IDgwMCBtcyIsCiAgICAiY291bnQiOiAxLAogICAgInBlcmNlbnRhZ2UiOiAxMDAKfSwKICAgICJncm91cDIiOiB7CiAgICAibmFtZSI6ICI4MDAgbXMgPCB0IDwgMTIwMCBtcyIsCiAgICAiY291bnQiOiAwLAogICAgInBlcmNlbnRhZ2UiOiAwCn0sCiAgICAiZ3JvdXAzIjogewogICAgIm5hbWUiOiAidCA+IDEyMDAgbXMiLAogICAgImNvdW50IjogMCwKICAgICJwZXJjZW50YWdlIjogMAp9LAogICAgImdyb3VwNCI6IHsKICAgICJuYW1lIjogImZhaWxlZCIsCiAgICAiY291bnQiOiAwLAogICAgInBlcmNlbnRhZ2UiOiAwCn0sCiAgICAibWVhbk51bWJlck9mUmVxdWVzdHNQZXJTZWNvbmQiOiB7CiAgICAgICAgInRvdGFsIjogMC4xMjUsCiAgICAgICAgIm9rIjogMC4xMjUsCiAgICAgICAgImtvIjogMAogICAgfQp9CiAgICB9Cn0KCn0=", "filename": "stats.json"}]}]} |
This is just an example of one possible integration, you can reuse it or come up with one that better suites your needs.
As we saw in the above example where we produced Xray Json reports with the result of the tests, it is now a matter of importing those results to your Jira instance. You can do this by simply submitting automation results to Xray through the REST API, by using one of the available CI/CD plugins (e.g. for Jenkins), or using the Jira interface.
In this case we will show how to import via the API.
APIOnce you have the report file available you can upload it to Xray through a request to the REST API endpoint. To do that, follow the instructions in v1 or v2 (depending on your usage) to obtain the token we will be using in the subsequent requests. AuthenticationThe request made will look like:
The response of this request will return the token to be used in the subsequent requests for authentication purposes. Xray Json resultsOnce you have the token we will use it in the API request with the definition of some common fields on the Test Execution, such as the target project, project version, etc.
With this command we are creating a new Test Execution that will have the results of the Tests that were executed. Once uploaded the Test Execution will look like the example below With Title and Description we have defined in the code and linked to the Tests we have created beforehand to hold Performance results. In order to check the details, you can click on the details icon next to each Test (below the red arrow in the screenshot), this will take you to the Test Execution Details Screen In the details we have the following relevant information:
Bringing the information of performance tests to your project will allow a complete view over the testing process and bring that visibility up front for the team to have all the elements necessary to deliver a quality product. |
.toc-btf { position: fixed; } |