Page History
| Table of Contents |
|---|
About TestNG
TestNG is a testing framework for Java, mostly focused on unit testing but not only.
...
| Info | ||
|---|---|---|
| ||
Xray supports TestNG 6.14 XML reports. |
TestNG Basic Concepts
In TestNG, you have Test methods, Parameterized Test methods, Test classes, Test groups, suites.
...
| Info | ||
|---|---|---|
| ||
Please look at the basic Java example: Testing using TestNG in Java. |
Importing TestNG XML reports
Below is a simplified example of a TestNG XML report containing a Test Suite with some Test Cases.
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
<?xml version="1.0" encoding="UTF-8"?>
<testng-results skipped="0" failed="2" ignored="0" total="8" passed="6">
<reporter-output>
</reporter-output>
<suite name="TestAll" duration-ms="25" started-at="2018-03-06T17:22:48Z" finished-at="2018-03-06T17:22:48Z">
<groups>
</groups>
<test name="calculator" duration-ms="25" started-at="2018-03-06T17:22:48Z" finished-at="2018-03-06T17:22:48Z">
<class name="com.xpand.java.CalcTest">
<test-method status="PASS" signature="setUp()[pri:0, instance:com.xpand.java.CalcTest@36d4b5c]" name="setUp" is-config="true" duration-ms="5" started-at="2018-03-06T17:22:48Z" finished-at="2018-03-06T17:22:48Z">
<reporter-output>
</reporter-output>
</test-method> <!-- setUp -->
<test-method status="PASS" signature="CanAddNumbers()[pri:0, instance:com.xpand.java.CalcTest@36d4b5c]" name="CanAddNumbers" duration-ms="2" started-at="2018-03-06T17:22:48Z" finished-at="2018-03-06T17:22:48Z">
<reporter-output>
</reporter-output>
<attributes>
<attribute name="test">
<![CDATA[CALC-2]]>
</attribute> <!-- test -->
<attribute name="requirement">
<![CDATA[CALC-1234]]>
</attribute> <!-- requirement -->
<attribute name="labels">
<![CDATA[core addition]]>
</attribute> <!-- labels -->
</attributes>
</test-method> <!-- CanAddNumbers -->
<test-method status="PASS" signature="CanAddNumbersFromGivenData(int, int, int)[pri:0, instance:com.xpand.java.CalcTest@36d4b5c]" name="CanAddNumbersFromGivenData" duration-ms="0" started-at="2018-03-06T17:22:48Z" data-provider="ValidDataProvider" finished-at="2018-03-06T17:22:48Z">
<params>
<param index="0">
<value>
<![CDATA[1]]>
</value>
</param>
<param index="1">
<value>
<![CDATA[2]]>
</value>
</param>
<param index="2">
<value>
<![CDATA[3]]>
</value>
</param>
</params>
<reporter-output>
</reporter-output>
<attributes>
<attribute name="test">
<![CDATA[CALC-1]]>
</attribute> <!-- test -->
<attribute name="requirement">
<![CDATA[CALC-1234]]>
</attribute> <!-- requirement -->
<attribute name="labels">
<![CDATA[]]>
</attribute> <!-- labels -->
</attributes>
</test-method> <!-- CanAddNumbersFromGivenData -->
<test-method status="FAIL" signature="CanAddNumbersFromGivenData(int, int, int)[pri:0, instance:com.xpand.java.CalcTest@36d4b5c]" name="CanAddNumbersFromGivenData" duration-ms="1" started-at="2018-03-06T17:22:48Z" data-provider="ValidDataProvider" finished-at="2018-03-06T17:22:48Z">
<params>
<param index="0">
<value>
<![CDATA[2]]>
</value>
</param>
<param index="1">
<value>
<![CDATA[3]]>
</value>
</param>
<param index="2">
<value>
<![CDATA[4]]>
</value>
</param>
</params>
<exception class="java.lang.AssertionError">
<message>
<![CDATA[expected [4] but found [5]]]>
</message>
<full-stacktrace>
<![CDATA[java.lang.AssertionError: expected [4] but found [5]
at org.testng.Assert.fail(Assert.java:93)
at org.testng.Assert.failNotEquals(Assert.java:512)
at org.testng.Assert.assertEqualsImpl(Assert.java:134)
at org.testng.Assert.assertEquals(Assert.java:115)
at org.testng.Assert.assertEquals(Assert.java:388)
at org.testng.Assert.assertEquals(Assert.java:398)
at com.xpand.java.CalcTest.CanAddNumbersFromGivenData(CalcTest.java:40)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:661)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:869)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1193)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:126)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:744)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
at org.testng.SuiteRunner.run(SuiteRunner.java:289)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1226)
at org.testng.TestNG.runSuites(TestNG.java:1144)
at org.testng.TestNG.run(TestNG.java:1115)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:283)
at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:120)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
]]>
</full-stacktrace>
</exception> <!-- java.lang.AssertionError -->
<reporter-output>
</reporter-output>
<attributes>
<attribute name="test">
<![CDATA[CALC-1]]>
</attribute> <!-- test -->
<attribute name="requirement">
<![CDATA[CALC-1234]]>
</attribute> <!-- requirement -->
<attribute name="labels">
<![CDATA[]]>
</attribute> <!-- labels -->
</attributes>
</test-method> <!-- CanAddNumbersFromGivenData -->
<test-method status="PASS" signature="CanAddNumbersFromGivenData(int, int, int)[pri:0, instance:com.xpand.java.CalcTest@36d4b5c]" name="CanAddNumbersFromGivenData" duration-ms="0" started-at="2018-03-06T17:22:48Z" data-provider="ValidDataProvider" finished-at="2018-03-06T17:22:48Z">
<params>
<param index="0">
<value>
<![CDATA[-1]]>
</value>
</param>
<param index="1">
<value>
<![CDATA[1]]>
</value>
</param>
<param index="2">
<value>
<![CDATA[0]]>
</value>
</param>
</params>
<reporter-output>
</reporter-output>
<attributes>
<attribute name="test">
<![CDATA[CALC-1]]>
</attribute> <!-- test -->
<attribute name="requirement">
<![CDATA[CALC-1234]]>
</attribute> <!-- requirement -->
<attribute name="labels">
<![CDATA[]]>
</attribute> <!-- labels -->
</attributes>
</test-method> <!-- CanAddNumbersFromGivenData -->
<test-method status="PASS" signature="CanDivide()[pri:0, instance:com.xpand.java.CalcTest@36d4b5c]" name="CanDivide" duration-ms="1" started-at="2018-03-06T17:22:48Z" finished-at="2018-03-06T17:22:48Z">
<reporter-output>
</reporter-output>
<attributes>
<attribute name="test">
<![CDATA[]]>
</attribute> <!-- test -->
<attribute name="requirement">
<![CDATA[CALC-1237]]>
</attribute> <!-- requirement -->
<attribute name="labels">
<![CDATA[]]>
</attribute> <!-- labels -->
</attributes>
</test-method> <!-- CanDivide -->
<test-method status="PASS" signature="CanMultiplyX()[pri:0, instance:com.xpand.java.CalcTest@36d4b5c]" name="CanMultiplyX" duration-ms="0" started-at="2018-03-06T17:22:48Z" finished-at="2018-03-06T17:22:48Z">
<reporter-output>
</reporter-output>
<attributes>
<attribute name="test">
<![CDATA[]]>
</attribute> <!-- test -->
<attribute name="requirement">
<![CDATA[CALC-1236]]>
</attribute> <!-- requirement -->
<attribute name="labels">
<![CDATA[]]>
</attribute> <!-- labels -->
</attributes>
</test-method> <!-- CanMultiplyX -->
<test-method status="FAIL" signature="CanDoStuff()[pri:0, instance:com.xpand.java.CalcTest@36d4b5c]" name="CanDoStuff" duration-ms="0" started-at="2018-03-06T17:22:48Z" finished-at="2018-03-06T17:22:48Z">
<exception class="java.lang.AssertionError">
<message>
<![CDATA[null]]>
</message>
<full-stacktrace>
<![CDATA[java.lang.AssertionError: null
at org.testng.Assert.fail(Assert.java:93)
at org.testng.Assert.assertNotEquals(Assert.java:897)
at org.testng.Assert.assertNotEquals(Assert.java:902)
at com.xpand.java.CalcTest.CanDoStuff(CalcTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:661)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:869)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1193)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:126)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:744)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
at org.testng.SuiteRunner.run(SuiteRunner.java:289)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1226)
at org.testng.TestNG.runSuites(TestNG.java:1144)
at org.testng.TestNG.run(TestNG.java:1115)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:283)
at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:120)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
]]>
</full-stacktrace>
</exception> <!-- java.lang.AssertionError -->
<reporter-output>
</reporter-output>
</test-method> <!-- CanDoStuff -->
<test-method status="PASS" signature="CanSubtract()[pri:0, instance:com.xpand.java.CalcTest@36d4b5c]" name="CanSubtract" duration-ms="0" started-at="2018-03-06T17:22:48Z" finished-at="2018-03-06T17:22:48Z">
<reporter-output>
</reporter-output>
<attributes>
<attribute name="test">
<![CDATA[]]>
</attribute> <!-- test -->
<attribute name="requirement">
<![CDATA[CALC-1235]]>
</attribute> <!-- requirement -->
<attribute name="labels">
<![CDATA[core]]>
</attribute> <!-- labels -->
</attributes>
</test-method> <!-- CanSubtract -->
<test-method status="PASS" signature="tearDown()[pri:0, instance:com.xpand.java.CalcTest@36d4b5c]" name="tearDown" is-config="true" duration-ms="0" started-at="2018-03-06T17:22:48Z" finished-at="2018-03-06T17:22:48Z">
<reporter-output>
</reporter-output>
</test-method> <!-- tearDown -->
</class> <!-- com.xpand.java.CalcTest -->
</test> <!-- calculator -->
</suite> <!-- TestAll -->
</testng-results> |
Entities
Test Cases are imported to Xray’s Generic Test issues. The “classname” and “methodname” attributes are concatenated and mapped to the Generic Test Definition field of the Generic Test.
...
Parameterized Tests are imported to the same Test Run, where each set of parameters is identified by a different context.
Mapping of fields from the report to the Test issue
| TestNG XML report | Test in Jira |
|---|---|
"methodname" attribute | Summary field |
"classname" attribute + "." + "methodname" attribute | Generic Test Definition custom field |
| groups | labels |
| "labels" attribute under the <attributes> tag, within some <test-method>, containing one or more labels delimited by space | labels |
| "test" attribute under the <attributes> tag, within some <test-method>, containing a Jira key of the Test issue | identification of Test issue key in Jira |
| "requirement" attribute under the <attributes> tag, within some <test-method>, containing a Jira key of requirement(s) | link to requirement(s) |
...
- linked requirements may be identified by setting an attribute named "requirement" with the respective issue key at the Test method level, using the ITestResult object"; multiple requirements may be specified by using the space delimeter
- the Test to report results against to may be identified by using and attribute named "test";
- if the "Test" attribute is used explicitly, then the Test must exist or else it will not be imported.
Examples
Consider running some Tests implemented in the following Java class.
...
| Info | ||
|---|---|---|
| ||
| It's possible to use Java annotations to more easily mark the Test methods with the linked requirement(s), Test or with the labels you wish to add. Please have a look at the examples shown in Testing using TestNG in Java . |
Status
The status of the Test Run will be set based on the Test case result:
...
Condition | Overall status of the Test Run |
|---|---|
If all the mapped results of the Test Case was PASS | PASS |
If any of the mapped results of the Test Case was FAIL | FAIL |
Other cases | TODO |