About TestNG
TestNG is a testing framework for Java, mostly focused on unit testing but not only.
Similar to JUnit, TestNG is also used for writing integration and acceptance tests, making use of other libraries such as Selenium.
TestNG XML reports are normally creating in the context of Java applications; however, so other languages and test automation frameworks may also be able to generate this kind of reports.
TestNG is quite feature-complete as may be seen in its online documentation.
Supported versions
Xray supports TestNG 6.14 XML reports.
TestNG Basic Concepts
In TestNG, you have Test methods, Parameterized Test methods, Test classes, Test groups, suites.
- A Test method is a test case;
- Parameterized Tests are a way of specifying input values for a given Test (similar to an example in a Cucumber Scenario Outline). Parameters may come from the XML configuration file used by TestNG or from a data provider (i.e. a method that generates a set of values);
- Tests code is implemented within Test classes;
- Test groups are somehow similar to using Jira labels in Jira issues, in the way that they're used to mark the Test methods as belonging to some "category"; these groups can be used afterwards to more easily select the Test to be run;
- A suite is a configuration file (i.e. a XML file) that is used to enumerate all the Tests to be run, based on the Test classes, Test groups, packages, Test methods.
TestNG uses "attributes" in order to ascribe behavior/characteristics to certain parts of your automated test code; these attributes can be set in the Test method's code.
Learn in practice
Please look at the basic Java example: Testing using TestNG in Java.
Importing TestNG XML reports
Below is a simplified example of a TestNG XML report containing a Test Suite with some Test Cases.
Entities
Test Cases are imported to Xray’s Generic Test issues. The “classname” and “methodname” attributes are concatenated and mapped to the Generic Test Definition field of the Generic Test.
If a Test already exists with the same Generic Test Definition, then it is not created again.
Test Cases are imported to a new (or a user-specified) Test Execution in the context of some project, along with their respective execution results.
Parameterized Tests are imported to the same Test Run, where each set of parameters is identified by a different context.
Mapping of fields from the report to the Test issue
| TestNG XML report | Test in Jira |
|---|---|
"methodname" attribute | Summary field |
"classname" attribute + "." + "methodname" attribute | Generic Test Definition custom field |
| "labels" attribute under the <attributes> tag, within some <test-method>, containing one or more labels delimited by space | labels |
| "test" attribute under the <attributes> tag, within some <test-method>, containing a Jira key of the Test issue | identification of Test issue key in Jira |
| "requirement" attribute under the <attributes> tag, within some <test-method>, containing a Jira key of requirement(s) | link to requirement(s) |
Notes:
- linked requirements may be identified by setting an attribute named "requirement" with the respective issue key at the Test method level, using the ITestResult object"; multiple requirements may be specified by using the space delimeter
- the Test to report results against to may be identified by using and attribute named "test";
- if the "Test" attribute is used explicitly, then the Test must exist or else it will not be imported.
Examples
Consider running some Tests implemented in the following C# class.
package com.xpand.java;
import org.testng.Assert;
import org.testng.annotations.Test;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.DataProvider;
import org.testng.Reporter;
import org.testng.reporters.XMLReporter;
import org.testng.ITestResult;
import com.xpand.annotations.Xray;
public class CalcTest {
@BeforeSuite
public void setUp() throws Exception {
}
@AfterSuite
public void tearDown() throws Exception {
}
@DataProvider
public Object[][] ValidDataProvider() {
return new Object[][]{
{ 1, 2, 3 },
{ 2, 3, 4 }, // error or the data itself :)
{ -1, 1, 0 }
};
}
@Test(dataProvider = "ValidDataProvider")
@Xray(requirement = "CALC-1234", test = "CALC-1")
public void CanAddNumbersFromGivenData(final int a, final int b, final int c)
{
Assert.assertEquals(Calculator.Add(a, b), c);
}
@Test
@Xray(requirement = "CALC-1234", test = "CALC-2")
public void CanAddNumbers()
{
Assert.assertEquals(Calculator.Add(1, 1),2);
Assert.assertEquals(Calculator.Add(-1, 1),0);
ITestResult result = Reporter.getCurrentTestResult();
result.setAttribute("requirement", "CALC-1234"); // Xray will try to create a link to this requirement issue
result.setAttribute("test", "CALC-2"); // Xray will try to find this Test issue and report result against it
result.setAttribute("labels", "core addition"); // Xray will add this(ese) label(s) to the associated Test issue
}
@Test
@Xray(requirement = "CALC-1235", labels = "core")
public void CanSubtract()
{
Assert.assertEquals(Calculator.Subtract(1, 1), 0);
Assert.assertEquals(Calculator.Subtract(-1, -1), 0);
Assert.assertEquals(Calculator.Subtract(100, 5), 95);
}
@Test
@Xray(requirement = "CALC-1236")
public void CanMultiplyX()
{
Assert.assertEquals(Calculator.Multiply(1, 1), 1);
Assert.assertEquals(Calculator.Multiply(-1, -1), 1);
Assert.assertEquals(Calculator.Multiply(100, 5), 500);
}
@Test
@Xray(requirement = "CALC-1237")
public void CanDivide()
{
Assert.assertEquals(Calculator.Divide(1, 1), 1);
Assert.assertEquals(Calculator.Divide(-1, -1), 1);
Assert.assertEquals(Calculator.Divide(100, 5), 20);
}
@Test
@Xray(test = "CALC-6")
public void CanDoStuff()
{
Assert.assertNotEquals(true, true);
}
}
In the first Test, Xray would try to find a Test with the key "CALC-10". If it does not exist, the Test won't be created.
The test would be linked to the Requirement with key "CALC-69", using the "tests" association, if the requirement exists.
If there is a requirement with the key "CALC-1", it would create the link to that requirement. If "CALC-1" corresponds to a Test issue, then "CALC_1" is ignored.
If the Category "CALC_1" does not map to either a requirement or a Test, it would be added to the Test as a label.
In the second Test, a Generic Test with the summary "CanSubtract" would be created.
The Test is only created if no Test with same Generic Test Definition already exists or if "CALC-2" does not correspond to a Test issue.
If there is a requirement with the key "CALC-2", it would create the link to that requirement.
If the Category "CALC_2" does not map to either a requirement or a Test, it would be added to the Test as a label.
package com.xpand.java;
import org.testng.Assert;
import org.testng.annotations.Test;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.DataProvider;
import org.testng.Reporter;
import org.testng.reporters.XMLReporter;
import org.testng.ITestResult;
import com.xpand.annotations.Xray;
public class CalcTest {
@BeforeSuite
public void setUp() throws Exception {
}
@AfterSuite
public void tearDown() throws Exception {
}
@DataProvider
public Object[][] ValidDataProvider() {
return new Object[][]{
{ 1, 2, 3 },
{ 2, 3, 4 }, // error or the data itself :)
{ -1, 1, 0 }
};
}
@Test(dataProvider = "ValidDataProvider")
@Xray(requirement = "CALC-1234", test = "CALC-1")
public void CanAddNumbersFromGivenData(final int a, final int b, final int c)
{
Assert.assertEquals(Calculator.Add(a, b), c);
}
@Test
@Xray(requirement = "CALC-1234", test = "CALC-2")
public void CanAddNumbers()
{
Assert.assertEquals(Calculator.Add(1, 1),2);
Assert.assertEquals(Calculator.Add(-1, 1),0);
ITestResult result = Reporter.getCurrentTestResult();
}
@Test
@Xray(requirement = "CALC-1235", labels = "core")
public void CanSubtract()
{
Assert.assertEquals(Calculator.Subtract(1, 1), 0);
Assert.assertEquals(Calculator.Subtract(-1, -1), 0);
Assert.assertEquals(Calculator.Subtract(100, 5), 95);
}
@Test
@Xray(requirement = "CALC-1236")
public void CanMultiplyX()
{
Assert.assertEquals(Calculator.Multiply(1, 1), 1);
Assert.assertEquals(Calculator.Multiply(-1, -1), 1);
Assert.assertEquals(Calculator.Multiply(100, 5), 500);
}
@Test
@Xray(requirement = "CALC-1237")
public void CanDivide()
{
Assert.assertEquals(Calculator.Divide(1, 1), 1);
Assert.assertEquals(Calculator.Divide(-1, -1), 1);
Assert.assertEquals(Calculator.Divide(100, 5), 20);
}
@Test
@Xray(test = "CALC-6")
public void CanDoStuff()
{
Assert.assertNotEquals(true, true);
}
}
Status
The status of the Test Run will be set based on the Test case result:
Test Cases (NUnit3.0) | Test status | |
|---|---|---|
Failed | Error | FAIL |
Passed | Success | PASS |
Other Cases | Other Cases | TODO |
Note: Test Cases with the status FAIL may have a failure message displayed in the Test Run screen, under the Results section.
If the same Test Case has been executed in multiple Test Suites, then the result for each Test Suite will be shown. A parameterized Test will also appear with its own specific context.
<REPLACE IMAGE BELOW>
Whenever a Test Case is executed in multiple contexts, the overall status of the Test Run will be calculated as a joint value.
Condition | Overall status of the Test Run |
|---|---|
If all the mapped results of the Test Case was PASS | PASS |
If any of the mapped results of the Test Case was FAIL | FAIL |
Other cases | TODO |