(For more resources related to this topic, see here.)

Logging and reporting

Reporting is the most important part of any test execution, reason being it helps the user to understand the result of the test execution, point of failure, and reasons for the failure. Logging, on the other hand, is important to keep an eye on the execution flow or for debugging in case of any failures.

TestNG by default generates a different type of report for its test execution. This includes an HTML and an XML report output. TestNG also allows its users to write their own reporter and use it with TestNG. There is also an option to write your own loggers, which are notified at runtime by TestNG.

There are two main ways to generate a report with TestNG:

Listeners : For implementing a listener class, the class has to implement the org.testng.ITestListener interface. These classes are notified at runtime by TestNG when the test starts, finishes, fails, skips, or passes.

Reporters : For implementing a reporting class, the class has to implement an org.testng.IReporter interface. These classes are called when the whole suite run ends. The object containing the information of the whole test run is passed to this class when called.

Each of them should be used depending upon the condition of how and when the reports have to be generated. For example, if you want to generate a custom HTML report at the end of execution then you should implement IReporter interface while writing extension. But in case you want to update a file at runtime and have to print a message as and when the tests are getting executed, then we should use the ITestListener interface.

Writing your own logger

We had earlier read about the different options that TestNG provides for logging and reporting. Now let's learn how to start using them. To start with, we will write a sample program in which we will use the ITestListener interface for logging purposes.

Time for action – writing a custom logger

Open Eclipse and create a Java project with the name CustomListener and with the following structure. Please make sure that the TestNG library is added to the build path of the project.

Create a new class named SampleTest under the test.sample package and replace the following code in it:
package test.sample; import org.testng.Assert; import org.testng.annotations.Test; public class SampleTest { @Test public void testMethodOne(){ Assert.assertTrue(true); } @Test public void testMethodTwo(){ Assert.assertTrue(false); } @Test(dependsOnMethods={"testMethodTwo"}) public void testMethodThree(){ Assert.assertTrue(true); } }
The preceding test class contains three test methods out of which testMethodOne and testMethodThree will pass when executed, whereas testMethodTwo is made to fail by passing a false Boolean value to the Assert.assertTrue method which is used for truth conditions in the tests.

In the preceding class test method testMethodThree depends on testMethodTwo.

Create another new class named CustomLogging under the test.logger package and replace its code with the following code:
package test.logger; import java.text.DateFormat; import java.text.SimpleDateFormat; import java.util.Date; import org.testng.ITestContext; import org.testng.ITestListener; import org.testng.ITestResult; public class CustomLogging implements ITestListener{ //Called when the test-method execution starts @Override public void onTestStart(ITestResult result) { System.out.println("Test method started: "+ result.getName()+ "
and time is: "+getCurrentTime()); } //Called when the test-method execution is a success @Override public void onTestSuccess(ITestResult result) { System.out.println("Test method success: "+ result.getName()+ "
and time is: "+getCurrentTime()); } //Called when the test-method execution fails @Override public void onTestFailure(ITestResult result) { System.out.println("Test method failed: "+ result.getName()+ "
and time is: "+getCurrentTime()); } //Called when the test-method is skipped @Override public void onTestSkipped(ITestResult result) { System.out.println("Test method skipped: "+ result.getName()+ "
and time is: "+getCurrentTime()); } //Called when the test-method fails within success percentage @Override public void onTestFailedButWithinSuccessPercentage(ITestResult result) { // Leaving blank } //Called when the test in xml suite starts @Override public void onStart(ITestContext context) { System.out.println("Test in a suite started: "+ context.getName()+ "
and time is: "+getCurrentTime()); } //Called when the test in xml suite finishes @Override public void onFinish(ITestContext context) { System.out.println("Test in a suite finished: "+ context.getName()+ "
and time is: "+getCurrentTime()); } //Returns the current time when the method is called public String getCurrentTime(){ DateFormat dateFormat = new SimpleDateFormat("HH:mm:ss:SSS"); Date dt = new Date(); return dateFormat.format(dt); } }
The above test class extends the ITestListener interface and defines the overriding methods of the interface. Details of each of the methods are provided as comments inside the previous code. Each method when executed prints the respective test method name or the suite name and the time when it was called. The getCurrentTime method returns the current time in HH:mm:ss:SSS format using the Date and DateFormat class.

Create a new testing XML file under the project with name simple-logger-testng.xml and copy the following contents onto it:
<suite name="Simple Logger Suite"> <listeners> <listener class-name="test.logger.CustomLogging" /> </listeners> <test name="Simple Logger test"> <classes> <class name="test.sample.SampleTest" /> </classes> </test> </suite>
The preceding XML defines a simple test which considers the class test.sample.SampleTest for test execution. The CustomLogging class which implements the ITestListener is added as a listener to the test suite using the listeners tag as shown in the preceding XML.

Select the previous testing XML file in Eclipse and run it as TestNG suite. You will see the following test result in the Console window of Eclipse:

The following screenshot shows the test methods that were executed, failed, and skipped in the test run:

What just happened?

We created a custom logger class which implements the ITestListener interface and attached itself to the TestNG test suite as a listener. Methods of this listener class are invoked by TestNG as and when certain conditions are met in the execution, for example, test started, test failure, test success, and so on. Multiple listeners can be implemented and added to the test suite execution, TestNG will invoke all the listeners that are attached to the test suite.

Logging listeners are mainly used when we need to see the continuous status of the test execution when the tests are getting executed.

Writing your own reporter

In the earlier section we had seen an example of writing your custom logger and attaching it to TestNG. In this section we will cover, with an example, the method of writing your custom reporter and attaching it to TestNG. To write a custom reporter class, our extension class should implement the IReporter interface. Let's go ahead and create an example with the custom reporter.

Time for action – writing a custom reporter

Open the previously created project named CustomListener and create a package named reporter under the test package.

Create a new class named CustomReporter under the test.reporter package and add the following code to it:
package test.reporter; import java.util.List; import java.util.Map; import org.testng.IReporter; import org.testng.ISuite; import org.testng.ISuiteResult; import org.testng.ITestContext; import org.testng.xml.XmlSuite; public class CustomReporter implements IReporter { @Override public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) { //Iterating over each suite included in the test for (ISuite suite : suites) { //Following code gets the suite name String suiteName = suite.getName(); //Getting the results for the said suite Map<String, ISuiteResult> suiteResults = suite.getResults(); for (ISuiteResult sr : suiteResults.values()) { ITestContext tc = sr.getTestContext(); System.out.println("Passed tests for suite '" + suiteName +
"' is:" + tc.getPassedTests().getAllResults().size()); System.out.println("Failed tests for suite '" + suiteName +
"' is:" +
tc.getFailedTests().getAllResults().size()); System.out.println("Skipped tests for suite '" + suiteName +
"' is:" +
tc.getSkippedTests().getAllResults().size()); } } } }
The preceding class implements the org.testng.IReporter interface. It implements the definition for the method generateReport of the IReporter interface. The method takes three arguments , the first being xmlSuite, which is the list suites mentioned in the testng XML being executed. The second one being suites which contains the suite information after the test execution; this object contains all the information about the packages, classes, test methods, and their test execution results. The third being the outputDirectory, which contains the information of the output folder path where the reports will be generated.

The custom report prints the total number of tests passed, failed, and skipped for each suite included in the particular test execution when added to TestNG as a listener.

Create a new file named simple-reporter-testng.xml to the project and add the following code to it:
<suite name="Simple Reporter Suite"> <listeners> <listener class-name="test.reporter.CustomReporter" /> </listeners> <test name="Simple Reporter test"> <classes> <class name="test.sample.SampleTest" /> </classes> </test> </suite>
The preceding XML is a testng XML configuration file. It contains a single test with the class test.sample.SampleTest to be considered for test execution. The CustomReporter class is added as a listener to the test suite using the listeners and listener tag as defined in the previous file.

Select the preceding XML file and run it as TestNG test suite in Eclipse. You will see the following test results under the Console window of Eclipse:

What just happened?

We successfully created an example of writing custom reporter and attaching it to TestNG as a listener. The preceding example shows a simple custom reporter which prints the number of failed, passed, and skipped tests on the console for each suite included the said test execution. Reporter is mainly used to generate the final report for the test execution. The extension can be used to generate XML, HTML, XLS, CSV, or text format files depending upon the report requirement.

TestNG HTML and XML report

TestNG comes with certain predefined listeners as part of the library. These listeners are by default added to any test execution and generate different HTML and XML reports for any test execution. The report is generated by default under the folder named testoutput and can be changed to any other folder by configuring it. These reports consist of certain HTML and XML reports that are TestNG specific.

Let's create a sample project to see how the TestNG report is generated.

Time for action – generating TestNG HTML and XML reports

Open Eclipse and create a Java project with the name SampleReport having the following structure. Please make sure that the TestNG library is added to the build path of the project.

Create a new class named SampleTest under the test package and replace the following code in it:
package test; import org.testng.Assert; import org.testng.annotations.Test; public class SampleTest { @Test public void testMethodOne(){ Assert.assertTrue(true); } @Test public void testMethodTwo(){ Assert.assertTrue(false); } @Test(dependsOnMethods={"testMethodTwo"}) public void testMethodThree(){ Assert.assertTrue(true); } }
The preceding test class contains three test methods out of which testMethodOne and testMethodThree will pass when executed, whereas testMethodTwo is made to fail by passing a false Boolean value to Assert.assertTrue method.

In the preceding class test method testMethodThree depends on testMethodTwo.

Select the previously created test class and run it as a TestNG test through Eclipse.

Now refresh the Java project in Eclipse by selecting the project and pressing the F5 button or right-clicking and selecting Refresh , and this will refresh the project. You will see a new folder named test-output under the project.

Expand the folder in Eclipse and you will see the following files as shown in the screenshot:

Open index.html as shown in the preceding screenshot on your default web browser. You will see the following HTML report:

Now open the file testing-results.xml in the default XML editor on your system, and you will see the following results in the XML file:

What just happened?

We successfully created a test project and generated a TestNG HTML and XML report for the test project. TestNG by default generates multiple reports as part of its test execution. These reports mainly include TestNG HTML report, TestNG emailable report, TestNG report XML, and JUnit report XML files. These files can be found under the output report folder (in this case test-output). These default report generation can be disabled while running the tests by setting the value of the property useDefaultListeners to false. This property can be set while using the build tools like Ant or Maven as explained in the previous chapter.

Generating a JUnit HTML report

JUnit is one of those unit frameworks which were initially used by many Java applications as a Unit test framework. By default, JUnit tests generate a simple report XML files for its test execution. These XML files can then be used to generate any custom reports as per the testing requirement. We can also generate HTML reports using the XML files. Ant has such a utility task which takes these JUnit XML files as input and generates an HTML report from it. We had earlier learnt that TestNG by default generates the JUnit XML reports for any test execution. We can use these XML report files as input for generation of a JUnit HTML report. Assuming we already have JUnit XML reports available from the earlier execution let's create a simple Ant build configuration XML file to generate an HTML report for the test execution.

Time for action – generating a JUnit report

Go to the previously created Java project in Eclipse.

Create a new file named junit-report-build.xml by selecting the project.

Add the following code to the newly created file and save it:
<project name="Sample Report" default="junit-report" basedir=".">  <property name="junit-xml-dir" value="${basedir}/test-output/junitreports"/> <property name="report-dir" value="${basedir}/html-report" />  <target name="junit-report">  <delete dir="${report-dir}" failonerror="false"/> <mkdir dir="${report-dir}" /> <mkdir dir="${report-dir}/Junit" />  <junitreport todir="${report-dir}/Junit"> <fileset dir="${junit-xml-dir}"> <include name="**/*.xml" /> </fileset> <report format="noframes" todir="${report-dir}/Junit" /> </junitreport> </target> </project>
The preceding XML defines a simple Ant build.xml file having a specific Ant target named junit-report that generates a JUnit report when executed. The target looks for the JUnit report XML files under the directory test-output/junitreports. For the Ant configuration file the default target to execute is configured as junit-report.

Open the terminal window and go to the Java project directory in the terminal.

Run the command ant –buildfile junit-report-build.xml and press Enter .

Once executed a JUnit HTML report will be generated in the configured directory /html-report/Junit.

Open the file named junit-noframes.html on your default web browser. You will see the following HTML report:

What just happened?

In this section we have seen how to use the JUnit XML report generated by TestNG and generate HTML report using Ant. There are two kinds of reports that can be generated using this method: frames and no-frames. If the report generation is configured with frames there will multiple files generated for each class and the main report will connect to them through links. A no-frames report consists of a single file with all the results of the test execution. This can be configured by providing the respective value to the format attribute of the report task in Ant.

Generating a ReportNG report

We had earlier seen that TestNG provides options to add custom listeners for logging and reporting. These listeners can easily be added to the TestNG execution and will be called during the execution or at the end of the execution depending upon the type of listener. ReportNG is a reporter add-on for TestNG that implements the report listener of TestNG. ReportNG reports are better looking reports compared to the original HTML reports. To generate a ReportNG report we have to add the reporting class to the list of listeners of TestNG while executing the tests. Let's see how to add ReportNG listener to TestNG and generate a ReportNG HTML report. In the following example we will use an Ant build XML file used to run our tests.

Time for action – generating a ReportNG report

Go to the earlier created Java project SampleProject in Eclipse.

Download ReportNG from http://reportng.uncommons.org/ download site.

Unzip and copy the reportng-<version>.jar and velocity-dep-<version>.jar from the unzipped folder to the lib folder under the project.

Download guice from guice site https://code.google.com/p/google-guice/downloads/list.

Unzip the downloaded guice zip file and copy the guice-<version>.jar to the lib folder under the project.

Create a new file named testng.xml under the folder and add the following content to it:
<suite name="Sample Suite"> <test name="Sample test"> <classes> <class name="test.SampleTest" /> </classes> </test> </suite>

Create a new file named reportng-build.xml by selecting the project.

Add the following code to the newly created file and save it:
<project name="Testng Ant build" basedir=".">  <property name="report-dir" value="${basedir}/html-report" /> <property name="testng-report-dir" value="${report-dir}/TestNG-report" /> <property name="lib-dir" value="${basedir}/lib" /> <property name="bin-dir" value="${basedir}/bin-dir" /> <property name="src-dir" value="${basedir}/src" />  <path id="test.classpath"> <pathelement location="${bin-dir}" /> <fileset dir="${lib-dir}"> <include name="*.jar" /> </fileset> </path>  <target name="init"> <delete dir="${bin-dir}" /> <mkdir dir="${bin-dir}" /> <delete dir="${report-dir}" /> <mkdir dir="${report-dir}" /> </target>  <target name="compile" depends="init"> <javac srcdir="${src-dir}" classpathref="test.classpath" includeAntRuntime="No" destdir="${bin-dir}" /> </target>  <taskdef name="testng" classname="org.testng.TestNGAntTask" classpathref="test.classpath" />  <target name="testng-execution" depends="compile"> <mkdir dir="${testng-report-dir}" /> <testng outputdir="${testng-report-dir}"
classpathref="test.classpath" useDefaultListeners="false"
listeners="org.uncommons.reportng.HTMLReporter">  <xmlfileset dir="${basedir}" includes="testng.xml" /> <sysproperty key="org.uncommons.reportng.title" value="ReportNG Report" /> </testng> </target> </project>
The preceding XML defines a simple Ant build XML file that generates a ReportNG report when executed. The said XML compiles and runs the TestNG tests. ReportNG is added as a listener and the default listener of TestNG is disabled by setting a false value to the useDefaultListeners attribute while using the testng Ant task.

Open the terminal window and go to the Java project directory in the terminal.

Run the command ant -buildfile reportng-build.xml testing execution and then press Enter .

Once executed a ReportNG HTML report will be generated in the configured directory html-reportTestNG-reporthtml under the said project directory.

Go to the said directory and open the index.html file on your default web browser. You will see the following HTML report:

By clicking on the Sample test link, you will see details of the test report as shown in the following screenshot:

What just happened?

In the previous section we learned how to generate a ReportNG HTML report for our test execution. We disabled the default TestNG reports in the previous Ant XML file but, if required, we can generate both the default as well as ReportNG reports by enabling the default report listeners. In the previous example the title of the report is configured by setting the property org.uncommons.reportng.title. There are other configuration options that we can use while generating the report, and we will cover these in the next section.

ReportNG configuration options

ReportNG provides different configuration options based on which the respective HTML report is generated. Following is a list of configurations that are supported:

org.uncommons.reportng.coverage-report: This is configured as the link to the test coverage report.

org.uncommons.reportng.escape-output: This property is used to turn off the log output in reports. By default it's turned off and is not recommended to be switched on as enabling this may require certain hacks to be implemented for proper report generation.

org.uncommons.reportng.frames: This property is used to generate an HTML report with frameset and without frameset. The default value is set to true and hence it generates HTML reports with frameset by default.

org.uncommons.reportng.locale: Used to override the localized messages in the generated HTML report.

org.uncommons.reportng.stylesheet: This property can be used to customize the CSS property of the generated HTML report.

org.uncommons.reportng.title: Used to define a report title for the generated HTML report.

Have a go hero

Write an Ant file to configure ReportNG to generate an HTML report without any frames for the TestNG execution.

Generating a Reporty-ng (former TestNG-xslt) report

While looking at the test report, senior managers might like to see the report in a graphical representation to know the status of the execution with just a glance. Reporty-ng (formerly called TestNG-xslt) is one such add-on report that generates a pie chart for your test execution with all the passed, failed, and skipped tests. This plugin uses the XSL file to convert the TestNG XML report into the custom HTML report with a pie chart. To use this plugin we will write an Ant target which will use the TestNG results XML file to generate the report.

Let's go ahead and write an Ant target to generate the report.

Time for action – generating a Reporty-ng report

Open the previously created SampleReport in Eclipse.

Download the Reporty-ng from the URL: https://github.com/cosminaru/reporty-ng
At the time of writing this book the latest version available was Reporty-ng 1.2. You can download a newer version if available. Changes in the installation process should be minor if there are any at all.

Unzip the downloaded zip and copy a file named testng-results.xsl from srcmainresources onto the resources folder under the said project.

Copy the JARs saxon-8.7.jar and SaxonLiason.jar from the unzipped Reporty-ng lib folder to the project lib folder.

Create a new Ant XML configuration file named reporty-ng-report.xml and paste the following code onto it:
<project name="Reporty-ng Report" default="reporty-ng-report" basedir=".">  <property name="xslt-report-dir" value="${basedir}/reporty-ng/" /> <property name="report-dir" value="${basedir}/html-report" /> <property name="lib-dir" value="${basedir}/lib" /> <path id="test.classpath"> <fileset dir="${lib-dir}"> <include name="**/*.jar" /> </fileset> </path> <target name="reporty-ng-report"> <delete dir="${xslt-report-dir}" /> <mkdir dir="${xslt-report-dir}" /> <xslt in="${basedir}/test-output/testng-results.xml"
style="${basedir}/resources/testng-results.xsl"
out="${xslt-report-dir}/index.html"> <param name="testNgXslt.outputDir" expression="${xslt-report-dir}" /> <param name="testNgXslt.sortTestCaseLinks" expression="true" /> <param name="testNgXslt.testDetailsFilter"
expression="FAIL,SKIP,PASS,CONF,BY_CLASS" /> <param name="testNgXslt.showRuntimeTotals" expression="true" /> <classpath refid="test.classpath" /> </xslt> </target> </project>
The preceding XML defines Ant build XML configuration, it contains an Ant target which generates the Reporty-ng report. The input path for testng results XML is configured through the in attribute of the xslt task in Ant. The transformation from XML to HTML is done using the testng-results.xsl of Reporty-ng, the location of which is configured by using the style attribute of the xslt task. The output HTML name is configured using the out attribute.

Different configuration parameters for Reporty-ng are configured using the param task of Ant as shown in the preceding code.

Now go to the said project folder through the command terminal and type the command ant –buildfile reporty-ng-build.xml and press Enter .

You will see the following console output on the terminal:

Now go to the configured report output folder reporty-ng (in this case) and open the file index.html in your default browser. You will see the following test report:

On clicking the Default suite link on the left-hand side, a detailed report of the executed test cases will be displayed as shown in the following screenshot:

logging-and-reports-img-15

What just happened?

In the previous example we learned how to generate a Reporty-ng report using Ant. The report is very good from a report point of view as it gives a clear picture of the test execution through the pie chart. The output report can be configured using different configurations, which we will cover in the next section.

Configuration options for Reporty-ng report

As said earlier there are different configuration options that the Reporty-ng report supports while generating the report. Following is the list of supported configuration options and how they affect the report generation:

testNgXslt.outputDir: Sets the target output directory for the HTML content. This is mandatory and must be an absolute path. If you are using the Maven plugin, this is set automatically so you don't have to provide it.

testNgXslt.cssFile: Specifies an alternative style sheet file overriding the default settings. This parameter is not required.

testNgXslt.showRuntimeTotals: Boolean flag indicating if the report should display the aggregated information about the method durations. The information is displayed for each test case and aggregated for the whole suite. Non-mandatory parameter, defaults to false.

testNgXslt.reportTitle: Use this setting to specify a title for your HTML reports. This is not a mandatory parameter and defaults to TestNG Results.

testNgXslt.sortTestCaseLinks: Indicates whether the test case links (buttons) in the left frame should be sorted alphabetically. By default they are rendered in the order they are generated by TestNG so you should set this to true to change this behavior.

testNgXslt.chartScaleFactor: A scale factor for the SVG pie chart in case you want it larger or smaller. Defaults to 1.

testNgXslt.testDetailsFilter: Specifies the default settings for the checkbox filters at the top of the test details page. Can be any combination (comma-separated) of: FAIL,PASS,SKIP,CONF,BY_CLASS.

Have a go hero

Write an Ant target in Ant to generate a Reporty-ng report with only Fail and Pass filter options.

Pop quiz – logging and reports

Q1. Which interface should the custom class implement for tracking the execution status as and when the test is executed?

org.testng.ITestListener

org.testng.IReporter

Q2. Can we disable the default reports generated by the TestNG?

Summary

In this chapter we have covered different sections related to logging and reporting with TestNG. We have learned about different logging and reporting options provided by TestNG, writing our custom loggers and reporters and methods to generate different reports for our TestNG execution. Each of the reports have certain characteristics and any or all of the reports can be generated and used for test execution depending upon the requirement.

Till now we have been using the XML configuration methodology of defining and writing our TestNG test suites. In the next chapter we will learn how to define/configure the TestNG test suite through code. This is helpful for defining the test suite at runtime.