Listeners in Selenium

What Are Listeners in Selenium?

**Listeners** in Selenium are used to monitor the execution of tests. They are interfaces that can be implemented to listen to different events that occur during test execution, such as when a test starts, passes, fails, or skips. Listeners allow for customized actions to be triggered based on these events.

Listeners are especially useful for:

Logging test events
Taking screenshots on test failures
Retrying failed tests
Generating custom reports

Types of Listeners in Selenium

Selenium supports several listeners, but the most commonly used ones are:

ITestListener: Used to listen to events related to individual test methods.
ISuiteListener: Used to listen to events related to the entire test suite.
IRetryAnalyzer: Used to retry a failed test automatically.

ITestListener Interface

The **`ITestListener`** interface allows you to listen to events such as test start, test success, test failure, etc. This interface is often used to log test results, take screenshots on test failure, or generate custom reports.

Methods in `ITestListener`:

onTestStart(ITestResult result): Invoked when a test method starts.
onTestSuccess(ITestResult result): Invoked when a test method passes.
onTestFailure(ITestResult result): Invoked when a test method fails.
onTestSkipped(ITestResult result): Invoked when a test method is skipped.
onTestFailedButWithinSuccessPercentage(ITestResult result): Invoked when a test method fails but is within the success percentage defined in TestNG.
onStart(ITestContext context): Invoked when the test context starts.
onFinish(ITestContext context): Invoked when the test context finishes.

Example: `ITestListener` Implementation


import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class CustomTestListener implements ITestListener {
    @Override
    public void onTestStart(ITestResult result) {
        System.out.println("Test started: " + result.getName());
    }

    @Override
    public void onTestSuccess(ITestResult result) {
        System.out.println("Test passed: " + result.getName());
    }

    @Override
    public void onTestFailure(ITestResult result) {
        System.out.println("Test failed: " + result.getName());
        // You can also add code here to take screenshots
    }

    @Override
    public void onTestSkipped(ITestResult result) {
        System.out.println("Test skipped: " + result.getName());
    }

    @Override
    public void onStart(ITestContext context) {
        System.out.println("Test context started: " + context.getName());
    }

    @Override
    public void onFinish(ITestContext context) {
        System.out.println("Test context finished: " + context.getName());
    }
}

ISuiteListener Interface

The **`ISuiteListener`** interface listens for events related to an entire test suite. It is used for setting up and tearing down resources for a whole suite of tests.

Methods in `ISuiteListener`:

onStart(ISuite suite): Invoked when the suite starts.
onFinish(ISuite suite): Invoked when the suite finishes.

Example: `ISuiteListener` Implementation


import org.testng.ISuite;
import org.testng.ISuiteListener;

public class SuiteListener implements ISuiteListener {
    @Override
    public void onStart(ISuite suite) {
        System.out.println("Test Suite started: " + suite.getName());
    }

    @Override
    public void onFinish(ISuite suite) {
        System.out.println("Test Suite finished: " + suite.getName());
    }
}

IRetryAnalyzer Interface

The **`IRetryAnalyzer`** interface is used to retry failed tests automatically. It allows you to specify how many times a test should be retried before it's marked as failed.

Methods in `IRetryAnalyzer`:

boolean retry(ITestResult result): Determines whether a failed test should be retried. Returns `true` to retry the test, and `false` to mark it as failed.

Example: `IRetryAnalyzer` Implementation


import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryAnalyzer implements IRetryAnalyzer {
    private int retryCount = 0;
    private static final int maxRetryCount = 3;

    @Override
    public boolean retry(ITestResult result) {
        if (retryCount < maxRetryCount) {
            retryCount++;
            return true;
        }
        return false;
    }
}

@Listeners Annotation

The **`@Listeners`** annotation is used to associate listener classes with a test class in TestNG. This allows the listeners to be applied to specific test classes or test methods.

Example: Using `@Listeners` Annotation


import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(CustomTestListener.class)
public class TestClass {
    @Test
    public void sampleTest() {
        System.out.println("Running test...");
    }
}

Detailed Explanation of `ITestResult`, `ISuite`, and `@Listeners`

1. ITestResult Interface

The **`ITestResult`** interface in TestNG provides information about the status and result of a test method during its execution. It stores details like whether a test has passed, failed, or skipped, and contains other useful metadata about the test.

Common Methods in `ITestResult`:

getName(): Returns the name of the test method.
getStatus(): Returns the status of the test (SUCCESS, FAILURE, or SKIP).
getThrowable(): Returns the throwable (exception) that caused the test to fail.
getStartMillis(): Returns the start time of the test method in milliseconds.
getEndMillis(): Returns the end time of the test method in milliseconds.
getMethod(): Returns the `ITestNGMethod` object, which provides details about the method.

Example: Using `ITestResult` in `ITestListener`


        import org.testng.ITestContext;
        import org.testng.ITestListener;
        import org.testng.ITestResult;
        
        public class CustomTestListener implements ITestListener {
        
            @Override
            public void onTestStart(ITestResult result) {
                System.out.println("Test started: " + result.getName());
            }
        
            @Override
            public void onTestSuccess(ITestResult result) {
                System.out.println("Test passed: " + result.getName() + " (Duration: " + 
                    (result.getEndMillis() - result.getStartMillis()) + "ms)");
            }
        
            @Override
            public void onTestFailure(ITestResult result) {
                System.out.println("Test failed: " + result.getName());
                System.out.println("Reason: " + result.getThrowable());
            }
        
            @Override
            public void onTestSkipped(ITestResult result) {
                System.out.println("Test skipped: " + result.getName());
            }
            
            @Override
            public void onStart(ITestContext context) {
                System.out.println("Test context started: " + context.getName());
            }
        
            @Override
            public void onFinish(ITestContext context) {
                System.out.println("Test context finished: " + context.getName());
            }
        }

Explanation:

The `ITestResult` object is passed to all `ITestListener` methods, providing detailed information about the test execution, such as its name, status, duration, and any exception thrown.

2. ISuite Interface

The **`ISuite`** interface provides information about a TestNG suite, including the results of its tests, configuration details, and the suite's methods. It is used to handle and monitor suite-level events and access suite-related data.

Common Methods in `ISuite`:

getName(): Returns the name of the suite.
getResults(): Returns a map of all the tests and their results in the suite.
getAllMethods(): Returns a list of all the methods that are part of the suite.
getParameter(String parameterName): Retrieves the value of a suite parameter from the TestNG XML configuration.

How to Execute Only Failed Test Cases in TestNG

When running automated tests, some test cases might fail due to unexpected conditions such as network issues, application state, or environment configuration. Instead of re-running the entire test suite, **TestNG** allows us to re-execute only the failed test cases using a **rerun mechanism**. This can save time and help identify flaky tests.

Steps to Re-Execute Only Failed Test Cases in TestNG

TestNG provides the ability to automatically re-execute failed tests in a subsequent run using the **testng-failed.xml** file, which is generated after a test run with failures. This file contains only the failed test cases, and running it will only execute the failed tests.

Here’s the basic flow:

Run the full test suite (including all tests).
If any tests fail, TestNG automatically generates a `testng-failed.xml` file inside the **test-output** folder.
Use this `testng-failed.xml` file to rerun only the failed test cases.

Step-by-Step Example

Let’s create an example test class where some tests will fail, and then we’ll re-run only the failed ones using TestNG's **testng-failed.xml** file.

Step 1: Create a Test Class with Failing Tests


            import org.testng.Assert;
            import org.testng.annotations.Test;
            
            public class FailedTestExample {
            
                @Test
                public void testA() {
                    System.out.println("Test A passed.");
                    Assert.assertTrue(true);
                }
            
                @Test
                public void testB() {
                    System.out.println("Test B failed.");
                    Assert.fail();  // This test will fail
                }
            
                @Test
                public void testC() {
                    System.out.println("Test C passed.");
                    Assert.assertTrue(true);
                }
                
                @Test
                public void testD() {
                    System.out.println("Test D failed.");
                    Assert.fail();  // This test will fail
                }
            }

In this example, `testB()` and `testD()` are designed to fail, while `testA()` and `testC()` will pass.

Step 2: Run the Test Suite

When you run the entire test suite, TestNG will execute all the tests. Since `testB()` and `testD()` fail, TestNG will generate a **testng-failed.xml** file, which includes only the failed tests.

Step 3: Locate the `testng-failed.xml` File

After running the test suite, navigate to the **test-output** folder. You will find a file named `testng-failed.xml`. This file is automatically created by TestNG whenever any tests fail.

Step 4: Re-Run the Failed Test Cases

To re-run the failed test cases, simply execute the **testng-failed.xml** file as a TestNG suite:


            java -cp "path-to-your-project-libs/*;bin" org.testng.TestNG test-output/testng-failed.xml

This command will execute only the failed test cases (in this case, `testB()` and `testD()`).

Example of `testng-failed.xml`

The `testng-failed.xml` file generated after the failed test cases might look like this:


            <suite name="Failed suite">
                <test name="Failed tests">
                    <classes>
                        <class name="FailedTestExample">
                            <methods>
                                <include name="testB"/>
                                <include name="testD"/>
                            </methods>
                        </class>
                    </classes>
                </test>
            </suite>

When this file is executed, only `testB()` and `testD()` from the **FailedTestExample** class will be re-run.

Handling Failed Tests Programmatically with IRetryAnalyzer

If you want to retry failed tests automatically instead of using the `testng-failed.xml` file, you can use the **IRetryAnalyzer** interface in TestNG. This interface allows you to retry a test based on a condition or a maximum retry count.

Example: Retrying Failed Tests with `IRetryAnalyzer`


            import org.testng.IRetryAnalyzer;
            import org.testng.ITestResult;
            
            public class RetryFailedTests implements IRetryAnalyzer {
            
                private int retryCount = 0;
                private static final int maxRetryCount = 2;  // Maximum retry limit
            
                @Override
                public boolean retry(ITestResult result) {
                    if (retryCount < maxRetryCount) {
                        retryCount++;
                        return true;  // Retry the test
                    }
                    return false;  // Do not retry the test
                }
            }

Now, you can apply this retry analyzer to your test methods using the `@Test` annotation:


            import org.testng.Assert;
            import org.testng.annotations.Test;
            
            public class TestWithRetry {
            
                @Test(retryAnalyzer = RetryFailedTests.class)
                public void testRetry() {
                    System.out.println("Executing testRetry...");
                    Assert.fail("Test failed, retrying...");
                }
            }

Advantages of Re-Running Failed Test Cases

Efficient Execution: Instead of re-running all tests, only failed tests are executed, saving time during the testing cycle.
Flaky Tests Identification: Re-running failed tests can help identify flaky tests (tests that fail intermittently).
Improved Debugging: Running only failed tests allows you to focus on debugging the specific issues without the overhead of executing the entire suite.

Using Suite-Level Annotations and Listeners in Selenium with TestNG

TestNG offers annotations like @BeforeSuite and @BeforeTest for test setup at different levels of execution. Additionally, **Listeners** allow monitoring the test flow and capturing events such as test success, failure, or skip to generate logs or reports dynamically.

Understanding Suite-Level Annotations

@BeforeSuite: This method runs before any tests in the entire suite execute. It is used for global setup tasks like initializing reports or setting up database connections.
@BeforeTest: This method runs once before each test tag in the testng.xml file. It is commonly used to initialize browsers or set configurations for tests defined in the specific test tag.

TestNG Listener: Monitoring Test Execution

TestNG listeners allow implementing custom actions when certain events occur (e.g., test pass, fail, or skip). This is helpful for generating detailed reports, logging information, and capturing screenshots during failures.

Maven Dependencies

<dependencies>
                    <!-- Selenium Java -->
                    <dependency>
                        <groupId>org.seleniumhq.selenium</groupId>
                        <artifactId>selenium-java</artifactId>
                        <version>4.12.1</version>
                    </dependency>
                
                    <!-- TestNG -->
                    <dependency>
                        <groupId>org.testng</groupId>
                        <artifactId>testng</artifactId>
                        <version>7.7.1</version>
                    </dependency>
                </dependencies>

Code Example: Using Suite-Level Annotations and Listeners

import org.openqa.selenium.WebDriver;
                import org.openqa.selenium.chrome.ChromeDriver;
                import org.testng.Assert;
                import org.testng.annotations.*;
                
                public class SuiteLevelAnnotationsExample {
                
                    WebDriver driver;
                
                    @BeforeSuite
                    public void beforeSuite() {
                        System.out.println("Before Suite: Initialize Extent Report or global resources");
                    }
                
                    @BeforeTest
                    public void beforeTest() {
                        System.out.println("Before Test: Launch browser or configure test settings");
                        driver = new ChromeDriver();
                    }
                
                    @Test
                    public void googleTitleTest() {
                        driver.get("https://www.google.com");
                        String title = driver.getTitle();
                        System.out.println("Test: Google title fetched - " + title);
                        Assert.assertEquals(title, "Google");
                    }
                
                    @AfterTest
                    public void afterTest() {
                        System.out.println("After Test: Close the browser");
                        if (driver != null) {
                            driver.quit();
                        }
                    }
                
                    @AfterSuite
                    public void afterSuite() {
                        System.out.println("After Suite: Flush reports and clean up resources");
                    }
                }

Implementing TestNG Listener for Monitoring

import org.testng.ITestContext;
                import org.testng.ITestListener;
                import org.testng.ITestResult;
                
                public class TestListener implements ITestListener {
                
                    @Override
                    public void onTestStart(ITestResult result) {
                        System.out.println("Test Started: " + result.getName());
                    }
                
                    @Override
                    public void onTestSuccess(ITestResult result) {
                        System.out.println("Test Passed: " + result.getName());
                    }
                
                    @Override
                    public void onTestFailure(ITestResult result) {
                        System.out.println("Test Failed: " + result.getName());
                        // Capture screenshot logic can be added here
                    }
                
                    @Override
                    public void onTestSkipped(ITestResult result) {
                        System.out.println("Test Skipped: " + result.getName());
                    }
                
                    @Override
                    public void onFinish(ITestContext context) {
                        System.out.println("All Tests Finished");
                    }
                }

Using the Listener in TestNG

To use the listener, update your testng.xml file:

<suite name="ExampleSuite">
                    <listeners>
                        <listener class-name="TestListener" />
                    </listeners>
                
                    <test name="GoogleTest">
                        <classes>
                            <class name="SuiteLevelAnnotationsExample" />
                        </classes>
                    </test>
                </suite>

Explanation of Code

@BeforeSuite: Initializes resources like reports or global setups needed for all tests in the suite.
@BeforeTest: Launches the browser before each test group in the testng.xml.
TestNG Listener:
- onTestStart: Logs when the test starts.
- onTestSuccess: Logs test success.
- onTestFailure: Can capture screenshots or logs when a test fails.
@AfterSuite: Flushes reports or cleans up resources after all tests complete.

When to Use Suite-Level Annotations and Listeners

@BeforeSuite: When you need to set up resources like **Extent Reports, database connections, or environment configurations** that are shared across all tests.
@BeforeTest: When each test group (defined in testng.xml) needs **separate browser sessions or specific configurations**.
Listeners: When you need to **monitor the test flow** dynamically for **custom logging, taking screenshots on failure, or generating reports** in real-time.
Continuous Integration: Useful for integrating with **Jenkins or CI pipelines** to track test failures, successes, or skips automatically.