In this post I will describe how to use RunListener in Android instrumented tests (i.e. UI Automator or Espresso). But before getting into the nitty-gritty details, let’s discuss if you need to use it at all.
First let me explain why I started looking into RunListener. Our test framework, based on UiAutomator and JUnit4, was taking screenshots in TestWatcher::failed() method, but was performing cleanup in teardown() method annotated with @After. The problem was that other actions performed in teardown() would often change the current activity, and thus any screenshot taken after that would would be useless. The simplest solution was to always take screenshot in teardown() and then delete the unnecessary ones in TestWatcher::passed() method. However, this solution had one downside – it would require the developer to always remember to take a screenshot in teardown() when writing a new test class. Alternative implementation would be to have base test class and implement taking screenshot in base teardown(), but then developer would have to always remember to call super.teardown() in a subclass. Granted, those are not big problems, but I was wondering if there wasn’t better approach – something that is executed after @After, but before TestWatcher.
And this is how RunListener came to light. Unfortunately, after playing with it I’ve discovered that it’s called after both @After and TestWatcher methods are executed, and therefore it wasn’t solution of our problems.
For reference, here is an order of execution of @Before/@After, TestWatcher, RunListener and @BeforeClass/@AfterClass.
- RunListener::testRunStarted
- @BeforeClass
- RunListener::testStarted
- @Before
- Test itself
- @After
- TestWatcher::succeeded (failed)
- RunListener::testFinished (testFailure)
- @AfterClass
- RunListener::testRunFinished
Considering this order of execution and that it is easier to use TestWatcher and @Before/@After annotations then RunListener, I don’t really know what would be the good use case for RunListener. Nevertheless, since I’ve spend some time on figuring out how to make it work in Android instrumented tests, I’ve decided to share this knowledge with you.
But wait, you might ask, what about your screenshots? Well, I found a good solution, but that will be a topic of another post.
RunListener in JUnit4
First let me remind you how RunListener is used in non-Android JUnit4.
We start by creating our listener class which extends RunListener. This step is the same for Android instrumented tests as well.
package com.example.myapp;
import org.junit.runner.Description;
import org.junit.runner.Result;
import org.junit.runner.notification.Failure;
import org.junit.runner.notification.RunListener;
public class TestListener extends RunListener
{
/**
* Called before any tests have been run.
* */
public void testRunStarted(Description description) throws java.lang.Exception
{
System.out.println("Number of tests to execute : " + description.testCount());
}
/**
* Called when all tests have finished
* */
public void testRunFinished(Result result) throws java.lang.Exception
{
System.out.println("Number of tests executed : " + result.getRunCount());
}
/**
* Called when an atomic test is about to be started.
* */
public void testStarted(Description description) throws java.lang.Exception
{
System.out.println("Starting execution of test case : "+ description.getMethodName());
}
/**
* Called when an atomic test has finished, whether the test succeeds or fails.
* */
public void testFinished(Description description) throws java.lang.Exception
{
System.out.println("Finished execution of test case : "+ description.getMethodName());
}
/**
* Called when an atomic test fails.
* */
public void testFailure(Failure failure) throws java.lang.Exception
{
System.out.println("Execution of test case failed : "+ failure.getMessage());
}
/**
* Called when a test will not be run, generally because a test method is annotated with Ignore.
* */
public void testIgnored(Description description) throws java.lang.Exception
{
System.out.println("Execution of test case ignored : "+ description.getMethodName());
}
}
Next we need to register our listener with test runner. There are two ways to achieve that – one is to register the listener with JUnitCore, and another – to create a custom runner.
Register with JUnitCore:
package com.example.myapp;
import org.junit.runner.JUnitCore;
import com.example.myapp.SampleTest;
public class TestRunner
{
public static void main(String[] args)
{
JUnitCore runner = new JUnitCore();
runner.addListener(new TestListener());
runner.run(SampleTest.class);
}
}
Custom runner:
package com.example.myapp;
import org.junit.runner.notification.RunNotifier;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.InitializationError;
public class CustomRunner extends BlockJUnit4ClassRunner {
public MyRunner(Class<?> klass) throws InitializationError {
super(klass);
}
@Override public void run(RunNotifier notifier){
notifier.addListener(new JUnitExecutionListener());
notifier.fireTestRunStarted(getDescription());
super.run(notifier);
}
}
If using approach with custom runner, we need to specify it in our test class using @RunWith annotation:
package com.example.myapp;
import org.junit.Test;
import org.junit.runner.RunWith;
import static org.junit.Assert.assertTrue;
@RunWith(CustomRunner.class)
public class SampleTest {
@Test
public void test(){
assertTrue(false);
}
}
RunListener with Android instrumented tests
The problem when using RunListener with Android instrumented tests lies in registering the listener. Android tests are executed using AndroidJUnit4 runner, which is declared as final class, and therefore cannot be extended for creating a custom runner.
Fortunately Google provides solution for this problem – listeners can be provided as arguments to AndroidJunitRunner as described here. There are to ways to do that – one is to specify listeners as arguments to adb shell am instrument command, the other – to specify them in manifest file. I think that very few people execute Android tests using am instrument command, and most use the gradle connectedAndroidTest task for that. For that reason we’ll use approach with manifest file.
The following tags need to be added to manifest in order to register TestListener listener.
<instrumentation
android:name="androidx.test.runner.AndroidJUnitRunner"
android:targetPackage="com.example.myapp">
<meta-data
android:name="listener"
android:value="com.example.myapp.TestListener" />
</instrumentation>
However, here is the tricky part – if you add this to your main manifest file it won’t work. The reason is that instrumented tests use they own manifest file which is automatically created by gradle, and this listener registration code simply won’t be there (you can check that by looking at contents of final manifest file here app/build/intermediates/packaged_manifests/). Therefore, we need to create our own manifest file in app/src/androidTest folder (filename should be AndroidManifest.xml), so that gradle would use it as a base for the automatically generated one.
The manifest file can be as simple as this:
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.example.myapplication">
<instrumentation
android:name="androidx.test.runner.AndroidJUnitRunner"
android:functionalTest="false"
android:handleProfiling="false"
android:label="Tests for com.example.myapplication"
android:targetPackage="com.example.myapplication">
<meta-data
android:name="listener"
android:value="com.example.myapplication.TestListener" />
</instrumentation>
</manifest>
That’s all that’s we need to use RunListener in Android tests. Now you can simply run your tests and check the logcat output – log messages generated by our TestListener will be there.
Please let me know in the comments what you think, and especially if you know any good uses for RunListener.
very nice article, thanks.
I am not able to see logs from functions other than testStarted