Test Automation Chronicles

This is my take on the “most common test automation mistakes” topic. However, I’m not going to delve into the mistakes SDETs make when creating test automation strategy or using Selenium etc. Instead, I’m going to concentrate on mistakes on a coding level, which are relevant to many testing tools, frameworks and programming languages.

Note: Java and JUnit4 will be used in my examples.

1. No error messages in assertions.

assertTrue(isOrderCompleted());

Without an error message a reader of a test report will not be able to see why the test failed and so they will have to look at the source code to find that out.

Here is how the test result looks for the assertion above. Not very informative, right?

java.lang.AssertionError
 at org.junit.Assert.fail(Assert.java:87)
 at org.junit.Assert.assertTrue(Assert.java:42)
 at org.junit.Assert.assertTrue(Assert.java:53)
 at com.example.test(MyTest.java:38)

The proper way to write the assertion above would be something like below. In general, the more information is provided in the error message, the better it is.

assertTrue("Order was not completed", isOrderCompleted());

And the test result:

java.lang.AssertionError: Order was not completed at org.junit.Assert.fail(Assert.java:87) at org.junit.Assert.assertTrue(Assert.java:42) at org.junit.Assert.assertTrue(Assert.java:53) at com.example.test(MyTest.java:38)

Now better, right?

There might be some cases when assertion library is intelligent enough to automatically generate informative error messages, but in all other cases it is better to write the error message yourself.

2. Using assertTrue for everything.

assertTrue(orders == 2);
assertTrue(orders < 2)
assertTrue(orderDesc.contains("pancakes"));

This issue might be partially caused by the pathetic lack of stock assertions in JUnit framework, but considering that there are libraries like Hamcrest matchers, Google Truth, AssertJ etc. that is not an excuse.

For example, the assertions above could be rewritten using Google Truth library like this:

// JUnit stock assertion
assertEquals("Values or 'orders' is not as expected", orders, 2); 
// Google Truth assertions
assertWithMessage("Values or 'orders' is not as expected").that(orders).isLessThan(2);
assertWithMessage("Values or 'orderDesc' is not as expected").that(orderDesc).contains("pancakes");

Why is this better? Because of the better readability of the error message in the test results. Compare the error message for the old assertions and the new ones:

java.lang.AssertionError
 at org.junit.Assert.fail(Assert.java:87)
 at org.junit.Assert.assertTrue(Assert.java:42)
 at org.junit.Assert.assertTrue(Assert.java:53)
at com.example.test(MyTest.java:38)

vs

java.lang.AssertionError: Values or 'orders' is not as expected 
Expected :3
Actual   :2
at com.example.test(MyTest.java:38)
Values or 'orders' is not as expected
expected to be less than: 2
but was                 : 3
at com.example.test(MyTest.java:38)
Values or 'orderDesc' is not as expected
expected to contain: pancakes
but was            : fish
at com.example.test(MyTest.java:38)

3. Empty catch blocks

try {
    doSomething();
} catch (SomeException ex) {}

Why it is a bad to have empty catch blocks in test automation code (it is wrong in any code, but for slightly different reasons)? Because the test will most likely fail anyway (if it is a well designed and correctly automated test, that is), but it will be much more difficult to understand why did it fail, as compared to having the exception to bubble all the way up to the test report.

4. Assertions outside the test

def myTest() {
  String orderDescription = OrderPage.getOrderDescription();
  assertThat(orderDescription).contains("pancakes");
}
class OrderPage {
  public static String getOrderDescription() {
    String orderDescription = page.getElementText("orderDescId");
    assertNotNull(orderDescription);
    return orderDescription;
  }
}

If there are assertions outside the test (e.g. in page objects, helper methods etc.) then reading and maintaining the existing tests, as well as writing the new ones, becomes difficult. When reading the test you will have to look inside all of it methods in order to find out what kind of checks it performs. When modifying the existing tests you might need to modify not only the test case, but assertions in the methods it calls, with sometimes undesirable impact on other test cases which use same methods. And when writing new tests you might not need those checks to be performed at all, or even need them to be not performed.

5. No logging

def myTest() {
  String orderDescription = OrderPage.getOrderDescription();
  assertThat(orderDescription).contains("pancakes");
}
class OrderPage {
  public static String getOrderDescription() {
    return = page.getElementText("orderDescId");
  }
}

Logging is truly an indispensable tool when debugging or doing root cause analysis (and believe me, you will do that a lot), and therefore should not be omitted (maybe with exception of a very simple automation script).

def myTest() {
  logger.info("Starting test: " + testname);
  String orderDescription = OrderPage.getOrderDescription();
  assertThat(orderDescription).contains("pancakes");
  logger.info("Finished test: " + testname);
}
class OrderPage {
  public static String getOrderDescription() {
    logger.debug("Getting order description text");
    return page.getElementText("orderDescId");
  }
}

Please note that the logging statements above are rather a pseudo-code for simplicity sake and a real logging is done slightly differently in Java and JUnit.

Note: This is Part 1 of a two-part series on most common test automation mistakes. Part 2 is here.