Is “Coverage” synonym for “Quality”?

100%

As a good tester, surely more than once you have considered looking at the coverage of your tests on the scope of a project. This indicator allows you to know, through the execution of a test suite, which lines of code have been executed and which have not, allowing programmers to see if there are parts of the code that are not being tested. Many times the detail ends up summarized in a percentage of code that is being executed, which is the measure that is most often taken into account when talking about this concept.

Many of the programmers I know establish as a goal to reach 100% coverage of their projects, in the aims of achieving a code robust in terms of quality. They tend to believe that covering the whole code will make sure you do not leave anything to check, and thus satisfy all testing needs.

But, is that statement true? 100% coverage makes me sure that my code will be wonderful? Can I say that my code is well tested if I have a good coverage?

It’s good wanting to reach 100%, but we must be aware of why we do not get there yet

Let’s set up an example case of analysis. Suppose we have a folder with two classes, AController and BController. When executing all the functional test suites, which we believe cover all the possible use cases of our application, we obtain this coverage metrics:

AController.php - 100% coverage
BController.php - 0% coverage

In this case, of course, I have defined very extreme numbers to exaggerate more the evidence I want to talk about. Plus, note that despite dealing with PHP files in this case, we can say pretty much the same for most code languages.

Returning to the topic, why the code in class B has not been executed? Two major possibilities to consider are:

  • I’m not testing one or more cases that would make these lines execute. Seeing a coverage of less than 100% would indicate then that we are missing some tests to write that could help us to certify the code works well. 
  • It’s code that is not used anywhere in our application. Seeing a coverage of less than 100% then would indicate we should perform some clean up of our code.

But what is the problem? Many times, stubborn in our search for the perfect 100%, we will not stop to analyze which of the reasons could be the one that causes this low value. The “simplest” for the programmer, in this case, usually implies creating a new test that includes this code in the test execution, in order to increase the coverage. This often is a huge mistake, since we are thinking of raising a number, and not in the good of our application.

Aiming for a coverage of 100% must surely be a goal, but not the main one. We must never lose the focus of why we test: we test to be sure that our code works as it is intended to work. And the test cases must be a reflection of this fact.

Therefore, more important than having 100% is to have, during our development timeline, a lower number to be able to act accordingly: even, if necessary, questioning the functionality of the application.

When achieving 100% blurs your sight

Let’s analyze in more detail the AController class. When opening it, we observe the following code:

<?php

use App/Domain/ItemRepository;
use Framework/Response;

class AController
{
    /** @var ItemRepository **/
    private $itemRepository;

    public function item(int $itemId)
    {
        $item = $this->itemRepository->byId($itemId);
        return new Response(200, $item->name());
    }
}

We will assume that we have a testing database that is always minimally filled with an item with ID 1.

AControllerTest contains code similar to the following:

<?php

class AControllerTest
{
    private $browser;

    //...

    public function testItem()
    {
         $response = $browser->visit('/a/item/1');
         $this->assertSame(200, $response->getStatusCode())
    }
}

With a single and simple test, we have easily achieved a 100% coverage. We can already say that our code is well tested! Turn off the computers go home, and tomorrow will be new day, right? … OK, no.

We have mentioned it before: we do not test to reach a number, we test to prove that everything works well and that we do not have unexpected surprises. At first glance, we can already identify two things that we did not do well:

  • We are not verifying that the response has the body we expect, that is, the name of the item. Checking such a simple fact is not banal: it’s possible that one of the developers, while developing the functionality, left a test return statement with a hardcoded object creation inside the implementation of the byId() function, which ends up always returning the same test object, and not the one with ID 1, and mistakenly uploaded the change to our VCS. Sounds unlikely, but the worst always can happen. 
  • And what if the item does not exist? In this case, if we had a test case verifying the behaviour of the application when an item does not exist, we would see that the application throws an error instead of, for example, a 404 code, since we have not contemplated this case during our implementation.

If we had not been focused on thinking about metrics, we might have put more focus on monitoring more unlikely cases. These type of situations are examples of what someĀ methodologies, such as TDD or SDD, try to avoid, focusing on verification and specification prior to development. We have been guided by the ambition of having good metrics, and we have forgotten that what matters the most is to verify that what we do is correct.

Closing up, coverage can be a great help in many aspects when testing, but we should never rely the quality of our work just in that. Experience and carefulness, in cases like this, ends up being much more powerful tools.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s