Why code coverage matters - Blog - Open Source - schlitt.info

schlitt.info - php, photography and private stuff

Trace this

  • Revision 1
  • by Tobias Schlitt
  • Wed, 19 Nov 2008
  • 23:29:46 +0100

Hire me

  • Hire a PHP quality expert

Surprise me

  • Amazon wishlist

Follow this

  • RSS syndication
  • Follow me on twitter

Read this

  • eZ Components book

Get more

  • Get more on Ohloh
  • Get more on Xing

Praise these

  • Thanks to Lighttpd
  • Thanks to PHP
  • Thanks to SQLite
  • Thanks to Gentoo Linux
  • Thanks to Subversion
  • Thanks to WCV

Respect this

  • Licensed under by-nc-sa

Conforms to

F**k spammers

  • F**k spammers

Why code coverage matters

I'm a fan of PHPUnit code coverage reports. And with this sentence I can see a lot of the developers out there shiver, because they are of the opinion, that code coverage reports for unit tests are nonsense and cannot give you any hint on the quality of a test suite. I see it a bit differently. Surely, a high code coverage rate of a test suite does never indicate, that code is well tested (if you have not written the code and tests yourself). But the other way around works: A small code coverage rate definitly means, that the test suite is not sufficient. But let me dig a bit deeper into code coverage and what it gives you.

The PHPUnit code coverage report indicates, how many lines of the code you are testing have been executed during the test run. It shows you the figure for each directory (aggregated from the files and directories contained) and each file of the tested code and gives you some color indication, if you have a high, medium or low code coverage rate. Beside that, it shows you the source code of each of your files and indicates, which code lines got executed, which not and which are unreachable.

So, basically you can check, which lines of the code to test are covered by the unit tests and, more important, which are not. A covered line does actually not mean, that this code line is properly tested, but a not covered line definitly means, that this one is not tested at all. For you as a developer, the latter fact is quite important, since it gives you an indicator for test cases you still need to create.

Creating a unit test is over all not easy. To create a proper unit test you need to think of several things: First you need to know, what a specific method should be doing. That is kinda easy, if you wrote (usually will write) the method yourself, but can be kinda hard, if you didn't. Second, you need to know what the preconditions of the method are (which attributes the methods accesses, what the values of these must be to achieve the desired result, which other objects must be instanciated,...). Third you need to know, how the method should behave for different input parameters (and with "behaviour" I mean success and failure).

If you have all these information collected, you can start writing the test cases for your method. Still it may sometimes be hard, if you have a method that performs a more complex operation, since you need to think of all possible (and in most cases you should also think about impossible) combinations your method can be called with. But I'm sure, if you are not lazy, you can figure out some sensible combinations of preconditions, input parameters and expected result.

But how can you figure out, if you really covered all necessary cases? Well, that is even harder than creating the test cases, because you usually think in limited dimensions and can not think of every possible case how a co-developer (or user, if you're developing a library) may try to use your method. Anyway, at this point, code coverage comes into the game. Although it cannot tell you, how a possible user of your method might try to interact with it, it can give you a hint on which code was not tested by you so far. If you see some red lines in the code coverage report, you definitly missed some cases you had in mind while developing the method and which are not tested.

Let me give a small example:

class DiceGamePlayer { public $playerName; // ... some more attributes and methods here... public function throwDice( $diceSize ) { if ( !is_int( $diceSize ) && $diceSize < 2 ) { throw RuntimeException( "A dice with $diceSize values does not make sense! Player $player must be cheating!" ); } return mt_rand( 1, $diceSize ); } }

This tiny method could be from a dice game, that contains several different dices (e.g. roleplaying games usually have). It is quite small, so it should be easy to think of unit test cases. The method obviously has the attribute $playerName as its only pre-condition and expects and integer value larger than 2 as its parameter. The desired return value is an integer value between 1 and the given dice size. A typical test case would be, to give it the number 6 and expect that it returns an integer value between 1 and 6. With that, you already have the desired functionality of the method covered. At least, you might think so.

But a look at the code coverage will indicate, that you only covered 50% of the (executable) code, because you only checked the functionality for a correct input value and did not test the method with an incorrect value. That means: More test cases are necessary to fully test the method. Since the method expects an integer value, it would be a good idea to test if the exception is thrown if you give it a string value like "foo". Well, another look at the code coverage will indicate that now 100% of the code is tested... and this is the point, where code coverage cannot help you any further.

Although you covered the full code now, you still have a bug in it. Another test case is necessary: What happens, if you give that method the integer 1? This is an integer value but smaller than 2, so the new test case would expect the RuntimeException to be thrown. This test case would actually fail, because the checking condition is incorrect. You would need to replace the logical-and operator with a logical-or to have it working correctly.

So, why do I tell you all this, if code coverage can not help you in this place? Well, it already helped you, with indicating, that the given portion of code was not tested. But that does not mean, that you can switch off your brain at this point. You still need to think about all possible cases this part of the code should cover. But still the code coverage report gave you a good hint, that you missed to test something at all.

What is the conclusion of all this? First of all: Writing sensible unit tests is not easy and needs at least a good imagination. A large code coverage rate does not mean that your test cases are good and that every possibility is tested. But a small code coverage number always means, that you missed to test parts of your code. And the report itself can then give you sensible hints, which parts of the code still need explicit testing. And if you you are capable of writing sensible test cases for these parts of your code and are sure (sure, you never can be!) that you already wrote sensible test cases for the covered parts, a large code coverage rate can also indicate a good test suite.

I think a much better alternative for testing the quality of your test suite is the so called "code mutation", which will hopefully be developed for PHPUnit during this years Google Summer of Code. A code mutation tool will try to change small bits of the code to test over and over again, until a test case will fail. If your test suite bails out with (almost) any change in the tested code, this indicates, that you have covered a lot of imaginable cases and in that way, that you tested well (remember, that you can never be really sure!).

So long I can only recommend to check your code coverage report and see where you really missed to test code and then switch on your brain to create sensible test cases for these parts. I'm pretty sure, that this will already raise the quality of your code a lot. At least, this is what I experienced again and again in the past, when looking at my code coverage reports and creating test cases for the not executed parts. Be sure, that you can already clean up a lot of tiny (and possibly larger) bugs and that you can ensure to not break BC much better than before.

So long, happy testing! :)

If you liked this blog post or learned something, please consider using flattr to contribute back: .

Trackbacks

Comments

Add new comment

Fields with bold names are mandatory.