Bug Problems with 100% Code Coverage

Test coverage has many limitations in the programming world. When I first heard of tools like Emma I really thought using them would make my programs totally robust and bug-less. However after reading articles like this, and working on getting 100% code coverage with my Stack project, I am starting to think otherwise. Would it be possible to introduce a full fledged bug into code that gets full code coverage?

INTRODUCING A BUG

My programming partner Daniel Arakaki and I decided to create a version of our previous stack system that would have 100% coverage in Emma using the existing unit tests, as well as a significant bug that could ruin the system if it were ever put into real use. After contemplating it for awhile, we came up with a plan to introduce a bug by changing a basic function call in the getTop() function of the ClearStack class.

by simply changing this method:

  public Object getTop() throws EmptyStackException {
  return this.top();
}

to:

  public Object getTop() throws EmptyStackException {
   return this.pop();
 }

we effectively introduced a bug into the system that would (using our current test cases) get 100% code coverage using Emma. We ran it through JUnit and Emma and sure enough it all worked fine. Final stack system with introduced bug: Download.

Imagine if this stack system was put into any sort of real use. There would be EmptyStackExceptions all over the place and if you ever did want to see what object was at the top of the stack, you would effectively screw up the stack structure. Any calls to pop after that would return a different object than expected. 

The funny thing about it is that all this could potentially happen just because of one wrong letter in the method (p instead of t). To put it in perspective, imagine if this bugged stack system was implemented on a vending machine. If there was a button to check item availability and a user pressed it, they would get a free item. In the long run the numbers wouldn't add up for the company supplying the vending machine and money would be lost. If a programmer bases all their tests on code coverage results, their code will be flawed. 

CONCLUSION

I think code coverage tools like Emma are very useful for programming. Code testing and optimization becomes more efficient, and it can at least help one to test their code better. The thing is, you cannot rely on them for all your testing needs. It was very easy to introduce a significant bug into the stack system and still get full coverage. It was only when I added another test case that checked for multiple getTop() calls that the problem was found. I, for one would not have though about this potentially being a problem in the original stack program. I guess it comes down to how thorough you want to test your code, and this cannot be fully based on code coverage. It is only a tool to help you with it.

In addition, another downfall of code coverage tools like Emma is that they cannot account for lines of code that should have been written, which means code that has 100% coverage could still have bugs. Overall though, I still think code coverage tools should be used to help in testing. They are quick and efficient and can really improve testing. I will definitely will use Emma as I write tests to my programs knowing all of this.