Code Coverage Using Emma

Emma is an open-source Java code coverage tool that I recently added to my collection of Java quality assurance tools [Ant, JUnit, CheckStyle, PMD, FindBugs]. What it essentially does is check that every line of code in the source files is run though using class, method, line, and basic block coverage methods. It can be a very handy tool for testing as it makes sure that every line of written code is tested (though there still may be bugs). It is another form of white box testing since it analyzes the internals of the program. 

To see how it all worked, I decided to run Emma on the stack system I worked on last week using the other Java tools. Initially, the method, block, and line-level coverage of the system was around 75%. For a small program like this, test coverage of 75% could be fully improved by adding more comprehensive test cases. It was my goal to get the entire program to 100% coverage. 

PROCESS & PROBLEMS

I decided to work on this exercise with another classmate Arthur Shum, to get a feel for paired programming. We both worked on our own computers but we went through everything together and resolved problems that arose. It turned out to be a very productive programming session. The installation and set up of Emma was very easy, just like the other Java tools. I was surprised at how nice the Emma html output is. The two classes Stack and ClearStack had color coded coverage markers - Green for covered code and Red for non-covered lines of code. This made it extremely easy to point out what needed to be tested. 

To get full coverage I had to go in and make more unit tests for the toString and getTop methods of the Stack object. I also had to make some new unit tests for the getTop method of the ClearStack Object. This turned out to be a rather simple task, but it was great because I got to write some more of my own JUnit tests. I feel a lot more comfortable writing tests with JUnit. 

The biggest problem I ran into during this whole process was having my correct code suddenly error in eclipse.  This would happen to me almost every time I made a change to one of my test classes. Arthur was having the same sort of problems with his project and eventually he found out that if you rename the containing directory each time, the problem was fixed. It was really a pain though. I just recently found out this was happening because I was running Ant (from a terminal window) and Eclipse simultaneously. Both were compiling the class files into the same directory and this was confusing the system. 

Eventually got 100% coverage in the stack project. Arthur did the same and it was all done rather quickly. Stack project with 100% code coverage: stack-tylerwolff-6.0.929.zip

One interesting thing I observed during this exercise was a line of code that produced a yellow coverage marker in the results. I noticed this happening in Arthur's code for his Stack class and it took us awhile to figure it out. After analyzing my test cases for that class we realized that he was not checking both cases for the if statement. To get full coverage, we needed to test all boolean cases, both true and false. This, I thought, is a pretty smart function.

However, I can see where getting 100% code coverage does not equal a fully tested, bug free program. Emma really only checks that all lines are tested. What would happen if some important factors were left out in an if statement? Or what if the system utilized a function that deleted something when it was supposed to only view it? All this would still get 100% code coverage if there were basic unit tests, yet it would still be buggy. I realize that it is not enough to rely on code coverage tools like Emma. Still, I think it is definitely a worthwhile tool to use on projects. It works quickly, efficiently and can improve the robustness of your code.