Code Coverage Using Emma

Emma is an open-source Java code coverage tool that I recently added to my collection of Java quality assurance tools [Ant, JUnit, CheckStyle, PMD, FindBugs]. What it essentially does is check that every line of code in the source files is run though using class, method, line, and basic block coverage methods. It can be a very handy tool for testing as it makes sure that every line of written code is tested (though there still may be bugs). It is another form of white box testing since it analyzes the internals of the program. 

To see how it all worked, I decided to run Emma on the stack system I worked on last week using the other Java tools. Initially, the method, block, and line-level coverage of the system was around 75%. For a small program like this, test coverage of 75% could be fully improved by adding more comprehensive test cases. It was my goal to get the entire program to 100% coverage. 

PROCESS & PROBLEMS

I decided to work on this exercise with another classmate Arthur Shum, to get a feel for paired programming. We both worked on our own computers but we went through everything together and resolved problems that arose. It turned out to be a very productive programming session. The installation and set up of Emma was very easy, just like the other Java tools. I was surprised at how nice the Emma html output is. The two classes Stack and ClearStack had color coded coverage markers - Green for covered code and Red for non-covered lines of code. This made it extremely easy to point out what needed to be tested. 

To get full coverage I had to go in and make more unit tests for the toString and getTop methods of the Stack object. I also had to make some new unit tests for the getTop method of the ClearStack Object. This turned out to be a rather simple task, but it was great because I got to write some more of my own JUnit tests. I feel a lot more comfortable writing tests with JUnit. 

The biggest problem I ran into during this whole process was having my correct code suddenly error in eclipse.  This would happen to me almost every time I made a change to one of my test classes. Arthur was having the same sort of problems with his project and eventually he found out that if you rename the containing directory each time, the problem was fixed. It was really a pain though. I just recently found out this was happening because I was running Ant (from a terminal window) and Eclipse simultaneously. Both were compiling the class files into the same directory and this was confusing the system. 

Eventually got 100% coverage in the stack project. Arthur did the same and it was all done rather quickly. Stack project with 100% code coverage: stack-tylerwolff-6.0.929.zip

One interesting thing I observed during this exercise was a line of code that produced a yellow coverage marker in the results. I noticed this happening in Arthur's code for his Stack class and it took us awhile to figure it out. After analyzing my test cases for that class we realized that he was not checking both cases for the if statement. To get full coverage, we needed to test all boolean cases, both true and false. This, I thought, is a pretty smart function.

However, I can see where getting 100% code coverage does not equal a fully tested, bug free program. Emma really only checks that all lines are tested. What would happen if some important factors were left out in an if statement? Or what if the system utilized a function that deleted something when it was supposed to only view it? All this would still get 100% code coverage if there were basic unit tests, yet it would still be buggy. I realize that it is not enough to rely on code coverage tools like Emma. Still, I think it is definitely a worthwhile tool to use on projects. It works quickly, efficiently and can improve the robustness of your code.

Bug Problems with 100% Code Coverage

Test coverage has many limitations in the programming world. When I first heard of tools like Emma I really thought using them would make my programs totally robust and bug-less. However after reading articles like this, and working on getting 100% code coverage with my Stack project, I am starting to think otherwise. Would it be possible to introduce a full fledged bug into code that gets full code coverage?

INTRODUCING A BUG

My programming partner Daniel Arakaki and I decided to create a version of our previous stack system that would have 100% coverage in Emma using the existing unit tests, as well as a significant bug that could ruin the system if it were ever put into real use. After contemplating it for awhile, we came up with a plan to introduce a bug by changing a basic function call in the getTop() function of the ClearStack class.

by simply changing this method:

  public Object getTop() throws EmptyStackException {
  return this.top();
}

to:

  public Object getTop() throws EmptyStackException {
   return this.pop();
 }

we effectively introduced a bug into the system that would (using our current test cases) get 100% code coverage using Emma. We ran it through JUnit and Emma and sure enough it all worked fine. Final stack system with introduced bug: Download.

Imagine if this stack system was put into any sort of real use. There would be EmptyStackExceptions all over the place and if you ever did want to see what object was at the top of the stack, you would effectively screw up the stack structure. Any calls to pop after that would return a different object than expected. 

The funny thing about it is that all this could potentially happen just because of one wrong letter in the method (p instead of t). To put it in perspective, imagine if this bugged stack system was implemented on a vending machine. If there was a button to check item availability and a user pressed it, they would get a free item. In the long run the numbers wouldn't add up for the company supplying the vending machine and money would be lost. If a programmer bases all their tests on code coverage results, their code will be flawed. 

CONCLUSION

I think code coverage tools like Emma are very useful for programming. Code testing and optimization becomes more efficient, and it can at least help one to test their code better. The thing is, you cannot rely on them for all your testing needs. It was very easy to introduce a significant bug into the stack system and still get full coverage. It was only when I added another test case that checked for multiple getTop() calls that the problem was found. I, for one would not have though about this potentially being a problem in the original stack program. I guess it comes down to how thorough you want to test your code, and this cannot be fully based on code coverage. It is only a tool to help you with it.

In addition, another downfall of code coverage tools like Emma is that they cannot account for lines of code that should have been written, which means code that has 100% coverage could still have bugs. Overall though, I still think code coverage tools should be used to help in testing. They are quick and efficient and can really improve testing. I will definitely will use Emma as I write tests to my programs knowing all of this.

Using Ant/QA Tools to Build a Robust Program Package

As programmers there are many tools out there that streamline the writing process and optimize code. Some tools create all in one programming environments to work in (IDE's), some tools help you to build projects, and others do all sort of cool things that clean-up and optimize code. Previous to taking ICS413 I was never exposed to these technologies (apart from IDE's). I never knew the power these sorts of things hold in a programming environment. 

TASK 1

The main goals of this exercise were to 1) become familiar with the ANT build system, 2) become familiar with Open Source Java automated quality assurance tools, and 3) learn more about the build system and packaging standards for our ICS413 class. The first task was to simply download and install the specified Java tools on our system:

• Checkstyle 5 Beta - checks source code mainly for layout issues based on a coding standard.
• PMD 4.2.3 - Checks source code for things like bugs, dead code, overcomplicated expressions, etc.
• JUnit 4.5 - A framework for writing and running automated tests.
• FindBugs 1.3.5 - Uses static analysis of byte code to look for possible bugs.
• Ant 1.7.1 - A Java based build tool. 'Make, but without Makes wrinkles'

In addition we were to download the source stack package to test these tools on. It turns out that this task was one of the more challenging parts of the assignment. After I downloaded each tool, I had to go and edit some environment variables on my machine so they would point to the respective tool. Since I use a mac I had to edit my .profile file to add some environment variables like this:

export ANT_HOME="/Users/tylerwolff/Documents/ICS/Tools/apache-ant-1.7.1"
export CHECKSTYLE_HOME="/Users/tylerwolff/Documents/ICS/Tools/checkstyle-5.0-be$
export FINDBUGS_HOME="/Users/tylerwolff/Documents/ICS/Tools/findbugs-1.3.5"
export JUNIT_HOME="/Users/tylerwolff/Documents/ICS/Tools/junit4.5"
export PMD_HOME="/Users/tylerwolff/Documents/ICS/Tools/pmd-4.2.3"

Luckily a friend in class helped me get to this point. Otherwise I would have been at a loss at what to do. I also needed to change the path to ANT 1.7.1. After I finally managed to finish this all, I ran some tests to see if I had installed everything correctly and it worked just right.

TASK 2

The default stack project was already set up with all the xml files needed for the ANT build system. All I had to do was go in and rename some of the variables to add my name to the project . The most complicated thing I had to do in copying the project was to go into the verify.build.xml file to remove the statements referring to the 'emma' tool which is not in use yet. I found the ANT xml files to be extremely easy to understand and decode. Its almost like an upgraded MAKE system. I guess thats why they themselves call it 'Make, but without Make's wrinkles.'

TASK 3

After copying the project, I worked on and finished fixing the problems associated with it during class, with the help of Daniel Arakaki. We used each tool, found the bugs or errors associated with the project, and continued on to fix the problems within the source code. All in all, I have to say it was a very streamlined process using the QA tools. They run instantaneously to produce a very readable page of errors. Although I was not surprised to see errors arise after invoking each tool, I was surprised at how easy it was to understand what the errors actually were. Many of the tools, like PMD and FindBugs even go a step further by linking the errors to full descriptions with examples. This made it really easy to fix some of the errors. For instance in the Stack.java file PMD suggested that instead of using implementation types like ArrayList, use an interface. I was a bit confused by this at first so I clicked the error link and it gave me a concrete example of what they mean and how to implement it. It's a great system!

The rest of the errors were checked and now the project verifies perfectly.

Finalized stack system: stack-tylerwolff-6.0.923.zip 

Overall I feel comfortable using these programming tools now. I don't think I will program anything from now on without them. ANT especially makes it really easy to compile and build entire systems. I like how you can use it to invoke quality assurance tools as well. Its so easy to go through and use these automated tools to better your program. I really think its a good thing to do. I love that fact that you can even learn from using these tools as they show you new errors and changes to make to them.

Thoughts on Automated vs. Manual Quality Assurance

When programming it is always hard to get the code looking and working right the first time around. More often than not, we miss the littlest, most minute details in our program code and it fails because of it. Those are the times when it is ultimately useful to get an objective, outside view of your code. 

In addition to having another programmer look and check for errors in your code, there are many automated tools that scan your code for possible errors. Some of these tools like Checkstyle and PMD scan your source code for things like formatting and code efficiency errors. Other tools like FindBugs scan actual byte code for bugs. Using these sort of tools can make your code optimal and up to standard. But how do they compare to actual human quality assurance checks?

I ran Checkstyle, PMD, and FindBugs on my CodeRuler solution to see what sort of errors were found, and compared these results to what two of my other classmates found in my code. Here's what the automated tools found:

• Checkstyle : 24 errors in MyRuler.java (67 errors total including IBM classes)

1.         20 errors due to lines over 100 characters long
   
2.         3 Javadoc errors
   
3.         1 error due to using a import wildcard statement '*'
   

• PMD: 10 errors in MyRuler.java (36 errors total including IBM classes)

1.         All priority 3 errors with code suggestions
   

• FindBugs: 0 errors in MyRuler.java

As this was my first time using these automated tools I was surprised at how easy and simple they are to use. Instantaneously errors are found and presented in an extremely efficient way. Checkstyle proved to find many formatting errors and described them all well. With PMD I was surprised to find that they not only show the errors but links to full descriptions with examples. How great is that! FindBugs found no errors but I'm sure if the code was more substantial and complicated it would have.

CONCLUSION

Compared to Manual QA checks, I think the automated tools did an amazing job (at least with my CodeRuler classes). When Arthur Shum and Aric West did a check on my code they found many of the same problems - wildcard import statement, long lines of code, javadoc problems, etc. However, what they were able to find were more complex formatting problems such as redundant inline-comments, confusing code segments, and bracket placement. A lot of these problems would essentially make the code even more readable and Checkstyle did not find them. I think a Manual QA check is essential if your code needs to adhere to very specific standards. 

PMD found some errors that I think would take another programmer awhile to point out, especially with long code. It is possible that a programmer could find all these same errors. However, PMD finds them instantly and presents them so well. I actually learned a thing or two from looking at the errors in my code. The errors gave great suggestions into making the code more efficient and readable. 

Overall, what I think makes Automated QA so great is that is instant. These tools can be acquired and used in a matter of minutes. What's more is that using them can even teach you new tricks. If there are still problems in your code, such as logical errors, then manual checks can be done. From what I've seen by doing this, using a combination of both automated and manual quality assurance checks can make your code optimum.

Code Ruler Peer Review [Flestado]

When I first opened the MyRuler.java file from Mari-Lee Flestado I did a quick scan over all the code to get an overall impression of it. At first glance I found the code to be clear and organized. I saw lots of clear, logical statements and many single line comments. It took slightly longer to realize the strategy the code implemented but overall I had a really easy time understanding the specifics of the program. Her ruler worked very well for what it was when I tested it out. It implemented exactly what her strategy specifics outlined. When I first started coding out my codeRuler I thought of some of the same strategies she used like creating more knights and peasants when the opposer has more. 

The javadocs throughout the code were there but weren't too specific. The single method added had a simple overall description that outlined what the code basically did, although it was not formatted correctly.

The overall structure of the class was done quite well. If I were to code something out like this I would hope to have it organized in this same manner. Its clear, concise, and easy to read. The logic of it all was simple and the strategies implemented were clearly done.

After scouring the code for any sort of violations I found a few that were not up to the standards outlined in The Elements of Java Style (many violations occurring in the example IBM code borrowed). Here is a table of my findings.

CODE/FORMATTING VIOLATIONS FOUND

File	Lines	Violation	Comments
MyRuler.java	44, 53, 57, *	EJS-7	no whitespace included in line between () and {
MyRuler.java	38, 79, 87, *	EJS-7	no blank line between logical sections of code
MyRuler.java	1,2,3, *	EJS-8	Tabs used, not converted to whitespace
MyRuler.java	109	EJS-9	'np' is not a meaningful name
MyRuler.java	38, 43, 52, *	EJS-37	comments repeat code, does not add useful information
MyRuler.java	105, 110, 120, *	EJS-76	Expression statements used instead of block statements
MyRuler.java	20, 98	ICS-SE-Java-6	Does not start with a capital and end with period. Should be a more complete sentence.

Overall, I found Mari-Lee's code to be very clean and easy to read. With a little bit of time editing, her code will be looking top-notch and up to standards.

Lessons learned from Code Ruler

CodeRuler is an application provided by IBM <LINK> where one is challenged to create the best possible ruler using Java programming. The game premise is simple: you must battle other rulers for land and dominance. For each game you start off with one castle, ten knights, and ten peasants. You get points for claiming land and capturing other ruler's things. 

Team Members: 

Tyler Wolff

Daniel Tian

For this project I worked with Daniel Tian. He and I consulted a lot about what our strategy was going to be and it was ever evolving. We started with the basic code given on the IBM site and initially decided to optimize some basic things: 

1. Command our castles to create more knights when knight count is low
2. Program our peasants to capture land more effectively
3. Program our knights to go after the closest enemy (other knight/castle) immediately in a large group. After all are gone, go after other peasants.

What we eventually created did in fact do all these things and more. However, we basically threw out all of the start up code from the IBM site and put in our own codes. We programmed out very efficient peasants who are constantly claiming new lands. We programmed intelligent castles to create what's needed most. Probably the most efficient thing we programmed was the knights. What the knights do is basically gang up on the nearest enemy, be it another enemy knight or castle. Normally it can take them all out. Then it goes for the left over peasants. More details can be found in the code javadoc. The source code and javadocs for our implementation can be found here [tylerwolff-daniel.f.tian.zip].

Evaluation Results

Opponent Name	Opponent Score	twolff-daniel.f.tian score	Winner
Split Up Ruler	70	788	twolff-daniel.f.tian
split Up Ruler	86	797	twolff-daniel.f.tian
Split Up Ruler	58	801	twolff-daniel.f.tian
Migrate Ruler	0	864	twolff-daniel.f.tian
Migrate Ruler	0	814	twolff-daniel.f.tian
Migrate Ruler	0	639	twolff-daniel.f.tian
Gang Up Ruler	84	712	twolff-daniel.f.tian
Gang Up Ruler	133	739	twolff-daniel.f.tian
Gang Up Ruler	60	809	twolff-daniel.f.tian

As shown above, our ruler does super good against all of the built in sample rulers. It will always beat them in a one on one game. The hardest sample ruler, 'smart split up ruler' will always loses to our ruler. Even in a 6 player free for all match our ruler wins 70% of the time. What I found to be the hardest part in winning was making the 0.5 second time restraint per turn. If at any moment a turn took longer than that, our guys would freeze up due to a disqualification. I figured this had to do a little with my computer speed (2GHz intel core duo) since it almost always ran fine on Daniel's computer. When this didn't happen, it almost always ran smoothly and efficient.

Lessons Learned

Overall, working on this CodeRuler project was very new and insightful for me. On the one hand it dealt with gaming strategies, something I've never really been exposed to much. I tried my best to think like a gamer but I missed a lot of little things here and there. I was so grateful to have Daniel on my team because he is really interested in game development and he always had clever ideas for improving our ruler. I liked how we would push each other to make better and better code. I have a real good idea of what the best strategies are for codeRuler now. I could probably apply some of the concepts to other games.

Another thing that was new to me was working in a group for a programming project. This is the first time I've programmed something with another person and it was a lot different. Initially it was hard to coordinate times to meet up to work on the project since we both are very busy outside of school. However, we found that we could almost always reach the other somehow, either by phone, email, or IM. A hard thing too was trying to sync up our ideas, even if they were similar. In coding out what the peasants do, we were always throwing back ideas of how to change things to make it better and it was quite hard to come up with a final solution that worked for both of us. It was great working in a group though because it really took what I would have done to a whole other level. Our ideas were more refined working together and our code runs really well.

Using Eclipse for this project was a dream once again. It was my first time using it to create javadocs and it was so simple to do so! That really is a useful thing that can make it easier for anyone editing or adding to it. It was nice to have the application right there next to the raw code too. It made it incredibly easy to edit and test things. Doing this exercise also allowed me to get some experience in using collections. I never really messed around with it before and I really like how simple it can make things.