Bug Problems with 100% Code Coverage

Test coverage has many limitations in the programming world. When I first heard of tools like Emma I really thought using them would make my programs totally robust and bug-less. However after reading articles like this, and working on getting 100% code coverage with my Stack project, I am starting to think otherwise. Would it be possible to introduce a full fledged bug into code that gets full code coverage?

INTRODUCING A BUG

My programming partner Daniel Arakaki and I decided to create a version of our previous stack system that would have 100% coverage in Emma using the existing unit tests, as well as a significant bug that could ruin the system if it were ever put into real use. After contemplating it for awhile, we came up with a plan to introduce a bug by changing a basic function call in the getTop() function of the ClearStack class.

by simply changing this method:

  public Object getTop() throws EmptyStackException {
  return this.top();
}

to:

  public Object getTop() throws EmptyStackException {
   return this.pop();
 }

we effectively introduced a bug into the system that would (using our current test cases) get 100% code coverage using Emma. We ran it through JUnit and Emma and sure enough it all worked fine. Final stack system with introduced bug: Download.

Imagine if this stack system was put into any sort of real use. There would be EmptyStackExceptions all over the place and if you ever did want to see what object was at the top of the stack, you would effectively screw up the stack structure. Any calls to pop after that would return a different object than expected. 

The funny thing about it is that all this could potentially happen just because of one wrong letter in the method (p instead of t). To put it in perspective, imagine if this bugged stack system was implemented on a vending machine. If there was a button to check item availability and a user pressed it, they would get a free item. In the long run the numbers wouldn't add up for the company supplying the vending machine and money would be lost. If a programmer bases all their tests on code coverage results, their code will be flawed. 

CONCLUSION

I think code coverage tools like Emma are very useful for programming. Code testing and optimization becomes more efficient, and it can at least help one to test their code better. The thing is, you cannot rely on them for all your testing needs. It was very easy to introduce a significant bug into the stack system and still get full coverage. It was only when I added another test case that checked for multiple getTop() calls that the problem was found. I, for one would not have though about this potentially being a problem in the original stack program. I guess it comes down to how thorough you want to test your code, and this cannot be fully based on code coverage. It is only a tool to help you with it.

In addition, another downfall of code coverage tools like Emma is that they cannot account for lines of code that should have been written, which means code that has 100% coverage could still have bugs. Overall though, I still think code coverage tools should be used to help in testing. They are quick and efficient and can really improve testing. I will definitely will use Emma as I write tests to my programs knowing all of this.

Using Ant/QA Tools to Build a Robust Program Package

As programmers there are many tools out there that streamline the writing process and optimize code. Some tools create all in one programming environments to work in (IDE's), some tools help you to build projects, and others do all sort of cool things that clean-up and optimize code. Previous to taking ICS413 I was never exposed to these technologies (apart from IDE's). I never knew the power these sorts of things hold in a programming environment. 

TASK 1

The main goals of this exercise were to 1) become familiar with the ANT build system, 2) become familiar with Open Source Java automated quality assurance tools, and 3) learn more about the build system and packaging standards for our ICS413 class. The first task was to simply download and install the specified Java tools on our system:

• Checkstyle 5 Beta - checks source code mainly for layout issues based on a coding standard.
• PMD 4.2.3 - Checks source code for things like bugs, dead code, overcomplicated expressions, etc.
• JUnit 4.5 - A framework for writing and running automated tests.
• FindBugs 1.3.5 - Uses static analysis of byte code to look for possible bugs.
• Ant 1.7.1 - A Java based build tool. 'Make, but without Makes wrinkles'

In addition we were to download the source stack package to test these tools on. It turns out that this task was one of the more challenging parts of the assignment. After I downloaded each tool, I had to go and edit some environment variables on my machine so they would point to the respective tool. Since I use a mac I had to edit my .profile file to add some environment variables like this:

export ANT_HOME="/Users/tylerwolff/Documents/ICS/Tools/apache-ant-1.7.1"
export CHECKSTYLE_HOME="/Users/tylerwolff/Documents/ICS/Tools/checkstyle-5.0-be$
export FINDBUGS_HOME="/Users/tylerwolff/Documents/ICS/Tools/findbugs-1.3.5"
export JUNIT_HOME="/Users/tylerwolff/Documents/ICS/Tools/junit4.5"
export PMD_HOME="/Users/tylerwolff/Documents/ICS/Tools/pmd-4.2.3"

Luckily a friend in class helped me get to this point. Otherwise I would have been at a loss at what to do. I also needed to change the path to ANT 1.7.1. After I finally managed to finish this all, I ran some tests to see if I had installed everything correctly and it worked just right.

TASK 2

The default stack project was already set up with all the xml files needed for the ANT build system. All I had to do was go in and rename some of the variables to add my name to the project . The most complicated thing I had to do in copying the project was to go into the verify.build.xml file to remove the statements referring to the 'emma' tool which is not in use yet. I found the ANT xml files to be extremely easy to understand and decode. Its almost like an upgraded MAKE system. I guess thats why they themselves call it 'Make, but without Make's wrinkles.'

TASK 3

After copying the project, I worked on and finished fixing the problems associated with it during class, with the help of Daniel Arakaki. We used each tool, found the bugs or errors associated with the project, and continued on to fix the problems within the source code. All in all, I have to say it was a very streamlined process using the QA tools. They run instantaneously to produce a very readable page of errors. Although I was not surprised to see errors arise after invoking each tool, I was surprised at how easy it was to understand what the errors actually were. Many of the tools, like PMD and FindBugs even go a step further by linking the errors to full descriptions with examples. This made it really easy to fix some of the errors. For instance in the Stack.java file PMD suggested that instead of using implementation types like ArrayList, use an interface. I was a bit confused by this at first so I clicked the error link and it gave me a concrete example of what they mean and how to implement it. It's a great system!

The rest of the errors were checked and now the project verifies perfectly.

Finalized stack system: stack-tylerwolff-6.0.923.zip 

Overall I feel comfortable using these programming tools now. I don't think I will program anything from now on without them. ANT especially makes it really easy to compile and build entire systems. I like how you can use it to invoke quality assurance tools as well. Its so easy to go through and use these automated tools to better your program. I really think its a good thing to do. I love that fact that you can even learn from using these tools as they show you new errors and changes to make to them.

FizzBuzz Development in Eclipse

TASK & PURPOSE:
The task was to implement the FizzBuzz program: 

"A program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”."

For me this seemingly simple program was more than just an easy assignment, it was an exercise in using Java/Eclipse and an introduction into using JUnit. I decided to go about creating FizzBuzz using a test-driven development style, where the program is created through a series of small test case iterations. For each case, the code necessary to pass the test is implemented, and after all cases tested pass, the program becomes correct. This type of development style is really different for me as I usually go about things in the opposite way - the creation of code, then testing. Creating the FizzBuzz program was the perfect chance for me to try it out through the use of JUnit.

Using Eclipse again was a dream. I forgot how much it simplifies things and how much more efficient it makes programming on a whole. It took me around 15 minutes from the startup of the program to complete the FizzBuzz program. Although that might sound long for a small program like this, the actual code for FizzBuzz flowed swiftly and smoothly. What took the most time was setting up the test cases in JUnit. Since it was new technology for me, I inevitably ran into some problems. 

PROBLEMS:
The main problem I ran into while doing this was creating the program using the test case development style described earlier. I had a hard time linking the test class up to the FizzBuzz class. It would not recognize my getValue() function and it was confusing me.

SOLUTIONS:
I finally realized the simple solution to my problem was to use 'FizzBuzz.getValue()' instead of just 'getValue()'. This also brought upon another error that was fixed (using hints from eclipse!) by inserting 'import static org.junit.Assert.*;' Eclipse is wonderful.

CONCLUSION
For me doing this program was a nice little refresher into programming again in java. It also was really nice to go in and use the JUnit test case development in Eclipse. Although test-driven development is a bit overkill for a simple program like FizzBuzz, I can see how on a larger scale this development system could be really efficient. It would be easier to solve a large problem by first testing out and solving smaller problems. I like the simplicity of this. It seems like it would lead to less bugs and smoother overall development.

The creation of FizzBuzz has shown me that this class (ICS413) is going to be very practical and fun. I am excited to learn about tools that real software engineers use as well as concepts they use in creating software. I feel more motivated to learn about this sort of thing as it will make me a better programmer. Overall, I am looking forward to the class.

SOURCE CODE:
FizzBuzzTest.java


import static org.junit.Assert.*;
import org.junit.Test;

public class FizzBuzzTest {

@Test
public void testFB() {

assertEquals("Test 1", "1", FizzBuzz.getValue(1));
assertEquals("Test 3", "Fizz", FizzBuzz.getValue(3));
assertEquals("Test 5", "Buzz", FizzBuzz.getValue(5));
assertEquals("Test 15", "FizzBuzz", FizzBuzz.getValue(15));
assertEquals("Test 100", "Buzz", FizzBuzz.getValue(100));

}

}


FizzBuzz.java

/**
* A Simple program that runs through the numbers 1 - 100,
* printing either "Fizz" for multiples of 3, "Buzz" for
* multiples of 5, "FizzBuzz" for multiples of both, or just
* the number itself for all others.
*
* @author Tyler Wolff
*/

public class FizzBuzz {

public static String getValue(int iNum) {

if ((iNum % 3 == 0) && (iNum % 5 == 0)) {
return "FizzBuzz";
} else if (iNum % 3 == 0) {
return "Fizz";
} else if (iNum % 5 == 0) {
return "Buzz";
} else {
return String.valueOf(iNum);
}
}

public static void main(String[] args) {

for (int i = 1; i <= 100; i++) { System.out.println(getValue(i));
}

}

}