2.3.1.1. Test Cases and Suites¶

OK, if the goal is to execute a program (or part of one) with the intent of finding an error, then let’s run it! But what input value(s) or user interaction sequence(s) should we use?

As an example, suppose you are testing something very simple, like a single function called isEven() that takes an integer value and returns true if the integer is even, or false if it is odd.

Remember that just running the program doesn’t really count as testing; instead, we must be trying to demonstrate failures of some kind. That means carefully planning out exactly how we are going to run the program–what input values or situations we will use, and what specific behavior or result we will be looking for.

A test case is a single scenario under which the software will be run to attempt to demonstrate a failure, and typically has three basic parts.

1. First, a test case must define all of the input values, conditions, or variables used in the test. For example, we can imagine a scenario where we wish to test isEven() on the input value 2.

2. Second, a test case must define a procedure for exercising the software under test. In the case of isEven(), the procedure can be very simple: call the method with the given input value, and store the boolean result. If we were testing a larger software part or an entire application, however, we may need to perform a longer, more complicated series of actions to exercise it, and this procedure is part of the test case.

3. Third, a test case must define the expected behavior that should result when the software is exercised in this way. For example, if we call isEven() with a value of 2, we would expect it to produce the result true.

Further, a test case is worked out (and written down) before the test is carried out. That is, we figure out what we will do and what result we expect it to produce before running the test. That differentiates testing from simply “running the code to see what happens”.

Of course, it will probably take more than one test case to determine that some required behavior is fully satisfied. For example, an implementation of isEven() that always returns true, regardless of the input value, will pass the example test case described above. As a result, programmers typically design collections of test cases that all work together. A collection of test cases designed for exercising the same piece of software is called a test suite.

For example, one might design five test cases for isEven(): one test for a positive even integer, one for a positive odd integer, one for a negative even integer, one for a negative odd integer, and one for zero. Another person might come up with a different test suite for this method, but this group would have a good chance of successfully demonstrating most of the simple bugs that could occur in the method in question.

2.3.1.2. Good Test Cases¶

Even when testing a single method like isEven(), there are a huge number of test cases to choose from. For 32-bit integers, there are 2³² possible input values we could try. So why would we pick any one of these values over another to use in a test case? What makes a test case a “good” test case?

According to Glenford Myers, a good test case is one that:

• Has a high probability of finding an error. Since our goal is to demonstrate faults, we should pick test cases that we think have the greatest chance of succeeding.

• Is not redundant. If we already have a test case for the value 2, we don’t need another one. Further, why not use some other integer, like 147652? Well, is there any reason to think this test case will have a high probability of finding an error? If we already have other test cases that cover even numbers, why test this number as well? If it simply retests behavior that has already been adequately demonstrated, then it is redundant–it exercises behavior already tested by other test cases in our test suite.

• Is “best of breed”. In other words, if there are several redundant test cases you might consider that all cover the same behavior, pick the test case that has the highest probability of finding errors. For example, if we want to test isEven() on an even negative integer, which value should we pick? Will any random value do? In this case, it might be better to pick the largest negative integer (-2³¹), since it will also check that the method works on an extreme boundary of the integer range.

• Is neither too simple nor too complex. Suppose you are testing a group of methods on a class, rather than just focusing on isEven(). You could have a test case that called every single method in the class in some order (and maybe call some methods more than once!). Such a test case might cover a lot of ground–all of the methods would have to work correctly on at least some test values in order for this test case to pass. However, if the test reveals a defect, where is it? It could be in any method, almost anywhere. This is an example of a test case that is too complex, since it involves too many behaviors all in the same series of actions. It would be better to use many smaller test cases, each focused on examining a single behavior, so that the test results would be more meaningful. Similarly, a test case that simply constructed the object and did nothing else might be too simple to reveal any defects. Strive for narrowly focused test cases that are still likely to reveal defects in the behavior they are exercising.

2.3.1.3. Software Testing Approaches¶

Clearly, exhaustive testing, where every possible execution sequence is tested, is impractical, even for very simple pieces of code like isEven(). There are just too many possibilities to try, and it takes too long. As a result, we need a way to select an appropriate set of test cases (a test suite) that has a high likelihood of revealing the most probable defects. Many different testing methods have been developed for devising test cases, all with the goal of helping you pick an effective test suite.

The two biggest groups of testing approaches are black-box testing approaches and white-box testing approaches, although others exist as well.

Testing activitives are also characterized according to the nature of the units being tested, and the focus of what you are trying to verify. Some of the most common testing activities are unit testing, integration testing, and system testing.

Black-box testing or functional testing is a term used for a family of testing approaches where the test cases are derived from a specification, an interface definition, or from a behavioral description. In using this strategy, the tester views the software under test as a “black box” whose behavior can only be determined by studying its inputs and the related outputs. For students, this is the approach you use when you think about writing tests using an assignment description.

In contrast, another family of testing approaches is called white-box testing, structural testing, or logic-driven testing. White-box testing approaches require the tester to examine the internal structure of the software under test. In using such a strategy, the tester derives test cases after reviewing the program’s internal logic and structure.

Unit testing is used to describe activities where one is testing a single software “unit” in isolation, independently of any of the other code being written. This is usually the technique you use when you are starting out, writing tests for one class at a time.

Integration Testing: Once individual program components or classes (units) have been tested, they must be integrated to create a partial or complete system. The integration process involves testing combinations of units working together for problems that come up with component interaction. The combination being tested may start out as small as two units interacting, and grow to be the full application. Usually, this is the time when pieces of a full system written by different developers are being tested together.

System testing is where an entire, fully integrated program is tested with respect to its typical working environment. This involves combining the code from all the developers into the full, complete software product.

2.3.1.4. Unit Testing in More Detail¶

Unit testing is used to describe activities where one is testing a single software “unit” in isolation, independently of any of the other code being written.

Exactly what unit means can vary from one programming language to another, from one programming paradigm to another, and from one organization to another, but is intended to mean a cleanly delimited, identifiable piece of software that can be executed independently. In Pascal, a “unit” is typically a procedure or a function. In object-oriented languages, a “unit” is often a single class, although sometimes it can be a single method. Unless otherwise specified, from here on out we will interpret “unit” to mean a single class in an object-oriented language, unit under test (UUT) to mean the class we are currently testing or writing tests for.

Unit testing is usually carried out by the programmer or programmers that wrote the unit under test (UUT), before the unit is combined with other pieces of software to form a larger application. The goal is to confirm to the greatest extent possible that the unit has no errors of its own before combining it with other software. That is because the smaller the piece of software you are testing, the easier it is to locate and remove defects that are revealed.

For students, it turns out that most of the testing you do on your own assignments is unit testing: testing individual classes in isolation to ensure they meet their individual design requirements.

2.3.1.5. The Benefits of Software Testing¶

Let us consider some of the benefits of software testing. This should give us a perspective for why we want to employ software testing in our assignments:

• It increases your confidence in your code

• It increases your understanding of the requirements

• It Preempts “big bang” integration problems

• It improves your grades

2.3.1.6. Test Driven Developement (TDD)¶

Test-driven development (TDD) is a programming technique that involves constantly alternating between writing one or more small test cases, and then writing a small increment of code, so that you can gradually build up a working code base one piece at a time. There are three principal ideas behind TDD:

• Test first. That is, each time you are about to write some part of your solution, first write down the test cases that are necessary to confirm your solution works the way you want, and only then write the code. For this reason, TDD is also known as test-first coding.

• Write in tiny increments. Rather than writing large chunks of code at once, you should add new code in “baby steps”: one small method, or one small piece of a method at a time, writing a new test case or two for each small bit.

  For example, the isEven() discussed earlier is small enough that you could write test cases for it, then write the method body in one step (just one line of code is needed), and finally run your test cases and debug as necessary.

  However, if you were writing a method with more complex behavior, it can be far too complex to write in one step. Consider a method that takes three numbers representing the lengths of the three sides to a triangle, where the method is supposed to return whether the corresponding triangle is equilateral, isosceles, or scalene, while reporting that the numbers do not form a triangle if no triangle exists with those side lengths.

  If you are writing tests for this method, you will need to check many distinct types of situations. Does it handle zero-length sides? Does it handle negative numbers? Does it handle lengths that do not form any triangle? What about isosceles? Equilateral? Scalene? You can separate each of these “cases” or branches in your solution’s logic into a separate small increment. Write the test cases you want to have for zero length sides. Then just implement that part of the method and run your tests. Next, add test cases for negative numbers, add just the extra code for this case, and re-run all your tests. Proceed through the remaining cases one step at a time, writing a few tests and then writing the code that implements the corresponding behavior.

  Using case analysis–that is, breaking a problem down into two or more subproblems, and defining the conditions under which one or another of these subproblems applies–is a powerful problem-solving tool. It comes up all the time in computer science, and provides a neat way to break apart complicated methods into smaller steps that can be incrementally tested.

• “When the bar is green, the code is clean.” This maxim of TDD characterizes the third key idea: each time you add a small piece of code, you re-run all of the tests you have for the unit under development, and you do not move on to the next step until all of your tests pass 100%.

  In short, add a small number of tests first, then add the corresponding (small) piece of code, run all your tests, and debug any problems immediately. You’re never ready to move to the next coding step (or finished with your solution, or ready to contribute your code to an open source project, etc.) unless all your tests pass. Your test cases are your expression of what “correct behavior” is for your code. Thus, these test cases are your yardstick for measuring your success, and as you incrementally grow your test suite, you can see how close you are to completing all the required behavior.

  Most testing tools for automatically running your test cases will show you a progress bar as the tests run, and color it green as long as tests are successful, and red when any tests fail.

A successful test in traditional testing finds one or more defects. But in TDD, when a test fails you have made progress because you now know that you need to resolve the problem. TDD increases your confidence that your system actually meets the requirements defined for it and that your system actually works. It is said that you should “test with a purpose” and know why you are testing something and to what extent it needs to be tested. Also with TDD, when you achieve a well tested program then every single line of code is tested. In general, this is something that traditional testing does recommend, but does not guarantee.

2.3.1.7. Learn More About TDD¶

TDD is all about writing “clean code that works.” Here are some quotes (from Kent Beck) that intuitively describe TDD:

The style here is to write a few lines of code, then a test that should run, or even better, to write a test that won't run, then write the code that will make it run. ... [After figuring out how to write one small piece of code ...] Now, instead of just coding on, we want to get immediate feedback and practice "code a little, test a little, code a little, test a little." [... So we immediately write a test for it.]

TDD grew out of Extreme Programming and evolved from the simple XP idea of “build a little, test a little” during coding. Basically, your code always has a complete set of tests that exercises its capabilities, and you write new tests as you add to your code.

A significant advantage provided by TDD is that it enables you (encourages you!) to take small steps when writing software. For instance, suppose you add a small piece of new code, compile, and test it. Sooner or later when you do this, one or more of your tests will fail because of one or more defects in your code. However, by proceeding in small steps, it is much easier to find and fix those defects. The problem is most likely in the tiny bit of code you just wrote, since all the other code passes all the other tests you’d previously written. If some previously working behavior breaks, again it is probably a result of interference caused by the newly added code. And bugs are so much easier to find if you have only written five new lines of code rather than five hundred, or five thousand.

2.3.5.1. Passing Information using Parameters¶

Some of the methods we have used require arguments, which are the values you provide when you invoke the method. For example, a Jeroo has two methods with the name ‘hop’. Calling hop() will cause the jeroo to hop one space ahead. However if you specify a number inside the parentheses like this: hop(4), the jeroo will hop four spaces ahead.

When you use a method, you provide the arguments. When you write a method, you name the parameters. The parameter list indicates what arguments are required.

For example:

public void turnAndDisable(RelativeDirection direction)
{
    this.turn(direction);
    this.toss();
}

To invoke this method, we have to provide a relative direction as an argument:

turnAndDisable(RIGHT);

This will cause the jeroo to turn right and disable a net.

2.3.5.2. Using Multiple Parameters¶

Here is an example of a method that takes two parameters:

public void turnThenHop(RelativeDirection direction, int hops)
{
    this.turn(direction);
    this.hop(numHops);
}

To invoke this method, we have to provide an integer and a relative direction as arguments:

turnThenHop(RIGHT, 7);

This would cause the jeroo to turn right and then hop seven times.

2.3.7.1. Logical NOT and the If-Else Structure¶

One thing to consider is that when writing if/else statements, starting with a ! usually makes code harder to read.

if (!this.isClear(AHEAD))
{
    this.toss();
}
else
{
    this.hop();
}

It’s easy to miss the ! above and misread what this conditional does. Instead, it’s preferable to phrase the same condition like this:

if (this.isClear(AHEAD))
{
    this.hop();
}
else
{
    this.toss();
}

You can see that logically these two if-then-else structures achieve the same thing, but one is easier to read.

2.3.7.2. Too Many Conditionals¶

Another thing to keep in mind is writing too many conditions. When solving a complex problem it can be tempting to just keep adding new conditions for every new scenario you find yourself in. However, this is both harder to read and can introduce bugs into your code that could be hard to find later.

Take for example:

if (this.isClear(AHEAD))
{
    this.hop();
}
else if (!this.isClear(AHEAD))
{
    this.toss();
}
else
{
    this.turn(RIGHT);
}

Logically, the if and else-if branch of this conditional do the same things as we saw above. However, there is a third branch here that will never execute. This is because the area ahead of the jeroo will always be either clear or not clear. The code will always find a branch to execute between the first two choices, and there is never any situation where the else branch will ever be applicable.

In computer terms, code that you write that can never be executed under any possible circumstances, is called unreachable code. Such code is usually a programming problem, since the reason it can never be executed is often due to improperly constructed programming logic, as in the example here. The first two branches cover all possible situations, so the third option is useless.

If you’re not entirely sure if two boolean statements are equivalent, it can be helpful to write out a truth table. For example, we can see below that writing b and !!b are equivalent.

Whatever value b has, we can see that !!b matches it!

2.3.7.3. Empty Condition Branches¶

It is also good practice not to leave empty conditions in your code.

if (this.isClear(AHEAD))
{
    // do nothing
}
else
{
    this.turn(RIGHT);
}

It is always preferred to have just one if statement rather than an empty if-else.

if (this.isClear(AHEAD))
{
    // do nothing
}
else
{
    this.turn(RIGHT);
}

Here, it would be preferred to use the ! operator rather than to have empty conditions:

if (!this.isClear(AHEAD))
{
    this.turn(RIGHT);
}

2.3.7.4. Many Conditions vs Compound Conditions¶

Taking a look at the following code snippet:

if (this.isClear(AHEAD))
{
    if (this.seesNet(RIGHT))
    {
        this.turn(RIGHT);
    }
}

Here we see one condition nested within another. It is generally preferable to instead write the same condition like this:

if (this.isClear(AHEAD) && this.seesNet(RIGHT))
{
    this.turn(RIGHT);
}

2.3.7.5. More Complex Conditionals¶

Sometimes you want to check related conditions and choose one of several actions. One way to do this is our cascaded if structure:

if (molly.isFacing(NORTH))
{
    molly.hop();
}
else if (molly.isFacing(SOUTH))
{
    molly.hop(2);
}
else if (molly.isFacing(EAST))
{
    molly.hop(3);
}
else
{
    molly.hop();
    molly.toss();
}

These chains can be as long as you want, although they can be difficult to read if they get out of hand.

You can also make complex decisions by nesting one conditional statement inside another. We could have written the previous example as:

if (molly.isFacing(WEST))
{
    molly.hop();
    molly.toss();
}
else
{
    if (molly.isFacing(NORTH))
    {
        molly.hop();
    }
    else if (molly.isFacing(SOUTH))
    {
        molly.hop(2);
    }
    else
    {
        molly.hop(3);
    }
}

The outer conditional has two branches. The first branch tells the jeroo to hop once and toss, and the second branch contains another conditional statement, which has three branches of its own.

These kinds of structures are common, but they get difficult to read very quickly. Good indentation is essential to make the structure (or intended structure) apparent to the reader.

2.3.10.1. Relational Operators with Primitive Data Types¶

Relational operators are used to check conditions like whether two values are equal, or whether one is greater than the other. These kinds of operators do not work on objects at all, so you cannot use them on jeroos. However, they work on numeric values such as ints. The following expressions show how they are used.

The result of a relational operator is one of the two Boolean values: true or false. These values belong to the data type boolean; in fact, they are the only boolean values.

You are probably familiar with these operations, but notice that the Java operators compare program values. They behave similar to the mathematical operators you are familiar with, but are not written the same way as mathematical symbols like =, ≤, and ≠.

A common error is to use a single = instead of a double == when you wish to compare two values. Remember that = is the assignment operator, and == is a comparison operator. Also, writing =< or => by accident will produce a compiler error. The equals sign always comes after the < or >, just like when you say the names of those comparisons in English: “less than or equal” has the less than symbol first, followed by the equal sign second.