Creating A Fortran Unit Testing Framework (part 3): The First Routine

The Form of Our Assertions

We’ll begin by implementing ASRTT. However, it will take us a bit of time to get there. One point that wasn’t made clear in the previous post is whether our assertions are subroutines or subroutine functions. The main difference is that subroutine functions return a value, while subroutines do not. Functions are used to return a single value, while subroutines are return no result. Subroutines are also used when we want to return multiple values. Fortran uses pass by reference semantics.

In our case, the chief difference is one of usage:

1. LOGICAL OK
   OK = ASRTT(.TRUE.)
2. CALL ASRTT(.TRUE.)
3. LOGICAL OK
   CALL ASRTT(.TRUE., OK)

In the first and third versions, the logical variable OK holds the result of the test. In the second version, we ignore the result of the test; the subroutine doesn’t return any value.

The first form seems more natural in terms of the intent of the assertion-—returning .TRUE. or .FALSE. according to the assertion. However, Fortran requires that we always assign the result of the function call to some variable. In most other languages, we can ignore the returned value and write

ASRTT(.TRUE.).

This won’t compile in Fortran; we have to assign the result:

OK = ASSTT(.TRUE.)

The third format removes the need to assign the result. However, it makes the signature of the subroutine more complicated, requiring the user to add a LOGICAL variable to the list of arguments.

The second form is closest to the usage in xUnit frameworks. We generally process the test outside of the main flow (either by updating a data structure or by throwing an exception). Handling the outcome of the test is the work of the framework, and shouldn’t intrude into the user’s code by requiring a dummy variable to accept the return value from a function.

I went back and forth several times, but eventually selected the second form. It is closest to other xUnit frameworks. It has the simplest semantics in using the framework, and so is the least intrusive to the user. Finally, it reflects a proper division of labor by pushing the complication of handling the result of the assertion onto the testing framework where it belongs.

Test-Driven Development

I’m going to take my own advice and write unit tests as we develop the framework. Each aspect of the framework will have one or more tests associated with it, verifying that it works. These tests will be written before each routine is added, a practice called “Test-driven development” (TDD). The idea behind TDD is that writing the tests before the code gives two advantages:

1. We make sure that the code implements only the functionality that is needed. If it isn’t required to make one of the tests pass it doesn’t get written.
2. By forcing us to write code that exercises the routines that we are developing, we create an application programming interface (API) that makes sense from the outside, and therefore is simpler to use. An API is the collection of subroutines and functions that can be called by outside code.
3. We have a suite of tests that allow us to refactor with confidence. Our test suite will need maintenance, just like any other code, and so we are kind to ourselves by writing those tests now.
4. Writing the test at the same time serves as a form of documentation. The tests show how the framework’s API is intended to be used.
5. Writing the tests at the same time as the code, the tests are in sync with the code; we don’t have to recall what we were thinking days (or weeks, or months) later.

The philosophy of TDD goes beyond what we will explore here. There are many who feel that TDD is an inappropriate name. They have rechristened it “Behavior-driven development,” to shift focus from the “tests” to the specification of the “behavior” of the code being written. This difference in focus leads to some interesting differences in outlook and focus. I encourage you to take a look at Dan North’s blog introducing BDD, and the behavior-driven development site (note the spelling "behaviour", rather than the American "behavior"). There are several open source projects dedicated to BDD. The oldest/best known are JBehave and easyB for Java, and RSpec for Ruby are a few of the most active. There is also a book coming out on RSpec (currently available in "beta") that is well-worth reading.

Implementing RESCHR Using TDD

The simplest, most basic piece of functionality that we want is to print out “.” if the test has passed, and “F” if the test has failed. So how do we test that? We break it into 2 pieces:

• A routine that chooses “.” If it receives a .TRUE. argument and “F” if the argument is .FALSE. This second routine will then call the first routine with the correct character.
• A routine that prints out the results of the test

We begin by assuming that we have a function that does what we want: RESCHR (for result character). It takes a LOGICAL and returns the correct character. The final twist is that FORTRAN 66 doesn’t have characters. Instead, we have to encode the character as a Hollerith constant in an integer, using something like

---

J = 1HF

---

We begin by writing a test to verify that expectation.

---

      PROGRAM TCASES

      CALL TSTCHR

      STOP

      END

C

C TEST CASE FOR THE ROUTINE THAT RETURNS THE CORRECT

C CHARACTER TO PRINT OUT

      SUBROUTINE TSTCHR

      INTEGER RESCHR

      INTEGER ACTUAL

      INTEGER EXPECT

      EXPECT = 1H.

      ACTUAL = RESCHR(.TRUE.)

      IF (EXPECT .EQ. ACTUAL) GO TO 100

      WRITE(6, 1000)

      GO TO 300

  100 CONTINUE

      WRITE(6, 2000)

  300 CONTINUE

      EXPECT = 1HF

      ACTUAL = RESCHR(.FALSE.)

      IF (EXPECT .EQ. ACTUAL) GO TO 400

      WRITE(6, 1000)

      GO TO 500

  400 CONTINUE

      WRITE(6, 2000)

  500 CONTINUE

 1000 FORMAT(1HF, $)

 2000 FORMAT(1H., $)

      RETURN

      END

---

Basically, we check that the integer returned matches our expectation. The return value ACTUAL is compared to EXPECT. If they agree, a "." is printed; if they don’t agree "F" is printed. The WRITE statement uses a hardcoded reference to unit 6 (the default output), which is the screen on my laptop. [1].

The next step in TDD is to do the minimum possible to get this to compile and see the test fail. This step seems silly. I know the test is going to fail. However, when we are testing more complex code, I have had tests that should fail pass. It is rare, but always provides extremely valuable feedback; there is something important that I don’t understand, and I need to figure that ASAP. Anyway, the minimal code is:

---

C

C RETURNS THE CHARACTER '.' IF THE ARGUMENT IS .TRUE.

C 'F' OTHERWISE

      INTEGER FUNCTION RESCHR(A)

      RESCHR = 1HX

      RETURN

      END

---

So, I compile and run, and the test does indeed fail; I get

---

FF

---

as output. The next step is to get the test to pass. We change RESCHR:

---

C

C RETURNS THE CHARACTER '.' IF THE ARGUMENT IS .TRUE.

C 'F' OTHERWISE

      INTEGER FUNCTION RESCHR(A)

      LOGICAL A

      IF (.NOT. A) GO TO 100

        RESCHR = 1H.

        GO TO 200

  100 CONTINUE

        RESCHR = 1HF

  200 CONTINUE

      RETURN

      END

---

And we are rewarded with passing tests:

---

..

---

(trust me, it is much more exciting in person…)

The final step is to tighten up the code—refactor. I pull the constants 1HF and 1H. into variables, to make the code more communicative:

---

C

C RETURNS THE CHARACTER '.' IF THE ARGUMENT IS .TRUE.

C 'F' OTHERWISE

      INTEGER FUNCTION RESCHR(A)

      INTEGER PASS, FAIL

      DATA PASS/1H./, FAIL/1HF/

      LOGICAL A

      IF (.NOT. A) GO TO 100

        RESCHR = PASS

        GO TO 200

  100 CONTINUE

        RESCHR = FAIL

  200 CONTINUE

      RETURN

      END

---

I run the tests, and both still pass. The other piece is to tighten up the test code as well. We are going to be running this a lot, so we might as well be nice to ourselves.

---

      PROGRAM TCASES

      CALL TSTCHR()

      STOP

      END

C

C TEST CASE FOR THE ROUTINE THAT RETURNS THE CORRECT

C CHARACTER BASED ON THE VALUE OF THE TEST

      SUBROUTINE TSTCHR()

      INTEGER RESCHR

      INTEGER I

      INTEGER PASS

      INTEGER FAIL

      INTEGER ACTUAL

      INTEGER EXPECT

      LOGICAL VALUES

      DIMENSION EXPECT(2)

      DIMENSION VALUES(2)

      DATA (VALUES(I), I=1,2)/.TRUE., .FALSE./,

     &    (EXPECT(I), I=1, 2)/1H., 1HF/,

     &    PASS/1H./, FAIL/1HF/

  DO 300 I = 1, 2

      ACTUAL = RESCHR(VALUES(I))

      IF (EXPECT(I) .NE. ACTUAL) GO TO 100

        WRITE(6, 200) PASS

        GO TO 300

  100 CONTINUE

        WRITE(6, 200) FAIL

  200 FORMAT(A1, $)

  300 CONTINUE

      RETURN

      END

---

There are three things about this code that may seem unusual. One is the empty parentheses after the call to TSCHR. This is not idiomatic Fortran. However, I like to do it because it is consistent with other languages, and makes the calls stand out more in the code. Secondly, I have put each variable’s definition on a separate line. This makes the listings somewhat longer, but also makes it easier for me to find each variable, or to change their type if I need to. Similarly, the DIMENSION statements are on their own line, for the same reason. These are simply my stylistic, idiosyncracies; do whatever feels right for you.

One thing I do suggest is that you explicitly declare every variable, rather than relying on Fortran’s implicit typing. I have been bitten by this repeatedly, and strongly recommend it. Even more useful is the IMPLICIT NONE in later versions of the language. Why? When (not if) I misspell a variable’s name, the compiler blissfully ignores my blunder and creates a new variable. These kinds of typos can be a nightmare to track down, because I tend to see what I intended to type, rather than what is actually there.

We looked at whether to make our assertions functions or subroutines. I somewhat arbitrarily opted for subroutines, largely because Fortran demands that I assign the results of a function to a variable. I find that an annoying intrusion, so I went with subroutines

We took a look at test driven-driven development. Here the tests are written before the actual code. In my humble opinion, TDD has several advantages, but the two that are most important to me are:

• creating the tests forces me to think through what the routine should do. If I can’t figure out how to test it, then I don’t understand it well enough to code it. The example we looked at was trivial, but as we get into more complicated cases, my experience is that it is well-worth the effort to figure out what the code should do before I write it.
• TDD also makes us look at the code from the point of view of a user of our routines (from the outside), rather than focusing simply on how they work (from the inside). This seems like a trivial difference, but I have found that it consistently helps me write better designed code.

Finally, we used our knowledge of TDD to create our first tiny piece of the framework, the RESCHR subroutine. We

1. Wrote the test.
2. Wrote the minimum code to get the test to run and fail.
3. Write the minimum code to get the test to pass.
4. Refactored the framework code to make it cleaner, ensuring all along that the tests still passed.
5. Refactored the test code, again making sure that all of the tests passed.

Next time we’ll develop a bit more, reinforcing these steps in a slightly more complicated setting.

[1] Unit 6 is typically mapped to the default output device, and I will assume throughout this series that we are using a version of Fortran that allows us to specify file names and descriptors directly. If you are working in a different environment (IBM mainframe for example) you know only too well that you have to map the units in a different way, such as JCL. You have my deepest condolences.

Creating A Fortran Unit Testing Framework (part 2)

Today we'll take a quick look at the existing open source frameworks for unit testing Fortran code. I'll briefly mention why none of these exactly fits the needs of our problem. Finally,we'll begin designing the skeleton of our framework.

Existing Possibilities for a Fortran Testing Library

There are a few open source alternatives for testing Fortran available at the time of this writing; the Wikipedia entry for unit testing frameworks lists four:

1. FRUIT,
2. FUnit,
3. Ftnunit, and
4. pfUnit.

The first two projects (FRUIT and FUnit), rely on the Ruby language. While Ruby is wonderful for many things, using a framework based on it requires that the language be installed and that the developer has at least basic familiarity how it works (installing Ruby Gems, etc.).

Ftnunit is closest in spirit to what we want. It is part of flibs. Unfortunately it is written in Fortran 90, which makes it less than ideal for our goal of testing old Fortran as we modernize it. Mixing old and modern code gives many compilers conniptions. Some features found in FORTRAN IV or FORTRAN 66 have been removed from the language. I know little of pfUnit. It appears to be under active development (the latest upload was 89 days ago from 12/27/2009). However, there are no files available for download thus far.

Beginning the Framework

I’ve decided to begin with Fortran 66. It was the first version to be standard (ANSI), contains most of the oldest and creakiest features. It is also weird in ways that make it a cool technical challenge from a coding perspective.

xUnit Frameworks

The vast majority of testing frameworks are based on a Java framework called JUnit[www.junit.org], which was in turn developed based on and older framework written in the SmallTalk language called SUnit. JUnit has been re-written (“ported”) to a huge variety of languages. Wikipedia contains one list; another can be found on Ron Jeffries site in the table titled “Unit Tests”. Collectively these implementations are termed “xUnit”, because of the tendency to prefix “unit” with some indication of the language that it is written in.

At its root, xUnit is built on the concept for an assertion; we assert that some property of the code is correct. The xUnit framework then evaluates that assertion. If it is true, the test passes. If it is false, the framework signals that it fails [1]. The traditional way of signaling that a test passed is to print a period (“.”); the way of signaling a failure is to print an “F” to the screen. Thus, a collection of 10 tests might display:

….F…..

In this case, the fifth test failed.

Unit tests are meant to evaluate very small parts of the code. Thus a unit test for the ABS intrinsic function might be

assert (1, ABS(-1))

The first number is the expected value, and the second number is what we are testing.

In object-oriented languages, each test typically evaluates a single function/subroutine (also termed “methods”). These assertions are wrapped in a function. Each function can contain one or more assertions (whether a function should contain more than one assertion is an issue that sparks near-religious debate). Each test function is called, and produces a single “.” or “F”. Taking into consideration the nature of older FORTRAN, we will write out a “.” or “F” based on the result of each assertion.

Assertions

The names of the assertions will follow the xUnit conventions as closely as possible. However, FORTRAN 66 only allows 6 characters per name, which limits our ability to choose expressive names. Furthermore, many programming languages are case sensitive, which allows us to be more expressive. We can combine multiple words, with the convention that a capital letter indicates a new word (called “camel case” because the capital letter in the middle pokes up like a camel’s hump). So, it we want to create a function that asserts that its logical argument is true, we would call it assertTrue(arg). Older FORTRAN is not case-sensitive, and is typically written in all capital letters, which limits us even further. We will adopt the convention that the root name for “assert” will be abbreviated “ASRT”. That leaves us two letters to describe the assertion. The first letter traditionally describes the kind of argument a function takes:

• I - INTEGER
• A - REAL
• D - DOUBLE PRECISION
• C - COMPLEX
• L - LOGICAL

Our first two assertions will simply determine whether their argument is true or false. The final letter will convey our expectation, “T” for .TRUE., “F” for .FALSE.

• ASRTT(arg) asserts that the LOGICAL arg is .TRUE.
• ASRTF(arg) asserts that the LOGICAL arg is .FALSE.

The next set of assertions will compare an expected value with the actual value to see whether they two are equal. The final letter of these assertions will be “E”, to convey we are testing for equality. The first letter will reflect the type of the argument. So:

• LASRTE(EXPECT , ACTUAL) asserts that the LOGIAL value ACTUAL is equal to the value that we EXPECT.

• IASRTE(EXPECT , ACTUAL) asserts that the INTEGER value ACTUAL is equal to the EXPECT value.

• AASRTE(EXPECT, ACTUAL, EPSILN) asserts that the REAL value ACTUAL is equal to the EXPECT value. Due to the inherent imprecision of floating point numbers, we do not actually compare the two values for equality. Instead we compute whether ABS(EXPECT – REAL) .LE. EPSILN, where EPSILN is a small positive number, such as 1.E-4 or 1.E-6.

• DASRTE(EXPECT, ACTUAL, EPSILN) asserts that the DOUBLE PRECISION value ACTUAL is within EPSILN (DOUBLE PRECISION) of the EXPECTED value (see the description of AASRTE).

• CASRTE(EXPECT, ACTUAL, EPSILN) asserts that the COMPLEX value ACTUAL is within EPSILN (COMPLEX) of the EXPECT value. Determining “within EPSILN” for COMPLEX numbers is a bit tricky. We let the user choose, comparing
  ABS(REAL(EXPECT) – REAL(ACTUAL)) .LE. REAL(EPSILN) and
  ABS(IMAG(EXPECT) – IMAG(ACTUAL)) .LE. IMAG(EPSILN)

If both are true, the assertion passes; if either is false, it fails.

In the next installment, we’ll look at implementing each of these assertions. I’ll also introduce test-driven development, a style of programming that leads to very robust code with a minimum of errors.

[1] Previous versions of JUnit (e.g., 3.8) contained a third outcome "E" for error/exception. This signaled a problem with the code, often a compilation problem or unexpected runtime error. The most recent version (4) of JUnit removed "E"; only "." and "F" are outcomes of tests.

Creating A Fortran Unit Testing Framework (part 1)

I’ve decided to develop a simple unit testing framework for Fortran[1]. The next several posts will document how the library is built, what decisions I’m faced with, and how they are resolved. At the end of the series, we will have a library that we can use to help us to develop new code, and to modernize existing code.

Here’s the problem that started the whole thing off. I am working on a series of simulations of the distribution of a statistic. We have derived the asymptotic distribution of the statistic, and want some idea of what sample size is sufficiently large to allow us to use the result. The distribution in question involves the sum of weighted chi-squared variates. This isn’t simple (or obvious) to compute, and there aren’t easily available, ready-made implementations to grab off the shelf and use. Moreover, I don’t much care about how to compute this quantity. It is a necessary piece of a larger puzzle, but not a piece I care much about.

Doing the obvious web searches yielded a couple of good alternatives. One in particular on Statlib seemed to be just perfect solution. The problem was that the original was in ALGOL or PASCAL, neither of which I work in on a daily basis. I could do the translation into something else (Fortran or Java or C# or any of the other, more modern languages that I know), but I am very leery of introducing a bug during the translation. A bit more web work revealed that the author of the original article had already done a translation into Fortran. So I downloaded it, happy to be able to move on to my real question. But, when I opened up the file, I found that it was written in an old version of Fortran, one that my compiler wasn’t going to compile without a lot of complaints, if it compiled it at all. So, same problem, different language. SPAM!

Modernizing Old (“Legacy”) Code

There is a lot of legacy code out there, much of it written in ancient versions of Fortran. I have code that I want to use that is written in FORTRAN 66, and there is even FORTRAN IV (or FORTRAN II!) floating around. A quick Google search also reveals a small but ongoing problem of being handed code that works but is written in an archaic version of Fortran.

As my experience demonstrates, finding one of these examples is always a pain. What starts as joy over finding a subroutine or library that does what you want turns into despair as your realize that you can’t use it. At this point, you know you have to convert it to a more modern version or the language (Fortran 95 or 2003) to incorporate into your program. Or, if you want to translate it into another language, you have the same problems, plus the differences between how one does things in Fortran and how they are done in your language of choice.

In general, these algorithms are sound, but the code often won’t even compile under a modern compiler. For example, the venerable Hollerith constant was deprecated in Fortran 90, and removed from the language entirely in Fortran 95.

So what to do? One possibility (if the code is based on an article in a professional journal or conference paper) is to roll up your sleeves, dig into the equations/derivations and write a program in your language of choice. While this certainly works, it defeats the whole benefit of finding the solution to our problem. In general, this solution requires that we deeply understand the logic underlying the original implementation, which is a lot of work that feels unnecessary, and is usually tangential to the real goal of our study.

The other alternative is to re-write the code in a modern version of the language. This always has the risk of introducing an error into the code. Because we intend to treat this code pretty much as a black box (we send values in, and new values come out, and we don’t much want to know how it does its work). We don’t want to burrow into the logic.

This is a well-known problem in professional (as opposed to academic/scientific) computing. Companies rely on code written years (or in some cases, decades) before. Often these programs are key to the business’ functioning, perhaps handing all of its billing, or being the central to a key inventory control system. The function often isn’t sexy, but it is vital to the company’s survival. Those who are old enough to have paid any attention during the late 1990’s will recall the “Y2K” problem. This was the classic maintenance nightmare. Code that was often decades old had to understood and modified. The original programmers had often retired (or died).

The defnitive book on dealing with this problem is "Working Effectively with Legacy Code" by Michael Feathers. It is brilliant, and well worth 10 times whatever Addison-Wesley is charging for it. In a nutshell, Feathers describes legacy code as code without tests. The ideal is to have a blanket of tests completely cover the code we want to change. Each test is small, and test a relatively small portion of the code. These are called "unit tests." We also want tests of the overall function ("functional tests"), but unit tests give us a fine-grained picture of what the code should do. Feathers' book is dedicated to ways that we can create unit tests.

Refactoring

The goal of modernizing any program is to change its internal structure (e.g., to update to a more modern version of a language). But as we do this restructuring, we want to insure that we don't change the function of the code. This process of changing the internal state while maintaining the external behavior is called "refactoring." One of the key ideas in refactoring is that we make very small changes, and verify that we haven't broken anything after each step. To do this, we need a good set of unit tests.

But why is this important? With a good set of unit tests, we can safely make changes to the code. If our change introduces an error, running the unit tests will let us know; a passing test will now fail. Because we have only made a small change since the last time we ran the tests, we know exactly what the change was that introduced the error. Fixing the error we introduced will make all of tests pass again.

To summarize:

1. We have legacy code that we want to convert to a modern version of the same language (Fortran)
2. Modernizing the code is a form or refactoring. We want to change the internal
3. To refactor the code with confidence, we need to create a good set of unit tests. Each unit test examines a small piece of functionality; the set of tests provides good coverage of the all of the functionality of the legacy code.
4. In refactoring, we proceed by making a series of very small changes to the code. After each, we run the tests. If we introduce a bug, one or more of the tests will fail. Because we have made only small changes, we can quickly identify the bug. Once we've fixed it, all of the tests will pass again.
5. My web surfing hasn't turned up a good unti testing framework for old Fortran, so I'm going to write one. I'll document the development here, describing the decisions made and why I made them.

Next time, I will take a look at existing unit testing frameworks for Fortran, and start developing my framework.

[1] Fortran v. FORTRAN: The original name of the language was "FORTRAN", an abbreviation fo FORmula TRANslation. The name was written all in capital letters. As of Fortran 90, the name of the language has been changed to "Fortran", with only the first letter capitalized. I'll use Fortran when speaking generically about the language. When I'm speaking of a specific version, I will use the correct capitalization for that version.

Why Simluate?

So you want to do a simulation. Go lie down and wait for the feeling to go away. Really, you'll thank me.

The most important thing to ask yourself is: "why?" There are a couple of very bad reasons to do a simulation, and a couple of really bad ones.

The bad ones are:

• It seems easy. That may be true, but it isn’t easy to do right. There are many mistakes to make (trust me, I've made most of them). Doing it right takes a lot of thought and an immense effort.
• Well I’ve got this computer sitting here doing nothing…. There is an old saying, “A fool with a tool is still a fool." or is that "... still a tool." No matter. As a scientist you have only one thing of value, your reputation. If you get the reputation for doing simple, careless work, then it doesn't matter how brilliant hyou are. No one will listen or care.

The only good reason to to a simulation is:

• To understand something that we can’t develop a good analytic result for.

That’s right. That is the only reason to do statistical simulation is as a poor proxy for an analytic result.

Remember:

One good derivation wipes away several decades of simulations.

So why is simulation worth doing at all?

1) We can’t do the derivation. This may be the fault of our personal training, or the difficulty of the task, whatever. But realize that if we are turning to simulation, we are announcing that we

2) The derivation is asymptotic. Most of us don’t live in the blessed land of Asymptopia, the land of infinite sample sizes where everything is linear and/or distributed normally. Many of the statistical tools that we have (Taylor series, first-order asymptotics, etc.) rely on large sample theory in order for their results to hold.

3) The procedure makes an assumption (normality, independence of observations, etc.) that probably isn’t true. As practitioners, we would like know how sensitive to the assumptions the procedure is.

4) We want to look at the properties of a particular implementation of a procedure. In the early 1980’s, confirmatory factor analysis and structural equations became practical. There was a great deal of interest in comparing software for estimating the parameters: particularly LISREL v. EQS. In IRT we have LOGIST v. BILOG v. PARSCALE v. MULTILOG. More recently, methods of handling missing data (based on Little and Rubin’s (1976) work) became available in general-use software such as SPSS and SAS, and there was a flurry of work comparing the implementations.

So, make sure you have a good reason for entering into the simulation arena. A good simulation brings useful information to a generation of practitioners. But, beware; here be dragons. There are an awful lot of ways to go wrong.