It's all production

Every now and then, I come across a discussion about whether test code deserves as much care as production code. I understand what is meant (the answer is yes), but I don't accept the premise of the question.

All code is production code. Deployment scripts, tests, database migrations and Java controllers all play a part in delivering value to the users. If a piece of code really has no effect on production, why was it ever written?

Overlapping magisteria

I've written that unit tests and acceptance tests represent two non-overlapping magisteria. I've realised I need to revise this.

Unit tests and acceptance tests specify internal and external behaviour respectively. There is no inherent reason why external behaviour should be trivially translatable to internal behaviour, because the structure of the internals need not bear any resemblance to the model the system presents to the world.

However, that is only true of systems in general. Systems produced using a domain-driven design methodology deliberately enforce structural similarity between the system's implementation and the domain it represents.

Because of this, it's possible to make inferences between external and internal properties of the system. For example, if a

LoginFailure

object returns

true

to the method

shouldLockAccount()

when its

int

field called

previousAttempts

is

2

or greater, there's a good chance that if a user thrice enters the wrong password that their account will be disabled.

If we know something about a well-designed system's external behaviour, we can generally extrapolate about its internal behaviour (and vice-versa).

Acceptance tests make big claims

If we have good acceptance tests, why do we need unit tests?

Acceptance tests tell us a lot. When they pass, they tell us things like:

• A feature works from an end-user perspective.
• All the components required to serve user requests have been wired up correctly. 
• The supporting infrastructure has been installed and configured in the test environment.
• The user interface has not significantly changed since the acceptance tests were first written.
• An intermittant network outage did not occur.

But when they fail, they tell us very little. They make such ambitious claims about the state of the software that failures could indicate a large number of disparate underlying issues.

Unit tests, on the other hand, claim very little. If a unit test fails, it is immediately obvious that there is a problem with a very specific section of code.

In general, the easier it is for a test to fail, the less it tells us when it does. Failures from tests that intermittently break for no reason at all aren't very informative. But if there's a little red dot next to that test that "couldn't possibly fail" then you've just discovered something startlingly new about your system.

Relying on acceptance tests makes sense if the code-base is relatively static, because failures will be relatively rare. But software that must change to remain valuable (i.e. almost all software) also needs fine-grained unit tests to give specific and actionable feedback on failures.

Eliot on projects without Continuous Integration

What are the roots that clutch, what branches grow
Out of this stony rubbish? Son of man,
You cannot say, or guess, for you know only
A heap of broken images, where the sun beats,
And the dead tree gives no shelter, the cricket no relief,
And the dry stone no sound of water.

- T. S. Eliot, The Waste Land

Communicative testing

A couple of weeks ago I proposed that tests could be thought of as facts that have to be 'explained' by code. In a comment on that post, p.j. hartlieb pointed out that this paradigm relies high tests dependability. And @hlangeveld suggested that test runs should be seen as analogous to experiments.

p.j. hartlieb and @hlangeveld help drive home the point that the purpose of tests is to provide information. If your tests aren't telling you anything, they're useless.

Normative tests

Test runs tell you whether you've finished new features and if you've broken old ones. I would call that normative information, because it reports on conformance to requirements. That kind of knowledge can answer questions like "Is this change ready to commit?" or "Can we go live on Monday?".

Management love normative information because it helps them make decisions and measure progress. This naturally leads to an over-emphasis on tests' role as a source of normative information.

Informative tests

Good tests are also be informative. They explain the meaning of failures and communicate intent. Tests can serve as alternative requirements documentation. Indeed, systems like Fitnesse unify the two concepts by converting requirements into executable acceptance tests.

The audience for informative tests is almost exclusively the development team. Informative tests provide an intimate perspective on the system's concepts that's necessary to work with the software on a daily basis. This is not information required by management, so the impetus to improve the tests' informative qualities needs to come from the development team themselves.

A Selenium system test that reports failure by dumping a raw exception stacktrace serves its normative function perfectly well. There has been a regression. We are not ready to release. Someone tell management so that they can manage the client's expectations. From Issue 658 in the Selenium bug tracker:

org.openqa.selenium.ElementNotVisibleException: Element is not currently visible and so may not be clicked
System info: os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.6.1', java.version: '1.6.0_15'
Driver info: driver.version: remote
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:94)
 at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:327)
 at org.openqa.selenium.firefox.FirefoxDriver.execute(FirefoxDriver.java:191)
 at org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:186)
 at org.openqa.selenium.remote.RemoteWebElement.click(RemoteWebElement.java:55)
 at org.openqa.selenium.internal.seleniumemulation.Click.handleSeleneseCommand(Click.java:33)
 at org.openqa.selenium.internal.seleniumemulation.Click.handleSeleneseCommand(Click.java:23)
 at org.openqa.selenium.internal.seleniumemulation.SeleneseCommand.apply(SeleneseCommand.java:30)
 at org.openqa.selenium.WebDriverCommandProcessor$1.call(WebDriverCommandProcessor.java:271)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:637)

If this was all that appeared in your test log, it would be very difficult to interpret the failure. There is no context. It's not apparent what functionality the user has lost, whether the error was handled gracefully or even if the problem is a conflict between user stories.

One way to make the result above more informative would be to catch the exception and log a message like "Error when an administrator attempted to reactivate a blocked account". Product owners don't care about the presence of divs. They care about functionality.

Communicative tests

Testing consumes a lot of effort. The return for that investment is readily available information on the state of the software. The more useful and accessible that information is, the more valuable the tests are.

Donald Knuth's description of literate programming is even more pertinent to testers than other programmers because the only purpose of tests is "explaining to human beings what we want a computer to do."

Blunt quantitative statements are sufficient to communicate normative information to people outside the development team. But to fulfill their potential within the team, test results must be qualitative, explanatory and communicative.

Tests are facts. Code is theory.

Programmers have turned to science to help resolve the software crisis. But they're doing it wrong.

Science envy

Programmers have science envy. We feel that, unlike much of our code, science works. Scientists have spent hundreds of years honing a methodology that helps them assimilate new knowledge and correct error, while we have spent decades frantically accumulating complexity that we can't handle. Strangely, scientific theories become more accurate over time, whereas software systems often decay.

The software industry has tried to learn from science and engineering's success. We call our programming degrees "Computer Science" and "Software Engineering", though they are neither. "Computer Science" students do almost no experiments. The "Software Engineering" concept of exhaustive up-front design has become so discredited that even those who can't imagine any other way feel obliged to pretend that they "don't do Waterfall".

Of course, science and engineering are just analogies when applied to programming. They are meant to be useful ways of imagining our profession, not to be literally true. But in their naive form, I don't think analogies between programming and science are very useful. If we want to benefit from scientific rigour, we need to be more rigorous in how we appropriate scientific concepts.

Scientific testing

Some software testers have used the scientific method as a way of framing their testing activities. For example, David Saff, Marat Boshernitsan and Michael D. Ernst explicitly cite Karl Popper and the scientific method in their paper on test theories. Test theories are invariant properties possessed by a piece of code which Saff et al attempt to falsify over a wide range of data points with an extension to the JUnit testing framework.

I find the reciprocal of this approach useful when debugging. I start with a defect, form a theory as to its cause, then design a test to try and falsify that theory. If I suspect that the issue is caused by rogue javascript, I'll disable javascript and attempt to reproduce the issue. If I can, I've disproved my theory and I need to find another explanation. This helps me to eliminate false causes and gradually home in on the bug.

The problem with analogies that treat tests as theories and code as a phenomena is that they tell us nothing about how to write code. The software under test is like gravity, a chemical reaction or the weather. It may or may not have an underlying structure and beauty, but any insights we gain during testing are inevitably after-the-fact.

Worse, they are static models. When software changes over time, the knowledge gathered through "scientific testing" may no longer apply. The scope of scientific testing is confined to a specific version of the software. For example, a tested and verified "theory" about the memory profile of an application may become invalid when a programmer makes a small change to a caching policy.

Tests are facts. Code is theory.

Science's strength is its ability to assimilate new discoveries. If we want to share in its success, a scientific model of software development needs to preserve science's adaptability.

We can go some way to achieving this by reversing the roles of testing and coding in the scientific testing model. Tests are facts. Code's role is as a theory that explains those facts as gracefully and simply as possible.

New requirements mean new tests. New tests are newly discovered facts that must be incorporated into the code's model of reality. Software can be seen as a specialised theory that attempts to embody what the stakeholders want the application to do.

How does that help us?

Once we accept that code as a theory, we are then in a position to justify employing the most powerful weapon in science's armoury - Occam's razor. Our role is to write the simplest possible code that is consistent with the facts/tests/requirements. Whenever we have the opportunity to eliminate concepts from our code, we should.

Simple code isn't just cheaper. It's more valuable too, because it's easier to change and extend. We can justify this with reference to scientists' experience that the simplest theory is the most likely to survive subsequent discoveries.

As new requirements arrive and our understanding of the domain deepens, we have the opportunity to refactor. Refactoring isn't rework or throwing away effort. Refactoring is enhancing code's value by incorporating new knowledge on what we want our software to do. This could be by adding functionality, or in reducing complexity. Either makes the software as a whole more valuable.

Science celebrates refactoring. Each new piece of evidence clarifies scientists' understanding of phenomena and helps yield more useful theories. Often these refinements are small, but occasionally Einstein will have an insight that supercedes Newton's laws of motion. Domain driven design founder Eric Evans describes such pivotal moments on software projects as "breakthroughs".

Non-developers often assume an application is invariably more valuable with a feature than without it. Yet the example of special relativity allows us to explain otherwise. Newton's laws of motion are perfectly adequate for ordinary use. Unless we are interested in bodies moving close to the speed of light, it's not worth bothering with the additional complexity Einstein's theories bring.

If stakeholders are willing to accept that the application targets the common case and excludes troublesome edge cases, they will enjoy software that is simpler and therefore cheaper and more valuable. Sometimes, there is value in absent features. Always, there is value in simpler code.

Crave simplicity. Celebrate deletion. If science responded to new information by adding special cases then science would be in as big a mess as the software industry. As you incorporate new requirements, attempt to refine your code so that it remains flexible enough to accomodate tomorrow's requirements. Otherwise, your code will become less and less fit for its purpose, which is to provide business value.

Conclusion

When John Maynard Keynes was attacked for repeatedly revising his economic theories, he said, "When the facts change, I change my mind – what do you do, sir?" Take the same attitude with your code, but treat requirements and tests as your facts. And remember, your code is just your best approximation of what your stakeholders want it to be.

When is the right time for the fancy stuff?

Derek Featherstone wrote an interesting post on When is the right time for accessibility? His thesis is that accessibility should be planned for at the design stage, but that implementing it should not be a high priority early on. The idea is that accessibility should be worked on as the product matures and included in a subsequent release.

As far as web accessibility and interaction-heavy sites are concerned, he is asking the wrong question. What he should be wondering is, When is the right time for the fancy stuff?

Sites with a lot of flash and javascript tend to be the worst accessibility offenders because the meaning of the site is only apparent by interacting with scripts on the page. The content is not comprehensible from the DOM itself. If you're sight impaired, don't use a mouse or don't have the reflexes of a twenty-something-year-old flash developer then you might be out of luck.

Derek is of course aware of the importance of accessibility. However his approach seems to be to build the stairs first and then put in the wheelchair ramp later, and no matter how well you plan you will always come across implementation problems you had not considered.

My main criticism of Derek's article is that he constructs a dichotomy between business imperatives (getting the site up) and doing the right thing (implementing accessibility). However there are tangible benefits for your project in getting accessibility right early on.

Most importantly, there is one blind and deaf user that every web developer should be concerned about: Googlebot. If a site is not accessible for a human with a disability then it almost certainly will not be indexed properly by Google. You cannot improve your site through user feedback if you have no users because no one can find your site.

The other main advantage of tackling accessibility early is that progressive enhancement is a sounder development methodology than building everything big-bang style. Build your pages so that they work without any flash or javascript. Once you have that working, you have a sound basis on which to build your incredibly sexy ajax effects.

That affords your testing finer granularity. You can test the plain version of your page before you spoon on the flash and javascript. If there is a bug, you will know whether it occurs as part of the form-submission process or in the interaction layer. That beats monitoring HTTP requests with Firebug trying to work out where the hell the problem is.

Of course, a big motivation web developers to make their sites accessible is that it's the right thing to do. And it is. But if you follow progressive enhancement and make accessibility part of your development process then you'll get more out of it than just a warm fuzzy feeling.