Eco on over-engineering

This triumph of the mechanism produced a kind of giddiness, whereby the importance attached to what the machine actually produced was less than that attached to the extravagance of the mechanical resources required to produce it, and frequently many of these machines displayed a wholly disproportionate relationship between the simplicity of the effect they produced and the highly sophisticated means required to obtain it.


- Umberto Eco, On Beauty: A History of a Western Idea

Tests are facts. Code is theory.

Programmers have turned to science to help resolve the software crisis. But they're doing it wrong.

Science envy

Programmers have science envy. We feel that, unlike much of our code, science works. Scientists have spent hundreds of years honing a methodology that helps them assimilate new knowledge and correct error, while we have spent decades frantically accumulating complexity that we can't handle. Strangely, scientific theories become more accurate over time, whereas software systems often decay.

The software industry has tried to learn from science and engineering's success. We call our programming degrees "Computer Science" and "Software Engineering", though they are neither. "Computer Science" students do almost no experiments. The "Software Engineering" concept of exhaustive up-front design has become so discredited that even those who can't imagine any other way feel obliged to pretend that they "don't do Waterfall".

Of course, science and engineering are just analogies when applied to programming. They are meant to be useful ways of imagining our profession, not to be literally true. But in their naive form, I don't think analogies between programming and science are very useful. If we want to benefit from scientific rigour, we need to be more rigorous in how we appropriate scientific concepts.

Scientific testing

Some software testers have used the scientific method as a way of framing their testing activities. For example, David Saff, Marat Boshernitsan and Michael D. Ernst explicitly cite Karl Popper and the scientific method in their paper on test theories. Test theories are invariant properties possessed by a piece of code which Saff et al attempt to falsify over a wide range of data points with an extension to the JUnit testing framework.

I find the reciprocal of this approach useful when debugging. I start with a defect, form a theory as to its cause, then design a test to try and falsify that theory. If I suspect that the issue is caused by rogue javascript, I'll disable javascript and attempt to reproduce the issue. If I can, I've disproved my theory and I need to find another explanation. This helps me to eliminate false causes and gradually home in on the bug.

The problem with analogies that treat tests as theories and code as a phenomena is that they tell us nothing about how to write code. The software under test is like gravity, a chemical reaction or the weather. It may or may not have an underlying structure and beauty, but any insights we gain during testing are inevitably after-the-fact.

Worse, they are static models. When software changes over time, the knowledge gathered through "scientific testing" may no longer apply. The scope of scientific testing is confined to a specific version of the software. For example, a tested and verified "theory" about the memory profile of an application may become invalid when a programmer makes a small change to a caching policy.

Tests are facts. Code is theory.

Science's strength is its ability to assimilate new discoveries. If we want to share in its success, a scientific model of software development needs to preserve science's adaptability.

We can go some way to achieving this by reversing the roles of testing and coding in the scientific testing model. Tests are facts. Code's role is as a theory that explains those facts as gracefully and simply as possible.

New requirements mean new tests. New tests are newly discovered facts that must be incorporated into the code's model of reality. Software can be seen as a specialised theory that attempts to embody what the stakeholders want the application to do.

How does that help us?

Once we accept that code as a theory, we are then in a position to justify employing the most powerful weapon in science's armoury - Occam's razor. Our role is to write the simplest possible code that is consistent with the facts/tests/requirements. Whenever we have the opportunity to eliminate concepts from our code, we should.

Simple code isn't just cheaper. It's more valuable too, because it's easier to change and extend. We can justify this with reference to scientists' experience that the simplest theory is the most likely to survive subsequent discoveries.

As new requirements arrive and our understanding of the domain deepens, we have the opportunity to refactor. Refactoring isn't rework or throwing away effort. Refactoring is enhancing code's value by incorporating new knowledge on what we want our software to do. This could be by adding functionality, or in reducing complexity. Either makes the software as a whole more valuable.

Science celebrates refactoring. Each new piece of evidence clarifies scientists' understanding of phenomena and helps yield more useful theories. Often these refinements are small, but occasionally Einstein will have an insight that supercedes Newton's laws of motion. Domain driven design founder Eric Evans describes such pivotal moments on software projects as "breakthroughs".

Non-developers often assume an application is invariably more valuable with a feature than without it. Yet the example of special relativity allows us to explain otherwise. Newton's laws of motion are perfectly adequate for ordinary use. Unless we are interested in bodies moving close to the speed of light, it's not worth bothering with the additional complexity Einstein's theories bring.

If stakeholders are willing to accept that the application targets the common case and excludes troublesome edge cases, they will enjoy software that is simpler and therefore cheaper and more valuable. Sometimes, there is value in absent features. Always, there is value in simpler code.

Crave simplicity. Celebrate deletion. If science responded to new information by adding special cases then science would be in as big a mess as the software industry. As you incorporate new requirements, attempt to refine your code so that it remains flexible enough to accomodate tomorrow's requirements. Otherwise, your code will become less and less fit for its purpose, which is to provide business value.

Conclusion

When John Maynard Keynes was attacked for repeatedly revising his economic theories, he said, "When the facts change, I change my mind – what do you do, sir?" Take the same attitude with your code, but treat requirements and tests as your facts. And remember, your code is just your best approximation of what your stakeholders want it to be.

Deconstructing The Savoy

The software industry has long looked to the construction industry for inspiration.

We appropriate its vocabulary - programmers "build" software designed by "architects". We draw on its ideas - the seminal Gang of Four book adapted architect Christopher Alexander's concept of design patterns for use in software construction. And the discipline of software engineering was founded on a desire to employ civil engineering practices to help us build complex software systems.

For me, the most striking similarity between the two industries is the frequency of budget blowouts and schedule overruns. The great thing about this for software developers is that it gives us a tangible way of describing our otherwise inexplicable travails and catastrophes to ordinary people.

Yesterday in The London Evening Standard I read an article about the renovations of The Savoy, the famous London hotel. It read almost word-for-word like a story about an overly ambitious IT migration project.

When The Savoy closed on 15 December 2007 for a planned 16-month, £100 million makeover, it was hoped the hotel would quickly resume its status, buffed and restored to its former glory.

In the planning stages, optimism rules. The project is so large and complex that we can't possibly plan accurately, so in the absence of evidence one way or the other we assume the best.

The original £100 million estimate for the work has been ripped up and although operators Fairmont Hotels and Resorts will not disclose the actual figure, the 15 months of work suggests it could be close to double that sum.

Once work begins, the fragility of the initial estimates is exposed. Often, the 'estimate' of how much a project will cost is as much based on the depth of the client's pockets as the actual effort required to get the job done (which of course no one knows in advance anyway).

Part of the problem was The Savoy's unplanned, organic growth.

We have to live with the sins of those who came before us. In my experience, the quality of a system's legacy code base has more impact on a project than the inherent difficulty of the project in question.

Although we had done two years of planning and tried to assess the level of issues behind the walls, it's only when you close the doors and open it up that you realise the amount of work is much more serious and extensive than first envisaged.

Unsurprisingly, once you are up to your elbows in a system's viscera, you have a much better idea about what you're in for. Exploratory surgery is the only way to be certain of how long changes will take.

"What was an open courtyard suddenly became a room, with a mix of internal and external walls."

A system accrues idiosyncrasies because it is inevitably patched, hacked and enhanced. Modules that were designed with one use-case in mind are re-purposed as business needs change. And scar tissue accumulates.

Digging up the roadway in Savoy Place, off the Strand entrance - still the only place in the UK where one must legally drive on the right - he found a huge gulley running around the perimeter, instead of a solid foundation. "We don't even know what it's for."

Users will adapt to visible peculiarities. They may even grow attached to them, even if the rationale for them has become obsolete (cars drive on the right in Savoy Place so that hansom cab drivers could open the door for their customers without leaving their seat).

But much more frightening are the dull, blank blocks of obsolete code that loom like monoliths erected by a vanished civilisation. No one remembers what they were originally intended to do, and you can never be quite sure that the earth won't mysteriously stop turning if they are ever removed.

The huge expense and loss of revenue mean The Savoy has to "hit the ground running" when it reopens if the money is ever to be recouped.

Counter intuitively, the level of optimism rises as the schedule slips. It's very tempting to think that although we fell behind in phase 1, we can make up the time in phase 2. However, it's much more likely that if one part of a project runs into trouble, the rest will too.

As a manager at one said: "Hotel travellers are very promiscuous, they will, as it were, sleep around. While you are off the scene many will have happily moved on and it could take years to get them back."

Software users are even more promiscuous, especially on the web. They will cheerfully see your competitors behind your back, even in the good times. They will not tolerate a prolonged outage and they will complain loudly if your service is unavailable even for an hour.

But I am not entirely pessemistic about the software development process. There is one important attribute that software has that buildings don't - malleability.

We are able to follow agile methodologies and incrementally improve our programs. We don't have to follow The Savoy's example and attempt to implement an enormous modification in one go. We can embrace change and use the information we gather along the way to improve the end product. And if we refactor as we develop, we can reduce the amount of technical debt we bequeath to our successors.

Context is sticky

Code reuse is one of the holy grails of the software engineering movement. Across the world, developers are frantically reinventing the wheel. The web groans under the weight of piles of functionally equivalent PHP applications for rendering the contents of database tables.

If a larger portion of this torrent of code could be reused then an enormous amount of effort could be saved. Perhaps this effort could be diverted into improving the software quality and we could finally make a dent in the software crisis.

But though everyone has been talking about code reuse for decades, there has been very little progress.

The code that has enjoyed a significant degree of reuse has been specifically designed for that purpose. Frameworks, libraries and plugin architectures are widespread. Even the mighty operating system exists to share functionality between applications. But serendipitous reuse of code that was originally designed to solve a singular problem is rare.

I think that the reason that code reuse is hard is the same reason that the semantic web has failed to materialise. This makes sense, because code is just a particular kind of semantic content.

As Clay Shirky has argued, the the semantic web is a problematic ambition because it requires a universal worldview. The semantic web project envisages that information interoperability will be achieved by employing universal data formats. But data formats are contingent on worldview, which can never be universal. Shirky takes genetics as an example:

It would be relatively easy, for example, to encode a description of genes in XML, but it would be impossible to get a universal standard for such a description, because biologists are still arguing about what a gene actually is. There are several competing standards for describing genetic information, and the semantic divergence is an artifact of a real conversation among biologists. You can't get a standard til you have an agreement, and you can't force an agreement to exist where none actually does.

Even something as apparently clear-cut as genetic science resists universal semantic presentation because the data is contaminated by its original context.

The opinions, prejudices, needs and worldview of a programmer are imprinted on their code to a far greater degree. That class you wrote the other day to process form values assumes that every field has exactly one value. The HTML the form was displayed in uses classes unique to your site's CSS. And the coding standards the class conforms to differ from standard PHP conventions because your organisation wants to achieve consistency with its .NET projects.

You might be able to shoehorn this code into the next project you complete for the same organisation, but there is little chance of your form-processing class ever being used by someone else entirely. The cleaner and more decoupled your code is, the more use it might be to someone else, but you cannot entirely erase the imprint of its original context because context is what gives your code meaning.

The way you can best foster reuse is to engineer a situation where the worldview embedded in your code is adopted by the reuser. Take Firefox as an example. The core functionality of the browser is leveraged by thousands of plugin developers. But the API these extensions work with was laid down by the developers of Firefox and has meaning only in the context of the Firefox browser.

A cross-browser extension API would be very convenient, but the task of creating a plugin model that would apply as well to Chrome as to Firefox would be gargantuan. Witness how difficult it is to even get HTML and CSS to render the same in more than one browser. A cross-browser API would take the compatibility issues from the DOM and spread them to every aspect of the browsing experience.

Commonly-used frameworks also owe their success to prescribing a worldview. The only painless way to work with a framework is to follow the Rails way, the Django way or the Drupal way. To reuse someone else's code you must make concessions to their way of doing things.

There are a couple of current developments in software engineering that will help with the code reuse problem. Test driven development helps to make the assumptions embedded in code explicit by describing them using unit tests. The referential transparency fostered by the functional programming paradigm controls context by quarantining side-effects.

But code reuse will always be intrinsically hard because context is sticky.