Tolstoy on iterative development

But their mother country was too far off, and a man who has six or seven hundred miles to walk before reaching his destination must be able to put his final goal out of his mind and say to himself that he will 'do thirty miles today and then spend the night somewhere'; and during this first stage of the journey that resting-place for the night eclipses the image of his ultimate goal and absorbs all his hopes and desires.

- Leo Tolstoy, War and Peace

Fear-based estimation

I prefer agile development because early feedback mechanisms like TDD, pair-programming and frequent releases reduce my fear of the unknown. I don't like feeling afraid, but when I am I try to take notice, because it's usually an indication that something is not right.

A great agile tradition is acknowledging the human element in software development. In that spirit, I'd like to suggest that developers' fear can be harnessed to improve estimation.

Humans are notoriously bad at temporal reasoning and programmers are even-more-notoriously bad at guessing how long a given piece of work will take. Agile teams work around this by estimating effort in 'story points' that measure the comparative size of tasks. We might not be sure how long designing the new schema will take, but we assign it 2 story points because we know it will take twice as long as another task we gave 1 point.

The rate at which a team completes points is known as its 'velocity' and can be determined by examining the team's progress over time. It's a robust, self-correcting system, but I've observed two drawbacks on projects I've worked:

• The confidence of estimates isn't captured.
• The technical debt incurred by a particular piece of work is ignored altogether.

A story that has a chance of going horribly wrong and that will in any case cripple the system's architecture could receive a low story point score because it (probably) won't take long to implement. Confronted with such a story, developers are terrified - and helpless.

I propose that we estimate in terms of 'fear points' to give product owners a disincentive to prioritise dangerous stories. The question we ask in estimation meetings shouldn't be "how big is this task" but "how afraid does this task make you?".

So long as the developers have an appropriate level of professional cowardice, a story's fear points will reflect the danger the story represents to the project's current and future timelines. And I think that 'danger' is as useful to communicate to management as 'size'.

If product owners want to negotiate a reduction in a story's fear points then they need to reduce uncertainty, remove risky features and compromise on ideas that would incur a lot of technical debt. They could also prioritise other stories that improve the team's confidence in the development process, like improved Continuous Integration.

Whenever I have seen developers afraid of a feature, it turns out badly. It ends up buggy, expensive to change and probably doesn't serve its intended purpose. By gauging developers' fear, agile organisations have an opportunity to avoid trouble before it happens.

If your developers are afraid of a feature - be very afraid.

Safe habits

When I was growing up, I found some of my parents' habits irritatingly conservative. Examples included:

• Always lock doors from the outside using the key
• Always turn on the cold tap before the hot tap

The idea was that following these habits guaranteed you could never burn yourself or lock yourself out. But I found these precautions frustrating because I knew that it was unlikely that I'd forget my keys or put my hand in the scalding water.

However, I was mistaking the most likely outcome for the entire distribution. In Life's Grandeur, Stephen Jay Gould calls this "reification" - fixating on the average case and ignoring variation and atypical outcomes.

On any given occasion, my cavalier attitude would probably suffice. But sooner or later, if I locked doors by setting the snib and pulling them shut, my keys would not be in my pocket.

My parents understood my fallibility better than I did. And they were in a better position to appreciate the myriad of door-locking scenarios that would confront me over my lifetime.

That's why I like to think that if my mother were a programmer (and she'd make a good one) she would advocate TDD, which goes as follows:

1. Write a failing test
2. Write the minimum of code to pass the test
3. Refactor

If you make TDD a permanent habit then you are inevitably at most 5 or 10 minutes away from working, tested code. And you know that every bit of code you write has a corresponding test.

Be mindful of your development habits, and design them for the bad days, not the average days.

Descartes on Unit Testing

For he who attempts to view a multitude of objects with the one and the same glance, sees none of them distinctly; and similarly the man who is wont to attend to many things at the same time by a single act of thought is confused in mind.

- René Descartes, 'Rules for the direction of the mind' in Key Philosophical Writings

Deconstructing The Savoy

The software industry has long looked to the construction industry for inspiration.

We appropriate its vocabulary - programmers "build" software designed by "architects". We draw on its ideas - the seminal Gang of Four book adapted architect Christopher Alexander's concept of design patterns for use in software construction. And the discipline of software engineering was founded on a desire to employ civil engineering practices to help us build complex software systems.

For me, the most striking similarity between the two industries is the frequency of budget blowouts and schedule overruns. The great thing about this for software developers is that it gives us a tangible way of describing our otherwise inexplicable travails and catastrophes to ordinary people.

Yesterday in The London Evening Standard I read an article about the renovations of The Savoy, the famous London hotel. It read almost word-for-word like a story about an overly ambitious IT migration project.

When The Savoy closed on 15 December 2007 for a planned 16-month, £100 million makeover, it was hoped the hotel would quickly resume its status, buffed and restored to its former glory.

In the planning stages, optimism rules. The project is so large and complex that we can't possibly plan accurately, so in the absence of evidence one way or the other we assume the best.

The original £100 million estimate for the work has been ripped up and although operators Fairmont Hotels and Resorts will not disclose the actual figure, the 15 months of work suggests it could be close to double that sum.

Once work begins, the fragility of the initial estimates is exposed. Often, the 'estimate' of how much a project will cost is as much based on the depth of the client's pockets as the actual effort required to get the job done (which of course no one knows in advance anyway).

Part of the problem was The Savoy's unplanned, organic growth.

We have to live with the sins of those who came before us. In my experience, the quality of a system's legacy code base has more impact on a project than the inherent difficulty of the project in question.

Although we had done two years of planning and tried to assess the level of issues behind the walls, it's only when you close the doors and open it up that you realise the amount of work is much more serious and extensive than first envisaged.

Unsurprisingly, once you are up to your elbows in a system's viscera, you have a much better idea about what you're in for. Exploratory surgery is the only way to be certain of how long changes will take.

"What was an open courtyard suddenly became a room, with a mix of internal and external walls."

A system accrues idiosyncrasies because it is inevitably patched, hacked and enhanced. Modules that were designed with one use-case in mind are re-purposed as business needs change. And scar tissue accumulates.

Digging up the roadway in Savoy Place, off the Strand entrance - still the only place in the UK where one must legally drive on the right - he found a huge gulley running around the perimeter, instead of a solid foundation. "We don't even know what it's for."

Users will adapt to visible peculiarities. They may even grow attached to them, even if the rationale for them has become obsolete (cars drive on the right in Savoy Place so that hansom cab drivers could open the door for their customers without leaving their seat).

But much more frightening are the dull, blank blocks of obsolete code that loom like monoliths erected by a vanished civilisation. No one remembers what they were originally intended to do, and you can never be quite sure that the earth won't mysteriously stop turning if they are ever removed.

The huge expense and loss of revenue mean The Savoy has to "hit the ground running" when it reopens if the money is ever to be recouped.

Counter intuitively, the level of optimism rises as the schedule slips. It's very tempting to think that although we fell behind in phase 1, we can make up the time in phase 2. However, it's much more likely that if one part of a project runs into trouble, the rest will too.

As a manager at one said: "Hotel travellers are very promiscuous, they will, as it were, sleep around. While you are off the scene many will have happily moved on and it could take years to get them back."

Software users are even more promiscuous, especially on the web. They will cheerfully see your competitors behind your back, even in the good times. They will not tolerate a prolonged outage and they will complain loudly if your service is unavailable even for an hour.

But I am not entirely pessemistic about the software development process. There is one important attribute that software has that buildings don't - malleability.

We are able to follow agile methodologies and incrementally improve our programs. We don't have to follow The Savoy's example and attempt to implement an enormous modification in one go. We can embrace change and use the information we gather along the way to improve the end product. And if we refactor as we develop, we can reduce the amount of technical debt we bequeath to our successors.

Clean code

At the beginning of Clean Code, Uncle Bob Martin enlists various well-respected programmers to explain what code cleanliness means to them. These luminaries include Bjarne Stroustrup (inventor of C++) and Ward Cunningham (inventor of the wiki).

Here is my definition:

Clean code imposes minimal impedance between the reader and the intent of the author. It contains little accidental complexity and its meaning can be easily understood, verified and manipulated.

When I work with clean code I have a sensation of reaching through the code to directly engage with the system's concepts.

When I work with unclean code my vision is clouded by weak naming, murky structure, inadequate commenting, convoluted dependencies and duplicated logic. Unclean code makes me afraid, because I cannot predict or understand the consequences of my changes.

Thinking from the G.U.T.

A mathematical physicist friend of mine once said that he looks forward to the day when physicists will produce a Grand Unified Theory (GUT) that will consolidate strong nuclear force, weak nuclear force and electromagnetism into a single interaction. This would be a stepping stone to the creation of a Theory of Everything that would also assimilate gravitation and thus unite all the strands of modern physics.

I was struck by his certainty that such a theory is possible. He seemed to be making a scientific prediction based on an aesthetic sensibility - unified theories are more beautiful therefore a unified theory is correct.

Unification is certainly an important part of the progress of a science. Occam's razor is a well-established principle for judging the utility of a theory. Entia non sunt multiplicanda praeter necessitatem - entities are not to be multiplied more than is necessary. In other words, the simplest theory that fits the data is the best.

When Sadi Carnot showed the equivalence of heat and mechanical work it was a victory for physics because physicists could now explain natural phenomena using less entities. At a superficial level, his discovery was useful because students now had to tax their brains with less concepts in order to understand both heat and motion.

But Carnot's unification had also produced a theory was more correct than earlier theories that thought of heat as a substance called "caloric". The mechanical theory of heat turned out to explain more phenomena than Carnot had originally considered. The laws of thermodynamics could not have been formulated without Carnot's insight.

Why should the simpler theory prove more correct?

Marcus Hutter believes that understanding is fundamentally an act of mental unification. The Hutter prize offers a reward for anyone able to produce a better compression of Wikipedia. A high compression ratio requires a deep understanding of the corpus - in this case a snapshot of human knowledge. If unification is in some sense equivalent to understanding then a unified theory is more likely to be correct because it is a better approximation of the phenomena in question.

(Interestingly, Hutter is also an advocate for a physical Theory of Everything)

However, it is a leap of blind optimism to assume that a G.U.T. is possible just because if it existed it would be useful, beautiful and likely to yield futher insight. Desirability does not imply feasibility.

In The Mythical Man Month Fred Brooks draws a distinction between essential and accidental complexity in software systems that is very pertinent to the possibility of a G.U.T.

Accidental complexity is caused by defficiencies in the solution. This kind of complexity can be eliminated by improving the approach to the problem. I would argue that the seperation of heat and mechanics was an example of accidental complexity caused by a lack of understanding of the nature of heat. The mechanical theory of heat was a successful simplification because it removed complexity that was never part of the phenomena itself.

Essential complexity, on the other hand, is inherent in the problem. It is impossible to build a solution that is less complex than the problem it is designed to solve.

The universe, like any other corpus, has an uncomputable Kolmogorov complexity that limits how simple a correct theory of physics can be. Though we cannot ever know the essential complexity of the universe, it does have one. There is an unknown and absolute limit to the unifying efforts of physics, so we cannot ever be sure that further unification will be possible.

Perhaps physics will encounter new phenomena that require new multiplication of entities to explain. Perhaps we are close to the limit and though we might incrementally simplify our theories we will never be able to reduce physics to less than four fundamental interactions.

We cannot hope to make out theories more unified than the phenomena they describe and still hope to make them correct. As Albert Einstein said (my italics):

Make everything as simple as possible, but not simpler