It's time

EuroClojure 2012 was almost a month ago. Now that my thoughts have settled I've made an attempt to distill what the event meant to me.

First of all, I was impressed at the intellectual engagement of the speakers and attendees. This is a grown-up community. Talks were less about "How do I do X in Clojure" and more about using the language to move the industry forward.

If you're a developer who's interested in the avant-garde, you should be paying attention to what this community is coming up with. I left feeling that Clojure is less a programming language and more a delivery mechanism for the Clojure philosophy.

But what is the Clojure philosophy?

Whether it was Rich Hickey railing against in-place database updates, Nick Rothwell animating art, Max Weber explaining CQRS or Paul Ingles exploring event-sourced analytics, an underlying theme was that time deserves first-class representation in interesting systems.

Even more directly, Bruce Durling's Incanter incantations and Jeff Rose and Sam Aaron's waveform wizardry manipulated time-series data before our eyes to great effect.

In an age where we can no longer assume that the universe marches to the beat of a single CPU, implicit time-management is no longer a convenience but a crippling limitation.

So when someone asks me "Should I adopt Clojure yet? What is the killer application?" I will answer them:

It's time.

Unthreading

Multi-threading is hard. To fully understand a piece of multi-threaded code the reader must mentally interleave two or more competing timelines. Our poor primate brains have enough trouble managing one thread of execution.

But what if we didn't have to carry around the idea of time at all? What if we could move about the codebase without having to mentally pause, play, fast-forward and rewind a hypothetical execution instance of the program?

The venerable and venerated Structure and Interpretation of Computer Programs comments with sadness on the entrance of the mutation-snake into the Lisp Garden of Eden: 

With objects, we must be concerned with how a computational object can change and yet maintain its identity. This will force us to abandon our old substitution model of computation in favor of a more mechanistic but less theoretically tractable environment model of computation. The difficulties of dealing with objects, change, and identity are a fundamental consequence of the need to grapple with time in our computational models. These difficulties become even greater when we allow the possibility of concurrent execution of programs.

Immutability makes code easier to understand because it allows the reader to use a simpler mental model.

Code that exclusively employs immutable (immortal) variables is effectively unthreaded from the reader's point of view. The meaning of the code is no longer dependent on time at all. The value of a symbol is a function of its definition - history is irrelevant.

Single-threaded code is easier to understand than multi-threaded code. But unthreaded code is best of all.

Overlapping magisteria

I've written that unit tests and acceptance tests represent two non-overlapping magisteria. I've realised I need to revise this.

Unit tests and acceptance tests specify internal and external behaviour respectively. There is no inherent reason why external behaviour should be trivially translatable to internal behaviour, because the structure of the internals need not bear any resemblance to the model the system presents to the world.

However, that is only true of systems in general. Systems produced using a domain-driven design methodology deliberately enforce structural similarity between the system's implementation and the domain it represents.

Because of this, it's possible to make inferences between external and internal properties of the system. For example, if a

LoginFailure

object returns

true

to the method

shouldLockAccount()

when its

int

field called

previousAttempts

is

2

or greater, there's a good chance that if a user thrice enters the wrong password that their account will be disabled.

If we know something about a well-designed system's external behaviour, we can generally extrapolate about its internal behaviour (and vice-versa).

Continuous receipt - sacrificing versions

Last week I received an email that convinced me that the way we think about versioning has irrevocably
changed. Here is the meat of the mail:

IntelliJ's Perforce plugin now has job support.

While this is a valuable feature for me in my current project, it's hardly a major event for software in general.

What struck me was the author's implicit assumption that IntelliJ is a single continuous entity that changes state. IntelliJ "now" behaves differently to what it did a month ago.

To a user, "IntelliJ" is becoming less a description of an immutable executable than the name of an evolving organism of features. Whereas once we spoke of the 2010 incarnation of a product verses the 2011 product, soon we will speak of how a piece of software behaved in 2012 compared to 2013.

Web applications like Facebook, Twitter and Gmail have been doing this for some time. It's becoming the norm for desktop apps too.

The frantic pace of continuously delivered versions is obliterating the end-user's concept of a version altogether. Where previously we expected to pull new features by explicitly choosing to upgrade, we now accept that our software will frequently be pushed changes - almost in real time.

This large number of minutely differing versions is similar to how animation generates the illusion of motion. The more frequent the changes, the smoother the effect. As video games have increased their frame-rates they have made older games look clunky and awkward by comparison. Publishing games with 1995's industry standard frame-rate would put a studio at a serious competitive disadvantage today.

Developers who are unable to embrace the concepts of continuous delivery will find users perceive their software in much the same way.

Designing Haskell

The other day I was struggling through quirk of Java's generics (which are implemented with type erasure). I pondered that it would have been wonderful if generics had been properly baked into Java from the start instead of being superficially retrofitted.

While I'm strengthening Java's type system, I thought, I should wish for type inference to cut out all those redundant type annotations. And if I'm going to do that, I might as well ask for closures!

Of course, some of the worst things about Java are awkward and commonly misused features, like nullability. I occasionally use it myself, but I'd prefer it if there were an alternative. And I'd gladly give up mutability, primitive types and implementation inheritance if only every other Java programmer had to give them up too.

But, I considered, would I just be imposing my personal taste on others? I mean, I like significant whitespace, but I know some other programmers inexplicably prefer curly braces.

If only there were already a magical programming language like this that I could switch to without disturbing the Java developers who are happy with things the way they are.

Eco on over-engineering

This triumph of the mechanism produced a kind of giddiness, whereby the importance attached to what the machine actually produced was less than that attached to the extravagance of the mechanical resources required to produce it, and frequently many of these machines displayed a wholly disproportionate relationship between the simplicity of the effect they produced and the highly sophisticated means required to obtain it.


- Umberto Eco, On Beauty: A History of a Western Idea

Procrustean sizing

Conventional agile estimation involves assigning story points based on the relative size of each piece of work. Most teams refuse to accept stories whose size falls outside a certain limit and send them back for further analysis. Teams using Procrustean sizing take this to the extreme and only accept stories of a single, fixed size.

Procrustes was mythical ancient Greek bandit whose speciality was capturing unwary travellers and making them fit his iron bed. If they were too short, he would stretch them. If they were too tall, he would cut off an appropriate section of their legs.

Procrustean has come to describe the coercion of data into an arbitrary container or structure. The term is often used perjoratively to refer to a simplistic, one-size-fits all approaches.

However, I believe that a Procrustean story sizing is a viable technique for teams using kanban to manage their work. Limiting all stories to a pre-determined size places extra restrictions on story formation but it provides stronger assurances about how work and value will flow through your team.

When all stories are of the same size it becomes easier to reason about the backlog as a whole. Adding up stories of different sizes is dangerous because teams sometimes have a tendency to over or under-estimate differently for larger stories. An eight point story might take more or less than four times as long as a two-pointer when there is not a strictly linear relationship between story points and effort.

This effect is stronger with greater variability. Some teams avoid the temptation to add up stories of different sizes by estimating in animals or some other arbitrary non-numeric scale. However, when there is only one size this complication is avoided and calculations of the total size of the backlog become more reliable.

More importantly, limiting variability helps to maintain a constant flow. Production levelling (as described by Taiichi Ohno in his account of the Toyota Production System) is essential for reducing waste because it ensures that all parts of the pipeline recieve work at a rate they can handle. Uneven flow (mura) results in work building up in some areas while other areas are forced to remain idle and disrupts feedback.

Setting limits on work-in-progress (WIP) is also simpler when there is no distinction between points and stories. A WIP limit of story points can mean that there are too few stories in development to usefully work on in parallel, but a WIP limit of stories can allow too much work to be in progress at once if large stories are in development. When there is too much WIP, it takes longer for problems to show. Also, it becomes difficult to change priorities because expedited stories are blocked by large pieces of work already in progress.

The most obvious disadvantage of the technique is that stories may need to be divided, even when the product owner envisages them as a single unit. This is a problem for any team that does not accept stories of an unlimited magnitude, but it happens more frequently when using Procrustean sizing.

Adopting Procrustean sizing is a tradeoff. If reliable flow is more important than the narrative integrity of stories, then it can assist a team's development process. If preserving stories in the form that product owners originally imagined them is important for communicating with stakeholders, then levelling your stories may not be good option.

Procrustean sizing forces teams to organise plan stories in the form that they can most easily be worked on. It's the software equivalent of Ohno's advice to reduce batch size in manufacturing and, like reducing batch size, takes discipline and effort. Procrustean sizing is a simple constraint, but kanban has already demonstrated the value and power of simple constraints correctly applied.

NB: I have not used this technique in its pure form on a real project. This post is intended as an RFC and exposition of the Procrustean sizing concept. I would be very interested in hearing others' experiences with this or similar ideas.