Goldilocks services

Github is a document store. Each repository is a document. Transactions are easy to execute within a

document and only eventually consistent across documents.

If your documents are too large, the scope of your transactions will be excessive, and you will get contention. If your documents are too small, the scope of your transactions will be insufficient, and you will be forced to manage changes across multiple transactions.

Each service is an aggregate root, so design them so that they can be evolved independently. 

Second order poka-yoke

Poka-yoke is the practice of "mistake-proofing" - designing interfaces in ways that prevent users from making mistakes.

Continuous integration is a form of poka-yoke for software development. Developers are prevented from releasing (obviously) broken software to production by a battery of tests.

However, poka-yoke is about designing to mitigate specific user mistakes (at least as I understand Shingo's definition). In other words, it's about putting up safety barriers between users and anticipated failure modes.

How much better to develop in a software ecosystem that discourages not only the occurrence of mistakes, but encourages the growth of designs that do not admit the possibility of these mistakes.

My colleague Gurpreet Luthra cites the example of making heat-resistant bolts a different size to ordinary ones so that they cannot be accidentally interchanged. However, were the design to be simplified so that the engine was a single piece of metal, bolts would not be required at all.

Sometimes simplification is possible. Other times essential elements of the domain force us to introduce scope for error. Perhaps engines could be constructed differently to simplify assembly. Perhaps not. The only way to tell the difference is to be constantly vigilant for the possibility of eliminating accidental complexity.

Having thorough test coverage is good. Preventing the complexity that disrupts developers' ability to comprehend the system well enough to avoid mistakes is better.

Simple designs can not only make mistakes avoidable, they can make them literally inconceivable. By that construction, you could consider a disciplined pursuit of simplicity second-order poka-yoke.

Unthreading

Multi-threading is hard. To fully understand a piece of multi-threaded code the reader must mentally interleave two or more competing timelines. Our poor primate brains have enough trouble managing one thread of execution.

But what if we didn't have to carry around the idea of time at all? What if we could move about the codebase without having to mentally pause, play, fast-forward and rewind a hypothetical execution instance of the program?

The venerable and venerated Structure and Interpretation of Computer Programs comments with sadness on the entrance of the mutation-snake into the Lisp Garden of Eden: 

With objects, we must be concerned with how a computational object can change and yet maintain its identity. This will force us to abandon our old substitution model of computation in favor of a more mechanistic but less theoretically tractable environment model of computation. The difficulties of dealing with objects, change, and identity are a fundamental consequence of the need to grapple with time in our computational models. These difficulties become even greater when we allow the possibility of concurrent execution of programs.

Immutability makes code easier to understand because it allows the reader to use a simpler mental model.

Code that exclusively employs immutable (immortal) variables is effectively unthreaded from the reader's point of view. The meaning of the code is no longer dependent on time at all. The value of a symbol is a function of its definition - history is irrelevant.

Single-threaded code is easier to understand than multi-threaded code. But unthreaded code is best of all.

Overlapping magisteria

I've written that unit tests and acceptance tests represent two non-overlapping magisteria. I've realised I need to revise this.

Unit tests and acceptance tests specify internal and external behaviour respectively. There is no inherent reason why external behaviour should be trivially translatable to internal behaviour, because the structure of the internals need not bear any resemblance to the model the system presents to the world.

However, that is only true of systems in general. Systems produced using a domain-driven design methodology deliberately enforce structural similarity between the system's implementation and the domain it represents.

Because of this, it's possible to make inferences between external and internal properties of the system. For example, if a

LoginFailure

object returns

true

to the method

shouldLockAccount()

when its

int

field called

previousAttempts

is

2

or greater, there's a good chance that if a user thrice enters the wrong password that their account will be disabled.

If we know something about a well-designed system's external behaviour, we can generally extrapolate about its internal behaviour (and vice-versa).

Domain-driven teams

I just watched Phil present on why enterprise architecture usually sucks, and it struck me that his talk resembled a formal argument:

1. Design always reflects team structure (Conway's law)
2. Design should reflect the business domain (Domain-driven design)

It seems to me that the following conclusion is inevitable:

3. Therefore, team structure should reflect the business domain

Logically, Phil's maxim that "what’s easy to change in the business model should be easy to change in your architecture" can only be achieved by structuring teams around domain concepts and avoiding cross-cutting concerns.

The waterfall mindset

Some developers are reluctant to delete code. If I understand their logic correctly, it goes something like this:

Writing code takes effort. Therefore, removing code means wasted effort.

This relies on two fallacies.

Firstly, it assumes that code is valuable in of itself. But code is a liability, not an asset. Removing code while maintaining functionality creates value because it improves agility without costing anything that stakeholders care about.

Secondly, it assumes that code/functionality is the only purpose of development. But agile practitioners use development as a way of gaining information. So even if a change is fully backed-out and never restored, the process of developing that change yielded an improved understanding of the solution space.

I realise that deconstructing fallacies is somewhat of a fallacy itself. Counterproductive practices are more likely to be driven by psychological factors than by logical errors. In this case, I think that a reluctance to delete code is motivated by an attitude I call the waterfall mindset.

Within the waterfall mindset, coding proceeds through a series of decisions that are never revisited once made. It's very similar to a sequential development process, except that the phases can be in the mind of a single developer. Like normal waterfall development, it relies on the supremely unlikely possibility of getting decisions right the first time:

• Generating reports is too slow? Let's cache them on the webserver.
• Cached files are out of date? Let's write a cron job to renew them every night.
• The webserver is under unacceptable load during the regeneration process? Let's delegate to a separate report server.
• Report generation fails silently when there are network problems? Let's develop a custom protocol with failure semantics so that the webserver knows when to re-send messages to the report server.
• And so on...

Or perhaps the report generation process itself could be profiled and optimised, eliminating the need for a caching mechanism altogether.

When I discover developers (including myself) in the grip of waterfall coding, I'm reminded of the nursery rhyme about the woman who swallowed a fly. It didn't work out well for her either.

Freedom of movement

One of the key disciplines of agile software development is avoiding Big Up Front Design. The conscientious XP developer will resolutely refuse to build anything that isn't required for the current iteration. Speculative code is poison to a project's agility, because it costs effort to build, effort to maintain and obscures the code that is performing useful functions. And often, as Ron Jeffries' adage goes, you ain't gonna need it.

Quality is ensured by a kind of mathematical induction. The original version of the application is small, focused and well-tested, which makes it responsive to change. By writing the minimum of new code every time a new feature is added, developers ensure that the code-base never accrues excess baggage that might impede it in the future.

This approach is counter-intuitive at first, because most people associate planning ahead with saving effort. The idea is that tackling future challenges before they arise can be cheaper. The first catch is that if a developer guesses wrongly they will have spent effort actively contaminating the project with detritus. The second catch is that the first catch happens every time.

However, I do not take this to mean that a developer should never look ahead when making decisions about the present. Simplicity is merely a means to an end, which is the greatest possible flexibility to take the project where it might need to go.

Flexibility must be evaluated relative to the needs of the project in question. If a website is likely to be promoted to a mass audience, then preserving code quality means maintaining the option to introduce caching in a later iteration. It would be a mistake to introduce any caching logic "just in case", because performance optimisations are notoriously hard to second-guess. But it would also be a mistake to take a decision that would make caching hard or impossible to implement.

The minimal solution is not the one with the least files, classes or lines of code. Minimal code makes the least possible number of assumptions about the future. Developers need to interpret what that means for their particular project by maintaining an awareness of upcoming requirements.

Freedom of movement is only meaningful if you know what moves you might need to make.

Architecture is the opposite of surprise

Architects often disagree on technical matters. But there's also a surprising amount of disagreement on what software architecture actually is. Here are a few definitions that I've come across:

• an abstract description of the system
• work done by developers with the title "architect"
• that diagram with clouds and arrows that somebody put on the network drive at the start of the project before we really knew what we were building
• that document signed by the client at the start of the project before we really knew what we were building

The problem with all these definitions is that they imply that systems that don't have architects don't have architectures. Any definition that can't cope with the majority of applications that have emerged without considered architectural analysis isn't very useful.

Martin Fowler's definition is more useful and widely applicable. He suggests that software architecture is the set of “things that people perceive as hard to change.” [PDF] This characterisation is successful because it shifts the focus from the production to the consumption of architecture.

Fowler's definition challenges the intentional fallacy as it applies to software architecture, which is the idea that the meaning of a text belongs to its author. Fowler's architecture can therefore include elements that were never deliberately envisaged by an architect, which in turn lets us consider systems that never had an architect.

A similar idea was advanced in the essay The Death of the Author by the literary critic Roland Barthes:

As soon as a fact is narrated no longer with a view to acting directly on reality but intransitively, that is to say, finally outside of any function other than that of the very practice of the symbol itself, this disconnection occurs, the voice loses its origin, the author enters into his own death, writing begins.

Substitute "architect" for "author" and "development begins" for "writing begins" and Barthes could be talking about what happens when a carefully prepared architecture document is handed over for implementation.

Claude E. Shannon's information theory formally analyses the consumption of texts. He measured the information content of written English by showing test subjects a truncated piece of English text and asking them to guess what letter would come next. They guessed correctly about half the time (which means that English contains roughly 1 bit of information per letter).

The beauty of this experiment is that Shannon didn't need a model of his subjects' knowledge of English. All he had to do was observe was what happened when they applied that knowledge.

Implicit in Shannon's experiment is the idea that English is the sum of all cues that inform speakers as to what could come next. Following his approach, I would define architecture as the sum of all cues that suggest to a developer how a feature should be implemented in a particular system.

These cues can take many forms. Perhaps the arrows and clouds diagram tells a developer in which tier of the system to put a particular piece of logic. But developers are also guided by the language used by stakeholders, organisational structure and the culture of the technology stack.

Under this definition, the more prescriptive a system's architecture, the less information developers need to absorb in order to understand a given feature. In other words, architecture is the opposite of surprise.

Vision is a feature

A few weeks ago Mark Whiting and I had a brief Twitter conversation about his suggestion that as design quality increases the designer disappears. He went on to suggest that the formalism we when recognising a designer's work is as much an imperfection of the design as a feature.

I was not so sure. There are definitely instances where the designer's mark seems to contribute to the design. Programming languages are a good example. Ruby would not be what it is without the strength of Matz's personal vision.

On the other hand, I do get annoyed when a designer's vanity tempts them to graffiti their signature onto a design that would have been better left alone. I'm thinking here of 'clever' designs like teapots with two spouts.

The difference between these two scenarios is, in my opinion, is whether or not the design space is convergent. I mean the term in the same sense as convergent evolution. In a convergent design space, the differences between designs will gradually disappear over time as individual designers are gradually more successful at approximating the best solution to the problem at hand.

In such a domain, it follows that any deviation from the one true design is noise. The designer's personal touch therefore detracts from their attempt to produce good design. A double-spouted teapot might help the designer express their individuality, but the result is just slightly less convenient tea.

However, it's rare to find a design space where a Platonic 'best' design exists. When have the various stakeholders in the construction of a new building ever agreed what is best? And to revisit my earlier example, which language is 'best' is one of the most common topics of programming flame wars.

Designers usually have to balance competing interests. How much should the finished product cost? What kind of user/customer should it be optimised for? What about older users/customers, or ones with disabilities? And not least, when is the deadline for the completed design? How designers balance these interests will inevitably affect the design. There is rarely any objective way to balance these subjective interests, so there is rarely an objective best design.

In such open design spaces, the designer's vision serves an important purpose - coherence. There are so many elements in a complicated design that it can be hard to take them in all at once. A strong authorial vision helps users/customers by giving them a guide to predict and/or remember the designer's choices.

Many Ruby admirers speak of the Principle of Least Surprise. Ruby is comparatively easy to learn and understand because its design choices aim to produce the least astonishment in the programmer. But since every programmer comes from a different background, they will each have different expectations and standards of astonishment.

So more precisely, Ruby was designed according to the Principle of Matz's Least Surprise. Once the programmer gets a handle on Matz's programming aesthetic, they can make educated guesses about parts of the language that they have not yet encountered.

So in conclusion, the formalism we when recognising a designer's work is a feature because it makes understanding complicated design simpler.