gets gets my goat

I'm participating in a virtual study group on the book Seven languages in seven weeks. The first week has been on Ruby, which has been a liberating and and frictionless experience - except for one glitch.

One of the exercises in the book involves writing a primitive grep in Ruby. This program reads text line-by-line from stdin and echos the lines that contain a fragment specified as a command-line argument.

My first attempt looked something like this:

#!/usr/bin/ruby 
while line = gets
    puts line if line.match ARGV[0]
end 

But when I tried to run it...

> ./grep.rb foo

./grep.rb:3:in `gets': No such file or directory - foo (Errno::ENOENT)
 from ./grep.rb:3

After a lot of head-scratching, I finally figured out that gets behaves differently if there are command-line arguments. From the Ruby documentation on gets:

Returns (and assigns to $_) the next line from the list of files in ARGV (or $*), or from standard input if no files are present on the command line.

So, I actually needed to do this:

#!/usr/bin/ruby 
while line = STDIN.gets
    puts line if line.match ARGV[0]
end 

After becoming a little frustrated at this seemingly inexplicable overloading of gets, I took a moment to think about why it bothered me so much.

One benefit that Ruby advocates often cite about the language is its adherence to the Principle of Least Surprise. So in the case of this particular unpleasant surprise, I felt cheated.

But then I realised that the strength of my reaction was a backhanded compliment to Ruby. The gets incident stuck in my mind only because it contrasts so strongly with the rest of my experience of Ruby as a consistent and well-designed programming language.

Agile teams need the same skills

There is occasionally anxiety about how project managers, business analysts, QAs and other traditional roles fit into agile projects. A project manager new to agile development might feel that their skills and experience aren't valued because they don't see their old job title explicitly mentioned in a description of an agile team.

Agile schedules still need to be managed. Agile requirements still need to be analysed. Agile codebases still need to be tested. The difference is that agile teams don't assume that these activities have to be done by dedicated team members.

A cross-functional team needs a wide variety of skills to succeed. Sometimes these will be provided by specialists and sometimes by generalists. The balance has to be struck for the specific needs of a project and can't be determined by a crude, one-fits-all rule.

Though agile is a great step forward in software development, it would be arrogant and unfair to think that there is no place in agile teams for software professionals that happen to come from traditionally structured organisations.

Elastic teams absorb shocks

I've come across the idea that because cross-functional teams need a variety of skills, members of the team should aspire to be generalists. I think that is overstating the case somewhat.

Agile teams need to be able to handle workloads requiring varying mixtures of skillsets. In order to cope, teams need elasticity, that is they need to be able to temporarily boost their capacity in a particular kind of work as the situation requires.

This can be better achieved by a mixture of flexibility and focus than by either radical specialisation or radical generalisation.

Specialisation has the obvious drawback that if the specialist in a given area is at full capacity the team cannot take on any more of this kind of work. This leads to the situation where some team members aren't fully utilised but the team cannot commit to taking any more stories to "done done" in the current iteration.

Total generalisation doesn't produce idle team members, but it does prevent a team from reaching its maximum possible potential. Software development is full of difficult and technically-specific problems that may not be understood by someone who has not deeply immersed themselves in that discipline. Furthermore, it's hard for someone who isn't fully comfortable with a domain to coordinate others at the same level.

I prefer teams where there is an expert ready to take charge in the face of any given challenge and who can rely on the rest of the team to fall in behind them. This happens best when team members are specialists in their own discipline and generalists in everything else.

To return to the metaphor in the title of this post, effective agile teams have a definite shape but they have enough elasticity to absorb the shocks of atypical workloads.

Domain-driven teams

I just watched Phil present on why enterprise architecture usually sucks, and it struck me that his talk resembled a formal argument:

1. Design always reflects team structure (Conway's law)
2. Design should reflect the business domain (Domain-driven design)

It seems to me that the following conclusion is inevitable:

3. Therefore, team structure should reflect the business domain

Logically, Phil's maxim that "what’s easy to change in the business model should be easy to change in your architecture" can only be achieved by structuring teams around domain concepts and avoiding cross-cutting concerns.

Declarative management

Broadly speaking, programming languages can be categorised as either imperative or declarative. This post will explain the difference, and also argue that a declarative approach to management leads to better outcomes.

Imperative programming

Imperative programs are lists of commands. Here is a piece of C code that tells the computer how to make a meal based on a recipe. Get a bowl, start at the first ingredient, add and mix ingredients one at a time and stop when there are none left:

Bowl make_recipe(char** ingredients, int num_ingredients)
{
  int bowl_size = number_of_ingredients / 3;
  Bowl bowl = fetch_bowl(bowl_size);

  for(int i=0; i < number_of_ingredients; i++)
  {
    add_ingredient(ingredient_list[i], bowl);
    mix(bowl);
  }
  return bowl;
}

For some recipes it doesn't matter what order ingredients are added. It wouldn't make a whole lot of difference to your pancakes batter if you add the milk before the eggs or the other way around. But this code doesn't trust the computer to decide when rearranging ingredients is safe; they must be added in the exact order that they appear on the list.

Declarative programming

Prolog is a kind of declarative language known as a logic programming language. I write Prolog code by declaring facts (Susan is Mark's parent) and rules (a grandparent is a parent of a parent):

parent(susan, mark).
parent(gerald, susan).

grandparent(X, Z) :- parent(X, Y), parent(Y, Z).

I can now ask Prolog questions based on these declarations:

| ?- grandparent(gerald, mark).
Yes

| ?- grandparent(mark, susan).
No

I didn't describe how to arrive at those answers. I merely provided the necessary information and left the rest up to Prolog. If someone changes details in Prolog's algorithm it won't affect my program so long as it continues to interpret facts and rules in the same way. There are many different kinds of declarative programming language but they all focus on what needs to be achieved and leave the how to the implementors' discretion.

Declarative management

Declarative management describes the desired outcome but give the team freedom to achieve it in the way that appears best to them. One way that we encourage this on agile projects is to make sure that all our stories have a "so that" clause that explains their intent. This allows requirements to be much more concise, because we concentrate on the things that matter - in other words the what.

If the story says "As a user, I want the system to know my favourite colour, so that the site can be personalised to suite my aesthetic taste", then the team are free to bring their expertise on the problem. The solution they come up with might even surprise the person who originally came up with the story. In the agile world, we call this "delighting the customer".

On the other hand, imperative management is very specific about exactly how the things must be done. To follow the recipe example above, workers in a fast-food chain might be given exact steps on how to put together a hamburger.

There is a place for both imperative and declarative programming. Declarative programming is a good choice when it's important to make intent very clear or when the computer is in a position to make better optimisations than the programmer. Imperative languages are often the only sensible choice when the how needs to be tightly controlled e.g. to ensure high performance in a video game.

However, there is very little justification for imperative management. The only time that its helpful to give a team the how is when you believe that they would come up with a worse solution than you. If you think that's true, you're probably either wrong or you haven't properly trained your team.

I'm sure there are rare cases when imperative management can help the business. In the fast food example, the fastest possible way to assemble a burger might not be independently arrived at by every part-time employee. Just like a C++ programmer might use her detailed understanding of a PS3's hardware to force it to behave in a certain way, central management might find it advantageous to promulgate an optimised burger production process that's faster than what employees could come up with themselves. But as a general rule this doesn't happen, because while computers are naive and unable to show initiative, your employees are not.

Imperative management is micromanagement. Declarative management is empowerment. And since your employees are (hopefully) better at their jobs than you are, empowerment is more likely to achieve the results you are after.

The waterfall mindset

Some developers are reluctant to delete code. If I understand their logic correctly, it goes something like this:

Writing code takes effort. Therefore, removing code means wasted effort.

This relies on two fallacies.

Firstly, it assumes that code is valuable in of itself. But code is a liability, not an asset. Removing code while maintaining functionality creates value because it improves agility without costing anything that stakeholders care about.

Secondly, it assumes that code/functionality is the only purpose of development. But agile practitioners use development as a way of gaining information. So even if a change is fully backed-out and never restored, the process of developing that change yielded an improved understanding of the solution space.

I realise that deconstructing fallacies is somewhat of a fallacy itself. Counterproductive practices are more likely to be driven by psychological factors than by logical errors. In this case, I think that a reluctance to delete code is motivated by an attitude I call the waterfall mindset.

Within the waterfall mindset, coding proceeds through a series of decisions that are never revisited once made. It's very similar to a sequential development process, except that the phases can be in the mind of a single developer. Like normal waterfall development, it relies on the supremely unlikely possibility of getting decisions right the first time:

• Generating reports is too slow? Let's cache them on the webserver.
• Cached files are out of date? Let's write a cron job to renew them every night.
• The webserver is under unacceptable load during the regeneration process? Let's delegate to a separate report server.
• Report generation fails silently when there are network problems? Let's develop a custom protocol with failure semantics so that the webserver knows when to re-send messages to the report server.
• And so on...

Or perhaps the report generation process itself could be profiled and optimised, eliminating the need for a caching mechanism altogether.

When I discover developers (including myself) in the grip of waterfall coding, I'm reminded of the nursery rhyme about the woman who swallowed a fly. It didn't work out well for her either.

Acceptance tests make big claims

If we have good acceptance tests, why do we need unit tests?

Acceptance tests tell us a lot. When they pass, they tell us things like:

• A feature works from an end-user perspective.
• All the components required to serve user requests have been wired up correctly. 
• The supporting infrastructure has been installed and configured in the test environment.
• The user interface has not significantly changed since the acceptance tests were first written.
• An intermittant network outage did not occur.

But when they fail, they tell us very little. They make such ambitious claims about the state of the software that failures could indicate a large number of disparate underlying issues.

Unit tests, on the other hand, claim very little. If a unit test fails, it is immediately obvious that there is a problem with a very specific section of code.

In general, the easier it is for a test to fail, the less it tells us when it does. Failures from tests that intermittently break for no reason at all aren't very informative. But if there's a little red dot next to that test that "couldn't possibly fail" then you've just discovered something startlingly new about your system.

Relying on acceptance tests makes sense if the code-base is relatively static, because failures will be relatively rare. But software that must change to remain valuable (i.e. almost all software) also needs fine-grained unit tests to give specific and actionable feedback on failures.