tag:blogger.com,1999:blog-86482683645499559592024-03-19T07:25:18.865+00:00the literate programmer"Let us concentrate on explaining to human beings what we want a computer to do"ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.comBlogger66125tag:blogger.com,1999:blog-8648268364549955959.post-779930757816965982022-03-20T20:58:00.001+00:002022-03-20T20:59:50.582+00:00The unreasonable effectiveness of data-oriented programming<p>In 2009, I left my graduate job in Australia and moved to the UK with my British partner. I'd been working as a programmer for about two years. The economy was reeling from the 2008 crash, so I took whatever programming jobs that I was offered. And what I was offered was PHP, specifically <a href="https://www.drupal.org/">Drupal</a>.</p><p>Ever since I stumbled across programming during an electrical engineering degree at Melbourne University, I've loved programming languages. I love their differences. I love their quirks. Most of all I love the model of the world that is embedded in every programming language's design choices.</p><p>I loved C's memory manipulation. I loved Haskell's elegant types. I loved Prolog's single-minded obsession with modelling every computation using reversible predicates (it reminded me of Wittgenstein and Russell). I still remember the warm glow I felt leaving the lecture on software design where I learned about Java's interfaces could be used to implement polymorphism without inheritance.</p><p>But I didn't love PHP. I knew it as a language that grew rather than was designed. I thought it was ugly. I I was a snob. But I knew how to program in it. "What is there even to know?" I thought.</p><p>The killer apps of the PHP world were all open-source content management systems (CMSes). Drupal. <a href="https://wordpress.com/">Wordpress</a>. <a href="https://www.joomla.org/">Joomla</a>. <a href="https://github.com/wikimedia/mediawiki">MediaWiki</a>, which still powers Wikipedia today. You had to hand it to PHP. It was good at getting content in front of people.</p><p>Drupal 6 was my tool of choice. I'd used it at university and luckily it was quite a common requirement of the digital agency jobs that I found in the south-west of England. I learned it. I learned it by buying the book and reading it cover to cover and I learned it by FTPing files directly onto production servers and seeing what happened.</p><p>To learn Drupal meant to learn to use its extensions. That's where the power of the PHP CMSes lay. There were extensions for everything and the community culture encouraged sharing and open sourcing the extensions that you made to other peoples extensions.</p><p>To use and write Drupal extensions meant to understand the hook system. The hook system is a kind of reflection-based event system. Extensions implement a hook via specially-named functions. When the hook is activated, all functions with a matching name will be called.</p><p>But what would they be called with? What kind of arguments would be passed to the hook functions? PHP inherited convenient syntax for nested associative maps from Perl, so Drupal used those. You'd read the documentation to discover what keys to expect in the associative map and which ones you were expected to return. Sometimes you'd make a mistake, but you could usually figure out what had gone wrong by inspecting the input and output associative maps.</p><p>I remember being uneasy that our code was just functions that accepted and returned data. It felt like cheating somehow. "If we were doing this properly," I'd tell my coworkers with the scintillating arrogance of a programmer with two years professional experience, "we'd be grouping these functions into classes and encapsulating this data in objects." "Yeah?" they said patiently, "And how would that be better?" I had no reply (I know there are replies but I didn't have them at the time).</p><p>Later, I got into Clojure. I learned it so that I could use <a href="https://twitter.com/rosejn">Jeff Rose</a> and <a href="https://twitter.com/samaaron">Sam Aaron</a>'s <a href="https://overtone.github.io/">Overtone</a> audio environment, that let me synthesise sound using programming (Sam now works on <a href="https://sonic-pi.net/">Sonic Pi</a>). I started attending and then organising the <a href="https://londonclojurians.org/code-dojo/">London Clojure Dojo</a>.</p><p>I learned a lot of other things about Clojure. I learned about the power of the REPL and then about using <a href="https://github.com/vim-scripts/VimClojure">VimClojure</a> to evaluate pieces of code inside the editor itself (I recommend <a href="https://github.com/tpope/vim-fireplace">Fireplace</a> today for Vim users). It was always easier to experiment with pieces of code that accepted data that I could directly enter and returned data I could easily inspect.</p><p>I loved <a href="https://twitter.com/weavejester">James Reeve</a>'s <a href="https://github.com/ring-clojure/ring">Ring</a>, which gave me a clear programming model for handling HTTP requests. A HTTP request is an associative map, and if I apply a series of functions to it I can turn it into an HTTP response, which is also an associative map. So simple it almost felt like cheating.</p><p>I was lucky enough to meet Jeff, Sam and James at the dojo. Inspired by Ring, I wrote a library called <a href="https://github.com/ctford/leipzig">Leipzig</a> that applied this data-oriented style to Overtone programs. I carved out a niche for myself on the functional programming conference circuit <a href="http://literateprogrammer.blogspot.com/2016/09/music-as-code-talks.html">speaking about music theory</a> modeled using little data-in data-out functional programs.</p><p>Through all this, I didn't have a good way of explaining what this style of programming was. It definitely wasn't object-oriented. It didn't quite seem to be functional programming either, at least not functional programming as I'd practised it writing Haskell. Someone who thinks in types might describe it in negative terms, as functional programming without the guard rails and affordances of types. But I didn't experience this data-first approach as an absence of anything. It felt to me like a different style of programming altogether, one with its own strengths, weaknesses, tricks and traps.</p><p>In <a href="https://www.manning.com/books/data-oriented-programming">Data-Oriented Programming</a>, <a href="https://twitter.com/viebel">Yehonathan Sharvit</a> gives a comprehensive account of what it means to write programs data-first. I haven't read anything like it before. You can watch <a href="https://www.youtube.com/watch?v=YR5WdGrpoug">Rich Hickey's talks</a> and get some of the ideas that Yehonathan covers. You can read books on Clojure and other functional programming languages. But nowhere else have I ever read a complete description of what it means to put data at the heart of your programs.</p><p>Yehonathan uses Javascript for his examples but the specific programming language is not the point. He writes about techniques including separating code and data, validation and state management that you can implement in any programming language.</p><p>Data-oriented programming starts with data modeling and treats functions as connectors that get you from one format to another. Unlike objects and higher-order functions, it offers a model that can be extended beyond individual programs to the system level. And it offers the flexibility to create designs more lucid and more opaque than any other style I've programmed with. It's not cheating and it does require discipline. Data-oriented programming is great when done right. But how can you do it right if no one tells you how to do it?</p><p>That's why I'm really glad that Yehonathan has taken the time to write this book, still in Manning's early access programme. Data-oriented programming is a thing, a different thing to other styles of programming. And now there's a reference detailing what it is.</p>ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com5tag:blogger.com,1999:blog-8648268364549955959.post-49730107046090457362017-01-14T11:53:00.002+00:002017-01-14T11:56:24.659+00:00Expressive types, not oppressive types<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
<div>
<div>
Uncle Bob wrote a recent post in which he warns programmers against the <a href="http://blog.cleancoder.com/uncle-bob/2017/01/11/TheDarkPath.html">"dark path"</a> some modern languages have taken - that is to "double down" on static typing. He cites <a href="https://developer.apple.com/swift/">Swift</a> and <a href="http://kotlinlang.org/">Kotlin</a> as examples, though his argument is meant to be interpreted more generally.</div>
<div>
</div>
<div>
I share many points of view in common with Uncle Bob. I find the dynamically typed <a href="https://clojure.org/">Clojure</a> programming language beautiful and expressive - most of my personal projects are written in Clojure. I think that TDD (test-driven <i>design</i>) is a valuable and important discipline - I work for <a href="https://www.thoughtworks.com/">an agile consulting company</a> where most of our projects include helping clients to get better at testing.</div>
<div>
<br /></div>
<div>
But I disagree strongly with the way Uncle Bob frames this discussion on static types.</div>
<div>
<br /></div>
Uncle Bob looks at advanced type systems and sees them as <i>more</i> <i>oppressive</i> rather than <i>more</i> <i>expressive.</i>
Being able to describe whether or not a function can return null is an
opportunity, not a constraint. Being able to use types to describe your
code's intent is an opportunity, not a constraint. Being able to reason
about the behaviour of a function based on its type signature is an
opportunity, not a constraint.<br />
</div>
</div>
<div>
The kicker is that this
is almost exactly the fallacy about TDD that we have railed against for
years. We call it "test-driven design" because we know that
evolving code in response to examples is a great way to inform a
design. Folks who have not learnt to listen to their unit tests see them
as nagging constraints that prevents them from writing code in the way
they'd like. A master of TDD uses tests as feedback for their design.<br />
<br />
Anyone
who sees unit test as mere "checks" that make changing code needlessly
difficult isn't getting the most out of test-driven design. Anyone who
sees static types as mere "checks" that make changing code needlessly
difficult isn't getting the most out of type-driven design.</div>
<div>
</div>
<div>
Based on his
post Uncle Bob falls into the latter category. He sees types as ad hoc antidotes for specific mistakes rather than tools for thought -
"Every time there’s a new kind of bug, we add a language feature to
prevent that kind of bug."</div>
<div>
</div>
<div>
If that's Uncle Bob's experience of Swift and Kotlin, he
should try <a href="http://elm-lang.org/">Elm</a>. Or <a href="http://fsharp.org/">F#</a>. Or <a href="https://www.haskell.org/">Haskell</a>. If his experience is anything like mine, he would find that more sophisticated types lead to less ad hockery, not more.</div>
<div>
<br /></div>
<div>
In <a href="http://blog.cleancoder.com/uncle-bob/2017/01/13/TypesAndTests.html">a follow-up post</a>, Uncle Bob is explicit about what he wants in a programming language - "There is a <i>balance point</i> after which every step down <i>The Dark Path</i> increases the cost over the benefit. I think Java and C# have done a reasonable job at hovering near the balance point."</div>
<div>
</div>
<div>
I couldn't disagree more. Java and C# have two of the most onerous and least beneficial type systems. Their complexity and absence of type inference force excessive bookkeeping on the programmer. They lack of basic features like <a href="https://www.schoolofhaskell.com/school/to-infinity-and-beyond/pick-of-the-week/sum-types">sum types</a>, which denies the programmer an important expressive idiom.</div>
<div>
</div>
<div>
Java and C# represent the nadir of the type system trade-off, not the zenith. Type systems are tools. Better tools help us write better code. We should welcome each and every advance in the tools we use to do our job, because frankly we could do a lot better than what we have now.</div>
<div>
<br /></div>
<div>
To argue that employing more expressive types is a "dark path" that leads
developers away from personal responsibility isn't accurate or helpful.</div>
<i></i></div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com63tag:blogger.com,1999:blog-8648268364549955959.post-3271814452836605672016-11-05T12:10:00.002+00:002016-11-05T12:10:55.799+00:00Computational Musicology, ????, Profit<div dir="ltr" style="text-align: left;" trbidi="on">
This year I had the pleasure of attending <a href="http://functional-art.org/2016/">FARM</a> at <a href="http://icfpconference.org/">ICFP</a>. As well as demoing <a href="http://ctford.github.io/klangmeister/">Klangmeister</a>, I gave a paper on what computational musicology means for the study of music. The abstract is as follows:<br />
<br />
<i>In this paper I examine the relationship that complexity theory and<br />disjunctive sequences have to music, music-generating programs<br />and literary works. I then apply these ideas by devising a program<br />to generate an infinite ‘Copyright Infringement Song’ that contains<br />all other songs within it. I adopt literary modes of analysis and<br />presentation, which I motivate by arguing that music is a cultural<br />and artistic phenomenon rather than a natural one.</i><br />
<br />
<a href="http://delivery.acm.org/10.1145/2980000/2975987/p54-ford.pdf">The full paper is available online via the ACM</a>. <br />
<br />
Most of the FARM papers focused more on general analysis of the structure of music than the interpretation of the meaning of specific pieces. I find the general analytic approach fascinating, but as I argue in my paper, I think computational musicology can be more than that.<br />
<br />
I'm grateful to the FARM organisers for accepting a work that is a little loose with the genre conventions of a computer science paper. ICFP is a great conference, but its usual standard of worthwhile research is inherited from mathematics and the sciences. With notable exceptions like <a href="https://www.youtube.com/watch?v=a5Q7td6t-iw">James Noble's work on postmodern programming</a>, I don't see many examples of academics employing computational thinking for humanities research.<br />
<br />
The paper is based on a talk I gave at <a href="https://thestrangeloop.com/">Strange Loop</a> last year called <a href="https://www.youtube.com/watch?v=Qg3XOfioapI">Kolmogorov Music</a>.</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com4tag:blogger.com,1999:blog-8648268364549955959.post-31167177764765884732016-09-19T14:01:00.002+01:002023-10-26T12:05:53.481+01:00Music as code talks<div dir="ltr" style="text-align: left;" trbidi="on">
<ol style="text-align: left;">
</ol>
I've been giving talks about music theory and code for a few years now, so I thought I'd collect them all together in one place. They are based on <a href="http://overtone.github.io/">Overtone</a> and my <a href="https://github.com/ctford/leipzig">Leipzig music composition library</a>.<br />
<ol style="text-align: left;">
<li><a href="https://www.infoq.com/presentations/music-functional-language">Functional Composition</a>, about music theory from sine waves through to canons, given at Lambda Jam 2013 (<a href="https://github.com/ctford/functional-composition">code</a>).</li>
<li><a href="https://www.youtube.com/watch?v=Qg3XOfioapI">Kolmogorov Music</a>, about music and complexity theory, given at Strange Loop 2015 (<a href="https://github.com/ctford/kolmogorov-music">code</a>).</li>
<li><a href="https://skillsmatter.com/skillscasts/7240-keynote-dueling-keyboards">Dueling Keyboards</a>, about temperament and tuning systems, given at Clojure eXchange 2015 (<a href="https://github.com/ctford/dueling-keyboards">code</a>).</li>
<li><a href="https://vimeo.com/channels/flatmap2016/165917501">Klangmeister</a>, about <a href="http://ctford.github.io/klangmeister">my online live coding environment,</a> given at FlatMap 2016 (<a href="https://github.com/ctford/klangmeister">code</a>).</li>
<li><a href="https://www.youtube.com/watch?v=EK4qctJOMaU">African Polyphony and Polyrhythm</a>, about music from the Central African Republic, given at Strange Loop 2016 (<a href="https://github.com/ctford/african-polyphony-and-polyrhythm">code</a>). <a href="https://ctford.github.io/african-polyphony-and-polyrhythm">Slides</a> are online.</li>
<li><a href="https://www.youtube.com/watch?v=Lxmekot9WfA">It Ain't Necessarily So</a>, about the psychology of musical perception, given at Curry On 2018 (<a href="https://github.com/ctford/it-aint-necessarily-so">code</a>). <a href="https://ctford.github.io/it-aint-necessarily-so/slides.html">Slides</a> and <a href="https://ctford.github.io/it-aint-necessarily-so/">demo</a> are online.</li><li><a href="https://www.youtube.com/watch?v=OCYU0LtqRH0">Birdsong-as-code</a>, about the music theory of birdsong, given at Strange Loop 2023 (<a href="https://github.com/ctford/birdsong-as-code">code</a>).</li>
</ol>
If you're interested in my personal music, check out <a href="https://soundcloud.com/ctford/whelmed">Whelmed</a> (<a href="https://github.com/ctford/whelmed">code</a>) or <a href="https://www.youtube.com/watch?v=BRBR45k2Yw4">this performance with keytar accompaniment</a>.<br />
<ol style="text-align: left;">
</ol>
</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com1tag:blogger.com,1999:blog-8648268364549955959.post-62694128800587873782016-07-03T16:57:00.000+01:002016-07-03T16:57:08.848+01:00Falsehoods programmers believe about music<div dir="ltr" style="text-align: left;" trbidi="on">
In the spirit of Patrick McKenzie's great post on <a href="https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/">falsehoods programmers believe about names</a>, I am trying to write an equivalent one for music. Any false assumption that might be made in codifying music is a candidate for inclusion. Suggestions are very welcome.<br />
<ol style="text-align: left;">
<li>Music can be written down.</li>
<li>Okay, maybe not with European notation, but there'll be a specialist notation for that kind of music.</li>
<li>Music is finite in duration.</li>
<li>Music has a composer.</li>
<li>Music is about harmony.</li>
<li>Music uses scales. </li>
<li>Music uses equal temperament.</li>
<li>Music uses tones and semitones.</li>
<li>Music and dance are separate activities.</li>
<li>Playing and listening to music are separate activities.</li>
<li>Musicians can play their part separately from the overall composition.</li>
<li>Music is performed by professional musicians.</li>
</ol>
</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com27tag:blogger.com,1999:blog-8648268364549955959.post-33678292027620449472016-06-26T16:20:00.000+01:002016-06-26T16:20:49.307+01:00GEB<div dir="ltr" style="text-align: left;" trbidi="on">
Douglas Hofstadter's <a href="https://www.amazon.co.uk/Godel-Escher-Bach-Eternal-Golden/dp/0465026567/">Godel, Escher, Bach</a> is one of my favourite books. Commonly referred to as GEB, this book is a mesmerising meditation on consciousness, mathematics and creativity. The central idea is that of a "strange loop", in which the same message is interpreted on multiple semantic levels.<br />
<br />
GEB was the first place I came across the idea of a <a href="https://en.wikipedia.org/wiki/Canon_(music)">musical canon</a>. A canon is a beautifully austere form of composition that was popular in the Baroque period. A canon consists of a <i>dux</i> part which sets out the base melody accompanied by a <i>comes</i> part which is some kind of transformation of the <i>dux</i>.<br />
<br />
Here is the structure of a canon described using the programming language Clojure. <span style="font-family: "courier new" , "courier" , monospace;">f</span> stands for the transformation selected by the composer.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">(<span class="pl-k">defn</span> <span class="pl-e">canon</span> [f notes]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> (->> notes</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> (</span><span class="pl-en" style="font-family: "courier new" , "courier" , monospace;">with</span><span style="font-family: "courier new" , "courier" , monospace;"> (</span><span class="pl-en" style="font-family: "courier new" , "courier" , monospace;">f</span><span style="font-family: "courier new" , "courier" , monospace;"> notes))))</span><br />
<br />
For example, the <i>comes </i>might be formed by delaying the <i>dux </i>by a bar and raising every note by a third. In my talk <a href="https://www.youtube.com/watch?v=Mfsnlbd-4xQ">Functional Composition</a> I show how computer code can be used to explain music theory, focussing on JS Bach's Canone alla Quarta from the <a href="https://en.wikipedia.org/wiki/Goldberg_Variations">Goldberg Variations</a>. Canone alla Quarta is an unusually complex and beautiful canon where the transformation is composed of a delay of three beats (a simple canon), a reflection (a mirror canon) and a pitch transposition down a fourth (an interval canon).<br />
<br />
Here is the transformation from Canone alla Quarta written in Clojure. <span style="font-family: "courier new" , "courier" , monospace;">comp</span><span style="font-family: inherit;"> is a Clojure function for composing multiple transformations together.</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">(defn canone-alla-quarta [notes]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> (->> notes</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> (canon</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> (comp (interval -3) mirror (simple 3))</span><span style="font-family: "courier new" , "courier" , monospace;">)))</span><br />
<br />
I was working on a talk for last year's <a href="http://www.thestrangeloop.com/">Strange Loop programming conference</a> (itself a reference to Hofstadter's work) and I decided that I wanted to create my own canon as a tribute to GEB as a finale. Rather than use an ordinary musical transformation for my <i>comes</i>, I wanted to pick something that spoke to the idea of composing music with computer code. I also wanted to incorporate GEB's theme of interpreting messages on multiple levels.<br />
<br />
I took the letters G, E and B, and used the <a href="https://en.wikipedia.org/wiki/ASCII">ASCII</a> codes that represent these letters as though they were <a href="https://en.wikipedia.org/wiki/MIDI">MIDI</a> pitch codes. This gave me my <i>dux</i>. I then took the same three letters and interpreted them as the musical notes G, E and B. This gave me my <i>comes</i>. I had obtained a canon based not on musical concepts like delay or transposition, but on encoding schemes used in computer programming.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">(defn canone-alla-geb [notes]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> (->> notes</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> (canon</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> #(where :pitch ascii->midi %)</span><span style="font-family: "courier new" , "courier" , monospace;">)))</span><br />
<br />
I elaborated the harmonies provided by this canon into a complete track, composed via computer code. The <i>dux </i>and the <i>comes</i> are joined by various other parts, some using <a href="https://en.wikipedia.org/wiki/Polyrhythm">polyrhythms</a> to generate apparent complexity from underlying simplicity.<br />
<br />
Eventually, the <i>dux</i> and the <i>comes</i> are accompanied by a third canonic voice, in which the names of Godel, Escher and Bach are read out by a text-to-speech program. So the theme of three notes G, E and B becomes a canon of three voices musical, technical and allusive to the three great creative spirits Godel, Escher and Bach.<br />
<br />
<a href="https://soundcloud.com/ctford/geb">Listen to the recording.</a><br />
<br />
<a href="https://github.com/ctford/kolmogorov-music/blob/master/src/kolmogorov_music/geb.clj">Read the code.</a><br />
<br />
<a href="https://www.youtube.com/watch?v=Qg3XOfioapI">Watch the talk.</a></div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com0tag:blogger.com,1999:blog-8648268364549955959.post-45554922368854119352015-05-17T13:14:00.001+01:002015-05-17T13:14:48.569+01:00Lanham on explicit data dependencies<div dir="ltr" style="text-align: left;" trbidi="on">
<blockquote>
Who's kicking who?<br />
<br />
- Richard Lanham, <i>Revising Prose</i></blockquote>
</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com0tag:blogger.com,1999:blog-8648268364549955959.post-8199268853398390162015-01-20T11:56:00.003+00:002015-01-20T11:56:32.092+00:00Korzybski on story points<div dir="ltr" style="text-align: left;" trbidi="on">
<blockquote>
The map is not the territory.
<br />
<br />
- Alfred Korzybski</blockquote>
</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com0tag:blogger.com,1999:blog-8648268364549955959.post-51905249313884295582014-10-19T17:04:00.003+01:002014-10-19T17:04:58.912+01:00Types don't substitute for tests<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
When reading discussions about the benefits of types in software construction, I've come across the following claim:</div>
<div>
<blockquote>
When I use types, I don't need as many unit tests.</blockquote>
</div>
<div>
This statement is not consistent with my understanding of either types or test-driven design. When I've inquired into reasoning behind the claim, it often boils down to the following:<br />
<blockquote>
Types provide assurance over all possible arguments (universal quantification). Unit tests provide assurance only for specific examples (existential quantification). Therefore, when I have a good type system I don't need to rely on unit tests.</blockquote>
This argument does not hold in my experience, because I use types and unit tests to establish different kinds of properties about a program.<br />
<br />
Types prove that functions within a program will terminate successfully for all possible inputs (I'm ignoring questions of totality for the sake of simplifying the discussion).<br />
<br />
Unit tests demonstrate that functions yield the correct result for a set of curated inputs. The practice of test-driven design aims to provide confidence that the inputs are representative of the function's behaviour through the discipline of expanding a function's definition only in response to an example that doesn't yet hold.<br />
<br />
All of the examples that I use in my practice of test-driven design are well-typed, whether or not I use a type system. I do not write unit tests that exercise the behaviour of the system in the presence of badly-typed input, because in an untyped programming language it would be a futile exercise and in a typed programming language such tests would be impossible to write.<br />
<br />
If I write a program using a type system, I still require just as many positive examples to drive my design and establish that the generalisations I've created are faithful to the examples that drove them. Simply put, I can't think of a unit test that I would write in the absence of a type system that I would not have to write in the presence of one.<br />
<br />
I don't use a type system to prove that my functions return the output I intend for all possible inputs. I use a type system to prove that there does not exist an input, such that my functions will not successfully terminate (again, sidestepping the issue of non-total functions). In other words, a type checker proves the absence of certain undesirable behaviours, but it does not prove the presence of the specific desirable behaviours that I require.<br />
<br />
Type systems are becoming more sophisticated and are capable of proving increasingly interesting properties about programs. In particular, dependently typed programming languages like <a href="http://www.idris-lang.org/">Idris</a> can be used to establish that lists are always non-empty or <a href="http://brianmckenna.org/blog/evenodd_agda_idris_haskell_scala">the parity of addition</a>.<br />
<br />
But unless the type system proves that there is exactly one inhabitant of a particular type, I still require a positive example to check that I've implemented the right well-typed solution. And even if the type provably has only one inhabitant, I would still likely write a unit test to help explain to myself how the abstract property enforced by the type system manifests itself.<br />
<br />
A type system is complementary to unit tests produced by test-driven design. The presence of a type system provides additional confidence as to the correctness of a program, but as I write software it does not reduce the need for examples in the form of unit tests.</div>
</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com5tag:blogger.com,1999:blog-8648268364549955959.post-43206102172135882922014-10-12T18:42:00.002+01:002014-10-12T18:42:21.679+01:00Hiding the REPL<div dir="ltr" style="text-align: left;" trbidi="on">
I depend heavily on Clojure's REPL, because it's where I write <a href="https://soundcloud.com/ctford/whelmed">music</a>. Over time, however, I've become less focused on directly interacting with the REPL, and pushed it more and more into the background.<br />
<br />
I use <a href="https://github.com/tpope/vim-fireplace">Vim Fireplace</a>, which gives me the ability to evaluate forms in a buffer by sending them to an nREPL instance. There's also a REPL available within Fireplace, but I find I only use it for simple commands like stopping a piece of music or printing a data structure.<br />
<br />
<a href="https://twitter.com/AidyLewis/status/521281104117248000">Speaking to Aidy Lewis on Twitter today</a>, I've come to realise that there may be two different models for REPL-driven development.<br />
<br />
Aidy described a model where the REPL is ever-present in a split-window. This brings the REPL to the foreground, and makes it conveniently available for experimentation. I would describe this as a side-by-side model.<br />
<br />
On the other hand, I treat the buffer itself as my REPL. I write and refine forms, evaluating them as I go. If I want to experiment, I do so by writing code in my buffer and either evolving it or discarding it. My navigation and interaction are as they would be in any other Vim session, punctuated by occasional re-evaluation of something I've changed. This seems to me more like a layered model, with my buffer on the surface and the REPL below.<br />
<br />
The reason I value this mode of interaction is it makes me feel more like I'm directly interacting with my code. When I make a change and re-evaluate the form, I have the sense that I'm somehow touching the code. I don't have a mental separation between my code-as-text and the state of my REPL session. Rather they're two ways of perceiving the same thing.</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com6tag:blogger.com,1999:blog-8648268364549955959.post-35710055395585092512014-06-07T17:24:00.000+01:002014-06-07T20:21:38.439+01:00Pragmatic people<div dir="ltr" style="text-align: left;" trbidi="on">
She came from Greece. She had a thirst for knowledge.<br />
She studied Haskell at Imperial College -<br />
that's where I caught her eye.<br />
She told me monads are burritos too.<br />
I said, "In that case Maybe I have a beef with you."<br />
She said, "Fine",<br />
and then in thirty metaphors time she said,<br />
<br />
"I wanna live like pragmatic people.<br />
I wanna do whatever pragmatic people do.<br />
Wanna sleep with pragmatic people,<br />
I wanna sleep with pragmatic people like you."<br />
Oh, what else could I do?<br />
I said, "I'll see what I can do."<br />
<br />
I showed her some JavaScript.<br />
I don't know why,<br />
but I had to start it somewhere,<br />
so it started there.<br />
<br />
I said, "Pretend undefined == null"<br />
And she just laughed and said, "Oh you're so droll!"<br />
I said, "Yeah?<br />
I can't see anyone else smiling in here."<br />
<br />
"Are you sure you want to live like pragmatic people?<br />
You wanna see whatever pragmatic people see?<br />
Wanna sleep with pragmatic people,<br />
You wanna sleep with pragmatic people like me?"<br />
But she didn't understand.<br />
And she just smiled and held my hand.<br />
<br />
Rent a flat above a shop,<br />
Cut your hair and get a job.<br />
Write a language in 10 days<br />
and you're stuck with it always.<br />
<br />
But still you'll never get it right,<br />
'cos when you're debugging late at night,<br />
watching errors climb the wall,<br />
if you just used types you could stop it all, yeah.<br />
<br />
You'll never live like pragmatic people.<br />
You'll never do whatever pragmatic people do.<br />
You'll never fail like pragmatic people.<br />
You'll never watch Curry-Howard slide out of view.<br />
Time flies like a Kleisli arrow<br />
because your perspective is so narrow.<br />
<br />
Sing along with the pragmatic people.<br />
Sing along and it might just get you through.<br />
Laugh along with the pragmatic people.<br />
Laugh along even though they're laughing at you.<br />
And the stupid things that you do,<br />
because you think that proof is cool.<br />
<br />
Like a dog lying in a corner,<br />
faults will bite you and never warn you.<br />
Look out they'll tear your insides out.<br />
Pragmatists hate a tourist,<br />
especially one who thinks it's all such laffs.<br />
Yeah and the stains of faulty reasoning<br />
will come out in the maths.<br />
<br />
You will never understand<br />
how it feels to live your life<br />
with no meaning or control.<br />
And I'm with nowhere left to go.<br />
You are amazed untyped languages exist<br />
and they burn so bright<br />
while you can only wonder.<br />
<br />
Why rent a flat above a shop,<br />
cut your hair and get a job?<br />
Dynamic languages are cool.<br />
Pretend you never went to school.<br />
<br />
Still you'll never get it right,<br />
'cos when you're debugging late at night,<br />
watching stacktraces 10 screens tall,<br />
if you had types you could stop it all, yeah.<br />
<br />
Never live like pragmatic people.<br />
Never do what pragmatic people do.<br />
Never fail like pragmatic people.<br />
Never watch Curry-Howard slide out of view,<br />
and on error resume,<br />
because there's nothing else to do.<br />
<br />
With apologies to <a href="http://jarviscocker.net/">Jarvis Cocker</a>.</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com4tag:blogger.com,1999:blog-8648268364549955959.post-18303322852596051672014-04-27T22:10:00.000+01:002014-04-27T22:10:58.709+01:00Microservices relax<div dir="ltr" style="text-align: left;" trbidi="on">
Recently I blogged explaining that I view microservice architectures as <a href="http://literateprogrammer.blogspot.co.uk/2014/03/the-microservice-declaration-of.html">a collection of strategies to independently vary system characteristics</a>. I suggested that these strategies could be framed in terms of a generic architectural constraint:<br />
<blockquote>
X can be varied independently of the rest of the system.</blockquote>
<a href="http://www.markbaker.ca/">Mark Baker</a> pointed out that while I employed the language of architectural constraints, my one <a href="https://twitter.com/distobj/status/445422032050593792">"isn't upon components, connectors, and data."</a> He referred me to <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm">Roy Fielding's excellent thesis</a>, famous for defining the REST architectural style, for a rigorous description of how architectural constraints can be described.<br />
<br />
Thinking about the problem further, I considered what kind of data flows through the microservice systems I've worked with. I realised that many of the qualities that characterise a microservice architecture are not visible during a single system run. Microservice architecture can only be explained in terms of components, connectors and data if the constituent services are themselves regarded as data, and the processes that evolve them as components and connectors.<br />
<br />
I don't think that's too much of a stretch. For systems that are deployed via continuous delivery pipelines, the automated deployment infrastructure is part of the system itself. If the build is red, or otherwise unavailable, then the production system loses one of its most important capabilities - to evolve. The manner in which the code/data that specifies the behaviour of the system evolves can be subject to similar constraints to the introduction of novelty into conventional system data.<br />
<br />
Viewed that way, it becomes apparent that the constraint I proposed was the inverse of what it should have been, namely:<br />
<blockquote>
X's variation must be synchronised with the rest of the system.</blockquote>
Monolithic architectures are subject to this constraint for various values of 'X'. Microservice architectures are produced via the relaxation of this constraint along different axes.<br />
<br />
The process of starting with a highly constrained system and gradually relaxing constraints that it is subject to is the opposite of the approach Fielding advocates in his thesis. When deriving REST, <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm#sec_5_1_1">Fielding starts with the null style</a> and layers on six constraints to cumulatively define the style.<br />
<br />
However, the microservice style is a reaction against highly-constrained monolithic architectures, so it has taken the opposite approach, which is to peel away constraints layer by layer in order to introduce new flexibility into the system.<br />
<br />
For example, a monolithic system might consist of five functional areas, all integrated within a single codebase. For a functional change to make its away to production, the data describing the system's behaviour (its source code) is modified, checked-in and moves through the deployment pipeline into production. This must happen for all five functional areas as one, because the transaction of deployment encompasses all five pieces of data.<br />
<br />
This system is devolved into a system of five microservices, each living in an independent source control root. A change to any one of those services can be checked in and move through its pipeline to production independently of the others. The deployment transactions are now smaller in scope, encompassing only a single service at a time.<br />
<br />
The change in the scope of a single deployment generates options for the maintainers of the system. They can now release changes to one of the five functional areas without downtime to any of the other four.<br />
<br />
But, as is the case with real options, this comes at a price. If the five areas interact, their APIs need to be versioned and kept backwards compatible. Changes that crosscut multiple services need to be carefully orchestrated to avoid breaking changes, whereas in the original monolithic system such refactorings might have been made with impunity.<br />
<br />
Generally speaking, for each constraint that is relaxed in a microservice architecture, there is a corresponding real option generated and a corresponding coordination cost incurred. For good explanations on real options and how they apply to software development, see the various works of <a href="http://theitriskmanager.wordpress.com/">Chris Matts</a>, including the book he wrote with <a href="https://twitter.com/OlavMaassen">Olav Maassen</a> - <a href="http://commitment-thebook.com/">Commitment</a>.<br />
<br />
I think that it's a productive approach to describe microservice architectures in terms of the the relaxation of architectural constraints relative to a monolithic architecture. This provides us with a model not only to analyse the runtime properties of the system, but also its evolution and the options it affords its maintainers.</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com0tag:blogger.com,1999:blog-8648268364549955959.post-29951212849296970872014-04-13T21:58:00.002+01:002014-04-13T21:58:21.379+01:00Review: Adaptive web design<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="http://www.amazon.com/Adaptive-Web-Design-Experiences-Progressive/dp/098358950X"><i>Adaptive web design</i> by Aaron Gustafson</a> is a good book on an important topic. How the web evolves, and how it caters to the broad spectrum of its users, that now make up the majority of the population in affluence societies, is a crucial consideration.<br />
<br />
The book is a curious mixture of tips, philosophy and polemic. Gustafason includes specific technical detail, for example microformats, structuring CSS selectors and <a href="http://www.w3.org/TR/wai-aria/roles">ARIA roles</a>.<br />
<br />
He also treats progressive enhancement as a moral issue, which is both a reasonable point of view and a rhetorical device. For example, he describes the technique of graceful degradation as "fault tolerance's superficial, image-obsessed sister" and that eventually "smart folks working on the web began to realize that graceful degradation's emphasis". He thus frames issues in a way as to try and ensure the reader's conclusion cannot be anything other than agreement, which is a time-honoured technique of polemic.<br />
<br />
This book is a good overview of a topic that can fade into the background under the pressure of deadlines and corporate disregard for users, so it's great that Gustafason has put his energy into a book that software developers can rally around.<span class="Apple-tab-span" style="white-space: pre;"> </span><br />
<br />
However it has two serious flaws, one of omission and one of commission, that make it less useful to a contemporary software team than it otherwise might have been.<br />
<br />
The flaw of omission is that it does not engage with the recent crop of client-side MVC applications. <i>Adaptive web design</i> was published in 2011, two years after the release of Google's framework <a href="http://www.angularjs.org/">AngularJS</a>, but it contains no mention of this style of application development, which challenges some of the fundamental assumptions of the progressive enhancement movement.<br />
<br />
Client-side MVC is, literally speaking, progressive enhancement, but it's not of the sort that Gustafason lionises. These applications, usually constructed with the help of extensive JavaScript frameworks, place rich interactivity on top of APIs, where as traditional progressive enhancement layers unobtrusive interactivity on top of HTML.<br />
<br />
The major difference with Gustafason's vision is that the HTML comes on top of the JavaScript rather than the other way around. Data is pulled into the browser by Ajax calls, before being rendered using client-side templating engines directly into the DOM. This allows the application to keep a model of the application's state and rules in the browser, and to update that model and apply those rules in the interests of a rich experience without the latency of calls back to the server.<br />
<br />
I don't know how such applications should be regarded. They don't, for example, allow users to access content if JavaScript is unavailable. This may or may not have the same urgency as it did when <i>Adaptive web design</i> was written three years ago, but in this book at least, Gustafason is unable to give us guidance.<br />
<br />
That substantially diminishes the relevance of this book, as the most pressing question most contemporary teams will have about progressive enhancement is how it relates to the major movement in the JavaScript world towards client-side applications.<br />
<br />
The flaw of commission, that makes it difficult to treat <i>Adaptive web design</i> as even a point-in-time reference, is the way that Gustafason uses definitions and terminology.<br />
<br />
One such example is the term "accessibility". Gustafason takes the perfectly reasonable view that accessibility is a broader issue than simply people who use assistive technologies. The problem with this definition is that it's incompatible with the common usage of the term.<br />
<br />
Interestingly, this is an example of a failure of progressive enhancement of language itself. Specialising normal language into <a href="http://legal-dictionary.thefreedictionary.com/Term+of+Art">terms of art</a> allows the common punter to at least get the gist of what's being discussed. Expanding the definition of terms is much more confusing, because it leads to sentences where the word simply does not make sense under its old definition.<br />
<br />
But all this is bunk when we get to the "Accessibility" chapter itself, where Gustafason reverts to the classic idea of accessibility being about people with physical needs that might impede them from accessing content. I guess Gustafason just found it too difficult to sustain his own revised definition.<br />
<br />
But the terminology that most grates with me is the assertion that progressive enhancement can be defined as "fault tolerance". The most obvious issue is that there are many instances of progressive enhancement that don't involve "fault". For example, it doesn't make sense to talk about the fault of my browser not specifically recognising the <a href="http://www.gmpg.org/xfn/">XFN</a> microformat, but my ability to browse the page regardless is, as Gustafason points out, a classic case of progressive enhancement.<br />
<br />
The greater problem is that there are times when fault tolerance and progressive enhancement are actively opposed. For example, when browsers tolerate malformed markup it makes it more difficult for later changes to e.g. HTML to be added that bring new features because they are more likely to clash with the "fault tolerant" HTML implementations of older browsers.<br />
<br />
The pity is that I don't think this particular confusion is necessary. Progressive enhancement is a very specific kind of fault tolerance, and Gustafason's attempt to compare progressive enhancement to Galapagos finches is entirely misleading, if entertaining. If Gustafason had chosen an alternative term to "fault tolerance", perhaps "graceful gradation" (to contrast with "graceful degradation") then the same point could have been made with much greater clarity.<br />
<br />
<i>Adaptive web design</i> is a solid book, written with energy and with a strong sense of the historical development of the web. If a team reads this book together, it will spark a lot of important discussions, but its occasional problems with terminology and its failure to engage with client-side MVC means it will not settle them.</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com2tag:blogger.com,1999:blog-8648268364549955959.post-48087801938239911162014-03-16T23:02:00.000+00:002014-03-16T23:02:40.645+00:00The microservice declaration of independence<div dir="ltr" style="text-align: left;" trbidi="on">
Recently, <a href="http://martinfowler.com/articles/microservices.html">an article on microservice architectures</a> by my colleagues <a href="http://martinfowler.com/">Martin Fowler</a> and <a href="https://twitter.com/boicy">James Lewis</a> has caused some mild controversy in the Twittersphere. For example, <a href="https://twitter.com/kellabyte">Kelly Sommers</a> said:<br />
<blockquote>
<a href="https://twitter.com/kellabyte/status/445263201563713536">I’m still trying to figure out what is distinct period. Maybe I’ll accept the name once I see a diff</a></blockquote>
I have yet to find someone who believes it a bad idea to build systems from small, loosely-coupled services that can be independently changed and deployed. The hesitance that some have shown for the term "microservice architecture" derives from their doubt that the term adds anything above and beyond the well-worn term <a href="http://www.infoq.com/articles/tilkov-10-soa-principles">service-oriented architecture</a> (here described by <a href="https://twitter.com/stilkov">Stefan Tilkov</a>).<br />
<br />
This is a serious charge. The value that concepts like "microservice" have is in their use for communication and education. If all a term does is to add to the menagerie of poorly-defined architectural buzzwords, then we might be better off sticking with the poorly-defined buzzwords we're already familiar with.<br />
<br />
On the other hand, that ideas of such uncontroversial value are still worthy of discussion is some indication that there is room for improvement in the current conceptual landscape.<br />
<br />
In order for "microservice architecture" to be a useful way of characterising systems, it needs an accepted and well-understood definition. <a href="https://twitter.com/distobj">Mark Baker</a> laments the lack of a precise description, because he feels its absence compromises the value of the term:<br />
<blockquote>
<a href="https://twitter.com/distobj/status/445235879418683392">Really unfortunate that the Microservices style isn't defined in terms of architectural constraints #ambiguity #herewegoagain</a></blockquote>
In their article, Martin and James have this to say about the style's definition:<br />
<blockquote>
We cannot say there is a formal definition of the microservices architectural style, but we can attempt to describe what we see as common characteristics for architectures that fit the label.</blockquote>
I'm still optimistic that a clearer picture is yet to emerge, though it might not be as unambiguous as <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm">the REST architectural style</a>, which is defined in terms of six constraints.<br />
<br />
My own opinion is that microservice architectures can be understood through a single abstract architectural constraint which can be interpreted along many different degrees of freedom.<br />
<blockquote>
X can be varied independently of the rest of the system.
</blockquote>
What might X be? There are many qualities that might be locally varied in an architecture to beneficial effect. Here are some ways we could introduce novelty into a service without wishing to disturb the greater system:<br />
<ul style="text-align: left;">
<li>Select implementation and storage technology</li>
<li>Test</li>
<li>Deploy to production</li>
<li>Recover from failure</li>
<li>Monitor</li>
<li>Horizontally scale</li>
<li>Replace</li>
</ul>
<div>
There are other aspects of systems that are harder to formalise, but which the human beings that work on our software systems might wish to do with without being troubled by the fearsome complexity of the whole architecture:</div>
<div>
<ul style="text-align: left;">
<li>Understand</li>
<li>Describe</li>
<li>Work on with a close-knit team</li>
<li><a href="http://vimeo.com/29719577">Share two pizzas in celebration of a new feature</a></li>
</ul>
<div>
I don't wish to imply that this is an exhaustive list. To me, the commonality lies in the concept of independent variation, not the kinds of variation themselves.</div>
</div>
<div>
<br /></div>
<div>
Monolithic architectures are monolithic in the sense that they have few or no degrees of freedom. This may be a perfectly valid choice. The more degrees of freedom available in a system, the more coordination is required to keep the disparate parts of the system working together.<br />
<br />
For example, a human shoulder is well articulated. This makes it an amazingly flexible subsystem. I can touch my toes, reach above my head and even scratch my own back. <strike>On the other hand</strike>, the infrastructure required to support such free movement is quite complex and therefore fragile. My right shoulder is currently broken. However, if it were simpler and flexed only in one direction there is a good chance that it would have been able to withstand the impact undamaged.<br />
<br />
(Luckily, my shoulders are able to fail independently and therefore my left is still in good working order!)</div>
<div>
<br class="Apple-interchange-newline" />
Systems suffer when architectures are not articulated in the directions they need to be. We may wish to horizontally scale, but are unable to isolate two parts of the system which exhibit very different behaviour under load. We might wish to enable two teams of people in different countries to work on different features, but find that they step on each others' toes because they are hacking on the same deployable artifact.</div>
<div>
<br /></div>
<div>
I value the concept of independence because it gives me a way of evaluating architectural choices beyond "microservices are good" and "microservices are bad". Degrees of architectural freedom are valuable, but they may or may not be worth the effort required to sustain them.</div>
<div>
<br /></div>
<div>
For example, independent deployment is useful because you don't have to stop the whole world to introduce a change in a small part of the system. To steal an example from conversation with James, a bank would not want to stop processing credit card transactions in order to release a new feature on its mobile app.</div>
<div>
<br /></div>
<div>
However, once you move away from monolithic deployments, API versioning and backwards compatibility are required to coordinate releases. That can be hard work, and if you never need to exercise your ability to upgrade a part of the system in isolation, then that work might be wasted.</div>
<div>
<br /></div>
<div>
I've worked on a system which had two services that talked to each other through a poorly designed API. Every time one service changed, the other had to as well. Backwards compatibility was almost impossible to maintain due to some very bad design choices we made early on. Consequently, we almost always had to do double deployments.</div>
<div>
<br /></div>
<div>
In the end we merged the two repositories holding the two services and started deploying them simultaneously. We lost the ability to take one to production without the other, but in our situation that was more a trap than a valuable feature. The two services continued to be distinct as runtime entities, but they now evolved along a single timeline.</div>
<div>
<br /></div>
<div>
This brings me to how I think service-oriented architecture might relate to microservice architecture. Stefan's fourth principle of service-oriented architectures is that services be autonomous:</div>
<div>
<blockquote>
Services can be changed and deployed, versioned and managed independently of each other.
</blockquote>
Personally, I see that as the principle attribute from which all others derive. However, under that definition the two services that my team merged no longer qualify as such, at least not in terms of traditional service-oriented architecture. We consciously gave up the ability to change, deploy and version them independently.</div>
<div>
<br />
However, I would argue they still qualify as micro-services. We could understand and discuss them separately. They had distinct responsibilities. We could monitor, diagnose and restart them separately in production. If we'd wanted to horizontally scale one without the other we could have. It just happened that on the evolution axis we did not require independence.<br />
<br />
According to the terms of the suggested constraint family that "X can be varied independently of the rest of the system", traditional service-oriented architecture could be seen as a specific set of values for X. Some of the degrees of freedom that were important 10 years ago are less important now. Most are still relevant. New kinds of variation that have risen to prominence.<br />
<br />
I don't think that microservice architectures are a break from the past. I would probably characterise the movement as a kind of neo-SOA that is trying to get back to the roots of the original movement. But I do think that microservice architectural analysis has a different and more general preoccupation than service-oriented architectural analysis because it revolves around the notion of independent variation itself, rather than prescribing certain kinds of independence.</div>
</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com4tag:blogger.com,1999:blog-8648268364549955959.post-4303407102402341592013-08-20T11:53:00.000+01:002013-08-20T12:03:29.381+01:00Real live music<div dir="ltr" style="text-align: left;" trbidi="on">
<blockquote>
<i>
“Yes,” said Golg. “I have heard of those little scratches in the crust that you Topdwellers call mines. But that’s where you get dead gold, dead silver, dead gems. Down in Bism we have them alive and growing. There I’ll pick you bunches of rubies that you can eat and squeeze out a cupful of diamond juice. You won’t care much about fingering the cold, dead treasures of your shallow mines after you have tasted the live ones in Bism.”</i>
<br />
<br />
- C. S. Lewis, <a href="http://www.amazon.com/Silver-Chronicles-Full-Color-Collectors-Edition/dp/0064409457">The Silver Chair</a></blockquote>
I am fascinated by our collective fascination with live music. Why does it make a difference whether we access a piece of music through the thick dark air of a crowded pub or via a commercial recording and distribution process?<br />
<br />
I question the mechanism, but I have no doubt as to the potency of its effect. I revel in live performance. I even listen to live albums - which I simultaneously find conceptually daft and utterly compelling.<br />
<br />
My own pet theory is that liveness (life?) is a perception of the possibility of being otherwise. As each note is struck, plucked or sung, the audience knows that though the piece may be written a certain way, in a live rendition <a href="http://www.youtube.com/watch?v=pLLoNi4qHPg">it ain't necessarily so</a>. Any given note has the potential to be substituted, varied or flubbed.<br />
<br />
But once a note has transformed from a musical intention into a perturbation of air and propagated through the room at (approximately) 340.29 metres per second, it drops down dead to the floor. The high note has been hit, or not. The <a href="http://www.freesound.org/people/TicTacShutUp/sounds/426/">high hat</a> has been hit, or not. The blue note has been blown and the possibility of being otherwise has ceased.<br />
<br />
Music-as-code offers a variation on this theme. If I, for example, describe <a href="https://github.com/ctford/whelmed">my music</a> with <a href="http://clojure.org/">Clojure</a> and <a href="https://github.com/overtone/overtone">Overtone</a>, then I can render my music to my speakers. But I can also alter it at will. If I take care to <a href="http://dddcommunity.org/learning-ddd/what_is_ddd/">directly represent the deep structure</a> of my music in code, then I can easily make changes to instrumentation/tempo/swing that even editors and sequencers of binary sound files cannot.<br />
<br />
I can bring the power of iteration, of version control, of collective review, of unit testing, of my combined past and present selves, to bear on a piece of music. The possibility of being otherwise is preserved.<br />
<br />
Music created with Overtone isn't just live - in a sense it's immortal. What's more, we now have the perfect retort to the unkind critic - "Pull requests accepted".</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com3tag:blogger.com,1999:blog-8648268364549955959.post-31572564594976226272013-06-12T19:53:00.000+01:002013-06-12T19:53:50.955+01:00Chomsky on static typing<div dir="ltr" style="text-align: left;" trbidi="on">
<blockquote>
<i>Colorless green ideas sleep furiously.</i><br />
<br />
- Noam Chomsky, Syntactic Structures</blockquote>
</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com3tag:blogger.com,1999:blog-8648268364549955959.post-73982020270314081372013-05-25T10:20:00.000+01:002013-05-25T10:20:23.435+01:00Goldilocks services<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="p1">
Github is a document store. Each repository is a document. Transactions are easy to execute within a</div>
<div class="p1">
document and only eventually consistent across documents.</div>
<div class="p2">
<br /></div>
<div class="p1">
If your documents are too large, the scope of your transactions will be excessive, and you will get contention. If your documents are too small, the scope of your transactions will be insufficient, and you will be forced to manage changes across multiple transactions.</div>
<div class="p2">
<br /></div>
<div class="p1">
Each service is an aggregate root, so design them so that they can be evolved independently. </div>
</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com0tag:blogger.com,1999:blog-8648268364549955959.post-79085418238382049412013-05-18T15:08:00.000+01:002013-05-21T07:33:24.111+01:00Second order poka-yoke<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="http://en.wikipedia.org/wiki/Poka-yoke">Poka-yoke</a> is the practice of "mistake-proofing" - designing interfaces in ways that prevent users from making mistakes.<br />
<br />
Continuous integration is a form of poka-yoke for software development. Developers are prevented from releasing (obviously) broken software to production by a battery of tests.<br />
<br />
However, poka-yoke is about designing to mitigate specific user mistakes (at least as I understand <a href="http://en.wikipedia.org/wiki/Shigeo_Shingo">Shingo</a>'s definition). In other words, it's about putting up safety barriers between users and anticipated failure modes.<br />
<br />
How much better to develop in a software ecosystem that discourages not only the occurrence of mistakes, but encourages the growth of designs that do not admit the possibility of these mistakes.<br />
<br />
My colleague Gurpreet Luthra cites <a href="http://techie-notebook.blogspot.com/2012/07/poka-yoke-applying-mistake-proofing-to.html">the example of making heat-resistant bolts a different size to ordinary ones</a> so that they cannot be accidentally interchanged. However, were the design to be simplified so that the engine was a single piece of metal, bolts would not be required at all.<br />
<br />
Sometimes simplification is possible. Other times essential elements of the domain force us to introduce scope for error. Perhaps engines could be constructed differently to simplify assembly. Perhaps not. The only way to tell the difference is to be constantly vigilant for the possibility of eliminating accidental complexity.<br />
<br />
Having thorough test coverage is good. Preventing the complexity that disrupts developers' ability to comprehend the system well enough to avoid mistakes is better.<br />
<br />
Simple designs can not only make mistakes avoidable, they can make them literally inconceivable. By that construction, you could consider a disciplined pursuit of simplicity second-order poka-yoke.</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com0tag:blogger.com,1999:blog-8648268364549955959.post-82501601094610195972013-05-11T09:30:00.001+01:002013-05-11T09:30:26.519+01:00Fixation and variability<div dir="ltr" style="text-align: left;" trbidi="on">
While explaining <a href="http://nealford.com/memeagora/2013/01/22/why_everyone_eventually_hates_maven.html">Why Everyone (Eventually) Hates (or Leaves) Maven</a>, Neal Ford divides extensible software into two categories: composable and contextual.<br />
<br />
Contextual software is best exemplified by plugin architectures - new features are added by slotting code into explicitly provided extension points. Composable software is best exemplified by the Unix ecosystem - new features are added by wiring together a combination of existing and custom components.<br />
<br />
So what is it about contextual systems that provokes such hatred?<br />
<br />
The key problem when making software extensible is where to put the variability. I have a fixed set of core features, but I also want others to contribute extra features that vary depending on their needs. How do I bind the fixed and variable functionality together so that they form a coherent system?<br />
<br />
<a href="http://maven.apache.org/">Maven</a> accommodates variability like a classic contextual system. The promise (and the curse) of Maven is the consistency of its build lifecycle. There is a place for everything, and everything must be in its place. Maven affords very little flexibility on the relationship between variable additions and its fixed feature set.<br />
<br />
<a href="http://rake.rubyforge.org/">Rake</a>, on the other hand, empowers developers by allowing extension via a general purpose programming language. Rake qualifies as a composable system because any means of adding new functionality that can be expressed in Ruby can be used to extend it.<br />
<br />
In composable systems, orchestration is variable. In contextual systems, orchestration is fixed. Keeping the means of combination in <a href="http://www.catb.org/jargon/html/U/userland.html">userland</a> is an additional burden on those who would extend your system, but it's more than offset by the flexibility and power they enjoy in the long term.<br />
<br />
Your software will grow, but your relationship with Maven won't. <strike>If</strike> When that happens, it's time to leave.<br />
<br />
<br /></div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com0tag:blogger.com,1999:blog-8648268364549955959.post-10419035418538486142012-12-16T12:02:00.000+00:002012-12-16T12:02:14.309+00:00Dickens on Ruby on Rails<div dir="ltr" style="text-align: left;" trbidi="on">
<blockquote>
<i>There is much good in it; there are many good and true people in it; it has its appointed place. But the evil of it is that it is a world wrapped up in too much jeweller's cotton and fine wool, and cannot hear the rushing of the larger worlds, and cannot see them as they circle round the sun. It is a deadened world, and its growth is sometimes unhealthy for want of air.</i>
<br /><br />
- Charles Dickens, Bleak House
</blockquote>
</div>ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com1tag:blogger.com,1999:blog-8648268364549955959.post-46777575779725523122012-08-17T22:30:00.000+01:002012-08-17T22:30:37.763+01:00It's all production<div dir="ltr" style="text-align: left;" trbidi="on">
Every now and then, I come across a discussion about whether test code deserves as much care as production code. I understand what is meant (the answer is yes), but I don't accept the premise of the question.<br />
<br />
All code is production code. Deployment scripts, tests, database migrations and Java controllers all play a part in delivering value to the users. If a piece of code really has no effect on production, why was it ever written?</div>
ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com4tag:blogger.com,1999:blog-8648268364549955959.post-15846149916861565282012-06-20T10:20:00.001+01:002012-06-20T10:20:44.933+01:00Einstein on Clojure<div dir="ltr" style="text-align: left;" trbidi="on">
<blockquote>
<i>It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.</i><br />
<br />
- Albert Einstein, The Herbert Spencer Lecture, delivered at Oxford (10 June 1933)</blockquote>
</div>ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com4tag:blogger.com,1999:blog-8648268364549955959.post-33744098175301573632012-06-16T14:09:00.001+01:002012-06-16T14:09:42.610+01:00It's time<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="http://euroclojure.com/2012/">EuroClojure 2012</a> was almost a month ago. Now that my thoughts have settled I've made an attempt to distill what the event meant to me.<br />
<br />
First of all, I was impressed at the intellectual engagement of the speakers and attendees. This is a grown-up community. Talks were less about "How do I do X in Clojure" and more about using the language to move the industry forward.<br />
<br />
If you're a developer who's interested in the avant-garde, you should be paying attention to what this community is coming up with. I left feeling that Clojure is less a programming language and more a delivery mechanism for the Clojure philosophy.<br />
<br />
But what is the Clojure philosophy?<br />
<br />
Whether it was <a href="https://twitter.com/richhickey">Rich Hickey</a> railing against in-place database updates, <a href="https://twitter.com/cassieldotcom">Nick Rothwell</a> animating art, <a href="https://twitter.com/webermaximilian">Max Weber</a> explaining CQRS or <a href="https://twitter.com/pingles">Paul Ingles</a> exploring event-sourced analytics, an underlying theme was that time deserves first-class representation in interesting systems.<br />
<br />
Even more directly, <a href="https://twitter.com/otfrom">Bruce Durling</a>'s Incanter incantations and <a href="https://twitter.com/rosejn">Jeff Rose</a> and <a href="https://twitter.com/samaaron">Sam Aaron</a>'s waveform wizardry manipulated time-series data before our eyes to great effect.<br />
<br />
In an age where we can no longer assume that the universe marches to the beat of a single CPU, implicit time-management is no longer a convenience but a crippling limitation.<br />
<br />
So when someone asks me "Should I adopt Clojure yet? What is the killer application?" I will answer them:<br />
<br />
It's time.</div>ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com0tag:blogger.com,1999:blog-8648268364549955959.post-40114987683153243112012-06-10T11:19:00.000+01:002012-06-10T11:19:57.027+01:00Unthreading<div dir="ltr" style="text-align: left;" trbidi="on">
Multi-threading is hard. To fully understand a piece of multi-threaded code the reader must mentally interleave two or more competing timelines. Our poor primate brains have enough trouble managing one thread of execution.<br />
<br />
But what if we didn't have to carry around the idea of time at all? What if we could move about the codebase without having to mentally pause, play, fast-forward and rewind a hypothetical execution instance of the program?<br />
<br />
The venerable and venerated <a href="http://structure%20and%20interpretation%20of%20computer%20programs/">Structure and Interpretation of Computer Programs</a> comments with sadness on the entrance of the mutation-snake into the Lisp Garden of Eden: <br />
<blockquote>
With objects, we must be concerned with how a computational object can change and yet maintain its identity. This will force us to abandon our old substitution model of computation in favor of a more mechanistic but less theoretically tractable environment model of computation. The difficulties of dealing with objects, change, and identity are a fundamental consequence of the need to grapple with time in our computational models. These difficulties become even greater when we allow the possibility of concurrent execution of programs.
</blockquote>
Immutability makes code easier to understand because it allows the reader to use a simpler mental model. <br />
<br />
Code that exclusively employs immutable (immortal) variables is effectively <i>unthreaded</i> from the reader's point of view. The meaning of the code is no longer dependent on time at all. The value of a symbol is a function of its definition - history is irrelevant.<br />
<br />
Single-threaded code is easier to understand than multi-threaded code. But unthreaded code is best of all. </div>ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com2tag:blogger.com,1999:blog-8648268364549955959.post-48929684084286995452011-11-25T13:05:00.000+00:002011-11-25T13:05:53.668+00:00Overlapping magisteria<div dir="ltr" style="text-align: left;" trbidi="on">I've written that <a href="http://literateprogrammer.blogspot.com/2011/05/tdd-is-from-venus-bdd-is-from-mars.html">unit tests and acceptance tests represent two non-overlapping magisteria</a>. I've realised I need to revise this.<br />
<br />
Unit tests and acceptance tests specify internal and external behaviour respectively. There is no inherent reason why external behaviour should be trivially translatable to internal behaviour, because the structure of the internals need not bear any resemblance to the model the system presents to the world.<br />
<br />
However, that is only true of systems in general. Systems produced using a domain-driven design methodology deliberately enforce structural similarity between the system's implementation and the domain it represents.<br />
<br />
Because of this, it's possible to make inferences between external and internal properties of the system. For example, if a <pre>LoginFailure</pre>object returns <pre>true</pre>to the method <pre>shouldLockAccount()</pre>when its <pre>int</pre>field called <pre>previousAttempts</pre>is <pre>2</pre>or greater, there's a good chance that if a user thrice enters the wrong password that their account will be disabled.<br />
<br />
If we know something about a well-designed system's external behaviour, we can generally extrapolate about its internal behaviour (and vice-versa).</div>ctfordhttp://www.blogger.com/profile/05464902188219000642noreply@blogger.com0