I just figured out why callback style APIs are harder to test (and harder to deal with in general). It’s because they break one of the essential ingredients of a hexagonal architecture: the middle hexagon should be independent of the adapters. Here’s why…

In order to use a callback API, one or more of our application classes must implement the callback method(s), and must therefore conform to some abstraction defined by the API’s provider. So our classes must depend on the API. Which means that the API can’t be easily mocked or stubbed. We have to treat our callback objects as being part of the adapter for that API, and test the rest of the application by mocking or stubbing them.

In fact, now I’ve written it down it’s trivially obvious, and hardly worth saying.

As you were.

Warning: academic theorizing and hypothesizing follow. Oh, and half-baked pontification.

I just finished refactoring reek to drive in a major new chunk of functionality (configuration files) which I’ll release soon, when I’ve had time for some thorough testing.

The refactoring needed to accommodate the change was huge, occupying much of my free time over the course of two months. Pretty much the whole of the tool’s original architecture has been revised. Why so big and so complex? Because the original code relied heavily on constants and class methods; they helped me get the early versions written quickly, but they represented a significant barrier to long-term flexibility. I’ve been wondering why that should be; why do constants and class methods stand in the way of adaptable test-driven code?

I think the answer lies in viewing the application through the lens of Hexagonal Architecture. Let me explain…

It seems to me that constants, global variables, classes, class methods, etc all live in a space that’s “anchored” to the runtime environment, which is itself a singleton. Anything anchored to that singleton is going to hinder the independence and isolatedness of unit tests, and also reduce the application’s flexibility in well-known ways. So far so standard. Now, suppose we model the singleton as a notional point that is external to the application. Hexagonal Architecture tells us we must access the singleton via an Adapter — in this case, an Adapter provided by the programming language and/or runtime. I’ll refer to the singleton as the application’s Anchor, and therefore claim that it is accessed through language features in an Anchor Adapter.

Now, I believe that the Domain Middle should not depend directly on Adapters. So any code that makes direct use of the Anchor Adapter must therefore be considered outside of the Domain Middle, and hence part of an Adapter — and hence also inherently outside the space where unit tests live comfortably.

Which is why constants and class methods add friction to unit testing.

Or rather: This model fits nicely with my penchant for Hexagonal Architecture, and lets me justify my unease at testing in and around class methods. And probably adds nothing to our understanding of software development.

Dave Verwer at NWRUG has been crazykind enough to invite me to re-run a version of my hexagonal architecture workshop this month at the MDDA. This is the session that, two years ago, had seventy of the world’s leading agilists standing on one leg. If you’re in Manchester on the evening of November 20th, why not hop along?

In Edge Case Bill de hÓra points out, as does Tony Coates, that a DataClass is not always a bad idea. Both are responding to an assertion made by Martin Fowler, that a DataClass is usually a sign of poor design. And both use serialised objects as a counterexample. I guess I’ve never thought of serialised objects as capable of having behaviour (since they generally live in media that don’t support it); and so perhaps I don’t even think of them as being sufficiently objecty to even count in the discussion. So to me it’s a non-disagreement.

More interesting to my eyes is Bill’s use of the term “application edges”:

“But when it comes to working at the application edges – at the network boundary, over HTTP, between applications, intra-app messaging – well, “I have a doubt.” [...] There’s a lot of system edges these days, and agreeing where the edges are is hard.”

There are indeed a lot of edges these days. And trying to think about them in the context of a layered model of the application’s architecture is likely to cause brain-ache. Which is why I keep pushing the hexagonal architecture model. It moves the application and domain objects into the centre, and surrounds them with adapters that connect to the rest of the world. In this architecture, the “edges” have a natural symmetry, and their role in the application becomes easier to visualise.

So in the context of hexagonal adapters, some of Bill and Tony’s data classes will likely be GoF Mementos, others are probably Whole Value objects, and the rest probably really are blobs of data. I would expect the Whole Value objects to have behaviour, but not the others. But the real point, for me, is that the hexagonal architecture approach makes the use of these patterns clearer…

During my ‘hexagonal architecture’ session at XPday Benelux, the discussion gave me some clues as to why I feel the “standard” layered architecture model is sub-optimal: I realised that I feel as if I’m looking at a picture of a pile of stuff from the side. Contrast this with a hexagonal model of the same system, in which I feel as though I’m looking down on the picture.

Why is this important? And what relationship does it have to being agile?

The answer, I believe, lies in Lakoff’s theory that metaphor shapes much of our thinking. When I look at any architecture model I subconsciously impose a point of view on the picture, because my mind relates what I see now to previous experiences. A layered model “looks like” a pile of books or building bricks; a hexagonal model “looks like” an island on a map (another metaphor in itself!) or a table with chairs arranged around it. The choice of metaphor is made deep in my perceptual system, helping me to make sense of anything I see. And once the metaphor has been selected, my mind will then automatically supply it with a whole load of related beliefs, many learned as a baby. Among these are the effects of gravity and mass, together with related implications of downward dependency.

These associations cause me to believe that the things at the bottom of the pile are hard to move or change. Whereas in the hexagonal view I instinctively feel the system’s components are more loosely coupled – perhaps because they are associated only by proximity, and not by gravity.

So because of these deep-seated metaphorical associations, maybe we build less adaptable systems when we think of them in layers…?

Last month Alistair began using a new name for the Hexagonal Architecture pattern – he now calls it the Ports and Adapters Architecture.

Now I believe that names are immensely important, and so I applaud Alistair’s search for the right name for this concept. But for some reason PortsAndAdapters rings no bells for me. Maybe I’ve simply become used to the old name, or maybe I’m averse to change. (That would be ironic for an agile coach, but I guess we all have comfort zones…) So what is it with this new name – or what does the old name have instead?

Obviously I have no quibble with the ‘Adapters’ part, because I’ve been using that term for long enough myself. But ‘Ports’? From that term I would tend to infer a fixed set of pre-existing conduits, sending and receiving signals from something specific. Whereas I tend to think of a hexagonal architecture as something much more flexible, able to post adapters (more on this soon) to a less prescribed set of real-world entities.

I tend to view the physical world as an amorphous soup to which the software system can connect adapters in fluid ways. I know that’s inherently impractical, but it makes for a great mental model when whiteboarding a system. And I think this gets near to the crux of my unease at the new name. I think of a hexagonal architecture as a whole-system view, whereas PortsAndAdapters seems to look only at the system’s connections with the physical world. The key, for me, is the symmetry, and the consequent removal of the constraints of the layered model. I don’t care about the Ports; I’m interested instead in the central core where the knowledge lives, and the ability to interact with that core in flexible ways. To me, hexagonal architecture is all about inner and outer, core and periphery, business knowledge and interfaces.

Why so much about just a name? After all, it’s clear that neither name comes close to expressing the above…

I like simple rules of thumb that will help developers create better, more adaptable designs. So here’s another: Never throw an exception. And if someone throws one at you, catch it immediately and don’t pass it on.

Why? Aren’t exceptions one of the mainstays of modern programming? Joel Spolsky outlines the basic arguments against throwing exceptions:

“…I consider exceptions to be no better than “goto’s”, considered harmful since the 1960s, in that they create an abrupt jump from one point of code to another. In fact they are significantly worse than goto’s…”

But then he goes on to give advice that I consider questionable at best, by suggesting that you catch exceptions quickly and convert them into return codes. I agree that they should be caught at the earliest possible moment, but I disagree with converting them into return codes. Because although that certainly makes the mechanism visible to the reader of the code, it still represents strong coupling where there should be none. The caller – or someone further back along the call stack – has to know about, and take action in response to, someone else’s problem. It seems to me that such coupling violates the pragmatic programmers’ tell don’t ask principle.

So what can we do when someone throws an exception at us? To avoid strong coupling between parts of the system we shouldn’t inform the caller that we had a problem. Instead, the object that catches the exception should set its own state such that it will answer future messages in the light of the problem that occurred. (In particular, the NullObject or SpecialCase pattern is useful here.)

For example, suppose we have a system that uses an external search engine, displaying the results on a GUI. And suppose that the API to the search engine throws an exception if the engine cannot be contacted – for example if the network connection is not available. If this occurs, we wish the GUI to display an error message instead of the search results. The above argument states that we shouldn’t put a try-catch block in the GUI, with the catch part containing the code to display the error. Instead, the SearchResult object returned by the engine could be special, having internal state that accords with the failure. Further, the tell-don’t-ask rule tells us we still shouldn’t query the SearchResult’s state in order to discover what to display. Instead the GUI could tell the object to display itself. Then a normal SearchResult object would display a table of results, while a special SearchResult object would display an appropriate error message.

Finally, note that the above also has implications for hexagonal architectures. The only code in our system that is permitted to throw exceptions is now the APIs we use. And the adapters that connect us to those APIs must catch any exceptions immediately, before they can propagate up into the inner hexagon where the domain and applicationm objects live. The presence of try-catch blocks is therefore yet another difference in code style between the inner and outer hexagons.

In his article Domain-Centric Programming J.B.Rainsberger tells a very nice story about how a project was improved by leaving the database development to the very last.

“The last thing I did was to build a simple, four-table database schema and a small component that read from and wrote to the live database. Since I had practiced test-driven development … I had high confidence that I had implemented everything correctly. Since I had focused on the domain, rather than the database, I was even more confident that I had gotten the database right the first time.”

(Thanks to Brian Marick for bringing the article to light, via the agile-testing group on Yahoo.)

By pure coincidence I’ve been telling very similar stories on a project I’m coaching at the moment. We’re currently investigating approaches for the architecture of a new product line, and the group thinks that we might gain a lot by thinking in terms of a hexagonal architecture. During the last couple of weeks I’ve been thinking a bit more about such models, and I’ve come to a rather remarkable conclusion: the database is not on the outside. I now believe that the database is merely one of the technologies that together create the nutrient environment supporting the objects in the middle hexagon.

As J.B. said in his article, the database exists solely to bring persistent state to the domain (and application) objects. Logically, then, it isn’t part of the “outside world”, to be connected by an adapter to the domain objects. Instead, it’s part of the memory model in which the action takes place. It’s one of the tools used to build the middle hexagon. (I feel instinctively that the adapters shouldn’t have persistent state. I’m probably wrong.)

Think of the domain objects as being “always there”. When the adapters are talking to them, they’re in primary memory. But when no-one is interested in them they may be archived out to secondary storage. If someone needs one of them again, the middle hexagon arranges to “re-inflate” any archived objects. This may show up as a slight delay – just as when you re-access an application that was paged out by your operating system. The delay in opening a connection to any particular domain object depends on “how far away” it has been archived: instantaneous when it’s active in memory, slightly longer when it has to be awakened from a database, longer still if it was archived to an old guy in a back room with a quill pen. Whatever the technology used, though, it’s still just technology. Still nothing more than the life-support environment in which the domain objects ply their business.

I discovered during our workshops last week that this is a huge topic. I’ve only scratched the surface here, and I haven’t had the time to work through all of the consequences in practice. But the ideas do seem to hang together, at least in this person’s head if nowhere else. More to come, I expect…

So if we can completely remove all duplicate conditionals, what will our code be like? Well the only remaining conditionals (if statements, switch statements etc) will be those that test something that’s set or defined outside of the system, ie. something that’s part of the runtime environment. The two or more code branches selected by such a conditional will tend to consist entirely of the creation/execution of different sets of objects. Each of these sets of objects must by definition ‘know’ what event or state occurred, probably because it was passed parameters that embody the environment’s state or prior decisions. And so within those invoked object sets there need be no conditional code at all.

This situation now dovetails neatly with Alistair Cockburn’s hexagonal architecture pattern…

The conditional statements themselves, because they form part of the response of the system to it’s environment, should be in the outer hexagon, in what Alistair calls the ‘transformers’ (and what I tend to call ‘adapters’). And if there really is no duplication of conditional statements, the inner hexagon (ie. the application and domain objects in Alistair’s model) will contain no conditional logic whatever! The inner hexagon is purely a soup of objects waiting to be invoked and parameterised in response to external stimuli.

So here’s a very simple rule you can apply to any system: Look at every conditional statement and check that it’s sitting in one of the adaptors connecting to the environment. If it isn’t it must be duplicating some knowledge that the system already holds elsewhere…

Last week I spent an enjoyable day at XPdays Benelux in Rotterdam. I ran a couple of sessions, attended a couple more, and met up with friends old and new. Here are brief recollections of the highlights in my day…

The day began with every presenter offering a 1-minute sales pitch for their session. I got to do two: one for Hexagonal Architecture and one for Jidoka. A minute is longer than you’d think, and I’d guess that most of the pitches were over inside twenty seconds. I decided to be a little different, so I had everyone stand on one leg – hopefully to demonstrate that the standard layered architecture model compromises agility…

I then went to the first half of Joseph Pelrine’s Cynefin workshop. I had to leave at the midpoint, due to the timing of my Hexagonal session; but what I saw in the first half intrigued me. I regret having paid little attention to Cynefin thus far, although after Joseph’s introduction I plan to catch up as quickly as I can.

Hexagonal architecture was a late substitution after another session dropped out. I’ve done a poor job at selling this topic to conferences: reviewers commented repeatedly – and correctly – that I failed to explain its connection with agile development. Consequently this is the only conference that has accepted the session, and so I wasn’t sure what to expect on its first outing. So earlier, while thinking about what to say for my 1-minute sales pitch, I had tried to work out why I instinctively feel there’s a connection to be made. I had concluded that the layered architecture model makes it harder to design adaptable systems, simply because we tend to think of things at the bottom of a stack as being harder to remove or replace (hence asking evefryone to stand on one leg). So the session turned into a lively debate about the power of metaphors in shaping our unconscious thinking, together with interesting discussions about the role of databases, and the use of throw-away user interfaces in the early stages of developing the domain objects. I felt it went well, due to the range of opinion in the group and everyone’s enthusiasm for engaged debate.

At lunch, the refreshment on offer was milk. Very Dutch, and a nice touch.

Then it was on to Christophe and Emmanuel’s coder’s dojo. As with so many things, a coding dojo is something I’ve been meaning to experience for a long time; I wasn’t disappointed. The first half of the session consisted of Christophe and Emmanuel running through a prepared kata, in which they had to develop a flight scheduling algorithm in Ruby. The point here is that, although they had worked through the exercise together previously, the session wasn’t just a code walkthrough. Instead, they wrote all code and tests from scratch, so that the audience was fully engaged in solving the problem together. (In retrospect, the customer acceptance tests should perhaps have been expressed in prepared FIT tables. Instead, they were developed “live” using RubyUnit, which confused the group into thinking in terms of unit tests and TDD too early in the process.) After the tea-break the session switched to become a randori. The group selected a problem from a shortlist of 4-5 – we chose conversion of Roman numerals into Arabic numbers – and developed a Ruby implementation test-first. The catch is that each pair only gets 5 minutes at the keyboard; when time’s up, the driver exits, the co-pilot becomes driver, and a new co-pilot volunteers to step up out of the audience. The session participants demonstrated a wide variety of approaches to TDD, including Microsoft’s “write all the tests up front”. This somewhat derailed the flow, although much was accomplished in the time available. I suspect that the dojo concept will work best when the group has time, over a number of sessions, to evolve a common style and approach – as the Paris dojo group has done during the course of this year.

Finally, my second session of the day: jidoka. This topic has proved extremely popular; I’ve already run it for AgileScotland, and I’ll be running it again at XPday London and at SPA2006. The beauty of the discussion format is that the session is never the same twice, and so it proved again here. The AgileScotland group stuck very much with the themes of stopping the production line and root-cause analysis, while this Benelux crowd veered all over the place, finally settling on a lively debate about the use of weekly retrospectives to drive process evolution. This was a big group, and with a fair amount of Devil’s Advocacy in some quarters we had a broad spread of informed opinion, and I enjoyed the session immensely.

Willem has posted photos of the conference (including one showing me and the attendees standing on one leg!)

In A thought on mocking filesystems Brian Marick provides yet another reason to think in terms of a hexagonal architecture. Discussing the writing of mock objects for huge components such as a filesystem, he writes:

“So it seems to me that a project ought to write to the interface they wish they had (typically narrower than the real interface). They can use mocks that ape their interface, not the giant one. There will be adapters between their just-right interface and the giant interface. Those can be tested separately from the code that uses the interface.”

Mocks/stubs and adapters are natural bedfellows, and definitely help to keep code clean and testable. I once worked on a project in which the “filesystem” was a JNDI directory service. And then later that was scrapped in favour of a plain old filesystem. How I wish I had known about hexagonal architectures then, because it was a real nightmare to disentangle the JNDI dependencies from the rest of the system.

My Jidoka and Hexagonal Architecture sessions have been accepted for XPday Benelux 2005 in Rotterdam this November. Looks like a great programme – shame I won’t be able to attend everything!

In Pseudo-OO Design Dadi Ingolfsson talks about that all-too-common phenomenon – a design in which the code and the data are separate.

“This relates to the Microsoft-style of a DAL (Data Access Layer), a BLL (Business Logic Layer) and a UI, where the data that travels through the layers might be, for instance, dumb Customer objects or DataSet objects. The objects in these different layers are very procedural in nature as they usually only operate on data in objects that are sent to them rather than the data being their own. It´s what Fowler called an Anemic Domain Model. This is not OO design even though many programmers seem to think code is OO if it has classes and objects.”

By coincidence, this week I found myself workshopping agile programming with a team whose legacy C++ system is just like this. My approach to gradually improving the situation is to look for instances of the feature envy smell. Bit by bit, as we find each of these smells, we move the code into the objects where it belongs. Bit by bit, encapsulation improves. Bit by bit, the domain model emerges in the code. And very often, as we introduce abstractions to improve testability, bit by bit a hexagonal architecture emerges too.

Why is this important? Why is it bad to separate the code from the data? Because it stops the code changing at the pace of the business. Distributed behaviour and state mean that each piece of business knowledge is also distributed throughout the system. Which means that any new or changed requirement will cost more to implement.

The standard three- or four-layer models of application architecture seem to dictate the direction of the dependencies between the various “kinds” of object in the system: The UI depends on the application layer, because the UI “drives” what happens “below”. The application layer depends on the business objects, which do all the domain-specific things. The business objects use (and hence depend on) the persistence layer and comms layer, both of which in turn use and depend on external APIs. It is perfectly natural to read a layered model in this way – indeed, that was its purpose.

ui ---> application ---> domain ---> (persistence, comms)

And it is this very interpretation that has made so many applications difficult or impossible to test. Each of those perceived dependencies ends up enshrined in the code structure, and by stealth the design has cast in concrete the layered model. At which point someone like me comes along an exhorts the developers to improve their test coverage. I’m told: “We tried that, but it takes too long to manage the Oracle testdata we’d need to test the GUI,” or something similar. Or the business decides to switch to a new database or enterprise comms architecture, requiring huge swathes of business logic to be re-engineered.

Alistair Cockburn’s hexagonal architecture model introduces an axis of symmetry that can help us see how to solve these kinds of problem:

Instead of the above-and-below of the layered model, we look at exactly the same application (and exactly the same code) using an inside-outside model. The domain objects sit in the middle, and are connected to the real world by what I call Adapters – a GUI, CGI scripts, a command-line, hardware controllers, a print spooler, etc. Each adaptor connects the domain objects to a different part of the ‘real’ world.

So let’s look at our untestable or non-portable application from the “hexagonal” point of view. (Remember, we’re still talking about the same codebase here: only our discursive model has changed.) The dependencies that have made testing so difficult, and which exist in response to a naive interpretation of the layered model, clearly violate the symmetry of the hexagonal model.

ui ---> (app, domain) ---> services

If we wish to be able to easily port the application to use different services, we’ll need to be able to easily replace the adapters that connect us to those services.

ui ---> (app, domain) ---> new services

And if we wish to test the domain objects independently from the hardware, we’ll need to be able to replace various adapters with stubs, shunts or Mock Objects.

fit ---> (app, domain) ---> mock services

We can’t do this currently because the design violates Bertrand Meyer’s Open-Closed Principle (“A system should be open to extension, but closed to modification“). To solve the problem we need to use Uncle Bob’s Dependency Inversion Principle: We introduce abstractions into the middle hexagon, such that the current adapters implement those abstractions.

ui ---> (app, domain ---> abstraction) <--- service

Symmetry is now restored: every adapter depends on the middle hexagon, and the middle hexagon (our domain objects) is independent of the physical context in which it is used.

adapter ---> middle <--- adapter

Now we can more easily see how to peel off any adapter(s) we wish and replace them with mock objects or different services. The adapter/middle axis of symmetry is the key ingredient in improving both testability and portability of enterprise applications.

Update, 16 aug 05
In TDD Pattern: Do not cross boundaries Revisited William Caputo provides both UML and C# code to illustrate his design pattern for removing the dependency on an adapter. This follows his earlier post, which argues that test-driven coding would naturally tend to produce the de-coupled form of the design.

Ever since I heard of it (around 1998 or so) I’ve been trying to persuade people to think in terms of Alistair Cockburn’s hexagonal architecture model. To me, it takes away so many of the pointless debates, and makes modelling and developing any new system so much simpler. Thanks Alistair!