Agile

One of the questions that was posed at ACCU conference was “What are the downsides of Agile development?” – and “What does Agile failure look like?”

I have tried to answer the first question before but failed myself.  One downside is that the Product Owner role – typically filled by a Business Analyst, Product Manager, Customer or other proxy-customer – is given an immense work load.  This is a subject James Noble has done some work on.

What happens is that Product Owners have to keep doing their original job and service the needs of an Agile team.  As the Agile team becomes more productive they ask more and more of the owner.  Consequently the owner gets maxed out and themselves becomes unproductive.

However when I thought about this I don’t think the failure is a failure of Agile, I think it is a failure of the business.  Put it this way: for years “the business” has been complaining that software teams do not deliver what they want.  Now along comes Agile and says “we will deliver exactly what you want” but “you need to tell us exactly what you want.” 

It is no longer enough to set half a dozen developers working on a problem and expect them to come up with the right thing.  If customers want what they ask for then they need to tell the developers exactly what they want in detail.

The second question, “What does Agile failure look like?” is hard to answer because we don’t have many case studies of failed projects.  This isn’t just true of Agile projects, it is true of many IT projects.  Companies don’t like to talk about failure. 

Still I can see several failure modes for Agile:

• Teams fall back to the old way of doing things: I’ve only heard about this not seen it myself.  Typically it occurs when you bring in a lot of consultants to make your team and process Agile.  Things work when the consultants are there but when they leave teams fall back to the old way of doing things.  Unit test break and aren’t fixed, planning meeting are missed, quality slips and deadlines are missed.  The presence of consultants gives the team a backbone but knowledge is not really transfered to the existing team.
• Failure to get management buy in: Agile is often introduced at the bottom of the hierarchy, by developers at the code face.  Eventually there comes a point when managers need to support it or stop it.  This doesn’t happen and the development process is left in a random state.
• Failure to get developer buy in: this occurs when management buy into Agile but developers don’t.  They don’t want to write tests, do code reviews or pair, believe in technology over visual planning, etc.  Again the process is left in a random state.  I expect to hear more about these types of failure in future.

The common theme here is not that Agile fails, after all IT projects and processes fail all the time so there is nothing new or special here.  Failure of an Agile project looks a lot like the failure of a Waterfall or RUP project.

Rather Agile fails when you fail to see the benefits of the new approach.  So when you do a little bit but not enough to improve quality, improve delivery dates, and make things predictable.

Both questions are good questions and deserve more attention.

As I mentioned last time, I came back from the ACCU conference with a bunch of notes and a lot of thoughts.  Recording them in this blog seems like a good way of remembering and sharing these thoughts.

Last time I referred to the Great TDD debate.  At the conference Mary Poppendieck reminded us of the key principle behind TDD and other forms of unit (especially automated) testing when she quoted from one of the original Toyota sources:

• Testing to find defects is waste: the defects should not be there in the first place so any attempt to find them is waste.
• Testing to prevent defects is essential: defects slip in, we should catch them as soon as possible so create systems with tests that prevent defects.

Hopefully it should be obvious were TDD fits it: TDD is there to prevent defects and reduce waste.  However, testing later in the cycle, e.g. system test, is waste and although it may be necessary right now we want to get away from faults, testing and waste.

On the face of it this is bad news for software testers.  If we start testing to prevent waste rather than find waste then they are out of a job.  Well I have three reassuring thoughts for software testers, they will not be unemployed anytime soon because:

• Few organizations can actually implement this principle, some will always need testers.
• We have a lot of code out there where nobody prevented the defeats and therefore needs testing.
• Testing itself is an activity that can add value to products –  as Oracle considered last year.

Also, most of the software test managers I have ever worked with have been keen to point out that they would rather conduct Quality Assurance than Test.  Too often these activities are lumped together but when you think about it they are very different.

So, in the short term testers have plenty of work, in the long term their future is in quality assurance – a more value add activity.

Now at this point a lot of people tell me I’m an idealist.  After all, everyone knows software is buggy, has always been buggy, and always will be buggy.  I remember being taught at University how every piece of software contained an infinite number of bugs.

I asked Mary Poppendieck about this, her answer was basically: you get what you expect, if you expect to get bugs then you will.

Well, maybe I am an idealist.  But what is wrong with wanting software without bugs?  What is wrong with setting my sights higher?

I got back yesterday from the 2007 ACCU Conference. Once again this was a great conference. Lots of good stuff on Agile and Lean (Mary and Tom Poppendieck delivered a keynote), software management, programming in general, Java, C# and C++.

I mainly kept myself to the management and agile sessions. Also I was involved with a lot of ACCU business type stuff occurring around the conference. It is about the time when all the key players in the ACCU are in the same place and a lot of association business gets done. As well as delivering my own session and taking the stage with Mark Shuttleworth and Mark Poppendieck I was in meetings about the ACCU journals, growing membership, conference organising and a host of discussions about the association.

Over the next few blog entries I’ll try and debrief myself and highlight the things I feel I learnt and need to remember. My comments from this time last year still stand, the best speakers are no necessarily the ones you expect them to be. (Apologies for that blog entry, its one of the ones with formatting problems.)

Right now here are a few bullet points.

I met three people who are separately working on projects that compile the same code base to Java and C#. These people have the code in one language and have some pre-processor step which makes the syntactic modifications so it can compile for the other. Is this a trend? Will we one day have Java# ?

Ed Sykes and Jan-Klass Kollhof did a really good session. They set out to produce a distributed algorithm using Python and JavaScript in web browsers. They just about succeeded but actually their session was far more interesting for what it said about systems with emergent behaviours. They also took a stab at futurology and suggested Web 4.0 Distributed Computing. You heard it here first!

The great Test Driven Development debate rumbled on. Well, when I say debate it is a bit one sided. There are a few people who think TDD is not a good practice then there are the vast majority of people who have tried it think it is a good practise that improve quality.

Mary Poppendieck presented some figures from Mike Cohn’s book Agile Estimating and Planning. Before introducing TDD Cohn’s company had 10 faults per 1000 lines of code, after 3 months of TDD this had fallen to 3 per 1000 lines.

For me this debate is a little like the Black Adder scene were the Captain (Tom Baker) says to Black Adder (Rowan Atkinson):

“Crew? Opinion is divided on the matter, all the other captains say you need a crew and I say you don’t”.

Gail Ollis did a good session on Advocating Agility, one of her first slides quoted the start of A.A.Milne’s Winnie the Pooh:

“HERE is Edward Bear, coming downstairs now, bump, bump,
bump, on the back of his head, behind Christopher Robin. It is,
as far as he knows, the only way of coming downstairs, but
sometimes he feels that there really is another way, if only he
could stop bumping for a moment and think of it.”

Gail had spotted the great sentiment expressed here: we do painful things because we simply don’t know how else we might do them and don’t take the time to find out. I left intending to use this quote myself.

In looking for this quote on the web just now I seen that neither Gail or myself is the first to spot this, its all over the place and often associated with change.

OK, thats a first stab at debriefing the conference, over the next few entries I’ll write more.

I attended a project retrospectives training session last week.  Although I’ve run quite a few retrospectives over the years I’ve never been trained in them.  My guide has been Norm Kerth’s Project Retrospectives book and a whole bunch of ideas I’ve picked up over the years.

This training session was organised and run by Rachel Davies and Diana Larson.  It drew on Diana’s new book Agile Retrospectives. 

Although the book and the training is aimed at Agile practitioners it could be used by other teams – its just that Agile teams are more likely to do this stuff.

The world outside of software development is also catching onto retrospectives.  In part this is thanks to the US Army and Marines who call them After Action Reviews (AAR).  Now they have a macho name and the backing of some generals they seem to be cropping up all over the place.

That said, I still think Retrospectives – or AAR to give them a more fashionable name – are still more talked about then practised.  The training session gave me an opportunity to reflect on this paradox (as well as learning some new techniques and improving my facilitation style.)

So why is it retrospectives get a lot of talk and little action?

There seem to be three basic failure modes.  First is retrospectives that don’t happen.  Usually the reason given is lack of time.  However I think this only an excuse to hide behind.  When people want to do something they do it, if something is important time is made for it.  Conversely when something is low priority or someone doesn’t really want to do it then there is not enough time.  We are all busy people, lack of time really means “is not a priority for me.”

I think one of the reasons people don’t want to do retrospectives is because they are scared of what might come out of it.  Individuals could be criticised, companies could be found wanting or someone may say something they are not supposed to say. 

Retrospectives expose the lie that your boss/manager/leader knows best.  By definition retrospectives are about asking teams what they can do better.  Traditionally we are brought up to expect that someone in authority, someone with a “big brain”, knows best.  So retrospectives can be threatening to managers who feel they are loosing some power.

Which brings us to the second common failure: ineffective results.  Simple formulaic retrospectives, perhaps they simply ask “What do we do right?  What do we do wrong?” and perhaps they limit the whole thing to an hour, and perhaps the team leader or project manager acts as facilitators.

Such retrospectives are unlikely to uncover deep thoughts because they are not asking the questions that will cause people to reflect deeply about the whole project.  Having a authority figure lead the retrospective will also cut off reflection, by leading this person is exercising their authority and sending a clear message “I am still in charge.”

Finally, the third failure mode of retrospectives is: Lack of action.  Things the team decide to do aren’t acted on.  This is actually closely related to the first two.  If it is hard to find the time for a retrospective do you have the time to make the change?  More likely you are risk averse.

And if the authority figure isn’t open themselves then they are not going to be fully behind the change.  This will be clear to anyone tasked with making the change.  That is, if anyone can be persuaded to take on the task.

Arie de Geus pointed out in his book The Living Company that when you have more people involved in making a decision it might take longer to actually make the decision but taking action once the decision is made will be quicker.  This is because more people were involved in making the decision.  If a retrospective is half hearted then any decisions will not have wide buy-in so taking action will be a slow process.

So what’s the answer to these problems?

Well I think I’ve realised were I was making my mistake.  Like many other people I’ve been seeing retrospectives in isolation.  As some kind of plug in solution, nicely self-contained.  But they are not, this was the mistake.

I’ve realised that retrospectives need to be embedded in a learning culture.  I used to think they could be one of the early tools to use in creating such a culture, now I realise they are an advanced tool to be brought in once you have the culture in place. 

An organization that does retrospectives is an advanced learning organization.  Most organisations are not learning organisations let alone advanced ones so it is hardly surprising that retrospectives are missing.

When you have such a culture not only do people value learning but it is a priority for them.  They make time for learning events.  They value the results and they take action.

In such a culture people feel less threatened by what might come out of a retrospective because they are accustomed to deep thinking and reflection.  Everyone knows the rules and people are open to insights and change.

In a learning culture managers have already accepted that the team knows best.  They have already changed the view of their role from command-and-control and “boss knows best”.  They define their role more in terms of helping individuals and teams learn and advance.  Neither are they threatened by loss of authority.

Finally, when you have such a culture, when people are not scared or threatened, when everyone appreciates the value of holding a retrospective then change does happen, people do follow through on the action identified in the retrospective.

So, how do we create such an advanced learning organization?  Well, that will have to wait for another blog entry and even my book.  In the meantime, if I can help with a retrospective or creating a learning culture please get in touch.

The SPA conference is on this week and I’m not there. SPA is a good conference but being so close to the ACCU conference I need a very good reason to go. Since I’m not speaking there I’ve skipped it.

The BCS SPA group also holds monthly meetings which I sometimes go along to. About three weeks ago I went along to a very interesting talk entitles “Large-Scale Agile Application Development”. In fact the talk was actually about how BT had adopted Agile development techniques. This was quite a challenge as those who know BT know that it is a very large company, a traditional telco and at one time a nationalised industry. There could be few more challenging environments into which to introduce Agile.

BT haven’t completed their Agile transition but it seems to be going well. Fortunately they had high level management backing on their side and all the kind of resources big companies can bring to bear on these problems: training, off-site boot-camps, consultants (mainly from Exoftware) and other resources.

I was particularly impressed by two of their ideas.

First was a plastic credit card like “pledge card.” I’ve seen these done by political parties in the past but this one was for “BT Core Agile Delivery Practises.” On one side was the Agile manifesto and on the other was BT’s core practises:

• • Customer collaboration

• • User stories

• • Iterative development

• • Automated testing
  • Continuous integration

The card strikes me as good because a) it was a constant reminder of the practises, b) it was very corporate so endorsed the idea that the company wanted this to happen.

The second idea was a pre-printed task card with the main fields a developer would use. I’ve seen index cards put through a laser printer before but this was more structured. It contained fields to help describe the work task, the priority and the acceptance criteria. Again good for re-enforcing the corporate message (“This is what we want”) and a reminder to teams as to what they needed to do. My only criticism might be that for such a small card it had a lot of fields to complete.

BT seems to have invested a lot in this programme. They had sent lots of people on training courses and held lots of off-site meetings and training.

They also spent a lot of time and money using coaching techniques. In fact they used pair-coaching with an experienced coach working with a new coach to help train the coach. To my mind this approach vindicates the coaching approach and shows how powerful it can be.

That say, it does seem the coaching model in use was very much drawn from the Agile coaching model rather than the model used by business coaches. The difference is that the Agile model is much more directive about what needs doing while the business model is much more open. Both have their place, I just wish Agile coaches were more aware of the other model and how it can help.

I went to a meeting of the UK Product Managers Forum last night.  This was the first time I have been to a meeting of this group and it is the first time that the group has held an event in London.  Naturally these two firsts are not unconnected!

I enjoyed the meeting, Ian Mapp of Respond spoke about the development of the product management function within the company.  This was very interesting itself, and as is so often the case the conversations with the other attendees was even more interesting.

These conversations support my belief that Product Management is not a well established role in the UK software industry.  When the product management doesn’t exist in a company there is a vacuum.  Without any guidance software developers will just develop the things they think are a good idea.  If there is a compelling need for your product this isn’t a problem in the short run: people need your product, good or bad they will buy it.

Nature abhors a vacuum and the same is true when product management is missing.  Somehow the vacuum will get filled.  In small companies it is often the founder(s), the CEO, CTO or similar who fulfils the role.  They often do it on gut feeling. If they are lucky they are right and the companies grow but then they don’t have the time to do it.

Sooner or later someone steps into the role.  It might be a project manager, a developer, an architect or even a tester.  These people do their best but they have other demands on their time – the job they should be doing.  There is competition not just for their time but their prioritisation decisions will be influenced by their duel roles.  Neither have they been trained for the role.

A lot of software development does wrong right here.  Without good product management to give customers a voice inside the organization companies will spend more time developing more features to sell fewer products.

(Product management can exist under the wrong name, I’ve met Architects who are really Product Managers and I’ve seen Project Manager roles advertised which are really Product Management jobs.  This just confuses things!)

The second thing I noticed at the meeting was how tight the profession is.  Most people knew of the Silicon Valley Product Group, 280 Group and Pragmatic Marketing.  Those of us who have ad product management training had had it through one of these groups in the US.  Again this is a sign of how underdeveloped the role is in the UK.  I believe this is about to change, Co-herence have licensed courses from the US and plan to deliver them in London.

The UK is not short of software companies.  It is short of product managers and product management skills.  Fixing the development process must involve product management.

If you are developing software application there are three things you must do:

1. Create a product management role (business analysis if you are in house)
2. Use source code control
3. Have a repeatable build process and run it at least once a night

If you aren’t doing these things then don’t bother leaving home, in fact, don’t even bother getting out of bed.  The scary thing is: most companies struggle to score two out of three on this list. 

(If I just scared you and you want help e-mail me, more on this next week.)

As an aside, Respond’s product is a customer feedback (i.e. complaints) management system.  Ian suggested that in some industries, notably airlines and mobile telecoms, the company got very little traction.  His conclusion was that for these companies customer service is not a high priority.  I can see his logic, if the companies in these sectors are not willing to spend money then it is reasonable to conclude that the companies do not see customer service as a priority.  One of the audience, form a major UK mobile phone operator took exception to this.  They claimed their company did take the issue of customer service and retention (churn) seriously. 

I’m in the process of reviewing my mobile telephone.  I’ve been with the same provider for 5 years now but I don’t feel like a valued customer.  The company has a memory like a goldfish – i.e. none – I might as well have been with them a year or a month.  I’d like to switch to another provider but a) I’m not convinced anyone else is better, b) it looks like my provider is actually the cheapest!

Wednesday’s Financial Times carried the monthly technology review. A couple of pieces caught my eye, both individually and when put together.

First this report In an e-world, IT controls the customer experience (subscription required) makes the point that if your business is conducted online then your customer experience is an electronic customer experience which in turn means the customer experience is decided by the IT department. This is a point I’ve made myself in the past – although I’ve never came up with such a compact phase to describe it.

In posts in January 2006 and December 2006 I sounded off against poor travel company web sites. My customer experience is poor because the web sites are poor. It doesn’t have to be this way but unless the IT department think about the customer experience, or the people responsible for the product involve themselves with IT this is what happens. Unfortunately IT departments have spent years refining their image as uncooperative.

Given the increased use of e-commerce by all businesses – not just those like Amazon which only exist on the web but everyday companies like the travel companies I dislike – then any business that does any e-retailing has this issue. The online brand can damage the off-line brand.

The second piece that caught my attention was this one Can HR handle IT recruitment? According to this article human resource departments do not have the skills and experience required to hire IT people. The story rings true from my own experiences and from many anecdotes I’ve heard over the years.

Although these stories all ring true I’ve also known several good HR people. People who have their heads screwed on and want the best for the business. Plus I’ve read a fair few HR text books and articles and I know HR people aren’t really as bad as IT people make out. So what’s going on?

Well I have some possible explanations.

Theory 1: Many companies have bad HR departments. There are some good HR departments out there but possibly most of them are not worth the salaries we pay them.

Theory 2: Everyone has these problems with HR, it isn’t just IT. HR is just badly understood by the rest of the business.

Notice that these explanations are not incompatible. Neither are they incompatible with the FT article. Which raises the questions: What are HR actually doing? Why aren’t HR departments getting to grips with IT and trying to rectify the problem? After all, strictly speaking, it is the HR departments job to hire people so it is they who have the problem. They can’t blame IT for this problem.

Put all these theories together and you get: HR is important, HR doesn’t understand IT, many HR departments are poor, many HR departments fail to hire the right people. As a result many IT companies do not hire the best staff. And because good IT depends on good people few companies will have good IT.

Proof for this theory comes from Google and Microsoft. Both of these companies have built hiring machine. They both need a lot of IT staff and they have made sure HR can do the job properly.

So the lesson is: If you want good IT you start by fixing your HR department.

Finally, put those two articles together.

IT controls the customer experience but HR do not know how to hire the people IT needs. In other words: HR departments do not know how to hire people who create the customer experience. Am I the only one who sees this as a serious problem here?

A student at Fullerton University in California e-mailed a lot of people on the Lean Software Development mailing list with some questions for his masters thesis.  Since I had a little time on my hands I wrote him some lengthy answers.  Having done that I though I would share them here.  I’ve removed a few questions because I don’t think the question or answer are particularly interesting.

1.      How would you define lean development?

The strictest definition of Lean Development would be to say it is a methodology proposed by the Poppendiecks (Lean Software Development)based on lean manufacturing and product development.  This definition is useful for deciding who is doing lean and who is not but is probably too strict.

Another definition of Lean (Software) Development is that it is the application of the principles of Lean Manufacturing ( The Machine That Changed the World, Womack, Jones & Roo ) and “Knowledge Based Product Development” (Product Development for the Lean Enterprise, Kennedy).

Personally my definition is:

• Lean Software Development is one application of the principles of Organizational Learning to the software development environment. 
• Learning Organizations may come up with a different development way of thinking but I am not aware of any others.
• Agile Software Development is a further refinement from principles of Lean to practices.
• The various Agile development methodologies are further refinements of these with further practises.
• Underlying everything is the idea of a Learning Organization (Senge, Argyris, etc.)

So you can create a pyramid, more general at the bottom, more specific at the top: 

– XP –	->	Specific practices
— Agile —	->	General practices
—- Lean —-	->	Ways of thinking
Learning Organization	->	Concepts

To put it another way:

• XP (or SCRUM, or DSDM, etc) are a specific set of practices for doing Agile software development
• Agile software development is set of principles and a specific set of lean practices
• Lean is a set of principles and a specific set of learning practices
• Organizational Learning is a set of principles to which some authors have added some practices

Note that you can, in theory, adopt an Agile methodology with very little learning. You could simply adopt, say, SCRUM practices and through focus and authority make people use these.  You probably would see some benefits however the team would not be learning and future change would not come naturally.

I examined Software Development as Organizational Learning for my own Masters dissertation

2.      What percentage of developers do you think are using Lean Development approach?

Very difficult to say, I think some ( less than 10%) will have picked a few practices, I think few will be working in a Poppendieck or Kennedy like way –  maybe less than 1% on this definition.

On the other hand, I think many are undertaking some organizational learning and using some agile practises but progress is very slow.

3.      How has the usage increased from previous years?

Slowly.

4.      Why do you think the usage has increased?

Various:

• The term is now commonplace thanks to Poppendiecks and others.
• Lean (and Agile) tell a very good story, developers want to work like this.  Managers see the parallels with lean manufacturing so also hear a good story.  It is an aspiration change.
• Natural selection: those who “get it” will out compete competitors.

5.      How do you expect the usage to be in the next five years?

Slowly.  I see an Agile backlash building, I hope lean will not be caught in it too but I think it will.

6.      Have you found resistance or pessimistic thinking towards Lean usage?

Yes – “Idealistic”, “Will not work here”

7.      What kind of unintended results have you seen by using Lean?

Lean relies on exposing problems so they can be fixed.  (Removing “camouflage” to use Argyris term and rolling back what Senge would call “shifting the burden.”)

As such it looks like things are getting worse to start with.  Problems that were hidden are now apparent, problems that could be ignored before need fixing now.  Therefore, Lean looks like it is making things worse before things get better.

When developers or junior managers get it but senior managers don’t there is a mismatch.  Junior managers pay a heavy emotional toll and sometimes pay with their jobs.

8.      What suggestions do you have for a shop considering Lean?

• Emphasis the learning. 
• Take it from both ends: senior managers down and shop floor up.
• Get help, specifically coaching.

10.     Can Lean development be applied to non-Agile (i.e. waterfall) methodologies?

You could make a waterfall leaner but I doubt we would ever call it lean.
In suggesting a lean waterfall you are saying: “We want to improve X but we don’t want to change Y”.  You have put boundaries on the problem.  In doing so you are putting some things out of bounds.  This means you can’t engage in Systems Thinking properly and are likely to come up with sub-optimal processes.

11.     How difficult is for a team become Agile and Lean after being non-Agile?

Very difficult.  You can start in a small way but you need to break with the past.

13.     What are the key indicators to tell if a team will succeed in becoming Lean?

The presence of a learning culture:

• Is the team holding retrospectives?
• Is the team changing their process?
• Is the team improving?

14.     Is it better to be Lean and then Agile or Agile and then Lean?

Given my definition above this question does not make sense.  There is no temporal relationship.

If you are Agile then you are (to some degree) Lean.  If you are Lean then you are Agile.

15.     What did I forget to ask you about lean?  Is there something about lean that you consider very important that we did not discuss?

The “change question”.  We know how to be Lean and Agile but how so you change the organization?

Then, when you have become Lean how do you carry on changing?

Organizational change is more important than the practices.  The practices support learning and change but change is not a rational thing.  You don’t do a value stream map and solve everything. 

OK, I promise this is the last entry on the Nurnberg funnel, after this I’ve worked it through my system…

In my entry on Nurnberg funnel and system development I wondered is system documentation actually works. Well I missed one piece of evidence that it does work. Go into your local bookstore, Waterstones, Borders, even Amazon online and look at the books on computing. Specifically look at the technical books that describe things like programming languages, operating systems, APIs and the like. If system documentation does not work how do we explain the existence of all these books?

After all, publishers are in the business of making money. If these books did not work – i.e. did not help learning and transfer knowledge – then presumably nobody would buy them, there would be no profit in publishing them and consequently they wouldn’t be taking up real-estate space in the book store.

So how do we explain these books if technical documentation doesn’t work?

I don’t have any figures on what is happening but here is my guess.

Firstly a whole load of this documentation does not work. Many books get bought despite being bad. Many more books are bad and simply don’t get bought, publishers swallow the loss and pulp the books. I would guess this category covers many many books. Lets take a stab and say 40% of all technical books.

A second category of books do work and they work for exactly the reasons Carroll outlined in the Nurnberg funnel. They are written along the lines of minimalism. My guess it that a lot of those “For Dummies” and similar books fall into this category. Still, this type of book represents a small percentage of all books. Lets say 5% – probably an overstatement but it keeps the numbers easy.

OK, I still have 55% to account for – and I’m deliberately not saying if these are percentage of books sold or books published.

Next up I think there are some books which do not directly take on board the minimalist ideas but have a similar approach and work for similar reasons. I’m specifically thinking here of patterns books like Design patterns and Pattern Languages of Program Design 5 (to which I was a contributor).

In part patterns are popular and accessible because they start with a problem and describe a solution. This is a little like the task orientation that Carroll advocates. I’m sure patterns books are not alone here but there aren’t very many books in this category so lets just say another 5%.

This also starts to hint at another explanation and one which must account for many of the remaining 50% of books. Simply minimalism is but one technique for creating documentation, it works particularly well in one field – user manuals – but it is not the only one.

For example I know of one other method, story telling. This is usually advocated as a management or knowledge management activity to be conducted verbally. (See books like The Springboard and Storytelling in Organizations). However there are technical books out there that take the same track.

For example, I found the original version of Inside Windows NT quite story like. Although perhaps this is a sign of how techno-geeky I was in the mid-late 1990’s. Somehow the second edition of the book never had the same plot or interest for me.

Less technical books about technical subjects are better examples of story telling. Take The Soul of a New Machine for example. This is a must read, I left it far too long before I read it but its right up there with The Mythical Man Month, Peopleware and The Psychology of Computer Programming if you want to know how software (and hardware) really get created. But Soul of a New Machine is also a story, it can even be bedtime reading.

Books like these are more the exception than the rule so they are few and far between. Lets say 1% are story books.

Another category is reference books like Charles Petzold’s “Programming Windows”. I defy anyone to actually read this books cover to cover, you delve into when you have a specific problem or need to know some specific detail. But reference books probably constitute a lot of the books written, say 20%.

That takes me to 71%. I still can’t account for 29% of books. I give up.

The important thing is that people read books for different reasons. Our motivation for reading one of these books will be different to the motivation for reading another. That will change the way we approach buying and reading books.

Which in a strange way brings us back to minimalism. People use Petzold’s books when they have a task to accomplish. People pick them up under certain circumstances and like a Dummies book expect to get something specific from them. But that doesn’t mean all books should be written in the minimalist style. Minimalism is just one of many styles.

So actually, we’ve come back to one of the re-occurring themes of this blog: know your customers, service their needs with a product that is designed to meet those needs.

Products development doesn’t end when the code is written, compiled and burnt onto a CD. It has to include the documentation, service and support that is supplied with the products. It also means ensuring that future product development can continue, whether through just-in-case documentation or just-in-time networks of people who know.

Unfortunately obvious solutions – like big thick manuals – are not necessarily the best solutions. We need to be on the look out for new solutions and new ideas.

And we need to understand the costs and benefits of all this. This is difficult bit because so often we don’t know the numbers and we can only speculate on what they are. But somehow we need to put all of that together.

After reading Minimalism beyond the Nurnberg Funnel I have been thinking what implication these theories could have for software development itself.

First for system design. Minimalist documentation aims at being task oriented. The idea is that the readers (users) are trying to do something and need to solve a problem – how do I? – so they turn to the documentation. What if we apply these ideas to system design?

Well I think we would come up with simplified systems that allowed users to perform tasks. The software would treat users as problem solvers and allow them to combine the tasks in what ever way they like to solve their wider problem. Now this starts to sound like Naked Objects.

Second: what if we turn our attention from user documentation to system documentation? Usually the documentation software developers write to describe the insides of the computer system is based on the Nurnberg funnel principle. We assume that is a software designer writes down everything they think they know about the software then someone who reads it will know the same information – open programmer’s head, insert Nurnberg funnel, pour documentation…. programmer can now understand software!

Obviously this isn’t going to work. This isn’t only true about existing software it is also true about systems we are going to write. Traditional software design (specifically the waterfall model) mandates that Architects and Designers – who in the traditional model are different to Programmers – should “design” a system, document the design and hand the design over to the Programmers so they will know how to turn the design into code.

Again, this is the Nurnberg funnel model and it isn’t going to work. When we follow this model we are lying to ourselves. So what are we to do?

Minimalism probably isn’t an answer in this case. Firstly the potential audience is so small it will be impossible to justify the extra cost. Secondly minimalism relies on describing the tasks people will want to perform. In general we don’t know the tasks that future system developers might want to undertake on our software. Yes we can guess what they might want to do but this really is little more than a guess. I have designed and built systems in the past with specific extension points for future developers – just add a new object here. But these are simple changes. The majority of the changes I’ve done to software involve expanding it in a way that wasn’t thought of in advance.

So while the thinking behind documentation minimalism can show the failings of traditional documentation it cannot provide a solution. We need to look elsewhere. I’m not sure I know the solution but I’ll make one suggestion.

The problem of making software developers familiar with a new code base is not a problem of documentation. Rather it is a problem of knowledge management and specifically knowledge transfer. Documentation is one tool in this arena but not the only one. Traditional documentation relies on a just-in-case model. Managers who are worried about risk have developers document the system just in case something needs to be done. This looks like a strategy to mitigate risk because we have “captured the knowledge” but in fact the risk still exists because the documentation is not very useful.

The alternative model is just-in-time knowledge management. Here we ensure that should a new developer need to know about an existing system they can ask someone who knows when they need to. The knowledge transfer then takes place directly from one developer to another.

Instead of the solution being based on something you have it is based on someone you know. On the face of this it this model looks costly. In order to get the knowledge we need a one-to-one relationship rather than the one-to-many relationship documentation provides. But since documentation does not work this argument does not hold.

In fact, writing documentation that might not be read, and if it is will not be understood represents waste. It is a waste of the developer’s time to write the documentation, it is a waste of paper to print the documentation, it is a waste of space to store the documentation and it is a waste of time to read the documentation. The only purpose it serves it to allow a box to be ticked on a risk form.

Using the just in time model all this waste is eliminated. If, and only if, the knowledge is needed then the cost of knowledge transfer will be incurred. And since the cost will be incurred in the future it is cheaper today. (I probably don’t need to spell this out but just in case: this model matches the ideas of Lean manufacturing, product development and software development.)

However, we are still left with the problem of mitigating the risk. To do this we need to keep track of the programmers who developed the system and have made changes since. This potentially changes the nature of the software development model but in today’s Internet connected world should not be insurmountable.

The problem then is that people, even developers, do forget over time. Still, forgetfulness might not be that much of a problem reality.

There are two types of system: those that are used and those that are not. The ones that are used tend to need changes over time; because these systems are living people think of new features and they need to be changed as the environment changes (think Y2K.)

Then there are the systems that don’t get used. Well if these systems are not used it is unlikely they need to be changed. Obviously for those systems that are being changed there are people who know how they work. For those that don’t change there might not be anyone who knows how they work but neither is there anyone who needs to know.

Of course there will always be the odd exception, a program that runs for 10 years without a change and then needs updating; but how many systems are there like this? Does the existence of a few systems that require very occasional changes justify writing all that documentation for all those systems were is isn’t needed, won’t be read and won’t be understood if it is?