allan kelly

I got back yesterday from the 2007 ACCU Conference. Once again this was a great conference. Lots of good stuff on Agile and Lean (Mary and Tom Poppendieck delivered a keynote), software management, programming in general, Java, C# and C++.

I mainly kept myself to the management and agile sessions. Also I was involved with a lot of ACCU business type stuff occurring around the conference. It is about the time when all the key players in the ACCU are in the same place and a lot of association business gets done. As well as delivering my own session and taking the stage with Mark Shuttleworth and Mark Poppendieck I was in meetings about the ACCU journals, growing membership, conference organising and a host of discussions about the association.

Over the next few blog entries I’ll try and debrief myself and highlight the things I feel I learnt and need to remember. My comments from this time last year still stand, the best speakers are no necessarily the ones you expect them to be. (Apologies for that blog entry, its one of the ones with formatting problems.)

Right now here are a few bullet points.

I met three people who are separately working on projects that compile the same code base to Java and C#. These people have the code in one language and have some pre-processor step which makes the syntactic modifications so it can compile for the other. Is this a trend? Will we one day have Java# ?

Ed Sykes and Jan-Klass Kollhof did a really good session. They set out to produce a distributed algorithm using Python and JavaScript in web browsers. They just about succeeded but actually their session was far more interesting for what it said about systems with emergent behaviours. They also took a stab at futurology and suggested Web 4.0 Distributed Computing. You heard it here first!

The great Test Driven Development debate rumbled on. Well, when I say debate it is a bit one sided. There are a few people who think TDD is not a good practice then there are the vast majority of people who have tried it think it is a good practise that improve quality.

Mary Poppendieck presented some figures from Mike Cohn’s book Agile Estimating and Planning. Before introducing TDD Cohn’s company had 10 faults per 1000 lines of code, after 3 months of TDD this had fallen to 3 per 1000 lines.

For me this debate is a little like the Black Adder scene were the Captain (Tom Baker) says to Black Adder (Rowan Atkinson):

“Crew? Opinion is divided on the matter, all the other captains say you need a crew and I say you don’t”.

Gail Ollis did a good session on Advocating Agility, one of her first slides quoted the start of A.A.Milne’s Winnie the Pooh:

“HERE is Edward Bear, coming downstairs now, bump, bump,
bump, on the back of his head, behind Christopher Robin. It is,
as far as he knows, the only way of coming downstairs, but
sometimes he feels that there really is another way, if only he
could stop bumping for a moment and think of it.”

Gail had spotted the great sentiment expressed here: we do painful things because we simply don’t know how else we might do them and don’t take the time to find out. I left intending to use this quote myself.

In looking for this quote on the web just now I seen that neither Gail or myself is the first to spot this, its all over the place and often associated with change.

OK, thats a first stab at debriefing the conference, over the next few entries I’ll write more.

I attended a project retrospectives training session last week.  Although I’ve run quite a few retrospectives over the years I’ve never been trained in them.  My guide has been Norm Kerth’s Project Retrospectives book and a whole bunch of ideas I’ve picked up over the years.

This training session was organised and run by Rachel Davies and Diana Larson.  It drew on Diana’s new book Agile Retrospectives. 

Although the book and the training is aimed at Agile practitioners it could be used by other teams – its just that Agile teams are more likely to do this stuff.

The world outside of software development is also catching onto retrospectives.  In part this is thanks to the US Army and Marines who call them After Action Reviews (AAR).  Now they have a macho name and the backing of some generals they seem to be cropping up all over the place.

That said, I still think Retrospectives – or AAR to give them a more fashionable name – are still more talked about then practised.  The training session gave me an opportunity to reflect on this paradox (as well as learning some new techniques and improving my facilitation style.)

So why is it retrospectives get a lot of talk and little action?

There seem to be three basic failure modes.  First is retrospectives that don’t happen.  Usually the reason given is lack of time.  However I think this only an excuse to hide behind.  When people want to do something they do it, if something is important time is made for it.  Conversely when something is low priority or someone doesn’t really want to do it then there is not enough time.  We are all busy people, lack of time really means “is not a priority for me.”

I think one of the reasons people don’t want to do retrospectives is because they are scared of what might come out of it.  Individuals could be criticised, companies could be found wanting or someone may say something they are not supposed to say. 

Retrospectives expose the lie that your boss/manager/leader knows best.  By definition retrospectives are about asking teams what they can do better.  Traditionally we are brought up to expect that someone in authority, someone with a “big brain”, knows best.  So retrospectives can be threatening to managers who feel they are loosing some power.

Which brings us to the second common failure: ineffective results.  Simple formulaic retrospectives, perhaps they simply ask “What do we do right?  What do we do wrong?” and perhaps they limit the whole thing to an hour, and perhaps the team leader or project manager acts as facilitators.

Such retrospectives are unlikely to uncover deep thoughts because they are not asking the questions that will cause people to reflect deeply about the whole project.  Having a authority figure lead the retrospective will also cut off reflection, by leading this person is exercising their authority and sending a clear message “I am still in charge.”

Finally, the third failure mode of retrospectives is: Lack of action.  Things the team decide to do aren’t acted on.  This is actually closely related to the first two.  If it is hard to find the time for a retrospective do you have the time to make the change?  More likely you are risk averse.

And if the authority figure isn’t open themselves then they are not going to be fully behind the change.  This will be clear to anyone tasked with making the change.  That is, if anyone can be persuaded to take on the task.

Arie de Geus pointed out in his book The Living Company that when you have more people involved in making a decision it might take longer to actually make the decision but taking action once the decision is made will be quicker.  This is because more people were involved in making the decision.  If a retrospective is half hearted then any decisions will not have wide buy-in so taking action will be a slow process.

So what’s the answer to these problems?

Well I think I’ve realised were I was making my mistake.  Like many other people I’ve been seeing retrospectives in isolation.  As some kind of plug in solution, nicely self-contained.  But they are not, this was the mistake.

I’ve realised that retrospectives need to be embedded in a learning culture.  I used to think they could be one of the early tools to use in creating such a culture, now I realise they are an advanced tool to be brought in once you have the culture in place. 

An organization that does retrospectives is an advanced learning organization.  Most organisations are not learning organisations let alone advanced ones so it is hardly surprising that retrospectives are missing.

When you have such a culture not only do people value learning but it is a priority for them.  They make time for learning events.  They value the results and they take action.

In such a culture people feel less threatened by what might come out of a retrospective because they are accustomed to deep thinking and reflection.  Everyone knows the rules and people are open to insights and change.

In a learning culture managers have already accepted that the team knows best.  They have already changed the view of their role from command-and-control and “boss knows best”.  They define their role more in terms of helping individuals and teams learn and advance.  Neither are they threatened by loss of authority.

Finally, when you have such a culture, when people are not scared or threatened, when everyone appreciates the value of holding a retrospective then change does happen, people do follow through on the action identified in the retrospective.

So, how do we create such an advanced learning organization?  Well, that will have to wait for another blog entry and even my book.  In the meantime, if I can help with a retrospective or creating a learning culture please get in touch.

The SPA conference is on this week and I’m not there. SPA is a good conference but being so close to the ACCU conference I need a very good reason to go. Since I’m not speaking there I’ve skipped it.

The BCS SPA group also holds monthly meetings which I sometimes go along to. About three weeks ago I went along to a very interesting talk entitles “Large-Scale Agile Application Development”. In fact the talk was actually about how BT had adopted Agile development techniques. This was quite a challenge as those who know BT know that it is a very large company, a traditional telco and at one time a nationalised industry. There could be few more challenging environments into which to introduce Agile.

BT haven’t completed their Agile transition but it seems to be going well. Fortunately they had high level management backing on their side and all the kind of resources big companies can bring to bear on these problems: training, off-site boot-camps, consultants (mainly from Exoftware) and other resources.

I was particularly impressed by two of their ideas.

First was a plastic credit card like “pledge card.” I’ve seen these done by political parties in the past but this one was for “BT Core Agile Delivery Practises.” On one side was the Agile manifesto and on the other was BT’s core practises:

• • Customer collaboration

• • User stories

• • Iterative development

• • Automated testing
  • Continuous integration

The card strikes me as good because a) it was a constant reminder of the practises, b) it was very corporate so endorsed the idea that the company wanted this to happen.

The second idea was a pre-printed task card with the main fields a developer would use. I’ve seen index cards put through a laser printer before but this was more structured. It contained fields to help describe the work task, the priority and the acceptance criteria. Again good for re-enforcing the corporate message (“This is what we want”) and a reminder to teams as to what they needed to do. My only criticism might be that for such a small card it had a lot of fields to complete.

BT seems to have invested a lot in this programme. They had sent lots of people on training courses and held lots of off-site meetings and training.

They also spent a lot of time and money using coaching techniques. In fact they used pair-coaching with an experienced coach working with a new coach to help train the coach. To my mind this approach vindicates the coaching approach and shows how powerful it can be.

That say, it does seem the coaching model in use was very much drawn from the Agile coaching model rather than the model used by business coaches. The difference is that the Agile model is much more directive about what needs doing while the business model is much more open. Both have their place, I just wish Agile coaches were more aware of the other model and how it can help.

I went to a meeting of the UK Product Managers Forum last night.  This was the first time I have been to a meeting of this group and it is the first time that the group has held an event in London.  Naturally these two firsts are not unconnected!

I enjoyed the meeting, Ian Mapp of Respond spoke about the development of the product management function within the company.  This was very interesting itself, and as is so often the case the conversations with the other attendees was even more interesting.

These conversations support my belief that Product Management is not a well established role in the UK software industry.  When the product management doesn’t exist in a company there is a vacuum.  Without any guidance software developers will just develop the things they think are a good idea.  If there is a compelling need for your product this isn’t a problem in the short run: people need your product, good or bad they will buy it.

Nature abhors a vacuum and the same is true when product management is missing.  Somehow the vacuum will get filled.  In small companies it is often the founder(s), the CEO, CTO or similar who fulfils the role.  They often do it on gut feeling. If they are lucky they are right and the companies grow but then they don’t have the time to do it.

Sooner or later someone steps into the role.  It might be a project manager, a developer, an architect or even a tester.  These people do their best but they have other demands on their time – the job they should be doing.  There is competition not just for their time but their prioritisation decisions will be influenced by their duel roles.  Neither have they been trained for the role.

A lot of software development does wrong right here.  Without good product management to give customers a voice inside the organization companies will spend more time developing more features to sell fewer products.

(Product management can exist under the wrong name, I’ve met Architects who are really Product Managers and I’ve seen Project Manager roles advertised which are really Product Management jobs.  This just confuses things!)

The second thing I noticed at the meeting was how tight the profession is.  Most people knew of the Silicon Valley Product Group, 280 Group and Pragmatic Marketing.  Those of us who have ad product management training had had it through one of these groups in the US.  Again this is a sign of how underdeveloped the role is in the UK.  I believe this is about to change, Co-herence have licensed courses from the US and plan to deliver them in London.

The UK is not short of software companies.  It is short of product managers and product management skills.  Fixing the development process must involve product management.

If you are developing software application there are three things you must do:

1. Create a product management role (business analysis if you are in house)
2. Use source code control
3. Have a repeatable build process and run it at least once a night

If you aren’t doing these things then don’t bother leaving home, in fact, don’t even bother getting out of bed.  The scary thing is: most companies struggle to score two out of three on this list. 

(If I just scared you and you want help e-mail me, more on this next week.)

As an aside, Respond’s product is a customer feedback (i.e. complaints) management system.  Ian suggested that in some industries, notably airlines and mobile telecoms, the company got very little traction.  His conclusion was that for these companies customer service is not a high priority.  I can see his logic, if the companies in these sectors are not willing to spend money then it is reasonable to conclude that the companies do not see customer service as a priority.  One of the audience, form a major UK mobile phone operator took exception to this.  They claimed their company did take the issue of customer service and retention (churn) seriously. 

I’m in the process of reviewing my mobile telephone.  I’ve been with the same provider for 5 years now but I don’t feel like a valued customer.  The company has a memory like a goldfish – i.e. none – I might as well have been with them a year or a month.  I’d like to switch to another provider but a) I’m not convinced anyone else is better, b) it looks like my provider is actually the cheapest!

Wednesday’s Financial Times carried the monthly technology review. A couple of pieces caught my eye, both individually and when put together.

First this report In an e-world, IT controls the customer experience (subscription required) makes the point that if your business is conducted online then your customer experience is an electronic customer experience which in turn means the customer experience is decided by the IT department. This is a point I’ve made myself in the past – although I’ve never came up with such a compact phase to describe it.

In posts in January 2006 and December 2006 I sounded off against poor travel company web sites. My customer experience is poor because the web sites are poor. It doesn’t have to be this way but unless the IT department think about the customer experience, or the people responsible for the product involve themselves with IT this is what happens. Unfortunately IT departments have spent years refining their image as uncooperative.

Given the increased use of e-commerce by all businesses – not just those like Amazon which only exist on the web but everyday companies like the travel companies I dislike – then any business that does any e-retailing has this issue. The online brand can damage the off-line brand.

The second piece that caught my attention was this one Can HR handle IT recruitment? According to this article human resource departments do not have the skills and experience required to hire IT people. The story rings true from my own experiences and from many anecdotes I’ve heard over the years.

Although these stories all ring true I’ve also known several good HR people. People who have their heads screwed on and want the best for the business. Plus I’ve read a fair few HR text books and articles and I know HR people aren’t really as bad as IT people make out. So what’s going on?

Well I have some possible explanations.

Theory 1: Many companies have bad HR departments. There are some good HR departments out there but possibly most of them are not worth the salaries we pay them.

Theory 2: Everyone has these problems with HR, it isn’t just IT. HR is just badly understood by the rest of the business.

Notice that these explanations are not incompatible. Neither are they incompatible with the FT article. Which raises the questions: What are HR actually doing? Why aren’t HR departments getting to grips with IT and trying to rectify the problem? After all, strictly speaking, it is the HR departments job to hire people so it is they who have the problem. They can’t blame IT for this problem.

Put all these theories together and you get: HR is important, HR doesn’t understand IT, many HR departments are poor, many HR departments fail to hire the right people. As a result many IT companies do not hire the best staff. And because good IT depends on good people few companies will have good IT.

Proof for this theory comes from Google and Microsoft. Both of these companies have built hiring machine. They both need a lot of IT staff and they have made sure HR can do the job properly.

So the lesson is: If you want good IT you start by fixing your HR department.

Finally, put those two articles together.

IT controls the customer experience but HR do not know how to hire the people IT needs. In other words: HR departments do not know how to hire people who create the customer experience. Am I the only one who sees this as a serious problem here?

A student at Fullerton University in California e-mailed a lot of people on the Lean Software Development mailing list with some questions for his masters thesis.  Since I had a little time on my hands I wrote him some lengthy answers.  Having done that I though I would share them here.  I’ve removed a few questions because I don’t think the question or answer are particularly interesting.

1.      How would you define lean development?

The strictest definition of Lean Development would be to say it is a methodology proposed by the Poppendiecks (Lean Software Development)based on lean manufacturing and product development.  This definition is useful for deciding who is doing lean and who is not but is probably too strict.

Another definition of Lean (Software) Development is that it is the application of the principles of Lean Manufacturing ( The Machine That Changed the World, Womack, Jones & Roo ) and “Knowledge Based Product Development” (Product Development for the Lean Enterprise, Kennedy).

Personally my definition is:

• Lean Software Development is one application of the principles of Organizational Learning to the software development environment. 
• Learning Organizations may come up with a different development way of thinking but I am not aware of any others.
• Agile Software Development is a further refinement from principles of Lean to practices.
• The various Agile development methodologies are further refinements of these with further practises.
• Underlying everything is the idea of a Learning Organization (Senge, Argyris, etc.)

So you can create a pyramid, more general at the bottom, more specific at the top: 

– XP –	->	Specific practices
— Agile —	->	General practices
—- Lean —-	->	Ways of thinking
Learning Organization	->	Concepts

To put it another way:

• XP (or SCRUM, or DSDM, etc) are a specific set of practices for doing Agile software development
• Agile software development is set of principles and a specific set of lean practices
• Lean is a set of principles and a specific set of learning practices
• Organizational Learning is a set of principles to which some authors have added some practices

Note that you can, in theory, adopt an Agile methodology with very little learning. You could simply adopt, say, SCRUM practices and through focus and authority make people use these.  You probably would see some benefits however the team would not be learning and future change would not come naturally.

I examined Software Development as Organizational Learning for my own Masters dissertation

2.      What percentage of developers do you think are using Lean Development approach?

Very difficult to say, I think some ( less than 10%) will have picked a few practices, I think few will be working in a Poppendieck or Kennedy like way –  maybe less than 1% on this definition.

On the other hand, I think many are undertaking some organizational learning and using some agile practises but progress is very slow.

3.      How has the usage increased from previous years?

Slowly.

4.      Why do you think the usage has increased?

Various:

• The term is now commonplace thanks to Poppendiecks and others.
• Lean (and Agile) tell a very good story, developers want to work like this.  Managers see the parallels with lean manufacturing so also hear a good story.  It is an aspiration change.
• Natural selection: those who “get it” will out compete competitors.

5.      How do you expect the usage to be in the next five years?

Slowly.  I see an Agile backlash building, I hope lean will not be caught in it too but I think it will.

6.      Have you found resistance or pessimistic thinking towards Lean usage?

Yes – “Idealistic”, “Will not work here”

7.      What kind of unintended results have you seen by using Lean?

Lean relies on exposing problems so they can be fixed.  (Removing “camouflage” to use Argyris term and rolling back what Senge would call “shifting the burden.”)

As such it looks like things are getting worse to start with.  Problems that were hidden are now apparent, problems that could be ignored before need fixing now.  Therefore, Lean looks like it is making things worse before things get better.

When developers or junior managers get it but senior managers don’t there is a mismatch.  Junior managers pay a heavy emotional toll and sometimes pay with their jobs.

8.      What suggestions do you have for a shop considering Lean?

• Emphasis the learning. 
• Take it from both ends: senior managers down and shop floor up.
• Get help, specifically coaching.

10.     Can Lean development be applied to non-Agile (i.e. waterfall) methodologies?

You could make a waterfall leaner but I doubt we would ever call it lean.
In suggesting a lean waterfall you are saying: “We want to improve X but we don’t want to change Y”.  You have put boundaries on the problem.  In doing so you are putting some things out of bounds.  This means you can’t engage in Systems Thinking properly and are likely to come up with sub-optimal processes.

11.     How difficult is for a team become Agile and Lean after being non-Agile?

Very difficult.  You can start in a small way but you need to break with the past.

13.     What are the key indicators to tell if a team will succeed in becoming Lean?

The presence of a learning culture:

• Is the team holding retrospectives?
• Is the team changing their process?
• Is the team improving?

14.     Is it better to be Lean and then Agile or Agile and then Lean?

Given my definition above this question does not make sense.  There is no temporal relationship.

If you are Agile then you are (to some degree) Lean.  If you are Lean then you are Agile.

15.     What did I forget to ask you about lean?  Is there something about lean that you consider very important that we did not discuss?

The “change question”.  We know how to be Lean and Agile but how so you change the organization?

Then, when you have become Lean how do you carry on changing?

Organizational change is more important than the practices.  The practices support learning and change but change is not a rational thing.  You don’t do a value stream map and solve everything. 

OK, I promise this is the last entry on the Nurnberg funnel, after this I’ve worked it through my system…

In my entry on Nurnberg funnel and system development I wondered is system documentation actually works. Well I missed one piece of evidence that it does work. Go into your local bookstore, Waterstones, Borders, even Amazon online and look at the books on computing. Specifically look at the technical books that describe things like programming languages, operating systems, APIs and the like. If system documentation does not work how do we explain the existence of all these books?

After all, publishers are in the business of making money. If these books did not work – i.e. did not help learning and transfer knowledge – then presumably nobody would buy them, there would be no profit in publishing them and consequently they wouldn’t be taking up real-estate space in the book store.

So how do we explain these books if technical documentation doesn’t work?

I don’t have any figures on what is happening but here is my guess.

Firstly a whole load of this documentation does not work. Many books get bought despite being bad. Many more books are bad and simply don’t get bought, publishers swallow the loss and pulp the books. I would guess this category covers many many books. Lets take a stab and say 40% of all technical books.

A second category of books do work and they work for exactly the reasons Carroll outlined in the Nurnberg funnel. They are written along the lines of minimalism. My guess it that a lot of those “For Dummies” and similar books fall into this category. Still, this type of book represents a small percentage of all books. Lets say 5% – probably an overstatement but it keeps the numbers easy.

OK, I still have 55% to account for – and I’m deliberately not saying if these are percentage of books sold or books published.

Next up I think there are some books which do not directly take on board the minimalist ideas but have a similar approach and work for similar reasons. I’m specifically thinking here of patterns books like Design patterns and Pattern Languages of Program Design 5 (to which I was a contributor).

In part patterns are popular and accessible because they start with a problem and describe a solution. This is a little like the task orientation that Carroll advocates. I’m sure patterns books are not alone here but there aren’t very many books in this category so lets just say another 5%.

This also starts to hint at another explanation and one which must account for many of the remaining 50% of books. Simply minimalism is but one technique for creating documentation, it works particularly well in one field – user manuals – but it is not the only one.

For example I know of one other method, story telling. This is usually advocated as a management or knowledge management activity to be conducted verbally. (See books like The Springboard and Storytelling in Organizations). However there are technical books out there that take the same track.

For example, I found the original version of Inside Windows NT quite story like. Although perhaps this is a sign of how techno-geeky I was in the mid-late 1990’s. Somehow the second edition of the book never had the same plot or interest for me.

Less technical books about technical subjects are better examples of story telling. Take The Soul of a New Machine for example. This is a must read, I left it far too long before I read it but its right up there with The Mythical Man Month, Peopleware and The Psychology of Computer Programming if you want to know how software (and hardware) really get created. But Soul of a New Machine is also a story, it can even be bedtime reading.

Books like these are more the exception than the rule so they are few and far between. Lets say 1% are story books.

Another category is reference books like Charles Petzold’s “Programming Windows”. I defy anyone to actually read this books cover to cover, you delve into when you have a specific problem or need to know some specific detail. But reference books probably constitute a lot of the books written, say 20%.

That takes me to 71%. I still can’t account for 29% of books. I give up.

The important thing is that people read books for different reasons. Our motivation for reading one of these books will be different to the motivation for reading another. That will change the way we approach buying and reading books.

Which in a strange way brings us back to minimalism. People use Petzold’s books when they have a task to accomplish. People pick them up under certain circumstances and like a Dummies book expect to get something specific from them. But that doesn’t mean all books should be written in the minimalist style. Minimalism is just one of many styles.

So actually, we’ve come back to one of the re-occurring themes of this blog: know your customers, service their needs with a product that is designed to meet those needs.

Products development doesn’t end when the code is written, compiled and burnt onto a CD. It has to include the documentation, service and support that is supplied with the products. It also means ensuring that future product development can continue, whether through just-in-case documentation or just-in-time networks of people who know.

Unfortunately obvious solutions – like big thick manuals – are not necessarily the best solutions. We need to be on the look out for new solutions and new ideas.

And we need to understand the costs and benefits of all this. This is difficult bit because so often we don’t know the numbers and we can only speculate on what they are. But somehow we need to put all of that together.

After reading Minimalism beyond the Nurnberg Funnel I have been thinking what implication these theories could have for software development itself.

First for system design. Minimalist documentation aims at being task oriented. The idea is that the readers (users) are trying to do something and need to solve a problem – how do I? – so they turn to the documentation. What if we apply these ideas to system design?

Well I think we would come up with simplified systems that allowed users to perform tasks. The software would treat users as problem solvers and allow them to combine the tasks in what ever way they like to solve their wider problem. Now this starts to sound like Naked Objects.

Second: what if we turn our attention from user documentation to system documentation? Usually the documentation software developers write to describe the insides of the computer system is based on the Nurnberg funnel principle. We assume that is a software designer writes down everything they think they know about the software then someone who reads it will know the same information – open programmer’s head, insert Nurnberg funnel, pour documentation…. programmer can now understand software!

Obviously this isn’t going to work. This isn’t only true about existing software it is also true about systems we are going to write. Traditional software design (specifically the waterfall model) mandates that Architects and Designers – who in the traditional model are different to Programmers – should “design” a system, document the design and hand the design over to the Programmers so they will know how to turn the design into code.

Again, this is the Nurnberg funnel model and it isn’t going to work. When we follow this model we are lying to ourselves. So what are we to do?

Minimalism probably isn’t an answer in this case. Firstly the potential audience is so small it will be impossible to justify the extra cost. Secondly minimalism relies on describing the tasks people will want to perform. In general we don’t know the tasks that future system developers might want to undertake on our software. Yes we can guess what they might want to do but this really is little more than a guess. I have designed and built systems in the past with specific extension points for future developers – just add a new object here. But these are simple changes. The majority of the changes I’ve done to software involve expanding it in a way that wasn’t thought of in advance.

So while the thinking behind documentation minimalism can show the failings of traditional documentation it cannot provide a solution. We need to look elsewhere. I’m not sure I know the solution but I’ll make one suggestion.

The problem of making software developers familiar with a new code base is not a problem of documentation. Rather it is a problem of knowledge management and specifically knowledge transfer. Documentation is one tool in this arena but not the only one. Traditional documentation relies on a just-in-case model. Managers who are worried about risk have developers document the system just in case something needs to be done. This looks like a strategy to mitigate risk because we have “captured the knowledge” but in fact the risk still exists because the documentation is not very useful.

The alternative model is just-in-time knowledge management. Here we ensure that should a new developer need to know about an existing system they can ask someone who knows when they need to. The knowledge transfer then takes place directly from one developer to another.

Instead of the solution being based on something you have it is based on someone you know. On the face of this it this model looks costly. In order to get the knowledge we need a one-to-one relationship rather than the one-to-many relationship documentation provides. But since documentation does not work this argument does not hold.

In fact, writing documentation that might not be read, and if it is will not be understood represents waste. It is a waste of the developer’s time to write the documentation, it is a waste of paper to print the documentation, it is a waste of space to store the documentation and it is a waste of time to read the documentation. The only purpose it serves it to allow a box to be ticked on a risk form.

Using the just in time model all this waste is eliminated. If, and only if, the knowledge is needed then the cost of knowledge transfer will be incurred. And since the cost will be incurred in the future it is cheaper today. (I probably don’t need to spell this out but just in case: this model matches the ideas of Lean manufacturing, product development and software development.)

However, we are still left with the problem of mitigating the risk. To do this we need to keep track of the programmers who developed the system and have made changes since. This potentially changes the nature of the software development model but in today’s Internet connected world should not be insurmountable.

The problem then is that people, even developers, do forget over time. Still, forgetfulness might not be that much of a problem reality.

There are two types of system: those that are used and those that are not. The ones that are used tend to need changes over time; because these systems are living people think of new features and they need to be changed as the environment changes (think Y2K.)

Then there are the systems that don’t get used. Well if these systems are not used it is unlikely they need to be changed. Obviously for those systems that are being changed there are people who know how they work. For those that don’t change there might not be anyone who knows how they work but neither is there anyone who needs to know.

Of course there will always be the odd exception, a program that runs for 10 years without a change and then needs updating; but how many systems are there like this? Does the existence of a few systems that require very occasional changes justify writing all that documentation for all those systems were is isn’t needed, won’t be read and won’t be understood if it is?

I am still thinking through the implication of Minimalism Beyond the Nurnberg Funnel. One of these ideas concerns the benefits to the software producer of supplying minimalist documentation. One of these is the idea of documentation as a product strategy. This is covered by Karl L. Smart in an essay in the book.

Software companies all too often view the production of documentation as a cost, not an opportunity. Documentation is seen as a necessary evil since supplying software without documentation is not acceptable but writing it will push up costs. So companies produce documentation but try to limit the costs and miss the opportunity.

In my experience this usually means that too few technical authors are employed, those that are over worked and their activity is something of an after thought. They are not integrated into the software development regime; they get the product late and are expected to document everything they can. To cap it all technical authors are often among the first to be laid off when a company needs to make redundancies.

(Of course they assume that the role of technical author is even recognised, many producers do not see technical author as a role in its own right. Such companies make developers, managers and particularly testers write documentation in addition to their main job.)

Overall technical documentation is a lose-lose. Companies don’t see the benefits of providing it, users demand it but do not have high expectations and when they do turn to the documentation it is pretty useless.

I have seen a variation on this theme a couple of times, this is the highly complicated product. There is no way the user can learn the product from the documentation so they need training courses. Training courses can be a nice revenue earner for the supplier but they don’t normally don’t manage training very well so don’t make money from it. To the buyer training is yet another hidden cost. To the user there is not enough training, it is not good enough and it is another cost that can’t get signed off. Meanwhile the software continues to get complicated and everyone hopes the training and documentation will act as a sticking plaster.

Smart argues that providing quality documentation is actually a win-win for supplier and customer. Minimalism provides the road to useful quality documentation which benefits the product in a number of ways:

• Good documentation differentiates the product from its competitors
• Good documentation is more useful to users who in turn use the product more – and exercise the features – so they extract more value from the product so they are more likely to buy more product.
• Good documentation means users are more likely to solve their own problems than call the technical support line. Therefore the supplier can save money on technical support.
• Not only the supplier saves on tech support but so does the customer. Research shows that for every $1 of direct technical support costs a user needs they use $3 of informal support when they ask colleagues to help them.
• Minimalist manuals are usually shorter than traditional one so they are cheaper to manufacture. Although now documentation is usually supplied in electronic format this argument is reduced.

Producing minimalist documentation might cost more because it requires more time and analysis from writers but this is offset against savings suppliers can increase profits by producing minimalist documentation. (A word of warning: minimalist documentation does not mean producing less of the same documentation, there is more to minimalism in documentation than simply doing less. It means doing things differently.)

Because better documentation makes the product a better product it enhances the sale-ability of the product and therefore improving documentation is a strategic move on the part of the supplier.

This also means that documentation should be an issue for product managers. In addition to specifying features and requirements of the software product managers need to RTFM and find a way of improving the manuals.

I can’t remember where I came across Minimalism Beyond the Nurnberg Funnel but I’m glad I did. Still I think I may have read the wrong book, a better starting place for this subject would of been the authors first book on the subject The Nurnberg Funnel. Both books discuss the same concept, namely minimalism in technical documentation. However the newer book (Beyond) is not so much an introduction to the subject but a review of how the ideas had developed in 8 years. So I think the earlier book may have been a better starting position. Now, 9 years after the second book I assume things have moved on again.

The authors’ (Carroll is the editor of the new book but wrote the original one) concern is how people learn, and specifically how they learn from documentation. To this end the books are about technical communication. I think the ideas suggested would be interesting to user interface designers to look at too. The reference to the Nurnberg funnel comes from mythology.

Apparently – and I’ve not come across this myth elsewhere – the funnel was a device that allowed knowledge to be poured into someone’s mind. Imagine knowledge as some kind of liquid, insert the funnel into the head then pour the liquid into the head. Thats it! You know have the knowledge! This sounds like a great device.

Although I’ve never actually seen such a funnel they must exist because I know plenty of people and corporations who behave as if such a funnel exists. So it must just be that I’ve been unlucky.

Anyway, get back to the book… The originator of minimalism in documentation, John Carroll, suggests that traditional technical documentation is written on this principle. The authors working on a technical product (most of his examples come from word processors) fill the manual with everything they can think of. And over time the manuals grow as new features are added. For the users of the product the manual can be worse than nothing.

Research by Carroll and others shows that users just don’t absorb everything in the manual through Nurnberg like reading. Users find it difficult to find things and when they do they don’t want pages of explanation and technical details, they simply want to be told what to do. Indeed there is a paradox: people don’t sit down and read manuals. They want to accomplish tasks so they just get on with using the software. When they get stuck then they turn to the manual but they still do not want to read the manual, they want to do their task.

At this point a manual that explains technical details and system philosophy is exactly the wrong thing. Users will get frustrated and skip parts of the manual until they find the bit they want – that is, the bit that tells them what to do.

Actually I’m glad to hear this. This is the way I use manuals, I always suspected that other people used them “properly” but now I know: everyone else does the same as me.

The minimalist solution proposed by Carroll is to make the manuals task oriented. Work out what the tasks the users will want to perform and explain how to do these tasks. In the process recognise that different users will require different tasks so document them as such. Manuals should be organized so users can find these tasks rather than by technical logic.

There is more to minimalism than this but I’m not a technical author so I won’t go into detail. The upshot of it all is: users learn how to use the software more quickly, are more satisfied, are more productive and the manuals are often smaller than the non-minimalist equivalents. But what I do recognise here is that a lot of these ideas overlap with the user interface ideas proposed by Alan Cooper in his book The Inmates Are Running the Asylum . And the same idea populates good product management or business analysis:

Identify your customers, understand what they want to do, really understand what they want to do, then cater for that need specifically.

Whether this is in the product you build, the user interface you design or the documentation you provide it all boils down the same thing: understand and be specific. Don’t provide technology, provide the means to do something.

(Interesting I was talking to Tom Gilb at XTC on Tuesday. In his opinion one of the main reason projects fail is that they are not specific enough about what they are doing. There is a big conversation to have around the idea but not here, not today.)

Getting back to the book itself. If you are involved in technical authoring then I think it would be well worth reading either The Nurnberg Funnel or Minimalism Beyond the Nurnberg Funnel. If you are not technical writer then its not very clear if you would benefit from reading the book. If you want to know more about how people learn, or you are a product manager looking for ways to improve your products then it is worth familiarising yourself with these ideas.

I didn’t read all of this book, I read enough to understand the key ideas and understand what advantages they brought. This book is quite academic in style so it might be worth seeking out other books on minimalist documentation unfortunately I don’t know any so I can’t recommend any. There are more implications of this work than I have covered here. I hope to come back to these at some later date.