Agile

The summer lull has finally allowed me to get a lot done. In particular I’m very pleased to have finished a piece of writing I began some months ago: Bugs – Management Strategies.

This is an important piece for two reasons. Firstly I’ve been wanting to write this essay for a few years but never found the time. It has taken about four months in total and it runs to about 30 pages, now I know why it took so long.

Secondly, this is one of the corner stones for Xanpan volume 2. Volume 2 was going to be subtitled “Management Heuristics” but I’m now thinking it will deal more with organising so the sub-title might become “Management Heuristics and Organisation” or “Heuristics for organising software development” or some combination.

When I have a couple more chapters of volume 2 in place I’ll make it available on LeanPub. Until then you can register your interest in volume 2 right there.

In the meantime please read “Bugs – Management Strategies” and let me have your feedback. At this point I’m really posting it to get some feedback.

And if you haven’t already bought and read Xanpan volume 1: Team Centric Agile Software Development please do today!

(And if you prefer the a hard copy then get Xanpan at Lulu rather than LeanPub.)

I’m awake. I’m lying in my bed. I’m sweating but I’m cold.

Its the small hours of the morning and the dream is as vivid as it is horrid….

I’m standing in a clients offices, I’ve been here before, I know whats happening.

They are building an website. Quite a complex one, this will be the primary purchasing venue for many customers. This will project the company image – and with the right bits it can up-sell to customers – it can even help reduce costs by servicing the customers after the sale. All good stuff.

But it is atrociously “behind” schedule, someone said it would be finished in a year, that was three years ago before any code was written. Now its two years to completion but in my dream people say 2+3=2.

How can that be?

I can’t say it but the only way out I can see is cancellation. If I was suddenly in charge of the client I’d cancel the thing. I’d salvage what I could and I’d launch a new, smaller, initiative to replace the website.

But its too big to fail, even the board knows how much money they are spending. Who’s going to walk in there and say: “Scrap it.”

Saying “Scrap it” would be to admit one failure and invite a messenger shooting.

And if I was the head of the supplier I’d say the same thing. I’d say to my customer: “I know I’m earning oodles of cash out of this, I know its a high profile feather in our cap but really its out of control you really shouldn’t continue.”

But of course they won’t.

Forget the money they’d lose, they weren’t hired to answer back – like my tailor friend.

And of course I’m neither of those. I’m just the guy having the nightmare and in the nightmare I’m the consultant who is trying to fix, it. In the nightmare I’m not fixing it I’m providing cover, while I’m there its Agile, while its Agile its good, Agile is a good drug and I’m the pusher.

“You can’t cancel it because all the competitors have one and so we must have one” tells me a ghostly apparition.

“We must be best in class” says another apparition.

“We must be head-and-shoulders above the opposition” says third – aren’t the opposition seven times the size? And don’t the competition buy large parts of their solution off the shelf?

But every time I look the work seems to grow. Every discussion ends in more stories. Not just stories, epics, super-stories, sub-epics, mezzanine-stories.

But its OK, this is Agile.

The business keeps throwing new requests at it which are just accepted – because they are Agile!

Some of these are quite big. But that’s OK because the team are Agile. And Agile means the team do what the business want right?

I watch the Analysts work over the stories in the backlog, as they do each grows and replicates like an alien parasite. The Analysts find more edge cases, extra detail which need to be included, more scenarios which need to be catered for. Each becomes a story itself.

But that’s OK because the team are Agile.

And those damn competitors don’t stop adding and improving their site which mean the client must add that too.

But that’s OK because the team are Agile.

And the points…. points are the new hours, in the dream I have a book “The Mythical Man Point”. The backlog is measured in thousands of points. The burn-down charts go down – but only if you look at the sprint burn-down, hunt around Jira and you can find a project wide burn-down, O my god, no….. its full of stories!

This is not a burn-down chart carrying us to a safe landing, its a fast climbing interceptor…

The backlog is a demon… its… its… undead.

The faces of those who’ve seen the chart are prematurely aged. Open Jira and show someone the chart and…. their hair turns grey, the wrinkles appear, in moments they are….

One man is immune. As the points grow his power grows, he is… he is… The Product Owner.

He introduces himself: “Snape is the name, Severus Snape” – I knew I’d seen him somewhere before.

In the planning meeting, he sees the poker cards pulled out, he focuses on the developer with the highest score, there is a ray of cutting sarcasm… he withers. The developers submit, the numbers are lowered. The Product Owner chuckles to himself – no over estimating on his watch!

One of the developers suggest “Maybe we should wait until we finish the current work”

Snape sneers: “I thought you were Agile boy?”

“If you can’t handle it I have some friends in Transylvanian who are really Agile…. do you want to lose the contract boy? … Off-shore is so so cheap…”

There is a reality distortion field around the Product Owner. Show him a burn-down chart and it looks good, his presentations to the steering committee always show perfect burn-down.

I’m in my pyjamas standing outside the building at night: a sinister looking figure is breaking and entering, he sneaks into the building, he opens Jira and … inserts stories! His mask falls, it is….The Product Owner! Of course, without stories in the backlog he would cease to exist, his power comes from the size of the backlog, more stories more power.

Ever since his boss came down with a rare form of chronic flu a link in the reporting chain has been missing. Made worse when the next man up was dismissed for inappropriate behaviour in the canteen. Since when the Product Owner reports to the COO, a COO who doesn’t really have time for him and only has a shaky understanding of any IT related topic.

I do the maths. The backlog isn’t so much a backlog as a mortgage, and the team are under water! The payments they make against the mortgage aren’t even covering the growth in stories. The backlog growth is an interest rate they can’t pay.

It takes months for stories to progress through the backlog and reach developers. When work finally gets to developers they too uncover more edge cases, more details, more scenarios, more of just about everything. Why didn’t the Analysts find these? Did they find them and then lose them?

Then there is a stream of bugs coming in – oozing through the walls. The technical practices aren’t solid, they are… custard! Bugs get caught but more get through!

Bugs can’t be fixed because: “bugs are OpEx and we are funded from CapEx.”

Someone has slain the Bug Fixing Fairy, her body is found slumped in the corner, a nice your girl straight out of college. They are hiring another fresh young graduate to send to the slaughter, fortunately Bug Fixing Fairies are Plug Compatible with one another.

Release dates can’t be honoured.

Woody Allen and Anne Hall walk in – since when did Woody Allen do horror films?

‘two elderly women are at a Catskill mountain resort, and one of ’em says, “Boy, the food at this place is really terrible.” The other one says, “Yeah, I know; and such small portions.”’

I have X-Ray vision: I can see WIP where it lies, there are piles of it on the floor. Its stacked up like beer barrels in a brewery. But the beer isn’t drinkable. Its a fiendish plan. If anyone drinks that beer, if the WIP is shipped, they will discover…. its full of holes! Quality control is… offshore.

Why is there so much WIP lying around? Why is the WIP rising?

Because they are Agile comes the reply… the business can change their mind at anytime, and they do.

I’m drowning in WIP.

WHIP to the left of me, WHIP to the right of me.

The developers are half way through a piece of work and the team are told to put it to one side and do something else. Nothing gets delivered, everything is half baked. WHIP – work hopefully in process that is.

When, and IF, the team return to a piece of WHIP things have changed, the team members might have changed, so picking it up isn’t easy.

WHIP goes off, the stench of slowly rotting software.

But that’s OK because the team are Agile.

Arhhh, the developers are clones, they are plug compatible, you can switch them into and out as you like… but they have no memory….

It gets worse, the client has cunningly outsourced their network ops to another supplier, and their support desk to another one, and the data-centre to the another… no one contractor has more than one contract. Its a perverse form of WIP limit, no supplier is allowed more than one contract.

O my god, I’m flying through the data centre, the data centre supplier has lost control, the are creepers everywhere, each server is patched in a different way, there is a stack of change configuration requests in a dark office, I approach the clerk, its its…. Terry Gilliam, the data centre is in Brazil….

Even when the business doesn’t change its mind the development team get stuck. They have dependencies on other teams and on other some other sub-contractor. So work gets put to one side again, more WIP.

All roads lead to Dounreay in Scotland, a really good place if you want to build something really dangerous, but why does this project require a fast breeder nuclear reactor?

But that’s OK because the team are Agile.

The supplier is desperate to keep their people busy, if The Product Owner sees a programmer who’s fingers are not moving on the keyboard he turns them to stone. The team manager is desperate to save his people, he rummages in the backlog and finds… a piece of work they can do.

(With a backlog that large you can always find something even if the business value is negative – and there are plenty of them.)

You can’t blame the development team, they need to look busy, they need to justify themselves so they work on what they can.

But that’s OK because the team are Agile.

Get me out of here!!!!!

I’m in my kitchen.

My hands are wrapped around a hot-chocolate, I need a fresh pair of dry pyjamas but that can wait while I calm down. I’ve wrapped a blanket around me and have the shivers under control.

Are they Agile? Undoubtedly it was sold as Agile. It certainty ain’t pretty but it is called Agile.

They have iterations. They have planing meeting. They have burn-downs. They have a Scrum Master and they have Jira. They have User Stories. They have some, slow, automated acceptance tests, some developers are even writing automated unit tests. How could it have gone so wrong?

Sure the development team could be better. You could boost the supply curve.

But that would be like administering morphine. The pain would be relieved for a while but the fever would return and it would be worse. The real problem is elsewhere. The real problem is rampant demand. The real problem is poor client management. The real problem is a client who isn’t looking at the feedback.

The real problems are multiple, thats what is so scary about the dream. They are all interconnected.

In the wee-small hours I see no way of winning, its a quagmire: To save this project we need to destroy this project. But we all know what happened in Vietnam.

What is to be done? – I can’t go back to sleep until I have an answer.

Would the team be better off doing Waterfall?

The business would still change its mind, project management would put a change request process in place and the propagation delay would be worse. There would probably be more bugs – testing would be postponed. Releases would be held back. This would look better for a few months until they came to actually test and release.

If they did waterfall, if they did a big requirements exercise, a big specification, a big design, a big estimation and a big plan they might not choose to do it. But frankly Agile is telling them clearly this will never be done. In fact its telling them with a lot more certainty because they are several years in and have several years of data to look at.

Agile is the cover. Because they are Agile they are getting more rope to hang themselves with.

But all this is a dream, a horrid dream, none of this ever happened.

Earlier this week I was the guest of a large bank in the City, OK Canary Wharf actually. They had their own little internal Agile conference. As well as myself some of the usual suspects were on parade as well as some internal speakers. It was very enjoyable and as usual the C-List speakers were some of the most interesting.

Later in the day I found myself in conversation with two people concerned about software testing. They posed the question: “How do we make testing faster?” Specifically, how do we make SIT (“System Integration Testing”) and UAT (“User Acceptance Testing”) faster?

Now of these solutions are far from unique to this particular bank. Indeed after listening to the question and situation I recalled several discussions I’d had, problems I’d seen and solutions I’d suggested before. (OK bad consultant me, I should have enquired into the problem, understood the context fully, coached her to find her own solutions etc. etc. But we only had 30 minutes and I wasn’t being paid for a full consulting session – nor am I charging you for reading this! – so lets not invent the wheel, lets fill in with assumptions and given some general answers.)

The solutions we came up with were:

1. More testing environments: it is shocking how often I hear testers say they do not have enough test environments. They have a limited number of environment and they have to time-share or co-habit. There are two answers: either increase the number of test environments or reduce the number of test and testers. I’m sorry I can’t give you a magic sentence that persuades the money people to buy you more environments. Now we can virtualise machines there is very little excuse for not creating more test environments. The usual one is cost. But even here the costs are not that high. (Unless that is you need to buy a new mainframe.)
2. Testers integrated & co-located with developers: if you want to go faster I’ve yet to find anything that beats face-to-face direct verbal communications. If you want/need people distributed then you should accept you will not be as fast as you could be. I’ve said it before and I’ll keep saying it: Testers are first class citizens, treat them as you would developers.
3. Automate SIT: simple really, automate the hell out of testing. Well easy to say, harder to do but far from impossible. Since most UAT/Beta testing is exploratory in nature it is largely SIT that need automation. And please please please, don’t spend money on tool test tools.
4. Make UAT just UAT: Recognise that some of what passes as UAT is actually SIT and automate it. In large corporates an awful lot of what is called “User Acceptance Testing” isn’t “User Testing” – although it may well be “Acceptance Testing.” Look carefully at what happens in this phase: that which is genuine Acceptance Test can – indeed should – be specified in advance (of the coding), can be moved into the SIT phase and automated. The true User bit is going to be much more explorative in nature.
5. Improve development quality with TDD and do it test first: one reason for long SIT and UAT phases is that they find bugs. One way to significantly reduce the number of bugs is to have developers practice Test Driven Development, i.e. build in quality earlier. This may mean coding takes longer in the first place but it will speed up the wider process. TDD is still quite new to many developers so you can speed up the process by giving them some help, specifically TDD coaching from someone who has done lots of this before. Until developers make the jump from “post coding automated unit tests” to “pre-coding test first development” you will not maximise the benefits of TDD, you will have more code in your system than you need and you will have a poorer overall design, and both of those make for more tests later on and potentially more problems. When tests are written before code TDD becomes a design activity.
6. Reduce batch size: i.e. reduce the size of the release, and probably do more releases. Do less, do a better job, push it out more often, reduce risk. Think dis-economies of scale. Many large corporations use a large batch size simply because any release is so painful. This is completely the wrong approach, rather than running scared of releases take them head on.
7. Done means done and tested in the iteration – at the end of the iteration it is releasable: when you reach the end of an iteration and declare work “Done” that means “Done ready for release (to customers).” Well, that should be your aim anyway and you should be working towards that point. If “Done” currently means “Done development, ready for SIT” then you should be working towards bringing SIT inside the iteration. If that looks hard right now then look again at items 1 to 6 in this list. And when you’ve moved SIT inside the iteration start on UAT. Until both those phases (and any other phases in the tail) are inside the iteration they can throw up problems (bugs) which will disrupt a future iteration which implies the work you said was Done is not Done.

Now the catch. Money.

Some of these things require money to be spent. In a rational world that would not be a problem because there is plenty of evidence that increased software quality (by which I mean specifically few bugs and higher maintainability) leads to shorter schedules and therefore reduced costs. And as a plus you will get more predictability. What’s not to like?

So rationally its a case of spend money to save money. Quality is free as Philip Crosby used to say.

But unfortunately rational engineers often find it difficult to see rationality inside big corporation operating annual budgeting cycles. One of the reoccurring themes this week – although not one I think was given official time – was the constraints of the budgeting cycle. The rational world of the engineer and the rational world of the account and budgeteer seem a long way apart.

There is a solution but it is a big solution which will scare many people. It is Beyond Budgeting, and that is why I was so keen to invite Bjarte Bogsnes – the author of Implementing Beyond Budgetting – to talk about Beyond Budgeting at this years Agile on the Beach conference. And its probably why Bjarte recently tweeted “Invitations to speak about Beyond Budgeting at Agile and HR conferences are now as frequent as Finance conference invitations. Wonderful!”

“Gentlemen, you are getting married. You go to the best tailor in town, and you say: ‘I’m getting married, my bride is beautiful, make me look one million dollars, I don’t care what it costs.’

The tailor measures you very carefully then he stands back. He rubs his chin, he is clearly troubled by something. ‘Tell me’ you say, ‘Please tell me.’

‘Sir’, he replies, ‘I don’t think it is my place to say’

‘Tell me’ you beg, ‘I must looks fantastic for my girl’

‘Well Sir, …’ he talks slowly and cautiously, ‘. if you really want to look $1,000,000 can I suggest you lose 5kg?’

What do you say?”

This is the dilemma many a consultant find themselves in. More importantly it is the dilemma that I find again and again in corporate IT:

• Is IT a service department which is expected to build what is requested without answering back?

Or

• Is are the IT departments the experts in maximising benefit and value out IT investment?

Is the tailor a master of clothes who just makes beautiful clothes? Or are they, because of their long experience making people look beautiful in clothes, an image consultant who can help you look beautiful?

Is your IT department a tailor who is expected to produce something brilliant no matter what the request is? Or are they people who, because they work with information technology every day, have experience and knowledge to help their (internal) customers maximise their benefit? (Even if that means the original request should be reconsidered.)

I distinctly remember a senior IT manager at an American bank telling me that his department was most definitely a service department, it wasn’t his job to answer back to business requests, his job was to build what was asked for. The implication being that even if his people knew the (internal) client was approaching the issue from the wrong point, and even asking for the wrong system, then his people should not correct them. They should just build what was asked for.

Perhaps this dilemma is most accurately felt by Business Analysts who are sometimes expected to write down the orders that someone else wants to give to IT even if they know better. Fortunately some Business Analysts can go further and advise their “customers” on what the bigger questions.

The original metaphor come from Benjamin Mitchell. As I recall it was part of a very deep conversation we had in Cornwall back in 2011 and which also led to the “Abandon hope all ye who enter Agile” blog post. I’ve embellished and elaborated the story over the years.

I reminded Benjamin of this story in January – he’d forgotten about it, thats how brilliant he is, he has so many great ideas he can’t keep track of them all. Once I’d reminded him he immediately made it better.

“How would you know?” he asked, meaning, “How can you determine what is expected of you?”

For me the story is about missed opportunities, about frustration and about what you might do to change the position. For Benjamin it is about asking “Do we all expect the same thing? – If you think they consider you a service group how can you confirm this?”

Either way the core problem is about a mismatch of expectations. Your IT department might be a service department, that approach could work, its how many big corporations operate (I my experience).

Or the IT department could be the experts in technology, how to apply technology, how to frame problems so technology can help and how to maximise the return from technology.

Both are valid options.

But the real problem is when someone in the department thinks the department is should be doing more than the rest of the company does; or when the rest of the company expects more from the department but the department sees itself as a service group.

Finally, go back to my series on the economics of software development, the supply and demand curve, and apply this story. If the IT department is seen as a tailor who shouldn’t answer back they will find it very difficult, if not impossible, to change the demand curve. Such a tailor can only work on the supply side.

In my last post (Inconvenient Truths of Project Status Reporting) I used an expression which I think deserves highlighting and explaining:

        False Projects

False projects occur when we use the word “project” for work which is not really a project. For example:

• There is no end date to the work, it goes on and on, rightly or wrongly.
• The team is not broken up, it goes on and on, rightly or wrongly, i.e. the team is not a temporary structure.
• The output of the work is not defined in advance, or might not even be defined.

Some might think I’m playing word games here but I think its important, I think we need to be clear about our terms. Please, hear me out. (This is really important when we talk about #NoProjects and #BeyondProjects. If this is a new idea to you see my Why Projects Don’t Make Sense post.)

PRINCE 2 (the UK project management standard) offers the following definition of “a project”

“A temporary organization that is needed to produce a unique and predefined outcome or result at a pre-specified time using predetermined resources.”

This is the definition which is most commonly violated. PRINCE 2 helpfully offers a second definition of a project:

“A management environment that is created for the purpose of delivering one or more business products according to a specified business case.”

This seems like a pretty open ended definition, none of the terminating conditions of the first, thats part of the problem. Potentially they are very different. Under this definition a false project would be a piece of work which is called “a project” but for which there is no specified business case or defined products.

Lest I be too UK centric, the American Project Management Institute offers this definition (I have highlighted the parts which concern me most):

“More specifically, what is a project?

It’s a temporary group activity designed to produce a unique product, service or result.

A project is temporary in that it has a defined beginning and end in time, and therefore defined scope and resources.

And a project is unique in that it is not a routine operation, but a specific set of operations designed to accomplish a singular goal. So a project team often includes people who don’t usually work together – sometimes from different organizations and across multiple geographies.”

As with the first PRINCE 2 definition the emphasis is on an activity that ends. In this definition the team “don’t usually work together”, there comes a point when working together becomes normal.

An awful lot of software development work just doesn’t correspond to any of these definitions but we call it a project. And a lot of work – whether we call it a project or not – is managed by a Project Manager which kind of implies someone, somewhere, thinks of it as a project.

I call such work a False Project.

The problem is: when we use the language of projects – when some people assume the work conforms to one of the above definitions, specifically when they think there is an end, or a temporary arrangement – then we create a conflict. We create confusion. We create a cognitive dissonance.

When we manage a piece of work as “a project” when it isn’t then at the very least we use the wrong language and tools. Worse still we create expectations that can’t be met. Part of the #NoProjects / #BeyondProjects logic is about fixing this conflict, closing this gap.

Personally I have a little hypothesis but I don’t know how to prove it – or disprove it. If someone has employment statistics on managers, middle managers and project managers for the last 30 years we might be able to find some supporting evidence.

My hypothesis is: starting in the 1980’s middle managers became unfashionable, many of them got laid off in mass downsizings. But that doesn’t mean all the management work went away, sure some did but some remained. I suspect the growth in projects and project management stems from this time. I suspect that much of what gets called project management is in fact good old fashioned junior or middle management dressed up so it might be added back into organizations which have previously rejected middle management or want “flat hierarchies” (and other such oxymorons).

Long time readers of this blog may recall that I subscribe to the high-brow MIT Sloan Management Review. They may also remember that I’m never quite sure it is worth the rather high subscription fee. But once in a while an article comes along which is worth a year’s subscription. The “Pitfalls of Project Status Reporting” is this year’s article.

As the title suggests the article looks at why project status reporting, specifically for IT projects, so often fails to alert companies of the problems in project work. (Clearly the authors are unfamiliar with #BeyondProjects/#NoProjects but I suspect most of the same problems can be found in the false-projects so beloved of IT organizations.)

If you want the full story you will need to buy your own copy of the article (Amazon have it for £2.50) but for the rest of you here are the highlights, the 5 points the authors called “Inconvenient Truths” plus some of the text and a couple comments from from myself. The comments are all my own but the quotes – and the headlines in bold – are from the article which is research not conjecture, something we could do we more of!

1. Executives can’t rely on project staff and other employees to accurately report project status information and to speak up when they see problems.

“[software project managers] write biased reports 60% if the time and that their bias is more than twice as likely to be optimistic.”

Basically people don’t like reporting bad news and worry that reporting it will reflect badly on themselves.

2. A variety of reasons can cause people to misreport about project status; individual personality traits, work climate and cultural norms can all play a role.

“employees who work in climates that support self-interested behaviour are more likely to misreport than employees who work in climates based on ‘rules and code’.” Why do I think of banks and bonuses when I read this?

3. An aggressive audit team can’t counter the effects of project status misreporting and withholding of information by project staff.

“Once auditors are added to the mix… a dysfunctional cycle that results in even less openness regarding status information.”

“Diminished trust in the senior executive who controlled their project was associated with an increase in misreporting.”

More reporting will make things worse, if auditors arrive then people don’t feel they are trusted and guess what? They might not show the auditors the dirty washing and might tend to paint things as good.

4. Putting a senior executive in charge of a project may increase misreporting.

“research suggests that the stronger the power of the sponsor or the project leader, the less inclined subordinates are to report accurately.”

I find this suggestion very worrying because some people in the Scrum community insist that the Product Owner must be an executive (“the real product owner”) who has real power to secure the resources and changes a make the project a success. This research suggests that having a strong, senior, Product Owner could make things worse.

5. Executives often ignore bad news if they receive it.

I am reminded of one client where my own attempts to raise a red flag have gone no-where. On my final visit to the client I spoke with the senior architect to again voice both my concerns over the work and the failure of anyone to listen. Unfortunately he had the same problem. He saw the same problems but couldn’t find anyone willing to listen.

My initial thought on all of this is that this is all the more reason to base reporting on deliverables, i.e. what has actually been delivered that is working and in use, that is rather than reporting on proxy “status” criteria, report actual progress, actual deliverables.

A few weeks ago I did a presentation entitled “Conway’s Law and Continuous Delivery” – although it was also at some point entitled “Conway’s Law and Organisational Change” or possibly “Conway’s Law and Change for Continuous Delivery” – to the Pipeline Conference in London.

The presentation was very well received at the time and I spent most of the next hour talking with people who wanted to talk about the presentation. One tester came to be later and said “Thank you, I understand what I’m doing now, I’m testing the organisation.”

Since then the presentation has had a lot of interest on Twitter and on SlideShare – at one time it made the front page of SlideShare (my 15 minutes of fame I guess).

I should say I took a slight liberty with the “organisational change” bit. I didn’t so much describe how to bring about organisational change as describe how the organization might (even should) change to align itself with Conway’s Law. Sorry about that, I don’t have a potted “how to change your organisation” recipe (although if you want to call I’m available for hire!)

Actually the Pipeline conference was the second time I’d made this presentation. The first outing was for the London Continuous Delivery group and hosted at the Financial Times. If you missed these two outings the good news is that the FT delivery was recorded and is available on Vimeo. This presentation is also on SlideShare but isn’t significantly different from the Pipeline version.

(Personally I feel that was the better of the two presentations but others have told me the second was the better, perhaps what the second lacked in raw freshness was more than made up for in being a little more polished and debugged.)

I’m really grateful to Matthew Skelton, Steve Smith and Chris O’Dell who are the people behind both London Continuous Delivery and Pipeline. Although I’ve known about the idea of continuous delivery for a while I know much more and I’m really excited by it.

They have also furnished me an opportunity to revisit Conway’s Law. Nearly 10 years ago I ran a workshop exploring Conway’s Law at EuroPLoP 2005. Both my co-organizer and I learned a lot, in fact the fuss about our write up rambled on for a couple of years – I’m not going to reopen that here and now. (The full report “What do we think of Conway’s Law now?” is available for download from allankelly.net.)

Ever since then one of my mental models has been the organisation structure is the software architecture, I may no longer change code but by changing organizations I am practicing software architecture.

Right now my head is buzzing with thoughts on Continuous Delivery, Conway’s Law, Team Structure, Scaling Agile and how this all fits with the Beyond Projects / No Projects agenda. I feel that if I can just make sense of all this I’m on the verge of some great breakthrough!

‘‘Knowledge work is not easily defined in quantitative terms, . . . To make knowledge work productive will be the great management task of this century, just as to make manual work productive was the great management task of the last century.’’ Peter Drucker (1969)

In recent months I have repeatedly been reminded of an argument, or perhaps a hypothesis, I advanced in my 2008 book Changing Software Development. In the introduction chapter I suggested that software developers might be regarded as the prototype of future knowledge workers. I’ve republished “Prototype of Future Knowledge Workers” as a blog post yesterday.

I’ve been reminded of this in a number of casual conversations recently, like the one I had with Laurie Young (@wildfalcon), again when rereading some Peter Drucker and because of media pieces like one in the Financial Times (“Do it like a software developer” by Lisa Pollack) suggesting that companies with successful software development teams might have something to teach other companies, and also by the topics submitted to the Agile on the Beach conference.

Let me explain, firstly let me explain why I make this argument, then let me explain the significance – and with reference to Agile – and finally, for completeness let me quote from the book…

Software Developers – programmers, testers, and everyone else – are what academics would call knowledge workers. We work with our brains rather than our brawn. We generate value because of what we know and what that knowledge allows us to do.

There are lots of other knowledge workers, teachers, lawyers, doctors, and during the twentieth century the number of knowledge work professions and number of knowledge workers grew.

In many ways software developers are no different, but in one very important ways they are very different. Software developers are able to create new tools for machines which allow them to change the way they do knowledge work.

Think about some of the tools your common knowledge worker uses during the typical day:

• E-mail: invented by Ray Tomlinson, a programmer
• Spreadsheets: invented by programmers Dan Bricklin and Bob Frankston
• The Web: invented by programmer Tim Berners-Lee
• Wiki sites: invented by programmer Ward Cunningham

Even if Twitter, FaceBook, Video conferencing, Voice over IP and countless other knowledge sharing and collaboration technologies were not invented by programmers I will bet good money on one common factor: software developers, programmers, had early, if not first access to these technologies. That is very very significant.

Lets be clear:

• Unlike other knowledge workers software developers have the ability to create new tools which can change their work.
• Software developers have early, if not first, access to tools which benefit knowledge work.

One might also add that people working with technology welcome and embrace new tools and technologies (on the whole) so the barriers for technology change are lower.

This means those working in software technology – and particularly those developing software – have more experience and most history of using these tools, and consequently are the likely to be the most adapted to working with these tools.

Thus if you want to see how new technologies, new tools, will change knowledge work look at those who create the tools and those closest to them. This shouldn’t be a surprise, after all knowledge work is about managing and using information, and these technologists create the tools we use to store, retrieve, manage, manipulate, combine, and share information.

(And there is a difference between knowledge and information but I’m not going into that here, its in the book.)

By itself this argument is interesting and perhaps good for developer self-esteem. We could leave it there but…. it also begs a question:

If software developers are the prototype of future knowledge workers what does it suggest will happen next? Specifically, how can this insight help answer the challenge posed by Peter Drucker above?

As a step towards answering those two questions let us rephrase the question:

• What are software developers doing today which might benefit other workers?
• Or to put it in Drucker’s language: what can managers of non-software staff learn from software teams which will make their teams more productive?

One way would be to look at the tools software developers are using today which might benefit other groups.

Version control springs to mind but version control has existed for at least 30 years and hasn’t caught on with the masses. We could continue to examine the toolset, one of my favourites is Markdown. Or we could look at the way software developers are managed. This was the thrust of the FT piece which looked at personnel management (I refuse to say HR) practices.

But there is something else… Agile itself.

One of the reasons I believe Agile came about was that the tools we use changes. Cobol gave way to Java which gave way to Ruby. OS/360 gave way to VMS which gave way to Windows (or was it Unix?) which gave way to Unix derivatives. One might say paper gave way to the web and wiki’s, telephones gave way to VOIP, telegraphs gave way to Twitter.

Our tools changed which allowed – even forced? – our processes to change. As knowledge workers outside of IT embrace these these tools its possible that their working practices will change in a similar way. My favourite example is a presentation given by Kate Sullivan at Agile on the Beach two years ago in which she described how the legal department at Lonely Planet adopted Agile.

There are several Agile practices which area are easier to imagine outside of software than others. For example:

• Visual boards: lots of places!
• Standup meetings: example abound and usually predate Agile, e.g. bar staff, Japanese office workers, NATO commanders (in Bosnia and probably elsewhere)
• Test driven working: consider many of the business examples from Lean Start-Up
• Pair programming: airline pilots and surgeons have been doing this for a long time but Kate added Pair Lawyering
• Iterations: although I’ve yet to get a concrete example most news organizations clearly operate in daily iterations

During the second half of last year I was involved with a client whose project was to write documentation. Specifications to be precision but not specifications for software – there was an element of that but that was not the aim. We used iterations, visual board and other techniques. We even used a walking skeleton. It worked. We delivered to schedule.

We in software have developed something good. Something that is spreading beyond software. Now I wonder, whats next?