agile

Online User Stories tutorials now complete

Leave a Comment / Agile / By / October 18, 2021 / ,
Better User Stories
As a Product Owner I want to write better stories

I’m pleased to announce I’ve released the last of my online User Stories tutorials (part 5: Workflow and Lifecycle) and with that the whole series is complete. You can now buy the entire User Stories set of 5 tutorials as one package at a 40% discount to buying the tutorials individually.

The package includes over six hours of video commentary, exercises, quizzes, downloads and both ebook and audio book versions of Little Book of Requirements and User Stories.

Blog readers can get a further 25% off the price with the code: “blogoct21” until the end of this month, October 2021.

In addition, the first 3 people to use that code will receive a free print copy of The Art of Agile Product Ownership.

The 5 tutorials are:

These tutorials turned ouit to be a lot more work than I expected (where have i heard that before?). The core material is based on the Requirements, Backlogs and User Stories workshop that I have been running for a few years and last year converted to a series of online webinars. In the process the material has become a lot more focused.

Please, let me know what you think, in the comments section below or in the feedback forms at the end of each tutorial in the series.

Why on-ramps and off-ramps are more important than highways

Leave a Comment / Agile / By / August 13, 2021 / , ,

It begins with a simple request: “we need to know when it will be done.” Or, when there is an agile-savvy manager, “our velocity needs to be higher.” But the more I look the more it appears the dev team aren’t really that bad, in fact they might actually be good. And, if you doubled team productivity overnight it wouldn’t make a big difference. The problem is elsewhere.

Sure the dev team could be better in many ways but simply coding faster isn’t going to solve the problem. The on-ramp and the off-ramp are in need of improvement: the work intake and the work delivery mechanism – entry ramp (getting stuff in processed) and exit ramp (getting it out the door) are often more imporant.

As they say: its déjà vu all over again. I see this again and again. In my mind’s eye turning requests into working software is a freeway, a motorway, an autobahn – a controlled-access highway to use a technical term. Each piece of work is a car.

Most of our attention goes on the cars/work speeding down the lanes, that is where we assume time is spent. That is where most of the team are working, that is where we direct people to look for problems. If all goes well the work/car moves rapidly from one place to another. Sure things go wrong on that journey, in coding, sometimes other pieces of work get in the way, sometimes something goes wrong and there is a pile up with work/cars queuing behind. And sometimes the best way of improving the overall flow is to limit work in progress or reduce speed limits.

But, the actual speeding down the highway part is but one of three essential elements. Frequently this is not where time is lost, and even though work can be unpredictable it is not the most unpredictable part of the work.

It is fairly common for work to spend most of its life waiting to enter the system – the on-ramp, how cars get on the highway and how work enters the development processes.

And there is the off-ramp – how does work leave the system and reach the customer? – after cars only join the highway when they are coming from one place and want to get to another.

Most people working in the system see their job as driving cars and ensuring that a particular payload is delivered to the destination. Who looks at the overall system? Who manages the highway? Who optimises the flow? This is where I see my work. It is not enough to ensure a piece of work is delivered, it is not enough to ensure the cars are going fast, one has to see the whole system. Usually the on-ramp and off-ramp require far more work than the actual highway itself.

In other words: it is not about ensuring any one car arrives once. It is about ensuring the system for delivering cars works effectively. While the highway journey gets the attention the on-ramp and off-ramp are often far more important.

Consider the off-ramp: it is very common to find that development teams are working pretty well, but when work is “done coding” it queues to get through testing, queues to get into a release and queues to be released. In fact, it is almost normal in teams that work spends longer “getting out the door” than it spends being done.

The continuous delivery movement has done a lot to improve this and the best teams have streamlined and automated this part but problems remain. I’ll just mention two.

One: I just said “the best teams.” The best teams are few and far between. Yes they get lots of attention but most teams are a long way from this. It is not uncommon to find that teams consider some continuous delivery processes madness. I floated the idea of branchless development to a team this year and they took it as a sign that I didn’t understand their work. The idea that you might not use source control branches appeared like a naive beginners mistake.

Two: where do you put testing? If testing is considered a special activity that must happen as part of a release process then it occurs on the off-ramp. That off-ramp has limited capacity and any problems have big knock on effects – it is very risky.

However, testing can be considered part of the main highway experience. Developers can work to a high standard an incorporate practices like TDD and BDD which lesson the need for testing. Formal testing – probably by professional testers- can be positioned before the off-ramp if you design the highway/workflow correctly.

Now consider the on-ramp: the intake process, the requirements process, the work-before coding, work that is normally done by Product Owners/Product Managers or Business Analysts. This can cause even bigger hold-ups than the off-ramp.

I’ve written before about the fear many organizations have of actually coding. As a result work is held in perpetual review, estimation and planning before it is allowed anywhere near a coder.

Another cause of delay is the product backlog: in many places this is a bottomless pit of work to do. Every few weeks the Product Owner shifts through the backlog selecting a few pieces of work to get done. Most work doesn’t get done and falls to the bottom. It is unlikely to be done but gets in the way and distorts metrics. As a result most work spends most of its life cycle waiting to be done, waiting to get onto the highway.

There is a natural (and good) tendency to focus on the work in hand, to think “if I can only get this piece of work done…” Like Orwell’s Boxer pledging “I will work harder” to any problem. (There are plenty of none team members prepared to stand on the sidelines saying “If only they did work harder.”)

It is not enough for any one person to work harder. It is a system: the an on-ramp, a highway and and off-ramp all need to work together. Only looking at the whole do these things become clear. Improving this flow requires a different set of skills to those of writing code and testing – of course there is overlap in skills and of course people can learn; but again, if one simply pledges to “work harder” and write better code the improvement will be marginal.

Seeing the highway – the work flow – is something I would expect a development manager to do, and if not a development manager than the person I call and Agile Guide and most of the rest of the industry calls an Agile Coach.

Subscribe to my blog newsletter and download Continuous Digital for free – normal price $9.99/£9.95/€9.95

OKRs top-down? bottom-up? or ripples in a pond?

2 Comments / Agile / By / May 5, 2021 / ,

One of the great things about writing a book is that you get a greater understanding of the subject. So it was with “Succeeding with OKRs in Agile”. In particular writing the book forced me to think about how OKRs fitted with agile and hierarchal structures. I get the impression that many people are interested in using OKRs to align teams but not everyone has worked out all the nuances.

On the face of it OKRs are hierarchal: it seems “obvious” that someone, somewhere, is going to set a big goal, that goal will cascade down and every team will end up with their own mini version of the goal. As I said, this seems obvious, how else could it be? Especially in a large organization?

After all, is that how the not so distant ancestors of OKRs, Management-By-Objectives (MBOs), worked.

This also fits the engineer’s mind: the product team have a goal, and all the supporting teams – whether contributing components or services – make their goals subservient to the one goal. The classic inverted tree with each team doing what the node above asks.

But, top-down conflicts directly with reality and with agile. Teams don’t have one goal, they don’t answer to but one leader but to multiple leaders, multiple customers, multiple stakeholders and these don’t always agree. Agile folk have long railed against command-and-control from above while advocating self-managing or self-organising teams. Surely OKRs go against this philosophy?

So, if we are to use OKRs in an agile environment these positions need to be reconciled. In Succeeding with OKRs I described the process as more bottom-up than top-down. Thus, rather than a big boss saying what should happen and that being cascaded down the origanization to provide goals at every level, I describer the big boss setting out a vision, a goal, an objective but not describing details. Then they say to the teams: “Help, how can you help more us towards that goal?”

Now, only a few months after I wrote that my thinking has moved on. I don’t disagree with myself but I see the need for a more nuanced explanation and a revised model.

First off, I’m guilty of using the language of hierarchy: top-down and bottom-up. In so doing I’m supporting the view that hierarchies are the natural state of things and creating a, possibly false dichotomy: one thing or another. For years I’ve been thinking of organisations both as federal entities and as solar systems. While the leadership team may be central to decisions they are not all powerful . Teams have their own paths, leadership and leadership teams are the sun and teams/divisions orbit them. (If I recall correctly I picked this idea up in the Henry Mintzberg book Simply Managing.)

Bear in mind, as I say in everyone of my books: team are autonomous. We stive for independent teams with devolved authority. Each team exists to deliver a product or service and each team has multiple stakeholders – of whom the big boss is but one. Each team therefore has to decide how best to deliver benefit to (potentially competing) stakeholders. Sometimes that means co-ordinating with other teams and even other companies.

Put that together: teams are creating OKRs but they are not doing it in isolation, they are listening to the big bosses at the centre but also the teams they need to work with.

Recently I’ve started to think of these concentric circles less as planetary orbits and more as waves, or ripples to be precise. The big boss at the centre makes big ripples that carry out to the edges.

But leaders are not the only ripple makers. Teams, customers and other stakeholders also have an effect – like rain falling on water. Sometimes these waves some together and magnify each other, other times they cancel each other out, more often than-not they are out of sync and disrupt each other in ways too complex to predictable.

We think of leaders as single water droplets but inreality there are lots of drops making lots of ripples

OKRs are the messaging system that allows teams to signal what ripples they are creating and which they are reacting to. Teams are iterating – OKRs reset every 13 weeks – which means every quarter teams get a chance to react to other ripples and rest their own.

Thought of like this you also get a scaling model. Not so much a model of “how to do this to scale” but a mental model which describes how to think about scaling.

I have some upcoming presentations and webinars about OKRs if you would like to know more

Or, buy the book “Succeeding with OKRs in Agile”

Subscribe to my blog newsletter and download Continuous Digital for free – normal price $9.99/£9.95/€9.95

OKRs in Agile Q&A part 2: The Backlog

Leave a Comment / Agile / By / March 24, 2021 /

Continuing from my last post and more questions arising from my Reawakening Agile with OKRs talk.

In my OKR book I advice teams to forget about the backlog and instead use the OKRs as a story generating machine. As I expected, this grabs people’s attention. For many this might be the most surprising piece of advice in the boo, and perhaps the most controversial.

So it is hardly surprising that there were several questions around this:

Q1: If the backlog isn€™t a reflection of what we need to do in order to move towards our vision what is it for?

Exactly. If the backlog does not reflect your vision then what is the point of a backlog?

First off, if the backlog is aligned with your vision, and things are working well then why change? Certainly don’t add OKRs unless they are addressing a problem, and when you add OKRs see if there is anything else you can remove. If OKRs are simply repackaging the backlog then why both? Why add to the tools and processes in use?

For a few fortunate teams that is indeed the case. However, for many teams the backlog also contains a multitude of work which is not part of the vision and requested fixes. The backlog long ago became a dumping ground for requests.

Yet removing work from the backlog becomes hard. Product Owners feels they lack the authority or confidence to actually say “No, we will not do that.” At the same time the teams performance is judged by “how much backlog” gets done. Success or failure come down to “is the backlog done?” Thus the backlog comes into conflict with the OKRs.

In the book I introduce Jeff Bezo’s “Day 1” approach where company works as if today is the first day and questions what they are doing. OKR setting needs to be a day-1 experience.

Q2: Backlog needs reviewing to align with OKRs, surely?

That is one approach, set the OKRs and then before or during every planning meeting comb through the backlog and find work which will move you towards the OKR. That will work if you have a few dozen items in the backlog but what if you have a thousand or more? What if the backlog has been passed down from a previous product owner or a requirements document? In both cases that work will be harder.

The bigger problem is: what if you think of something that is not in the backlog and will move you towards the OKRs?

Do you say No?
I expect not, I expect you will quickly write it, slip it into the backlog and say “look what I just found”

Now the backlog is a collection of ideas which might, or might not, help achieve the OKRs but which you might not do.

At which point, what is the point? Why not just brainstorm what you can do?

Q3: Isn’t OKR then a guidance to create Backlog? or prioritize it?

Create a backlog, yes, the OKR is guidance to create a backlog – its a story generating machine. So what is the point of having 500 stories which describe work not related to the current OKRs? Will not get done anytime soon? And likely will never be done? But which confuse the governance process and drawn everyones morale because “we still have 500 PBIs to do.”

Once you have your OKRs then there is little point in creating any more backlog then you will do in the next three months. You may generate 12 months of work but since OKRs are reset every three months the chances are three quarters of those stories will be thrown away.

So every quarter reset the backlog and start over. That is pretty much what I’m advocating.

Prioritizing the backlog, see Q2 above.

Q4: How have you approached the removal of backlogs? small experiment?
Q5: Have you done this in real life? How did you persuade people?

OK, you found me out, we didn’t actually throw the backlog away. There was some history in there which was useful and more importantly it would have drawn too much attention to the team. Instead we just ignored it.

This started as a small experiment between me – as Agile Coach – and the Product Owner: we just opted to ignore it.

We set the OKRs based on current priorities and strategy. Then in the planning meetings if we knew of a story in the backlog we pulled it up. But actually, this turned out to be more work than it was worth. Plus, by challenging the team we got better answers and more involvement.

It didn’t take long before the team noticed what was happening but I don’t think they minded much. Again they might say “O I know there is a story in there to do…” but more generally we just wrote new stories there and then: the OKRs were a story generating machine.

In the book I describe a further experiment I ran with another coach, as a result she came to the same conclusion: OKRs before backlog. While we shared this with other coaches – and indeed anyone who wanted to talk about it – we didn’t make a big fuss or publicise it. I’m doing that now.

Succeeding with OKRs in Agile

I see this approach as the logical conclusion of working with OKRs so I don’t think you need to make a fuss. You can just do it and I expect more teams to reach this conclusion. Except of course, those backlog-slaves who are labouring under corporate agile to burn-down the backlog!

Succeeding with OKRs in Agile is available in print or ebook format from Amazon now, an audio version will be out in the next few weeks. I will be repeating Reawakening Agile for Agile Newbury next month and discussing OKRs with Adrian Reed in May.

Subscribe to my blog newsletter and download Continuous Digital for free – normal price $9.99/£9.95/€9.95

Purpose over Backlog

2 Comments / Agile / By / March 3, 2021 / , ,

Backlogs are a good idea. Backlogs ease the transition from the old “requirements up front” world to the new more dynamic agile world. Backlogs provide a compatibility layer for agile teams to interface to more traditional project management and governance. Backlogs even allow you to take a stab at done date!

Backlogs allow you to even out work between the quiet periods and the busy times. Backlogs give you a place to store good ideas which you can’t do just now. And because stakeholders can see their request is not forgotten they don’t need to shout for it today.

Listen to Allan Kelly reading Purpose over Backlog

Yes backlogs are good. I’ve seen them work well myself and I’ve taught many teams to effectively use backlogs.

But – you knew there was a but coming didn’t you? – but…

Backlogs have problems, too many teams are labouring under the Tyranny of the Backlog, they have become backlog-slaves and practice something we might call BLDD – Back Log Driven Development.

(To be clear, when I say “backlog” I am primarily thinking of the product backlog – the long list of all the things the team (might) do in the future. This is different to the sprint backlog (iteration backlog). The sprint backlog is a shorter list of things the team aims to do this iteration. I am using Scrum terminology but the ideas are pretty much “generic agile” and I’m thinking more broadly than Scrum. Many implementations of Kanban feature a product backlog of sorts so while Kanban is less prone to these problem it is not immune.)

1) Lump of Work Fallacy

There is usually an assumption that the backlog represents all the work to be done – an impression reinforced by early implementations of Scrum. In the short term that leads to agile teams being seen as inflexible and prioritising process over need because new work is not allowed in.

In some cases teams even struggle to get started on work because a big-up-front requirements gathering and analysts activity is required to create a backlog. In the worst cases that work is even estimated and end-dates forecast before a line of code is cut or developers hired.

In the longer term it is simply unrealistic to assume the backlog is fixed. Even with more and better analysis it is impossible to foreseen future requests. The agile adage “it is in doing the work that we understand the work” cuts both ways: coders understand what they need to build and customers/stakeholders/analysts understand what they want.

Work will arrive after you begin, any system that does not incorporate that truth will fail one-way or another.

2) Bigger than you think

Not only does the backlog grow with completely new work the work in it changes – and grows. There are many reasons this happens: new opportunities appear, hidden ones become clear, requests require more work than expected and so on.

Humans are very bad at estimating, especially about the future, and, it turns out, they are also very bad at estimating time spent in the past. If you want accurate forecasts you need to invest in them, you need to make structural changes and you need to use statistics.

However, because of the lump of work fallacy and the belief that humans can make estimates, poor end-date projections get made and when they are missed (because they were wrong to start with) everyone gets upset.

3) Fallacy of Done

Backlogs come with burn-down charts and an assumption that there is an end; and that end is when everything is “done.” The team will be done when the backlog is empty. That assumption is baked into BLDD, traditional project management and even governance.

I have long argued that software is never done. I’ll accept that I might be wrong, but in the digital age, when business runs on digital technology (i.e. software) your products are only done when you business is done. The technology is the business, and the business is the technology. Stop the backlog growing, stop growing you technology and you kill the business.

4) Backlog Bottomless pit

Put all those reasons together and the backlog becomes a bottomless pit. In the early days of agile, when I managed teams myself, the backlog would often sit on my desk, written out on index cards and held together with rubber bands. I could get a sense of how big the backlog was my looking.

Today everyone uses electronic tracking systems. Not only do these allow an infinite number of items they rob us of perspective. To paraphrase Comrade Stalin: “2 outstanding backlog items is tragedy, 200 is a statistic.”

5) Backlogs obscure strategy & purpose

With so many backlog items it is easy to get lost – you can’t see the wood for the trees. Arguments over what will be done next start to resemble deciding who should get a lifeboat place on a sinking ship, add in the demands “when will you be done?” (plus explaining why the date has changed) and “the bigger picture” gets lost.

In Back Log Driven Development the sense of purpose and strategic goals is lost as teams struggle with the day-to-day demands of just doing stuff.

6) Powerless product owner (i.e. backlog administrators)

Tyranny of the backlog seems worst where product owners lack real authority and skills. They are little more than backlog administrators. They spend most of the week adding requests to the backlog, then passing a few chosen items to developers in planning meetings. A vicious circle develops, the product owner can’t win so people trust them less, their authority wanes, and the backlog spirals.

Few organisations give product owners the power needed to get a grip on this situation. Indeed, many product owners are plucked from the ranks for development or support and given a battlefield promotion to product owner but lack the skills required. (See The problem with Product Owners.)

A solution?

For years I’ve been suggesting teams throw away the backlog – you will not forget the important things. But then how do you know what to do?

Take a step back, start with your purpose, your mission, the reason you team, your company, your organisation exists. What should you be doing? How can you fulfil that purpose and serve your customers?

This is where I see a role for OKRs and jobs to be done. Both these techniques – together, or separately – can be used as story generators. Every time you need to more work, more stories, you return to your OKRs and ask “what can we do now to move us towards our objective?”

When writing Succeeding with OKRs in Agile I became more and more convinced this is the path to take. Increasingly I sum this up as Purpose over Backlog.

Post publication addition: great suggestion fro Jari Mäkelä on Twitter to call this: Purpose Driven Development, PDD.

Step 1: Clarify your purpose – what is your overarching reason for existing?
Step 2: Clarify how your existing strategy builds towards that purpose, and if you don’t have a strategy create one.

Repeat steps 1 & 2 annually.

Step 3: Think broadly, set your OKRs as a team so you build towards your purpose by following your strategy.
Step 4: Spend the next 12 weeks executing against those OKRs

Repeat steps 3 & 4 every 3 months.

Step 5: In each planning meeting take stock of what you have done and progress against OKRs
Step 6: Ask “what do we need to do next to move towards the OKRs?”

Succeeding with OKRs in Agile

Repeat steps 5 and 6 every 2 weeks

And if you are Kanban’ing then keep steps 1, 2, 3 and 4, adjust 5 and 6 as appropriate.

Having finished, completed, published Succeeding with OKRs I really wish I had been clearer in the book. The ideas are there but with time they have become so much clearer… maybe I need another book.

Buy Succeeding with OKRs in Agile at Amazon today.

Subscribe to my blog newsletter and download Project Myopia for Free

Succeeding with OKRs in Agile book cover

Best seller – Succeeding with OKRs in Agile

Agile / By / February 10, 2021 / ,

I’m delighted that my new book Succeeding with OKRs in Agile went on sale at Amazon yesterday. By this morning it was the number #1 best seller in Amazon’s IT Project Management category – and not doing badly in Computer Programming and Business Management & Leadership either. (Although the publisher has some power over which categories a book is in it is still a black-art.)

It is hard to express just how great it is to see the book in the number #1 slot. While I hope it stays at #1 for a while I expect it will drop down before long.

Print and audio versions of the book are in the works and should be released in the next few weeks so if you would rather read a physical version or listen, watch this space as they say.

The book has taken a little under a year to write and a few more months to make production ready. The wonders of LeanPub mean many readers have already been enjoying early versions of the book. If you would like to read the book on iBooks or as a PDF then LeanPub is the place to buy from.

I recorded the little video below to explain why I wrote the book.

Warning sign

Warning signs of a failing outsourcer

Agile / By / January 11, 2021 / ,

It is 2021 and unfortunately on Friday I felt the need to repost “Dear Customer, The Truth about IT“. Little has changed in the 10 years since I wrote the original – if I was writing it today probably the only thing I would change is “IT”, I’d write “Digital” (I should probably also change Manchester United but …).

Unfortunately the vast majority of supplier’s are engaged on the basis of their marketing materials, sales pitch and promises. This tells you nothing about their actual ability to deliver working software. The suppliers can all hire great marketing people and use the same words. They can hire and incentivise the best sales people, and they can all take you out for a good meal, a round of golf or to a strip-club. (O, and they can all find a few “satisfied customers” to provide a testimony.)

The only real way to know if a supplier can deliver is to see them in action. So how can you tell things might be going wrong? What are the warning signs?

With help from Mike Burrows and John Clapham I’ve came up with this list of early warning signs. We were thinking in the context of a client-supplier (outsourced) relationship but many of them apply if you are working with internal teams too.

Staffing

1) Supplier loads teams up with extra managers: test managers a speciality
1.1) Team members don’t make decisions and defer problems to managers: there is a manager for every problem
1.2) Offshore teams have parallel management hierarchies
1.3) Suppliers feel the need to mark all your managers with their own manager (who is then duplicated offshore)

2) Inverted staffing pyramids (few devs at the bottom, lots of managers, BAs & other non-coders above)

3) People get swapped by suppliers with little notice
3.1) Short term substitutions are made: I once saw a supplier bring in a temporary SAP HR consultant to cover the usual consultant’s 2-week holiday. There was no way the substitute could come up to speed in that time let alone contribute positively.
3.2) People bait & switch: the people you meet first met didn’t last long, they were substituted for inexperienced people
3.3) “I can do that” – you get people new to their role, you get who they have available, people with experience in one role fill another role; a project manager plays coach, a delivery manager plays scrum master

4) Part time assignees (particularly managers): work a few hours a week on the project, see 1.1.

Get ready

5) Long running “set up” phases
5.1) You spend longer pondering the future than the time it takes to create the future
5.2) A lot of time is spent agonising about infrastructure changes rather than just doing them
5.3) Team advocates for, and does, investment in infrastructure and “reusable code” before anything is usable is actually delivered

Reporting not delivering

6) Supplier does not deliver working software

7) Supplier does not deliver working software every two weeks

In 2021 delivering working software to production every two weeks, or at least usable, potentially releasable software, is table stakes. The best teams deliver multiple times a day. If the supplier can’t deliver something by the end of week 4 you have a second rate supplier. Get out now.

8) Reporting hours done rather than demonstrating working software and stories

9) “Watermelon report” Green on the outside when everything inside is Red; impressive looking reports which don’t distract from the fact that nothing, or very little, was actually complete
9.1) Claiming stuff is done when it hasn’t finished testing
9.2) A Definition of Done which leaves work not-done – Mike has a good post at agendashift.com/done.

Other warning signs

10) You invest as much time in their org design as your own, if this starts to include people performance monitoring and management what are you gaining over using your own people?

11) Suppliers always say yes: no push back and no negotiation, feedback and scrutiny of your requests are signs they are paying attention to your needs. It you ask for the impossible it is better the supplier tells you so than accepts what you ask for. Ideally you want a supplier who can highlight the difficulties with your suggestion and work with you to achieve something akin to what you want even if you have to rethink your request.

12) Your own people are disenfranchised/disgruntled/frustrated by the arrangement. Particularly noticeable where people are expected to work in a different time zone to suit the other partly and when outsourcer staff are elevated (faster, smarter, etc) over the existing people.

In most of these cases the supplier is working around their own constraints rather than putting your needs first.

Subscribe to my blog newsletter and download Project Myopia for Free

An unhappy customer complains

Dear customer, the truth about IT

Agile / By / January 13, 2021 / , ,

10 years on I feel the need to repost this classic letter from the IT industry to our clients.

Audio version, read by Allan Kelly.

Dear customer,

I think it’s time we in the IT industry came clean about how we charge you, why our bills are sometimes a bit higher than you might expect, and why so many IT projects result in disappointment. The truth is that when we start an IT project, we don’t know how much time and effort it will take to complete. Consequently, we don’t know how much it will cost. This may not be a message you want to hear, particularly since you are absolutely certain you know what you want.

Herein lies another truth, which I’ll try to put as politely as I can. You are, after all, a customer, and, really, I shouldn’t offend you. You know the saying “The customer is always right”? The thing is, you don’t know what you want. You may know in general terms, but the devil is in the detail – and the more detail you try to give us beforehand, the more likely your desires are to change. Each time you give us more detail, you are offering more hostages to fortune.

Software engineering expert Capers Jones believes the things you want (‘requirements’, as we like to call them) change 2% per month on average – thats close to 27% over a year once you compound changes. Personally, I’m surprised that number is so low.

Just to complicate matters, the world is uncertain. Things change, and companies go out of business. Remember Enron? Remember Lehman Brothers? Customer tastes change. Remember Cabbage Patch Kids? Fashion changes, governments change, and competitors do their best to make life hard. So, really, even if you do know absolutely what you want when you first speak to us, it is unlikely that it will stay the same for very long.

I’m afraid to say that there are people in the IT industry who will take advantage of this situation. They will smile and agree with you when you tell them what you want, right up to the point when you sign. From then on, it’s a different story; they know that changes are inevitable, and they plan to make a healthy profit from change requests and late additions at your expense.

While I’m being honest, it is true we sometimes gold-plate things. You might not need a data warehouse for your online retailer on day one. Yes, some of our engineers like to do more than what is needed, and yes, we have a vested interest in getting things added so that we can charge you more.

It is also true that you quite legitimately think of features and functionality you would like after we’ve begun. You naturally assume something is ‘in’ when we assume it is ‘out’. And, in the spirit of openness, can you honestly say that you’ve never tried to put one over on us? (Let’s not even talk about bugs right now: it just complicates everything.)

Frankly, given all this, it is touching that you have so much faith in technology to deliver. But when IT does deliver, does it deliver big. Look what it did for Bill Gates and Larry Page, or Amazon and FedEx. Isn’t it interesting that when the IT industry develops things for itself, we end up with multi-millionaires? When we develop for other people, they end up losing money.

How did we ever talk you into any of this? Well, we package this unsightly mess and try to sell it to you. To do this, we have to hide all this unpleasantness. We start with a ritual called ‘estimation’ – how much time we think the work will take. These ‘estimates’ are little better than guesses. Humans can’t estimate time. We’ve known this since at least the late ’70s, when Kahneman and Tversky described the ‘planning fallacy’ in 1979 and went on to win a Nobel Prize. Basically, humans consistently underestimate how long work will take and are overconfident in their estimates.

To make things worse, we have a bad habit we really should kick. Between estimating the work and doing the work, we usually change the team. The estimate may be made by the IT equivalent of Manchester United or the New York Yankees, but the team that actually does the work is more than likely a rag-tag bunch of coders, analysts and managers who’ve never met before.

Historical data – data about estimates, actuals, costs, etc – can help inform planning, but most companies don’t have their own data. For those that do have data, most of it is worse than useless. In fact, Capers Jones suggests that inaccurate historical data is a major cause of project failure. For example, software engineers rarely get paid overtime, so tracking systems often miss these extra hours. Indeed, some companies prohibit employees from logging more than their official hours in their systems.

So we make this guess (sorry, ‘estimate’) and double it – or we might even triple it. If the new number looks too high, we might reduce it. Once our engineers have finished massaging the number, we give it to the sales folk, who massage it some more. After all, we want you to say “yes” to the biggest sticker price we can get. That might sound awful, but remember: we could have guessed higher in the first place.

Please don’t shoot me: I’m only the messenger.

We don’t know which number is ‘right’, but to make it acceptable to you, we pretend it is certain and we take on the risk. We can only do this if the number is sufficiently padded (and, even then, we go wrong). If the risk pays off, we get a fat profit. If it doesn’t, we don’t get any profit and may take a loss. If it’s really bad, you don’t get anything and we end up in court or bust.

The alternative is that you take on the risk – and the mess – and do it yourself. Unfortunately, another sad truth is that in-house IT is generally even worse than that provided by specialists. For a software company development is a core competency – such companies live or die by their ability to deliver software, and if they are bad, they cease to trade. Evolution weeds out the poor performers. Corporate IT on the other hand rarely destroys a business – although it may damage profits. Indeed, Capers Jones’ research also suggests specialist providers are generally better than corporate IT departments.

Sales folk might be absent, but the whole estimation process is open to gaming from many other sources and for many other reasons. The bottom line: if you decide to take on the risk, you may actually increase risk.

I know this sounds like a no-win scenario. You could just sit on the fence and wait for Microsoft or Google to solve your problems with a packaged solution, but will your competitors stand still while you do? Will you still be running a business when Google produces a free version?

Beware snake oil salesmen selling off-the-shelf applications. Once people start talking about ‘customisation’ or ‘configuration’, you head down a slippery slope. Configuring a large SAP installation is not a matter of selecting Tools, Options and then ticking a box. Configuring large packages is a major software development activity, no matter what you have been told. The people who undertake the configuration might be called ‘consultants’, but they are really specialist software developers, programmers by another name.

There really isn’t a nice, simple solution to any of this. We can’t solve this problem for you. We need you, but you have to work with us. As the customer, you have to be prepared to work with us, the supplier, again and again in order to reduce the risk. Addressing risks in a timely and cost-effective manner involves business-level decisions and trade-offs. If you aren’t there to help, we can either make the decision for you (adding the risk that you disagree), or spend your time and money to address it.

You need to be prepared to accept and share the risk with us. If you aren’t prepared to take on any risk, we will charge you a lot for all the risk we take on. Sharing the risk has the effect of reducing the risk, because once the risk is shared you, the customer, are motivated to reduce risk. One of the major risks on IT projects is a lack of customer involvement. You can help with that just by staying involved.

Ultimately all risk is your risk: you are the customer, you are paying for the project one way or another. If it fails to deliver value, it is your business that will suffer. When you share risks, when you are involved closely, risks can be addressed immediately rather than being allowed to fester and grow.

Finally, you may have grand ambitions, but we need to work in small chunks. I know this may not sound very sexy, but software creation works best when small. Economies of scale don’t exist. In fact, we have diseconomies of scale, so we need to work in tiny pieces again, again and again. If you are prepared to accept these suggestions, then let’s press ‘reset’ on our relationship and talk some more.

Yours sincerely,

The IT Industry

Dear Customer was first publishing this blog nearly 10 years ago, a polished version became famous in Agile Journal (now Agile Connection) a few months later and forms the prologue to Xanpan, 2015.

Subscribe to my blog newsletter and download Project Myopia eBook for Free

My story, my why

1 Comment / Agile / By / January 5, 2021 / , ,

I thought I’d open 2021 year with a personal story of how I got where I am today (no, I’m not in San Francisco, although that is the Golden Gate in the picture)

Allan Kelly on the north side of the Golden Gate bridge, 2001.

I started programming when I was 12 (ZX81 then BBC) and then led a very successful career into my 30s – including a spell in California. Increasingly I found the code was not the challenge, I could make the code do what I wanted. The problem was the way we were managed, or mismanaged, the things we were ask to do and the way we were organised.

So began my journey into “management”. Determined to be a better manager than those I had worked for I took myself back to school. During a year in business school I learned a lot of good stuff, I discovered “organisational learning” and I reconnected with my dyslexia.

“Agile” was just breaking at the time and in agile I saw the same ethos of learning I was getting so excited about. The reports of agile teams I read described the best aspects of the developments I had worked on. For me, managing software delivery and enhancing agile are the same thing.

My mission became to help my younger self: help technologists deliver successful products and enjoy satisfying work. Most of what I do falls under the “agile” banner but really it is about creating the processes and environments were people can learning, thrive and excel.

When people are getting satisfaction from their work delivering great products, businesses succeed and grow. And as software has come to underpin every digital initiative my work has expanded.

For my latest blog posts, give aways and special offers on books and training subscrbe to my newsletter – and as a thank you download my Project Myopia eBook for free

Programmer’s Rorschach test

Agile, Business / By / August 1, 2017 / , ,

The picture above, I recently added this picture to Continuous Digital for a discussion of teams. When you look at it what do you see:

An old style structure chart, or an organization chart?

It could be either and anyone who knows of Conway’s Law shouldn’t be surprised.

When I was taught Modula-2 at college these sort of structure charts were considered superior to the older flow charts. This is functional decomposition, take a problem, break it down to smaller parts and implement them.

And that is the same idea behind traditional hierarchical organizational structure. An executive heads a division, he has a number of managers under him who manage work, each one of these manage several people who actually do the work (or perhaps manage more manager who manage the people who do the work!)

Most organizations are still set up this way. It is probably unsurprising that 50 years ago computer programmers copied this model when designing their systems – Conway’s Law, the system is a copy of the organization.

Fast forward to today, we use object oriented languages and design but most of our organizations are still constrained by hierarchical structure, that creates a conflict. The company is structurally decomposed but our code is object oriented.

The result is conflict and in many cases the organization wins – who hasn’t seen an object oriented system that is designed in layers?

While the conflict exists both system and organization under perform because time and energy are spent living the conflict, managing the conflict, overcoming the conflict.

What would the object-oriented company look like?

If we accept that object oriented design and programming are superior to procedural programming (and in general I do although I miss Modula-2) then it becomes necessary to change the organization to match the software design – reverse Conway’s Law or Yawnoc. That means we need teams which look and behave like objects:

  • Teams are highly cohesive (staffed with various skills) and lightly coupled (dependencies are minimised and the team take responsibility)
  • Teams are responsible for a discrete part of the system end-to-end
  • Teams add value in their own right
  • Teams are free to vary organizational implementation behind well defined interface
  • Teams are tested, debugged and maintained: they have been through the storming phase, are now performing and are kept together

There are probably some more attributes I could add here, please make your own suggestions in the comments below.

To regular readers this should all sound familiar, I’ve been exposing these ideas for a while, they draw on software design and Amoeba management, I discuss them at greater length Xanpan, The Xanpan Appendix and now Continuous Digital – actually, Continuous Digital directly updates some chapters from the Appendix.

And like so many programmers have found over the years, classes which are named “Manager” are more than likely poorly named and poorly designed. Manager is a catch all name, the class might well be doing something very useful but it can be named better. If it isn’t doing anything useful, then maybe it should be refactored into something that is. Most likely the ManagerClass is doing a lot of useful stuff but it is far from clear that it all belongs together. (See the management mini-series.)

Sometimes managers or manager classes make sense, however both deserve closer examination. Are they vestige from the hierarchal world? Do they perform specialist functions which could be packaged and named better? Perhaps they are necessary, perhaps they are necessary for smoothing the conflict between the hierarchal organization and object oriented world.

Transaction costs can explain both managers and manager classes. There are various tasks which require knowledge of other tasks, or access to the same data. It is cheaper, and perhaps simpler, to put these diverse things together rather than pay the cost of spreading access out.

Of course if you accept the symbiosis of organization and code design then one should ask: what should the organization look like when the code is functional? What does the Lisp, Clojure or F# organization look like?

And, for that matter, what does the organization look like when you program in Prolog? What does a declarative organization look like?

Finally, I was about to ask about SQL, what does the relational organization look like, but I think you’ve already guessed the answer to this one: a matrix, probably a dysfunctional matrix.