Agile

I recently came across the expression “The Scrum Wall”, as in the expression “Hitting the Scrum Wall”. Its akin to “the pain barrier”, or “feel the burn” in aerobics workouts of old.

Once I heard it I knew what it was, I’ve talked about this before, now I know it has a name.

The Scrum Wall is the thing Bob Martin described at the ACCU conference. And it is probably why Jeff Sutherland endorses Test Driven Development and other technical practices from XP.

You hit the Scrum wall when you adopt Scrum and everything goes well, then, after a few Sprints things don’t work any more – to use an English expression, they go pear shaped. You can’t keep your commitments, you can’t release software, your customers get annoyed and angry, it looks like Scrum is broken.

This is what happens when you adopt Scrum without technical practices such as Test Driven Development, continuous integration and refectoring. When teams adopt the Scrum process, they go faster, show progress, things look good… and then the quality becomes a problem. Now the team are fighting through quick sand.

The code quality is poor and developers are expected to continue to make progress. Maybe the Scrum Master/Project Manager reverts to past behavior and demands overtime and weekend working. Maybe the team start busting a gut to keep their commitments. Either way the team is heading for burn-out.

That’s the wall.

About a decade ago I worked for a major supplier of financial information in London. The company decided it wanted to improve its software development process and the answer to this problem was to adopt CMM – the Capability Maturity Model. Initially the become level 2 and then move on the level 3.

The little group I worked in interfaced to financial trading institutions in the City of London. My project in particular had to respond quickly to changes at the institutions.

The CMM roll-out was a akin to a neutron bomb in reverse. The people were left in place, but their processes and practices were replaced en-mass. Developers attended CMM training and instructed in how to develop software according to the new process.

The effect was devastating. One manager likened it to pouring quick setting concrete on the railway tracks of progress.

I went out and I bought myself a book on CMM so I could understand it. What I learned was that CMM was a way of measuring maturity. It was a meter, imaging it as a ruler. It was a way of assessing your organization. You could measure yourself as 1 CMM, or 2 CMM, up to 5 CMM. It was not a method of working.

The CMM body of knowledge also contained recommendations, “best practices” and examples of good practices but it was not itself a process. It was a meter for measuring your maturity.

From the book I learned that Watts Hymphrey advised companies to: focus on improving their processes first and let the CMM measures improve by themselves.

What I observed in practice was the destruction of working practices an processes. I never observed any attempt to involve those doing the work in process improvement. The consultants knew best. Opposition was to be overcome. Process improvement was something that was done to others.

This wasn’t the company for me and I left. The company was eventually certified as CMM Level 2 and was expected to move to level 3 in time. Time went by and I don’t think the company ever made level 3. I heard that internally CMM was recognised as a mistake and the work was quietly allowed to die. The company also announced major moves to offshore development.

Fast forward to 2009. I recently met someone who was an outside consultant to the company and advised them on CMM adoption. He told me that he, and the other consultants, had advised the company to build on the practices they had and improve them. They specifically advised them not to write new processes and impose them on people.

He also told me that when the company did its own retrospective on the CMM project it was seen as a mistake to have written and imposed new processes on developers. A far better way would have been to improve what was in place with the people who did the work.

So the moral of the story:
• There is a lot of good in CMM – and the more recent CMMI.
• CMM itself is not a process, it is a meter, it is a means of measuring yourself.
• The key is process improvement, focus on improving and the levels will come.
• Don’t replace, improve, evolve. Involve your own people.

I recently gave some coaching to a big, well known, UK company get an Agile team started. Its interesting, this team have a few novelties all of their own, I’ll blog about that some other time. But they also have all the same questions and doubts that other teams have.

So here is a kind of Agile FAQ, the questions this team asked which all teams seem to ask sooner or later.

Q: What do we do about work items that are too big to fit in one iteration?
A: We will break the work down. We will break business requests into developer tasks. We will implement the minimally marketable feature the business agree to. We will talk to the business customer about what is required, we will implement the smallest thing that will work. It might be new, it might take some time to get the hang of it but it is doable.

Q: Should we have longer iterations? (Because our work items are so large)
A: No, if you have doubts you should have shorter iterations so that a) you see problems sooner, b) you get more practice at breaking work down.

Q: How will I know what to do if I don’t write a technical design?
A: Design is good but do not do more than is needed. Writing a document can be a useful learning exercise but there are other, and faster, ways to do similar things. Technical designs are best done at the white board with a colleague. If you need to document technical knowledge then document it after you have written the code/done the configuration.

There is no place for architects and software designers who are removed from the team. Design and architecture will evolve over time so needs guiding. This means experienced designers need to be involved (preferably coding) with the team daily.

Q: How will the business know what we are doing if we don’t write a functional design?
A: The business will be working with the team in planning meetings and will be seen the work as it is delivered.

Q: How do we manage dependencies between tasks?
A: Identify them and order the tasks accordingly. If need be break them down further.

Q: What if the business wants us to do A and B but not C?
A: Then do A and B but not C. The business pays the bills. If you need to do C for technical reasons – such as system quality – then do it. Don’t do C if its a feature the developers thing is worth having but the business don’t prioritise.

Q: What if the business wants us to do A then B but it would be quicker to do B then A?
A: Explain that to the business but respect their final decision. If they want B sooner then do it first.

Q: What if there is something technical which the business don’t understand and we need to do? The business won’t let us!
A: Explain it in terms the business will understand – this might require some learning and practise.

Q: Shouldn’t we have a detailed plan for future Sprints?
A: There is no point; requirements will change, your knowledge and understanding will increase and you will be better at estimating later. Only plan the next sprint in detail, you can sketch the work beyond that but don’t spend a lot of time on it.

You might sketch out what you expect to do in future sprints in “Release Roadmap|” but remember this will change as you go. Keep the roadmap under review.

Q: Should I include testing a documentation in my estimates? Won’t that make the estimates too large?
A: Yes. If that is work that needs doing it should be included. Not to include it would start to build an additional, hidden, backlog of work. Equally the customer should see the real price of the work, not the price you want them to see with more to come. And if the project manager gets a shock then so be it.

Q: Doesn’t Agile invite scope creep?
A: Yes but in general I find scope reduces on Agile projects. I call it running Scope Creep Backward.

The online Technology Management journal has published an article by me entitled “A new governance model for agile work.”

In the article I argue that Agile based software development needs a new governance model. The old model, based on defined requirements is not applicable – simply because the requirements evolve when working in an Agile fashion. Instead a governance model based on the venture capital investment cycle is more appropriate.

I used to regular review books on this blog, somewhere along the line I stopped doing it – too many other things to blog about I guess. Plus, while I still read a lot I find it hard to find books which say anything news.

And, I increasingly find I’m reading books before they are published. My friends Rachel Davies and Liz Sedley have been beavering away at their book on Agile Coaching for over a year now. I’ve been reviewing the drafts as they go and I’ve seen it change and improve over time.

The book, Agile Coaching is now available as a Beta download from the Pragmatic Programmers press and the print version will be out in August.

For me the book is a book of two halves. As you would expect half of it is about coaching Agile teams. In that you’ll find that it covers similar ground to my own Changing Software Development – both are about introducing change. While I cover more management and background they cover more personal stuff.

The other half of the book is a nice, modern, discussion of how Agile teams work. Its not Scrum, Kanban, XP or any other method. It describes what you find. Its one of the best introductions to current Agile practices you’ll find. Plus, it incorporates a lot of experience which earlier books couldn’t.

Anyway, the waiting is over see for yourself.

Interesting piece in last weeks Economist which suggests the Hawthone effect doesn’t stand up to analysis. (For good measure the Financial Times covered the same story over the weekend.)

For those who don’t know, the Hawthorne effect is named after the Chicago telephone factory were it was originally observed. The researchers suggested that the act of experimenting on people changed their behaviour.

Back in 1924 researchers changed the lighting at the Hawthone factory and observed productivity increased. Then they reduced lighting and to their surprised it increased again. The conclusion of the study is widely cited to show that when workers know they are experimented on their performance increases. Another spin on the research suggests it is when workers feel valued their performance increases.

Now it seems the original data has been re-analysed using modern econometrics techniques. What it actually shows is the workers were more productive at the start of the week (lighting changes happened on Sunday.)

(Have a look at the Economist piece or the research itself to understand the subtleties I’ve glossd over.)

Anyone who saw another Economist report from last year knows already that most published research is wrong. That study showed that most of the research published in the top journals is disproved within three years.

Which raises the question of whether less well known and influence journals are actually more reliable but lets leave that to one side for now.

Perhaps the only surprise about the Hawthone effect is that it took over 80 years to be exposed.

And what lesson are we to draw from what we now know?

Well, maybe, if workers are more productive at the start of the week then keep that first two or three days clear for core work. Push meetings such to the end of the week.

I can never decide if I actually like foie gras as a food. The taste is always secondary to the thought of the force feed ducks. This production process has led to attempts to ban foie gras – as in Chicago a few years ago.

Personally I’d like to end a practice I’ve seen too many times in the software industry and which I’ve come to call foie gras recruitment.

Foie gras recruitment is the forced expansion of a development team beyond the ability of the team to absorb new people. It happens when an organization decides it needs more software produced and decides to do so by hiring lots of new developers.

For example, an internet company I worked with a few years ago. The company faced a rapidly expanding market and needed more from the development team. I was asked help them improve their development processes and practices (“get agile”). When I turned up on day-1 I found four new developers were starting that day and another the following week.

This was on top of the two new developers who had been hired two months previously. And as if that wasn’t enough the company was opening a US development office and had three new recruits starting there the following week.

Given that the original development team was only four strong that is more that is more than a 100% increase. Unfortunately it was too late for me to stop any of this, I just had to pick up the pieces.

This was not the first time I had seen this problem and I’m sure it won’t be the last. The reason why companies do this is because they feel the need to get more done. However the immediate effect of stuffing the development group with new people is to slow everything down. The more you force feed the group the greater the slow down. Hence foie gras recruitment.

Most developers know Brooks Law: “Adding developers to a late project will make it later.” This can be generalised as “Adding developers slows work down.”

This is because new developer need time to learn their way around the system, the tools, the process and practices. When they don’t know they have to ask. People need to tell them things. Those people are the current developers. So they get less done. The more people you hire the slower things go.

(A related problem is hiring just developers rather than a balance of developers, testers and BAs/Product Owners but we’ll leave that for another day.)

In my experience a new developer needs at least three months to become productive. In the book Organization Patterns of Software Development Jim Coplien and Neil Harrison suggest the figure is closer to one year.

I’m sure someone will say: “But if the business want that” or “If we really must deliver more we need more people” but the reality is: more means slower. It is one of those occasions were someone need to explain to “the business” that more isn’t what they think it is.

This gives rise to the latest Kelly’s Law of development: In the short term your development resources are fixed – they may only be reduced.

Of course in the long term you may need to increase your development team and you should hire new developers. The right way to do this a rolling programme of recruitment. Aim to hire, say, one new developer every six months.

The frequency with which you hire will depend on things like: your current team size (big teams can hire more frequently) and the structure, number and interdependency of the existing team(s).

In any company there will be some level of natural loss and you may well want to set you hiring level to allow for this.

In the short term, if you really need to improve your through put then you need to look inside the team at the process and practices and see what can be changed there. Even here it takes time to improve throughput.

So, good bye to foie gras recruitment, hello to a little and often.

Please forgive my slight departure from my usual posts on software development, Agile, patterns and so on. But I’ve had cause in the last month to think hard about what the internet means to us all.

I’m probably one of the last generation of Europeans to remember life without the internet. My first contact with e-mail was about 25 years ago when a friend and me looked after the school computers and got access to the first inter school e-mail system – The Times Network for Schools.

(Access didn’t last long once the school saw the phone bill. Just long enough for us to discover that none of the other local schools had not changed their default password.)

As my professional career has developed so has the internet. But I can remember time before the internet so I know the world could survive without it again. A different world yes but it would survive.

Last month the internet took on a dark side and my relationship with it changed. Death came to cyberspace.

I’ve been following the blog of that old school friend for a few years. Nothing that interesting but it felt like I was in contact with him. Then in late April his post read like this: “I am an alcoholic, I have been for nearly 10 years. At the weekend I attempted suicide. The police broke into my flat and saved me because someone saw my update on Facebook.” Presumably his Facebook updated read like “Dave is … attempting suicide with pills and beer.” Since then he’s been blogging every few days about his attempt to give up drink.

Think about that. A suicide attempt on Facebook. A confession of alcoholism in a blog. A public dairy of a recover attempt.

Sure you could say it was a very big plea for help. Publicity for himself yes, and maybe it was self indulgent but it also a sign of how we live out lives.

Then yesterday, a Facebook message from someone I didn’t know. I though it was spam and almost ignored it but I didn’t. It was that friend’s sister. She wanted to tell me he’d died.

I’ve long thought e-mail was the worst possible way to give people bad news but I was thinking of news like “Your project is late” this is an order of magnitude more.

Facebook was probably the only way she had of finding me quickly and I’m glad she did. But that changes Facebook for me. Until last month Facebook was about friends, pictures, smiles and so on. Its now changed.

I guess its like the telegrams we used to get: “Regret to inform you Bill Smith passed away STOP.”

And what is the protocol here? Do I go and delete him from my friends list? Or do I leave him there Zombie like? How long do I leave his blog on my reader?

Who tells Facebook to remove his account? Do they need to see a death certificate?
How long does Blogger leave his blog there before it is deleted by some robot?

Web-Archive will always have his posts and the websites he created.

Gradually he’ll be removed from friends lists, links to his blog will be dropped and the great garbage collector in the sky will remove his cyber-soul.

For now Dave still lives in cyberspace, he just hasn’t posted for a few days. Soon it will be a week, then a month, then a year. He’s already left more of a life history than most people who have ever lived, and its far more accessible. In 500 year historians will have a better understanding of everyday life because of the likes of him.

Jeff Sutherland was back in London last week and was good enough to give a talk to ACCU London. As one of the organisers of the ACCU London events I feel I’m justified in giving myself (and my co-organiser James Slaughter) a well deserved pat on the back. This was something of a coup for us.

Over 80 people registered for the event – in fact there was a waiting list of people we couldn’t fit in – and I counted at least 70 people in attendance.

As this event was somewhat bigger than our usual events we had to find a bigger room. Fortunately JP Morgan came to the rescue with a room at the top of their London tower. So not only did we have Jeff but we had a great view of London. Thanks JP Morgan!

Jeff is a good presenter and had some interesting things to say. Although I know most of the Scrum-stuff I still picked up some good information. A few points I thought writing down:

• When Scrum was introduced at Palm it was found that 1 hour of postponed testing resulted in 23 hours of testing later.
• The Chaos reports from the Standish group find that 76% of requirements change after a project has started.
• Data from Yahoo shows that teams adopting Scrum on their own show a 35% productivity improvement. Teams which are coached through Scrum adoption shows a 300-400% improvement. Good news for those of us like me who make a living as an Agile Coach, usually with Scrum. (For those who want it the original source of this statistic is “Rolling Out Agile at a Large Enterprise” by Gabrielle Benefield at the Hawaii International Conference on Software System, 2008.)
• Not only does Jeff endorse the addition of XP technical practices (TDD, refactoring, pair-programming, etc.) to Scrum he believes these are what allow Scrum to scale up. This is important as it fits in with Bob Martin’s comments at the ACCU conference last month.

At the end of the session there was a very good Q&A session – so good we ran out of time and had to cut it short.

In response to one question Jeff mentioned a paper he wrote a couple of years ago called The Future of Scrum. What is interesting here is, Jeff foresees changes in the way Scrum is practices. This contradicts with Ken Schwaber’s statement that “There is only on Scrum.”

I don’t want to get into fighting about whether there is one Scrum, or a future Scrum or what ever. What I think is interesting here is that there are different views on the subject.

While the Agile community as a whole sometimes seems unable to agree on “what is Agile” the Scrum community seems steel like in their understanding of what Scrum is. The truth, for both groups, is somewhere between the two extremes.

I stumbled across a programme called English Heritage on BBC 2 last night (Friday 15 May). Enjoyable for three reasons: first it was an insight into the work of English Heritage, second it was about renovation of Kings Cross station but while both of these were interesting the what I found fascinating was; third – the work of architects.

That’s physical, building, architects. Not the software engineering type.

Like many others in the software engineering community I’ve been guilty in the past of throwing the architectural metaphor around without really understanding what building architects do. Sure I know a little but much of what I think they do is guesswork.

So this programme grabbed my interest because it showed real architects at work. Here are some points that stood out to me:
• Work had begun on the project and the architecture plans were far from complete. Design was continuing and evolving as work continued.
• There was a budget and work had to be within the budget. It looked like the budget (and I suspect time) was set and design and construction occurred within that budget. I saw no sign of a up front master plan which had been broken down item by item, then estimated for time and cost and the sum total agreed.
• In some cases demolition had occurred and it wasn’t clear what would be build in place.
• Building areas where being found and possible work suggested.
• Most of the work was on the old Kings Cross station, some was redesign, some was addition, some was removal and some was simple refactoring.
• We never saw the architects at their drawing boards; we did see them walking around the site, investigating ongoing work; arguing, discussing and negotiating with clients and authorities.
• We saw architects having to change their designs because of business constraints (not enough money to renovate one area) and regulating authorities (English Heritage restricting what the architects could do and how they did it).
• We saw one set of architects disagreeing with another set (Network Rail’s architects wanted to do things want way and English Heritage’s architects thought it should be done another way.)

All in all I found it a fascinating insight into what architects do. As I long suspected, the practice of architecting buildings is not what a lot of software people assume it to be.