Uncategorized

Almost exactly two years ago I wrote a strident blog entitled “Agile will never work in Investment banks”. Its a blog post that keeps getting rediscovered by those who agree or disagree with my position and just such event has occurred on Twitter with Kevin Burns @martinburnsuk and several others @kev_austin @sandromancuso @gordonmcmahon

During the intervening two years I have had many conversations which support my point of view. Most of these conversations have been with people who have more, and more recent, experience in banks than I do. In particular I found Peter Pilgrim’s blog closely aligned with my own views.

Conversations with the likes of Andy and Chris (you know who you are!) and others have led me to believe that while retail banking might be more hopeful it isn’t a massively better there. In part that is due to the colonisation of retail banking by investment banking that has occurred over the last twenty years.

Lets be clear: although investment banking and retail banking share the word “banking” in their title they have nothing else in common. I’ll leave it to John Kay to say something about the why they are in conflict. (Broken link).

The move by many investment banks to occupy retail space has been purely opportunistic, they have sought cheap funds and to embed themselves in the economic fabric so they become “too big to fail” and “systematically important institutions”, thereby guaranteeing Government bail out, thereby reducing their funding costs and increasing banking bonuses. (See The Bankers New Clothes for a discussion of the many problems with current banks and solutions.)

If you wanted to summarise my argument in one line I’d say: Too big to fail means too big to manage.

Now there are two caveats on my original post which I want to make clear before expanding on the discussion:

1. Agile is interpreted by many in banking to mean Iterative Development – see my recent “3 Styles of Agile” post. This can work to some degree. Simply adding the technical practices, improving software craftsmanship, can improve IT. However if an Agile ever becomes more than – incremental or evolutionary – there then conflicts will be created within the management regime which may lead to its own destruction.
2. Incremental and Evolutionary Agile can exist in banking, and have existed in many banks I know of but only in the short to medium term. This kind of Agile requires a guardian angel. Again this will create conflicts, while the guardian angel is strong enough the team can be protected. However given time either the angel is successful and will presented with a better opportunity (probably at another bank in which case he might have the team follow him) or he is not successful and will be pushed to one side and his creation will be dismantled.

Turning specifically to retail banks, and thereby extending my argument. On the one hand I don’t believe retail banks suffer with quite the same rent-seeking culture found in investment banks so I’m more hopeful of them. But on the other hand retail banks have an additional set of problems:

Legacy retails banks are dependent on Cobol mainframe systems. This brings several problems. Firstly while you can do some Agile like things in a Cobol environment there are many things you can’t do.

Specifically TDD: as far as I can tell several attempts have been made to create a CobolUnit (this CobolUnit may be the same to different) and they are all pretty much dead. No action since 2009 or 2010 on those links.

We can’t expect the OpenSource community to come to the rescue here. Few people write Cobol at home and the generation of people who write Cobol are different to the OpenSourcers who created JUnit, nUnit, etc.

Unit IBM or MicroFocus get interested in a CobolUnit then I don’t expect much to happen – it seems IBM might be moving in this direction. But generally I don’t see the incentive for either unless they can see money.

Second these Cobol systems are ageing. They have patch upon patch applied to them and by some accounts are reaching the end of their lives. Much of the technology they are built with just isn’t as flexible or modifiable as the current generation. There is only so much you can do with a hierarchical IMS database or a mainframe batch processing system. (If you could do these things, and you could do them as easily as modern technology then we wouldn’t have needed to invent modern technology.)

Since retail bankers were early adopters of computers some of these systems are among the oldest systems in use. Many of them have been patched together through mergers and acquisitions making things even more difficult.

True I’d a great proponent of never rewrite and I think it would be a mistake for banks to rewrite these systems. Indeed, given the chronically poor management skills in these banks authorising a rewrite would be a waste of money. Which means you are damned if you do, damned if you don’t.

Third, compounding the first two: the people who wrote these systems are nearing retirement age. Not only do they understand the technology but they understand the business too. When they are gone these systems are lost.

Banks could have alleviated this problem by training replacements but they don’t generally do this. Actually they have made things worse in the last 20 years by outsourcing and offshoring work thereby discarding their own experience and acerbating the generation change. Some of the knowledge might now reside in the outsourced and offshore but getting this into the bank will prove difficult and expensive.

The solution for the banks is to buy off-the-shelf systems. But this is incredibly difficult so…

Fourth, buying an off the shelf product means not only process changes but potentially product changes. That lands the retail bank with change problems. They will need to customise off the shelf software and that isn’t cheap either.

Fifth, new “challenger” banks like Metro bank are already buying off the shelf. As the costs of maintaining legacy systems or migrating to new systems escalate these banks will have a greater and greater advantage over the legacy banks.

And its not just challenger banks: Paypal, Zopa, Funding Circle and a host of other online payment and loans providers are applying technology to reinvent banking. In doing so they are eating the retail banks market. (And the banks can’t respond because of the reasons listed here.) In other words: the banks are suffering from disruptive innovation

Sixth – something I just hinted at – the quality of IT management in banks is absolutely appalling. Again investment banks have made retail banks worse because the big money was in investment, anyone who was any good and wanted more money had an incentive to move from the retail to the investment side. Retail IT has been robbed of its best staff.

Management quality is not entirely down to investment banks. Banks are about money management after all. Promotion to the senior positions comes from the banking side, not the IT side. Almost by definition the senior managers in banks don’t understand IT. It also means that IT managers sooner or later face a ceiling on progression.

Many of these problems have been made worse because investment bankers controlled retail banks. Investment banking is essentially a rent extracting activity, and they have managed retail IT just like that. The existing IT has been milked and without adequate investment. Hence problems like well the documented RBS outages and Lloyds payment problems.

To make matters worse banks are where the money is so they have been preyed on by major systems companies and tool vendors. These companies sell technology fixes which may – or may no – address a problem but do little if anything to build capacity. Put it this way: banks are major buyers of automated testing tools which aren’t used.

You see my argument?

I’m sure Agile could help with some of these problems – not all but some – but the way I see it the problems facing the banks mean Agile will have a very hard time.

Finally, as others have pointed out many of my arguments can be applied to other types of large corporations. I agree, I confine my arguments to banks because I think the case is clearer there, banks are all very similar, because banks exhibit more systematic failures than most and banks are something we should be worried about – as customers, as tax payers, as citizens in economies that depend on banks.

If I’m being honest, I feel as if I don’t know much about scaling agile. But when I think about it I think the issue is really: What do you mean when you say “scaling agile?”


It seems to me you might mean one of three things:

• How do you make agile work with a big team? Not just 7 people but 27 people, or 270 people
• How do you make agile work with multiple teams within an organisation? i.e. if you have one team working agile how do you make another 2, or 20 or another 200?
• How do you make a large organisation work agile? – it’s not enough to have a team, even a large team working agile if the governance and budgeting systems them work within are decidedly old-school

When I rephrase the question like that I think I do have some experience and something to say about each of these. Maybe I’ll elaborate.

But first an aside: why have I been thinking about this question?

Well David Anderson pushed out a blog about the Scales Agile Framework (SAFe) which prompted Schalk Cronje to ask what I thought – and at first I didn’t have anything to say! But then Schalk pointed out that Ken Schwaber has blogged about the SAFe too. It’s not often that David and Ken find themselves on the same side of the argument… well, actually… they probably do but too many people are willing to see Kanban and Scrum as sworn rivals. I digress, back to SAFe and scaling.

I still don’t have anything original to say about SAFe, I simply don’t know much about it. However the points David and Ken make would worry me too.

I’m not about to rush out and read SAFe. I find I’m more likely to be told by my clients: “I can see how agile works in big companies, but we are a small company, we can’t devote people like that.” And while I do have, and have had, some larger clients I think that statement says a lot.

I have over the years built up some rules-of-thumb, heuristics if you prefer a fancy term, for “scaling agile”. I’ve never set them down so completely before so here you go:

1. Inside every large work effort there is a small one struggling to get out: find it, work with it
2. Make agile work in the small before you attempt to make it work at scale; if you can’t get a team of 5 to work then you won’t find it any easier to get a team of 10 or 100 working. Get a small team working, learn from it, and grow it…
3. Use piecemeal growth wherever possible: start with something small and grow it
4. Don’t expect multiple teams to work the same way – one size does not fit all! A new project team might be perfectly suited to Scrum, a maintenance team to Kanban and a BAU+Project team to Xanpan. Forcing one process, one method, one approach one different teams is a sure way to fail.
5. Manage teams as atomic units, aim for team stability, minimise moves between teams
6. Split big teams into multiple small, independent, teams with their own dedicated work streams, product focuses and even code bases        
7. Trust in the teams, delegate as far down as you can; give teams as much independence as possible; staff teams with all the skills they need – vertical teams, include testers, requirements people and anyone else
8. Minimise dependencies between teams; decouple deliveries, decouple teams and again, vertical teams, staff the team so they don’t need to depend on other teams
9. Use technical practices to automate as much of the dependencies between teams as possible. e.g. a good TDD suit and frequent CI will by themselves allow two related teams to work much closer together
10. Overstaff in some roles to reduce dependencies – overstaffing will pay back in terms of reduced dependency management work        
11. Learn to see Supply and Demand (a future blog entry is in the works on this): supply is limited and is hard to increase, you need to work on the demand side too
12. Recognise Conway’s Law: work with it or set out to use it; again piecemeal growth and start as small as possible will help
13. Use Portfolio Management to assess teams, measure them by value added against cost. Put in place a governance structure that expects failure and use portfolio management to fail fast, fail cheap, fail often
14. Ultimately embrace Beyond Budgeting and change your financial practices
15. Big projects, big teams are high risk and likely to fail whatever you do; some things are too big to try. Some big projects just shouldn’t be done

    
There is no method, framework, tool out there that will be your silver bullet, but if you think for yourself, and if your people are allowed to think and act then you might just be able to create something for yourself.

Doing back to the three questions I opened with:

• How do you make agile work with a big team? When the team gets too big split it along product lines or product subsystems; if you can’t then don’t do anything that big
• How do you make agile work with multiple teams within an organisation? Use multiple independent teams, minimise dependencies, decouple and use technical practices
• How do you make a large organisation work agile? Portfolio management and beyond budgeting

And remember: Don’t do big projects, do small ones and grow success. If that is not an option for you then brace.

My last blog post (Scrum has 3 Advantages Over XP) attracted a couple of interesting comments – certainly getting comments from Uncle Bob Martin and Tom Gilb shows people are listening. And it’s Tom’s comment I want to address.

But before I do, lest this come across as Scrum bashing I should say that I think Scrum has done a lot of good in the software industry. Yes I had my tongue-in-my-cheek when wrote “Scrum has 3 advantages over XP” but if Scrum hadn’t made “Agile” acceptable to management it is unlikely that XP ever would have take Agile this far.

I suppose I’m a little like Uncle Bob, its a bit of a surprise to me that Scrum Certification has come to mean so much to the industry when really it means so little. I’m also worried that brands poor people “certified”, good people “uncertified” and stifles innovation.

Second the ideas of the Scrum originators are good. And lets not forget that while Schwaber and Sutherland have become celebrities in the community there were five authors of the original “Scrum: A Pattern Language for Hyperproductive Software Development”, Beedle also co-authors one of the early Scrum books, Devos still works as a Scrum Trainer, while Sharon seems to Scrum’s Fifth Beatle – where are they now?)

Getting back to the point. Tom Gilb commented: “Sutherland did say in a speech that Scrum works best with XP”. Chris Oldwood also says he’s heard Jeff say this.

In fact, I think I’ve heard Jeff Sutherland say something similar, heck, not similar, the same and reported it in this blog. And Jeff’s not alone, I’ve heard the same thing from other Scrum advocates, trainers, and cheer-leaders.

But, this view is not universal. There are Scrum trainers who don’t. Specifically there are those who train Scrum and specifically say XP practices like TDD and on-site customer should not be used. And as I sarcastically commented previously, part of the attraction of Scrum to management types is that it doesn’t delve into the messy world of coding.

Lets just note: there are multiple interpretation of what is, and is not Scrum. Nothing wrong with that, it just complicates matters.

Back to XP though….

There are, broadly speaking, two parts to Extreme Programming (XP). The technical practices (Test Driven Development, Continuous Integration, Refactoring, Pair Programming, Simple Design, etc.) and the process management side (Iteration, Boards, Planning Meetings, etc.)

It is these technical practices that Jeff, and others, are saying Scrum software development teams should use. Fine, I agree.

Look at the other side of XP, the process side. This is mostly the same as Scrum. Do a little translation, things like Iteration to Sprint and it is the same.

(OK there are a few small differences. I just checked my copy of Extreme Programming (old testament) and to my surprise stand-up meetings aren’t there. I think many XP teams do stand-up but it isn’t originally part. The book does include “40 Hour Work week” which isn’t present in The Scrum Pattern Language, or “Agile Project Management with SCRUM” but I’ve heard Scrum advocates talk about sustainable pace. I’m still not sure how you marry this with commitment but I’ve written about that before. However these points are knit picking.)

Essentially Project Management XP style is Scrum.

This shouldn’t be a surprise. In the mid-90’s when Scrum and XP were being formed and codified many of these people knew one another. If nothing else they saw each other’s papers at PLoP conferences. The Scrum Pattern language was presented at PLoP 1998, Episodes (the basis for XP) was at PLoP 1995, and both draw on the work of Jim Coplien.

I have every reason to believe that XP and Scrum didn’t appear in insolation. There was cross-fertilisation. I have also been told there was competition.

If you accept that Scrum and XP project management are close enough to be the same thing, and you accept Jeff Sutherland’s recommendation (namely do XP technical practices with Scrum) then what do you have?

• XP = Technical Practices + Scrum Style Project Management
• Jeff Sutherland says effective development teams = Scrum + Technical Practices from XP

Think about that. Even talk about it, but don’t say it too loudly. For all the reasons I outlined before, XP isn’t acceptable in the corporation. Scrum is. Therefore, we must all keep this little secret to ourselves.

Having cleared up some preliminaries, i.e. What is architecture?, we are getting closer to the big question: what do architects do? But I’ll continuing to take this piecemeal. In this blog entry I’d like to dismiss too groups of people who carry the Architect title but are not Architects.

The first group are “Architects by Seniority.” Some years ago I held a post with the title “Senior Software Engineer.” At first I though this might mean I was “the senior software engineer” but quickly realised I was one of many “senior software engineers.” The company conferred this title on anyone who had more than a few, about five, years of experience working as a software engineer. Or as I used to joke “anyone over 30.”

Some Architects get their titles the same way. My guess is this is more common on the services side of the industry were engineers are sold by the hour to clients and Architects have a higher billing rate.

A few months ago I was told it was common in the Indian outsourcing industry to confer the title Architect on engineers with 3 years experience. This is one data point, I don’t know how common that really is. Anyone out their know?

Unfortunately, some of the people who are given the title Architect simply because they have been around a while let it go to their heads. Which brings us to the second group who are Architects in title but not in practice: “Divorced Architects” or, as I think Joel Spolsky christened them “Astronaut Architects.”

These are Architects who sit around thinking big thoughts about “the system” but aren’t connected with what is actually happening. Just because you have the title “Architect” does not give you the knowledge or right to tell people what to do without doing it yourself. As Jim Coplien and Neil Harrison put it “Architect also Implements.”

If you are lucky these architects are pretty much harmless, they cost the company money, the developers tip their flat-cap to them in the morning but ignore them when they do work. If your unlucky their crazy ideas result in a messed up system and their egos get in the way.

Years ago I worked on rail privatisation, the Railtrack A-Plan timetabling system to be exact. It was on Windows NT with Sybase (yes, thats how long ago it was) in C and C++ with a little Visual Basic. 120 people worked on the system at the peak, of which four were architects and about 12 were coders, OK, maybe 16 if you include the SQL and VB guys.

But the architects came from a mainframe Cobol background so they designed a batch processing system, set down constraints and ways of working which just didn’t make sense for a client-server system. The company had a ISO-9000 system in place with lots of management so the result of this architecture was a lots of problems. Once they got into the code the developer just did what they wanted, the architects would never know because a) they wouldn’t get their hands dirty with code and b) they didn’t really know C let alone C++.

The project wound down and went into maintenance mode so I left. A couple of years later I found myself back on a much reduced project to redevelop parts of the system. Now we had about five developers, one architect part-time and a couple of dozen people tops.

We mostly ignored the architect, he saw one system and we saw another. ISO-9000 was nominally in place but widely ignored. The process worked a lot better. Occasionally we wrote a formal document to keep the formal process and architect happy but the real documentation was contained in “Rough Guides” which didn’t formally exist.

Moral of the story: Just because you are called an architect, just because you go to meetings, you aren’t an architect.

For those who don’t know, Gartner group are a business/technology research outfit who produce reports on things technical and business related. They have a pretty good reputation and they have a couple of sidelines in consulting and event organizing (or is it the other way round?). Sometimes they are leading opinion and sometimes following. They sell their reports, their conferences are expensive and their consulting must be more so.

Anyway, the reason for saying this is that Gartner have decided “Pattern Based Strategy(tm)” is the next big thing: “Companies Must Implement a Pattern-Based Strategy to Increase Their Competitive Advantage.”

(Notice the “(tm)”. Somebody, somewhere owns the trademark “Pattern Based Strategy.” Hillside own the trademark “PLoP” but not “pattern”. Could the owner of this one be…?)

Now as many of my regular readers will know, I’ve been talking about Business Strategy Patterns for over five years – and you can read my Business Strategy Pattern papers for free. Since posting up the 2009 EuroPLoP paper I’ve spent some time on the whole, I know have a 230 page draft of a book and I’m starting to approach publishers. There is still a lot to do but I hope to have something out by the end of 2010.

Getting back to Gartner. I’d love to tell you how their patterns and mine link up but I can’t actually read the Gartner report. Call me penny-pinching if you will but $500 seems pretty tall. Actually, they have a bunch more papers on strategy patterns but again, its $500 a throw.

I asked Gartner for a copy of their research and they said: “Unfortunately, our research is only available to our clients, but I can provide you with the press releases we have done on the topic which you may find helpful.”

So there you go, a closed shop. They aren’t willing for other people to review their work, definately not the culture we in the patterns community have.

It also means I can’t see their sources. I’m naturally suspicious of organization that won’t disclose their sources. For all I know they could be be building on sand – they could even be referencing me left right and center. Never mind me, I’d like to see some third party research to support their ideas.

From the press release (the first link I gave) and some other downloads from the site it doesn’t look like they are talking about Patterns in the same sense that anyone familiar with Christopher Alexander’s work would understand it. And by extension, they are not linking their patterns work up with work from the software engineering community – Design Patterns (GoF, POSA, etc.). Neither are the talking about patterns as knowledge management from the likes of Mary Lynn-Manns and Daniel Mays, or the work on Business Strategy Patterns I and others have been doing.

At the basic level there are similarities about the patterns they see and the underlying ideas of Alexander. For both Gartner and Alexander it is about a reoccurring sequence of events in some setting – Alexander would say place, software folks would say context and Gartner say market.

I think Gartners work is more related to data mining and business intelligence than it is to Alexander. As such it may well have more to do with bird spotting. Some years ago Harvard Business Review ran a piece on how spotting patterns in birds could help with business strategy – see Spotting Patterns on the Fly from HBR November 2002.

Although we are all dealing with the same thing: reoccurring events, sequences, constructions and the forces that bring them about, I consider Gartner’s patterns as “patterns in the small.” Reoccurrence without the repetition, analysis and explanation that make up “pattern in the large”. Its the same word but used differently.

Gartner also discuss “weak signals” and scenario planning. This is interesting because I’ve always believed pattern thinking has a lot in common with scenario planning. It is about detecting the forces at work and seeing how they play out.

Gartner are mixing a number of different themes into their “Pattern Based Strategy(tm)”: IT systems, event monitoring and data mining. They talk about finding “new patterns” but how can you find new patterns if you don’t know the old ones? It would be nice to think Garner have some patterns they might share but I suspect they don’t.

OK, lets give it 10 years and see if anyone remembers Gartner’s “Pattern Based Strategy ™.” Alexander and Design Patterns will still be around – timeless.

Finally, I would imagine that Gartner are one of those companies with a reputation management system so someone in Gartner will shortly be alerted to this blog. I’d like to hear your comments.

Anyone who bought this weekends Financial Times might enjoy my contributions to the letters page today. If you missed it, then I hope this brings a smile to your face. (The piece I was commenting on appeared last week.)

Another year, another great ACCU conference. I counted 9 languages on the conference program, everything from Scala to C++ – yes there was more C++ that anything else but it was far from the only thing going on. Languages plus sessions on design, pattern, management, Agile and neuroscience.

Normally some of the most insightful comments come not in the scheduled sessions but in the discussions over coffee and in the bar afterwards. Sometimes you don’t know these are insightful until after the event when you’ve left and had time to think it all through.

This will always be a special ACCU conference for me because I was the keynote speaker on Saturday. My keynote was well received and the slides ….

So here is a brain dump.

The opening keynote on Wednesday was from Bob Martin – “Uncle Bob” – about software craftsmanship. (I’m watching the software craftsmanship movement with interest and will write a blog about it in future.) Bob suggested that the most successful Agile method – Scrum – was the accidental “winner” of the Snowbird summit in 2001. This was because the Agile manifesto and Agile values said nothing about technical practices.

Scrum, as is well know, does not contain any technical practices. Teams often borrow these from XP. Because of this Bob believes Scrum is a subset of XP – a subset which just deals with project management. Without the technical practices productivity falls off and teams make a “big mess faster.”

Bob’s answer is a renewed emphasis on technical practices – hence software craftsmanship.

Bob is a great showman and his talk was very entraining, if you get the chance to see him do so. Along the way made several Scrum jokes and was quite critical of Scrum Master Certification.

Someone like Bob can do this, Bob has the reputation and standing to be able to take pot shots from a safe(ish) position. This is good because someone needs act as a foil.

Jutta Eckstein’s session on transparency in Agile was interesting. It reminded me of the session I ran here a few years ago called “Exposing problems / Creating awareness.” The these was similar: “Agile as a problem detector.” The session was largely a discussion between Jutta and the various people in the room.

Keith Braithwaite presented the latest instalment of his “Measuring the effects of TDD” research project. I’ve seen Keith talk about this subject before and over time his presentation is getting more and more compelling. His sample size in increasing and its looking like a more convincing case.

If you don’t know the work have a look at Keith’s blog. To summarise it: you can measure code complexity using a measure called cyclomatic complexity. Keith’s research shows there is a correlation between complexity and automated until tests. Namely: code with a high level of automated unit tests exhibits lower complexity metrics and people comment they are “happier” with the code.

Having discovered the metric Keith is still trying to understand the cause-and-effect relationship and, perhaps, more importantly, uncover how this relationship can be used to improve matters.

Nicolai Josuttis finished the day with a second keynote. You could call it the antidote to software craftsmanship: Crappy code.

Although Nico made it jovially it had a serious point: there is a lot of bad code out there and it is likely to be around for a while to come. The economics of software development and business tend to perpetuate this situation. Therefore, we need to get used to crappy code and find ways of dealing with it.

While I agree with Nico I can’t help feeling it was a little defeatist. If we accept this poor state of affairs do we still have hope of a better tomorrow?

Between them Bob and Nico certainly set a debate raging – Good code, or Crappy code?

Wednesday finished with 40 of us in Chutney’s curry house.