Agile

An interesting prediction from Kevin Rutherford for 2010: “I predict that 2010 will be the year that eXtreme Programming returns to centre stage.”

I tend to agree with Kevin, while his logic is sound I would offer an alternative rationale for the return of XP. It goes like this…

XP can be viewed from two perspectives: project or engineering practices.

The first of these is basically Scrum. The cognoscenti might debate which came first and which borrowed from which but basically the approach is the same: short time-boxed iterations. Yes Scrum adds in Scrum Masters, Product Owners, self-organizing teams and such but while XP’s description is more basic they have broadly the same approach.

As to engineering practices, there is a growing Software Craftsmanship movement which is aiming to put these back on the agenda. While Scrum has been centre stage in the Agile world many of the engineering practices have been absent. Uncle Bob Martin has been talking about this for a couple of years and I’ve observed it myself (see my post about the Scrum Wall.)

Scrum gained a lot of popularity during 2008 and 2009 but too many of those teams adopted the project management bit without the engineering practices. While that approach can bring benefits it also stores up problems for the future. As more teams take this approach more teams will see these problems and more teams will need to add back the engineering practices and take craftsmanship seriously.

Add engineering practices to a Scrum team and it isn’t very different from XP.

The question Kevin leaves unanswered is: XP 1 or XP 2? For me its Extreme Programming, first edition, “the old testament.”

Now, to extend Kevin’s prediction. I hope we will see the return of “requirements” in 2010, or rather, more focus on “business value” delivery.

The “what are we building?” question has been underplayed in Agile too date. This means the Product Owner, Customer, Business Analysts and/or Product Manager roles (call it what you will) needs to have more attention. Fixing development is the easy bit, the difficult bit is doing the right thing. But, you can only really address that question once you can deliver.

Living and working in London means a lot of the software people I meet and talk to work in financial services. It amazes me the number of programmers etc. that banks employ. And you hear stories, some really awful stories about the state of IT in banks. And I’ve seen a few in my time.

So, for all those readers out there who have to work with banks and other financial services companies I recommend you get hold of a copy of this weeks Economist and read “Silo but deadly.” This article lends some weight to those stories I hear in pubs, at conferences and during training courses.

Some highlights:

• Financial services companies are estimated to spend $503bn on IT this year, thats more than Governments, more than manufacturing and more than any other sector
• The chief risk officer at Deutsche Bank says: “Banks are essentially technology firms”
• SOA architecture is a problem, that’s Spreadsheet Oriented Architecture
• Data quality is a recognised problem

The articles suggests that the woeful state of IT in banks may have contributed to the recent problems. This is something I’ve suggested other links before now, e.g. the Moody’s bug.

Personally, I think one of the reasons why banks and other financial companies have such poor IT goes back to that quote above: Banks are technology firms but they don’t view themselves that way. As a result the way they don’t necessarily take the right approach to people or projects, and they don’t take the right short-term v. long-term view and so on.

Lest we get too negative about our industry you might also want to read this recent survey from McKinsey which suggests IT isn’t doing such a bad job, and outside the IT department corporations are pretty happy with IT on the whole.

Twice this year I have visited companies which want to try a more Agile way of working but who have requirements coming of their ears. I’m talking hundreds of pages of requirements. Sometimes as use cases and scenarios. Sometimes as big meat documents written by expensive consultants and bearing names like “Requirements Specification” or “Business Design.”

Its hard to argue with such requirements documents, in part because its hard to read them and understand what they are about. The thing they have in common is a level of detail. They go into great detail on what needs doing and how you might go about doing it.

Its not the first time I’ve seen such documents. I’ve been on similar projects before. One variation is a project I saw last year were the documents were not requirements documents but strategy documents. Several big name consultancies had been into this (media) company and do strategic work around a new product. Lots of presentations, lots of SWOT analysis and so on.

What all these documents have in common, apart from being expensive, is that they are worse than useless. They give the impression that things are know but they aren’t know. They knowledge walked out the door, what you have is a snapshot of someone’s understanding. But its a snapshot that nobody can really take in because its so big. And it is a snapshot that is rapidly aging.

As I’ve said before: the bigger a document the less likely it is to be read, and if it is read, then the bigger it is the less that will be remembered of it.

To which we might add: the bigger the document is the more there is to go out of date.

Yet because these documents exist people feel compelled to use them. They may even be mandated to use them. They don’t feel they can say to their boss: this was a useful learning exercise, no we will do it for real. So we then embark on a process that try to use documents which only get in the way.

Having someone investigate requirements or strategy can be useful. But think of it as a learning and feasibility exercise. And above all: don’t let that person leave the project.

At best these documents are historical reference source; as worst they are an obstruction.

Set a clear goal for the work and work out what you need to do to meet it. As Tom Gilb says, this should fit on one page of A4.

As you work towards this goal you will uncover requirements. You might want to note them down but remember there is no point in running far ahead of the development team.

Once you have something up and running – even in demo – what people think they want will change. So start small, get something working and get a team that delivers.

Just say No to big requirements documents. Say Yes to short, concise, goal to aim at.

Continuing about patterns… At the beginning of the month I went out to Bulgaria to teach pattern writing to some academics and students at the University of Sofia. It was the first time I’d been to Bulgaria so I was interested to see the country which physically, historically and culturally sited between Europe and Russia.

We made the class very practical, no PowerPoint, in fact apart from a 5 minute video it was computer free. Yes we talked, yes we used flip charts but most of all it was practice, this was about writing after all. Only about an hour in and we had our participants putting pen to paper to write some patterns.

For this we used a video about milk packaging from TetraPak, there is an English version on You Tube, as it happens we used the short German version but it was the pictures not the words which were important. After watching this we did some brainstorming to identify what was happening in the video, i.e. what solutions were being used to solve which problems.

Initially we asked the participants to think about three things: the state before the solution, what the solution was and how things were different after the solution. Once the groups had grasped this we expended these into the traditional parts of a pattern: context, problem forces; solution and implementation; consequences and all the other bits (known uses, etc.)

During the writing myself and my co-presenters (EuroPLoP regulars Klaus Marquardt and Didi Schueltz) circulated between the groups to answer questions, provide advice and deliver a bit of shepherding.

And then we repeated the exercise with different subject material. We did some brainstorming to identify ideas which the participants would like to write patterns about, problems in their own domain. We repeated the writing process and finished the day with a writers workshop.

So in the space of one day we covered all the pattern basics. Although not in as much depth as maybe we could have. We carried on into a second day but only used half of the day. Next time I’ll be sure to use two whole days.

On the second day we covered the ideas behind shepherding, talked some more about pattern form, answered some questions and ran a second writers workshop, this time on an existing pattern by an experienced writer.

More than ever I left feeling that patterns are a generally applicable knowledge management technique. Not everything belongs as a pattern but the form is useful in many domains. And pattern thinking (or perhaps pattern analysis is a better term) is useful even when the thing you are writing (or just analyzing) is not a pattern.

And what did I learn? Well…

• I really enjoyed teaching pattern writing and hope to do it again
• Using a video to provide source material worked well
• Not using slides worked well, it requires confidence but the result was better
• You still need to give the attendees something to take away so having support material is necessary. Still, I’m always disappointed with slides which are little more than bullet points after the class. We did hand out some of my writing about patterns and patterns form, some patterns about patterns and some links to websites with more information and patterns.
• Having co-presenters was good. This was an intensive class so it needed more than one person
• You really need two days to do this properly
• Having attendees experience what you do was far more valuable than just lecturing them

Next time I would like to include a pattern reading exercise too. Take a well written pattern and read it through as a group. I’m going to be delivering a software patterns course this week and I plan to try this exercise in miniature.

In my last blog (Magic happens here – ERP, CRM, SAP, BPM) I mentioned change programmes in passing but didn’t elaborate. So I will now. This topic isn’t unique to the BPM work I have discussed in the last few blog entries but it is related.

When I think of “Change Programmes” I am thinking of the scenario where an organization decided something must change. Someone – consultants, managers, senior folk – decide what need sot change and how, e.g. introduce a new IT system, change the business processes, etc. – and then set about making it happen.

From time to time I run across articles online, in blogs and in prints journals and books which address the question like:

• how to you motivate people to change?
• how do you overcome resistance to change?
• why do so many change projects fail?
• why do people resist change?

And so on. The common theme is that a group of people, lets call then Workers, are required to change their behavior in response to some initiative from other people, lets call them Managers – although they often go by the name of Consultants.

This is a problem many IT projects of all shapes and sizes face at times. Its also a problem faced within IT when Managers want to change the way the team works.

I don’t believe people resist change. I think people, for Workers are always People, are happy to embrace change when they see the advantage, when they have choice and when they are involved actively in the change.

For example: many people are only too happy to get a new mobile phone when their supplier tells them they can have a free upgrade.

Change problems only exist when one group of people want to impose change on another group. Its not resistance to change, its it resistance to being changed. Been told what to do differently.

As long as one group of people think they know best and try to impose change on another group people will continue resist change, articles like I mention above will continue to be written and change initiatives will continue to fail.

The solution is to change the way you deal with the problem. Those who want change need to involve those who need to change. If you are a Manager who wants a team to improve their performance then ask team what they could do to improve. Ask the team what they would do differently. Take the teams ideas and help the team turn them into action.

If you are an IT expert charged with delivering a new IT system – to support business change or otherwise – then ask the people who will use the new system what it should do, give them demonstrations as the system developers, ask their opinion, deploy the system early and incorporate the feedback of the Workers.

I’m not saying you ignore long term issues, strategy, corporate objectives and such. Tell your Workers about these things and harness their ideas. Integrate the ideas and goals into a shared understanding.

This advice applies to all change initiatives and all IT change projects. (My “Agile approach to BPM” entry a few weeks ago put this in the context of Agile and BPM.)

Too often companies use IT as a blunt instrument to impose a change in process, behavior and working practices. I sometimes imagine it as a dice game. Faced with imagined resistance to change and complex systems Managers commission an IT projects in the hope that it will create change. They grab the dice – very expensive dice I should say – and roll them in the hope that they will roll a six and win.

Surely we can do better and move beyond luck for success.

I want to draw a line under the BPM/SAP mini-series of blogs. I am more aware than ever about the need to learn more in this area and actually I already know more thanks to people’s responses to these blogs – a few comments and a few e-mails. I hope to spend time in future finding out more about this arena and why it presents so many difficulties.

(And I’m hoping to see Jez Higgins talk at ACCU 2010, Wrestling with Giants.)

However, I continue to have this feeling that the SAP folks just don’t explain themselves. It feels like people are constantly telling me: “you don’t understand…”. What I would like is someone to explain please? Dont just tell me I don’t understand. Please, tell me: what is it that is different? What am I missing here?

Until that happens the BPM/SAP world will continue to feel like a secret society of people who “understand it” but will not explain it.. Until then all this is a mystery, SAP will be a box marked “magic happens here.” I’ve looked a some websites on SAP and I’ve looked at some books. The emphasis on results rather than how only deepens the mystery.

So, let me close by posing a hypothesis, maybe some researcher can pick this up and explore it.

I suggest companies buy big application suites one the assumption that:

Cost of Purchase + Cost of Customisation (programming) + Cost of Change Programme

is less than <

the Cost of Developing a Product from scratch

And that:

Cost of ongoing licencing + Cost of ongoing Maintenance

is less than <

Cost of Maintaining a Development Team to maintain a bespoke product

You might also add in a saving in management time, headaches from managing software developers, speed to market and a few other variables but that is the basic equation.

My belief – and I have no evidence for this so it is conjecture (don’t you just love blogging?) – is that companies underestimate the cost of installing and customising these big packages, and simultaneously overestimate the cost of developing their own solution.

That second part is probably the most controversial bit. I suggest that when run well these costs are not excessive and not run away. The current crop of development techniques (yes, the Agile and Lean ones) together with modern tools and existing libraries can help contain these costs.

A couple of cases might demonstrate my point….

As it happens, I do know of a company in London which has taken this route. They have four developer using PHP to continually develop their CRM system and other business functions.

A few years ago a company I worked with introduced SalesForce to help the sales team and started to extend SalesForce to marketing and support. At first it looked like a large part of my project would be thrown away as SalesForce replaced it. After many weeks of talking to the third party integrator it became clear there was something we needed and the system wouldn’t do. We developed the functionality ourselves for a fraction of the cost and time it would have taken to have the third party customise (programme) SalesForce. but then, it was a software company which knew how to do these things.

To finish off my BPM/BPE/SAP mini-series here are some other things I learned about SAP and BPM. These might be specific to the team I encountered or they might be generic. I think all of them present obstacles to doing SAP implementation Agile. Not impossible, just difficult.

Please, if I have any of this wrong correct me. I’d love to be wrong on most of these things. I’m also acknowledge my dataset is small so anything you can add to these notes would be great.

First a word on TLAs, it gets confusing:

• BPR – Business Process Re-Engineering; where it all began (sort of), comes from a book by Hammer and Champy and was very popular in the 1990’s. Largely discredited now, a blunt instrument used by many consultants which caused a lot of damage
• BPM – Business Process Management; and
• BPE – Business Process Engineering; I don’t know the difference between these two terms, they seem to be the same thing. They describe the thing that rose from the ashes of BPR. Still focused on business processes but removing some the neutron bomb like aspects
• BPM (again) – Business Process Modeling; the activity of examining your business processes, a step in applying BPM (Business Process Management), something you can buy tools for
• CRM – Customer relationship management; and
• ERP – Enterprise resource planning; two of the most common applications for which BPM, and SAP in particular, are used.

As you can see, it get confusing.

OK, things I learned about SAP:

• Actually understanding what SAP does, and does not do, is very difficult. Despite masses of literature very little of it contains hard data. Most of it is hand-waving generalist, marking speak or so focused on outcome that it is impossible to tell which bit SAP did all by itself, and which bit was a custom bit bolted on the side.
• Nobody is a developer, there are lots of designers, even the people who you might think did configuration and table updates are designers. They all expect to have someone below them to actually get their hands dirty.

• SAP culture seems decided un-Agile. There is an expectation that business processes are be defined up front, software design up front, etc. etc.
• At the end of the development everything is rolled out, i.e. change is imposed on others.

• SAP people are plug compatible, well so I was told. Apparently you can give the SAP HR person a week off, call up the consultancy they work for and have another SAP HR consultant for that week.
• SAP consultants have a lot of frequent flyer miles. Don’t expect them to attend a 9am Monday stand-up meting or hang around on Friday afternoons.
• SAP consultants tend to work for big name consultancies. These consultancies seem to have an attachment to waterfall development. Perhaps because it gives them some nice sign-off (i.e. billing) points. Strategy, requirements, design, implementation, testing, change management, ….
• SAP is all encompassing. It is a development environment, a programming system/langauage, a run time environment, a source code control system and many other things too.
• SAP “configrability” seems to be at a GUI level, cannot textualise the configuations. Thus very hard to automate tests, compare differences and version control.
• Similarly code reviews are difficult but pair-programming should work. But given the prices consultancies charge for their consultants I don’t see many takers.
• SAP source code control does not provide for branching or merging – and probably not importing or exporting.
• Some SAP configuration can only be done by hand. So no automated builds or unit tests. (SAP doesn’t even try to support them.).
• SAP has a programming language, ABAP. Its supposed to be a little like Cobol. There is an ABAP Unit test framework. It won’t do you much good because most SAP “programming” is done through configuration and there is no way to do unit testing on that configuration.
• (These and other aspects of the SAP tool set very old. This will make life difficult to Agile teams who may need to create new technology to address these issues.)
• Once you have finished your SAP application you pay SAP to “maintain” it. Or rather, pay them not to break it with a future SAP release.
• Nothing from SAP works out of the box. One of the first things you do in an SAP project is switch on the bits you want and switch off the bits you don’t. Some of these appear to be one time only decisions, you don’t get to change your mind.

Overall I got the impression that when you buy SAP you buy a legacy system. You start from nothing and you move immediately to legacy maintenance. That should be ideal but in buying SAP you buy lots and lots of functionality you don’t need and that imposes an horrendous maintenance burden.

I hope that doesn’t come across as SAP bashing. I’ve spoken to some people this year about other ERP and CRM type systems and it appears they are all largely the same. For example: there is no way to do unit testing in Microsoft Dynamics.

SAP present their product as a suit of unified tools. This seems to be pure marketing. It seems that the system has been developed by so many different companies, teams and at different times its a miracle that any of it works together. The Microsoft and Oracle equivalents seem to be in even more divergent as these have been assembled by buying the components in.

I also learned recently that a lot of these fancy “business process” applications are actually just a series of forms on screen. People fill in forms on the screen and it gets stored in a database. Or it triggers a dispatch. Or some such. Business Process Management is often just forms and workflow. Nowhere near as exciting at as “re-engineering” or the other fancy terms that get used in this space.

For the moment I think “Business Processes” is a very a grand title for what is basically: flow charts.

Whether it is exclusive locking on a source code control systems or flow charts, I kept feeling like I was back in the 1970’s, or maybe the 1980’s. Of course, I’m too young to remember those times in IT so I may be wrong. But this is a very different world from the internet world I’m use to.

In my research, I also learned that SAP have tried Agile, specifically Scrum, internally. However corporate anti-bodies killed most of the attempts. Some infection might remain. Having seen what I saw this year I think SAP corporate culture probably can’t stand the idea of Agile in reality.

Still, my experience of using Agile on SAP projects is limited. As you can see from the list above I think there are many obstacles. I think it can be done but I think you need to go back to basics, values and principles.

Continuing on the Business Process theme I want to think about what an Agile approach to BPM/BPR would be.

For me there are two scenarios: Business Process Management for new processes and the “re-engineering” version which looks improve existing processes.

In the first case the answer is: start small.
Don’t just look to develop a small piece of software;; look at the business you want to enter, look at the product or service you plan to offer. Find small pieces which can exist in their own right. Maybe you can start by offering it in a small geographic area. Maybe you can limit your offering to start off. Maybe you can shave off some of the frills to the offering.

Find the smallest possible product or service that you could enter the smallest possible market with. Build just enough (if any) technology to support that and enter the market. Then set about expanding your offering and you sales. As you do add the bare minimum of technology to do it.

Yes, its incremental growth. Yes, it is bootstrapping yourself to organic growth. That doesn’t look good on a business plan, doesn’t look good when you are GE and you need big new markets and big new products to make any difference to your balance sheet. Sorry about that.

It does mean the risk is less. It does mean you could fund a start-up to do it. It does mean you could make a lot of experiments to see which markets are attractive.

So it does mean you need to change the way you enter new businesses.

Second case: you have some existing processes at the moment.

Here you want to set in place a process of improvement. Not a big bank, sweep everything away but a million small changes that build to big change. Involve the people who are doing the work. Find out what they think would improve matters.

Perhaps you want to leave the IT to one side entirely to get get started with. See what changes and improvements you can make without any IT change. When you’ve made them use IT to cement the changes.

Again it doesn’t have the big bang mega-bucks appeal of the big project but its far more likely to lead to long term success. This is how you create and sustain a continuous improvement culture and a learning organization.

The traditional “Business process” mindset seems to view the business as a gaint state machine. It is the SAP teams job to help the people in that organization move through the states and transitions of that machine. This approach assumes “the big brains know best.” Which is exactly what has been discredited in the software development world but there is still an illusion that this can happen on the business side. Thats why we need an Agile approach to business processes even more than we need it in software development.

I keep thinking of Naked Objects. It seems to the Naked Objects approach could be a great approach to bootstrap a an Agile BPM programme.

I want to stay with the theme of Business Processes and Agile software development for a couple of entries. I’ll break it up into bite-sized chunks rather than one rambling entry. And they may come in quick succession.

Quite a lot has come out of my last entry (BPM – is it just programming the corporation? (And the SAP Irony)), comments on the entry and conversations I’ve had recently. I’m grateful to Hans for adding some more comments to the entry which I found very interesting.

Hans suggested that the way the term “agile” is used is different between the BPM/SAP type of people and the software development community. He suggests – if I understand him right – that Agile software people value the ability to change so “agile” means “can change (software) to respond to situations”; while BPM/SAP folk consider “agile” to mean “the technology works – and we can do stuff with it quickly – which free us up to think about other (more important) things.”

There is a subtle distinction here, I wouldn’t say it is two meanings of the word, I’d say it is two ways of looking at the problem. Two different aspects view, or opportunities to exploit.

The Agile software folk say: “look how quickly we can change this software, look how quickly you can get results, now think of the business opportunities that opens up.” In other words: being Agile in your software technology opens business opportunities itself.

To the BPM/SAP folk “agile software development” means: better.
So: software will be delivered sooner, with the features we actually asked for, within the budget we first thought of an maybe with less bugs. “With the hassle of software development out of the way we can use all the free time to think about other business opportunities.”

To me this is the difference in:

• Agile software development as a better way of developing software

and

• Agile software development as a way of creating new business opportunities

Ultimately its a question of perspective and what you want.

For me the first sees IT, and development in particular, as a cost and neglects the benefits. For some business that will be true. The ones that interest me are the ones that use technology to change the way they do business and gain advantage.

On and off I’ve run across Business Process Management – the descendent of Business Process Re-Engineering. Earlier this year I did a little work with a SAP team who wanted to be Agile, and a few weeks I was at the BPM conference in London.

What was noticeable at the BPM conference was both how dependent on IT it is, and how many of the techniques used by BPM practitioners look like IT techniques, specifically, the practices in software development.

Whenever I encounter BPM – or when I looked at BPR years ago – I get this feeling that it is an attempt to apply software development methodologies to the corporation. The BPM folks seem to regard the corporation as a machine to be programmed.

Processes are the sub-routines. These are programmed and linked together. Processes are re-usable – at least in theory. Under taking BPM seems to involve analysis, design and implementation (but no testing, strangely.) It sounds like Waterfall software development.

BPM folks want to create “the agile business.” They are talking the language of Agile and borrowing a few techniques. I even met a company at the BPM conference who have their own proprietary BPM process which looks, sounds and smells like Scrum.

Which leads me to ask: Is BPM just programming at the corporate level?

I think BPM folk would reject this suggestion simply because they view “programming” as dirty, cheap, low-end work for unwashed young people. What they do is expensive, strategic and practiced by ex-programmers (who would prefer not to talk about the past).

The other reason I think BPM people would rush to reject the idea of programming is because – as they are always keen to tell you – the software systems they use are not programmed. They may be configured or they may be customized. In the extreme you might write a customer module (or rather pay someone else to).

But, as all programmers know: programming is not just about code and its not just procedural. The configuration and customization are a form of programming. Some of the configuration tables in SAP look like table driven programming to me – not unlike a Turning machine, actually.

Now the BIG irony….

BPM folk are keen to talk about end-to-end activities and processes. Yet getting BPM tools – I’m thinking of SAP specifically – to do something means working with functional specialists who know their module and little else. An SAP Finance person would never think of working on SAP HR, an SAP HR person would never work on SAP Logistics.

A standard techniques in Agile teams is to have people work on difference parts of the system to do end-to-end. But this seems to be one thing SAP people can’t do. Ironic isn’t it, the people who advocate end-to-end processes can’t do it themselves.