allan kelly

A question in my mail box: “How does Agile relate to CMM Level 5?”

As I started to tap out the answer I thought: this might as well be a blog entry. So here it is.

Think of CMM, or rather CMMI which replaced CMM about 10 years ago, as a ruler. It is a way of measuring software development effectiveness. Is your development process 1cm good? 2cm good? or 5cm?

CMMI doesn’t say how you should do development, only how you should judge effectiveness. Agile is a way of doing things – as opposed to “the state of Agile” which is a measure of effectiveness.

You can use Agile to be any CMMI level you like. CMMI doesn’t care how you do it. Watts Humphrey (the father of CMM) used to say something along the lines of “Work on you quality and processes first, the levels will come by themselves.” That statement is very Agile. Unfortunately some organisations get it the wrong way around. I was at Reuters when they imposed CMM and destroyed a big chunk of their development capability.

The late Watts Humphrey did issue some process recommendations in the form of the Personal Software Process and Team Software Process.

That’s the easy bit.

“The state of being Agile” might, or might not, conflict with Level 5 CMMI simply because different things are considered important. You might be CMMI level 5 and decidedly unAgile.

Ironically CMMI level 5 might mean you have to investigate Agile. Because being level 5 means you are “self optimising”. If an organisation is level 5 and hasn’t looked at Agile it should because that may help them improve.

Paradoxically, being level 5 itself makes it becomes harder to improve. The risk of change is a lot greater because the organisation has more to loWaterfallse – and probably lots of procedures to update making any change expensive.

CMM(I) tends to be associated with a certain way of doing things. Partly this is historic: when CMM(I) appeared Waterfall was the dominant model, NASA was the first organization to reach level 5 and (at the time) it was very Waterfall based, and because CMM(I) tends to be more common in military work which are also big and paper work intensive.

So, some people believe that CMM(I) means following a very structured, heavy, WaterfallWaterfall process. It doesn’t have to mean this but historically the two do tend to coincide.

The Software Engineering Institute issued a report in 2008 which discussed how CMMI and Agile could work together: CMMI or Agile: Why not embrace both! I don’t agree with everything in the report but I do agree with the general tone. CMMI and Agile are not alternatives and can be complementary.

I have a piece on the InfoQ website today entitled Agile Contracts (well options for really).

A couple of footnotes to my last blog post on Capers Jones book, Applied Software Measurement. One of the points I noted was Jones suggestion that rising prices and costs in India and elsewhere would mean IT offshoring loose its financial benefits from about 2015 onwards.

Jones, by the way, later says that Indian and Chinese outsources are aiming to compete on quality not price in the long run. So we can expect to see the nature of offshoring change.

Shortly after posting that blog entry I noticed a piece in The Economist on the Indian ID programme, Identifying a billion Indians (27 January, subscription required.) The Indian Government is embarked on a scheme to give unique identity numbers to the entire population.

The first thing that first got my attention was not that this was an IT based scheme (no surprise there) but who was doing it. Not Tata (TCS), not Wipro, not Infosys. It is Accenture and L-1 of the US and Morpho of France. I’m sure much of the work is being done in India but the Indian Government has given the work to foreign firms. Lets give a round of applause to the Indian Government for not feeling compelled to give work to local firms.

And then lets note that offshoring/outsourcing goes both ways. Its not all about US and Europe loosing out to India or China. It goes the other way too.

The second thing that got my interest was this: the work is split between these three firms: 50%, 30%, 20%. Progress is reviewed regularly and work reallocated. The most effective firm during any one period gets the bulk of the work next time. That is a very enlightened way of working and gives real incentives the firms to do their best. It also shows flexibility with contracts.

Separately, some US readers might be please to learn that Financial Times reports that the big Indian outsources plan to do less work with US firms. European readers won’t be so happy to learn that Europe will replace the US as their target.

The reason: not competition, not prices or quality. US Visa prices.

I continue my reading of Capers Jones Applied Software Measurement as discussed a few entires ago – Software Facts and I’d like to report some more of Jones findings.

These numbers are very insightful but, and its a but Jones acknowledges, the data is very shaky. As Jones says “software measurement resembles an archeological dig. One shifts and examines large heaps of rubble, and from time to time finds a significant artifact.”

He readily admits that there is not really enough data to make solid conclusions (less than 1% of the data needed is available) but this is where we are at. This is as good as it gets. Before anyone rushes to say “Capers Jones is wrong” ask yourself: “Do I have better data to make conclusions one?”

I find two weaknesses in Jones work. Firstly his reliance on Function Points. I agree with him that lines of code is not a good measure but I have some doubts about function points for two reasons.

• They do not incorporate algorithm complexity
• Automating function point counting is difficult and becomes an approximation. Thus true function point counts are expensive which makes them difficult to work to calculate often

But, function points, for all their faults seem to work.

My second issue with Jones work concerns his assumption of the waterfall model. Yes he acknowledges Agile, he even says it is the most productive approach in some circumstances but all his data and assumptions are cut through with the waterfall. I suppose this is only natural, it was (is) the dominant model in the industry for 30 years or more.

OK, on with some numbers and conclusions, again these are almost entirely US based but are probably very similar elsewhere….

Productivity

• Jones repeatedly states and shows how quality and productivity are related. The most productive teams have the lowest bug counts and shortest schedules.
• The three biggest costs on a software project, in order: Rework (Bug fixing), paperwork and meetings & communication. Sometimes managerial costs are actually greater than coding costs.
• Jones states as little know fact “excessive staffing may slow things down rather than speed things up”
• Inspections (code, design, requirements) are the most efficient known ways of preventing problems.
• Studies show that some developers are 20 times more productivity than others, and some make 10 times as many errors as others.

Costs

• Paperwork (requirements, technical specs, etc.) on the largest systems can be larger than one person’s ability to read during an entire career. (Think about that: you join a new project at 21, by the time you hit retirement you haven’t finished reading the documentation.)
• Most corporate effort (time and money) tracking systems are incorrect and miss between 30% and 70% of the real effort.
• Thus this data is essentially useless for benchmarking, creates cost over runs because they are so inaccurate and is actually dangerous because it puts future projects at risk when the data is used for estimates. (Before you question this statement go and check, does your tracking system measure unpaid overtime?)

Projects

• “More than half of large projects are scheduled irrationally” – a delivery date is set external to the development group without reference to capabilities.
• Over 50% of large (more than 10,000 function points) projects are cancelled and the rest are almost always late and over budget.
• In 2008 at least 25% of new projects used some elements of Agile.
• Maintenance is now the dominant form of software development and is likely to stay that way.
• There are several Y2K like problems likely to occur in the next 40 years.
• Even waterfall projects overlap phases – typically 25% of any phase is incomplete when the next starts. This rises to 50% when the handover from design to coding is considered.
• It is even difficult to determine when a project really starts – less than 1% have certain start dates – and 15% have ambiguous end dates.

Outsourced & Offshore

• Outsourced software development companies are generally better than their clients and have a better record of delivering large systems.
• Offshore work tends to be less product than onshore work in the US – that does not mean it is not cost effective but it is less productive in terms of function points per man/time period
• Inflation and other cost changes mean that by 2015 the economic advantages of offshoring development work are likely to be gone. (Considering that it takes time to transfer work and offshore teams to become productive it is likely that very soon it will not make sense to send work overseas.)
• Worryingly Jones reports that 75% of companies have deficient requirements practices. Since offshore work is particularly dependent on good requirements I would expect offshore projects to be more troublesome in this regard.

Standards and Quality

• Jones finds no solid evidence that ISO 9000 improves quality or the ability to deliver; almost the opposite it does increase the cost of projects because it increases the amount of paperwork. Ironically some ISO 9000 advocates have defended the approach to him on the grounds that ISO 9000 is not designed to improve quality! (Sounds like many people have been misled over the years)
• Military systems are the largest systems in the world, some over 350,000 function points. They are also the most expensive and usually paperwork driven.

• Department of Defense (US) standards do not seem to improve software quality, they have sometimes been impediments and have certainly raised costs.
• Microsoft, Oracle and SAP are singled out as having very poor quality control.
• Interestingly he also suggests that once 25% of a large COTS application needs to be customised it is less expensive to build the same application from scratch.

Jones says that as of 2008 there were 700 known programming languages with one a month appearing. The average life of a language is 15 years but the average life of a major software system is 20 years. These figures might be a little misleading, C is now nearly 40 years old, other languages survive only a few. But the point is clear: there are lots of dead languages and orphaned languages out there (over 500 Jones thinks.)

As noted before, most schedules are planned with an externally determined date. Jones gives this as the number one reason for project failure – number two is excessive schedule pressure which is very similar. Later he proposed schedules are so bad as to constitute professional malpractice.

This is interesting. One the one hand businesses need those deadlines. On the other, if projects were left to run their natural duration would the failure rate be so high?

He also comments that US development teams tend to be strong on technical skills but are let down by weak management and organisational issues. Later in the book he attributes much of this down to western managements pre-occupation with cost control which he contrasts with eastern concern with quality.

This argument seems reasonable but I’m not sure it stands up to analysis. Although it might explain why Scrum can be “successful” even when technical practices are ignored.

In my experience US and European teams don’t always have the technical skills they should have – certainly talk to Steve Freeman or Jason Gorman and you will hear some horror stories. While India has more CMM certified organisations than anywhere else my personal experience is, while India has some very good developers the IT bubble has attracted in not just second rate developers but third rate ones as well. Its also worth pointing out that Japan isn’t a software powerhouse, Toyota even struggle to apply Lean thinking to software.

Jones believes, from his studies, that management is the greatest contributor to success and failure at the project and company level. In some ways that is reassuring, it shows that there might be something in good management.

Interestingly Jones also supports one observation I have frequently made myself: when redundancies occur Test and QA staff are usually the first to be let go, and this is counter productive in anything except the very short tun.

I’ll continue reading and maybe blog some more data. In general, the Applied Software Measurement has a tendency to repeat itself and I wish he used more graphs for his data but I still think it is worth reading and I recommend it.

My last blog post (Scrum has 3 Advantages Over XP) attracted a couple of interesting comments – certainly getting comments from Uncle Bob Martin and Tom Gilb shows people are listening. And it’s Tom’s comment I want to address.

But before I do, lest this come across as Scrum bashing I should say that I think Scrum has done a lot of good in the software industry. Yes I had my tongue-in-my-cheek when wrote “Scrum has 3 advantages over XP” but if Scrum hadn’t made “Agile” acceptable to management it is unlikely that XP ever would have take Agile this far.

I suppose I’m a little like Uncle Bob, its a bit of a surprise to me that Scrum Certification has come to mean so much to the industry when really it means so little. I’m also worried that brands poor people “certified”, good people “uncertified” and stifles innovation.

Second the ideas of the Scrum originators are good. And lets not forget that while Schwaber and Sutherland have become celebrities in the community there were five authors of the original “Scrum: A Pattern Language for Hyperproductive Software Development”, Beedle also co-authors one of the early Scrum books, Devos still works as a Scrum Trainer, while Sharon seems to Scrum’s Fifth Beatle – where are they now?)

Getting back to the point. Tom Gilb commented: “Sutherland did say in a speech that Scrum works best with XP”. Chris Oldwood also says he’s heard Jeff say this.

In fact, I think I’ve heard Jeff Sutherland say something similar, heck, not similar, the same and reported it in this blog. And Jeff’s not alone, I’ve heard the same thing from other Scrum advocates, trainers, and cheer-leaders.

But, this view is not universal. There are Scrum trainers who don’t. Specifically there are those who train Scrum and specifically say XP practices like TDD and on-site customer should not be used. And as I sarcastically commented previously, part of the attraction of Scrum to management types is that it doesn’t delve into the messy world of coding.

Lets just note: there are multiple interpretation of what is, and is not Scrum. Nothing wrong with that, it just complicates matters.

Back to XP though….

There are, broadly speaking, two parts to Extreme Programming (XP). The technical practices (Test Driven Development, Continuous Integration, Refactoring, Pair Programming, Simple Design, etc.) and the process management side (Iteration, Boards, Planning Meetings, etc.)

It is these technical practices that Jeff, and others, are saying Scrum software development teams should use. Fine, I agree.

Look at the other side of XP, the process side. This is mostly the same as Scrum. Do a little translation, things like Iteration to Sprint and it is the same.

(OK there are a few small differences. I just checked my copy of Extreme Programming (old testament) and to my surprise stand-up meetings aren’t there. I think many XP teams do stand-up but it isn’t originally part. The book does include “40 Hour Work week” which isn’t present in The Scrum Pattern Language, or “Agile Project Management with SCRUM” but I’ve heard Scrum advocates talk about sustainable pace. I’m still not sure how you marry this with commitment but I’ve written about that before. However these points are knit picking.)

Essentially Project Management XP style is Scrum.

This shouldn’t be a surprise. In the mid-90’s when Scrum and XP were being formed and codified many of these people knew one another. If nothing else they saw each other’s papers at PLoP conferences. The Scrum Pattern language was presented at PLoP 1998, Episodes (the basis for XP) was at PLoP 1995, and both draw on the work of Jim Coplien.

I have every reason to believe that XP and Scrum didn’t appear in insolation. There was cross-fertilisation. I have also been told there was competition.

If you accept that Scrum and XP project management are close enough to be the same thing, and you accept Jeff Sutherland’s recommendation (namely do XP technical practices with Scrum) then what do you have?

• XP = Technical Practices + Scrum Style Project Management
• Jeff Sutherland says effective development teams = Scrum + Technical Practices from XP

Think about that. Even talk about it, but don’t say it too loudly. For all the reasons I outlined before, XP isn’t acceptable in the corporation. Scrum is. Therefore, we must all keep this little secret to ourselves.

For corporations Scrum offers three advantages over Extreme Programming:

1) It offers certification – corporates can hire people who are “certified” and have their own people certified.
2) Scrum traces its roots back to the Harvard Business Review – so it must be serious
3) Scrum does not contain the words “extreme” or “programming”. No need to dirty our hands with messy code, keep that stuff as far away as possible (the further the cheaper, right?)

On the other hand, XP has two advantages over Scrum:

1) It actively seeks to address the quality problems most software development teams suffer from and which cripple productivity
2) It excites the people who actually do the work – dispensing with “resistance to change” in one fell swoop

At Agile Cambridge last year I saw a really good presentation from people at GE Energy about their Agile adoption. Frankly I’m sceptical about the ability of any large organisation to adopt Agile but these guys had some real success to show.

Two things stuck in my mind. When summing up the presenter said:

“If I was doing the same thing again I would start with the XP technical practices not the Scrum process” and he went on “We have adopted Scrum, we are now advancing to XP.”

XP and Scrum date from about the same time (mid-90’s). Perhaps because XP become popular first and Scrum later succeeded it as the Agile-poster-child many people seem to think Scrum is superior, or at least a superset of XP. Actually, the reverse is true.

XP is a superset of Scrum, and, in my humble opinion, superior.

About a year ago I needed some numbers about software development – industry norms really: effectiveness, productivity, bug counts etc. etc. Its actually pretty hard to get these numbers and after hunting around I found myself with a copy of Capers Jones Applied Software Measurement.

Jones, for those who don’t know, has made a career out of analysing software and software teams numbers. Applied Software Measurement is his latest work on the subject – although its nearly three years out of date.

Unfortunately, for me, Jones didn’t have the numbers I wanted – I forget what I wanted now. But he does have lots of interesting facts and numbers. You can dispute some of these numbers, you can argue with him but on the whole he knows more about this subject than you or me so he’s probably right. That said, there are a few points where I disagree with him and I might blog about these in future.

For now, I’d like to share some of the number in Applied Software Measurement. Overwhelmingly Jones researched American companies and American teams. I expect most of these numbers will be broadly the same in Europe and elsewhere. All figures are for 2008 unless specified.

Productivity

• Broadly speaking, in the USA, software productivity rates were the same in 2008 as they were in 1988
• The best US companies in terms of productivity and quality are about three times better than average (US)
• The worst US companies for productivity and quality are about 50% worse than average
• Only about 15% of companies are improving productivity and quality
• Similarly, about 15% of companies are getting worse in these terms
• About 70% of companies are neither improving or getting worse
• Where productivity is improving it is usually teams developing web applications who use Agile methods. Some benefits also accrue from use of Team Software Process and Personal Software Process
• Claims by tool vendors of 10-to-1 or 20-to-1 displacement of human activity are generally not justified. It is counter productivity to invest in tools before resolving organisational and methodology issues. i.e. save the tools until you’ve improved things (John Seddon should be happy with this finding.)
• Office environments can have as much of an impact on productivity as tools and methods
• Looking after employees well increases productivity and reduces turn-over
• Companies providing 10 day or more of training per employee per year have higher productivity rates than similar companies who do not
• Accounting errors (and use of unpaid overtime) can hide the true cost of software production by 100%, i.e. the work costs twice as much as the accountants say
• Formal design reviews and code inspections are effective but can fall into disuse because (new) managers do not understand this. (Tom Gilb is right!)
• “Software does not age gracefully, and tends to become increasingly unstable and difficult to modify safely. The Unites States and indeed much of the world are facing some huge geriatric problems with their inventories of ageing and decaying applications” – maintenance, renovation and enhancement are now the dominant activities in the industry and it is likely to stay that way
• On average military software is 25 times larger than end-user software, and the specifications and other documentation 200 times larger.
• Productivity and quality seem to be better in object oriented languages

Documentation & Bugs

• Producing paper documents for software development is more expensive than producing software itself
• Up to 400 words may be written in specification for every line of code in large systems.
• In a large software project requirements change and increase at the rate of 2% per month
• Repairing defects in software is more expensive than coding and producing documentation
• The finding and fixing software defects (bugs) is historically the most expensive activity and will account for 50% of total costs over the product life time
• Approximately 7% of changes and fixes contain new defects
• Most forms of testing finding fewer than 30% of all bugs

Projects

• Over 50% of all software projects are one-man projects
• Productivity and quality are directly coupled: projects with high quality have high productivity and vice versa
• IBM was the first company to discover the link between quality and productivity. They made the information public in 1975 but it is still not widely know.
• User satisfaction and defect counts are also directly correlated

I’ve mined these numbers out of Applied Software Measurement, I might mine some more and blog again. But really, it isn’t a book of numbers. Its a discussion of how to measure software and software teams.

Finally for now, I’m only at chapter 3, Jones says that currently available data on software production is less than 1% of what is needed. He also says that most databases of such data are privately held and not accessible by most of use. Well, I guess that explains why I couldn’t find the numbers I wanted when I wanted them.

Its nearly the end of the year, I’m about to close one Blog archive and start another – don’t worry, this means nothing for readers, its part of my admin. But it does mean that I put behind me all those half written blog posts, and single line ideas, that have been building up during the year.

Before I do so though, there are a couple of days to quickly get important backlogged entries out!

Two weeks I blogged, not for the first time, about Product Manager (Product Management an open secret, a differentiator). When I talk to people about Product Managers I often find myself in a conversation about marketing. Clued up people realise that good Product Management is, in part, a marketing function, others are surprised. They think marketing is about advertising and sales.

It is important, very important, to differentiate between the two sides of marketing: Inbound and outbound. So, for the record, I want to record it:

Outbound marketing: is the marketing most people think of when someone days “marketing.” This is about letting people know you, and your product, are here. Its advertising. Its public relations. It is generating sales leads. It is improving awareness. You get the picture?

It is called outbound because it is from you and your organisation to the wider world.

Inbound marketing: is from the wider world (customer, potential customer, competitors) to your company. It is about bringing that information inside and then acting on it. It is about building the products your customers want to buy. Get it right and sales should be like a knife through butter, people want your products.

In software companies, and other tech companies inbound is largely the role of Product Management. Product Marketing is about outbound. Although Product Manager is the usual job title some places call this role an Architect, others Programme Manager, and occasionally Project Manager. More often or not it is just absent.

Most companies realise, sooner or later, that outbound marketing is needed. Unfortunately, in all too many companies inbound marketing does not exist. Or inbound is “what the sales guy says the customer wants.”

As much as we love sales people (after all, they bring in the stuff that pays us) they are not the best people to decide what goes into a product. The reason why they are good at selling is that they get over customer objections and sell the product. This means the customer, the next customer, the next sale, is king. What the next customer wants isn’t what every customer wants.

Lets make this simple: if you are running a software product company and you don’t have a Product Manager then get one.

At the Skills Matter Agile Lean Kanban exchange the other week someone – sorry I missed you name – told me about a report from the BBC on Product Management. It turns out the report is from a branch of the BBC I didn’t know about, “BBC Academy” and it entitled “The State of Product Management 2010.” Its well worth reading if you have an interest in Product Management or the UK software development scene.

Although I’ve not blogged about it for a while Product Management is one of my passions.

Think of the development team as the beating heart of the work effort. Someone has to keep the blood flowing, someone has to keep the arteries clear and make sure the right requests for work can channelled to the development team.

This is a role. Someone needs to do this. Someone needs to understand customers and be in a position to prioritise. And someone needs to be able to have ongoing conversations with customers, developers and everyone else.

This is not a document, attempts to do this with a document makes things worse. Documents don’t have conversations. Documents are static.

One of the reason why Silicon Valley companies are so successful is that they understand this. In Silicon Valley there is a well developed role called the Product Manager. This is the person who looks at the customers, understands their problems and needs, looks at the market and a whole bunch of other stuff, and this person decides what is needed for a successful software product.

I point to Silicon Valley because certainly in Europe, and I suspect in much of the rest of the USA, this role isn’t so widely recognised.

This role doesn’t exist in corporate IT. Here the Product Manager’s cousin the Business Analyst fills a similar role but in a very different way. However, there is change afoot here, stay with me.

Here in the UK I find I have to explain this role to clients again and again. They make four common mistakes:

1. They assume the development team just know what is needed. Sometimes they do, sometimes they don’t. They may get luck once but can you rely on luck the second time? Third time?
2. They assume that the Project Manager knows. Again this is wrong, Project Manager training is about delivering a defined project to a date within constraints. It is not about understanding customers. (True there is overlap but Project Managers training is different.)
3. Experience IT Managers see the need and appoint a Business Analysts. This is the right thing to do when you are a corporate IT department, or if you are writing bespoke software for one customer but, if you are trying to create a product to sell to many customers it is the wrong choice.
4. Managers assume it is obvious, or that they, the “managers” know. If they are Product Managers or have been Product Manager they might be right. Even if they do know what the customers want someone has to explain it to developers and be on hand to answer their questions. Senior Managers just don’t have the time.

So its good to see the BBC highlighting this role. The description in this report is very media centric but it is recognisably the Silicon Valley Product Manager role. There are some good case studies in the report.

As I said, over in the corporate IT world this role doesn’t exist because customers are captive. One of the key aspects of the Product Manager role is understanding what customers (who have a choice) will pay for. In the corporate IT world this doesn’t exist: you get what you are given. So the BA role is similar but different. (There are other difference but they are for another day.)

(Interestingly several of the Product Managers quotes in the BBC report state they have Business Analysis backgrounds.)

The thing is: Business Analysts need to become more like Product Managers because more and more corporate IT departments are creating systems for external customers who do have a choice.

For example, 15 years ago just about all the IT created by travel companies was for their own use. Today travel companies have customer facing IT systems which can both win and loose the company sales. Online offerings are part of the buying experience, part of the customer experience and a potential product differentiator.

Even though the Product Manager role is not as well understood as it should be it is becoming more important. For those who get it Product Management is a potential differentiator.