Agile

Its nearly the end of the year, I’m about to close one Blog archive and start another – don’t worry, this means nothing for readers, its part of my admin. But it does mean that I put behind me all those half written blog posts, and single line ideas, that have been building up during the year.

Before I do so though, there are a couple of days to quickly get important backlogged entries out!

Two weeks I blogged, not for the first time, about Product Manager (Product Management an open secret, a differentiator). When I talk to people about Product Managers I often find myself in a conversation about marketing. Clued up people realise that good Product Management is, in part, a marketing function, others are surprised. They think marketing is about advertising and sales.

It is important, very important, to differentiate between the two sides of marketing: Inbound and outbound. So, for the record, I want to record it:

Outbound marketing: is the marketing most people think of when someone days “marketing.” This is about letting people know you, and your product, are here. Its advertising. Its public relations. It is generating sales leads. It is improving awareness. You get the picture?

It is called outbound because it is from you and your organisation to the wider world.

Inbound marketing: is from the wider world (customer, potential customer, competitors) to your company. It is about bringing that information inside and then acting on it. It is about building the products your customers want to buy. Get it right and sales should be like a knife through butter, people want your products.

In software companies, and other tech companies inbound is largely the role of Product Management. Product Marketing is about outbound. Although Product Manager is the usual job title some places call this role an Architect, others Programme Manager, and occasionally Project Manager. More often or not it is just absent.

Most companies realise, sooner or later, that outbound marketing is needed. Unfortunately, in all too many companies inbound marketing does not exist. Or inbound is “what the sales guy says the customer wants.”

As much as we love sales people (after all, they bring in the stuff that pays us) they are not the best people to decide what goes into a product. The reason why they are good at selling is that they get over customer objections and sell the product. This means the customer, the next customer, the next sale, is king. What the next customer wants isn’t what every customer wants.

Lets make this simple: if you are running a software product company and you don’t have a Product Manager then get one.

At the Skills Matter Agile Lean Kanban exchange the other week someone – sorry I missed you name – told me about a report from the BBC on Product Management. It turns out the report is from a branch of the BBC I didn’t know about, “BBC Academy” and it entitled “The State of Product Management 2010.” Its well worth reading if you have an interest in Product Management or the UK software development scene.

Although I’ve not blogged about it for a while Product Management is one of my passions.

Think of the development team as the beating heart of the work effort. Someone has to keep the blood flowing, someone has to keep the arteries clear and make sure the right requests for work can channelled to the development team.

This is a role. Someone needs to do this. Someone needs to understand customers and be in a position to prioritise. And someone needs to be able to have ongoing conversations with customers, developers and everyone else.

This is not a document, attempts to do this with a document makes things worse. Documents don’t have conversations. Documents are static.

One of the reason why Silicon Valley companies are so successful is that they understand this. In Silicon Valley there is a well developed role called the Product Manager. This is the person who looks at the customers, understands their problems and needs, looks at the market and a whole bunch of other stuff, and this person decides what is needed for a successful software product.

I point to Silicon Valley because certainly in Europe, and I suspect in much of the rest of the USA, this role isn’t so widely recognised.

This role doesn’t exist in corporate IT. Here the Product Manager’s cousin the Business Analyst fills a similar role but in a very different way. However, there is change afoot here, stay with me.

Here in the UK I find I have to explain this role to clients again and again. They make four common mistakes:

1. They assume the development team just know what is needed. Sometimes they do, sometimes they don’t. They may get luck once but can you rely on luck the second time? Third time?
2. They assume that the Project Manager knows. Again this is wrong, Project Manager training is about delivering a defined project to a date within constraints. It is not about understanding customers. (True there is overlap but Project Managers training is different.)
3. Experience IT Managers see the need and appoint a Business Analysts. This is the right thing to do when you are a corporate IT department, or if you are writing bespoke software for one customer but, if you are trying to create a product to sell to many customers it is the wrong choice.
4. Managers assume it is obvious, or that they, the “managers” know. If they are Product Managers or have been Product Manager they might be right. Even if they do know what the customers want someone has to explain it to developers and be on hand to answer their questions. Senior Managers just don’t have the time.

So its good to see the BBC highlighting this role. The description in this report is very media centric but it is recognisably the Silicon Valley Product Manager role. There are some good case studies in the report.

As I said, over in the corporate IT world this role doesn’t exist because customers are captive. One of the key aspects of the Product Manager role is understanding what customers (who have a choice) will pay for. In the corporate IT world this doesn’t exist: you get what you are given. So the BA role is similar but different. (There are other difference but they are for another day.)

(Interestingly several of the Product Managers quotes in the BBC report state they have Business Analysis backgrounds.)

The thing is: Business Analysts need to become more like Product Managers because more and more corporate IT departments are creating systems for external customers who do have a choice.

For example, 15 years ago just about all the IT created by travel companies was for their own use. Today travel companies have customer facing IT systems which can both win and loose the company sales. Online offerings are part of the buying experience, part of the customer experience and a potential product differentiator.

Even though the Product Manager role is not as well understood as it should be it is becoming more important. For those who get it Product Management is a potential differentiator.

As noted in my last blog entry RQNG has published my description of Agile planning using Iteration plans, Release Plans and Roadmaps. I’ve added a little bit to this an produced a longer version of the same piece – it only adds about a page.

Three Plans for Agile (long version) is now available on my website.

RQNG has published one of my articles entitled “Three Plans for Agile”. This looks at the role of iteration plans, release plans and roadmaps, and how the three fit together.

I have to say thanks to Jon Harley, we had an e-mail discussion months ago which was the seed of this article.

After writing this I discovered (I’d like to say remembered but truly, I’d forgotten) that I actually blogged about this early last year as part of my Project Plans mini-series.

Rather famously Tolstoy started Anna Korenina with the words:

“Happy families are all alike; every unhappy family is unhappy in its own way.”

I like many others have repeated this quote, and even applied it to business and, in particular the act of creating software. I think, although I dread to check now, that I even open one of the chapters of Changing Software Development with the quote.

I dread because I have come to the conclusion that it it the wrong way around – at least for companies, and specifically the bits of corporations which produce software but which sell something else (i.e. Corporate IT departments) the quote should be:

“Unhappy companies are all alike; every happy company is happy in its own way.”

Blame (credit?) Jason Yip and his “Problems I know you have” blog post that sowed the seeds. But after a series of corporate clients in the last couple of years I’d like to offer my own list.

Here is a quick list of the mistakes I see big corporate IT departments making in the development of software. Actually, there are too many points here, there will have to be a part 2 to this post…

Too much Work in progress
Companies ask so much of their software development groups and these groups don’t say No. As a result they have far more projects in the pipe than they can ever hope to deliver in the supposed timeframe. Earlier this year I met a manager who had about 10 or 12 people reporting to him, he was trying to schedule them to about 17 projects. Any attempts to do anything other than run all 17 at once were an anathema to him and his managers.

This leads to….

Resource foobar
For “resources” read people. There are only so many people in the corporate IT department and in an effort to do work people are assigned to work on multiple projects. So #1 their productivity drops because they are now task switching, #2 quality drops because they are thinking of so many different things, #3 predictability falls because you don’t know when they’ll be working on which project, #4 arguments follow over who should be working on what.

At one client I analysed the (highly inaccurate) time tracking system and found some people worked on 14 projects in one week. Although about 54% of employees only worked on one project a week those who worked on more than one averaged 2.7 projects per week each.

Part of the problem is…

Resource (people) pools
People aren’t allowed to stay with the same piece of code for very long. Which means those who manage them must argue over what project they are assigned to and when. And when they aren’t assigned to a piece of work, but they are the only person who can fix a problem, then all hell breaks loose which destroys the predictability of the project they should be on.

Successful sports teams (think Manchester United) don’t break the team up at the end of the season. Could you imagine the manager saying: “Well the season is over, I don’t need you for a few months so I’ll assemble a new team then, goodbye.”

Unsuccessful sports teams (think Everton, I know I do) usually sell the best players at the end of the season and need to rebuild the team at the start of the next.

Sometime they are pulled off work because of….

Year long portfolio planning
Which means the company decides on some arbitrary date what it will work on for the next year. Which supposes #1 the company has perfect foresight, #2 nothing will change, #3 all projects will run as planned.

Question: How can you be Agile if you decide at the start of the year what you will be doing at the end?
Question: Did your company have a year long plan on Friday 12 September 2008? Was the plan still valid on Monday 15 September 2008?

One side effect of this is: demand for “architects”, “designers” and such is very high at the start of the year, while at the end nobody wants an “architect” but everyone wants Testers. So you create your own resource crush.

Too many Chiefs, not enough….
Big companies don’t like coding, its dirty “engineer” work. So that is all sent offshore or hidden somewhere. Which means they need people to manage it: Development Managers, Project Managers, Programme Managers, Work Package Managers, Test Managers, Development Leads, Test Leads, and their friends the: Architects, SMEs, Business Analysts, etc. etc.

Rule of thumb #1: if the number of coders and testers working on a project is less than than half you have a problem.
Rule of thumb #2: if you find yourself sitting in a room with other (none coders or testers) discussing the work of coders and testers with none of them present then you have a problem

Of course, that assume you have a room….

Too many meetings; too few rooms (projectors, teleconferences, etc.)
When you can’t have a meeting because you can’t get a room its a sign that either you have too many meetings, or…. no, its just a sign that you have too many meetings.

When a company saves money on projectors, teleconferences kit, video conference kit, etc. it saves money on the bottom line but people waste far more value faffing around with making do with inadequate resources.

No quality practices at the code face
And when you have too many chiefs you are likely to find that all talk about “Agile” is about Iterations, Stand-up meetings, User Stories and so on.

Wake up and smell the coffee: if you can’t get quality code done Agile isn’t going to save the day. Invest in TDD, CI, Refectoring, etc. etc. While some big companies do this many don’t – many small ones don’t either but they are not the subject of this post.

There are assumptions at work here: quality is expensive, quality isn’t achievable (all software has bugs doesn’t it?), quality is something developers do, or should do, either way, its something people with dirty under their finger nails are involved with. (I talked about this at the Agile Business Conference last month, Quality – How much quality can we afford?)

Even raising the quality question can lead to accusations that the questioner doesn’t “see the bigger picture.”

(Sometime when you dig into things developers are already moving in this direction but they are doing so without making a song and dance about it.)

Hierarchy
This deserve a blog entry on its own. Inherent in Agile as a whole, but not spelt out explicitly, there is an assumption of a equality. That we are all in this together, we all have a common goal, and we aren’t going to let job titles, length of service, age or other stuff get in the way.

Unfortunately most of the corporate IT groups I’ve seen in the last few years have various people who do stand on hierarchy.

Functional silos
Why, o why, do companies organise themselves as: Test group, Development Group, Analysis Group, and so on?
No one group can deliver a product on its own.

Once you have silos you have people (managers) who need to justify and protect their silo. Solutions which mean people are more integrated worry them.

Lack of knowledge / Knowledgeable people
Another reoccurring bottleneck is the lack of people who actually understand the systems and code base. This is sometime made worse by earlier redundancy programmes.

Kelly’s Knowledge Law #1: It takes time for people to become expert on a software code base and system
Kelly’s Knowledge Law #2: Knowledge transfer programmes don’t fix this any time soon
Kelly’s Knowledge Law #3: You are where you are because of earlier decisions not to recruit more people, not to train more people and to fire people

Don’t believe me? Check out Brook’s Law

And to make things worse….

Knowledge doesn’t accrue in offshore partners
The likes of Wipro, TCS, InfoSys and so on rotate their people every few years. So just as Sid is getting to be proficient in your code base he is moved to another client.

That is, of course, assuming Sid stayed with your supplier long enough to get knowledge of your systems.
Assuming he did: remember he works for your supplier not you

Which leads to….

Hollowing out your knowledge pool
In the early days of outsourcing some really stupid companies gave their staff to the new supplier. When they tried to take IT back they found the people and knowledge stayed with the supplier.

So today most (clever) companies keep the key people – people with experience, “architects” and the like. But, these people are growing older, and sometimes they leave of their own accord. Which begs the question: Are you creating the next generation?

How does someone get to be an Architect? (i.e. a developer who might code)
Or an Subject Matter Expert? (e.g. a system analyst who doesn’t code)

Answer: they get to know this stuff because they once cut code.
These people became Architect and exports because you decided their knowledge was too valuable to have leave the company or do dirty things like coding.

So what are you going to do when they move on? Or retire?

Remember: the thing that makes them valuable is a) they work for you, b) they worked for you for so long they know how it works.
Outsource the doing today and you loose you knowledge tomorrow.

I see these issues repeated over an over again. Now I’ve told you it isn’t rocket science so why do companies do to themselves?

As Tolstoy might have said: “Unhappy companies are all alike; every happy company is happy in its own way.”

I’ll continue this rant in another blog posting soon.

Back at the beginning of September Kevin Rutherford tweeted:

‘Anyone know the Japanese for “red, green, refactor” ?’

Now as it happens I know a Japanese speaker very well. Red was easy “akai”. Green was a little tricky because the Japanese actually have two words for Green, “aoi” and “midori.” We went with midori.

Refactor was a little bit tricky. After all, there was no such word in the English language until Martin Fowler published Refactoring. Indeed, “refactor” still haven’t made it into my English dictionary.

My translator suggested the Japanese would probably stick with the English term so we got “akai, midori, Refactor.”

At which point Kevin asked: “akai, midori, kaizen?”

Its taken me a few weeks to have an in-depth conversation about this with my translator but it seems Kevlin is right.

Kaizen is actually two Japanese words:
Kai: change
Zen: good

So the net effect is: “to change [improve] for the good”

Which seems to sum up refactoring perfectly.

Now, any other Japanese speakers out there like to refactor?

And just a gentle reminder that Changing Software Development is now available in Japanese. My personal Japanese translator didn’t do this translation, it was handled by the Japanese publisher.

I’ve blogged before about “The Alignment Trap” (Originally in December 2007 and after my piece in The Agile Journal in February 2009). This is the counter-intuitive evidence that sometimes doing things right is more beneficial than doing the right think. I often discuss this in my presentations and there are two standard reactions: for some this research rings true while others question it.

Those who question it often ask: is there any other evidence?

Well, there just might be.

Delivering Value from Information Systems and Technology Investments: Learning from success is a survey report from Cranfield School of Management in the UK. I only heard about this report a few months ago and its is now four years old but I see little reason to suspect things have changed since 2006.

The researchers define four levels of maturity in organisations developing systems. The first level is project delivery with the focus on the supply side. In order to reach the higher levels organisations must first be able to successfully deliver the IT projects they undertake.

In other words: doing things right is a prerequisite to success.

(I should point out that this report has nothing to do with Agile, Waterfall or any other method. Its about the ability to realise benefits from IS/IT projects whatever the approach.)

The second level of maturity concerns realising the expected benefits from projects. Here attention expands from the supply side to look more at the demand side and whether the business cases proposed are realistic. It is here that “doing the right thing” starts to be the issue.

Levels 3 and 4 concern realising benefits consistently across the whole portfolio.

The report contains some other nuggets of information which are with reporting:

• 73% of survey respondents said significant improvements were needed to get satisfactory value from IS/IT. (This 73% is curiously close to the 71% of companies in the “Maintenance zone” in the Alignment trap report.)
• 38% of businesses openly admit benefits are overstates in business cases in order to obtain project funding
• 80% report that review of work against the original objectives is inadequate

Finally, the other statement is worth analysing from this report is:

“The data also indicates that organisations have undue faith in business cases and that the deployment of formal methodologies gives managers a false sense of security, perhaps an excuse for not becoming sufficiently involved.”

If we take that statement apart it supports the Agile approach:

• “Undue faith in business cases”: Implies more tools are needed to tackle value. Agile would suggest a try, experiment, fail-fast, fail-cheap approach
• “Formal methodologies … false sense of security”: That could be PRINCE2, Scrum or something else. The methodology doesn’t have the answers. You need to be pro-active and think for yourself.
• “Not becoming sufficiently involved”: Yet again we hear that lack of customer/user/manager involvement is a cause for failure. We have a few tools for working around these problems but nothing beats having actual involvement.

In my Agile Business Conference present (“How much quality can we afford?”) I talked about the Reuse Myth, this is something always touch on when I deliver a training course but I’ve never taken time to write it down. Until now.

Lets take as our starting point Kevlin Henney’s observation that “there is no such thing as reusable code, only code that is reused.” Kevlin (given the opportunity!) goes on to examine what constitutes “reuse” over simple “use.” A good discussion itself but right now the I want to suggest that an awful lot of code which is “designed for reuse” is never actually re-used.

In effect that design effort is over engineering, waste in other words. One of the reasons developers want to “design for reuse” is not so much because the code will be reused but rather because they desire a set of properties (modularity, high cohesion, low coupling, etc.) which are desirable engineering properties but sound a bit abstract.

In other words, striving for “re-usability” is a developers way of striving for well engineered code. Unfortunately in striving for re-usability we loose focus which brings us to the second consideration…. cost of re-usability.

In Mythical Man Month (1974) Fred Brooks suggests that re-usable code costs three times as much to develop as single use code. I haven’t seen any better estimates so I tend to go with this one. (If anyone has any better estimates please send them over.)

Think about this. This means that you have to use your “reusable” code three times before you break even. And it means you only see a profit (saving) on the fourth reuse.

How much code which is built for reuse is reused four times?

I would suggest the answer to this question is: very little.

Consequently development teams which write a lot of reusable code are costing their organisations a lot of time and money. Waste.

Reusable code is not the solution to any of our IT problems. It is a supply side only solution and not a very good one at that. While it may reduce the amount of code that is written it reduces it by artificially constructing supply rather.

Thus, reuse, as a solution to software supply problems, is a myth.

This leave two questions we need to answer.

First: how are we to get those desirably engineering properties if we can’t/don’t push reuse?

The good news here is that Test Driven Development, i.e. writing code that can be automatically tested with other code we write, also has the effect of promoting modularity, high cohesion, low coupling, etc. Thus the answer to this question is: Make your code testable.

This approach will retain focus and deliver worthwhile benefits.

The second question: How should we manage reuse? After all, there are some genuine situations were reuse is the right answer.

Here there are two answers, a micro and a macro answer.

The micro answer, when your working in a development team is: look for emergent reuse.

Don’t plan for reuse but look for opportunities to reuse something that has gone before. When you see the opportunity take the previous work and enhance it just enough to reuse it in your case. This way you only pay the price for what you actually need when you need it.

If you later find the same code is useful again then repeat the process. Improve it just enough to use it a third time.

Remember: you have the tests from the earlier answer to make it safe to do this. Without the tests things get difficult.

Now you’ve reused your code three times, you’ve pay the price a bit every time, and you have the tests to show it still works. By now the code is going to be getting pretty close to generically reusable.

Still, maybe you go round this look a third or fourth time. It doesn’t matter is making one piece of code reusable costs more in the long run than it would have done if you did it the first time because… You haven’t spent money making a lot more code reusable that never needed it.

Telling the future is hard. It is difficult to know whether code will be reused or not, so default to Not and save the money. Occasionally you will be wrong and it will cost more, but overall you will be right and you will save money.

The macro answer: if this code really is widely reusable by many people then, go into business. Market this as a library, a plug in, an application what ever. Let the market decide the economics of reusability.

My presentation from the Agile Business Conference (ABC) 2010 is now online: Quality – How much can we afford?