allan kelly

Here is an observation which I’ve been been playing back to clients for a year or so now:

• There are two types of company in the world, those who talk about automated testing and those who do automated testing.
• Those who talk about automated testing buy tools like Quick Test Pro, WinRunner and Quality Centre and Rational Test Workbench, Rational Quality Manager, i.e. expensive products from the likes of HP and IBM.
• Those who do automated testing use tools like Selenium, Fit & Fitnesse, Cucumber (inc. Gerkin, RSpec, etc.) and other Open Source products they don’t need to buy

For completeness I should add:

• There are a few, very few, companies which don’t talk about automated testing but on the whole everyone thinks it is a good idea and those who don’t do it would like to
• There are a some companies who talk about automated testing and have no tools; this is a better position to be in than talking and spending money

This is my observation, and if I’m being honest I did see a company 12 years ago which used WinRunner in a load testing environment. But, on the whole the software test suites from IBM and HP tend to be shelfware. In fact, this blog is motivated by a trip to a company which found my observation very funny because QTP is sitting on a shelf unused.

What I would really like to know is: has anyone else seen this? Or, can anyone give a counter example?

I think there is even a logic here. The IBM and HP products are expensive, so expensive you have to ask the price (actually IBM does give the price of Test Workbench at $5,500 for a single user license). Consequently they need to be sold, it also means that they need to be bought at a high level in the corporation. The people who are sold these products are very disconnected from the day-to-day work of developers and testers. This means the products might not be suitable and even if they are then developers and testers need to adopt them: they aren’t part of the change decision.

In addition: I just don’t think these tools are any good. To be fair, I’m no expert on testing tools but I’ve seen Quality Centre in action. Quality Centre (QC) isn’t an automated testing tool, its a very expensive test tracker. Testers write their test scripts in natural language, e.g. English, into the tool, the tool runs on the side and as they execute the tests they click Success or Fail. At the end QC gives a report.

To my mind QC is a fancy version of Microsoft Word. English language test scripts, if they are any good, are so detailed that they take almost as long to create as automatic scripts but they are expensive to execute because they are manual.

QTP and similar products are often linked to the OS or browser. They result in fragile tests which break easily when a box moves a few pixels or the browser changes.

HP also follow a razor blades model, I’m told: once you’ve bought QTP you need to buy a plugin for every browser to OS you want to use it with. Every time a new version of the browser comes out you need to buy a new plugin – or so I am told, tell me if I’m wrong.

The net result is: these tools are high maintenance and easily fall into disuse if they are ever used.

On the contrary: Open Source tools are adopted by people who do the work because they see the benefit they bring and they don’t need budget or signatures to get the tools. People get them because they want to use them not because a Salesman comes around and convinces the boss that you need a tool.

And because many of the Open Source tools require developers to create glue code to interface the tool to the application they lead to a more collaborative style of working.

(Perhaps some companies who just talk about automated testing have downloaded some of the Open Source tools but never use them tool. Since they are free nobody notices.)

So there is my theory. Anyone else seen this? Anyone else explain it? Anyone got a counter example?

Last week I made the London to Cornwall train journey for the first time in a few months. I made the journey many times during the course of the Agile Cornwall programme. This time however I was on my way to the Agile on the Beach 2012 conference. The conference itself is a product of the Agile Programme and looks set to run again in 2013 (although the final decision has yet to be taken.)

With the release of a report last week which claimed the Agile Cornwall programme created 50 jobs I felt Agile on the Beach is something of a celebration. I’ve now had a chance to read the report in full and while the report is not available for public download I would like to share a few comments from it.

First I should say that the report itself was conducted by a third party company commissioned by Oxford Innovation. Oxford Innovation runs the Grow Cornwall programme on behalf of public bodies and it was as part of this programme that Agile Cornwall was bored.

The report surveyed the companies who participated in the Agile programme to find out about their opinions of the programme and the changes which had resulted. They spoke to multiple people in most of the companies so it was pretty well grounded although it probably wouldn’t stand up to rigorous academic analysis. (If any academic out there would like to do a rigorous study then please get in touch.

Here are some highlights from the report:

• The Agile programme has met expectations for all companies who participated in the review. (I suspect one which was disappointed did not participate.)
• Businesses are finding they are more flexible and responsive to customers
• Money and customers
  • None of the business reported a decrease in turnover, profitability or customers since adopting Agile.
  • 7 companies reported increased turnover or revenue; 3 reporting the increase was significant
  • 6 reported increased profitability, 3 saying the increase was “significant”
  • 6 reported an increase in customer and/or business opportunities with 3 of the six saying the increase was significant

There are some downsides however:

• One company reported that it was now more difficult to hire because they could not find people with Agile skills.
• Small companies said it was difficult to find the time to get started on Agile

All of which makes me very happy.

There are also some very positive comments on the quality of the consultants used in the project, as I reported in my last blog it wasn’t all me. But, if I may be permitted to blow my own trumpet, I was there a lot, I worked with more companies more of the time than any other consultant. So I’ll be giving myself a pat on the back!

The report has, unfortunately, not been made public – although a summary version was given to attendees at Agile on the Beach. So if you would like to see the full report I suggest you go and knock on the door at Grown Cornwall / Oxford Innovation.

Regular readers will now that over the last two years I’ve done a lot of work in Cornwall, I jokingly refer to “the Cornish Software Mines” (A case study and Light Touch Coaching to name two blog entries.) To date the most visible result of this work was the Agile on the Beach conference which is running again next week.

There is now a report out which has looked at the programme overall and concluded that the Agile Cornwall Programme created 50 jobs. That might not be a big deal for London but in Cornwall that is a big deal. Particularly since half the jobs were high paying.

So far I haven’t actually seen the report so I just have this report on the report to go on. I’m trying to get a copy as we speak.

I can’t take all the credit, I had help. The thanks need to go to:

• Oxford Innovation for creating and running the programme
• Michael Barritt who was the driving force while of Oxford Innovation and has since moved to Sullivan Cuff Software (one of the programme’s success stories)
• Jon Jagger who provided most of the technical coaching
• Benjamin Mitchell who coached several of the clients when I was fully stretched and gave intensive help to one of the more difficult clients
• Plus Kevllin Henney and Sebastian Bergmann who provided additional training, and Nancy Van Schooenderwoert who provided specialist embedded consulting

This report also looks it should help providing some evidence for Agile.

There is still time to book tickets for Agile on the Beach if you want to meet all these people and some of the people who benefitted.

“The second is the most dangerous system a man ever designs” Fred Brooks, 1975 & 1995

Brooks was talking about software designers, architects, but I think the statement holds true not only for all software developers but for the business people who commission replacement systems.

From time to time I come across a team who want to rewrite their system. Usually this is the technology team who are saying “The code is too complex, the thing was written by a jerk, She cannae take it Captain”.

Sometimes is the business side realising that their system looks out of date – Visual Basic 6 controls on Windows Vista, or turn of the century web style.

Either way there is a “Stop the world I want to rewrite” mentality. Sometimes the powers that be buy the argument, sometimes they bless a rewrite or, as one team called it a “Technical Improvement Programme” – TIP for short.

Lets me come to the point: this is always and everywhere a mistake. You should never, ever, rewrite a system just because the technology is old.

This is a mistake because customers don’t want to change. The may think that VB6 controls look odd but they know how to use it. Even if change would benefit them they are scared of it. Plus, they have their own business to run – or life to live – and changing systems is only going to get in the way.

Second, this is an utter waste. Statistics regularly show that as much as 80% of features in a typical software system are not used. So why implement five times more functionality than you need?

One company I worked with commission an cheap Indian company to rewrite an old PowerBuilder application in Java. They were told to make it identical. So they did, they spent a lot of time making Java look like PowerBuilder. They copy every feature, even the ones not used. It was buggy as hell and the users didn’t want it.

Third: the “just a rewrite” mentality leads people to under estimate the complexity involved and to overlook the opportunities for improvement. Why produce a second system which is identical to the first?

Finally, and worst of all: the world doesn’t stop, you can’t afford to stop and rewrite, nor can you afford to waste the opportunity to produce a really good system, one that is better than competitors.

Yes, I agree your current system might be at the end of its life and you might need to throw it away but the solution is not a rewrite. When this happens you have two options.

The first won’t satisfy every problem but is the least risk: refactor your way out of the problem.

Actively encourage programmers to refactor code, of course you need to embraced Test Driven Development at both the development and testing level first of all and start putting tests in. Thats not to say you cover it in tests first and then refactor, rather you start putting the tests in as you go. This is a bit of high-wire act without a safety net at first so I strongly recommend you get some help if you’ve not done it before.

This is real engineering, this is changing the engine oil while the engine is running.

That is the easy answer but it won’t work every time. Sometimes the changes are too much, say from VB6 to C#.

The second answer to develop a new product, a competitor product. One for which your current system/product is the competition. This development effort needs to be market lead and focus on what customers/users want the the current system doesn’t provide.

I’ll say it again: Market led. The marketing department, the product managers and/or business analysts should decide what goes in. Not the developers.

When companies rewrite they jump to the assumption: “The developers know what the current system does so they just need to copy it.” That is the biggest mistake.

Next, get into the market fast, release early release often. Test what you are building in the market, show your company real, usable progress. Remember, your competitor is the current system, if you can’t do better than it you want to know fast.

The objective is to build a system that your current customers want to use, one for which they are prepared to give up the system they have at the moment. Consider this a variation on Same Customer, Different Product pattern (available online or in the Business Patterns book).

Emphasis the things the new system does that the old one doesn’t. Don’t worry about all the features the old one has that you think you need to launch. You might need them in the end but nobody is going to buy your software because it does the same thing. Add them as you need them, you’ll find you need to bring across far fewer than you initially think.

Now I apologise to all the developers out there who are begging their managers to let them rewrite the system. Your code base may be terrible, the technology may be out of date but a development led rewrite is not the answer. Few companies can survive such a thing.

The last two blog entries (“Scopeless contracts: the problem” and “Scopeless contracts: the solution”) have been working up to this one. Actually this is the blog entry I set out to write a few weeks ago but felt I had to explain why.

Even if you accept that leaving scope out of a contract for work is the right thing to do you still need to sell it to your customers. This is, perhaps, the real problem. However this is not always the case. Some clients will immediately recognise the problem, or might not wish to sign a big up front deal.

So, faced with a customer who wants to know “How much will it cost to do X ?” what should you say? Or rather, what can you emphasis to persuade them that you have a better idea?

(Note, on anything other than a trivial piece of work X will not be one thing but many, call them X1, X2, X3, … Xn.)

1. Flexibility: scope less contracts build in flexibility, if the client wants Y instead of, or in addition to X then thats OK. No need to scrap this contract and start again.
2. Work can go away: Suppose X3 become pointless, then it can just go away without any trouble. As opposed to the UK Government Connecting for Health project [[link]] were actually cancelling the project is more expensive than letting it finish.
3. Room for innovation: if you sign a big contract and later think of a bright idea then something has to give. Scopeless contracts allow for this because feature negotiation is ongoing.
4. Bounded by money: Rather than being bounded by scope contracts are bounded by the money available, or sometimes the time available. This forces the creators to work within a very real boundary rather than poorly defined scope and time estimates.
5. Governance control: bounding the contract in money provides for real governance rather than the governance by proxy that scope, features and line items provide. When the money runs out the work stops. (This of course assumes the team are making regular deliverables, if they aren’t you have even bigger problems.)
6. Inventive to reduce work: putting 2, 4 and 5 together you get an incentive to reduce the work to be done rather than inflate. In traditional work scope is problem because removing it creates contract problems and loss of face, but more importantly: there is little incentive to remove work until difficulties hit.
7. Reduce delay & waste, keep options open: All the time spent writing and negotiating scope delays the time when you actually start delivering something. If things change, and the longer you spend writing the scope the more likely things will change, then the more of that work which will be wasted. And the more you think you know what you want the more you constrain the options for producing the optimal solution. Selling without scope reduces this work because scope is derived as you go within the process and only enough is produced as is needed.

You can, and people do, do all these things within traditional scope based contracts but there is additional work to do to change the contract. And, more importantly, it starts the contract with an illusion, a myth: the idea that one day things might be done. Which gives us:

1. Traditional work relies on a scope myth “scope won’t change”. It does but we pretend it was because something was wrong. The result is we spend time and money managing the myth: managers need to revise contracts, staff change boards, write complex governance reports, etc. etc. In other words: because managing scope is doomed to failure the thing we call “scope” is a chimera which we waste time and money on “managing”.

Despite three blog entries I still don’t feel I’ve said what I wanted to say. This is a topic I need to return to….

In my last blog entry, “Scopeless contracts: the problem” I rehashed all the reasons why including scope in a contract for delivery is a bad idea. So having rubbished the most common way of working I had better explain what I suggest you do….

You have two choices.

Option 1: Set up a ‘Business as Usual’ contact.

The client nominates work to be done and the development team do it. Simple. This is the way most organisations work “business as usual” but once the idea of a project enters the scene people expect the project to have an end date, in order to have an end date they imagine that some amount of work needs to be done.

You can still have an end date but instead of measuring progress against “work to be done” (he scope, the backlog) measure progress against the value added.

After all, software is never finished. If you ever run out of requests for a software product it means nobody is using it. However it might not make sense to do all the work that is requested.

How do you size the team? You don’t, you start small. You start with just the money you can afford to loose. After a while, say three months you review what is bring done and decide whether you wish to go faster (expand) or slower (contract), or perhaps stop altogether.

(Another reason to start small is to reduce the effect of Conway’s Law: if you start big you will get a big programme of work, if you start small you might just come up with a small solution.)

Option 2: Set a Goal, work to the goal

If option 1 isn’t governed enough for you then set an overarching goal. Every three months (or what ever time you decide) ask yourself: are we making progress towards the goal?

You might even want some people on the team measure progress towards the goal (these would normally be Business Analysts). All the work you do should be tested against the question “Does this move us towards our goal?”

If you want more structure use my 10 Step Requirements model.

Yes, it really is that simple.

Drop the backlog. Set a goal. Work business as usual. 1, 2, 3.

The thing is, when you contract with someone to build you software its not like asking them to build you a house. You are buying a service not a thing. If you could buy a thing you probably would have done already. Once you make that mindset shift it all makes sense.

The only complication is, well there are two really: first you have to accept the project model is broken so you need to make more strategic decisions. If you need to keep projects for accounting purposes but projects should have nothing to do with development.

Second you need to do this on a rolling basis, review it every three months.

For most organisations the biggest problem is that they have come to believe the project myth, and they believe it all the way up the hierarchy. Until the people at the top really understand that IT doesn’t fit into the project model its difficult to change.

I’ve discussed Agile contract models before and I’ve agued against points based contracts (‘Points based contracts? Just say No’) but the more I think about client-supplier software development contracts the more I think including any scope in the contract is asking for trouble.

Before I explain let me say: Yes I know this isn’t going to be possible either with your clients or your own sales folk, I’ll follow up this blog entry with another setting out what you should sell instead.

Now I’m using the word scope, although I hate the word it is the word most people recognise. I mean: “the work which will be done.” Call it a requirements document, a PRD, a product backlog or anything else. Its a shopping list of things you think you want.

Right now, here is why I think fixing scope inside a contract is a bad idea:

Scope is always unknown: you may think you know it but you don’t because things change, learning occurs (on all sides), when someone see something they change their mind about what it is they want, or what they ask for, or what they want next.

I’ve been over this ground again and again, first in Why Do Requirements Change (2004), again in Changing Software Development (2008) and more recently in Dear Customer (2012).

And the bigger your project is the more your requirements will change and grow. Capers Jones gives the industry average as 2% per calendar month for large projects. If you’ve got a project/programme that is up and running you can go and measure it yourself. One team I know saw their product backlog increase by 50% between January and July.

Someone said to me recently: “Surely the team has got better at this now, they’ve learnt.” Well yes, they will have got better, that might take the edge off, might rule out some human factors but the world hasn’t stopped.

The World is Changing: lets suppose for a moment you are running a big project, say its been running 2 years and you think it has another 2 years to run. Cast your mind back 2 years, what was the world like in July 2010 ?

First some context: David Cameron has been British Prime Minister for 2 months, Osama bin Laden was alive and well, few people outside Japan had heard of Fukushima, Deepwater Horizon had just been capped and oil was under $50 a barrel.

• The first generation iPad had been on sale for 3 months, it was far from obvious that Apps were about to take over the world
• Nokia were still pushing Symbian phones and were still the biggest mobile phone seller and Android was in version 2.2 (Froyo)
• JP Morgan was fined £33m for failing to protect client money
• Greek debt was downgraded to junk in April
• Barclays were fixing Libor

Go back a little further: Windows 7 was release in October 2009. Further still: the iPhone was released in June 2007 and the AppStore in July 2008. Not that long ago.

Did any of those events effect your business?

Did any of those events, or countless others I haven’t mentioned effect what your business wanted?

Now ask yourself: what could happen in the next two years, to 2014, that would change what your business, your customers want?

President Romney? Grexit? Finexit?

Scope should reduce as well as increase: when you fix scope you also make it difficult for scope to reduce. Requirements should be reducing as much as increasing.

Marty Cagan recently blogged about “The Opportunity Backlog”. Your backlog is not a list of things that will be done, it is a list of things which could be done, and you should do as little of it as possible (its expensive), instead focus on the opportunities which deliver the most.

You human, you make mistakes: well maybe not you personally but the people who work for you.

There are mistakes in what you ask for, mistakes in what you build and mistakes in how people use it. Sometimes fixing a mistake is more important than doing something new.

Second system effect: there is a blog entry waiting to be written about version 2 projects, for now lets stick with what Fred Brooks said in 1975: “The second is the most dangerous system a man ever designs. … The general tendency is to over-design the second system, using all the ideas and frills that were cautiously side-tracked on the first one.”

Whether you are designing from a technical, user or business perspective you suffer from second system effect.

OK, am making my point?

Since requirements can and will change there is not point in writing them into a contract.

This point is doubly dangerous if you attached estimates and timelines to the contract because you will be wrong again. Human’s Can’t Estimate.

This is bad for the supplier who will sooner or later have to explain what is going on to the customer and its bad for customers who don’t get systems that work.

Following on from last blog entry, IT does matter, more important than ever, I’d like to highlight a study that may have slipped peoples attention. Hardly surprising because the study appeared in an academic journal which is a good place to hide such things – the MIS Quarterly March 2012 (paid for download) if your interested.

Let me cut to the important findings:

• “IT investments have a greater impact on firm profitability through revenue growth than through cost reduction”
• “IT investments have a positive and statistically significant correlation with sales and profitability”
• “an increase in IT expenditure per employee by $1 is associated with a $12.22 increase in sales per employee”

That is, rather than looking to IT to save the company money, i.e. reduce costs, IT spending on generating revenue – innovation, new products, new services, new ways of capturing customers, and so on – is more beneficial to the company.

Second:

• “the effect of IT investments on sales and profitability is higher than that of other discretionary investment such as advertising and R&D expediters.”

So, if you have a few spare dollars you want to invest in the company you are better off spending them on IT than advertising, or research and development. But then, since much of IT – specifically when developing new products – looks a lot like R&D perhaps its not always clear.

And yes, the authors are clear: IT spending has more benefit than advertising. That might come as a surprise.

The research paper speculates on why these findings might be. One reason is that in the Internet age things have changes and IT is now more valuable than it was before. Put this together with my comments in the last blog entry, namely that modern corporate IT may well be able to the customer experience and you can start to see why.

The authors also speculate that the reason why IT does not produce greater savings is a) that many of the savings available have already been taken and b) cost savings are easily copied by competitors while innovations – new products and services – are far more difficult to copy. Which also implies that there is more competitive advantage in using IT for innovation, products, services and improving the customer experience.

(Part of the reason cost saving IT projects are easy to copy might be that they often involve off the shelf software and third party providers – often consultants – who your competitors can also use.)

There is a third finding that is also worth noting:

• “IT has a greater effect on firm profitability in service industries than in manufacturing industries”

What the paper doesn’t discuss, and I think is missing, is: how do these findings stand up when you consider more-effective and less-ineffective IT. After all, less effective IT can be a good way to through money way. For companies with poor IT deliver performance is IT still a better bet than advertising?

After all, the Alignment Trap suggested that less effective IT performance could lead to a decline in sales. I’d like to see this research go further.

Just under 10 years ago Nicholas Carr wrote a (in)famous piece in the Harvard Business Review entitled “IT doesn’t matter”. The argument was, post dot-com-boom, that IT was now a commodity, companies didn’t need to spend big bucks on it because they could buy just about anything they wanted off-the-shelf.

At the time my response was, “Is IT worth it?” I, unsurprisingly said “Yes it is” and then looked at Carr’s example of American Hospital Supply. My argument here was that rather than be defeated by commoditisation of IT American Hospital Supply could have chosen to make IT their business. It was a strategy decision.

Anyway, 10 years on. Commoditisation of IT has continued but it is hard to imagine anyone arguing like Carr that IT doesn’t matter any more. Indeed, IT is more important and in more and more cases is becoming the business.

Technology has destroyed the music business as we knew it and is in the process of remaking publishing; television is going to be changed soon.

IT forms the marketing channel now – Facebook, Twitter, LinkedIn.

Logistics is IT based – DHL, drop shipping, Amazon.

iPads, iPhones, Web – are the way you talk to your customers.

And thats the vital bit: information technology, computer technology, software technology is no longer a back room cost cutting tool. It is the customer interface. It is the customer service experience.

Take the travel industry for example. Go back 15 years, 1997. I remember booking my first flight to the US that year. I went into a travel agent, a young lady sat on the other side of the table and poked at a green screen. We talked, she booked me a flight. In 1997 how many of us ever saw the inside of a travel companies IT system?

OK, in 1997 a few were appearing on the web, go back to 1992 and I can safely say: if you didn’t work in the travel industry you didn’t touch travel IT.

IT was a back room activity. Yes the company needed it but the dependency was recent. Customers were unaware of all this.

Today that has changed. This year I’ve booked flights on BA, KLM, S7 and Virgin Atlantic. How I book them – the company web site, Opodo, Expedia, SkyScanner has a lot to do with the customer service experience. I normal start a flight search on SkyScanner, switch to Opodo or Expedia to book and complete things on the airlines own site. If a site is difficult to use I stop and switch elsewhere.

Thats flights. Booking the family holiday is even more IT dependent. Crummy interface, can’t get the answer and I drop the holiday plan and go to another supplier.

I’ve been making this argument for a few years now. So it was good to see McKinsey & Company catch up with me in April when they published “Are you making the most of your company’s ‘software layer’?” Its the same argument:

Software now forms an integral part of the customer service (i.e. buying) experience. Get it right (Amazon) and win big. Get it wrong and you will never be heard of again.

Underlying your delivery might be commodity products but choosing when and where to use off-the-shelf and when to write your own is more important than ever.

Now, imagine for a moment you were a newspaper editor in 2003. You read Carr’s article, you agreed with it, you commoditised your IT. Does that still look like a smart decision?

IT systems might be commodities under the hood but your thinking needs to be ahead of the pack and you need to be able to deliver on what you decide.