allan kelly

Think about this. If these instruments had the correct rating people would have put a lower value on them to reflect the risk. And banks would have had a different value at risk so maybe they would have seen problems coming sooner.

How come Moody’s didn’t notice this sooner? Well it seems they did but they didn’t want to face up to it – management not wanting to hear what engineers say? – Sounds like the Challenger disaster again.

What happened to code reviews? What about Unit testing?

Let me guess, ‘this is the real world, we don’t have time for that’. The financial industry is build on software but banks haven’t really internalised that yet. The Moody’s bug is not an isolated instance.

Story 1, I was working at an investment bank in the late 1990’s. I was working on a risk system with the quants. They had branched the code base from the main developers months before – when I say branched, I mean it was put on another server so there was no way back.

The bank was silly enough to loose its development team – five of them left within a few weeks of one another. I was asked to take over the main code base and support the main overnight risk assessment system. There was a list of bugs they wanted fixing, most of which I’d fixed in the other code base, but before I could do so I needed to build the live system.

The two code bases on two different servers were different, not just diverged but contradictory. The code left on the developers machines was different again. And being C++ there were a lot of #ifdef’s and I couldn’t tell which ones were on when the live system was built.

I had no way of building the system to produce the same numbers as the live one.

I asked for help but nobody really wanted this news. I could build a system, several versions of it in fact, but none of them produced the same figures as the live system. And nobody had any time to help me work out what the figures should be, or even tell me how I could work them out.

The bank’s risk analysis software was effectively a random number generator.

Story 2, I heard this at SPA this year from “Bill” – a pseudonym. I think it was a derivatives or risk system but I’m not sure. Bill had been examining some figures one evening and they didn’t look right. The more he looked the more wrong they looked.

Bill found a developer and together they looked at the code. And in the code there was a sign error. Numbers which should be positive were negative and vice versa. They made a fix, and reported up. It went all the way up the chain and fast. The fix was signed off and in production very quickly.

I didn’t find out if the bank ever reviewed the numbers, or how long the error was live but, it was quite possible the bank was insolvent on several occasions without knowing it.

Moral? Increasingly our financial markets are built on systems which are not only difficult too understand but which we known contain errors. So what happens now?

Most likely Moody’s will survive – even if their shares fall. The moment the word “code” is used management will tune out, nobody wants this news so lets move on.

But if it is just possible the SEC, FSA and other regulators will get involved. This could be serious and they might demand something is done about it. Not just at Moody’s but elsewhere – clean up financial systems!

Now this is a more interesting scenario because two things might happen.

Firstly this could make Sarbanes Oxley regulation look like a walk in the park. What if the SEC introduce their own coding standards? What if all software needs to be formally approved?

This route would be a field day for the CMMI and ISO guys, the high priests of formal methodology and for document management systems. They would have so much work… and banks software would grind to a halt.

It would also be good news for vendors of shrink wrapped software. If developing your own costs so much more, and gets you tied in regulation then it is no longer cost effective so buy, don’t build.

There is another option, far less likely I’ll admit but…

What if you had an audit trail? You could build the system for any date at the push of a button.

What if code reviews did happen? Or what if code was pair programmed?

What if all source code was unit tested? And the tests kept? And the tests re-run regularly?

What if writing code became so difficult, expensive and risky we had to reduce our use of code? Well then we’d have to really code the right thing – improve our requirements.

Do you see were I’m going? Improving our code base, even regulating it, doesn’t have to be SarBox all over again. Many of the Agile methods are surprisingly well suited to this.

We will see. In the meantime, next time someone tells you you are wasting time writing a unit test remind them that Moody’s lost 15% of its value because of a bug and may have endangered a fair few banks.

I’ve always had my doubts: if we can learn from failure can’t we also learn from success? But how often do you hear people say: “Great success! Now what did you learn?”

All too often we don’t stop, examine our failures, take time to reflect and actually do the learning from them. We’ve failed, failure is painful, we want to put that behind us, forget about it. Maybe we could make time but do we really want to? Who wants to dwell on what has gone wrong? Naturally, after a failure we are defensive, and when we are defensive our learning process aren’t at their most effective.

In the IT world failure isn’t an absolute. Its not like a football match, one side goes home knowing they scored more goals than the other and therefore “won.”

If an IT project delivers late is it automatically a failure? What if it succeeds in the other goals?
What if it is late, and costs more, but delivers more value to the business?

I often tell story I found when I was writing my MBA dissertation. For this project I interviewed a variety of software developers about past projects, one guys, lets call him Paul (because that’s his name), he said: ‘The project was a great success’

He went on to describe how the technology was good, the product did what the customer wanted and everyone was pleased with the outcome. Then I said: ‘You say the project was a success, but before that you said it took 12 months to complete, and how it was originally estimated to take 3 months. Some people would say a project that was nine months late on a three month schedule was a failure’

To which he replied: ‘I never thought of it like that’

Failure is ambiguous. So how do we know when and what to learn?

In Wednesday’s FT there is a piece by David Story in which he looks at some statistics. He discusses the failure rate of entrepreneurs. If we learn from our failures we should expect entrepreneur who have failed in business to be more successful if they launch another business. But it turns out the chances of a business succeeding are the same whether the founder is a first time entrepreneur or an entrepreneur with a failure behind them.

You are as likely to success with your first business as your are the second or third. The odds don’t improve.

The only thing that does change is that you try again. If you never launch a business venture you can’t have a successful one. If you failure once you can’t have a success unless you try again. He likens it to buying tickets in the lottery.

If we don’t learn from our failures, we don’t learn from our success, and often we don’t even know what was successful and what was a failure, then how are we to learn?

We have to buy more lottery tickets. We have to institutionalise our learning, we have to create routines to learn and set our learning triggers.

I’ve written before that I believe Product Management is a much misunderstood role in the UK, and that I think it is an appreciation of this role that makes Silicon Valley companies so much more successful than their UK peers. Well, this conference might be a sign that things are changing. The fact that it existed at all, that it had sponsors, and that it was full is a good sign. However, I still came away with a feeling that only a few companies “get it.”

Practically the conference was well organised and free. It was sponsored by Telelogic (part of IBM) and Tarigo – a company I’ve not heard of before but I’m glad to say offer Product Management training in the UK. As with any vendor sponsored conference it was in some ways a marketing exercise by these two companies, but that said, the marketing wasn’t too heavy and they didn’t ram their products down your throat.

However, the conference was pretty tightly controlled, no questions until the end and no involvement of the attendees. Quite different to my usual type of conference, but then ACCU, SPA and PLoP’s just aren’t ordinary conferences, I’m spoilt.

So what did I learn?

Well I had my enthusiasm for Product Management warmed up again – it would be good to work as a Product Manager again!

Tarigo presented some estimates they had done of the number of Product Managers in the UK. Much to my surprise they believe there are about 90,000 to 110,000 Product Managers in the UK, which equates to one Product Manager for every nine developers.

On the face of it this is positive. I had no idea there were so many! And the ratio of 1:9 isn’t too far adrift of my own recommendations on what you need (1:3 for innovative rapidly changing products, 1:7 for less innovative more stable products) but, Tarigo has included Business Analysts in their figures.

I need to write a full entry on Business Analysts and Product Managers. In some ways the role is similar, in others it is different. I think most BAs could benefit from learning about Product Management and I think that is why Tarigo included them. But, on the other hand the roles are different. In the Q&A session I asked the panel about this and they agreed the roles are different.

So I think Tarigo’s 1:9 figure is optimistic.

And to prove the role is misunderstood I spoke to people at the conference who had nothing to do with technology. They are a different type of Product Manager again. Looking at the attendee list and their companies I think more fell into this category than I realised.

I also got some ideas for my growing business pattern language and confirmation of some of the things I’ve been writing in my latest set of patterns.

I learned a little about Telelogic’s product – I didn’t try to hard – and it confirmed another pattern I’m seeing: specialist database products. I first noticed this when I came across ChimSoft – Chimney Sweep Software – honest, as far as I can tell they are for real.

What I see is this: databases are wonderful things, they can do anything! But to use them effectively you need to program them. In any sector – chimney sweeping, product management – there are people who need a database and don’t have the skills to programme FoxPro or Excel let alone Oracle or DB2. Therefore, there is a market in specialised databases in specific sectors. And exploiting it is nine-tenths good product management because there really isn’t any new technology here.

Extend this argument upwards, to more complex domains, and you arrive at CRM and ERP systems. Both of these are in many ways databases with fancy utilities. But again, this is another blog entry.

Back to the conference.

Finally, perhaps the most surprising thing at the conference was the reaction to one session about Agile Software Development. Unfortunately the speaker didn’t link it up with Product Management very well so it was basically a “This is Scrum” talk but the audience got very very interested. Most of the Q&A session was about Agile in fact.

So why?

Well I think there were several reasons.

Firstly most of these people had not been exposed to the ideas behind Agile before. This just goes to show how things have yet to take off.

Second: the Agile story is very compelling if you know a little about software development. Agile promises products on time, with quality and everyone is happy.

The problem with the Agile story is: unless you know a little about software development already then it doesn’t sound any different to anything else. You abstract the message for those who don’t know software development (i.e. senior managers) and you get: improved productivity, reduced time to market, increased business value, etc. etc. At which point are we talking about Agile or Ration Rose? Or .NET? Or BoundsChecker? Or outsourcing to EDS? Or India?

I have more insights from the conference – so it was worth going to – but they are destined for my current crop of patterns.

Why should I blog about airports? – this blog is about the business of software development. Well I could argue that its all business and within remit but I’m still its stretching it a little.

The reason to blog about terminal 5 (T5) is because past I have held it up as an outstanding example of lean practices in action.

• Compare and contrast: Terminal 5 and Wembley Stadium
• Terminal 5 is Lean

T5 is part of Heathrow airport, and Heathrow is owned an operated by BAA. In the past I’ve talked about T5 in isolation but, as all good system thinkers know, we have to look outside the immediate to understand the whole.

I think there are some interesting lessons to be learned from the recent mess at T5 and at Heathrow in general over the last few years. Some of these lessons might just help software developers.

Lets start with the re-occurring horrors that are Heathrow airport – well talk about the T5 opening mess another day. Take some examples:

• Disruption and Flight backlogs
• Reoccurring delays
• Baggage handling problems and cancellations
• Knock on problems elsewhere
• General unhappiness with the airport – I agree, I live close enough to it to make it pointless trying to use another and the experience is ghastly.

The paradox at Heathrow is that while T5 has been an exemplar case-study in Lean Thinking the rest of Heathrow shows many of the faults of traditional thinking non-lean thinking.

Travelling through Heathrow is an exercise in queuing: Lean Thinking tells us how to manage queues. Heathrow is an operating at the absolute limit of capacity, therefore it is highly efficient but prone to disruption. The slightest problem and everything is messed up.

While BAA managers have claimed that the opening of T5 would solve the problems they ignored changes on the existing terminals that would make real improvements for passengers. They have looked for a single big fixed (T5) over a myriad of smaller fixes.

For example:
• Why not strip out some shops and provide more seating? Why not strip out some shops and make popular ones bigger? OK I know why, because they make more money, we’ll come to that.
• Why not copy Liverpool airport and give people the option to pay for fast-track security? People who have a genuine choice will be more accepting.
• Why not give free water bottles for people who have their water taken off them at security?
• Why make it worse for passengers with petty, pointless rules? – the one that caught me last year was taking a cup of coffee (bought airside) with me to the departure gate. This is not allowed. Why not?

(By the way: has anyone noticed the one place at Heathrow where you can’t by a Grande Cappuccino? – The baggage hall, waiting for a case which may, or may not, appear is devoid of shops. Yet it’s the place where I’m usually most gasping for a drink, chocolate or newspaper to read while I wait.)

Part of the reason BAA has gone for the one big fix is because the myriad of small fixes requires staff co-operation, and it turns out Heathrow is a bastion of confrontational management, highly unionised workforce and prescribed working practises.

Customers have very little choice: BAA control the three biggest airports in the London area. Once you are at the airport BAA control everything you can do – doubly once you get through security.

So what are the lessons for software developers?
• Lean/Agile has its limited: No matter how lean T5 construction was the rest of the airport is a disaster. One group doing lean will not automatically infect everyone else.
• Pay attention to your customers: if you treat them badly they will start to find ways to buy competitor products, or simply stop buying (travelling). Vendor lock-in will eventually be broken.
• Queues work, they make you more efficient but they reduced your ability to react and change. Nobody likes queues, they make us all miserable. The fact that many software queues are hidden is no better.
• Management need to co-operate with those who do the work. They need to accept that a million small solutions can be as worthy as one big one.
• Short term gain (all those shop rents) detracts from long term improvement and customer experience. Sometimes the bottom line has to take second place.

Fundamentally the economics of Heathrow (and BAA and BA) are just wrong. I might go into that some other time, for the moment I’ll refer you to the Economist. Their analysis and solution is spot on. The objectives of BAA, shop holders, Government security, airlines and passengers are at odds. Nobody seems to benefit.

Perhaps the biggest problem is the business models of BAA and British Airways (which dominated Heathrow and is dominated by the airport) are utterly dependent on squeezing as many people through the airport as possible. This is not good for our national economy – let along London – but because BAA and BA carry so much political clout they are able to keep the Heathrow economic-fantasy protected. Making Heathrow responsive to real economics (as we did with coal mines, ship yards, steel factories and car companies) would actually solve the problems.

So the last lesson: It’s the Economics, stupid.

Software companies, unlike Heathrow, live in the real economic world. In the short run a company may be able to run a business model which harms its customers but customers have choice. So you’ve got to get the economic incentives aligned. If you are developing software make sure your economics benefit everyone: what is good your customer should be good for you.

Jobs and banks

There were a lot of bankers at the conference this year – or rather developers who work in banks. In truth this is always the case but this year I think there were more. It also gave the opportunity to find out what was happening in the financial job market.

Before Christmas I think many people were expecting a shake-out similar to that of 2001/2002 but it seems the effect of the crisis or credit crunch is very mixed. I observed a few weeks ago that one effect has been to push up rates and while some banks do seem to be shedding staff others are still hiring and demand is strong. I talked to someone from one (American) bank which had drastically cut back staff, while people at another (English) bank were still hiring lots, and another (Scottish) bank which recently bought another (Dutch) bank is having to hire lots of people to help integrate the systems of the two banks.

Lies, Damned Lies and Statistics

I’ve had a downer on metrics and statistics for a long time. But I’ve also been aware that getting the right data and measuring the right numbers is often the key to success. Tom Gilb is firmly in the numbers and metrics camp and argues a good case for measurement and targeting. I was lucky enough to spend lots of time listening to Tom and talking to him.

It seems that, as with so many other things in life, the key is doing it right. It is very easy to set the wrong targets, to measure the wrong thing and produce side-effects you don’t want. But when you look at the right numbers, measure the right thing, and set the right targets you can get great results.

But it isn’t easy, most people get it wrong.

Tom is a fascinating guy, if you ever get the chance to hear him speak do so. A lot of his ideas come directly from Deming – who he knew personally. I don’t think it is too much of an exaggeration to say Tom may be the Software World’s own Deming. If nothing else Tom interprets Deming’s message for software development.

Software development success

In an aside Tom also pointed out that the UK now has Royal Academy of Engineering and that they (not so) recently produced a report on software development. Its a shame that you can’t down load the report, I’d like to read it.

I notice the comment that “only around 16% of IT projects can be considered truly successful”. That 16% seems very close to the figures given in the MIT Sloan Review piece on the IT alignment trap last year. That report said 7% of companies had effective IT departments which delivered on business objectives and another 8% who were effective but were not aligned with the business.

Unfortunately that means 85% of us are working on failing projects. Depressing.

The 1968 wrong turn reconsidered

I’ve been heard to say that somewhere about 1968 the software industry took a wrong turn. We went down the route of engineering, planning and tools rather than people and learning. But the conference made me wonder, maybe it wasn’t such a wrong turn, maybe it was a diversion we needed to take so we could solve some problems. Now those problems are kind of solved we need to refocus on the people.

Maybe.

Product Managers

I have long claimed that UK business do not get Product Managers, this might now be changing. The term Product Manager was in wide use and more people seemed to have a Product Manager on their team.

Lets hope I’m right.

Still, there are too few Product Managers, too few of them are really good, their role is still misunderstood and there is not enough training for them.

Next years conference

I’m no longer on the committee for the conference but I still have conversations about it. It is already taking shape in peoples heads and promises to be an even better conference. For better or worse the conference is unlikely to get any bigger. If it were to get bigger it might loose some of its flavour.

I think the next 2-3 years will be an interesting time in the UK and European conference scene. The arrival of the profit making QCon is having an effect and I know there is some debate about the future, style, content and so on of other conferences. At a guess I think you might see one or two new conference appear and others (perhaps) disappear or change.

My book

It was good to see Changing Software Development selling, and fun to sign copies!

Latest figures say it is selling well, yippee!!!

This Blog

I met several readers of this blog at the conference. Seems they want shorter entries. I’ll try my best. But I’ve said this before and failed!

So I took the opportunity at SPA to ask Steve Freeman about it. Unlike me, Steve has spoken to David so here are some of the points I got:
• Kanban teams have given up on planning and estimating entirely. The activities took more time than they were worth. Prioritisation is – as far as possible – handed back to the business. The highest priority is simply done.
• Kanban teams have a number of ‘slots’ by which I think they mean ‘tasks they can be working on at one time’. The business decided what to put through these slots.
• Teams track how long it takes for a piece of work to move through the system. I guess they just note the start date and end date. They can then tell what their throughput is which replaces the need to estimate.
• Work is tracked by cards on a board. This visibility together with the flow measurement helps show where the bottlenecks are.
• The business may (i.e. not all teams do this) offer the business a ‘Silver Bullet’. The business can use this to cheat: to fast track a feature through the system, to do a feature without any of the usual pre-work, or such. However, the business is only allowed one Silver Bullet at a time.

It all sounds very interesting and I hope to hear more. And if anyone reading this knows any more – or thinks I’ve got something wrong – then please add your comment.

According to Steve, David is finding that his bottleneck is not development. The bottleneck is earlier, its with the business deciding what to do and what needs to doing. This is something I have observed myself. In fact I thought I had written a blog entry on this subject, perhaps with the title ‘The Bottleneck has moved’ but I can’t find it so probably I haven’t written it just thought about it.

Anyway, I’ll say it now: The Bottleneck has moved.

In my own experiences with my Blue-White-Red agile method, I found that while it took time for developers to do work this was simply leg-work. More developers more work. Quality improvements (unit testing, code reviews, etc) helped and these made the teams more productive. But when ever the developers needed to consult the business (i.e. product management) things slowed down. Or, developers took the wrong decision and re-work resulted.

And if we want to practice Lean development then we can’t waste time building the wrong thing. We need the business to support us.

Over the last 20-30 years developers have got a host of new toys to make them more productive: their own (fast) machines, more productive languages, nice IDEs like Eclipse, and so on. Surely some of this technology has boosted their performance. On the other hand, the pipe coming into development hasn’t go any bigger. If anything it has got smaller.

What strikes me about Kanban is that it requires more business buy in, more business acceptance of the method and requires the business to know more about its role and what it wants. This is good because it makes development more responsive to the business.

This is entirely what I would expect because, while Agile/Lean can improve development alone, it ultimately needs to exist in an Agile/Lean environment. If it does not there will be tension, eventually the organization will kill its Agile development or Agile development will infect the wider company. This might be what David is seeing (or creating!) with Kanban.