Returning to my series of posts applying the tools of economics to software development – Supply & Demand in software development, Software supply over time and Software supply & demand – this time its Agile – it is time to turn our attention to the demand curve. First a reminder of how things start…

First I need to explain why I believe there is so much demand for software and why I believe the curve is inelastic, i.e. why a higher price doesn’t reduce demand very much. The reasoning here breaks into five groups: technology progress, separation of demand from cost, separation of benefit from delivery, lack of evaluation and the effect of “fixed” work – although the last in that list doesn’t always play a part.

It is important to remember in all this discussion that software is a derived demand. Nobody wants software for its own right, they want it to achieve some other aim.

Technology Progress

Moore’s Law implies that processor power doubles every two years (give or take a bit). Consequently the ability of computers to tackle ever more complex problems and fill more needs constantly increases. Supply creates demand, we want it because we can.

For example, I’m hoping Farther Christmas will bring me an activity tracker for Christmas, I didn’t know such devices existed until recently but I can see it solving one “problem” in my life. These devices didn’t exist until recently, I didn’t need one because they didn’t exist. The underlying problem existed but had been put in the “thats life” bin.

At a company level this means the benefits computers can bring to our companies are constantly increasing and there are new opportunities to save costs or generate revenue. And if the incumbent company doesn’t take these opportunities then Amazon, Google or some start-up will.

Moore’s Law is not alone, Metcalfe’s Law compounds the effect. The more devices we enable and the more devices we connect to the internet the greater the value and opportunities from connecting these and more devices.

Separation of Demand from Cost

Traditional development process – and even Agile processes – tend to separate the demand for software technology from the supply, consequently those requesting technology are isolated from the cost of that technology.

Many practices in software development make this problem worse. We send Business Analysts to talk to “users” about “What they Want” i.e. what is their demand. These BAs are sometime little more than order takers, waiters. People are encouraged to ask for more and more. Indeed, some of our practices incentivise people to ask for more and punish them for not asking for it.

Even if “users” don’t ask for it Analysts may be encouraged to include any need they can comprehend. I once had a BA tell me “If we miss a something in the requirements document it is seen as a black mark against the BA.”

Separation of Benefits from Delivery

To make things worse its not just demand which is separated from costs but so too are the benefits. Nobody wants IT for IT’s sake, they want it for some benefits. But achieving the benefits may well take time, it might be months or years after an IT system is delivered before the benefits are seen.

That is, if the benefits are ever seen, IT alone is not enough to bring about benefit. Users may need training, changes to processes may be needed, customer agreements may need to be changed and so on.

Although it is a few years old now Wired for Innovation contains a good, short, discussion of why IT benefits don’t appear in the way we would like them too (i.e. quickly and easily) and what can be done to help.

Lack of evaluation

Ideally companies would evaluate the benefits delivered by IT work but all too often this fails to happen. Instead companies may rely on the original claimed benefits. But this might the result of optimistic thinking itself: “38% of businesses openly admit benefits are overstates in business cases in order to obtain project funding” – from a Cranfield Business School study I reported in October 2010.

And when one manager exaggerates benefit to get his project funded it creates an incentive for the next manager to exaggerate the benefit of the next project and so on. Benefit inflation cascades through the organisation.

This isn’t helped by vendors who claim benefits for their product which might not apply to a particular business or in a particular context. If company A used product B to save $C millions then surely company X can use product B to save about $C millions, right?

In our personal lives we have become accustomed to technology getting cheaper and more powerful and we expect the same in business. But consumer technology is paid for by millions of people buying millions of products while business technology may have one customer. Consequently we don’t appreciate the costs involved in creating new software.

Goal Displacement: Floorded contracts, projects and governance

In an effort to control IT, to maximise the return organizations frequently fixate on controlling the costs. They demand things like price, time and scope are decided in advance and fixed. Sometimes this is done internally – projects are approved and must be finished according to some set budget or date. And sometimes it is done by finding a supplier who will agree to fixed price/time work. (I’ve written about this before in Dear Customer on this blog and on Agile Connection/TechWell.)

The problem is, when you put these arrangements in place the objective becomes: meeting the fixed criteria, not delivering value. It is what psychologists and management students call Goal Displacement.

Options to reduce scope, time and cost are lost in the process. The things which were meant to be maximums, ceilings, become minimums, floors. Opportunities to reduce costs are lost to more features and more software.

Net effect

Taken individually, together and in combination with other forces we end up with a demand curve which is very high and inelastic. Changing the price of the technology doesn’t reduce the demand by much, Moore and Metcalfs Law mean there are always new opportunities, failure to capture original benefits mean they are still there for the taking, and lack of evaluation means nobody is counting, boys will always make arguments for more toys.

In the next instalment of this series I intend to look at what Agile can do about this and whether Agile makes things better or worse.

Having outlined the theory of supply and demand in the software market I want to spend this, and the next few posts examining what this analysis can teach us. I’m starting with the supply curve and will look at the demand curve later.

First I want to consider what happens to a software system over time if nothing else is done. (Consider this as a small detour before we consider Agile.)

Over time changes to software systems will add code, this means there is more code to maintain, more code for engineers to understand, more code for testers to test. Internally code quality tends to decay, some people call it software entropy.

Engineers leave the team and new engineers (might) arrive thereby loosing knowledge of how the system works. At least the knowledge is diluted. And if maintenance is outsourced or offshore such knowledge may be destroyed altogether. The new, cheaper, outsourced developers need to relearn.

Meanwhile the business environment seldom becomes any simpler. New business rules get added one place or another – increasing complexity. New systems need to be connected – either as data sources or destinations – which again increase complexity.

This all gives rise to the Big Ball of Mud pattern.

Finally technology changes. The operating system and databases which form the foundations of the system may be upgraded or replaced. In the worse cases the language the system is written in dies.

The net result of all of this is that the system becomes harder to maintain. Harder translates to more time and more time translates to more money. Thus the system becomes more expensive to maintain. Consequently the supply curve for the system moves to the left. For the same price we can now get less software.

Consider the graph below. We have the Supply of Software (Ss) and Demand for Software (Ds) curves as before. Now we add the Supply of Software over Time (Sst) curve. Over time, of nothing is done, the curve itself moves, all other things being equal we do not move to a new position on the curve. We get an entirely new curve.

Now it is possible to fight against this trend. And certainly there are plenty of tool vendors out there who will happily sell you a tools which they claim will move the curve in the opposite direction. (In economics terms we break the “all other things being equal” assumption.”)

But either way: unless you fight it things will get more expensive for the same quantity of software – or you get less software for the same money.

(Whoops, I’m publishing a revision of this blog entry – something I don’t normally do. I realised after publication that I had mislabelled the graphs – they were correct but did not follow economists convention hence this revision. The core argument still stands although I have had to modify the reasoning slightly. Sorry.)

I’m proud to say I am, or at least was, a Software Engineer. I’m proud to say that I’m a third generation Engineer. My farther and grandfather got their hands dirty with the oily sort of engineering. But I’m also a frustrated Economist. Economics forms part of both of my degrees and on a couple of occasionsI’ve been temped to become a real Economist.

So I am always disappointed when someone talks about Software Economics. I have yet to find a serious study of software economics. Most of the studies, books and papers about “software economics” would be better called “software accounting.” They are about apportioning costs and adding up numbers. Economics is not Accountancy.

I have long believed software engineers and their managers would benefit from a better understanding of economics. In fact many of my clients will be used to hearing me refer to the supply side (development) and demand side (requirements, analysis, etc.).

I find it very useful to think of software development in supply and demand terms and use the tools of economists to understand what is happening. Usually thinking like this shows there supply and demand to be mismatched.

(This blog post that has been a long time coming. Several previous attempts have floundered because there is so much I want to say about software economics – and thinking in supply and demand terms. So while there might be little in this post which is earth shattering – especially to those of you who have studied economics – later posts will build on these ideas and analysis.)

Lets start with the fundamental problem which economic theory addresses: the allocation of scarce resources. After all we have scarce resources in software development so lets use an Economists tools: supply and demand.

Lets have a look at the supply curve:

The line marked Sn is a normal supply curve. When price is a bit greater than zero individuals will start supplying their services and companies will come into being to supply the market. As the price increases more individuals and companies will be attracted to the market and supply will increase, more resources are brought to play.

Note two things here:

1. Economists usually draw these curves for a product within a market. Here I am not examining the supply of multiple copies of a software product, rather I am examining the supply of software development capacity for a unique product. The marginal costs of supplying existing software are as close to zero as to be zero, the marginal costs of supplying new software products – i.e. the capacity to create a product – is significantly larger than zero and that is what we are examining here. In software terms we need to consider a software product, perhaps a single program, perhaps a “project” or a small system.
2. If you are not an economist don’t worry about the fact that these “curves” are straight lines, the basic analysis still holds, we hold actual curves for an advanced class.

The software supply curve – Ss – starts the same way. When price is zero nobody supplies – perhaps not entirely true if we consider open source but close enough. As price increases more resources are brought into play and supply increases.

Software development is not cheap. If you are buying software developer services they are rarely cheap. The only time you get “cheap” software is when you buy (some) off the shelf stuff. Consequently when the price is low there is no supply of custom software.

But – and this is the BIG BUT – supply raises very slowly. Two people might write twice as much code as one but when they work together the generalised form of Brook’s Law comes into play: adding people to a software development effort slows a team down. Therefore supply responds slowly to price increases – what economists call inelastic.

Now consider the demand for new software:

Here Dn represents demand for a normal product, when the price is high there is little demand for the product. As the price falls the demand increases because more people can afford it. (Remember an Economist’s definition of “Demand” is “A want with the ability to pay”. I may want a new iPad Air but if it is more than I can pay there is no demand – some might call this “latent demand.”)

For normal products, in time, the market will “clear” and balance will be reached when the curves meet:

At some price (Pn) the quantity supplied (Qn) will match that demanded. Now economists accept that this might not happen instantly – and there are various models for that – but in the scheme of things the time it takes for the market to settle is not a big issue. It will happen eventually.

Until that happens all sorts of problems will ensue, some people might make a lot of profit, some people might go without and some people might have to pay a non-financial price, e.g. queues will form. All good stuff we could consider but lets leave that for another day.

Returning to the demand for software. People and companies do not want software for its own sake, they want software because of what it allows them to do. What Economists call a “derived demand”.

The problem is, and the reason I’ve drawn the the Ds curve as highly inelastic is that we are faced with an ever increasing demand for software. Processor power doubles every 18 months to two years (Moore’s Law) which means the things our computers are capable of doing – with the right software – is forever expanding. When we get this right the benefits can be massive. Which increases the demand for software…

• How many readers wanted to read newspapers on their telephone three years ago?
• How many readers knew they needed to watch TV on an iPad five years ago?
• How many readers took digital pictures with their telephone and posted them online on a social network to instantly share them with friends and relatives seven years ago?

So lets look at the software supply and demand curves together:

For me, the problem in software economics:

• Demand is rampant but inelastic so as price rises it does not fall off very fast
• Supply is severely constrained and is very difficult to increase (particularly in the short run) so is also inelastic
• Consequently the market only clears at high prices and slowly: consequently markets take a long time to reach equilibrium, which in turn means we need to consider the tensions arising in a market which spends so much time out of equilibrium. Thus the time lags which normally only play a small part in classical economic analysis are more significant.

Each of these statements deserves further examination and explanation in their own right – future blogs, maybe.

More importantly, and also deserving of its own blog entry, is the question: what can be done to bring this market into equilibrium?

That is the question I find myself wrestling with on behalf of clients, and it is one I will return to.

My recent “#NoProjects – why projects don’t make sense” post stirred up a fair bit of discussion, both on the blog and especially on Twitter – check the #NoProjects hashtag. But, while that post railed against “projects” it didn’t say much about what we should do instead. Indeed as was pointed out it might be better to advocate and rally behind a positive alternative rather than negative idea.

So let me try and put in place the outline of how I see a #NoProjects world working.

(Strictly speaking it might be better to use the hashtag #BeyondProjects but a) #NoProjects has already built up momentum, b) #NoProjects actually describes what we are talking about quite well and c) #NoProjects has 4 fewer letters than #BeyondProjects.)

Stable teams: first off the teams that do the work need to be stable and they need to be kept together over the long run. Of course teams will sometimes loose people and they will sometimes expand. But over the long run we want team members who will live with their creation and grow with their creation.

Teams rooted in business: These stable teams needs to be based with their business users and customers. There should no divide between “the business” and “IT”. IT shouldn’t be hired guns who get called up when there is a problem.

Increasingly, as more and more everyday business depends on technology “the business” is a technology business. IT departments as we know them may cease to exist in time.

(Naturally there are some differences between Beyond Projects in a corporate IT environment and in a software vendor environment. I won’t go into the details here. But, on the whole it is corporate IT departments who have a project obsession and practice corporate psychopathy. Continuity, if not stability, is usual the norm for software vendors.)

Business as usual (BAU) is the source and destination: Everyday business, business as usual, is the source of requests for technology changes and the destination for such changes. Many of these changes will be small – incremental – but others will be big. The people who do the work will be involved with uncovering the need for change and in working with the change when it occurs.

The need for change management will decrease considerably because IT will no longer be inflicting change on unsuspecting users. Instead technologists will be bringing requested improvements to willing customers.

If you need change management then it too will be based in the business lines of work. So too will be testers and business analysts. In fact the role of business analyst will become far more important because they will be charged with work with real stakeholders and helping them find ways to improve the business processes, products and delivery. BAs will look more like internal consultants than the order takers they are sometime used as.

Continuous stream of value: the result of this model will be a continuous stream of value. Technologists will be delivering improvements to the systems regularly, every few weeks if not every day. The batch size for work request and delivery will be one.

Again, as more and more businesses become IT businesses – business which can only exist and compete through the use of IT and IS systems – the value delivered will be bottom line value allowing the business to compete and stay ahead of competitors.

Portfolio management: this is not to say that all lines of business – and technology teams – will exist for ever. Some lines will need be to closed. Some teams will need to survive with less technology. Conversely, work streams which are seeing growth and performing stronger than the wider company will benefit from more resources.

The activity of reviewing work streams, increasing capacity, reducing capacity, starting new initiatives and terminating existing ones needs to be pursued more rigorously than before.

True portfolio management needs to be practices on a regular basis – at least every three months. And it needs to look at value delivered, potential future value and costs. Portfolio management which looks at progress against plan, or measures success as “on time, budget and scope” has no place is the Beyond Projects world.

Like trains: trains run on schedules and so should software projects. When trains leave the station late and arrive late they are poor performers. Just the same goes for software work. We want work which begins on schedule and arrives on schedule.

When this happens co-ordination becomes simpler. The train schedules are known weeks and months in advance. Yes they change sometimes but not often. When you know the train you are going to get you arrange your journey around that.

When you have to take connecting trains you know how much time you need. You might allow a buffer or you might take a risk and run between platforms. How many of you readers have missed a train or a plane recently? Humans are very good at working to deadlines and very poor at estimate how long activities will take.

Co-ordinating multiple pieces of work between different teams becomes a process of putting the features/functionality you need from another team on their train. You inject work into the other team. And the receiving team will be able to tell you their schedule and whether the train you want to put a feature on is overloaded. They might even offer you a discount for going off-peak!

Project managers: obviously without projects there are no projects to manage. But does that mean the end for the people we call project managers?

I think not, there is still work to manage. I disagree with many in the Agile community – and particularly the Scrum community – because I don’t think all management work goes away. There might be less but I think those who argue “you don’t need managers” don’t really understand management work. (In truth most managers don’t either. Go and read Managing by Mintzberg then we can talk about what managers actually do.)

Many of the skills good project managers have will still be needed. They may take on another co-ordination role or they may become business-as-usual managers. Rather than having “Project Manager Call Centre Renewal Project” on their business card we might see “Call Centre Development Manager”.

Management roles will still exist but they will manage real work.

Goodbye matrix management: with projects gone and technologists based in the business much matrix management should disappear. Matrix management has never looked like a great idea, at best it is confusing at worst it is dysfunctional.

There will be more line management and there will be more need for business line managers to understand technology. Some of those surplus project managers will be well positioned to take up the new roles.

(Indeed I suspect that the proliferation of project managers in the last 20 years has much to do with the reduction of middle and junior managers that occurred in the 10 years before hand. First corporations cleared out their management ranks, then reinvented them under the cover of “projects.”)

There then are the basics of #NoProjects, Beyond Projects. I’m sure there is much more I could say and much more I will say in time.

If you have any opinions on this please add them below, I’d really like to hear what people think. And especially why you think I’m wrong, what have I missed? Go on, tell me!

Abi Crafter at Countersoft liked by “11 Myths and 2 Truths of Agile” blog entry so much he turned it into this Infographic. Thanks Abi!

(Some of you might have noticed that a reworked version of that blog appeared on Agile Connection as Top Twelve Myths of Agile Development.)

By the way, notice I’ve linked to Abi and his company Countersoft. Abi did something I like, Abi gave me something, I’m more than happy to give him a link for that. I don’t necessarily endorse his tool but I might well look at it.

I regularly decline – or plain ignore – requests to review particular products in this blog. Similarly I mark comments which don’t really say anything but include a link as spam. Unfortunately most such comments seem to come from Indian companies which reduces my opinion of the Indian software industry.