Agile

This isn’t a happy story, it has no happy ending, I suffered personally, its personal but I want to share.

Its about trying to solve a problem with the fashionable solution rather than rolling back the last fashionable solution you applied which created the problem to start with…

A long time ago, well, the best part of 10 years ago, in a town far far away (actually the other side of London) I went to help a small part of a very large company “become agile”.

The managers wanted to be “agile” and insisted they knew what it meant and what the implications were. So my job was merely to help the change resistant workers change.

Now this company, like so many others, had decided that coding was expensive and should be done in a far away place, and this time I do really mean far away, far enough away to be cheaper.

The system in question was big, and old, over 20 years old, and millions of lines of Cobol. Fortunately the company still employed most of the people who had built the system over 20 years. Unfortunately they were to forbidden to code. They were too expensive for that. So the far away cheap people coded. I forget the job title the old coders were given, maybe it was SME but I thought of them as Architects – although Systems Analyst might have been a better title.

The far away coders, employed by a “partner” (outsourcer) were young, they lack programming experience. I suspect they had learned Java at college and been given a Cobol boot camp when they joined the partner. From what I could tell they were quite capable of making a code change at the function level but… anything involving system structure, multiple functions or some of the larger (too big) functions required help.

They needed to talk to the architects.

To avoid this the big company had the architects write “design” documents. But still the coders needed regular conference calls.

These coders also had a habit of changing, after 18 months on the contract the outsourcer moved them on. But then, many of them didn’t last that long; many left of their own accord before then. Consequently, just as one of the coders got to the point off properly understanding Cobol and the system they were gone.

To make sure the right work was done the big company had Business Analysts detail the need in big documents. Lots more to read.

Of course all this required a lot of testing for the partner, plus another partner (in a third location), and internal “user acceptance testing” (in a fourth location.) Implicitly the process and managers accepted lots of failure and expected testing to generate a lot of (re)work.

Now something else happens when you use an outsourcer and so many people: you need more management, at both the client and supplier.

All this complexity (not to mention cost) piled up and made them unresponsive. Hence they wanted to be Agile! That’s why I was there.

Every so often one of the projects would get into a real mess and the architects would be allow to take over and code. When this happened it was completed in a few weeks.

Most of the offshore efforts took months.

Yes the company was saving money on programmers but…

• They were spending more on business analysts and architects
• They were spending a lot on test
• They were spending a fortune on managing

But most of all they were paying in late deliveries, new products not in the market, delayed cost reduction initiatives and so on. Plus they were pained by poor completing date forecasts.

And it was getting worse.

So of course Agile was the answer.

But the problem the company faced wasn’t one Agile sets out to solve. The problem was one of knowledge: the company had the knowledge but wasn’t using it effectively. While they didn’t use the knowledge they were losing the knowledge. Knowledge couldn’t just be moved from the heads of people in London to the heads of people in a far away place.

The company ignored knowledge, or at least thought it could be written down. They saw the problem as expensive typists.

And me?

I diagnosed the problem as managers failing to understand, hence I wanted to spend time talking to managers. But they said they already understood and Agile was the answer so my wanted to talk to them was not what they wanted.

I crashed and burned.

In The Living Company (one of my favourite business books, highly recommended) Arie de Geus describes the Mayor of Rotterdam Problem, it goes like this.

Imagine you have perfect foresight. You can see the future accurately. It is 1920 and you visit the Mayor of Rotterdam. You tell the mayor about how the events of the next 25 years, specifically about the destruction Rotterdam will suffer during 1940-1945.

de Geus asks: What is it reasonable to expect the major to do?

Think about this for a moment.

de Geus draws the following lesson:

The future cannot be predicted. But, even if we could, we would not dare act on the prediction.

So, I might say “You project is a train crash and destined to fail.” But if you have invested $100million do you dare to act on my information?

I draw another lesson from this story:

There is nothing we can do to change the future, even when one has considerable power.

Most people regard a Mayor as a powerful figure. But what could the mayor do in 1920 to stop Great Depression, the rise of Fascism and the pending war?

And thats assuming people believe you.

Why bring this up now?

Well… increasingly some of our most established companies face exactly this dilemma. The raise of digital business, coupled with disruptive innovation, agile working and new management models place CEOs of some of the largest business in exactly the Mayor’s position.

If you are the CEO of a business with a few million customers and you can see how digital disruptors will steal your business over the next 10 year d you dare act on the information?

Acting now might mean destroying profits for years to come, acting now might mean laying off thousands of workers and dropping millions of customer. Success if very very far from guaranteed. And thats assuming you can persuade people you were right.

There is one company where this has already happened: Kodak.

A recent MIT Sloan Review piece Willy Shih gave an account of Kodak which differs substantially to the commonly received view. Kodak is regularly held up as an example of a company that failed to recognise how digital would change their world and as a result went bankrupt. But…

Shih describes it differently, he says that Kodak managers did understand what was happening – after all they invented digital photography. The problem was: Kodak couldn’t change direction because of their ecosystem and the economics of their business.

Part of the Kodak problem was that had Kodak publicly acknowledged the decline in their traditional market they would have accelerated the change.

Kodak did try use their legacy business as a cash cow did enter the digital market this left a big problem: how do you manage, and motivate, people in the legacy business who will never make the transition?

The question to ask right now is:
• What other Kodaks are there out there? – unable to change direction because of their legacy.
• Which CEOs are Mayors of Rotterdam? – unable to act on accurate predictions of the future.

If you are the CEO of a Kodak an alternative strategy might be ignore the future, crank up the machine and do even more of the same – whatever your current business model is. Aim in the short run to minimise risk while maximising your personal return. Plan to jump out of the business yourself before it goes bang or just plan to blame the failure to predict the future.

This is the first blog post of 2017 so something a little different, a little more personal, a little speculative…

I’ve written before about my dyslexia (Dyslexia makes me stronger) and I’ve also noted that what is good for dyslexics what helps them learn, is usually good for the wider population too; i.e. if you make things dyslexic friendly non-dyslexics also find things easier.

A corollary for that might be: reading which many find a little difficult dyslexics find very difficult.

Now I’ve stumbled on something that has got me thinking, let me explain…

Something really unusual has happened to me for the second time within a year. I find myself working regularly inside a big corporate organization, specifically, I’m working with the same company regularly enough for them to have given me a company e-mail address, which means I need to log into corporation Outlook. It also means the corporation is sending me lots of internal stuff and people are communicating with me via e-mail and by referring me anonymous webpages and and and…

OK, for most of you this is normal, it was once too for me but I’ve spent most of the last 10 years working either with companies which aren’t big enough to do this sort of stuff or, I was wasn’t working with companies enough to get into this position.

I’ve noticed that I’m ignoring much of this material, I sometimes feel like I’m drowning. And I notice that I only skim read (at best) much of the stuff that is thrown at me. Its overwhelming and I have a very low tolerance level, perhaps because…

Above all what I notice is I feel troubled inside when confronted with pages and pages of corporate text. I even feel like crying. I switch off.

Little companies, growing companies, have this stuff too but much less of it. And usually you can find someone to ask. In the big company not only is there lots and lots but asking is hard – try calling IT support and you find out how much these people don’t want you to.

I’ve had this before but only now have I made a connection with my dyslexia because I’ve remembered where I’ve had these feeling before: at school and at University.

I realise that confronted with this mass of text I feel like I did when confronted with books at school.

It is not about reading, I love reading, I am reading several books for enjoyment at the moment and I continue to read the news papers and websites.

But confronted with masses of material I need to read, which not reading makes me in some way a failure, school text books and corporate webpages I feel inadequate, I feel intimidated, I want to cry and a bit of me feels like I’m at school.

It occurs to me that as companies rely more on automated systems, documented processes and policies, and they move away from conversation and human interaction then those of us who are dyslexic are at a disadvantage.

Sure I can read the words but I can’t process them.

Ironically, as I mentioned when I blogged about my dyslexia before, I think my dyslexia is an asset, I think dyslexia made me a better programmer. But equally dyslexia makes it harder for me to operate in a modern corporate environment. So by documenting themselves big corporates could be deterring or disabling the very people they want.

(I say disabling very specifically: Dyslexia is a socially constructed disability, it is only a disability because of the society we have created, in earlier, pre-text, societies it may be considered an advantage.)

Or is it just me?
Do other dyslexics feel this?
And what about the mass of non-dyslexics? – maybe they feel the same

This blog contains a lot of conjecture, free thinking if you prefer, I write to find out if others think this way – please comment.

The end of the year is almost here, a year which has seen two mini-series in this blog: Management and Agile ERP. Well, I was catching up with my reading backlog a little something about management by Tim O’Reilly caught my eye.

Writing about the future of management in MIT Sloan Management review O’Reilly makes he following suggestion:

“a large part of the work of these companies [Google, Facebook, Amazon.com]— delivering search results, news and information, social network status updates, and relevant products for purchase — is performed by software programs and algorithms. These programs are the workers, and the human software developers who create them are their managers.”

O’Reilly’s argument is that the traditional management role was about organizing workers. Now the workers are often electronic systems and their work is governed by programs then the programmers are in effect the managers. Programmers – and others in the development arena – instruct the workers using algorithms and code.

In my management series one of the arguments I made was: no matter how self-organizing and managing teams are there is a residual amount of “management work”. In a team without (or just fewer) managers this work is spread around. Among the most obvious examples are the NCO managers (Scrum Masters, Technical Leads, etc.) teams often have. And, because management work is spread around among more people there are more people who need to understand management and business.

O’Reilly’s argument leads you to the same place.

If we assume more management and business decisions are being taken at-the-code-face then more programmers (and other technical staff) need to learn about management and business. Paradoxically, firms should have fewer managers but more management training.

Some aren’t going to like that, some want to take an engineering view of everything.

I suspect there is a corollary: if technical staff need to spend more time considering management and business issues then there is less time for coding, less space for technologies which require really deep understanding (e.g. C++) and fewer opportunities for those who only want to code and ignore all the management, business and political stuff.

This blog has been a lot quieter in the last couple of months than normal. My energy has been going elsewhere.

If you would like to catch up with latest thinking – particularly #NoProjects – then please check out my latest work-in-progress book, #NoProjects: Correcting Project Myopia.

Early versions of this are available to buy and download now, most of my free time between now and the new year is directed at advancing this book (after one or two false starts.)

Come January my “spare time” time will be absorbed by reviewing the Agile on the Beach submissions (see the call for speakers). Which also helps explain the other reason for the quiet blog…

My summer programming endeavour is a thing I call Mimas – named after one of Saturn’s moons. It is a conference submission and review system for Agile on the Beach. I really should write a blog about the things I learned writing Mimas…

Like all software endeavours there is a lot more I could do. And like so many programmers I wonder…. could this be useful to other people?

(This is refactored version of my earlier blog post Technical Liabilities and not Technical Debt – its shorter and contains few of the references to banking and finance.)

Wherever I go I hear developers talk about “Technical debt”. Unfortunately the expression isn’t used in the way Ward Cunningham originally intended it.

The time has come to retire the technical debt metaphor, in its place we should talk about Technical Liabilities. This is a small change in language with big implications.

The big big problem with talking about “debt” is that debt is not necessarily a bad thing; debt is often good, particularly in a business context. This leads to misunderstanding between engineers and non-engineers and can even lead engineers into poor habbits.

To many people in business – and particularly bankers – debt is good. Debt allows you to grow business and improve earnings. When an engineer says “This will add technical debt” the engineer thinks they are warning about something bad, but to a non-engineer business person, debt is often the preferred form of funding. So buying new capability with (technical) debt sounds like a good deal.

Let me go a bit deeper…

The essential part of Wards explanation was this:

“there were plenty of cases where people would rush software out the door and learn things but never put that learning back into the program, and that by analogy was borrowing money thinking that you never had to pay it back.”

However the metaphor has been widely interpreted as something you might intentionally do. “Debt thinking” goes like this:

“We are (time and money) poor, but we can borrow from tomorrow’s riches, rather than write good code today we can save time (and money) by just getting something that works and later (when we are rich) we can come back and do all that good stuff which we should do. (And by the way, if it turns out that our initiative fails then we have saved effort and money.)”

This thinking is ripe both among developers and those who employ developers.

This kind of debt thinking is, perhaps, similar to a mortgage:

“I don’t have the money to buy a house right now but if I borrow the money from a bank I can pay back the money over time with the money I save on rent.”

Yet many engineers are the first to bemoan the fact that despite the best intentions they never get to pay back the debt. Instead they merely service the debt, that is to say, they (at best) pay the interest but never the principle debt remains. Indeed all too often the payments are not even sufficient to pay the interest which gets added to the principle.

With any debt the important question is: can you afford to payments? The bigger the debt, the higher the interest rate the greater the amount of todays resources (time, money, effort) that needs to be devoted to servicing the debt. Yesterday you borrowed so you could buy something beyond your resources, today you must devote some of your resources to pay those who provided the loan.

Over time “technical debt” massively hinders the ability of organizations to change both the software and themselves.

Technical debt is less more like a pay-day-loan than a mortgage: a loan you take when you have no choice, the high interest rate can be crippling, payments eat into your ability to do any useful work.

Only those who are desperate and/or financially naive would accept a pay-day-loan. Let me suggest those who willingly take on technical debt similarly desperate or technically naive.

This logic is why I say: There is no such thing as quick and dirty, only dirty and slow.

So, talking about technical debt is loaded with problems, when is a debt good? when is it bad? what is the interest rate? do those who advocate incurring debt know what they are saying? And do you have a repayment plan?

Simply changing our language from Technical Debt to Technical Liability removes these problems.

Liability is something neither business and technical folk consider good. If I look at my dictionary it tells me:

“Liability:

1. Subject to an obligation

2. Exposed to a possible risk

3. Responsible for

… and some more…”

These attributes, both in a technical setting and non-technical context, can be agreed by everyone.

Businesses have to list liabilities on their balance sheet and reducing the liabilities produces a more attractive balance sheet. (Debt on the other hand is listed as an asset on bank balance sheets, and even Apple sometimes chooses to issue debt rather than use capital.)

Let us let “Technical debt” go back to Wards original meaning. Lets talk about Technical Liabilities in our systems.

Technical liabilities that cost us because they create obligations – obligation which slow us down, obligations which must be repaid; and they create risks, we don’t know when an obligation is going to bite us.

I’ve had a very frustrating morning, and it all comes down the failure of bank IT systems – particularly a software release failure at HSBC. Gee, their systems are flaky.

I was trying to buy tickets at Eurostar.com for my trip to Lean Kanban France in a few weeks. First I tried to use my American Express card (I hate it, I only use it for work expenses) and I got the now traditional “Service not available -Unhandled Java exception …gavity.exceptions…” at the final payment stage – you know the pocky little box you have to put three letters into? – I’ve had this error before elsewhere and I think its because I block pop-ups on Firefox. Annoying but nobody at AMEX care so that is that.

So I repeated the transaction using Chrome and my Mastercard – issued by HSBC but branded John Lewis.

And again at the final security page – Arcot’s 3D Verified by Mastercard bit – the page errored. Not a horrible unhandled Java exception but an error just the same.

So I called Eurostar – because something similar happened a few weeks ago and they took my booking by phone. Only this time they insisted that it was the banks fault – like two banks fail? Didn’t anyone ever tell them “Select isn’t broken”?

This time they insisted the problem was with the banks, to make a booking over the phone would cost me £10 – something they waived last month. I even went as far as to speak to a supervisor who continued to insist he wouldn’t waive the fee – which I know from last month he has the ability to do.

Instead he told me it was probably my Mac that was the problem.

Seriously, he told me to use a phone instead. 10 years ago saying “don’t use a Mac” might have been acceptable but now?

Bad customer care on both points.

I gave up, I fell back on Safari and another card altogether to make the booking.

After several tries and a long wait I finally got to speak to HSBC Mastercard and this is what I discovered…

• HSBC Mastercard don’t currently use Arcot security but intended to start on 9 November
• But the release has been pulled because of technical difficulties, they don’t know when it will occur
• But the message had gone out to retails that they were to start (remember, today is 1 November so they change wasn’t scheduled for 7 more days)

This also explains a couple of other recent online purchase problems I’ve had with this card.

This implies that either:

• When a seller system (e.g. Eurostar) calls the HSBC system (“is_using_arcot_security()”) the HSBC system is replying “True” when it should say “False”

Or

• There is a piece of system configuration that HSBC publishes to sellers (“<is_using_arcot_security>True</is_using_arcot_security>”) which has been pushed out prior to the update and that a) this config has been released too early (its not 9 November yet!) or b) else has not been rolled back.

Either way HSBC have bungled a software release. Of course, it might not be HSBC, it could be Arcot, or rather their current owner CA – who were fingered in the RBS outage a couple of years ago.

Which also might explain why I spent over 30 minutes on hold to the call centre (plenty of failure demand).

And we still haven’t discussed by HSBC pulled their release in the first place, clearly something went wrong to cancel the 9 November release – my money is on low quality.

When will these companies learn: high quality in software is faster and cheaper.

Continuous delivery is about risk reduction.

(Well, they are probably mainframe systems so maybe I should be a little forgiving on the second point.)

Now someone at HSBC will probably get into trouble for sharing so much information with me. Well its a good thing they did because I wasn’t going to hang up until they did. Thank you, you did the right thing – unlike Eurostar.

I write this both to relieve my frustration and in the hope that it might make HSBC (and AMEX) do something about this.

(I should also point out: none of these companies have ever hired me. If they had I’d probably keep quite because I avoid blogging to specifically about clients.)

But you know the really worrying thing about all this:

Our modern banking system is based on the assumption that the Government will always bail out a failing bank – the fear of the payments system failing is so great governments will do anything, thats why we rescued RBS.

Now it seems banks are failing to maintain the payments system anyway. The banks are collecting technical debt the way they used to collect sub-prime. Central bankers throwing money around won’t work.

Who will save us this time?

To say last October’s post “Software has diseconomies of scale – not economies of scale” has been my most popular post ever is something of an understatement! It has been read tens of thousands of times after being picked up by some very popular publications and newsletters.

When I wrote the piece I was writing largely on intuition. That software development experiences diseconomies of scale started as hunch, gradually I could see more evidence but when I published last October I didn’t have anything that would stand up to academic scrutiny – not that I was aiming for a peer reviewed publication.

Since then I’ve become aware of several pieces of research which do show the same phenomenon and do stand up to academic standards, plus I’ve become aware of several people who have made the claim earlier than myself.

So, for those who want to dig into this subject some more let me record the evidence.

“From Aristotle to Ringelmann” (or better an open download of a preprint version): in this paper Ingo Scholtes, Pavlin Mavrodiev and Frank Schweitzer at ETH in Zurich examined open source code bases and team sizes. In their abstract they conclude:

“Our findings confirm the negative relation between team size and productivity previously suggested by empirical software engineering research, thus providing quantitative evidence for the presence of a strong Ringelmann effect.”

The term “Ringelmann effect” is new to me but describes what is seen in software teams – and elsewhere:

“The Ringelmann effect is the tendency for individual members of a group to become increasingly less productive as the size of         their group increases” Wikipedia

Another academic paper is Economies and diseconomies of scale in software development by Comstock, Jiang and Davies. Unfortunately, despite the title this paper focused on constructing a model for effort and value in software development than drawing conclusions about economies and diseconomies. So yes it is about software economics but not a lot about economies and diseconomies, more software accounting. The authors offer no conclusion of their own in that respect.

That said there is some evidence here. The authors note several earlier studies which had mixed results, some showed diseconomies of scale but some also showed economies. They also point out that at least two of the more established forecasting models (from traditional backgrounds), the Putnam and CJD models, assumed diseconomies of scale (I’d never heard of either of these models before!).

It a shame that some of this knowledge has existed in some academic circles for several years, it seems to be another example of how academic papers hidden behind paywalls prevent the spread of useful knowledge. (My initial attempts to get this paper met with paywalls but in the link above I tracked down a downloadable version.)

There is an important observation built into this paper, almost by accident: the optimal team size is not fixed, it will depend on the size of the undertaking and the duration of the effort. I am often asked “What size should a team be?” and several Scrum advocates have stated “the team should be 7 people plus or minus 2.” Clearly the right size for any undertaking will depend on multiple factors.

Away from the academic world there is some more support for diseconomies from other observations.

In a blog post about two and a half years before mine Jesus Gil Hernandez made exactly the same point, “Diseconomies of Scale in Software Development.” Jesus also makes the point that if we want to forecast and plan large software initiatives we need to take such diseconomies into account.

“Diseconomies of Scale and Lines of Code” is blog post from further back, 10 years ago, from Jeff Atwood’s Coding Horrors blog. Jeff also suggests diseconomies of scale but then his blog goes on to focus on lines of code and why its a poor metric (and it is). Jeff also points out that Steve McConnell discussed diseconomies of scale in his 2006 book, Software Estimation.

What is becoming clear is that the possibility of diseconomies of scale in software development have been known about for a long time and my argument has some validity. What we need now is more research to get to the bottom of this…

Unfortunately I don’t have the time or resources to get do such research, I’m happy to collaborate, any academic or PhD student out there fancy picking this up as a research question?

The story so far…

1. Agile & ERP?
2. The Good News about Agile & ERP
3. ERP Culture v. Agile Culture
4. Th Bad News about Agile & ERP        5.        

Where does this leave us?

For a modern software developer, who understand agile and is well versed in the current ways of working encountering an ERP system is like stepping back 20 or perhaps 30 years.

The mindset of in ERP is more akin to the software development mindset when I started working in IT. It is as if the last 20 years have passed them by – the technology is dated, the agile revolution hasn’t happened, users should be grateful for what they are given and the vendor is all powerful.

The deal with ERP is: the ERP system does so very much, why reinvent the wheel? Writing software is expensive (and fraught with failure) so why not use one we made earlier?

It’s the software reuse argument written BIG.

The downside is: the ERP is all encompassing, you need to use particular technologies and these technologies haven’t really advanced in 20 years. The skills are in short supply. Most importantly: all the ways we have found to make software development more effective are absent.

Let us be very clear: there are massive benefits from using an off-the-shelf system. ERP systems are big, really big, and complex, general ledgers, multi-country/multi-currency payroll, etc. etc. Why would you want to rebuild this stuff?

But: there are also massive costs.

The ultimate question is: are the benefits greater than the costs?

Its worth repeating what Capers Jones says at this point:

“As a rule of thumb, if COTS [Common Of The Shelf] packages require modification by more than 25% it may be less expensive in the long run to build the same kind of application.” Applied Software Measurement, Jones, 2008

Unfortunately he doesn’t elaborate on how the 25% is calculated, he probably means function points. He continues:

“Package modification is a high-risk activity with a significant chance of failure and disaster. Not only is productivity normally low for modifying purchased packages, but if the vendors are reluctant to provide support, modification can easily be more expensive than customer development.” Applied Software Measurement, Jones, 2008

When answering this question please remember licenses for ERP systems are far from cheap, and they reoccure. Plus, vendors may well charge extra fees to support parts of the system which have been changed or to ensure backward compatibility.

Once installed clients can be very, very, reluctant to update ERP system. This is hardly surprising once you realise that “configuring” a system like SAP may mean having developers change SAP code in the base system. If SAP release a new version they may have changed that code, you are lucky if you can get away with a merge from hell.

When you buy an ERP system you buy an existing legacy system. You then attempt to configure it to be your legacy system. (See my Peugeot post.)

This sounds like a bad deal but its a deal that makes sense for a lot of companies and has done for 20 or 30 years.

Now, here is my conclusion:

20 years ago buying a legacy ERP system and configuring it to be your own legacy may well have made financial sense.

But in the last 20 years technology has advanced, not just the CPUs but the programming languages, the programming tools and even the programming processes.

Buying a legacy ERP system today means buying into a legacy development process with legacy tools. You step back 20 years.

In the last 20 years the tools and techniques have got a lot better. While buying an ERP system may look like a sensible decision you also shackle yourself to 20 year old tools and processes.

The tools and processes have improved so much in the last 20 years that the equation has changed. Modern approaches (i.e. agile and test driven) using modern technology means the cost and time gap between a) developing your own system using modern techniques (including microservices, existing libraries, open source, etc.) and b) choosing legacy ERP system (reoccuring license fees plus configuration costs) is far smaller than it was.

I conjecture that the gap is getting smaller as we speak.

Since ERP systems last decades the 20 year old ERP system you are buying today might be expected to last another 20 years. Who knows how much better things will get in two decades.

This also means there is a market opportunity for “modern ERP” systems which use modern technologies (Java, JavaScript, Micro-services, etc.) to support modern working practices. Microsoft, Oracle and SAP are all trying to move forward but the legacy is very profitable.

Finally, one might ask: given all these problems, and the changes in technology why do companies buy these systems?

Nobody ever asked my opinion on this so I can’t really say but let me float an idea.

Only companies of a certain size, say $1 billion revenue can afford these systems. When a company reaches that size they are expected to buy such a system – having an expensive system is a sign that your company has arrived.

These systems are driven by the finance departments, the CFO. As companies get big they need such systems to control the company. But also, the CFO of a $1bn company wouldn’t look serious if he didn’t have one – after all, all the other $1bn companies have an ERP system so why not yours?

If you are the CFO of a $1bn company without an ERP system and something goes wrong it will look irresponsible. Having an expensive ERP system allows the CFO to say “We followed best practice and bought Oracle”. It reduces the risk of the CFO losing their job.

And buying an ERP system allows the CFO to claim more resources for his department.

And it allows the CFO to say at his next job interview “I introduced an ERP system to a $1bn company.”

So the next time you get appointed CFO of a company which as just broken the $1bn revenue barrier and you find there is no ERP system in place what are you going to do?

Engineers might think: I’ll set up a small team and start growing our own ERP system using modern technology.

To the engineering mind that approach stands the best chance of working and is financially the best route.

But it is also high risk simply because nobody else does it.

Everyone else says: “I’ll call SAP, I’ll spend a lot of money with them, Accenture, Cap-Gemini and CSC. When it goes wrong I can blame them. After all, thats what everyone else does so doing anything different would be high risk.”

The story so far…

• Part 1: Agile & ERP introduced
• Part 2: Good News: Agile can work with ERP!
• Part 3: Culture: How Agile & ERP cultures conflict

In this mini-series I’m discussing Agile development on ERP systems. If you are doing this and you’ve just Googled “Agile and ERP” there is a good chance you’ve found this blog because… bad news… there isn’t a lot of experience of doing Agile in an ERP environment.

Sure people have done it, sure some people have written about it in blog and journals but I said: “some”. Some means “not many”. One of the problems of doing Agile in an ERP setting is that there is a lack of knowledge and experience. People have done this, I’ve “done” it several times myself but… not as many people have done Agile in ERP as have done, say, Agile in Java Banking or some other domain. And very few of them have written their experience up. And even fewer have good stories to tell.

Partly it comes down to a question of “What do you mean by Agile?”. Sure you can do Agile in ERP if you define “Agile” as those “agile” practices which work in the ERP setting. On the other hand, when you do find the examples that are out there you will find many practices missing, specifically the technical practices.

Almost all the reports of Agile in ERP you will find are about process – much like my previous post . Doing ERP with test driven development, acceptance test driven development, simple design, emergent design and refactoring is almost unheard of.

Sure there are a few blogs out there which discuss it a bit but there is not much. I briefed a recruiter to find someone with (A)TDD and ERP experience. He came back a couple of weeks later and said “Its like looking for Unicorns, these people don’t exist.”

In part this is because ERP systems are sold on the promise of EVERYTHING. Their design dates back decades so they are monolithic and the culture that surrounds them is 20 or 30 years behind the current thinking.

My most recent run at this problem was the best yet but here is the bad news: with an ERP system once you move away from the Agile processes practices and start to look at the Agile technical practices things get very difficult.

The cultural problems (corporate IT and ERP specific) don’t help but the technology is the major obstacle. Lets start with testing…

Some test frameworks do exist (e.g. ABAP Unit and X++ Unit) but they are limited and, as with other languages, only a fraction of developers use them, and only a few of them do it properly.

One of the problems is an old one: Mocking and Stubbing. But in an ERP system the Mocking and Stubbing problems are far far worse because the system is provides so much already you need to do more mocking, the system is pervasive, there is just more system to mocks and stub.

Since ABAP is effectively Cobol you can forget reflection, annotation and polymorphic substitution. Heck, I don’t think it even supports macros!

Part of this problem is because of the architecture of ERP systems. The architecture is decades old, its decidedly not Object Oriented, as closer to working in embedded C than Java.

Another problem is because of lot of the system isn’t programmed in text. Remember I said some of the programming was table based? And some was pointy-clicky? That stuff is super difficult to test because it exists in a different paradigm. In some cases even when the “configuration” exists as a code like fragment it is considered data by the system and cannot be extracted.

In once case we exported “the code” only to find it wasn’t coded in ASCII. And if you’ve exported it you might not be able to import it again.

The designers of these systems actually expect their production code to be changed, tinkered with, messed with, hacked, by those configuring the system. Let me repeat: production code from the vendor may be changed as part of a configuration effort. Imagine that when you wanted to write a macro in Microsoft Word VBA you were allowed access to the underlying Microsoft Word code.

These systems are the exact opposite of what John Vlassides would advocate:

“the hallmark of good code is that it is changed by adding to it, not hacking it” John Vlassides, The C++ Report

Changing code by adding to it is a new idea in the ERP world, both for those “configuring” it and those building the product.

But the problems of Automated Unit Testing TDD pale into insignificance when compared to the problems of doing automated Acceptance Test Driven Development (ATDD).

Because ERP systems are monoliths it is hard, if not impossible, to find a way in to add ATDD style tests. These systems don’t just predate microservices they predate components, COM, CORBA, DLLs, shared libraries and even static libraries.

I’ve seen automated test tools work at the highest level, the UI level. I’ve heard Quick Test Pro can run against the UI on Windows interfaces and I’ve seen Selenium run against a Microsoft front end, I’ve even seen Selenium execute a Given-When-Then test but…

This is all at the UI level.

It seems impossible to set up tests anywhere between the very lowest level and the very highest.

If you are lucky the ERP system exposes some services or interfaces you can connect to, for example Rocket U-2 lets .NET code connect to it in places. But such interfaces are not always there, or if they are they are undocumented and subject to change. So most higher level testing needs to be at the UI level….

Since its at the UI level the tests are slow, fragile, slow, difficult to create in advance and slow; the whole system needs to brought up to run them so if something is found its a big pond to fish in.

Since the whole system is being powered up for a test you need test data and this is a problem. If you are very lucky you might be able to get to a relational database and insert your own data, if you are unlucky you won’t be able to access the database or you’ll find its an old propriety hierarchical database and you will need to load lots of data.

And did I mention its slow?

It is not just ATDD tests which are slow the build is slow too. Hours not minutes.

Building might not be much like you know it, I once had to explain source code control to an SAP team. Because all configuration is considered “data” the very idea of source code control can be strange – you are dependent on the vendor actually supporting source control which at one point SAP did not.

Sometimes actually saving “configuration” and loading it into another instance of the system isn’t supported to start with. You may need to build that plumbing.

And remember, these ideas may be completely foreign to the people “configuring” the system. These people are HR or supply chain experts who happened to pick up a bit of ERP and found it pays well. They don’t understand many software engineering concepts, they may not particularly want to learn them and their employers may not want them to either.

As I said before: these systems are large monoliths.

Many of the tools and techniques which software developers have created to manage large systems, to modularise them, to control them, to isolate parts of them, to allow change at different rates are simply absent. ERP systems are not just monoliths the are monopolies, they are intended to form the central nervous system of an organization and run the organization, they are a single thing.

That is their nature, their culture, their technology, even their raison d’être.

That might seem strong but think about it… Senior managers buy these systems to provide insight and control over large organizations. They want the large organization to function as a single entity, they don’t want duplication, they don’t want variation, they want a monolith and that is what the ERP sales people sell them.

So while you might be able to bring some Agile to ERP implementation there is a very deep seated conflict between the agile means and the decidedly non-agile end.