Agile

In general IT and Human Resource departments do not get on well together.  To be fair to IT I’m not sure any department in a company gets on brilliantly with HR but IT seems to get one worse than most.  I once worked for a company where one senior manager claimed “I only work for small companies, as soon as they get big enough to have an HR director I leave.”  True to his word within a year of the firm appointing a permanent HR director he was gone.

When you look at it the two departments really are opposites.

• HR is generally staffed by people who use people skills, they do touchy-feely, they do form filling and the do structure.  HR comes along late in a company’s life; nobody founds a firm by starting with an HR group.  HR is also, if we believe stereotypes, largely staffed by women.
• IT – and really I’m talking specifically about software development groups – are the reverse.  They are staffed by people who don’t know the meaning of the word touch, generally dislike form filling (why fill in a form when you can create an electronic systems?) and are creative.  Software developers are often the first people in a start-up company.  And on the whole the teams are male dominated.

To be fair to HR – and I have been known to stand up for them in the past – they do have good intentions.  HR departments can be a force for good in a company and they do work that does need doing in big companies.  On the whole IT people do not appreciate this.  Which just goes to make the conflict so much worse.

I’ve commented on this subject before.  Then I gave two reasons: firstly that HR groups just do not understand IT and secondly that you actually need damn good HR people if they are to benefit IT. 

So why are software developers so different form the other groups HR works with?

Well I think there are some characteristics of development teams which alienates them from HR people.  Other groups and individuals have these characteristics but in software development they are more common and more pronounced:

• Technical skills are to the fore and social skills can be non-existent.  I think the developers of tomorrow need for more social skills than those of yesterday but still technical skills are key.  This is the opposite of HR departments themselves where social skills are more important than technical skills.
• Software developers often appear to job-hop.  A year here, two years there, six months there.  To a classical HR person this looks risky, will this person be with the company in three years?  But actually this can give a software developer a lot more experience than they would get from working on the same code base year after year.

  And it is not really developers’ fault.  Much development work is project based, teams expand and contract, therefore people move teams and people move between companies to follow the projects.  Further more because so many software companies are start-ups the risk of company failure and redundancy is that much higher, it is an occupational risk.  Unlike HR departments which only exist in safe companies.

  The really odd bit is that the best developers may have the most fractured CVs.  If you are good you will be hunted.  If you are an average developer you are more likely to stay in one place because fewer people want you.  Simply good people move.  This is the opposite of what HR expect.  Part of this is because…

• Productivity varies:  good developers can be twice or ten times as productive as average developers.  In fact one study I saw put it as high as 27 times more productive.  Yet do you see this in their pay?  Many of us have met hyper-productive developers but have you ever met one who was paid twice or ten times more than the average one sitting next to him?

  These kind of difference mean that developers who are a little bit better than average can be significantly more productive.  Thus becomes more important to be able to differentiate between someone who is 10% better than average and someone who is 15%.

  As a result a good developer can consistently extract more economic-rent from an employer.  Because paying people these sorts of differentials is alien to HR groups they don’t do it thereby leaving the gate open for someone else to.

• Software developers often lack traditional career markers: in other words they generally don’t do sport.  They were never school team captain and – if they are like me – spent their teenage years in front of a computer honing their skills.  And they don’t move up the career paths into management.  If you have a great developer then keep him developing, do not force him into management if he wants to carry on cutting code.

This is all leading up to one more difference.  The best software developers are clever, very very clever.  There is a scene in the BBC film “Breaking the Code” where someone from the Ministry of War tells Alan Turning “You are much more intelligent than most of the people around us.  That makes them very nervous.”  I think that effect is at work too.

So what can be done?

There needs to be work on both sides.  HR needs to appreciate the special challenges and demands of managing IT resources and especially software developers.  IT needs to recognise how HR can help and the corporate need for HR.  But this may not be enough, we may need a structural solution.

A few months ago I visited the offices of one of the UK’s leading national newspapers.  There are two sides to a newspaper: the business side, including production, advertising, marketing, accounts, etc. and then there is the editorial side.  Because of the need to keep editorial independent this needs to be handled by journalists.  Therefore, there needs to be two HR departments.  The first is for the bulk of the employees and the second is just for the journalists.

I suppose the main HR group could handle the routine admin like payroll, tracking holidays etc.  but to have journalists governed by the same management structure could compromise editorial independence.  So they have their own HR group, staffed with managers from the editorial side.

This model could work in IT.  I can imagine a new role inside the group “Technical HR manager”.  This person would be involved with hiring, performance reviews, etc. etc. but they would know about technical people in depth.  They would come from a technical background themselves.

Although this might sound like a drastic change I think it is reasonable.  Companies which depend on software development really bet their future on their people.  They bet that their people can develop better technology than the competition.  What appears to be a technology based business is in fact a people based business.  There is vast difference between having above average people and above above average people.  Just go and ask Google or Microsoft.

Ironic isn’t it?  HR departments can’t be trusted when human resources are the most important thing.

And that’s another thing, why on earth did we dump the name Personnel Department? We are not talking about some plug-compatible resources here, we are talking about people.  Living breathing people, not resources.

My guess is that Personnel Departments dumped the title so they could put themselves on a part with the money and machines, the other resources; in doing so they lost sight of their key differentiator.  When people are key you have to treat them as such.

I’ve been saying for a while that there is no need to panic over the outsourcing/off-shoring of software development work to India.  As far as I am concerned the pie is big enough, it is expanding, and India is not the right place to do much of this work.  The right place is right next to the business.

Add to that economics.  India may produce a lot of computer science and engineering graduates but less than 10% of them are useful to the industry.  Of those that are there are not enough to satisfy demand, staff turn over is high and wages are rising.

Just before I left for EuroPLoP I saw this story in the FT “Bangalore wages spur reverse offshoring”.  The story even made it to the editorial column too  “Outsourced to the US”.  Basically wages are rising and the price advantage is disappearing.

As I predicted (December 2006, July 2006 and June 2007) India is pricing itself out of some types of work and it is more economic to do the work in the high-wage US or Europe.

Even if India staff are cheaper you have to factor in:

• Communication and travel costs.
• Additional processes, procedures, documentation, etc. which you don’t need when the guys are down the hall.
• Delays in responding.
• Cost correcting cultural differences, and extra costs in being specific.

It makes sense to offshore some work to India (or China, or Russia) but not all work.  You have to judge these things on a case by case basis. 

Unfortunately one side of this I haven’t accounted for before is management that believe the hype.  If you are attracted by offshoring to India you must go and do your own analysis: count all the costs, assess the risk, look at the time it will take, etc. etc. 

The unfortunate bit is I think some managers are not doing this.  They are jumping on the bandwagon when it doesn’t make sense.

More unfortunate, as far as I can tell there are far too many start-up India outsourcing companies willing to cater for unprepared managers who want to outsource.  I suspect the big, well established, companies want projects that succeed.  They will avoid work from unprepared managers looking to ride the bandwagon.

However, the smaller companies just want work.  They can’t afford the luxury of turning work away so they will accept it from even the most ill prepared companies.  So you get the unprepared working with the worst possible partners.

And that is why, some companies have great offshoring experiences and most have failures.

Still responding to the questions posted in response to my TDD and objectives entry. The question was about how to do TDD, specifically UI and also about what happens when the general consistency fails.

Well the UI question is easy: You can’t really do TDD for user interfaces.  You can, but you have to jump through hoops to get your test framework to push buttons.  Of course there are tools which will do this for you but these usually fall into the realm of ATDD (Acceptance TDD) and system testing.

But, what TDD can help you with is the logic behind the UI.  We all know we should separate program and business logic from the UI but we also all know it has a habit of getting in there.  Well if you are practising TDD you separate the two (as you know you should) and you can TDD the logic and leave the UI as a lot more trivial. (And testable with other tools.)

The other way of doing it is to test the output.  Increasingly application emit XML or HTML to generate their user interfaces and you can test that.  You can test small segments of XML to see if it contains the expected data.

For me the secret of TDD is that it is a Design technique, thats what the second D stands for after all.  It is about designing your code with tests rather than UML, or flow charts or what ever.  You do it before you write the code, it forces you to think about what you are about to write and that is design.

Second part of the question: general consistency.  Well, true, a little change might break a lot of tests, but actually you want that to happen.  If someone removed operator== then you want your tests to break because it highlights the change. 

A good set of tests round your code ensures that any unexpected changes are found quickly.

Of course, when you make a significant change in your code you might find a lot of tests broken, and you end up having to fix lots of tests but:

• Firstly a major change should product major breakage so you can get it right
• Second a well structured, properly separated design, will be less prone to breaking.  Writing tests is itself an engineering exercise.

All in all TDD promotes a lot of good practises which we know are good but somehow forget to do.

 

I don’t normally respond to comments on my blog.  Most of them don’t actually require anything more to be said but on this occasion I thought I should respond.  And keeping to my one entry one comment policy I’ll take them both separately.

So, the question was: Am I changing my view on objectives?

And my answer is: It depends; in this case I’ll wait and see what happens.

I think it is worth knowing what you are aiming for.  I think setting your own objectives is worth while and can help improve your performance.  And I think they are a good way of communicating priorities to and from management.

I also think that if you are going to have explicit objectives it helps if they can be more objective than subjective.  So, the SMART rules make sense.  But… well, there are problems….

Not everything you want to aim for can be specific or measured.  Quite often we operate with ambiguity.  A lot of my work is ill defined, trying to make it better defined can be useful but it can also be counter productive because things will change.  I’m not really sure what my goal is or should be, I just kind of know the general direction.

There is also a problem with the achievable and realistic part of the rule; in a word setting realistic goals encouraged people to satisfice.  If your promotion, pay rise or bonus depends on meeting an objective then there will always be a tendency to aim slightly lower, for something that is more achievable.

As much as they can improve communication between management and workers they can also be the source of misunderstanding; when they are imposed on people, when management sets the objective and forgets about them for a year, or when management sets them and then abdicates all responsibility for helping someone meet them.

So I stick by my answer: it depends.  I’m in favour of objectives when they are used appropriately.

I’ve been doing a round of objective setting for my client recently. When it was first announced the company wanted to go down this path I groaned. “Arhh” I thought, “I hate objectives, I hate people setting them, I hate all the baggage that comes along with them.” Neither was I impressed by the way my client’s HR department presented the idea or the proposal for implementation.

I’m a past master at criticising management by objective, it is quite easy really, just look at the side effects and read Goodhart’s Law. Anyway, its not turning out like that at all, little bit of imaginative thinking and things are looking good. Sure we are only in the objective setting phase so far so it is too early to say, much could happen but….

First I have rigorously applied the SMART approach – objectives need to be Specific, Measurable, Achievable, Realistic and Time boxed. Second I added an extra criteria “V” – call it the SMART+V formula. “V” stands for Value-add. (Only I now realise it should be V+SMART, if you can’t show value don’t continue.)

Next was the realisation that the development team all did pretty much the same job. Therefore one could expect them to have pretty similar objectives. And if they all had similar objectives it would aid team work.

Finally, I think I realised how to use this to my advantage. The team have not been doing code reviews or Test Driven Development (automated unit testing if you prefer) but everyone is willing to do both. So, its now turning out that most of the team have objectives around these area. Not everyone, and individual goals might have a slightly different take but they all build.

Half way through and this approach seems obvious. If a manager wants his team to do TDD then why not just set it as an objective? Surely this is the way it is meant to work: manager decides, manager aligns team, team does what manager wants.

If it is this obvious why bother with all the TDD evangelising, training, coaching and calls to do it. Why both with fancy change ideas. Just set it as a target and do it. Could it be so easy?

Of course it isn’t going to be that easy. For a start just because it is set as an objective doesn’t mean it is going to be met. Just because people want to do something doesn’t give them the knowledge to do it; at the start people don’t know how to do it. Then there are the usual problems like time pressures and people who will think “it won’t work here.”

Having set it as an objective I’m still going to have to work hard at giving people as much support, training, coaching and other assistance as I can.

One real danger to this approach is that the managers who set these kind of objectives might just believe that setting an objective is all that is required. As I just said: setting it as an objective doesn’t mean it will happen. Objectives are a two way street and managers need to help people meet theirs. Thats why I’m thinking of setting my own objective as: “Team reaches 80% of objectives.”

If this approach is so obvious why haven’t I done it before? Well apart from the problems I just outlined this particular tool has not been available to me in the past. Sometimes management wasn’t bought in to TDD, sometimes objectives weren’t in use and sometimes I wasn’t in a position to set the objectives.

At the moment it feels good and I think its going to work. I’ll report back…

I’m going through some notes from the ACCU conference in April and note I made after a conversation with someone – I forget who so sorry for not crediting you properly but I want to capture these thought….

We write software but we seldom write software for the sake of it. We write software as part of something else. Perhaps we write embedded software for a mobile phone, or we write an enterprise application for an account department – you get the idea.

The thing the software goes into – devices, process, etc. – is itself a system. A mobile phone is an electronics system, which is part of the cellular communications system; an accounts ledger is part of a business systems which tracks accounts.

Increasingly it is software that makes all of these systems possible. You could do a lot of this in hardware, albeit specialist hardware but doing it in software is easier and cheaper. Despite all the problems with writing a software application to manage a cell phone it is still a lot simpler, cheaper and faster than creating dedicated hardware.

Software allows you to use commodity hardware, commodity manufacturing and commodity infrastructure to create your product. The truly unique bit rests in the software. While you can copy the binary code or application for virtually nothing the creation of that software is unique. Knowledge is embedded within the software.

Ultimately all software systems are simply part of a bigger system. Systems have software inside.

This idea isn’t new. I’ve said it myself before, what is new is the term: Software Inside. I guess we can credit Intel with providing the inspiration for the term – thanks Intel!

This term allows us to focus not so much on the software but on the greater system: the phone, the accounts system, the car, the washing machine, the moon-landing, etc. The fact that we know about software can take us (the software developers) into many different and strange domains.

And once in them we ourselves, the software experts, need to adapt and learn these new domains. We need to be soft ourselves. Software developers need to be cameleons.

I’ve written before about the Agile backlash I see coming. But what I’ve been wondering is: what comes next?

One idea comes from James Noble and others who have suggested Post-Modern Programming. I like the idea, it helps us understand the world better and it helps reposition our view of software development but it doesn’t tell us to do anything differently. Post-Modern programming is more a philosophical understanding than a prescription.

Myself I’m a big proponent of Learning Based Software Development. While this changes the emphasis on in software development I’ve yet to develop the ideas into a proposal for anything more than … well ideas.

Actually Learning Based Software Development (yes I just invented the term so lets invent the abbreviation LBSD) can be seen as a software specific version of Knowledge Based Product Development. (Sorry, one of the entries with messed up formatting.)

This week I stumbled across a blog by Jason Gorman in which he discusses Post-Agile. This is the first time I’ve heard the term and it seems I’m a little late to the party. (Also a sign that I need to get to XTC a bit more regularly.)

It appears from what Jason says that opinion is divided on what “Post Agile” means yet but one possible explanation is: Being informed by the Agile/Lean movement but not following any prescribed methodology. If that is what it means than good, it could be what I’m looking for.

But boy, Post-Agile is such a pretentious title.

First an apology: I’ve been very busy the last few weeks so I’ve not been blogging so much. Naturally the creation and launch of Software Strategy has kept me busy and so has the company’s first client where I am far busier than I have been for a long time. Add to this friends and family visiting from abroad and a few things have had to give….

Despite this I wanted to capture a quote I read this week which I think is very wise.

Last weeks Economist carried a profile of Gerald Grinstein, no I hadn’t heard of him before either. Turns out he is the boss of Delta Airlines and what he said was this:

“Chance favours the prepared mind”

Which when you think about it is so true. If you spend time thinking about where you are, what needs doing, what options are available and how things might unfold then when chance occurs you know whether to grasp it or not.

This is what worries me about being so very busy. When I’m busy busy, and when see companies who are busy “doing stuff” I find so often that opportunities are missed and much of the stuff that gets done isn’t necessarily the right stuff.

For example: I often advise people to keep a work diary (or journal as some would call it.) Writing a journal is not really about recording what happened but thinking about what the implications are and what might happen next. As such it is an act of reflection. I find that when I get busy my journal (and blogging!) get squeezed, but when this happens I loose direction and although I’m busy I am less effective.

So this is a reminder that in the middle of this very busy period I still need to find time to think and prepare my mind.

A couple of weeks ago I raised the topic of How Agile Teams fail. Well in recent conversations and observations I’ve been reminded of two more modes of Agile failure. (Maybe reminded is the wrong word, maybe I could say I’ve spotted patterns, having one or two cases logged in my mind some more examples have focused my thoughts.)

One mode is where the developers simply state they are doing Agile. We can call this Fake-Agile. By simply stating they are Agile the team can then ignore documentation, avoid project management, and generally hack.

Frankly this isn’t Agile and it gives Agile a bad name. I’m sure some examples of this are genuine attempts to become Agile but without real understanding or experience it is difficult.

The lesson here is: Simply stating you are Agile does not make you Agile.

I’ve also been reminded of this by recent TDD experiences. A team I am working with has decided they want to start doing Test First Development. However nobody has done it before and three weeks after voting to do this there is little sign of it being done. This isn’t necessarily the team’s fault, although they want to do it they don’t really know how.

Think about it. When I decided I wanted to drive a car I hard to learn to do it. The same with skiing, wine tasting, and lots of academic subjects. Sure I can pick some of it up by just doing it, and if I’m really clever, lucky or something else I might just succeed – like the way I first learned to programme.

But generally, wanting to do something isn’t enough to actually do it. You need help and advice.

y solution is to arrange some training and coaching for the team to help them start doing TDD. Once we have that in place I expect things will change.

The second Agile failure mode is a bit more complicated. I’ve seen it a couple of times now and I suspect its behind many Agile failure. It happens like this…

A team are trying to adopt Agile, but they are also working with new technology. The new technology could be technology new to the team, it could be technology new on the market (perhaps a beta release from a vendor) or it could be newly introduced technology. Whatever, the team need to learn the technology.

So we have New Process and New Technology.

The first sign is exploding cards. Teams write cards to represent the work they are doing to do at the start of the iteration but when the iteration gets underway it turns out the cards were vast understatements, both in terms of the time required and the work required.

(By the way, you need a card and board system to spot exploding cards, a software tool will simply hide the problem.)

In some cases progress is slow and few, if any cards are complete. Perhaps people say things like “the card is 80% complete.” In Agile iterations 80% counts as zero. It is either done or it is not done.

In other cases, where teams recognise the extra work involved they may write more cards – hence the expression exploding cards. Here developers start work on a card and realise/find there is far more work than they anticipated. They break this down into several smaller tasks and the number of cards multiply. In one case I saw developers took to folding the cards in half to fit them on the white board.

There are techniques for dealing with this situation. Writing cards to represent the extra work is one, and you may need to add more frequent re-prioritisation.

Another technique is time-boxing: where developers have to guillotine their activities at some point and just go with the best they know so far.

And another is investigation: where an investigation task (probably time-boxed) is conducted before the actual work itself. This can help split tasks down.

But, all these extra techniques for dealing with this situation are relatively advanced. An advanced Agile team may simply take these problems in their stride. For a team new to Agile this situation can be off putting, some people may take it as evidence that Agile does not work and it may even lead to project collapse.

Such projects were probably going to end up in a mess whether they are run Agile or not. Because of the new technology the project is uncharted waters and the amount of learning and investigation needed is very high. Therefore, even with traditional project management techniques the project will probably hit problems.

Here the advantage of Agile is that you know the problems are happening much earlier and not just at the end. But, if they team is new to Agile then you may loose the project and the Agile practices too.

The solution here… Not sure I know one. Some projects will always slip and even fail no matter how well you try to manage them.

If there is a solution it lies in seeking help, involving people who have done these things before – either Agile or the technology. Training and coaching can help too, whether in the new technology or new process.

What is the lesson from all of this? Well if there is one it is simply that Agile Software Development may look easy but it isn’t. Getting to be Agile is not simply a case of reading one book, it is a case of reading many books, talking to many people, working with experienced practitioners, having training, coaching and trying again when things don’t work out.

It isn’t easy being Agile.

Something else I discovered at the ACCU conference: Bicycle Repair Workshop.  Should I name to originator?  Or should I let her live in quite anonymity?  Perhaps I should actually seek permission to talk about this here, but then was it not Picasso who said “great artists steal.”  (No I did ask permission.)

First the name: its a straight take from Monty Python.

What it is, is a forum where software developers at the company can gather and talk about the problems they are currently encountering.  The originator envisaged a forum for people to talk about process issues, and specifically Agile practices but its free form enough to let people take in what they want.

The workshop takes up an hour or so on a regular basis, I can’t recall it it is weekly, bi-weekly.  There is a bit of fun (cakes supplied) and people turn up only if they have something to contribute – so a self selecting group.

Like the book groups and Tech-Talks that I’ve created in the past the forum gives people the opportunity to talk about the issues they face.  By exposing, explaining and naming the problem/issue/challenge/opportunity it becomes more manageable.  By explaining it people are forced to make some sense of it to themselves and then to other people.  And when other people see it they can help.

In one of his books Gerry Weinberg (Secrets of Consulting I think) compared problems in companies to moss.  In the dark, moss grows and becomes bigger.  Exposed to light moss dies.  Problems are the same, when you hide them they become worse, expose them and it becomes easier to deal with.

I see this happening when people gather in a pub or coffee bar, or stand around the water cooler and moan about things in their work place.  They know what is wrong but they feel helpless to act.  So bar and water cooler conversations don’t result in much action.  Often these problems don’t get talked about in official forums either:

• because these problems shouldn’t exist, e.g. the according to the official process manual these things shouldn’t happen
• because office politics gets in the way, e.g. everyone knows that X needs to change but nobody feels they can say it
• because somethings we can’t talk about, e.g. Fred under performs and needs to be let go
• because solving the problem needs resources and we all know the company won’t spend money
• because we are embarrassed to admit to our own problems

The advantage of the semi-official forum, like Bicycle Repair Workshops and book groups is that people are freed of these constrains and can talk more freely.  For managers this posses a problem: too much support and involvement with the forum will make it official and kill it, too little support and it will turn into another moaning shop and nothing will happen.

By creating a forum people are allowed to talk about a problem and expose it to the light.  The advantage of the Bicycle Repair Workshop is that it is directly asking for and discussing these issues.  I occasionally facilitated a Tech Talk into such a forum, and the books we studied in book group lead to discussions about problems bit I never tackled the issues head on, on a regular basis.

Of course it mights be possible to have too many forums.  All of these activities take time and commitment.  A company running bicycle repair workshops, book groups, Tech Talks and what ever else may over tax people.

These three types of forum share another thing in common.  They can be created through bottom-up action.  You don’t need to be a manager to create them, you can just do it.

On the other hand, if you are a manager then then can create other types of forums. I’m working with a company now to help them revise their development process and become more Agile.  One of these things I’m doing it running twice monthly improvement meetings, kind of reverse-retrospectives.  These are more official so have a different flavour.

In all these cases the real problem is turning information and learning into action.