The Freedom to Fail

Recently I had a discussion with a colleague about his strong 'fear of failure'. We spoke about it and hopefully I put his fears into context, and I will try and build on that here.

Now the Fear of Failure is important, it helps keep us motivated, it keeps us diligent and for some people its is truly what drives them. However in our industry it also stifles growth, innovation and promotes over conservative behaviour.

I am quite lucky in my current role, I lead a team of engineers who have explicitly been given 'The Freedom to Fail' - it is actually in the team mission statement, how neat is that?

Now that doesn’t mean its ok to sit back relax and just let projects crash on to the rocks. There are some projects that we should not ever fail on, those where a solution is self-evident, where we have more than adequate resources and where risks are contained.

But what it does mean is, we can be more experimental in our approach, we can truly try and learn whilst tackling the challenge, and we can try things that normally would be considered risky. We get a chance to learn and grow.

In an organisation that in many ways is very conservative, and in a current climate that promotes that conservatism, we break the mold.

Now that doesn't mean we do things purely because we like the idea of them (well I lie, we occasionally do but as a learning exercise). But we question the traditional approaches, we ask ourselves if the "typical" solution is appropriate, if the corporate template is appropriate and seek out other approaches which may indeed prove to be a better fit.

Perhaps luckily our choices have proved successful more often than not. But things do go wrong, we on occasion do fail and wind up either having to restart, rework or abandon a piece of work.

It is not wrong to fail, however it is very very wrong to hide, deny or ignore failure. That failure may only be partial or minor, but it is still a failure and needs to be addressed.

So here is what is important...

Identify Failure

Have a way of identifying failure (and subsequently success). There are many useful techniques that can be applied here, including time boxing, acceptance testing and "done means done".

Time boxing can be vitally important. Estimated work often overruns its a fact of life, we are poor at estimating and often something simple takes a lot longer than expected. Using a time-box you can identify that you have failed, and you need help, need to change approach or need to give up.

Acceptance testing lets you prove that you haven't failed. In many cases I take a pessimistic view of success, if its not passing all the tests its a fail.

Done means done" - all too often you can ask if feature X finished and you'll get the reply "oh yes that's done but this part needs tweaking". Sorry pal, if that part needs tweaking its not done, now put that card back into "in progress" and finish it properly before declaring it done.

Understand the consequences of Failure

It is important to understand what the consequence of failure is at every point. There are varying magnitudes to this, both at a project planning "will we get this done on time" point of view and real life consequences of failure of the project after release.<

It is important to know what can be done (if anything) to mitigate the failure. Can a different approach be taken, can another team undertake the work, can we just ditch it?

Structure what you do to ensure any failure occurs as early as possible in the process. Failing quickly increases the chances that we can mitigate the failure or if that is not possible then stop work before more resources are wasted.<

Understand the Root Cause of the Failure

All too often the root cause of a failure is not fully understood. "We think its because the connection pool was used up and that caused ..." - wait up here, stop, "We think?"

In some cases the root cause is obvious and at other times it requires investigation (root cause analysis) however something that is often over looked is what I consider "people problems".

Dealing with people is hard, especially in an industry known for having anti-social tendencies. Quite often people without the correct skills, experience or support are given problems they are just not equipped to handle - and thats a management failure.

Disseminate the Failure

This is the bit a lot of people shy away from. I have a great deal of respect for someone who can stand up in a meeting and say, "yeah I screwed that up, I need help to find a way to resolve it".

In my experience the failure to discuss failure is worse than the original failure - it dooms others to repeat it. And covering up a failure is far far worse than than actually failing, perpetuating a lie that something works when it doesn't.

Honesty is indeed the best policy. Of course I don't mean that companies should publicly air their dirty laundry but rather within your team (and hopefully within your entire company) there should be open, honest and frank communication.


Final Whitterings

Does your team fear failure to the extent they don't think outside what they think is expected of them? Worse yet do they fear it to the extent they will actively hide or deny it? Are you holding your team back through your own fears?

Building a sense of trust within a team is vital, how you address the issue of failure I think is vital.

If you work in an environment where failure is expensive then try and create a space where it is acceptable. Code Dojo's, Hack Days and similar events give people a chance to flex their mental muscles, their creativity and lets them grow, it gives them that freedom to fail.

Can't afford to fail? Can't afford not to learn from failure.

In order to ...

Lately I've been doing some story generation for a few projects, and I've really been focusing on the idea of putting value first.

Its a subtle, but powerful idea. I was introduced to it by Dan North during his tutorial on BDD at QCon London 2009 and it really clicked.

I've been doing quite a bit of process work involving Kanban, which really focuses on value - so it has a kind of synergy (pardon me while I wash my mouth out for using management speak). Liz Keogh also wrote a good article for InfoQ recently on Kanban and BDD.

So the format we've chosen is the same Liz describes in her article:

In order to ...
As a ...
I want ...

And its working really well so far. By putting the value first it makes more sense to our domain experts, who typically being non-technical in most cases got stuck on "As a", and "I want". And we've found that we keep our scope better contained as people stop thinking about "I want" and concentrate on the end value of "In order to".

So if you are working on stories I would really encourage you to try this one simple change. Hopefully you'll see the benefits.

An Unexpected(?) Advantage of TDD

So there are a lot of advantages to using TDD, they are well documented and there are plenty of people who extol its virtues.

One of them however has resonated with me of late. And that is how it effectively builds a living document of the implemented functionality of your system.

I am working on a personal project, and the time I can spend on it is quite sporadic. Between work and a young family, time to code for fun is quite broken - 10 minutes here, an hour there, a weeks break, another 10 minutes and so forth. It doesn't make for much of a flow.

But what I have found is, by sticking to TDD and by using test names that are informative that I have built a living document of what I've implemented. So when I come back to the code even for a moment I know where I am. Run the tests, and green, there is all my functionality.

Better yet, the breaks force me to revisit my naming, and I actively refactor the code (including the tests) to try and make it more readable.

I've also found that by invoking the "if its not green at then end of the session then ditch it" rule quite effective too. Now you might say that if you only have five minutes to work on it then chances are you're not going to get to green. And a lot of the time its true - but before I stop, I roll back to green.

Now at first that seems counter productive, as the tests will show where its broken and you know where to come back to - but in practice I have found that it actually tends to mislead me, particularly if I have been away from it for a while. So better to return to a clean slate. Then what I find is that my subconscious tends to chew on the problem for a while, and when I come back to it I find my flow much quicker.

So its just another reason to use TDD and descriptive test names.

Building your Village

On Friday the 24th of April I ran an Agile day for my team and others at the BBC, I managed to get some good speakers on a variety of Agile and Development topics - it turned out to be a really good day. The official blog post on it is almost ready to go up.

During the day my mind drifted on to the importance of interactions.

I tend to see a project team as a village, the village should be as self contained as possible - the butcher, the baker and the candlestick maker. Everyone in a village has their part to play in its success. And in a strong village everyone pulls together when times are tough, they come to a fellow villagers aid in times of trouble.

So whilst we may have a team made of specific roles and skill sets the team is greater than the sum of its parts. So as a project suffers challenges the team should pull together to meet those challenges.

I have worked in teams that do this naturally, and teams where its just never quite happened and teams where it never had a chance.

The teams where it worked had a few things in common:

Experience - in all cases the team was experienced, both in software engineering and the problem domain. Each team had at least one problem domain expert, someone who could answer or anticipate the questions.

Attitude - in all cases the team had a strong professional attitude. Not only did the teams strive to minimise technical debt, but all had generally strong work practices. That sense of professionalism actually was strong enough that the team was able to "stand up" to management at those times when it was really needed.

Motivation - in all cases the team was motivated. And typically the motivation came from the enjoyment of working with a great team doing great stuff. Something that is often overlooked.

Now if you had these things you could argue that pretty much any process would work, and I would agree - not all these projects were what we would call Agile today. But they were Agile in the sense of putting People and Interactions, before Process and Tools.

So if your team is not pulling together in troubled times, look at the three points above. Does your team have experience, attitude and motivation? If it doesn't you need to look at how to address it. And you don't need to be the team leader or manager to address it - its amazing what influence any member of the team can have if they are prepared to talk to their team members.

So how do you build your village?

Firstly, interaction. How well does the team communicate? All the members of the team need to feel that they can speak up, and all the members of the team have an obligation to listen. Do you all sit together? If you can't sit together do you use the same watercooler or coffee machine? How often do you have lunch or go to the pub as a team?

I wrote a little while ago about development being a social activity - this follows on that great teams subconsciously understand this. Now by social I don't mean going out and partying, but I mean interacting.

So why do I think interaction on these levels is so important?

Well a team that communicates well will be more fun to work with, it will improve team motivation, it will lead to improved attitude (management by example or culture) and it will encourage everyone to use their experience, encourage people to improve or better themselves.

Its not going to solve the problem of having domain experts, java experts, jvm tuning experts - but you in the end you get to be an expert by doing and learning. Ideally by learning from another expert, but that doesn't have to be from the team, it could be from conferences, blogs, mailing lists, geek nights, books, consultants or from another team in the organisation. Building the right village will encourage the team to learn, to rise to the challenge.

Software Development is a Social Activity

Something I have been pondering for a while, is "Why is software so hard?". I know it is hard, I have spent a long time working in the industry, I have seen it fail - thankfully I have seen it succeed (albeit often "challenged") more regularly. I have read a lot of books and talked to a lot of people, but still I felt something was missing.

Computers are logical devices, they follow your instructions exactly without the slightest deviation - however being human we are not logical, and we do not execute or think naturally in the same precise logical manner. Our brains are non-linear, we make jumps, we shift through varying layers of abstraction and we rarely stay in that logical state for too long.

But that still doesn't explain why software is so hard. We also have the tools to focus us and guide us when it comes to writing the code. Compilers, unit tests, analysis tools, exploratory testing as well as a myriad of processes attempt to focus us on the problem at hand. So why is it still so hard?

I think one of the leading reasons is this, Software Development is a Social Activity and those that work in software development are stereo-typically anti-social in nature. Yes it is a stereo-type, but it does have a firm foundation in the truth. By in large, we are not social butterflies, we tend to prefer the company of machines to that of humans, the company of our problem domain to that of other people. I hear cries of disagreement from some quarters - yes it is not the rule, there are some highly sociable developers out there.

However time and time again, I have seen software projects flounder because of the lack of both information and 'social' bandwidth. By social bandwidth I mean the social interactions between members of the team. Good social bandwidth from my experience requires a very high information bandwidth - face to face communication is good example. This is somewhat difficult to explain, and something I feel that I am only beginning to grasp.
Email has perhaps one of the narrowest social bandwidths of all. You have no real sense of the other persons state of mind, their current emotion or their current physical situation. Its even narrower when one or even both parties don't recognise that fact. We have all witnessed email storms caused by an off the cuff comment at one time or other.

An area of high social bandwidth is the office coffee machine or water cooler. Where people tend to meet, talk and exchange idea's purely because of the constructed environment - you would not choose to go and chat to a random person every hour or so just for the hell of it (... or maybe you would, maybe you are one of those social butterflies).

Location is a key factor here, as we congregate together to work, we construct a village, a place of community. Now communities don't always get on, and certainly are not all best friends - but they do tend to build a spirit over time, especially where interaction is unavoidable.

The act of developing a piece of software is the act of a community, it requires a varied team. Just as the village requires the butcher, the baker and the candle-stick maker. The most productive teams recognise and encourage that variation in skills, and tend naturally strive to increase their social bandwidth to spread those skills and experience.

Too many times I have heard a senior manager, who having walked into the office for 30 seconds in a month, complain that a team is not working hard enough - when I know what they heard was actually the sound of productivity, the sound of high information and social bandwidth. A team with a strong bond, can have a chat or a laugh - a positive working environment increases productivity, it makes people want to come into the office.

Now I am not saying the team needs to hold hands and sing campfire songs, but rather the team needs interaction. It needs developers, testers, business analysts, project managers and customers to interact. And the team should foster that interaction both professionally and socially. I've seen project managers turn down a pub lunch request which would have done wonders in building a fragile relationship into something more substantial.

It is important that this social drive come from within the team, and not an enforced company social event, one which will leave some feeling awkward or even resentful, and actually discourage some from attending at all. A sense of professionalism can provide that balance, and it is more than achievable with the right team.

Continuous Integration - Moving beyond the build

So everyone (well I really hope everyone) is now using Continuous Integration of some form or other. However a lot of CI I have seen is purely build, or build and unit-test. This is really under utilising a potentially great resource.

These days developer time is expensive compared to processor cycles. So we need to maximise developer time spent on productive things. Automation can help us do this, and automation starts with CI.

So what can we do to make this a reality?

Firstly consider what tasks do you do repetitively that take up time? Deployments, Integration Tests, Acceptance Tests? What tasks would you do more often but they take too much time? Static Analysis, Performance Tests, Endurance Tests?

But, I hear you say, this takes too much time, we cannot spare that right now.

Well I disagree - the time that you spend now will pay you back later.

In anything from a few minutes to a few hours you can set up a nightly build, analysis, test cycle could save a whole lot of trouble and which can be incrementally improved to make life easier.

I plan to write some blog posts on how to do this, just need to find the time.

Test Driven ...

In the past I have been guilty of not testing enough (who hasn't). I went through a phase of code first and test as late as possible, then code first and write tests - the old idea of unit testing. However this always ends up in testing getting left behind as time pressures push the demand towards just getting the code done.

I was then introduced to Test Driven Development, it made sense, but I struggled with it. Even when I thought I was doing it right I would find as time pressures increased that it would slip, and it wouldn't be long before I was back in the old style of unit testing.

I used to grapple with design, either in my head or on paper, then write a test against that design and then try and code that design. The problem being, that the design was never quite right. Some requirement was always missed or not considered, or even worse non-required features would slip in because they seemed like a good idea. Going back and changing that design, meant going back and changing the tests.

The other problem was that the tests were always biased towards that vision instead of testing the requirements, increasing testing became a matter of just adding more and more tests, a lot of them testing the same thing and of dubious value.

So as a result things never flowed, there was always a discontinuity - and that leads to turbulence and drag. Drag means wasting energy, there in the temptation to ditch that perceived drag, the testing, instead of the real drag, a bad design.

I then had the pleasure of working on a project with some really focused guys (Jim, Pat, Alistair amongst others), those who knew what TDD really was, and how to really do it, how to use the tools in the right ways. To them it was just natural, everything flowed, there was no drag.

The 'click' was the realisation was not that I didn't know how to write unit tests, nor was it that the test had to be written before the code, but the test had to be written before the design had crystallised.

Now, I am not saying that the secret is no design. Indeed design is very important, but now I tend to do enough design when and as its needed. The final design emerges as I test, code, test, refactor, test.

Why the term "Sprint" Irks Me

Apart from being inherently lazy, there are other reasons why the term "Sprint" irks me....

Agile development is supposed to be about sustainability - in fact the Principles of the Agile Manifesto states ...

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

The term Sprint in my mind does not describe a constant pace, it conjures up that desperate dash to the finish line, giving it your all in one go.

The argument is however that "sprints are short", however they are repeated. Do we expect the Olympic 100m runners to do a 100m dash, and then do it all over again, and again, and again? All in a relatively short time frame, after all how long do you rest between week long Sprints?

The other argument is "its just a name", however software development is in a large part about naming. Writing clean code is more often than not seeking the right name, something that describes intent - so why should we strive for anything less in naming our processes and their elements?

So until I find something better I will continue to prefer iteration as to me it better describes the intent of repeated time-boxed work.