When the pressure is on

Ever been in that situation, its a deadline, you are under the gun, things are going wrong.  You are putting in the long hours, but no matter how hard you work things seem out of control.  I’ve been in that situation, a few times.  I’ve worked in a couple of start-ups, and one in particular had a couple of occasions where it almost all went wrong, we pulled it in, but boy it was close.  

So you think, we’ve done it before but we didn’t fail, we made that release or we went live with that customer... or maybe you didn’t... or maybe you did but in the process you burnt the relationship with the client... or burnt yourself (or your team) out or maybe you found yourself in firefighting mode for weeks post release scrambling to keep things running behind the schemes.


The problem is it can be addictive, it can be a rush to pull that rabbit out of the hat, and it tends to propagate the Hero culture and then it feeds on itself.  Eventually this will lead to an implosion where projects and possibly the company will fail.

How do we break that cycle, given that we are in the middle of it?

[Context note:  This is more aimed at small teams/startups and I am not going to go into how to avoid the situation in this post, it is about the situation at hand]

First, find some calm.

 

Second, step back, go for a walk, go to a cafe, take a nap - do what you need to do in order to get some effective head space.

Third, you need to plan, and I don’t mean a detailed plan, I mean just enough of a plan to keep you on the right track, a meta-plan if you like, something to guide you.

So you are now calm, you can judge risk more clearly, you can make decisions, you will make less mistakes.

If you are still doing features I recommend setting up a really simple “Kanban” board.  Something like [ backlog | dev | test | deploy | accept ], put up a few of the most important/riskiest tasks (negotiate, argue, decide amongst the team and the product owner) on the left, and pull them across one at a time right through to the end before starting the next - don't be tempted to work on too many at once, more team members than cards in flight is a good rule of thumb.  When that bunch is done, put up the next few important and repeat, keep the number of cards on the board small, don’t overwhelm it or yourselves.  Don’t get hung up on the column names and the card content/structure - just do the minimum to keep a tab on what you are doing, adjust as appropriate.

Its likely that you are close to release, perhaps your stuck in a nasty bug/quick fix cycle.  “This change will fix it, ah no, damn, ah, try this, and this..... “. Time to break cycle.  This sort of thing will destroy the confidence of your client, your credibility and damage the likelihood of future work.

So how do you break out of this?  Don’t guess, be data driven, work from facts and not supposition, that meas work from test results.  Ideally you’ll be using TDD or BDD style methods but if you aren’t you can still leverage some of those ideas to help break the cycle.  Reproduce failures, if possible work out how to automate that test - even a simple bash script using curl and grep with a simple assertion (exit 1) is better than nothing.  And will help prevent that problem creeping back in later.  You can take these tests and later build them into a proper BDD suite.  

Take small steps: when confidence is low move slow.  By taking small steps you can confidently move forward as you take less risk and its easier to roll back to a known, or at least better known state.

“But I don’t have an environment to test in” … so this is a huge risk, usually it is actually the case that the live environment is very different to the test one.  Time to change that, a few hours (even days) will make all the difference.  “We don’t have the hardware” … sorry that excuse is running out, AWS and other cloud providers allow you to put together a lot of hardware quickly at pretty reasonable cost that you can turn off when done - a few hundred $/£/€ may be worth it if you can solve the problems effectively.

Remember its not easy, if it was someone would have already done it, so looking for silver bullets is counter productive.  Whilst you can get away with cutting some corners (with acceptable consequences usually in the form of technical debt) there are some you can’t - sometimes you just have to accept spending that time and money to make sure you can test what you are doing adequately.  Keep note of those decisions, where you have incurred technical debt, so it can be addressed later.

The important thing is to maintain perspective.  Yes things are bad, but its possible to bring things under control, when things are in control you can make better decisions, you can make clear decisions on trade offs.

Setting up a ‘war room’ is sometimes a good idea, as long as it promotes that calm control, and doesn’t become a yelling chaos factory.  Protect your space, make it clear that you don’t want those who will disturb the groups balance invading that space.  Agree to provide status updates on your terms - ideally your simple Kanban board will give enough of that status for you!

Protect your relationship with the client/customer - keep a clear consistent communication channel, one person to inform, update and ask questions.  I sometimes call this an Anchor, like a news Anchor they are there to keep consistency and to keep it (the channel) together under pressure.  They can also help prevent the ‘quick fix/release/break’ cycle by keeping check on the team output.  They should be professional, keep emotion aside (including not reacting to emotion from the other side) and try to build a collaborative approach.

“It’s not our fault its broken … It works for us” … this is a very common attitude, however whilst at the time you may think you are right, all too often you turn out not to be.  That is embarrassing and possibly worse.  If its not your fault, be prepared to be able to prove it.  If you can’t prove it and more importantly if you can’t prove it in the environment that your client is seeing it in, then accept its more likely your problem than theirs.  The worst thing about this sort of response is that it breaks confidence, and reduces trust because once you do it once it becomes all the more difficult to work closely with the client to solve the next problem.

No plan survives contact with the enemy (http://bit.ly/kl5OgR) be prepared to adapt as necessary.  Periodically step back, breathe and review the board.  Take a quick poll of the team, chat to the client and sanity check your priorities.

One boss early in my career gave me this advice, “In the next quarter they are not going to remember that you were two weeks late, but they are going to remember if you delivered them a steaming turd on time”.

Now days I view this advice in terms of the time/scope/cost triangle - they won’t remember if you were missing a couple of features, but they will if it didn’t work at all.

The goal is to move toward sustainable development and to not be in this situation to begin with. But things don’t always work out, you and your team may not have the experience or the available resources, you may just be moving too fast so as not to miss the opportunity.  Some times you find yourself in a fire fight and you need some help with the hose and not a lecture on the dangers of playing with matches.

Importantly when the dust settles a bit you make the time to look back, do a retrospective, try and work out where things went wrong in the first place and look to improve.