Random Whitterings

Enter the Pipeline .... exploring Scala and Akka

2011-06-11T20:52:00.001+01:00

Sometime ago I did some interesting work with Akka. These days I don't code everyday, most of what I do is in my own time and I wanted to play with the latest Akka (1.1) and see if it was still as fun as it was.

I wanted something simple to connect to Twitter, read the stream, chop it up message by message, pass them to a JSON parsing actor, which it turn would pass them on to further actors for more processing. The concept can be extended in many ways.

The codebase as it stands is quite small, which is really thanks to standing on the shoulders of giants:

Scala 2.9.0
Akka 1.1
SJSON
Signpost OAuth library
Apache HTTP Client 4.1
ScalaTest
Mockito
SBT

A few comments about the code...

Firstly its not production code, it is an exploration of various things. I will outline some of what I think are the most interesting parts of the code. See the bottom of this post where to find the code.

Using Either and Option

One of the things that has always created confusion is the use of exceptions for things that are not really exceptional cases. Too often we see exceptions used for alternate path selection, often the less travelled path but still a valid path.

If you expect failure, which in a networked environment we should, then you should not be using exceptions, its not an exceptional case, its an expected one. However how do you represent this in code? Thankfully Scala gives you Either. Your methods can use Either to indicate that this method will return either an error or a result.

In the above we have defined a trait with a connect method that can return either an Error if something goes wrong, or a BufferedReader if all goes well.

Similarly you can use Option to indicate that a method may return a value or not. This is much more meaningful than returning null's as you cannot be sure what a null really means, whereas an Option explicitly states its Optional.

And its possible to combine the two...

So in the above we are saying, either expect a ReadError if there is a problem otherwise you will 'optionally' get a string.

Using 'become' to manange connection state

Akka provides the ability to change the receive method of an Actor using 'become', this can be very useful to manage the state of an actor based on the messages it receives. In this case we have two "states", active and inActive. The messages that are valid in each state are different. The actor can move between these states using 'become'.

Testing

I have written the tests with ScalaTest, in a Given-Then-When style. One of the challenges of testing an intergration point is verifying expected behaviour without actually having to connect to a real service. Using AkkaTestKit and Mockito I have attempted to test as close to the integration as possible without actually building a mock service.

Testing logging events

Akka provides an eventing service which is an alternative to traditional logging. Eventing decouples the action of logging from the source of the event, this is very useful in many respects including testing. It is possible to hook into the event service through the test, in this case I used an actor that simply places the events received on the queue. The test can then verify expected events are on the queue in the expected timeframe.

Whilst it is possible to get the Events sent to the TestKit testactor I found this tended to made the test confusing as essentially two streams of messages were being merged which in turn made the test logic quite complex. By using a separate queue it made it clearer (in my view) to understand.

Below you can see an example using Mockito as well as the event queue.

How to use it

Warning: whilst using this code for testing and experimentation does not (according to my understanding) break any Twitter T&C's you should satisfy yourself should you decide to use this code or alternatively base any future code upon it.

You will need to register an application with Twitter for which you will need to have a Twitter account, login and then go to https://dev.twitter.com/apps

Here you can register a new application, then make note of the consumerKey and consumerSecret. You will also need to make note of your Access Token (oauth_token) and Access Token Secret (oauth_token_secret).

Then you need to clone the source code and built it using sbt (http://code.google.com/p/simple-build-tool/)

The following transcript shows how to use it.

If you want to play with the code in Intellij, just run 'sbt idea' on the command line. The code is tagged with 'blog_post' at the point in time of writing this blog post.

When the pressure is on

2011-04-27T19:55:00.000+01:00

Ever been in that situation, its a deadline, you are under the gun, things are going wrong. You are putting in the long hours, but no matter how hard you work things seem out of control. I’ve been in that situation, a few times. I’ve worked in a couple of start-ups, and one in particular had a couple of occasions where it almost all went wrong, we pulled it in, but boy it was close.

So you think, we’ve done it before but we didn’t fail, we made that release or we went live with that customer... or maybe you didn’t... or maybe you did but in the process you burnt the relationship with the client... or burnt yourself (or your team) out or maybe you found yourself in firefighting mode for weeks post release scrambling to keep things running behind the schemes.

The problem is it can be addictive, it can be a rush to pull that rabbit out of the hat, and it tends to propagate the Hero culture and then it feeds on itself. Eventually this will lead to an implosion where projects and possibly the company will fail.

How do we break that cycle, given that we are in the middle of it?

[Context note: This is more aimed at small teams/startups and I am not going to go into how to avoid the situation in this post, it is about the situation at hand]

First, find some calm.

Second, step back, go for a walk, go to a cafe, take a nap - do what you need to do in order to get some effective head space.

Third, you need to plan, and I don’t mean a detailed plan, I mean just enough of a plan to keep you on the right track, a meta-plan if you like, something to guide you.

So you are now calm, you can judge risk more clearly, you can make decisions, you will make less mistakes.

If you are still doing features I recommend setting up a really simple “Kanban” board. Something like [ backlog | dev | test | deploy | accept ], put up a few of the most important/riskiest tasks (negotiate, argue, decide amongst the team and the product owner) on the left, and pull them across one at a time right through to the end before starting the next - don't be tempted to work on too many at once, more team members than cards in flight is a good rule of thumb. When that bunch is done, put up the next few important and repeat, keep the number of cards on the board small, don’t overwhelm it or yourselves. Don’t get hung up on the column names and the card content/structure - just do the minimum to keep a tab on what you are doing, adjust as appropriate.

Its likely that you are close to release, perhaps your stuck in a nasty bug/quick fix cycle. “This change will fix it, ah no, damn, ah, try this, and this..... “. Time to break cycle. This sort of thing will destroy the confidence of your client, your credibility and damage the likelihood of future work.

So how do you break out of this? Don’t guess, be data driven, work from facts and not supposition, that meas work from test results. Ideally you’ll be using TDD or BDD style methods but if you aren’t you can still leverage some of those ideas to help break the cycle. Reproduce failures, if possible work out how to automate that test - even a simple bash script using curl and grep with a simple assertion (exit 1) is better than nothing. And will help prevent that problem creeping back in later. You can take these tests and later build them into a proper BDD suite.

Take small steps: when confidence is low move slow. By taking small steps you can confidently move forward as you take less risk and its easier to roll back to a known, or at least better known state.

“But I don’t have an environment to test in” … so this is a huge risk, usually it is actually the case that the live environment is very different to the test one. Time to change that, a few hours (even days) will make all the difference. “We don’t have the hardware” … sorry that excuse is running out, AWS and other cloud providers allow you to put together a lot of hardware quickly at pretty reasonable cost that you can turn off when done - a few hundred $/£/€ may be worth it if you can solve the problems effectively.

Remember its not easy, if it was someone would have already done it, so looking for silver bullets is counter productive. Whilst you can get away with cutting some corners (with acceptable consequences usually in the form of technical debt) there are some you can’t - sometimes you just have to accept spending that time and money to make sure you can test what you are doing adequately. Keep note of those decisions, where you have incurred technical debt, so it can be addressed later.

The important thing is to maintain perspective. Yes things are bad, but its possible to bring things under control, when things are in control you can make better decisions, you can make clear decisions on trade offs.

Setting up a ‘war room’ is sometimes a good idea, as long as it promotes that calm control, and doesn’t become a yelling chaos factory. Protect your space, make it clear that you don’t want those who will disturb the groups balance invading that space. Agree to provide status updates on your terms - ideally your simple Kanban board will give enough of that status for you!

Protect your relationship with the client/customer - keep a clear consistent communication channel, one person to inform, update and ask questions. I sometimes call this an Anchor, like a news Anchor they are there to keep consistency and to keep it (the channel) together under pressure. They can also help prevent the ‘quick fix/release/break’ cycle by keeping check on the team output. They should be professional, keep emotion aside (including not reacting to emotion from the other side) and try to build a collaborative approach.

“It’s not our fault its broken … It works for us” … this is a very common attitude, however whilst at the time you may think you are right, all too often you turn out not to be. That is embarrassing and possibly worse. If its not your fault, be prepared to be able to prove it. If you can’t prove it and more importantly if you can’t prove it in the environment that your client is seeing it in, then accept its more likely your problem than theirs. The worst thing about this sort of response is that it breaks confidence, and reduces trust because once you do it once it becomes all the more difficult to work closely with the client to solve the next problem.

No plan survives contact with the enemy (http://bit.ly/kl5OgR) be prepared to adapt as necessary. Periodically step back, breathe and review the board. Take a quick poll of the team, chat to the client and sanity check your priorities.

One boss early in my career gave me this advice, “In the next quarter they are not going to remember that you were two weeks late, but they are going to remember if you delivered them a steaming turd on time”.

Now days I view this advice in terms of the time/scope/cost triangle - they won’t remember if you were missing a couple of features, but they will if it didn’t work at all.

The goal is to move toward sustainable development and to not be in this situation to begin with. But things don’t always work out, you and your team may not have the experience or the available resources, you may just be moving too fast so as not to miss the opportunity. Some times you find yourself in a fire fight and you need some help with the hose and not a lecture on the dangers of playing with matches.

Importantly when the dust settles a bit you make the time to look back, do a retrospective, try and work out where things went wrong in the first place and look to improve.

Estimation? No a distribution of confidence ...

2011-02-09T21:15:00.000Z

I have always been wary of estimation...

Well no, that's not quite true, I wasn't until I gave an estimate to a salesman, and then I found myself having to meet a hard release date... once bitten... twice bitten... well in reality I am a slow learner so it was probably a couple of dozen times.

Most estimates are really figures plucked from the air by a developer, or worse someone who doesn't actually work on the code, or even far far worse by someone who have never even seen the code.

So here is my current thoughts on estimation.. not fully formed, and probably poorly communicated, but I still think worth sharing, if only to improve my own understanding.

Even with Agile poker planning methods, estimation is still a dubious business.

An estimation is only valuable if you are aware of the context in which the estimation was made. Then you must remember that estimation is exactly that, an estimation - it is in fact some sort of distribution curve of the confidence of the person making the estimation.

Huh?

What if you plot the 'confidence' of the person making the estimation against the possible solution time. Typically this might look something like this.

The estimate they will give should be around the peak of this curve - well pessimistic estimators will be a bit to the right to give them some margin, and optimistic to the left.

Now the shape of this curve is what is interesting. It will be determined by how 'known' the problem is. So a really well known task may have a graph like so:

And a conversely really poorly understood problem may have a distribution like:

At either end of the spectrum shown here estimation is pointless.

Either you know the problem so well that time spent estimating is actually waste or you have no idea of the problem so it becomes essentially a random guess. A random guess is more dangerous than not estimating at all.

So estimation is of most value in this middle region, where you have a reasonable understanding of the problem, but not so well that estimation is just waste. A goldilocks zone if you like.

Good poker planning sessions often are more about the conversations and the discovery around a problem. These discussions attempt to drag the flat distribution into a more normal one where it is sufficient to make an estimate.

One of the things I have seen here is some teams then get into an obsessive perfection of estimation loop. Where every retrospective has a key outcome of 'we must improve our estimation'. And I've seen teams haggle for what feels like hours as to whether it really is a 7 hour or 13 hour story.

Their key outcome is wrong, they are focusing on the wrong thing. The real outcome should be, we should improve our understanding of the problems at hand.

I think that one way to avoid this obsession is to use T-Shirt sizing - step away from the numbers and use more coarse grain concepts like Small, Medium, Large.

Note that I think story points are better than real time measures but I still think the psychology of dealing with the numbers is the same.

T-Shirt sizes are coarse, deliberately so. You can start with a rough equivalent to 1 day, 3 day, 5 day - but they should not be thought of in that way when doing the estimation - you just try to group them into general categories of S, M, L.

Over time you can measure what your outputs are, and then you can get average times for each of your sizes and these then can be used by those interested in trying to predict the future.

So estimation or planning sessions should really be about discovery, learning and then putting that problem into some general category. Importantly don't be afraid to turn to the group and say - Hang on, I don't think we can even categorise this, it requires better understanding, more learning - lets take it out of this process and do some more focused work.

And even if you don't change a single thing about the way you work just think a little about the way you work. Remind yourself that estimation is just that, estimation and just as important remind those who use your estimations.

The Failure of the Skunkworks

2011-01-29T09:25:00.000Z

Something I have seen as a quite common approach in an organisation is to start a skunkworks project, to get something critical off the ground. I have been involved in several in the course my career, with varying success.

Typically these initiatives are actually done for the wrong reasons, but that’s not what I am going to talk about here.

What I have noticed is that more often than not, a skunkworks project is started; a small, skilled and motivated team is split off, it is isolated and allowed to focus on a specific issue. Now this (sometimes) results in fantastic results, the team is hugely productive and makes brilliant in-roads into the issue. Their progress reports are astounding and it promises great things.

Then they reach that critical juncture and the team is brought back into the fold. Unfortunately then more often than not things start to unravel. The business views the final outcome as if not a failure, then a partial one.

Looking at what has happened we can find a lot of potential ‘reasons’ for this:

The skunkworks team just failed, they misreported their progress, they built false hope.
The process of creating a skunkworks team created too much animosity, those left behind were resentful and they then either consciously or unconsciously sabotaged the final integration of the solution.
The team produced something which whilst it solved the problem, couldn’t realistically be integrated into the ‘real’ system, it was either too advanced or couldn’t be understood by the mere mortals expected to continue or support the work.

Regardless of these and many other possible reasons I actually think there is something very different at work here – I think the perceptions of the above don’t take into account something fundamental.

When you split off your skunk-works team, you removed them from your ‘System’. You gave them freedom, they probably self-organised, they were smart and motivated. In their work they created their own little system in a bubble, one that suited their needs and as a result their productivity exploded. They probably even ignored the remnants of the system they were supposed to use (timesheets... nah can’t be bothered waste of time ...) and got away with it, because they were the skunkworks team, they were special.

Then you dragged them out of that bubble.

Pulled back into the ‘System’, and you crushed them, that bubble dissolved, productivity plummeted and problems rocketed – you re-imposed those constraints which held them back in the first place.

Chances are some of that team you pampered by allowing that microcosm walked out the door, they saw what was possible and had it cruelly ripped out from under them. The ‘System’ defeated them.

You see the huge gains in productivity were not down to isolating them, giving them their own coffee machine or supplying them with pizza late into the evenings. Sure they loved that stuff, but what they really loved was the freedom – the new system they created specifically to meet their goals.

By System, I don't mean process (Waterfall, SCRUM, etc) , I don't mean the Feng Shui of the office, I don't mean free pizza - the system encompasses all of those things and much more. In turn the system produces a culture, and that culture plays a part in driving the behaviour of the people who work there.

Its an extremely complex system with positive and negative feedback loops and a million variables. More people, more variables, more loops.

I don’t have a magical solution if you are considering or have an existing a skunkworks project, every situation is different. But I do suggest you step back, and really think about why there needs to be a one. I am betting that stepping back, thinking about the way you work, the system and its culture, that it will help you find alternative paths to solve your problems.

The mysterious case of the slow Alpha

2010-11-27T19:40:00.005Z

One of the questions I like to ask candidates during the interview process is, "What is the most memorable problem you have had to solve". Often I am disappointed in the answers I get and thats with candidates with 10+ years of experience who I would have expected to come across some gnarly problem they had to solve.

Thinking about the question from my own experience I have several answers, but one in particular is now over 10 years old I thought was worth documenting before it fades from my memory.

In the late 90's and early 00's I worked in a startup in the telephony space, or more correctly the computer telephony integration space. It was a niche that was growing fast and one we were doing very well in. We had secured a contract with DEC (just as they merged with Compaq and since merged with HP) to port our software onto DECUnix (later Tru64). We had been running on Intel based systems for sometime using SCOUnix, Linux and Solaris. I had joined the team just after the port had been done, we had a couple of systems live and a couple running in labs.

There were some concerns raised on performance during integration testing with a client, and we embarked on some benchmarking. We pitted a brand new DS-20 on loan from DEC against our latest and greatest PIII-500 - in theory the DS-20 should have left the P-III in its dust. But it didn't, it didn't even come close to matching it, in fact in some tests it came in at almost 50% slower.

There was much gnashing of teeth. We were relying on the higher performance of the DS-20 to drive our SS7/ISUP solution into new markets at a scale our competitiors couldn't match.

I was still pretty early in my carreer, wet behind the ears if you like. I set about looking at what changes had been made during the port, and after a few days could not find any significant changes, those which were made I did some micro-benchmarks on and couldn't find any issue. We tried tuning the DECUnix kernel, we spoke to Compaq and adjusted everything to what was considered the optimum for our situation, no real change.

Tempers were rising, client was threating to pull the plug, my boss and I were spending hours on end, stepping through code, testing, tuning all to no avail. Eventually our CTO rang Compaq and gave them a barrage about how **** their Alpha's were and how we were so dissappointed with their performance we were going to drop our port and withdraw from the contract. After some too'ing and fro'ing Compaq flew an engineer out to take a look at our benchmarks and to see if they could help us out.

So the engineer arrived, I walked him through our tests, ran the benchmark and showed him the outputs. He looked over the tuning we'd done and agreed we'd done all the right things. I asked if he'd like to see the code, he said not yet, instead lets profile it. Profile? I'd not really thought of doing that on the Alpha, we'd profiled on the intel, ok so how can we do that on DECUnix? So he spent a few minutes showing me how to do it, I did a fresh build with profiling enabled, and then we re-ran the benchmark with profiling on. As expected it was slower still, but we generated a profile.

Right lets take a look.

The engineer looks at the results and within 30 seconds he said, 'Any reason you are opening /dev/null 11 million times during that test?'....huh, WTF?!

I looked through the application code, no references to '/dev/null' - it must be from somewhere else. Some find, strings and grep later we found the culprit, a shared library with no much in it, but it wasn't a DEC library. A search through our CVS repository, and I found a small module written by my predecessor who left a few days after I started - just after he'd finished setting up the new server.

In that module was a single function snprintf ... and a lot of comments abusing DEC for not including this function in their standard libraries......

And in that function was the reference to /dev/null

He had implemented his own snprintf, and thought he'd done a smart job. He was using the fact that fprintf would return the length of what was outputted, so in order to determine the length of the input he would open /dev/null, fprintf it, get the length, close /dev/null and then use that length to determine if the input needed truncation before calling sprintf. Oh crap.

Some quick hacking, a fresh build, and bang our Alpha was now screaming along just under twice as fast as the Intel.

Red faced, I thanked the engineer, and slunk off to explain to the CTO where the problem was. A day or so later we shipped a new version to our client using the GNU library instead of our insane one.

Contracts saved, faces red and a lot learnt.

QCon London 2010 Wrapup

2010-03-15T18:45:00.005Z

Last week I attended QCon London 2010. I was lucky enough to get to two days of tutorials as well as the full three days of the conference.

First up I attended the brilliant Rest In Practice tutorial run by Jim Webber and Ian Robinson. The tutorial is based around their forthcoming book with Savas Parastatidis which now in rough cuts

The tutorial covered a lot of ground, and was well presented. By using Hypermedia you can create very rich services and this tutorial provided a great insight on how and why you should do just that. It provided a good overview of what REST really is using the Leonard Richardson maturity model, and how most services that tout being RESTful are not actually fully RESTful (though they may indeed still be of value).

On day 2 I attended the Resource Oriented Architecture tutorial run by Brian Sletten. Before going in I wondered if I'd made a mistake selecting two REST tutorials - but it turned out to be a great choice, it was complementary to the day before.

One of the most memorable parts of the tutorial was the 'musical interlude' where Brian did a brilliant job of explaining derivative works, and how public data opens up a world of possibilities.

The conference proper kicked off on Wednesday. The opening key note by Bob Martin, whilst at times a bit preachy, was entertaining and re-enforced some good development practices.

As usual it was difficult at times to chose which talk to attend, which became irritating when you hit a slot where there were no talks that stood out. This year was also much busier, with rooms becoming crowded and at times overcrowded. I felt that it wasn't quite as well organised this year, things didn't flow quite as well and some of the speakers seemed ill prepared.

However many of the talks were of high standard, and very enjoyable.

Unlike last year I am not going to try and summarise every talk I went to, I just don't have the time these days. But some stand outs for me included:

The Counter-Intuitive Web by Ian Robinson

One of the best talks of QCon, Ian delved into the differences between Operation and Resource centric architectures - some overlap with Mondays tutorial, but was in many ways complimentary.

He talked about how to map Domains on to Resources and how HTTP can be used appropriately as the Application Transfer Protocol and not just treated as a Transport Protocol. One of the best tag-lines of the talk was 'Intention should be written on the wire'. The idea that contracts can be expressed simply in the combination of link relations, media types, HTTP idioms and an entry point URI.

Simplicity - The Way of the Unusual Architect by Dan North

Dan is a great speaker, entertaining and engaging. The talk started with a comical biblical style story with one of my favourite lines being - "And Maven brought forth a Plague of Apache Commons, and there was a flood of all the Libraries of the Internet as a judgement upon the people"

The talk was about how we over complicate things, gradually over time we introduce further and further complexity. Dan presented some techniques to try and address these tendencies.

How to do over 100K contended complex business transactions per second at less than 1ms latency - Dave Farley & Martin Thompson

Ok, the only thing I can say is - this hurt my brain. Their solution was computer engineering elegance, hitting the sweet spot at every point in the chain - really forced me to think about how to approach problems and really reminded me of the early days of PC's where every bit of performance was eeked out of the system.

The Freedom to Fail

2009-10-18T10:58:00.006+01:00

Recently I had a discussion with a colleague about his strong 'fear of failure'. We spoke about it and hopefully I put his fears into context, and I will try and build on that here.

Now the Fear of Failure is important, it helps keep us motivated, it keeps us diligent and for some people its is truly what drives them. However in our industry it also stifles growth, innovation and promotes over conservative behaviour.

I am quite lucky in my current role, I lead a team of engineers who have explicitly been given 'The Freedom to Fail' - it is actually in the team mission statement, how neat is that?

Now that doesn’t mean its ok to sit back relax and just let projects crash on to the rocks. There are some projects that we should not ever fail on, those where a solution is self-evident, where we have more than adequate resources and where risks are contained.

But what it does mean is, we can be more experimental in our approach, we can truly try and learn whilst tackling the challenge, and we can try things that normally would be considered risky. We get a chance to learn and grow.

In an organisation that in many ways is very conservative, and in a current climate that promotes that conservatism, we break the mold.

Now that doesn't mean we do things purely because we like the idea of them (well I lie, we occasionally do but as a learning exercise). But we question the traditional approaches, we ask ourselves if the "typical" solution is appropriate, if the corporate template is appropriate and seek out other approaches which may indeed prove to be a better fit.

Perhaps luckily our choices have proved successful more often than not. But things do go wrong, we on occasion do fail and wind up either having to restart, rework or abandon a piece of work.

It is not wrong to fail, however it is very very wrong to hide, deny or ignore failure. That failure may only be partial or minor, but it is still a failure and needs to be addressed.

So here is what is important...

Identify Failure

Have a way of identifying failure (and subsequently success). There are many useful techniques that can be applied here, including time boxing, acceptance testing and "done means done".

Time boxing can be vitally important. Estimated work often overruns its a fact of life, we are poor at estimating and often something simple takes a lot longer than expected. Using a time-box you can identify that you have failed, and you need help, need to change approach or need to give up.

Acceptance testing lets you prove that you haven't failed. In many cases I take a pessimistic view of success, if its not passing all the tests its a fail.

Done means done" - all too often you can ask if feature X finished and you'll get the reply "oh yes that's done but this part needs tweaking". Sorry pal, if that part needs tweaking its not done, now put that card back into "in progress" and finish it properly before declaring it done.

Understand the consequences of Failure

It is important to understand what the consequence of failure is at every point. There are varying magnitudes to this, both at a project planning "will we get this done on time" point of view and real life consequences of failure of the project after release.<

It is important to know what can be done (if anything) to mitigate the failure. Can a different approach be taken, can another team undertake the work, can we just ditch it?

Structure what you do to ensure any failure occurs as early as possible in the process. Failing quickly increases the chances that we can mitigate the failure or if that is not possible then stop work before more resources are wasted.<

Understand the Root Cause of the Failure

All too often the root cause of a failure is not fully understood. "We think its because the connection pool was used up and that caused ..." - wait up here, stop, "We think?"

In some cases the root cause is obvious and at other times it requires investigation (root cause analysis) however something that is often over looked is what I consider "people problems".

Dealing with people is hard, especially in an industry known for having anti-social tendencies. Quite often people without the correct skills, experience or support are given problems they are just not equipped to handle - and thats a management failure.

Disseminate the Failure

This is the bit a lot of people shy away from. I have a great deal of respect for someone who can stand up in a meeting and say, "yeah I screwed that up, I need help to find a way to resolve it".

In my experience the failure to discuss failure is worse than the original failure - it dooms others to repeat it. And covering up a failure is far far worse than than actually failing, perpetuating a lie that something works when it doesn't.

Honesty is indeed the best policy. Of course I don't mean that companies should publicly air their dirty laundry but rather within your team (and hopefully within your entire company) there should be open, honest and frank communication.

Final Whitterings

Does your team fear failure to the extent they don't think outside what they think is expected of them? Worse yet do they fear it to the extent they will actively hide or deny it? Are you holding your team back through your own fears?

Building a sense of trust within a team is vital, how you address the issue of failure I think is vital.

If you work in an environment where failure is expensive then try and create a space where it is acceptable. Code Dojo's, Hack Days and similar events give people a chance to flex their mental muscles, their creativity and lets them grow, it gives them that freedom to fail.

Can't afford to fail? Can't afford not to learn from failure.

Learning Scala when Sleep Deprived

2009-08-25T13:27:00.011+01:00

So I've not blogged anything in a while, usual excuse of being busy, the not so usual excuse of becoming a Dad for the second time. So as well as being busy I am somewhat sleep deprived.
I've been doing some stuff with Scala and Lift, and hit a really basic issue that took me a moment (in my current state that equals about an hour) to understand.
I had a couple of classes and was trying to access a field in one class from another ... well I thought it was a field.

package foo

class A( name: String ) { }

class B() {
  val myA = new A("foobar")
  println( myA.name )
}

When I compiled I got this error:

error: value name is not a member of foo.A

Huh? Fields are public by default in Scala, what is going on here. It really had me stumped, my addled brain failed to find any reason for this, and a google search found nothing really useful.

So I changed my code to...

class A( name: String ) {
  def getName = { name }
}

class B() {
  val myA = new A("foobar")
  println( myA.getName )
}

Now I was calling a method which simply returned name, this compiled and ran fine. So why did my original code not work? And why if I first thought that perhaps constructor parameter generated fields were private, but why did I not get an accessibility error?

Then I had a thought, how does the compiler know if 'name' is a val or a var? What if I explicitly state so?

class A( val name: String ) { }

class B() {
  val myA = new A("foobar")
  println( myA.name )
}

This worked. Interestingly using 'var' worked as well - I can only assume that the compiler is getting confused if its not specifying it and so silently doesn't create a field. Even though you can still reference it internally in the class it has no external representation.

Perhaps a case for either a warning or better error granularity?

Or perhaps just a case for me to get more sleep?

In order to ...

2009-06-04T19:36:00.003+01:00

Lately I've been doing some story generation for a few projects, and I've really been focusing on the idea of putting value first.

Its a subtle, but powerful idea. I was introduced to it by Dan North during his tutorial on BDD at QCon London 2009 and it really clicked.

I've been doing quite a bit of process work involving Kanban, which really focuses on value - so it has a kind of synergy (pardon me while I wash my mouth out for using management speak). Liz Keogh also wrote a good article for InfoQ recently on Kanban and BDD.

So the format we've chosen is the same Liz describes in her article:

In order to ...
As a ...
I want ...

And its working really well so far. By putting the value first it makes more sense to our domain experts, who typically being non-technical in most cases got stuck on "As a", and "I want". And we've found that we keep our scope better contained as people stop thinking about "I want" and concentrate on the end value of "In order to".

So if you are working on stories I would really encourage you to try this one simple change. Hopefully you'll see the benefits.

IVY

2009-05-28T21:05:00.003+01:00

One of the things I have been using lately is Ivy.

In the past I have had to use Maven on various projects - and must admit, I hated it. Whilst it might be really trendy at the moment to use "convention over configuration" to me doesn't sit well in a build system. Several times what I've needed to do hasn't fit into the Maven model and I really dislike having to "wedge things in" to appease a tool.

But one thing I did like was the dependency management.

Well Ivy gives you this, without all the other stuff you don't need. You use it from Ant, its clean and easy to manage dependencies, and it leverages the existing Maven repositories.

And something else neat is the ability to bootstrap it from Ant, so you don't have to rely on someone having Ivy already installed, you can install it from within your Ant build and use it on the fly.

What I do is this, create an install-ivy.xml with the following contents:

<project ivy="antlib:org.apache.ivy.ant">
  <property name="ivy.install.version" value="2.1.0-rc1">
  <property name="ivy.jar.dir" value="${basedir}/.ivy">
  <property name="ivy.jar.file" value="${ivy.jar.dir}/ivy.jar">

  <target name="download-ivy" unless="skip.download">
    <mkdir dir="${ivy.jar.dir}">
    <get src="http://repo1.maven.org/maven2/org/apache/ivy/ivy/${ivy.install.version}/ivy-${ivy.install.version}.jar" dest="${ivy.jar.file}" usetimestamp="true"/>
  </target>

  <target name="install-ivy" depends="download-ivy">
    <path id="ivy.lib.path">
      <fileset dir="${ivy.jar.dir}" includes="*.jar">
    </path>
    <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpathref="ivy.lib.path">
  </target>

  <target name="clean-ivy">
    <delete dir="${ivy.jar.dir}">
  </target>

  <target name="clean-ivy-cache" depends="install-ivy">
    <ivy:cleancache>
  </target>

 </project>

Then in the build.xml I add an:

    <import file="install-ivy.xml">

Then have a target like:

  <target name="resolve" depends="prepare,install-ivy">
    <ivy:retrieve>
  </target>

Then its just a matter of setting up your ivy.xml with your dependencies, which could be something simple like:

<ivy-module version="2.0">
  <info organisation="org.apache" module="hello-ivy">
  <dependencies>
    <dependency org="com.thoughtworks.xstream" name="xstream" rev="1.3.1">
    <dependency org="commons-lang" name="commons-lang" rev="2.4">
    <dependency org="junit" name="junit" rev="4.5">
    <dependency org="org.mockito" name="mockito-all" rev="1.7">
  </dependencies>
</ivy-module>

One of the good things I find is that, I work on a variety of machines, and I don't need to install ivy, my build scripts do it for me as and when.

An Unexpected(?) Advantage of TDD

2009-05-18T20:21:00.002+01:00

So there are a lot of advantages to using TDD, they are well documented and there are plenty of people who extol its virtues.

One of them however has resonated with me of late. And that is how it effectively builds a living document of the implemented functionality of your system.

I am working on a personal project, and the time I can spend on it is quite sporadic. Between work and a young family, time to code for fun is quite broken - 10 minutes here, an hour there, a weeks break, another 10 minutes and so forth. It doesn't make for much of a flow.

But what I have found is, by sticking to TDD and by using test names that are informative that I have built a living document of what I've implemented. So when I come back to the code even for a moment I know where I am. Run the tests, and green, there is all my functionality.

Better yet, the breaks force me to revisit my naming, and I actively refactor the code (including the tests) to try and make it more readable.

I've also found that by invoking the "if its not green at then end of the session then ditch it" rule quite effective too. Now you might say that if you only have five minutes to work on it then chances are you're not going to get to green. And a lot of the time its true - but before I stop, I roll back to green.

Now at first that seems counter productive, as the tests will show where its broken and you know where to come back to - but in practice I have found that it actually tends to mislead me, particularly if I have been away from it for a while. So better to return to a clean slate. Then what I find is that my subconscious tends to chew on the problem for a while, and when I come back to it I find my flow much quicker.

So its just another reason to use TDD and descriptive test names.

Building your Village

2009-05-08T21:12:00.003+01:00

On Friday the 24th of April I ran an Agile day for my team and others at the BBC, I managed to get some good speakers on a variety of Agile and Development topics - it turned out to be a really good day. The official blog post on it is almost ready to go up.

During the day my mind drifted on to the importance of interactions.

I tend to see a project team as a village, the village should be as self contained as possible - the butcher, the baker and the candlestick maker. Everyone in a village has their part to play in its success. And in a strong village everyone pulls together when times are tough, they come to a fellow villagers aid in times of trouble.

So whilst we may have a team made of specific roles and skill sets the team is greater than the sum of its parts. So as a project suffers challenges the team should pull together to meet those challenges.

I have worked in teams that do this naturally, and teams where its just never quite happened and teams where it never had a chance.

The teams where it worked had a few things in common:

Experience - in all cases the team was experienced, both in software engineering and the problem domain. Each team had at least one problem domain expert, someone who could answer or anticipate the questions.

Attitude - in all cases the team had a strong professional attitude. Not only did the teams strive to minimise technical debt, but all had generally strong work practices. That sense of professionalism actually was strong enough that the team was able to "stand up" to management at those times when it was really needed.

Motivation - in all cases the team was motivated. And typically the motivation came from the enjoyment of working with a great team doing great stuff. Something that is often overlooked.

Now if you had these things you could argue that pretty much any process would work, and I would agree - not all these projects were what we would call Agile today. But they were Agile in the sense of putting People and Interactions, before Process and Tools.

So if your team is not pulling together in troubled times, look at the three points above. Does your team have experience, attitude and motivation? If it doesn't you need to look at how to address it. And you don't need to be the team leader or manager to address it - its amazing what influence any member of the team can have if they are prepared to talk to their team members.

So how do you build your village?

Firstly, interaction. How well does the team communicate? All the members of the team need to feel that they can speak up, and all the members of the team have an obligation to listen. Do you all sit together? If you can't sit together do you use the same watercooler or coffee machine? How often do you have lunch or go to the pub as a team?

I wrote a little while ago about development being a social activity - this follows on that great teams subconsciously understand this. Now by social I don't mean going out and partying, but I mean interacting.

So why do I think interaction on these levels is so important?

Well a team that communicates well will be more fun to work with, it will improve team motivation, it will lead to improved attitude (management by example or culture) and it will encourage everyone to use their experience, encourage people to improve or better themselves.

Its not going to solve the problem of having domain experts, java experts, jvm tuning experts - but you in the end you get to be an expert by doing and learning. Ideally by learning from another expert, but that doesn't have to be from the team, it could be from conferences, blogs, mailing lists, geek nights, books, consultants or from another team in the organisation. Building the right village will encourage the team to learn, to rise to the challenge.

URIPlay is now OpenSource

2009-04-22T21:10:00.001+01:00

A little late but URIplay is now available as OpenSource, the BBC RAD Team has made the code available under the Apache 2.0 license

URIplay is a name service for media content, giving each file a URI and a simple description. Think of it as DNS for media. The service is not intended to be used directly by viewers. Like DNS, it works behind the scenes to make things easier.

You can see a simple demo here: http://open.bbc.co.uk/rad/projects/uriplay

The code has been picked up and is already evolving here: http://uriplay.org

Why do we assume they know what they are doing?

2009-04-14T20:10:00.002+01:00

I had a interesting thought whilst talking to someone about the whole 'Twitter using Scala' thing.

Now I have no vested interest in either Ruby or Scala. I like both languages, I tend to lean toward Scala as I am a staticly typed kind of guy, but in the end I like to think I am a pragmatist and will use the right tool for the job.

But what got me after looking at all the blog posts, all the tweets and email threads was, why do we assume that they (Twitter) know what they are really doing? Now I am by no means having a go a Twitter, I think it was a brilliant idea, it has captured the imagination of many and I see plenty of people that bag it still updating their status every hour or more.

But coming up with a great idea for a service doesn't mean you instantly know the right way to implement it - especially if its never been done before.

So Twitter, its kind of like a global IRC service, its web-scale. Did the guys that came up with the idea have any experience with building web-scale services?

(Hint: buy Jim Webber a few beers if you want to know how to play nicely at web scales and how to leverage its nature to your advantage).

So as the debate over Ruby-vs-Scala continues, which I think is irrelevant, the real questions on the learning Twitter has gone through linger on. Have they learnt enough about the domain (web scale messaging)? Has the underlying Architecture improved or does it remain the same regardless of the language of implementation?

My guess (and it is a guess) is the Architecture remains the same. In that case they may have undertaken what has been done many times before and just reimplemented it but in another language. They may have even done it right, managed to get a clean break from some technical debt that was dragging them under, but in my experience, that never quite happens either. So you are still faced with technical debt (possibly even more) as well as a sub-optimal architecture.

Having said all of that, it is worth keeping in mind that Twitter is still here today and is still increasing in popularity despite the "Fail Whales". And I will continue to use it.

And in reality this doesn't just go for Twitter, theirs just happens to be more public than most.

So a reminder (to myself as much as anything) - Before reaching out for that implementation solution, that golden hammer - its worth stepping back, taking a break and looking at the architecture and even the requirements and asking is that really valid, does it really have to be 'that' way? Are we making life more difficult for ourselves.

Fluent Java Interface for generating HTML

2009-04-14T13:22:00.003+01:00

Am very happy to see that Alistair Jones (of How Big is my Potato fame) has produced a neat Fluent Interface for generating HTML from Java: http://code.google.com/p/hypirinha/wiki/Tutorial

Software Development is a Social Activity

2009-04-13T08:51:00.004+01:00

Something I have been pondering for a while, is "Why is software so hard?". I know it is hard, I have spent a long time working in the industry, I have seen it fail - thankfully I have seen it succeed (albeit often "challenged") more regularly. I have read a lot of books and talked to a lot of people, but still I felt something was missing.

Computers are logical devices, they follow your instructions exactly without the slightest deviation - however being human we are not logical, and we do not execute or think naturally in the same precise logical manner. Our brains are non-linear, we make jumps, we shift through varying layers of abstraction and we rarely stay in that logical state for too long.

But that still doesn't explain why software is so hard. We also have the tools to focus us and guide us when it comes to writing the code. Compilers, unit tests, analysis tools, exploratory testing as well as a myriad of processes attempt to focus us on the problem at hand. So why is it still so hard?

I think one of the leading reasons is this, Software Development is a Social Activity and those that work in software development are stereo-typically anti-social in nature. Yes it is a stereo-type, but it does have a firm foundation in the truth. By in large, we are not social butterflies, we tend to prefer the company of machines to that of humans, the company of our problem domain to that of other people. I hear cries of disagreement from some quarters - yes it is not the rule, there are some highly sociable developers out there.

However time and time again, I have seen software projects flounder because of the lack of both information and 'social' bandwidth. By social bandwidth I mean the social interactions between members of the team. Good social bandwidth from my experience requires a very high information bandwidth - face to face communication is good example. This is somewhat difficult to explain, and something I feel that I am only beginning to grasp.
Email has perhaps one of the narrowest social bandwidths of all. You have no real sense of the other persons state of mind, their current emotion or their current physical situation. Its even narrower when one or even both parties don't recognise that fact. We have all witnessed email storms caused by an off the cuff comment at one time or other.

An area of high social bandwidth is the office coffee machine or water cooler. Where people tend to meet, talk and exchange idea's purely because of the constructed environment - you would not choose to go and chat to a random person every hour or so just for the hell of it (... or maybe you would, maybe you are one of those social butterflies).

Location is a key factor here, as we congregate together to work, we construct a village, a place of community. Now communities don't always get on, and certainly are not all best friends - but they do tend to build a spirit over time, especially where interaction is unavoidable.

The act of developing a piece of software is the act of a community, it requires a varied team. Just as the village requires the butcher, the baker and the candle-stick maker. The most productive teams recognise and encourage that variation in skills, and tend naturally strive to increase their social bandwidth to spread those skills and experience.

Too many times I have heard a senior manager, who having walked into the office for 30 seconds in a month, complain that a team is not working hard enough - when I know what they heard was actually the sound of productivity, the sound of high information and social bandwidth. A team with a strong bond, can have a chat or a laugh - a positive working environment increases productivity, it makes people want to come into the office.

Now I am not saying the team needs to hold hands and sing campfire songs, but rather the team needs interaction. It needs developers, testers, business analysts, project managers and customers to interact. And the team should foster that interaction both professionally and socially. I've seen project managers turn down a pub lunch request which would have done wonders in building a fragile relationship into something more substantial.

It is important that this social drive come from within the team, and not an enforced company social event, one which will leave some feeling awkward or even resentful, and actually discourage some from attending at all. A sense of professionalism can provide that balance, and it is more than achievable with the right team.

Continuous Integration - Moving beyond the build

2009-03-31T21:40:00.001+01:00

So everyone (well I really hope everyone) is now using Continuous Integration of some form or other. However a lot of CI I have seen is purely build, or build and unit-test. This is really under utilising a potentially great resource.

These days developer time is expensive compared to processor cycles. So we need to maximise developer time spent on productive things. Automation can help us do this, and automation starts with CI.

So what can we do to make this a reality?

Firstly consider what tasks do you do repetitively that take up time? Deployments, Integration Tests, Acceptance Tests? What tasks would you do more often but they take too much time? Static Analysis, Performance Tests, Endurance Tests?

But, I hear you say, this takes too much time, we cannot spare that right now.

Well I disagree - the time that you spend now will pay you back later.

In anything from a few minutes to a few hours you can set up a nightly build, analysis, test cycle could save a whole lot of trouble and which can be incrementally improved to make life easier.

I plan to write some blog posts on how to do this, just need to find the time.

QCon London 2009 Conference Day 3

2009-03-13T22:00:00.001Z

Today was the 3rd and final day of QCon London this year.

Pimp my Architecture with Dan North

Dan has a great presenting style, and during this presentation he emphasised the importance of the team Shaman, the team story-teller - and I think he proved himself to be one of the best through this presentation.

I really enjoyed this presentation, as it emphasised the importance of the oral tradition within a development team. And I do feel that is important, the best teams I have worked in have always had this, it meant that new team members came up to speed quickly on the quirks and history of a project, and really helped build the team.

I also enjoyed his 'Shanty Town' pattern - as opposed to the 'big ball of mud'. The truth is as he rightly points out, you just can't rush in and bulldoze, the people have to live somewhere - but you can work to improve it, piece by piece. You can build scaffolding or temporary architectures in the process of rebuilding the town.

Too much to convey it all or even a fraction, but I must say if you get the chance to hear Dan speak it is worth it, his energy and passion are contagious.

Thoughts on the Generic vs the Specifc Tradeoff with Stefan Tilkov

Stefan's talk was great, in particular my favourite snippet from his talk was:

The wise architect ...
question: *
answer: It depends.

And its so true.

Summing up his talk is difficult, as the two sides tend to cancel out, i.e. it depends. But it was good and he covered a lot of good specific versus generic examples.

Strategic Design with Eric Evans

Another really good talk, Eric covered some interesting perspectives on dealing with legacy systems, and the approach you could take in a pragmatic and progressive way. And along the way discussed various issues you need to deal with.

The strategic design comes from, making sure the most important parts of the system are well designed.

Some good points he made ..

One irresponsible programmer will keep any 5 top-notch programmers busy cleaning up after them.
The quality of the code is equiv to the second worst programmer on the team. (Not the worst, he is watched like a hawk)
Remember, the core domain is what makes the system worth writing.
Heroes work in the core domain (they make the system features the business needs and wants).

Why do irresponsible programmers so often become heros? Because they don't waste their time making other people more productive, they don't clean up other peoples mess, but they do produce new capabilities no matter the impact.

Five Considerations for Software Architects with Kevlin Henney

Well today turned into a treat, Kevlin was another good presenter, with a lot of information and some good techniques covered. The presentation was based around a visit to a pub where he and a couple of others (poor memory, cannot remember their names) and tried to distill 5 things every architect should take into mind. These distilled down to:

Economy - economy in code, in design and in architecture
Visibility - understanding not just the end product but how it came about
Spacing - the interesting thing is the space between the components not necessarily the components themelves
Symmetry - makes things easier to understand and maintain
Emergence - understand the emergent properties of the problem and the solution

Finally I attended the interview track for Stefan's interview of Ian Robinson and Jim Webber about REST and using the Web as an Integration platform. It was good to hear their views on REST and the pragmatic use of the web for machine-machine integration - I guess good because I agree with them.

A good day, and a good conference, I really enjoyed it - my brain is still soaking in a lot of what I heard - just need to let the R-mode chew on it a while :)

QCon London 2009 Conference Day 2

2009-03-12T09:31:00.008Z

Today was the 2nd day of the conference proper.

Transforming Software with Web as a Platform - Dion Hinchcliffe

This was the morning keynote. It was in a well presented style, however was primarily a 'state of the nation' kind of talk. Some of the interesting points for me included how building social networks around your site/product is very effective and beneficial for various reasons, including an almost off-loading some of the product support and testing into that community.

Dion covered how the biggest technology companies were hardware, then progressed to software and now were data companies. His mention of how the desktop was one of the last areas where we have "commercial software" I think this was slightly off, I think Open Source has made considerable progress and is making more all the time in that area - I think it is the Mobile desktop which is now the commercial software sweet spot - thanks to the likes of the iTunes AppStore.

He also covered cloud computing and how he felt we were at risk of falling into a very small number of large players that we trust with our data.

Agility: Possibilities at a Personal Level with Linda Rising

I chose to go to Linda's presentation as a couple of people had recommended listening to her speak if I ever get the chance. And I must say that I am glad that I did, not just for the content, but also for her relaxed and engaging presentation style.

So it was mainly a discussion about drugs, well caffeine, the worlds drug of choice, nothing else comes close with apparently around 90% of the worlds adult population consuming it daily in some form or other.

The talk (I feel) really centered around the Industrial Age and how we are still behaving and living in a manner consistent with that time. A lot of interesting facts, I knew about the drinking of beer due to bacteria in the water, but never put it together that accurate time pieces and tea/coffee became readily available at the same time - the time of the industrial revolution. And the process of making tea/coffee of course required boiling the water. So people needed to wake up for their shift at the factory, and instead of beer they would have tea/coffee which because of the boiled water was safe.

Linda spoke a lot about caffeine and its effects on the human body, some I knew, some I didn't - I certainly wasn't aware of the half-life of the effect on metabolism.

Another interesting snippet was that 80% of us think we are above average.

During the question time there was talk about adjusting work rythmns to suit peoples make up - morning person, evening person - indeed adjusting entire teams to work in this manner.

Forging ahead - Scaling the BBC into Web 2.0 with Dirk-Willem van Gulik

This talk I was looking forward to for a variety of reasons. But I must admit I felt let down, Dirk spoke very quickly and the presentation style was not great. A lot of detail was covered, but the interesting points I felt were lost.

The challenge was not just the technical one of moving from static webpages to a fully dynamic personalised Web 2.0 platform - but also the challenge of doing it in a timely and agile manner. There was coverage on how human and logistics do not scale, but no real description of how that is overcome.

There was a good point about how the BBC was funded and the impact that makes, being TV License funded more web traffic means essentially each request has to become cheaper to be done on budget.

Something that really made me think is the statement that "Content is our core business - not IT" which left me wondering why it all hadn't been outsourced, most organisations focus on their core business and bring in the experts to do the things they don't consider core.

The rest of the afternoon got whitewashed, as I bumped into Jim Webber and got talking about various things we were joined by Ian Robinson for a short while and then Martin Fowler. I was also then introduced to Paul Downey and had a interesting conversation about various things including Standards and Standardisation, Open Source development, company culture and the workings of the web.

I missed a session about Functional Programming I was keen on, but am glad I can catch it up online.

Another good day, looking forward to tomorrow.

QCon London 2009 Conference Day 1

2009-03-11T15:43:00.006Z

Today was the first day of the conference proper at QCon London 2009, an interesting day all around and again a lot to chose from.

Opening Key Note by Tony Hoare

A contrast on Computer Science and Software Engineering - for me it was very interesting as I've often internally and occasionally externally debated as to whether what I do is really Engineering. I hold a degree in Electronic Engineering, but I have always worked in Systems or Software. A lot of what we do can be considered craftsmanship - and it was good to hear Tony say that this is part of the early life-cycle of an Engineering discipline.

Too much info to cram into this and I know it will be up on the QCon site in the near future if you are interested.

SOA with Ian Robinson

Really good insight into starting a big project especially in an already running system - good to see SOA talked about as SOA (the approach) and not SOA as a vendor pitch to buy their latest golden hammer.

Scala with Jonas Boner
Very interesting JVM based language with very good integration with Java. Provided an elegant mix of OO and Functional programming - and so provided a clean path to move towards more functional models. The session went through some of the language features, but one of the most impressive things shown was a demo of a web chat system written in 30 lines of code.

A really nice concept of Traits which could be added at either class or instance levels - so much information that I have no hope of conveying it. But has inspired me to take a closer look.

Clojure with Rick Hickey

A "Lisp" Functional Language that runs on the JVM. Was a very interesting presentation and the language has some really interesting features including how it dealt with concurrency. Again too much information to try and convey.

Building for Performance and Scalability with Alois Reitbauer

Outlined the cause of many performance issues, and very high level ways of detecting these issues - but there was nothing new here, and no practical demonstrations of the methods, and no hard data.

This is an area I am quite familiar with having spent time 'in the trenches' dealing with these problems, and having spent time developing ways to help mitigate them. So I was disappointed - I felt there is so much more on this subject, particularly how easily steps can me made to introduce ways and means of detecting performance deviations early on in the process. Big expensive tool-sets are not necessary to begin this process (or at all in my view).

REST-based Integration Architecture for a Financial Business Service Phillip Ghadir

This talk interested me as it was about moving from WS-* to REST and using the likes of ATOM/ATOMpub.

Must admit that brain fry had begun to set by now and I didn't take it all in. But was interesting to hear Phillip's view on - use REST at the interface boundary but don't try and use it internally as the latency cascade will bite you.

Unfortunately I missed the evening Key Note - but no doubt it will be available online soon.

QCon London 2009 Tutorials Day 2

2009-03-10T18:49:00.004Z

So today was my second day of tutorials at QCon London 2009 - another tough choice as to what to go to - as much as I wanted to learn more about Erlang I chose instead to attend Advanced Ruby and Behaviour Driven Development.

Advanced Ruby by Sam Aaron

I thought I was being a bit ambitious selecting to go to a tutorial on Advanced Ruby given my Ruby knowledge is quite limited, i.e. only just above zero. But it turned out to be very good and quite insightful. It was the first time I had been to a Language tutorial that didn't in anyway get bogged down in syntax.

The tutorial covered the Language from different approaches of coding, such as prototype, class based and functional. It was useful to learn how Ruby handled inheritance, closures, creating methods on existing classes, detecting missing methods etc. It also provided to me a chance to reflect on the other languages I use and understand them better.

Must admit I felt a bit for Sam having Ola Bini in the room - but he handled it really well. Ola did provide some further insight into Ruby's internals which was good.

Behaviour Driven Design by Dan North

This was a session I was really looking forward to, and Dan didn't disappoint, his passion and energy for this stuff really shows through. It was interesting to hear his views on how Agile is progressing beyond things like XP - in a way its become more pragmatic and seasoned I guess, and it really got me thinking how to step beyond TDD or as he rightly pointed out "Code By Example".

There was a serious amount of information in the session and it was very engaging. It was great to go back to some reasoning as to why we work well in small teams (our brains are hardwired to work well in groups up to about 10 people - pre-history family group size perhaps?). It was also good to discuss things like why it is hard to introduce change, and how to reduce the likelyhood of falling back into 'the old ways'.

Trying to summarise Dan is almost impossible and its worth hearing him talk on the subject if you get the chance.

It was also refreshing for someone to say about their own work, 'this is really just the beginning, lets see where we go from here'.

All in all another good day, tomorrow the conference proper starts.

QCon London 2009 Tutorials Day 1

2009-03-09T19:47:00.007Z

So today was the first day of the QCon tutorials, and was a good day... we'll leave aside what TFL did to my oyster card, I don't want to get into a rant - lets just say if you buy Oyster online, don't try and get a travelcard and prepay on the same card.

GET Connected: A Tutorial on Web-based Integration

First tutorial was with Jim Webber on using the Web as an Integration platform. Jim was in his usual self-deprecating mood, keeping things light hearted with the ocassional rant. (I've known Jim a while, and I nicknamed him Mr.Angry, whilst he nicknamed me Mr.Grumpy - but thats for another post)

It was actually a great session, my only disappointment was that it wasn't a full-day as there was certainly enough material for it, Jim had to skim through the last third to the most important points. The presentation for me though was especially good, as it gave me a chance to reflect on what it really means to leverage the web - it also gave me some ideas on solving current problems by using ATOMpub to handle some eventing stuff.

Domain-Driven Design - a complete example in a current technology stack

This session wasn't as good in my view, the choice for the example I thought was pretty good, but I didn't find it engaging. It lacked something, and the code examples I felt could do with some more work. It didn't help that the first thing brought up in the IDE was a WSDL, something that sends a shiver down my spine - especially when you've just had a session on using the web and talking about hypermedia aware restful interfaces.

However one of the best parts of the day was at lunch being introduced to Michael Nygard author of one of my favourite books, ReleaseIT. A very interesting guy though I must admit I felt stumped on what to talk to him about aside from "love your book by the way". Thanks for the "FanBoy" comment Jim - at least I only bought the book once ;)

I met Ola Bini - one of the Guys behind JRuby and Ioke - interesting guy to say the least. Also met Sam Aaron, another Ruby guy and who will be running the Advanced Ruby session I am going to tomorrow. Had an interesting discussion with him and Michael about Ruby, SmallTalk, MetaClasses, Secret Classes and some other stuff that went over my head with my limited knowledge of Ruby and even more limited knowledge of SmallTalk.

A good day, and am looking forward to tomorrow.

Test Driven ...

2009-03-07T20:26:00.011Z

In the past I have been guilty of not testing enough (who hasn't). I went through a phase of code first and test as late as possible, then code first and write tests - the old idea of unit testing. However this always ends up in testing getting left behind as time pressures push the demand towards just getting the code done.

I was then introduced to Test Driven Development, it made sense, but I struggled with it. Even when I thought I was doing it right I would find as time pressures increased that it would slip, and it wouldn't be long before I was back in the old style of unit testing.

I used to grapple with design, either in my head or on paper, then write a test against that design and then try and code that design. The problem being, that the design was never quite right. Some requirement was always missed or not considered, or even worse non-required features would slip in because they seemed like a good idea. Going back and changing that design, meant going back and changing the tests.

The other problem was that the tests were always biased towards that vision instead of testing the requirements, increasing testing became a matter of just adding more and more tests, a lot of them testing the same thing and of dubious value.

So as a result things never flowed, there was always a discontinuity - and that leads to turbulence and drag. Drag means wasting energy, there in the temptation to ditch that perceived drag, the testing, instead of the real drag, a bad design.

I then had the pleasure of working on a project with some really focused guys (Jim, Pat, Alistair amongst others), those who knew what TDD really was, and how to really do it, how to use the tools in the right ways. To them it was just natural, everything flowed, there was no drag.

The 'click' was the realisation was not that I didn't know how to write unit tests, nor was it that the test had to be written before the code, but the test had to be written before the design had crystallised.

Now, I am not saying that the secret is no design. Indeed design is very important, but now I tend to do enough design when and as its needed. The final design emerges as I test, code, test, refactor, test.

ssh'ing rsync out of proxy hell...

2009-03-05T10:15:00.007Z

I had a situation last week where I needed to install macports but due to local proxy behaviour I could not get rsync to work.

However I did have ssh working through the proxy, and I had access to an external system which didn't have the proxy issues... so I created a tunnel.

Now this has to be done as root, as it needs to create a listening port below 1024.

First I changed the /opt/local/etc/macports/sources.conf to point to the localhost
#rsync://rsync.macports.org/release/ports/ [default] rsync://localhost/release/ports/ [default]

Then I created a tunnel through my external system using ssh
ssh -L 873:rsync.macports.org:873 user@my-external-system.com

On that ssh connection I did the following to keep that session active:
ping -i 60 localhost

Now macports quite happily rsync'd away oblivious to the plumbing I had to do. Gotta love ssh.

Using Groovy and GUnit to test Java 'privately'

2009-03-02T21:01:00.026Z

Normally if I am writing a class and find I need access to a private field in order to test correctly then I take it as a smell - something not quite right with the design. But every now and then I fail to find a way to change the design without corrupting its intent.

Now I know you can use reflection in Java to access private fields, but its a bit clunky. But recently I have been reading about Groovy, and found that it gives you full access to all your Java objects, but allows you to ignore private specifiers.
So as an example, say I have a counter class, one which allows users to get the next value of the counter, but cannot provide anyway of altering the counter for that would invalidate its purpose.

package net.usersource.example;

import java.util.concurrent.atomic.AtomicInteger;

public class Counter {
   private final AtomicInteger counter;

   public Counter() {
       counter = new AtomicInteger(0);
   }

   public int getNext() {
       return ensuringValueDoesNotRollIntoNegativeValues(counter.incrementAndGet());
   }

   private int ensuringValueDoesNotRollIntoNegativeValues(int value) {
       if( value < 0 ) {
            if( counter.compareAndSet(value, 0) ) {
                return 0;
            }
            else {
                return getNext();
            }
        }
        return value;
    }
}

Now, how do I test this without calling getNext() until it reaches Integer.MAX_VALUE? Do you know just how big MAX_VALUE really is, and how long that will take? Thats not really an option - so I need somehow to change the underlying counter.

Now I could do that with reflection, but I can do it much simpler using Groovy (which if you look closely, especially the way I have written it, looks like java) ...

package net.usersource.example

import org.junit.Test;
import org.junit.Assert;
import java.lang.Integer;

public class CounterTest {
    @Test
    public void verifyFirstObtainedCounterValueIsOne() {
        Counter c = new Counter();
        Assert.assertEquals( c.getNext(), 1 );
    }

    @Test
    public void verifyThatAtMaxIntTheNextValueIsZero() {
        Counter c = new Counter();
        c.counter.set(Integer.MAX_VALUE-1);
        Assert.assertEquals( Integer.MAX_VALUE, c.getNext() );
        Assert.assertEquals( 0, c.getNext() );
    }
}

This was TDD'd so I only knew the solution thanks to the test.

Unfortunately I have found some issues with the Eclipse Groovy Plugin - its auto-completion is a bit zealous in someways (wants to try and match everything bringing your machine to a crawl) and ignorant in others (not matching in imports). Also the GUnit results were reporting exceptions the line above the line with the issue - which caused me some confusion initially, and I need to investigate that further.

But to me this is a neat way of testing Java code, by stepping out of Java to something that makes it easier, whilst staying on the JVM.

Is JUnit Named Right?

2009-02-28T16:06:00.005Z

When JUnit first appeared, it was perfectly named, however as time has moved on so has its use.

Jim Webber's recent blog post (here) made a very valid point about when a Unit Test is not really a Unit Test. Test and Behavioural Driven Design push JUnit into an area that is not adequately described as unit testing. Driving a design through testing in this manner means you aren't (at least initially) writing unit tests.

I've also used JUnit in the past to do things which are decidedly not unit testing, its a useful framework to control other sorts of testing. And on more than one occasion have had to explain to a member of management why I used a free unit testing framework instead of using their big expensive shiny testing metaframework thingy.

I am not suggesting that Kent and Eric should rush out and re-release their work under a different name. It was (to me at least) interesting to reflect on naming, and how things progress over time, so once perfectly named entities may gradually slip to the almost perfect. And it is also a reminder that not quite perfect risks gradually slipping to the not relevant at all.

What the perfect name would be today, I don't know, I think the name is still perfect enough.

Why the term "Sprint" Irks Me

2009-02-20T15:03:00.007Z

Apart from being inherently lazy, there are other reasons why the term "Sprint" irks me....

Agile development is supposed to be about sustainability - in fact the Principles of the Agile Manifesto states ...

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

The term Sprint in my mind does not describe a constant pace, it conjures up that desperate dash to the finish line, giving it your all in one go.

The argument is however that "sprints are short", however they are repeated. Do we expect the Olympic 100m runners to do a 100m dash, and then do it all over again, and again, and again? All in a relatively short time frame, after all how long do you rest between week long Sprints?

The other argument is "its just a name", however software development is in a large part about naming. Writing clean code is more often than not seeking the right name, something that describes intent - so why should we strive for anything less in naming our processes and their elements?

So until I find something better I will continue to prefer iteration as to me it better describes the intent of repeated time-boxed work.