Enter the Pipeline .... exploring Scala and Akka

Sometime ago I did some interesting work with Akka.  These days I don't code everyday, most of what I do is in my own time and I wanted to play with the latest Akka (1.1) and see if it was still as fun as it was.

I wanted something simple to connect to Twitter, read the stream, chop it up message by message, pass them to a JSON parsing actor, which it turn would pass them on to further actors for more processing. The concept can be extended in many ways.

The codebase as it stands is quite small, which is really thanks to standing on the shoulders of giants:

 Scala 2.9.0
 Akka 1.1
 SJSON
 Signpost OAuth library
 Apache HTTP Client 4.1
 ScalaTest
 Mockito
 SBT

A few comments about the code...

Firstly its not production code, it is an exploration of various things. I will outline some of what I think are the most interesting parts of the code. See the bottom of this post where to find the code.

Using Either and Option

One of the things that has always created confusion is the use of exceptions for things that are not really exceptional cases. Too often we see exceptions used for alternate path selection, often the less travelled path but still a valid path.

If you expect failure, which in a networked environment we should, then you should not be using exceptions, its not an exceptional case, its an expected one. However how do you represent this in code? Thankfully Scala gives you Either. Your methods can use Either to indicate that this method will return either an error or a result.



In the above we have defined a trait with a connect method that can return either an Error if something goes wrong, or a BufferedReader if all goes well.

Similarly you can use Option to indicate that a method may return a value or not. This is much more meaningful than returning null's as you cannot be sure what a null really means, whereas an Option explicitly states its Optional.

And its possible to combine the two...



So in the above we are saying, either expect a ReadError if there is a problem otherwise you will 'optionally' get a string.

Using 'become' to manange connection state

Akka provides the ability to change the receive method of an Actor using 'become', this can be very useful to manage the state of an actor based on the messages it receives. In this case we have two "states", active and inActive. The messages that are valid in each state are different. The actor can move between these states using 'become'.




Testing

I have written the tests with ScalaTest, in a Given-Then-When style. One of the challenges of testing an intergration point is verifying expected behaviour without actually having to connect to a real service. Using AkkaTestKit and Mockito I have attempted to test as close to the integration as possible without actually building a mock service.

Testing logging events

Akka provides an eventing service which is an alternative to traditional logging. Eventing decouples the action of logging from the source of the event, this is very useful in many respects including testing. It is possible to hook into the event service through the test, in this case I used an actor that simply places the events received on the queue. The test can then verify expected events are on the queue in the expected timeframe.

Whilst it is possible to get the Events sent to the TestKit testactor I found this tended to made the test confusing as essentially two streams of messages were being merged which in turn made the test logic quite complex. By using a separate queue it made it clearer (in my view) to understand.

Below you can see an example using Mockito as well as the event queue.



How to use it

Warning: whilst using this code for testing and experimentation does not (according to my understanding) break any Twitter T&C's you should satisfy yourself should you decide to use this code or alternatively base any future code upon it.

You will need to register an application with Twitter for which you will need to have a Twitter account, login and then go to https://dev.twitter.com/apps

Here you can register a new application, then make note of the consumerKey and consumerSecret. You will also need to make note of your Access Token (oauth_token) and Access Token Secret (oauth_token_secret).

Then you need to clone the source code and built it using sbt (http://code.google.com/p/simple-build-tool/)

The following transcript shows how to use it.



If you want to play with the code in Intellij, just run 'sbt idea' on the command line. The code is tagged with 'blog_post' at the point in time of writing this blog post.