Real-world functional Scala

If Java is a get shit done™​ language then Scala is a get shit done fast™​ language. The reason is that the signal to noise ratio is very high and that concise code is easier to reason about which makes it less error prone.

When understanding Scala it is sometimes important to have a look at some real-world examples — especially when it comes to functional programming. When I first started using Scala I had a hard time understanding some of the constructs used by other people. In this blog post I want do explain some of them. The first example is a classic one. You want to debug some structure and output all elements which have a condition set.

Printing Elements of a Collection

You will find this in every first-steps guide for Scala. What I like about it is the fact that you have only little amount of code to write if you want to inspect a certain structure.

list filter { _.condition } foreach println

Flattening a Collection

Convert a collection of collections to a flat collection. For instance you have a List<List<String>> in Java and want to get a List<String> instead.

final List<String> newList = new LinkedList<String>();
for(final List<String> innerList : oldList) {

A lot of code for a common operation. Probably the reason why the Google Guava library supports Iterables.concat(oldList) to get the same result. So how is this done in Scala?

val newList = oldList.flatten

A simple operation which is very powerful. Let me give you another example. Suppose you have a List[Option[T]] for some T. Before understanding Scala very well and before reading this excellent article about monads I once wrote shameful code like this.

val listOfOptions: List[Option[Int]] = 
  List(Some(1), None, Some(3), None, Some(5))

// don't do this!
val listOfDefinedValues: List[Int] = 
  listOfOptions filter { _.isDefined } map { _.get }

println(listOfDefinedValues) //List(1, 3, 5)

When encountering such a construct an alarm should raise in your head, telling you there must be a better way to do this. flatten to the rescue. println(listOfOptions.flatten) would do the job.

val listOfOptions: List[Option[Int]] =
  List(Some(1), None, None, Some(2))

println(listOfOptions.flatten) //List(1, 2)

Using flatMap for good

Although flatMap might sound esoteric at first it is a very helpful operation. So first of all you know there is the map operation which is extremely useful in a sense like this.

"hello world" map { _.toUpper }

So what is the deal with flatMap? It would not work in this case and the compiler would raise a type error. That is because a collection is expected as the result.

flatMap makes a lot of sense when you have a collection and need to pass every element into a function that would then again return a collection.

val list = List(1,2,3,4)

def intToListOfChars(i: Int) = 

val listOfListOfChar: List[List[Char]] =
  list map intToListOfChars

As you can see calling map with intToListOfChars creates a list of lists and we could later flatten that list agin. Or we simply use flatMap which does all of this in one step.

val listOfChars = list flatMap intToListOfChars

Still this example is pretty awkward. But imagine you have a method like trimToOption(value: String): Option[String] which trims a string and returns None if the string is empty or null (for whatever reason).

If you write a webapplication which expects user input you are probably working with a lot of Option[T] values (or Box[T] in case of Lift). For example like this:

val description: Option[String] = 

Now you also want to trim that description. Here are a couple of ways on how to do it using trimToOption.

// not so nice
val trimmed1 = trimToOption(description getOrElse "")

// ugly as hell
val trimmed2 = description match {
  case Some(desc) => trimToOption(desc)
  case None => None

// kind of ugly
val trimmed3 = (description map trimToOption).flatten

// yey! :)
val trimmed4 = description flatMap trimToOption 

So you see that flatMap is actually a pretty usable function.

Passing a Tuple to a Function

Sometimes you have to pass a tuple to a function and instead of the tuple you would like to pass each value of the tuple as a parameter.

val listOfStrings =
  List("Hello", "World")

val withIndex = listOfStrings.zipWithIndex

println(withIndex) //List((Hello,0), (World,1)))

So now imagine you have a method like this:

def printStringWithIndex(value: String, index: Int) =
  println(index+": "+value)

In this case withIndex foreach printStringWithIndex would not work because we pass a tuple to that method. Scala has a nice built-in method called tupled which takes a function and converts it to one accepting a tuple.

withIndex foreach (printStringWithIndex _).tupled

Using the Compiler

Imagine you have a class with a type parameter like CustomList[T] and now you want to have a method which is implemented only for some kind of T. Obviously one way would be to use traits and mix them in on demand. But it gets more and more complex.

Fortunately we can let the compiler proof the evidence of a type at a position in our code.

class Foo[T](bar: T) {
  // only allow this method if T is a String
  def length()(implicit proof: T =:= String) = 

val foo1 = new Foo(123) //works just fine
foo1.length //does not compile.

val foo2 = new Foo("hello")
foo2.length //5

This is great because it allows us to introduce phantom types and to write less code for certain structures. For instance we have a Connector[A] in Audiotool. This connector accepts connections to other connectors. It is either an input or an output and it transmits an abstract signal. You can only connect an output with an input. A Connector accepts only connections of its own type and virtual connections. This means the following example would be uber-cool code. Unfortunately we did not write Audiotool in Scala ;)

class Connector[A] {
  add[B](conn: Conn[A, B])(implicit proof: A =:= B)
    { ... }

  add(connection: VirtualConn[_, _])
    { ... }

Note: This is in fact not a real example since it would not work. You could only connect an output with another output but I think you might get the idea and it is easier to follow.


Functional programming might seem weird. Constructs like =:= or flatMap are awkward at first but make complete sense. Using Scala and picking some of the functional idioms which you think are right and well understood in your team can increase productivity and lead to less bugs.

Go channels with Scala

Since I had to spend 10h in a train this weekend I was looking at some food for thought. For a long time I had a couple of slides about the Go language on my machine and so I started reading.

Go comes with a different concepts for synchronization. Channels are a construct baked into the language. They allow you to communicate between different go-routines etc. Basically I stopped reading after I came across the Channels section.

A channel allows you to send and receive data from it. Much like a Scala actor. By default a channel is performing blocking send and receive operations. Sounds pretty much like a blocking queue to me. The syntax is something like the following.

// Create a channel of int
var channel = make(chan int) 

func send() {
  // Send 1 to the channel.
  // This operation blocks ...
  channel <- 1

func receive() {
  // Receive from channel
  // This operation also blocks!
  message := <- channel 
  fmt.Println(message) //1

go send()
go receive()

So ultimately I thought it should be quite easy to achieve something similar using Scala. I think this is an excellent example of the scalable part when it comes to Scala.
Go channels are diveded into reciver and sender channels, with a bidirectional channel type. The same can be achieved using Scala traits very easily. I could also add a couple of nice syntactic sugar like adding a foreach method to a channel which allows you to use it with Scala for comprehensions.

Have a look at this Gist for the complete source code. First of all one part is a little bit unfortunate. You are not able to define <- or <~ as a unary operator in Scala. So I thought I stick with the ! operator which you already know from actors and it is the analog to Go’s <-.

At the end of the file you will find a re-implementation of the Go Producer-Consumer example.

Something really cool is the fact that you can use Scala’s for-comprehensions which automatically exit once a channel is being closed. I think Go allows you do the same by specifying range when iterating over a channel. However I am not sure since I only had a brief look.

So is this useful? I do not know since I would probably stick with Scala actors for most of my multi-threading needs but it is still a great example of Scala’s flexibility as a language.

Endless scrolling with Lift

For the upcoming Audiotool 2.0 launch in collaboration with Burn I am developing our new website. The framework choice was quite obvious for me since I love Scala and really like Lift.

Lift is very nice because of its excellent Ajax and Comet support. My main problem getting started with Lift was the lack of documentation and real world examples. E.g. if you want a custom 404 page without a redirect your only source of information is a thread in the mailing list. Lift also comes with its own terminology and a lot of custom DSLs.

Cricitcs aside: I am absolutely happy that Lift exists. You simply have to start using it. The documentation has also improved a lot. Simply Lift and Exploring Lift are available for free and the Wiki contains a lot of useful information too.

I want to give you a simple example of why I think that Lift is simply awesome and what sold me to the framework. You probably know those endless scrolling pages.

Let’s see how to develop something similar with Lift and Scala. First of all we have to prepare a little template.

<div class="lift:surround?with=default;at=content">
  <div class="lift:endless">

That is all the HTML we need. In fact we need even less but if you are going for designer friendly templates you could extend it with your usual page chrome.

There are two calls to Scala code in this template. lift:surround will trigger the SurroundSnippet and insert the current template into the default template at the content position.

Our duty is now the implementation of lift:endless. Lift comes with a dynamic convention-over-configuration lookup for the so-called snippets. I am not a fan of COC so I will stick to the manual mechanism of binding endless to a snippet. Lift offers the DispatchSnippet in this case.

You will have to add this line to Boot.scala:

LiftRules.snippetDispatch append Map("endless" -> EndlessSnippet)

It tells Lift to invoke the dispatch method of the EndlessSnippet object whenever a snippet named endless is encountered.

So the only missing piece is the implementationof the EndlessSnippet and this is where Lift really shines.

import net.liftweb.common._
import net.liftweb.util._
import net.liftweb.http._
import net.liftweb.util.Helpers._
import net.liftweb.http.js.jquery.JqJsCmds
import net.liftweb.http.js.JsCmds
import net.liftweb.http.js.JE
import scala.xml.{NodeSeq, Unparsed}

object EndlessSnippet extends DispatchSnippet with Loggable {
  def dispatch = {
    // If you want to execute render only when "" is called
    // you would match on "xyz" here.
    case _ => render 

  def render =  {
    // Generate a random identifier
    val containerId = nextFuncName

    // The page which we are currently looking at.
    var page = 0

    // Method to be called by JavaScript.
    def process(in: Any) = in match {
      case id: String if id == containerId =>
        page += 1
        logger.debug("Generating page "+page+".")

        // Generate new markup and append it to the list.
        JqJsCmds.AppendHtml(id, getItems(page))
      case _ =>
        logger.error("Illegal data has been sent.")

    // Here we marry the template with our code.
    // "*" is a CSS selector that matches anything and #> binds it to the content
    // we specify.
    // The most interesting part here is the JavaScript  which we bind to the process
    // closure. Lift does all the magic for us and calls the server-side method once the
    // user scrolls to the bottom of the page.
    "*" #> (<ul id={containerId}>{getItems(page)}</ul> ++ JsCmds.Script(JE.JsRaw(
  if($(window).scrollTop() == $(document).height() - $(window).height()) {"""+
    SHtml.jsonCall(containerId, process)._2.toJsCmd+"""

  def getItems(offset: Int): NodeSeq = {
    val count = 100
    val start = offset * count
    val end = from + count

    // We generate a couple of <li> elements here.
    for(i <- start until end) yield {

This little amount of code really shows the power of Lift. Simply write a closure and bind it to your JavaScript. Lift does all the plumbing for you. Then you only have to implement some logic on the server. In most cases you do not even have to write any JavaScript since Lift offers Ajax links, forms etc. already. SHtml.a(...) generates a link which will trigger a function on the server for instance.

But please beware: I am not sure if SHtml.jsonCall is actually the way how you should do this. There might be a better way of calling a server-side method via JavaScript.

There is of course more to Lift. Comet is where things get really cool and the record framework seems interesting as well. Hopefully this little excursion into Scala and Lift could quicken your appetite. Since I am usually not a frontend fan this made it interesting and fun again.

I will try to post a complete project in the next days as well. Before starting with the Audiotool website I wrote some test cases that might help other people getting started.

Software Transactional Memory and Audiotool

I have tweeted a while ago about the fact that Audiotool does heavily rely on a so called software transactional memory. In this blog post I want to discuss the implementation of our STM and why it is useful.

Wikipedia defines STM as:

In computer science, software transactional memory (STM) is a concurrency control mechanism analogous to database transactions for controlling access to shared memory in concurrent computing.

First of all when reading this you might think that an STM is not necessary since ActionScript does not support true concurrency and also the shared-nothing worker proposal will not allow you concurrent memory access. While this holds true you may not forget that concurrency becomes an issue when you have just access to a single core but on multiple machines.

A lesser known fact about Audiotool is also that we have a built-in system for real time collaboration since its beginning. Due to conceptional issues and other prioritized features we were never able to roll it out as a full feature but the technology base is there.

When it comes to multiplayer gaming and network latency you will always have to figure out a system which allows you to share a certain state across multiple machines. In some games the state is synchronized across a certain server. An example is an FPS like Quake3. The Audiotool is unlike Quake3 not a FPS and different conditions have to be met. And this is the reason why I have choosen to implement an STM with very optimistic concurrency control.

The basic idea of my system is to use atomic transactions with commit and revert operations. Those are handled by a transaction manager which performs the basic tasks of an STM. That means: execute all transactions until an error occurs, roll back and try again. Each transaction in the Audiotool contains a guard. The guard defines whether or not one should still execute the transaction.

Here is an example of why a guard is important and how all this stuff works. You have two Audiotool instances running that share the same state. We call them A and B. A and B are connected via P2P no an arbitrary network. This means exchanging messages introduces latency and we cannot guarantee that A is at the same state of B.

So what happens? t0 is committed by A, serialized and sent to B who is committing the same transaction. The same applies to t1. t2 is initiated by B, serialized and sent to A. Everything works fine until B commits t4. While t4 is being transferred to A, A created t5 and committed it. This means A never saw t4 when it committed t5 and B would continue with t6 ignoring t5. A will actually receive t4 after he commited t5 so the system will revert t5, commit t4, commit t5 if the guard does not prevent it, and both continue at t6.

To explain the guard: Imagine the t5 transaction includes modifications on a device in the Audiotool that is deleted by t4. In that case t5 will never be executed.

So what does this imply? You can guess that each modification of the Audiotool state is in fact handled by a transaction and we carefully have to pick the guards. E.g. if an editor is referenced by a transaction it is part of the guard. However this introduces a second issue: references. Sharing references of objects across a network is a tricky task. I am using “boxes” to solve this issue. A box can hold a value and it is guaranteed that it will hold the same value at the same state across the network. A box can be serialized and will be shared between commit and revert states. This means if tn creates an object which is referenced by tn+1 both will use the same box. This also means that I can safely revert tn+1 and then tn and commit them without having to worry that both still reference the same object. And this even works across the network. And in fact we can now also identify whether or not one transaction would conflict with a different one which is a very important condition for the actual implementation.

Right now you might have figured out that a transaction in the Audiotool has the following properties:

  • It is serializable so it can be transferred across the network.
  • All object references must be stored via “boxes”.
  • A guard is used to determine whether or not to commit a transaction.
  • Transactions can be committed and reverted.

A lot of stuff one has to reason about. But although we do not support live collaboration at the moment we are using the STM. Why? The reason is quite simple. The STM gives us history for free. Since everything is serializable via a very lightweight protocol we can take the last 500 transactions for instance and store them as a compressed ByteArray in memory instead of each as a single object with all its references.

Another very nice option is testing. A serializable history means that one simply has to press a super secret shortcut to dump the whole history and send it to the responsible developer who can reason about every single step that happened which ultimately led to the error. In development mode I usually commit, revert and commit each transaction. Then I also serialize and deserialize them and do the whole thing again.

Since transactions are known by the system we can also write a fuzzer that executes lots of crazy stuff. Sort of Android’s monkey mode. And ultimately I personally hope that we will get to the point where we will enable live collaboration in the Audiotool of course.

The last feature I implemented is the automatic transformation of arbitrary transactions to a compound transaction. You can guess what this feature actually means for the user :). In fact it is very complex and I implemented it with the STM using only a couple lines of code (of course that is just half the story …).

If you are interested in writing your own multi user application I think this is a very nice approach since collisions do not happen very often from my experience and the optimistic concurrency in a P2P environment achieves great results in terms of overall latency and feeling. The Google Wave protocol and their operational transform is also an interesting read.

Understanding Scala Implicits

Many languages make use of implicit conversions. Scala is the only language I know of which makes implicit definitions a feature.

It is very easy to understand an implicit conversion since most programming languages do this already for you. Here are two examples (AS3/Java):

trace("Implicitly converted: "+1)
System.out.println("Implicitly converted: "+1);

Read More »