Author Archive for joa

Project Hiddenwood

This years FOTB was special. At the end of my session I showed a sneak preview of project Hiddenwood. I demonstrated complete playback of Audiotool tracks on stage — in a browser. Now that does not sound too special…

But then again, the playback was done using JavaScript only and calculated in realtime.

Audiotool is a complex piece of software so you might ask how one could torture themselves by implementing it in JavaScript? We didn’t. Instead we started building our own vision of a cross-platform application framework a couple of months ago.

Introducing project Hiddenwood.

Hiddenwood is a collection of libraries and tools specifically designed to support different devices and platforms. The core libraries are the “driver layer” and always platform-specific with a platform-independent interface.
On top of that we provide a basic layer of libraries like our UI system, animation framework or managed collections which guarantee 0% garbage collection activity and have been battle-tested in Audiotool.

The framework is all about speed and consistency. The rendering pipeline is optimized for OpenGL and although we offer something similar to Flash’s display list a lot of features are not available because they would compromise the speed.

Speaking about speed: we are always interested in staying as native as possible on our target platform. So for the browser we emit JavaScript, for Android you will get the full DalvikVM performance and for the desktop you will get JVM performance. This approach has also another very important aspect. If you want to go platform-specific for certain features you can do that.
For instance if we want to render Audiotool songs on the server using a fork-join pool for our audio calculation this is possible and might not make sense on an Android device.

You write Java code and the supported platforms are native desktop applications, Android (minimum requirements are Gingerbread and OpenGL ES 2.0) and modern browsers. Now for browsers we even go one step further and support multiple options. That means if WebGL is not available we simply fallback to a normal canvas based render-engine. The same applies to some of the Android drivers.

iOS is of course important as well and we are actively researching the best option that will give us the most flexibility and performance.

We are currently working on two real applications built with Hiddenwood. So far it is a real pleasure to enjoy quick build times and simply test what you want on the desktop with great debugging capabilities. When you are ready you can try the same app on Android or in the browser — which might take a little bit longer to compile.

Because we see Hiddenwood as an application framework there are a lot of goodies built-in like a sprite-sheet based class generator. Think Image mixerBackground = Textures.mixer.background(); where mixer was the folder name and background the name of the file.

We believe that as a developer you really do not care about what kind of technology you are using and just want a great result. We also think that you should be able to reuse platform-independent code across multiple projects. However we do not want to take power away from the developer because if you know what you are doing: go for it.

Of course we are not the only ones with this idea. Nicolas Cannasse saw the signs years ago and invented haXe which gives you a comparable experience and Google released playN a couple of weeks ago which takes a similar approach (and requires another 25 committers :P).

But when we started Hiddenwood we wanted the Java tooling experience and playN was not public at that time. We also think that a game engine is not what you want to use for all kinds of applications. So we like to be able to give people the freedom to build their own game engine on top of Hiddenwood — and calculate physics in a different thread peut-être.
Speaking about threading: the only possible solution that works across all platforms is a shared-nothing architecture which we put in place. However if you write platform specific code you can use of course everything the platform offers and a lot of the Hiddenwood core libraries like the network- or cache-layer make use of multiple threads.

In the end what makes Hiddenwood special in my opinion is that we do not believe in write once run anywhere because that just does not make sense. The essence and philosophy behind Hiddenwood is to write platform-agnostic code using kickass-libraries and being able to reuse that. Audiotool on a tablet would look completely different from Audiotool running in a browser. And Audiotool on iOS would probably be also a little bit different from Audiotool on an Android device because there are simply different paradigms you should respect.

I hope that we can share more information with you soon. With the news of mobile Flash Player being deprecated and the ongoing demand for cross-platform development we have exciting times ahead of us. I am also super excited about the (beautiful <3) applications which we are going to release in the not so distant future.

First look at Dart

In this post I want to give you a short introduction to Google’s new language called Dart.
Continue reading ‘First look at Dart’

Real-world functional Scala

If Java is a get shit done™​ language then Scala is a get shit done fast™​ language. The reason is that the signal to noise ratio is very high and that concise code is easier to reason about which makes it less error prone.

When understanding Scala it is sometimes important to have a look at some real-world examples — especially when it comes to functional programming. When I first started using Scala I had a hard time understanding some of the constructs used by other people. In this blog post I want do explain some of them. The first example is a classic one. You want to debug some structure and output all elements which have a condition set.

Printing Elements of a Collection

You will find this in every first-steps guide for Scala. What I like about it is the fact that you have only little amount of code to write if you want to inspect a certain structure.

list filter { _.condition } foreach println

Flattening a Collection

Convert a collection of collections to a flat collection. For instance you have a List<List<String>> in Java and want to get a List<String> instead.

final List<String> newList = new LinkedList<String>();
for(final List<String> innerList : oldList) {
  newList.addAll(innerList);
}

A lot of code for a common operation. Probably the reason why the Google Guava library supports Iterables.concat(oldList) to get the same result. So how is this done in Scala?

val newList = oldList.flatten

A simple operation which is very powerful. Let me give you another example. Suppose you have a List[Option[T]] for some T. Before understanding Scala very well and before reading this excellent article about monads I once wrote shameful code like this.

val listOfOptions: List[Option[Int]] =
  List(Some(1), None, Some(3), None, Some(5))

// don't do this!
val listOfDefinedValues: List[Int] =
  listOfOptions filter { _.isDefined } map { _.get }

println(listOfDefinedValues) //List(1, 3, 5)

When encountering such a construct an alarm should raise in your head, telling you there must be a better way to do this. flatten to the rescue. println(listOfOptions.flatten) would do the job.

val listOfOptions: List[Option[Int]] =
  List(Some(1), None, None, Some(2))

println(listOfOptions.flatten) //List(1, 2)

Using flatMap for good

Although flatMap might sound esoteric at first it is a very helpful operation. So first of all you know there is the map operation which is extremely useful in a sense like this.

"hello world" map { _.toUpper }

So what is the deal with flatMap? It would not work in this case and the compiler would raise a type error. That is because a collection is expected as the result.

flatMap makes a lot of sense when you have a collection and need to pass every element into a function that would then again return a collection.

val list = List(1,2,3,4)

def intToListOfChars(i: Int) =
  i.toString.toList

val listOfListOfChar: List[List[Char]] =
  list map intToListOfChars

As you can see calling map with intToListOfChars creates a list of lists and we could later flatten that list agin. Or we simply use flatMap which does all of this in one step.

val listOfChars = list flatMap intToListOfChars

Still this example is pretty awkward. But imagine you have a method like trimToOption(value: String): Option[String] which trims a string and returns None if the string is empty or null (for whatever reason).

If you write a webapplication which expects user input you are probably working with a lot of Option[T] values (or Box[T] in case of Lift). For example like this:

val description: Option[String] = extractDescription(request)

Now you also want to trim that description. Here are a couple of ways on how to do it using trimToOption.

// not so nice
val trimmed1 = trimToOption(description getOrElse "")

// ugly as hell
val trimmed2 = description match {
  case Some(desc) => trimToOption(desc)
  case None => None
}

// kind of ugly
val trimmed3 = (description map trimToOption).flatten

// yey! :)
val trimmed4 = description flatMap trimToOption

So you see that flatMap is actually a pretty usable function.

Passing a Tuple to a Function

Sometimes you have to pass a tuple to a function and instead of the tuple you would like to pass each value of the tuple as a parameter.

val listOfStrings =
  List("Hello", "World")

val withIndex = listOfStrings.zipWithIndex

println(withIndex) //List((Hello,0), (World,1)))

So now imagine you have a method like this:

def printStringWithIndex(value: String, index: Int) =
  println(index+": "+value)

In this case withIndex foreach printStringWithIndex would not work because we pass a tuple to that method. Scala has a nice built-in method called tupled which takes a function and converts it to one accepting a tuple.

withIndex foreach (printStringWithIndex _).tupled

Using the Compiler

Imagine you have a class with a type parameter like CustomList[T] and now you want to have a method which is implemented only for some kind of T. Obviously one way would be to use traits and mix them in on demand. But it gets more and more complex.

Fortunately we can let the compiler proof the evidence of a type at a position in our code.

class Foo[T](bar: T) {
  // only allow this method if T is a String
  def length()(implicit proof: T =:= String) =
    bar.length
}

val foo1 = new Foo(123) //works just fine
foo1.length //does not compile.

val foo2 = new Foo("hello")
foo2.length //5

This is great because it allows us to introduce phantom types and to write less code for certain structures. For instance we have a Connector[A] in Audiotool. This connector accepts connections to other connectors. It is either an input or an output and it transmits an abstract signal. You can only connect an output with an input. A Connector accepts only connections of its own type and virtual connections. This means the following example would be uber-cool code. Unfortunately we did not write Audiotool in Scala ;)

class Connector[A] {
  add[B](conn: Conn[A, B])(implicit proof: A =:= B)
    { ... }

  add(connection: VirtualConn[_, _])
    { ... }
}

Note: This is in fact not a real example since it would not work. You could only connect an output with another output but I think you might get the idea and it is easier to follow.

Conclusion

Functional programming might seem weird. Constructs like =:= or flatMap are awkward at first but make complete sense. Using Scala and picking some of the functional idioms which you think are right and well understood in your team can increase productivity and lead to less bugs.

Go channels with Scala

Since I had to spend 10h in a train this weekend I was looking at some food for thought. For a long time I had a couple of slides about the Go language on my machine and so I started reading.

Go comes with a different concepts for synchronization. Channels are a construct baked into the language. They allow you to communicate between different go-routines etc. Basically I stopped reading after I came across the Channels section.

A channel allows you to send and receive data from it. Much like a Scala actor. By default a channel is performing blocking send and receive operations. Sounds pretty much like a blocking queue to me. The syntax is something like the following.

// Create a channel of int
var channel = make(chan int) 

func send() {
  // Send 1 to the channel.
  // This operation blocks ...
  channel <- 1
}

func receive() {
  // Receive from channel
  // This operation also blocks!
  message := <- channel
  fmt.Println(message) //1
}

go send()
go receive()

So ultimately I thought it should be quite easy to achieve something similar using Scala. I think this is an excellent example of the scalable part when it comes to Scala.
Go channels are diveded into reciver and sender channels, with a bidirectional channel type. The same can be achieved using Scala traits very easily. I could also add a couple of nice syntactic sugar like adding a foreach method to a channel which allows you to use it with Scala for comprehensions.

Have a look at this Gist for the complete source code. First of all one part is a little bit unfortunate. You are not able to define <- or <~ as a unary operator in Scala. So I thought I stick with the ! operator which you already know from actors and it is the analog to Go’s <-.

At the end of the file you will find a re-implementation of the Go Producer-Consumer example.

Something really cool is the fact that you can use Scala’s for-comprehensions which automatically exit once a channel is being closed. I think Go allows you do the same by specifying range when iterating over a channel. However I am not sure since I only had a brief look.

So is this useful? I do not know since I would probably stick with Scala actors for most of my multi-threading needs but it is still a great example of Scala’s flexibility as a language.

Endless scrolling with Lift

For the upcoming Audiotool 2.0 launch in collaboration with Burn I am developing our new website. The framework choice was quite obvious for me since I love Scala and really like Lift.

Lift is very nice because of its excellent Ajax and Comet support. My main problem getting started with Lift was the lack of documentation and real world examples. E.g. if you want a custom 404 page without a redirect your only source of information is a thread in the mailing list. Lift also comes with its own terminology and a lot of custom DSLs.

Cricitcs aside: I am absolutely happy that Lift exists. You simply have to start using it. The documentation has also improved a lot. Simply Lift and Exploring Lift are available for free and the Wiki contains a lot of useful information too.

I want to give you a simple example of why I think that Lift is simply awesome and what sold me to the framework. You probably know those endless scrolling pages.

Let’s see how to develop something similar with Lift and Scala. First of all we have to prepare a little template.

<div class="lift:surround?with=default;at=content">
  <div class="lift:endless">
    <ul>
      <li>item0</li>
      <li>item1</li>
      <li>item2</li>
    </ul>
  </div>
</div>

That is all the HTML we need. In fact we need even less but if you are going for designer friendly templates you could extend it with your usual page chrome.

There are two calls to Scala code in this template. lift:surround will trigger the SurroundSnippet and insert the current template into the default template at the content position.

Our duty is now the implementation of lift:endless. Lift comes with a dynamic convention-over-configuration lookup for the so-called snippets. I am not a fan of COC so I will stick to the manual mechanism of binding endless to a snippet. Lift offers the DispatchSnippet in this case.

You will have to add this line to Boot.scala:

LiftRules.snippetDispatch append Map("endless" -> EndlessSnippet)

It tells Lift to invoke the dispatch method of the EndlessSnippet object whenever a snippet named endless is encountered.

So the only missing piece is the implementationof the EndlessSnippet and this is where Lift really shines.

import net.liftweb.common._
import net.liftweb.util._
import net.liftweb.http._
import net.liftweb.util.Helpers._
import net.liftweb.http.js.jquery.JqJsCmds
import net.liftweb.http.js.JsCmds
import net.liftweb.http.js.JE
import scala.xml.{NodeSeq, Unparsed}

object EndlessSnippet extends DispatchSnippet with Loggable {
  def dispatch = {
    // If you want to execute render only when "embed.xyz" is called
    // you would match on "xyz" here.
    case _ => render
  }

  def render =  {
    // Generate a random identifier
    val containerId = nextFuncName

    // The page which we are currently looking at.
    var page = 0

    // Method to be called by JavaScript.
    def process(in: Any) = in match {
      case id: String if id == containerId =>
        page += 1
        logger.debug("Generating page "+page+".")

        // Generate new markup and append it to the list.
        JqJsCmds.AppendHtml(id, getItems(page))
      case _ =>
        logger.error("Illegal data has been sent.")
        JsCmds.Noop
    }

    // Here we marry the template with our code.
    // "*" is a CSS selector that matches anything and #> binds it to the content
    // we specify.
    // The most interesting part here is the JavaScript  which we bind to the process
    // closure. Lift does all the magic for us and calls the server-side method once the
    // user scrolls to the bottom of the page.
    "*" #> (<ul id={containerId}>{getItems(page)}</ul> ++ JsCmds.Script(JE.JsRaw(
"""$(window).scroll(function(){
  if($(window).scrollTop() == $(document).height() - $(window).height()) {"""+
    SHtml.jsonCall(containerId, process)._2.toJsCmd+"""
  }
})""").cmd))
  }

  def getItems(offset: Int): NodeSeq = {
    val count = 100
    val start = offset * count
    val end = from + count

    // We generate a couple of <li> elements here.
    for(i <- start until end) yield {
      <li>{"item"+i}</li>
    }
  }
}

This little amount of code really shows the power of Lift. Simply write a closure and bind it to your JavaScript. Lift does all the plumbing for you. Then you only have to implement some logic on the server. In most cases you do not even have to write any JavaScript since Lift offers Ajax links, forms etc. already. SHtml.a(...) generates a link which will trigger a function on the server for instance.

But please beware: I am not sure if SHtml.jsonCall is actually the way how you should do this. There might be a better way of calling a server-side method via JavaScript.

There is of course more to Lift. Comet is where things get really cool and the record framework seems interesting as well. Hopefully this little excursion into Scala and Lift could quicken your appetite. Since I am usually not a frontend fan this made it interesting and fun again.

I will try to post a complete project in the next days as well. Before starting with the Audiotool website I wrote some test cases that might help other people getting started.

Software Transactional Memory and Audiotool

I have tweeted a while ago about the fact that Audiotool does heavily rely on a so called software transactional memory. In this blog post I want to discuss the implementation of our STM and why it is useful.

Wikipedia defines STM as:

In computer science, software transactional memory (STM) is a concurrency control mechanism analogous to database transactions for controlling access to shared memory in concurrent computing.

First of all when reading this you might think that an STM is not necessary since ActionScript does not support true concurrency and also the shared-nothing worker proposal will not allow you concurrent memory access. While this holds true you may not forget that concurrency becomes an issue when you have just access to a single core but on multiple machines.

A lesser known fact about Audiotool is also that we have a built-in system for real time collaboration since its beginning. Due to conceptional issues and other prioritized features we were never able to roll it out as a full feature but the technology base is there.

When it comes to multiplayer gaming and network latency you will always have to figure out a system which allows you to share a certain state across multiple machines. In some games the state is synchronized across a certain server. An example is an FPS like Quake3. The Audiotool is unlike Quake3 not a FPS and different conditions have to be met. And this is the reason why I have choosen to implement an STM with very optimistic concurrency control.

The basic idea of my system is to use atomic transactions with commit and revert operations. Those are handled by a transaction manager which performs the basic tasks of an STM. That means: execute all transactions until an error occurs, roll back and try again. Each transaction in the Audiotool contains a guard. The guard defines whether or not one should still execute the transaction.

Here is an example of why a guard is important and how all this stuff works. You have two Audiotool instances running that share the same state. We call them A and B. A and B are connected via P2P no an arbitrary network. This means exchanging messages introduces latency and we cannot guarantee that A is at the same state of B.

So what happens? t0 is committed by A, serialized and sent to B who is committing the same transaction. The same applies to t1. t2 is initiated by B, serialized and sent to A. Everything works fine until B commits t4. While t4 is being transferred to A, A created t5 and committed it. This means A never saw t4 when it committed t5 and B would continue with t6 ignoring t5. A will actually receive t4 after he commited t5 so the system will revert t5, commit t4, commit t5 if the guard does not prevent it, and both continue at t6.

To explain the guard: Imagine the t5 transaction includes modifications on a device in the Audiotool that is deleted by t4. In that case t5 will never be executed.

So what does this imply? You can guess that each modification of the Audiotool state is in fact handled by a transaction and we carefully have to pick the guards. E.g. if an editor is referenced by a transaction it is part of the guard. However this introduces a second issue: references. Sharing references of objects across a network is a tricky task. I am using “boxes” to solve this issue. A box can hold a value and it is guaranteed that it will hold the same value at the same state across the network. A box can be serialized and will be shared between commit and revert states. This means if tn creates an object which is referenced by tn+1 both will use the same box. This also means that I can safely revert tn+1 and then tn and commit them without having to worry that both still reference the same object. And this even works across the network. And in fact we can now also identify whether or not one transaction would conflict with a different one which is a very important condition for the actual implementation.

Right now you might have figured out that a transaction in the Audiotool has the following properties:

  • It is serializable so it can be transferred across the network.
  • All object references must be stored via “boxes”.
  • A guard is used to determine whether or not to commit a transaction.
  • Transactions can be committed and reverted.

A lot of stuff one has to reason about. But although we do not support live collaboration at the moment we are using the STM. Why? The reason is quite simple. The STM gives us history for free. Since everything is serializable via a very lightweight protocol we can take the last 500 transactions for instance and store them as a compressed ByteArray in memory instead of each as a single object with all its references.

Another very nice option is testing. A serializable history means that one simply has to press a super secret shortcut to dump the whole history and send it to the responsible developer who can reason about every single step that happened which ultimately led to the error. In development mode I usually commit, revert and commit each transaction. Then I also serialize and deserialize them and do the whole thing again.

Since transactions are known by the system we can also write a fuzzer that executes lots of crazy stuff. Sort of Android’s monkey mode. And ultimately I personally hope that we will get to the point where we will enable live collaboration in the Audiotool of course.

The last feature I implemented is the automatic transformation of arbitrary transactions to a compound transaction. You can guess what this feature actually means for the user :). In fact it is very complex and I implemented it with the STM using only a couple lines of code (of course that is just half the story …).

If you are interested in writing your own multi user application I think this is a very nice approach since collisions do not happen very often from my experience and the optimistic concurrency in a P2P environment achieves great results in terms of overall latency and feeling. The Google Wave protocol and their operational transform is also an interesting read.

Understanding Scala Implicits

Many languages make use of implicit conversions. Scala is the only language I know of which makes implicit definitions a feature.

It is very easy to understand an implicit conversion since most programming languages do this already for you. Here are two examples (AS3/Java):

trace("Implicitly converted: "+1)
System.out.println("Implicitly converted: "+1);

Continue reading ‘Understanding Scala Implicits’

A Year Of Scala

About a year ago I started using Scala. A programming language I felt in love with after all. The one and only programming language that started making me smile when I was happy to see how elegant I can express my ideas.

I have been trying to spread the word about Scala during various Flash conferences and always tried convincing my colleagues and friends to learn this great language. A blog post by Alan Shaw about his experience with Haskell inspired me to write about my journey.

Continue reading ‘A Year Of Scala’

Adobe MAX 2010 Opener

This year I had the honor to be part of the opener show for Adobe MAX 2010. I really appreciate that the people at Adobe working on the keynote took the risk to try this out.Nokia Theatre

Mike Chambers asked me already in April this year if I would like to do it. He had the idea of Erik Natzke generating art, me doing live coding and a DJ playing music. Eric Clark has been a fantastic supporter of Audiotool so I thought it would be really nice to have him do it. Luckily he agreed.
Continue reading ‘Adobe MAX 2010 Opener’

Apparat 1.0 RC7 Is Here

I wanted to make a simple bug fix release for Apparat since I solved some bugs and issues. However there are two additional features that are worth mentioning. First of all Apparat comes now with an ASMifier tool. This allows you to let Apparat generate code from an existing SWF. It will create a file that contains a basic class structure with statements that you can directly use for __asm optimizations. You can also suppress all log output and turn Apparat into quiet mode by specifying -Dapparat.quiet=true.

Also worth mentioning: JITB ships now by default with Apparat. What does it mean? Quadruple-rainbow in your face. You can now run your SWF files with JITB if you follow the five golden steps to pure happiness:

  1. Download and install Scala 2.8.0
  2. Type scala in a terminal, then type println(System getProperty "java.library.path" split java.io.File.pathSeparatorChar mkString "\n") and remember one of the folders
  3. Download LWJGL 2.5 and extract somewhere
  4. Copy the native libraries which are located in lwjgl-2.5/native/YOUR OS/ into one of the folders that the Scala command showed you
  5. Open a terminal, go into the Apparat installation directory and type ./jitb some.swf to launch JITB

Now here is the problem. JITB is not complete and so it will probably fail with your SWF — In fact I can nearly gaurantee you that it will not launch your SWF. I included JITB basically because it already allows you to use GLSL shaders very easy and I know a lot of people would like to play around with that.

However I have created an archive of all files I showed at FOTB besides the files that came from David’s laptop since I do not have them on my machine. You might be interested in checking out Example12.as which is the one that uses GLSL shaders. If you want to compile your own version you will need to link apparat-ersatz.swc as an external library since it contains the ContinuousFileLoader and GLSLShader class. Happy hacking!

Update: Also read Ray tracing with Pixel Bender using JITB by David Laenarts for details on the FOTB demos.